ML14094A165
| ML14094A165 | |
| Person / Time | |
|---|---|
| Site: | Pilgrim |
| Issue date: | 04/02/2014 |
| From: | Entergy Nuclear Generation Co |
| To: | D'Antonio J Operations Branch I |
| Jackson D | |
| Shared Package | |
| ML13295A007 | List: |
| References | |
| TAC U01876 | |
| Download: ML14094A165 (45) | |
Text
Facility Comment I NRC Resolution Exam Item: JPM 56 EDG Monthly Surveillance Issue: JPM step 17 required unloading the EDG prior to opening the output breaker in response to load swings. Unloading the EDG was a critical step.
Facility Comment: In the case of load swings, there is time available to unload the EDG, but the breaker is designed to be opened under full load and no adverse consequences result from this operation. The actual breaker has been opened under full load as a result of a malfunction with no damage. Grade this step as non critical.
NRC Resolution: The examiner reviewed the references provided by the facility and agrees that while unloading the EDG prior to opening the output breaker is preferred, doing so is not critical.
JPM ID:
Title:
Issue:
Discussion:
NRC SIM JPM S6 Emergency Diesel Generator Monthly Surveillance Alternate Path Step 17 of this JPM required the unloading of Emergency Diesel Generator (EDG) A requires the EDG be unloaded following load swings was identified as a critical step within the JPM. Examiners identified that several candidates immediately tripped the EDG output breaker and therefore did not carry out the required actions of this step. The critical nature of this step was called into question. A determination was requested for whether the actions were necessary or the step was erroneously identified as actually critical.
The procedure for performing this JPM is covered by PNPS procedure, 8.9.1, "Emergency Diesel Generator And Associated Emergency Bus Surveillance". The EDG is manually loaded incrementally in power and reactive load to rated maximums of 2600 kW and 1250 kVAR stopping at predetermined plateaus for at least 8 minutes. A reverse unloading process is performed during cooldown following achieving maximum rated for a minimum of one hour.
A simulator malfunction results in power oscillations of +1-300 kW while raising EDG load to the plateau of 1800kW occurs during JPM step 17 that requires the candidate perform corrective action.
Specifically, PNPS 8.9.1 section 6.0 Precaution [10] states:
The surveillance shall be aborted by immediately unloading the EDG and opening diesel generator Circuit Breaker A609 or A509 if any of the following parameters exist:
(a) Oscillations in kVAR greater than 200 when at steady state.
(b) Oscillations in kW greater than 200 when at steady state.
The Task Standard for this JPM states, "The EDG surveillance will be performed IA W procedure 8. 9.1. The diesel will be successfully paralleled to the bus and load increased. When the governor becomes unstable the operator will unload the engine and open its output breaker. There shall be no failure of critical elements".
The requirement to unload the EDG prior to opening the output breaker was discussed with plant engineering personnel. While preferable to immediately tripping the output breaker, load swings do not represent an "emergency" and time exists to minimize the transient on the breaker by lowering load; however were the breaker to be opened immediately, no harm to the breaker would be expected since it is rated to be opened under full load (2600 KW or greater) conditions. Therefore there is not a negative consequence for the EDG to load be lowered prior to opening the output breaker.
Proposed Resolution:
References Included:
Proposed Corrective Action:
Additionally, during performance ofPNPS 8.9.1 in November of2011, a trip of breaker A609 (EDG 'B' output breaker) occurred. EDG 'B' was operating at full load for 45 minutes when a differential relay actuation resuling in automatic tripping of output breaker. The Apparent Cause Evaluation of this event found no adverse consequences to the EDG or output breaker from this transient (CR-PNP-2011-5152 ACE).
Evaluate candidate performance based on step 17 of the JPM being a non-critical step.
- 1.
NRC SIM JPM S6, "Emergency Diesel Generator Monthly Surveillance Alternate Path"
- 2.
PNPS procedure, 8.9.1, "Emergency Diesel Generator And Associated Emergency Bus Surveillance
- 3.
CR-PNP-2011-5152 "EDG B Output Breaker Trip", Apparent Cause Evaluation Revise existing PNPS JPM for the corresponding step to be non-critical.
APPendix C Page 1 of 14 PERFORMANCE INFORMATION Form ES-C-1 NRC SIM JPM S6 NRC JPM S6 Rev 020814.doc
Appendix C Page 2 of 14 PERFORMANCE INFORMATION Form ES-C-1 NUCLEAR PLANT OPERATOR JOB PERFORMANCE MEASURE (RO/SRO}
TITLE:
EMERGENCY DIESEL GENERATOR MONTHLY SURVEILLANCE ATERNATE PATH OPERATOR:
DATE: ---------------------
EVALUATOR:
EVALUATOR SIGNATURE: ---------------------
CRITICAL TIME FRAME:
Required Time (min):
N/A Actual Time (min):
PERFORMANCE TIME:
Average Time (min):
15 Actual Time (min):
JPM RESULTS*:
SAT UNSAT NEEDSIMPROVEMENT (Circle one) *Refer to Grading Instructions at end of JPM COMMENT SHEET ATTACHED: Yes I No (circle one)
(Required for UNSAT, Needs Improvement or Follow-Up Questions)
SYNOPSIS:
TASK STANDARD:
The reactor is at power with all house loads aligned to the Unit Aux Transformer. The monthly operability run of the "A" EDG is in progress. The "A" EDG has already been started and the Operator will commence the EDG run. As KW loading is increased, the EDG governor will become unstable as indicated by KW swings on the engine. The operator is expected to recognize indications of unstable operation and lAW the precautions of procedure 8.9.1 unload the engine and open its output breaker.
The EDG surveillance will be performed lAW procedure 8.9.1. The diesel will be successfully paralleled to the bus and load increased. When the governor becomes unstable the operator will unload the engine and open its output breaker. There shall be no failure of critical elements.
EVALUATION METHOD:
EVALUATION LOCATION:
X Perform Simulate NRC JPM S6 Rev 020814.doc X
Plant Simulator Control Room N/A
Appendix C Prepared:
Mark Santiago Reviewed:
Page 3 of 14 PERFORMANCE INFORMATION Form ES-C-1 Date:
02/08/14 Date:
Approved:
Superintendent, Operations Training (or Designee)
Date:
REVISION LOG Revision Number Date Description 0
10/15/07 New JPM to support the 2007 Annual Operating Exam 1
11/21/13 Updated JPM for 2014 NRC exam. Removed several non-critical steps to both shorten the JPM and which did not add any value to the evaluation of the operator.
2 02/08/14 Revised JPM following NRC Prep week.
NRC JPM S6 Rev 020814.doc
Appendix C Page 4 of 14 PERFORMANCE INFORMATION TASK
Title:
Task Number K&A SYSTEM:
RESPOND TO A DIESEL 264-04-01-004 264000 GENERATOR SPEED CONTROL GOVERNOR MALFUNCTION.
REFERENCES:
PNPS 8.9.1, EMERGENCY DIESEL GENERATOR SURVEILLANCE SIMULATOR CONDITIONS:
Form ES-C-1 K&ARATING:
A3.04 3.1/3.1
- 2. Place the DIESEL GENA TEST switch to "TEST" on Panel C3.
- 3. Place the DIESEL GENA GOVERNOR MODE SELECTOR switch to "DROOP" on Panel C3.
- 4. Start the EDG
- 5. Acknowledge the EDG trouble alarm
- 6. Create a lesson plan that will perform the following when KW load is> 1300 KW:
a) Using local controls Increase and then decrease KW loading so that-300 KW oscillations are seen on the EDG.
b) Be prepared to execute the step repeatedly so that continuous oscillations are evident until the EDG output breaker is opened.
- 7. An lOS operator is standing by to support the operator in responding to the JPM.
GENERAL TOOLS AND EQUIPMENT:
- 1. Key for the "A" EDG Test Switch
- 2. Copy of 8.9.1, completed Section 7.0 Prerequisites and Attachment 1 up through section 1.3, Diesel Start Local. Section 1.3 is to be signed off in its entirety.
CRITICAL ELEMENTS:
Critical elements are shaded in gray within the body of this document.
NRC JPM S6 Rev 020814.doc
Appendix C OPERATOR BRIEF:
Page 5 of 14 PERFORMANCE INFORMATION
- 1. State the following paragraph IF this is the first performance in this setting:
Form ES-C-1 a) "All actions associated with this job performance measure are to be performed. You will be provided access to any tools or equipment you determine necessary to perform the task. When a second checker is called for, the evaluator will perform the role of second checker and will always be in agreement with your actions. Before you start, the evaluator will state the task conditions and answer any questions, then provide a cue to begin".
- 3.
"The task conditions are as follows:
a) The UAT is supplying all the electrical buses b) The monthly operability run of the "A" EDG is in progress lAW procedure 8.9.1 c) The diesel has been started and is running unloaded.
d) The procedure has been completed through Attachment 1, Section 1.3-Diesel Start -
Local.
e) The diesel is ready to be paralleled to the bus.
f)
An operator is standing by in the diesel room.
g) No other testing or surveillances are currently being performed."
- 4. Allow the operator time to review the prepared copy of 8.9.1 prior to commencing and solicit and answer any questions the operator may have.
INITIATING CUE:
State the following:
"[State the operator's name], continue the operability run of the "A" EDG commencing with, Section 1.4, Diesel Start-Local, commencing at step [1 ].
This Task is not Time Critical NRC JPM S6 Rev 020814.doc
Appendix C PERFORMANCE:
Page 6 of 14 PERFORMANCE INFORMATION Form ES-C-1 Notes This task is covered in procedure 8.9.1.
All controls are located on panel C3.
All critical steps must be performed in order written unless otherwise noted START TIME:
- 1.
Procedure Step:
[1]
RECORD start initiation time on Attachment 1 C.
(a)
RECORD diesel VOLTAGE and FREQUENCY as indicated on Panel C3.
Standard Records start time on Attachment 1 C Records Voltage and frequency in the spaces provided on step [1 ].
Cue If asked, Cue that the EDG was started 5 minutes ago.
Notes Performance:
SATISFACTORY UNSATISFACTORY Comments:
- 2.
Procedure Step:
(b)
Using the DIESEL GENA VOLTAGE REGULATOR SETPOINT ADJUSTER, ADJUST the diesel generator output voltage to 4200 volts.
Standard Operator adjusts the output voltage to 4200 volts Cue None Notes Performance:
SATISFACTORY UNSATISFACTORY Comments:
- 3.
Procedure Step:
[2]
VERIFY diesel generator voltage and speed vary on demand.
(a)
Using the DIESEL GENA VOLTAGE REGULATOR SETPOINT ADJUSTER, RAISE the voltage regulator setpoint until the upper limit white light illuminates.
NRC JPM S6 Rev 020814.doc
Appendix C Standard Cue Notes Performance:
Comments:
- 4.
Procedure Step:
Standard Cue Notes Performance:
Comments:
- 5.
- 6.
Page 7 of 14 PERFORMANCE INFORMATION Form ES-C-1 Voltage is raised until the upper limit white light illuminates (light above the switch)
None SATISFACTORY UNSATISFACTORY (b)
LOWER the voltage regulator setpoint until diesel generator output voltage is approximately 4100 volts.
Voltage is lowered to 41 00 volts None SATISFACTORY UNSATISFACTORY
.. [SL',:~~t£1 the DIESEl GENA TO BUS A5 sync;hronizing switch to 80N 11 UNSATISFACTORY St!llncoming voltage slightly above the running voltage.
- UNSATISFACTORY NRC JPM S6 Rev 020814.doc
Appendix C I l
- 7.
Procedure Step:
Standard Cue Notes Performance:
Comments:
- 8.
Procedure Step:
Standard Cue Notes Performance:
Comments:
- 9.
Procedure Step:
Standard Cue Notes Performance:
Comments:
Page 8 of 14 PERFORMANCE INFORMATION Form ES-C-1 (b)
Using the DIESEL GENA GOVERNOR SPEEDCONTROL, RAISE frequency approximately 1 Hz.
Frequency is raised - 1 Hz as indicated on the frequency meter.
None SATISFACTORY UNSATISFACTORY (c)
LOWER frequency by approximately 2 Hz.
Frequency is lowered - 2 Hz as indicated on the frequency meter.
None SATISFACTORY UNSATISFACTORY NOTE If EDG "A" is to be paralleled with the Startup Transformer, then the degraded voltage protection for both A5 and A6 is to be declared inoperable in accordance with Technical Specifications Table 3.2.8.
02_erator reads the note and proceeds to the next step.
None SATISFACTORY UNSATISFACTORY NRC JPM S6 Rev 020814.doc
Appendix C Page 9 of 14 PERFORMANCE INFORMATION Form ES-C-1
- 10.
P~~~~efft; ;X4 * *'PAptAl..LEl Oiesef Generator "A.. to Bus As.
- 12.
- 13.
Procedure Step:
[5]
ADJUST OIE.SEt.. GENA GOVERNOR.. SPEED CONTROL to produce a slow rotation in the FAST direction.
UNSAnSFACTORY SUGHTL Y BEFORE an "in phase indication, CLOSE f!reaker A509:, DlESEL GEN A TO BUS A5.
UNSATI8FACTOAY
. IMMEDIATELY INCREASE toad to 500kW (450 to 550kW)
With th~ DIESEL.GENA GOVERNOR SPEED CONTROL.
SET the DIESEL GENA TO BUS AS synchronizing switch to "OFF".
Standard Synchronizing meter de-energizes when switch is placed to OFF.
Cue None Notes NRC JPM S6 Rev 020814.doc
Appendix C Performance:
Comments:
Page 10 of 14 PERFORMANCE INFORMATION SATISFACTORY UNSATISFACTORY Form ES-C-1
~~,:ltiQ.f!tfi4~E t~ctiv&J~d to 250kVAB (200 to30QkVAR)with the
.. * *. lrJIS.EL <9EN A VOLTAGE *REGULATOR SETPOINT ADJUSTER.
RECORD time:
. the.. :time is not a critlcaJ element of this critical ste.
- UtfSATISFACTORY EA the diesef has run tor at least 8 minutes:
- ... load to 1()00kW (950 to 1050kW) using the
- SeN A eOVERNOR.SPEEO CONTROL.
Rv**
I:NOfUEASE reactive toad to 500kVAR (450 to 550kVAR) using the:C>IISELGEN /(VOLTAGE RE<9ULATOR SET~OINT AD'JUSTER~..
. RSCORS time:
NRC JPM 86 Rev 020814.doc
Appendix C Page 11 of 14 PERFORMANCE INFORMATION Form ES-C-1 tbe time is not a *critical element of this critical. step.
- wiftbegin in the next step w~en KW loading exceeds AR'ER the die set has run tor at teast 8 minutes:
INCREASE load to 1800kW (1750 to 1850kW) using the OISSEl: GEN A GOVERNOR SPcED CONTROL.
- :~:mr~~flllaiwe shalf beaborted*bY*immediately unloading the EDG and
'oM
- itrsel generator Circuit Breaker A609 or A509 if. any of the following
.. exist:
at.lons Jn kVAR greater than 200 when atsteady state,
.,ns in: kW greater than 200'when at steady state."
- ** <'.*0.** ~or untoads the diesel Diesel Generator, Governor Control.switch, lowers the KW of the NRC JPM S6 Rev 020814.doc
Appendix C Page 12 of 14 PERFORMANCE INFORMATION UNS4TlSFACTORV Cue:
This completes this JPM.
STOP TIME:
NRC JPM S6 Rev 020814.doc Form ES-C-1
Appendix C Page 13 of 14 VERIFICATION OF COMPLETION Job Performance Measure No.:
2014 Systems-Control Room JPM S6 Examinee's Name:
Date Performed:
Facility Evaluator:
Number of Attempts:
Time to Complete:
Question Documentation:
Result:
SAT UNSAT Examiner's Signature:
Date:
NRC JPM S6 Rev 020814.doc Form ES-C-1
Appendix C INITIAL CONDITIONS:
Page 14 of 14 JPM CUE SHEET The UAT is supplying all the electrical buses Form ES-C-1 The monthly operability run of the "A" EDG is in progress lAW procedure 8.9.1 The diesel has been started and is running unloaded.
The procedure has been completed through Attachment 1, Section 1.3-Diesel Start
- Local.
The diesel is ready to be paralleled to the bus.
An operator is standing by in the diesel room.
No other testing or surveillances are currently being performed.
INITIATING CUE:
Continue the operability run of the "A" EDG commencing with Attachment 1, Section 1.4, Diesel Start-Local, commencing at step [1 )).
This Task is not Time Critical NRC JPM S6 Rev 020814.doc
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 1 of 28 Entergy Operations, PILGRIM NUCLEAR POWER STATION HT - Apparent Cause Evaluation Report Position Evaluator Responsible Manager CARB Chairperson (if applicable)
EDG B Output Breaker Trip CR-Unit-2011-05152; Event Date: 11-14-2011 REPORT DATE: 11-29-2011, Rev 2 Name Date Richard Morris 11/28/2011 Bruce Chenard 11/30/2011 EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 2 of 28 PROBLEM STATEMENT:
On November 14, 2011 during the monthly run Emergency Diesel Generator (EDG) 8 generator output breaker A-609 tripped. This breaker tripped occurred approximately 45 minutes into the full load portion of the test, the 8 EDG remained running. Monthly test was secured and the entered into LCO to troubleshoot.
Does this ACE report require an Equipment Failure Evaluation (EFE)? (See procedure steps 5.3[3](c) and 5.4)
[8J Yes 0No IF Yes, THEN complete EN-LI-119-01 Equipment Failure Evaluation AND attach in PCRS IF No, THEN an EFE analysis is not required.
Was an HPER assigned & performed for this CR?
(See procedure step 5.3[3](c))
DYes IF Yes, THEN ensure results of the EN-HU-103 HPER are discussed in the Event Description.
EVENT DESCRIPTION: (The How)
IZI No The trip of the EDG output breaker A609 is driven from the differential relay 187-609.
There are several deviations that can result in the trip of the differential relay. A Failure Modes Analysis (FMA) per procedure EN-LI-118-08 was used to determine potential causes and procedure EN-MA-125 was used to check causes and provide data in cause confirmation or elimination.
A brief causal time line is provided for reference.
October 16 to October 19 2011 EDG 8 planned overhaul. Scope of the overhaul required 2-yr, 4-yr and 8-yr PM activities. The 2yr PM includes a generator meggar test.
October 18 2011 Generator Meggar completed.
October 20 to 21 Post work testing from outage required several engine starts and runs.
Engine was returned to operable status October 22, 2011.
November 14 2011 Normal monthly surveillance test is performed by procedure 8.9.1. At 0230 the EDG output breaker trips open. This is the first EDG 8 run since the overhaul.
November 14, 2011 0630 Troubleshooting begins. First task assigned to Electrical Maintenance is to perform a generator meggar test to assure that windings are intact.
November 14, 2011 0900 FMA team convenes and lays out potential causes for breaker opening.
November 14 2011 1600 MA-125 re ared to perform field checkin of potential causes EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 3 of 28 developed by FMA.
November 14, 2011 1700 Elect Maintenance reports that field connect wire lug on B EDG Generator Lead Junction Box is broken. Lead goes to current transformer (CT) 5 on phase B of the generator output.
November 14, 2011 2345 Generator Meggar testing satisfactory.
November 15, 2011 1400 CT-5 terminal lead repaired under WO 296754. task 01. This is the first time on record that this connection has been disturbed.
November 15, 2011 2000 Confirmatory testing of other possible cause per MA-125 complete. Testing determines that other causes ruled out and broken lug is confirmed as cause of differential relay trip and subsequent breaker opening.
ANALYSIS and APPARENT/CONTRIBUTING CAUSE(S): (The Why)
The A609 breaker connects the output of EDG B to its associated emergency bus A6. There is a logic scheme to protect the integrity of the emergency power bus and its power supplies.
One protective feature is differential current. This scheme functions to compare current leaving the generator to the current entering the bus. If the currents do not match then the relay trips. This is known as a differential trip. Such a trip opens the source output breaker to permit an alternative source to power the bus. The symptoms described in CR-11-5152 match this type of trip. An FMA was performed to determine possible causes of the trip.
A major step of the action plan per the FMA was to gather as-found conditions on the differential relay, the A609 bus, and the current transformer wiring. A preliminary report came in from Electrical Maintenance that the terminal posts on the differential relay 187-609 were loose. Confirmatory testing for this condition was created in the MA-125 to perform in-situ testing to determine if the loose posts were the cause of the relay trip. The in-situ test proved that the loose posts were NOT the cause of the relay trip.
CR-PNP-2011-5166 and CR-PNP-2011-5181, conditions reports were the findings of the troubleshooting and identification of the broken ring lug on the 8-EDG current transformer sensing wiring and addressed by this Causal analysis.
A second finding of the MA-125 was inspection of the "as-found" condition of the current transformers (CTs) feeding the differential relay. Electrical maintenance found that a terminal on the B phase of the generator out CT was broken. The as-found condition (see photos 1 and 2 attached) shows that the ring lug on one of the two CT terminals was broken. This break interrupts signal to one side of the relay. The loss of signal was caused by a broken ring lug on the B phase generator output CT (CT-5). The orientation of the lug on the terminal was not consistent with good maintenance practice. Referring to photo 2; the ring lug was positioned such that the lug was sticking outward toward the cabinet cover. The wire for the broken ring lug is routed with several sharp bends and is not in-line with the long axis of the wire. The connection for the intact lug was gently swept from the wire harness to the EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 4 of 28 terminal with no sharp bends and that the wire remains in-line with the ring lug. Conversely the sharp bend to attach at the terminal adds stress to the connection. That orientation led to unnecessary stress on the terminal lug. It is not known when the lug got into this orientation but it is possible that this defect is from original construction. Field observation showed that the terminal screws were aged, had a thick patina indicating they had not been moved for several years, there were no tool marks. There is no prior history of CT testing, therefore there is no planned reason for lifting and re-landing these leads.
When the generator is meggar tested the heavy detached cables interact (photo 1) with the CT harness and add stress. It is probable that the lug is bent upward by personnel working in the cabinet disconnecting and reconnecting the generator leads. The act of bending the lug back down likely broke most of the ring lug web. Subsequent vibration within the cabinet led to final separation of the remaining connecting ligament. While the ring lugs are durable and rugged, they are also breakable from bending action after a few bending cycles. Photo 3 shows the break surface. There is a small area showing tarnish from a prior crack while the majority of the break surface still shows bright brass coloring.
EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 Page 5 of 28 WHY STAIR CASE Emergency Diesel B manually tripped approx. 45 minutes into full load run Emergency diesel tripped because generator output breaker auto opened.
Breaker opened because differential relay 187-609 tripped Differential relay 187-609 tripped because a current imbalance was detected A current imbalance causes a differential input to output on the relay TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Current imbalance caused by loss of signal from B phase CT DIRECT CAUSE Lon of signal from B phase CT cause by broken ring lug Broken ring lug caused by orientation of connection 0
I Orientation of connection cause stress by inadvertent contact and lift CC-01
~~A
~~
/
/
Inadvertent contact and lug lift occurs during bi-annual generator megger test
,..... -------------I I
v 1
Heavier Personnel I
1 cables lifted performing I
I during test I
generator inadvertantly 1
I megger test contact and I
I rest against bend the lug I
1 CT -wire eventually 1
harness breaking it I
I I
1 CC~2 I
ONTRIBUTING-r 4
_I CAUSE OP4A Insufficient Procedure Detail Procedure 3.M.3-61.5 does not instruct workers to check integrity of CT connections post Megger EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 6 of 28 O&P DISCUSSION:
Evaluation of Organization and Programmatic issues was addressed by a barrier analysis and classified with EN-Ll-118 attachment n9.5. To potential O&P issues were Identified:
OP2G - Organization to program interface weakness - Is there evidence of the lack of a procedure that should have been written but does not exist? and OP4A Programmatic deficiencies - Is there evidence that there is insufficient detail in a procedure to perform the task?
Since a procedure does exist it is determined that OP4A is the O&P mechanism involved. The procedure for meggaring the Generator should contain a step to check the integrity of the CT wiring connections. Affected procedure is 3.M.3-61.5 attachments 1 C and 2C.
EXTENT OF CONDITION:
The purpose of this review is to determine whether the likely cause of the EDG 8 differential lockout has common mode implications. A failure modes analysis was developed that lists the potential causes based on data collected from field as-left conditions and other sources.
Various possible causes were ruled out. An unexpected condition was identified.
Troubleshooting is complete.
The unexpected condition is a bent and broken ring lug found on the phase 8 current transformer. Circuit analysis indicates that opening of this connection would have resulted in the EDG response that was reported and found as-left in the field. The lug separation was in the neck of the lug. It was not due to loosening of the stud or wire crimping. The lug is of a rugged construction commonly used in such applications and where vibration is expected. In a like new condition, it is not considered likely that the low mass of wire connected to the lug would drive vibratory failure. It is more likely that a past mechanical overload (i.e. bending) of the lug from work in the cabinet initiated a substantial crack. The ring lug was positioned such that there was increased chance for mechanical interaction and additional stress. Subsequent operation of the EDG over a period of time provided enough vibratory flexing of the damaged lug to ultimately cause failure. This is supported by the fact that the EDG operated successfully multiple times since the last entry into the cabinet and the lockout occurred after most of the recent surveillance run. OE searches for broken wire lugs indicate that failures have occurred in conditions of high long term vibration (e.g. turbines, diesels, etc.). It is not clear from the documented information where the dominant failure location on the lug is. Causes include wires installed too tightly, numerous connection flexures from removing and reinstalling, etc. While vibration certainly contributed to the industry failures, weak work practices were also important. From a common cause perspective, it is not evident that a mechanical overload of the lug in one EDG would have occurred in the other EDG. Therefore, this potential cause is also not considered a likely cause that can affect the other EDG.
In addition to the above analysis the current transformer lugs on the A EDG are to be inspected as part of this extent of condition. Work Order WO 298446 is issued to perform this inspection. (CA-EOC). Although the S80 wiring configuration is not the same as found in the main EDG's the wiring should be inspected to identify any similar issues or EN-Ll-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 7 of 28 interferences. This will be captured by an enhancement to the SBO-DG maintenance program to add similar note of the "as found"- "as left" condition of all accessible wiring during the megger testing. (CA-EOC).
SAFETY IMPLICATIONS Any safety significance of the identified issue is bounded by the loss of a single emergency diesel generator. During this time frame all other AC sources were available/operable (i.e.,
startup transformer, shutdown transformer, unit auxiliary transformer, station blackout diesel generator, and B emergency diesel generator). The identified issue did not affect any of the other normal or emergency power sources. The EDG-A from a common cause perspective was determined not to be evident of a mechanical overload of it's current transformer (CT) lug. Input from PRA with regard to a EDG out of service showed no significant increase in core damage risk, even if we were conservatively unavailable for the 14 LCO period. There is no significant increase to industrial safety with a diesel out of service other than those workers involved with the troubleshooting.
INTERNAL (Site/Fleet) OPERATING EXPERIENCE:
A search of PCRS for conditions previously identified at Site/Fleet station was preformed using the keywords of "lug" and "broken" with a date range of 11/2007 to 11/2011 and obtained 139 hits. The review identified multilple incidents:
Broken strands Worn lugs Broken terminal screws Wrong size lug Improper lugs for application Failure due to tooth washer causing copper fatigue.
Internal OE with regard to broken lugs results:
CR-AN0-1-201 0-1022 Category 0 repair only no corrective actions.
CR-PLP-201 0-1866 Category D repair only no corrective actions.
CR-PLP-201 0-4983 Category C repair only no corrective actions.
EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 8 of 28 CR-PNP-201 0-3665 Category D repair only no corrective actions.
CR-PLP-2010-5197 Category 8 heavy use shortened life of lug.
CR-AN0-1-201 0-1022 Category D repair only no corrective actions.
CR-PLP-201 0-1866 Category D repair only no corrective actions.
CR-AN0-22009-1 09 Category D repair only no corrective actions.
CR-QN02-2011-732 Category D repair only no corrective actions.
Review of these internal OE did not lead to any proactive corrective actions that PNPS could take to prevent the type of failure found during this event.
EXTERNAL (Industry) OPERATING EXPERIENCE (see procedure step 5.3[3] (j))
Performed OE search for common cause evaluation with the following results.
OE search in IN PO database with the following used as keywords:
"Terminal lug failure" "Crimp connection failure" Electrical termination failure" "fatigue failure lug" "lug" No restriction was put on any of the OE searches except "lug" which was limited to date ranges of November 2007 to November 2011.
All searches performed resulted in 300 hits, (maximum allowed in OE search) with many overlapping hits between searches. Review of the search results identified various generic failures associated with lugged connections. The most applicable OEs were those associated with a failed or broken lug ( -11 ), the next most applicable would be loose connections ( -7) and lastly other ( -2).
The OE reviewed for failed or broken lug most likely propagated by lug flexing during maintenance activities, excessive movement of the lug while it was tightly fastened to its contact point, excessive bending due to installation configuration, and cyclic fatigue due to routine lifting and landing of the lug during preventive maintenance, Most relevant results follow OE21 075 (Brunswick) Wire movement/manipulations results in broken wire lug in breaker cubicle of instrument air compressor.
OE27124 (Millstone 2) Mechanical failure of a lug in the pilot wire circuitry that caused an open circuit. Excessive movement may have occurred during wiring configuration changes and activities after initial installation.
EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 9 of 28 OE15516 (LaSalle Unit 2) Investigation determined that installation of the lead into the lead box resulted in excessive bending stress on the cables. This stress combined with the size of the cables and a splice that had been inserted into each phase, cause the lug to fail in service.
OE14057 (Limerick) Failed wire lug in the Alterex phase differential current protective relay system. The lug failed due to cyclic fatigue. Incipient cracks developed from routine lifting and landing of the lug during preventative maintenance over several outages. Normal running vibration was sufficient to cause the cracks to propagate leading to the ultimate failure of the lug.
OE12764 (Salem 1) Main generator phase A differential current relay actuation was caused by a degraded crimp termination on the phase A neutral CT field wiring. It is believed that the degraded crimp termination caused either a high resistance in the CT secondary and/or an intermittent open circuit.
There were no corrective actions associated with the external OE searched that would have positively identified a weakened lug. Aside from the additional training purposed by this causal analysis and additional visual inspections that may pro-actively identify potential problems.
EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 10 of 28 APPARENT OR CONTRIBUTING CAUSE, OR EXTENT OF CONDITION ISSUE (Add PCRS CA #, if applicable)
ACTIONS COMPLETED (See EN-LI-119 step 5.3[3](k))
ACTION COMPLETED
[note any Work Orders/Requests, ER'S, other]
Date Completed Apparent Cause Current transformer phase B lead repaired by WO 11/15/2011 Direct Cause 296754. Generator output and relaying successfully APPARENT OR CONTRIBUTING CAUSE, OR EXTENT OF CONDITION ISSUE (Add PCRS CA #)
Revise 3.M.3-61.5 Attachment 1C &2C O&P action EOC EOC Enhancement -Lessons Learned Enhancement -Lessons Learned post work tested.
PROPOSED CORRECTIVE ACTIONS (See EN-LI-119 step 5.3[3](k))
CORRECTIVE ACTION DESCRIPTION
[note any Work Orders/Requests, ER's, other]
Add NOTE to inspect CT wiring during meggering of generator electrical end before Step [5] "Visually inspect all CT wiring as found" and again at end of Step [6] visually inspect "as left" CT wiring.
Perform inspection of A EDG current transformers for similar terminal wiring issues.
Perform inspection of SBO-DG wiring in junction box where meggering is performed.
Maintenance Depts, Electrical and I&C review the causes and consequences of this event with respective staffs. Use as an example where identifying a minor problems early may prevent a larger issue later.
System Engineering review the causes and consequences of this event with respective staff. Use as an example where identifying a minor problems early may prevent a larger issue later.
EFFECTIVENESS REVIEW PLAN (If Required)
(See EN-LI-119 step 5.3[3](k)(10))
LO Number Assigned Department Elec. Maint.
WO# 298446 Elec. Maint.
Maintenance Mgmt System Eng'rng.
Apparent Cause CA
Description:
Enhance existing 2 year PM for the EDG electrical Due Date 1/30/201 2
12/8/2011 4/6/2012 4/6/2012 4/6/2012 inspections as part of 3.M.3-61.5 Attachment 1 C/2C to visually inspect current transformer (CT) wiring "as found" and "as left" when meggering electrical end in G11/ G12 terminal box.
Description Method:
Document review EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 11 of 28 Attributes:
Component position Success:
Visual verification there has been no disturbance of wiring during meggaring.
Timeliness:
Performed as part of the existing 2 year PM when accessing terminal box G 11 /G 12 Owner Group:
Electrical Maintenance
\\Due Date:
\\6130/2012 EN-LI-119, Rev. 13, Attachment 9.1
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 12 of 28 TREND DATA (coordinate entry in the PCRS Trend Table of this CR):
Cause Codes:
Human Performance Causal Factor(s) (List all): None O&P Causal Factor(s) (List all): OP4A Insufficient details Equipment Causal Factors (List all): EF1 H Unforeseen failure (CT wiring has no routine PM check on terminals or wiring which would lead to visual inspection).
ES1 F Installation workmanship less than adequate on CT lug that was improperly routed to create a sharp turn and edge to enable failure.
MT1 Z Other Workmanship/procedural guidance during generator meggering does not identify potential issues with CT wiring.
EFE Codes ((see Procedure step 5.4 [4]):
INPO ER PO&C codes: None Failure Mode Codes: FM09 Cracked/Fractured FM29 Open Circuit FM45 Fatigue Attachments:
Failure modes analysis chart (EN-DC-118-08)
Barrier analysis Evaluation for 0 & P issues (EN-LI-118 Attachment 9.5)
Photographs Photo 1 Generator Lead Junction Box Photo 2 Current Transformer number 5 Connections Photo 3 Broken Ring Lug EN-LI-119, Rev. 13, Attachment 9.1
An~
... rtMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 13 of 28 CR-PNP-2011-05152 FAILURE MODES ANALYSIS Based on Attachment 1 EN-LI-118-08 Problem Statement: At 02:30 11/14/2011 with EDG Bin test, Received Alarms EDG B lockout; Generator Breaker Trip/lnop; Volt/ Frequency Abnormal.
Generator output breaker A-609 Tripped. The B edg remained running at 61 HZ, 0 voltage, 0 field amps. Develop troubleshooting to determine cause.
Initial Field observations: Excitation panei(Voltage regulator) resistor R-26 discolored and fastener loose. Differential relay (187-609) trip light ON.
No flags tripped on voltage balance relay, no flags tripped on overcurrent, no flags at engine. Engine went to 960 rpm as expected with mech governor.
WORKSHEET Tech Refuting Probability Failure Mode Supporting Evidence Evidence Assumptions Added Info Conclusion 4
Generator short/
Differential relay ( 187-no over current Do meggar Megger Ruled out as cause Fault 609) trip flags No reported completed or degraded.
abnormal smells/
and SAT at 2235 11-burning.
14-2011 PJK Meggar Sat.
4 Generator to breaker Differential relay ( 187-Meggar Sat.
Do meggar Megger Ruled out as cause cable short I fault 609) trip reported completed or degraded.
and SAT at 2235 11-14-2011 PJK 4
Exciter Series Boost Differential relay ( 187-Meggar Sat.
Ruled out as cause Transformers 609) trip or degraded.
4 Exciter Saturable None No Catastrophic Short to ground of Ruled out as cause Reactors damage and no Exciter Reactors or degraded.
voltage or Kvar would be swings Catastrophic and open reactor would produce Kvar issues 4
Generator breaker Generator breaker No visible damage Do detail inspection Ruled out as cause (A609} short/ fault trip/inop alram by inspection by adding visual or degraded.
Differential relay (187-inspection to MA-125 609) trip
ATTACHMENT9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 14 of 28 2
Differential relay Relay tripped as relay tripped from Validate 187-609 Loose terminal test (187-609) failure evidenced by trip light.
signal or other calibration & perform proved not to be a itself or setpoint drift Visual inspection found cause?
functional 3.M.3-1.
cause. Internal low.
very loose case tern inals Perform physical contact maintained by at terminals# 6-10 which inspection of case terminal strip spring tension. Bench test is C phase.
before removal.
showed correct relay Relay is mounted of settings. RULED door of A609 OUT Tech Refuting Probability_
Failure Mode Supporting_ Evidence Evidence Assum~ions Added Info Conclusion 5
Input CT's to 187-609 All CT's tested sat 3 at Bkr Visual inspection RULED out as (6 ea). 3 at Gen 3 at 3 at Gen.
CT testing cause Breaker. This item is (continuity, an actual failure of a excitation, turns ratio CT.
if possible) 1 CT secondary wiring Inspection found CT for B Loose wires/ or Inspection found CT DIRECT CAUSE phase at generator (CT-5) broken wiring for B phase at Lug was had broken spade lug.
connections generator (CT-8 had positioned in a Resulted in open circuit to Industry OE (PLP &
broken spade lug.
place where there Differential IPEC) and Pilgrim Resulted in open could have stress OE circuit to Differential interactions See Photos.
3 Shunt Reactor Differential relay (187-NOT Likely Cal test (along Waiting for bench Module 609) trip with187 relay) cal completion X
Voltage regulator Stable voltage and Burn result from New MOC Ruled out as cause stable Kvars event? NO burn Review photo record or degraded.
conditions before found MPR
- resistor R-26 event. Visual Associates similar condition in inspection no confirmed R26 file photos.
damage/ overheat resistor in normally of resistor discolored at other plants and is not a concern.
6 K1 relay failure Differntial relay ( 187 -609) loss of field relay K1 relay function to Ruled out as cause
_trie_ _________ did not trip short out field or degraded.
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 15 of 28 X
Notes Engine trip DID NOT HAPPEN Need to walk down all panels for as found conditions and flags, along with visual inspection. - COMPLETE Results below Differential relay is the device that actuated the generator lockout.
Ruled out as cause or degraded.
Follow-up Observations: Elab found terminal posts loose of differential relay #187-609 could be event cause. Need to confirm by test.
Field check of all other devices: Relay 140-609 Loss of excitation No trip flags; no flags on 164F-609 Ground detection relay; no flags on 132-609 anti-motoring relay;159-609/2 Breaker 609 closure permissive flag dropped as expected for an engine load run.
Breaker A609 inspection performed no physical damage evident.
Inspection of Gen Ditt CTs at G12 found broken lead at CT8 Discussion with MPR confirmed the dark R26 resistor is normal Report of 61Hz and 960 RPM is not consistent; most likely result of accuracy of instrumentation All Ct's have been tested satisfactorily.
In-situ test on 187 relay lugs proved it not to be the cause. Contact maintained by internal terminal spring tension. Ruled out Calibration test of the relay found relay to be satisfactory.
Summary Gen Lockout shorts out gen field to produce zero gen output voltage; expected result References Ops control room log, Procedure 8.9.1, Alarm Outputs Prints E17, E27 sh 2
ATTA... rtMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 16 of 28 ATTACHMENT9.1 BARRIER ANALYSIS Sheet 1 of 1 BARRIER ANALYSIS WORKSHEET CONSEOUEt.JCEIADVERSE EFFECT BARRIER THAT SM()QLD HAVE PRECLUDED BARRIER ASSEssMENT eoN$1!0UENCE (WHY THE BAARll:R FAILED)
(list one at a time - need not be in order)
{Identify all ~p!icabi& physical & administratPie (Identify if barrier was weak. missing or ineffective EDG-8 generator Output breaker A-609 Visual inspection and careful work No specific steps to check CT wiring as tripped due to generator differential relay practice around wiring in terminal box part of 2 year PM meggar testing.
actuation G12 OP4A insufficient procedure detail to
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 17 of 28 ATTACHMENT 9.5 EVALUATION FOR ORGANIZATION & PROGRAMMATIC ISSUES Sheet 3 of 14 Beginning of Form Include this Worksheet as an Attachment to the report. The questions are provided to promote consideration of like symptoms, not to define a specific failure mode. O&P causal factors are symptoms of the more basic causes of the event and are typically an action or condition that shaped the outcome of the situation.
For each causal factor block checked YES:
- 1.
Ensure it is appropriately represented in the WHY Staircase as a cause or contributor.
- 2.
In the BARRIER ANALYSIS, tie the O&P causal factors as appropriate to Barriers that failed, were weak, missing, or ineffective.
- 3.
Summarize in the O&P section of the report how the identified Organizational & Programmatic weaknesses caused or contributed to the event and identify the Barrier which should have prevented it.
O&P codes: OP2G lack of a procedure that that should have been written but does not exist; and/ or OP4A Insufficient details in a procedure to perform the task. Since a procedure does exist it is determined that OP4A is the O&P mechanism involved O&P Worksheet Contributed to Potential O&P Failure Modes (Causal Factors) or Caused RC# AC #CC#
Event?
YES NO I)
OP1 X - Organization to Organization Interface Weaknesses X
- Inadequate interface among Organizations (Good organizational structure but organizations don't communicate).
- Excessive or lack of overlap in functions (Overall structural design results in overlaps or holes between organizations) a)
OP1 A - Does there appear to be evidence of inadequate interface among X
organizations?
Problems in this area surface in the form of a high human error rate in tasks requiring communication among different organizations. Usually this is caused by a lack of interface formality (tailgate meetings, formal interface documentation or agreements, etc.), inadequate teamwork or trust among organizations, or inadequate physical settings.
b)
OP1 B - Is there evidence of excessive or lack of overlap functions between X
organizations?
Negative performance in this block is usually caused by a lack of organizational planning resulting in an inadequate definition of job functions between one or more organizations.
c)
OP1C -Is there evidence that the required notifications were not made when the job X
was begun, interrupted or completed?
Describes either the failure to perform the verbal communication of status when required by the process or the failure to design the process to require the verbal communication of status when successful implementation of the process depended upon this communication.
d)
OP1 D - Is there evidence that appropriate personnel and departmental interactions X
were not fully considered when new processes were created during the implementation phases of the change?
Changes to processes created new requirements for interaction between personnel or departments that were not considered in the implementation phase of the change.
e)
OP1 E - Is there evidence that planning was not coordinated with inputs from walk-X downs and task analysis?
Job plan did not incorporate information gathered during field visits or task analysis concerning the steps and conditions required for successful completion of the task.
f)
OP1 F-Is there evidence that planning was not coordinated with all departments X
involved in task Interdepartmental communication and teamwork were not supported by the planned work flow.
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 18 of 28 ATTACHMENT 9.5 EVALUATION FOR ORGANIZATION & PROGRAMMATIC ISSUES Sheet 4 of 14 O&P Worksheet Potential O&P Potential O&P Failure Modes (Causal Factors)
Failure Modes RC# AC#CC#
(Causal Factors)
YES NO
- 2)
OP2X *Organization to Program Interface Weaknesses X
Lack of commitment to program Implementation (organization never gets program off the ground)
Inadequate Program monitoring or management (organization does not monitor or manage the program effectively)
Lack of program evaluation process (program survives but Organization does not evaluate program, so it goes in the wrong direction)
Lack of organizational authority for program Implementation (organization starves to death because no one is protecting it) a)
OP2A - Is there evidence of a lack of commitment to program implementation?
X Usually evidenced by slow program implementation. The failure is generally due to excessive program implementation requirements or a lack of management support of the program.
b)
OP2B - Is there evidence of inadequate program monitoring or inadequate X
management skills?
Indicated by a lack of program improvement over time. Usually it is caused by inadequate staffing or inadequate management skills.
c)
OP2C - Is there evidence of a lack of a program evaluation process?
X This area is more reactive, in that a program failure occurs before action is taken. However, the same items contribute to negative performance, i.e., inadequate management practices, inadequate staffing for program implementation, or insufficient program design.
d)
OP2D - Is there evidence of a lack of organizational authority for program X
implementation?
This code usually is associated with an insufficient budget for the program or fragmented responsibility and/or accountability for the program. Potential causes include a lack of organizational planning or a lack of management commitment to program implementation.
e)
OP2E - Is there evidence of unclear or complex wording or grammar in program X
implementation documents?
Wording, grammar or symbols fail to clearly and concisely specify the required action; instructions provided for team of users fail to specify roles of each user.
f)
OP2F - Is there evidence of an omission of relevant information in program X
implementation documents that would have prevented an event from occurring (e.g.
insufficient information in graphs, tables or illustration; lack of instructions or data sheet documentation requirements, etc.)
Over reliance on user's training, skills or experience; lack of detail for infrequent, complex, crucial or error-prone tasks; insufficient information in graphs, tables or illustrations; lack of instructions for data sheet documentation requirements g)
OP2G - Is there evidence of the lack of a procedure that should have been written but X
implementing The process meets administrative requirements for having a procedure, but no procedure has procedure been written.
h)
OP2H - Is there evidence that policy guidance or management expectations were not X
well defined or understood by personnel involved in performing the task?
Personnel exhibited a lack of understanding of existing policy and/or expectations, or policy/expectations were not defined.
i)
OP21 - Is there evidence that job standards were not adequately defined or X
communicated?
Measurement of effectiveness could not be performed for specific job functions due to lack of defined standards.
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 19 of 28 ATTACHMENT 9.5 EVALUATION FOR ORGANIZATION & PROGRAMMATIC ISSUES Sheet 5 of 14 O&P Worksheet Potential O&P Potential O&P Failure Modes (Causal Factors)
Failure Modes RC# AC#CC#
(Causal Factors)
Yes No j)
OP2J - Is there evidence that personnel exhibited insufficient awareness of the impact of actions on safety and reliability?
X Management failed to provide direction regarding safeguards against non-conservative actions by personnel concerning nuclear safety or reliability k)
OP2K-Is there evidence that management follow-up or monitoring of activities was X
ineffective in identifying shortcomings in implementation?
I)
Management's methods for monitoring the success of initiatives were ineffective in identifying shortcomings in the implementation.
m)
OP2L - Is there evidence that causes of a previous event or known problem were not X
determined?
Analysis methods failed to uncover the causal factors of consequential or non-consequential events.
n)
OP2M-Is there evidence that the effects of changes on planned schedules were not X
adequately addressed prior to implementation?
Changes to processes which resulted in scheduled changes had effects on personnel or equipment that were not addressed in the change implementation.
o)
OP2N -Is there evidence that the job scoping process did not properly identify potential X
task interruptions or environment stress?
Work scoping process was not effective in detecting reasonable obstructions to work flow (e.g.,
shift changes) or the impact of environmental conditions.
p)
OP20 - Is there evidence that the job scoping process did not identify special X
circumstances or conditions that may be impacted or dependent on other circumstances or conditions?
Work scoping process was not effective in detecting work process elements having a dependency upon other circumstances or conditions.
q)
OP2P - Is there evidence that the field walk down input to design was less than X
adequate?
Design change and/or field change requests as a result of inadequate field walk downs to verify actual configurations of plant components, structures and systems that interface with or affect the designs.
- 3)
OP3X - Program to Program Interface Weaknesses X
Lack of interface requirements (no formal procedures to make sure two programs talk to each other)
Conflicting program requirements (conflicting actions required by two different programs)
Inadequate Interface requirements (information required is available but program interfaces are inadequate to get it) a)
OP3A - Is there evidence of a lack of interface requirements between two or more X
programs that are required to interface in that details necessary to ensure a consistent standard are not adequately covered in programmatic implementing documents?
Usually this is caused by inadequate program design or an inadequate work planning process.
b)
OP3B - Is there evidence of conflicting program requirements where one program has X
different actions from another program for the same issue?
This codes is used when different actions are required by two or more programs for the same situation. As a result, staff efficiency and accountability is negatively impacted.
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 20 of 28 ATTACHMENT 9.5 EVALUATION FOR ORGANIZATION & PROGRAMMATIC ISSUES Sheet 6 of 14 O&P Worksheet Potential O&P Potential O&P Failure Modes (Causal Factors)
Failure Modes RC# AC#CC#
(Causal Factors)
Yes No c)
OP3C - Is there evidence of inadequate interface requirements in that one program X
specifies actions different from another program for the same issue?
This code is used when actions are required by one program belonging to another program that is inadequate in perform the actions. The cause is usually inadequate program design and/or inadequate work planning processes
- 4)
OP4X - Programmatic Deficiencies X
Insufficient detail (This is my first time doing this, how am I supposed to know what "use normal process" means?)
Inadequate scope ("The procedure left out all the information on the electrical cables that need to be connected")
Excessive implementation requirements (so many requirements that people give up and don't try to follow the procedure)
Inadequate verification process ("We haven't really looked at our processes and given them a 'check up' for over 15 years")
a)
OP4A-Is there evidence that there are insufficient details in a procedure to perform the X
implementing When a program is vague regarding what is required in a particular situation, it is usually indicative procedure of an inadequate program design or insufficient feedback for individuals using the procedure.
b)
OP4B-Is there evidence of inadequate job scope (omission of necessary functions) in X
an implementing procedure because of an inadequate program design or inadequate feedback from the field?
Either inadequate program design or inadequate feedback from the field is usually taking place c)
OP4C - Is there evidence of excessive implementation requirements that result in X
portions of the program being ignored by the staff due to overload?
This can be caused by inadequate program design, lack of work prioritization, or inadequate staffing d)
OP4D-Is there evidence of an inadequate verification process (single human error, X
high program failure rate, poor procedure quality or inadequate program design?
Program breakdown by a single human error; high program failure rate, poor procedure quality.
Inadequate program design e)
OP4E-Is there evidence that there is a lack of responsibility by personnel because it is X
not well defined or personnel are not being held accountable?
Responsibility for process elements (procedures, engineering, training, etc.) was not placed with individuals or accountability for failures of those process elements was not placed with individuals.
f)
OP4F - Is there evidence that a response to a known or repetitive problem was X
untimely?
Corrective action for known or recurring problems was not performed at or within the proper time.
g)
OP4G - Is there evidence that needed changes to the plant were not approved or X
funded which resulted in a plant issue?
Corrective actions for existing deficiencies that were previously identified were not approved or funded.
h)
OP4H - Is there evidence that there was not a means or process to ensure procedures X
and documents were of adequate quality and up to date?
A process for changing procedures or other work documents to ensure quality and timeliness was nonexistent or inadequate
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 21 of 28 ATTACHMENT 9.5 EVALUATION FOR ORGANIZATION & PROGRAMMATIC ISSUES Sheet 7 of 14 O&P Worksheet Potential O&P Potential O&P Failure Modes (Causal Factors)
Failure Modes RC# AC#CC#
(Causal Factors)
Yes No i)
OP41 - Is there evidence that duties were not well distributed among personnel that X
contributed to a problem?
The work loading of individuals within a group or team did not adequately address training, experience, task frequency and duration, or other situational factors such that responsibility was inappropriately distributed.
j)
OP4J - Is there evidence that too few workers are assigned to perform a task that X
contributed to an issue?
Job planning did not allot a realistic number of man-hours based on the scope of work described.
k)
OP4K-Is there evidence that an insufficient number of training or experienced workers X
were assigned to a task?
Though the overall number of personnel assigned matched the planned man*hour allotment, organization methods failed to identify that personnel assigned did not have adequate experience or training to perform the work.
I)
OP4L-Is there evidence that there is a problem in perform repetitive tasks and sub X
tasks which contributed to a problem?
Work flow plan repeated tasks or sub tasks to the detriment of successful completion of the evolution.
m)
OP4M-Is there evidence that there was a less than adequate process for a X
configuration change to a design document?
Documentation generated as a result of a design change which renders the as-left configuration of affected components, structures and systems indeterminate.
n)
OP4N - Is there evidence that personnel exhibited insufficient awareness of the impact X
of actions on safety reliability because management failed to provide direction regarding safeguards against non-conservative actions by personnel concerning nuclear safety or reliability?
Management failed to provide direction regarding safeguards against non-conservative actions by personnel concerning nuclear safety or reliability.
o)
OP40 - Is there evidence that the planning process was not coordinated with inputs X
from walk downs and task analysis?
Job plan did not incorporate information gathered during field visits or task analysis concerning the steps and conditions required for successful completion of the task.
p)
OP4P - Is there evidence that previous industry or in-house operating experience was X
not effectively used to prevent problems and an event occurred because the information was not properly assimilated by the organization (missed opportunity)?
Industry or in-house experience relating to a current problem existed previous to the problem, but was not assimilated by the organization (missed opportunity).
- 5)
OPSX - Organizational Weaknesses X
Inadequate function or structure (poor internal organizational design that is missing vital functions)
Inadequate attention to emerging problems (organization doesn't pay attention to what is happening within it)
Inadequate work prioritization (organization doesn't prioritize their workloads so they waste time on unimportant things)
Inadequate communication within the organization (communication does not get up and down the organizational ladder)
Inadequate job skills, work practices, or decision making (organization had a problem because of its people)
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 22 of 28 ATTACHMENT 9.5 EVALUATION FOR ORGANIZATION & PROGRAMMATIC ISSUES Sheet 8 of 14 O&P Worksheet Potential O&P Potential O&P Failure Modes (Causal Factors)
Failure Modes RC# AC#CC#)
(Causal Factors)
Yes No a)
OP5A - Is there evidence of inadequate functions or structure which results in work not X
being performed due to a lack of organizational planning or inadequate staffing?
Specific issues or work are not performed or addressed, usually due to a lack of organizational planning or inadequate staffing.
b)
OP5B - Is there evidence of inadequate attention to emerging problems?
X Repetitive organizational crises in morale, work practice, or repeat events, etc. The causes are usually associated with a lack of strong self assessment, strategic planning, and root cause processes. Additionally, inadequate vertical information flow to the decision makers in the organization and inadequate prioritization of work can contribute to a breakdown in this area.
c)
OP5C - Is there evidence of an inadequate work prioritization process?
X Normally associated with staff work overload, over-run of the committed budget, and increasing backlog of work items. It is usually caused by inadequate work prioritization and inadequate vertical communication of the organization=s missions and goals.
d)
OP5D - Is there evidence of inadequate communication within the organization?
X Important issues are not being addressed or a breakdown of normal work processes has occurred.
Low staff morale is usually the long term result. Common causes for this include an inadequate information flow path, lack of a teamwork type of culture, or inadequate physical settings e)
OP5E - Is there evidence of inadequate job skills, work practices or decision making?
X Generally evidenced by low morale and excessive human error rates. Causes usually include a punitive management style, inadequate supervision, training, staff qualification, or vertical communication, and conflicting or unreasonable organizational goals.
f)
OP5F-Is there evidence that corrective actions for previously identified problems or X
event was not adequate to prevent recurrence (failed to take meaningful corrective actions for consequential or non-consequential events)?
Management failed to take meaningful corrective action for consequential or non-consequential events.
g)
OP5G - Is there evidence the supervisor was not properly notified of a suspected X
problem?
A problem requiring verbal communication with supervision arose, but was not verbally communicated to the supervisor.
h)
OP5H - Is there evidence of that pertinent information is not being properly transmitted X
verbally between the transmitter and the listener and vice versa?
The sender failed to verbally transmit information to the listener. This requires a sender and listener to be present, and is regardless of either individual's position in management chain.
i)
OP51 - Is there evidence that there are too many administrative duties assigned to X
supervisory staff to properly perform supervisory activities?
The administrative load on immediate supervisors adversely affected their ability to supervise ongoing activities j)
OP5J - Is there evidence that there is insufficient supervisory resources to provide the X
needed supervision to plant personnel?
Supervisions resource is less than that required by task analysis considering the balance of procedures, supervision and training.
k)
OP5K-Is there evidence that there is insufficient manpower to support the identified X
goals and objectives of the plant?
Personnel are not available as required task analysis of goal/objective.
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 23 of 28 ATTACHMENT 9.5 EVALUATION FOR ORGANIZATION & PROGRAMMATIC ISSUES Sheet 9 of 14 O&P Worksheet Potential O&P Potential O&P Failure Modes (Causal Factors)
Failure Modes RC# AC#CC#
(Causal Factorsl Yes No I)
OP5L-Is there evidence that sufficient resources are not provided to ensure adequate X
training is provided and maintained?
Training resources are not available as required by task analysis.
m)
OP5M - Is there evidence that there is not adequate availability of appropriate materials X
and tools to do the job?
A process for supplying personnel with appropriate materials or tools did not exist.
n)
OP5N - Is there evidence that there is not a means provided for ensuring adequate X
equipment and quality/reliability/operability for personnel equipment?
A process for ensuring personnel's equipment was satisfactory did not exist.
o)
OP50 - Is there evidence that personnel selection did not ensure an appropriate match X
to ensure a motivation for the worker?
Personnel selection processes failed to determine a mismatch between motivation and job description prior to task.
p)
OP5P - Is there evidence that tasks and individual accountability were not made clear to X
the worker?
Tasks (and the individual accountability for the task) that were outside written guidance or training were not made clear to the worker.
q)
OP5Q - Is there evidence that the progress and status of task is not adequately tracked X
by supervision?
Supervision did not take the appropriate actions to monitor the task progress or status.
r)
OP5R - Is there evidence that there is not an appropriate level of in-task supervision X
planned prior to the task being performed?
Supervision did not adequately assess the task for points of supervisory interaction prior to assignment to workers.
s)
OP5S - Is there evidence that direct supervisory involvement in the task interfered with X
the overview role of supervision?
Supervision became so involved with the actual task steps that overall command and control were adversely affected t)
OP5T - Is there evidence that emphasis on the schedule had an impact on doing a X
quality job and accepted standards were not met as a result of this emphasis?
Accepted standards for methods were not met due to supervision's focus on completing the activity within a certain time frame.
u)
OP5U - Is there evidence that job performance and self checking standards were not X
properly communicated to the organization performing the work prior to the job being performed?
Supervision failed to adequately communicate how standards for job performance and self-checking could be applied to the actual job at hand v)
OP5V-Is there evidence that too many concurrent tasks were assigned to the worker X
that were beyond the individual's abilities?
Supervision failed to detect that concurrent job assignments for an individual exceeded the individual's abilities.
w)
OP5W - Is there evidence that there is frequent job or task shuffling without adequate X
time to shift attention away from the previous task?
Supervision transferred a worker from one task to another without adequate time to shift attention away from previous task.
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 24 of 28 ATTACHMENT 9.5 EVALUATION FOR ORGANIZATION & PROGRAMMATIC ISSUES Sheet 10 of 14 O&P Worksheet Potential O&P Potential O&P Failure Modes (Causal Factors)
Failure Modes RC# AC# CC#
(Causal Factors)
Yes No X)
OP5X-Is there evidence that supervision did not consider the worker's need to use a X
higher order of skills that consider the workers talents and strengths?
Supervision did not consider the worker's talents or innovative strengths that could be used to perform more challenging work.
y)
OP5Y - Is there evidence that worker assignments did not consider the worker's X
previous task?
Supervision did not adequately assess the previous task's impact upon the worker's ability to implement the current task.
z)
OP5Z - Is there evidence that a workers assignment did not consider the worker's X
ingrained work patterns and necessary work patterns for successful completion of the current task?
Supervision failed to assess the incompatibility between worker's ingrained work patterns and necessary work patterns for successful completion of the current task.
aa) OP5AA - Is there evidence that there is too an infrequent contact with the workers to X
detect work habit and attitude changes?
Supervision not aware of deviation from desired work habits/attitudes due to lack of interaction with personnel.
bb) OP5AB - Is there evidence that supervision provides feedback on negative performance X
of an individual but not on positive performance?
Worker's performance adversely affected by supervision's focus on negative performance feedback.
cc) OP5AC - Is there evidence of a lack of teamwork as a result of inadequate training X
content?
Training content did not adequately address actions individuals must take in order for the crew or team as a whole to be successful.
dd) OP5AD - Is there evidence of a lack of evaluation of risk and consequences prior to X
making a change that would have an adverse impact as a result of the change?
Elements of the process change were not recognized as having adverse impact or increased risk of adverse impact prior to implementing the change.
ee) OP5AE - Is there evidence that personnel exhibited insufficient awareness of the impact X
of actions on safety and reliability?
Management failed to provide direction regarding safeguards against non-conservative actions by personnel concerning nuclear safety or reliability.
ff)
OP5AF-Is there evidence that causes of a previous event or known problem were not X
determined?
Analysis methods failed to uncover the causal factors of consequential or non-consequential events.
gg) OP5AG - Is there evidence that a response to a known or repetitive problem was X
untimely?
Corrective action for known or recurring problems was not performed at or within the proper time.
hh) OP5AH - Is there evidence that needed changes were not approved or funded that X
resulted in a plant problem?
Corrective actions for existing deficiencies that were previously identified were not approved or funded.
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 25 of 28 ATTACHMENT 9.5 EVALUATION FOR ORGANIZATION & PROGRAMMATIC ISSUES Sheet 11 of 14 ii)
OP5AI - Is there evidence that a means was not provided to ensure procedures and X
documents are of adequate quality and up to date?
A process for changing procedures or other work documents to ensure quality and timeliness was nonexistent or inadequate.
jj)
OP5AJ - Is there evidence that planning was not coordinated with inputs from walk X
downs and task analysis?
Job plan did not incorporate information gathered during field visits or task analysis concerning the steps and conditions required for successful completion of the task.
End of Form
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 26 of 28 Photo 1 Note in circled area that a lead to CT-5 for the B phase generator output is raised. This lead was found disconnect at the terminal lug and was raised for this photo. See Photo number 2 for additional detail.
Also note that large gauge wires are the Generator output leads. The Cables are lifted in the photo for meggar testing of the generator and associated circuits. These cables are heavy and stiff. Note that some cables are mechanically interacting with the CT wire harness.
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 27 of 28 Photo 2 Close up of CT -5. Note that the screw for the ring lug in the foreground positioned the lug so that it was sticking outward. The wire for the broken ring lug is routed with several sharp bends and is not in-line with the long axis of the wire. Also note that the connection for the intact lug is gently swept from the wire harness to the terminal with no sharp bends and that the wire remains in-line with the ring lug. Conversely the sharp bend to attach at the terminal adds stress to the connection. See photo 3 for ring lug break details
ATTACHMENT 9.1 TYPICAL HIGHER TIER APPARENT CAUSE EVALUATION & RESPONSE TEMPLATE Page 28 of 28 Photo 3 Ring lug break detail. The ring remained intact the lug broke at the shank right at the wire bundle ends. The break is on a slant and shows a small area where the lug was previously cracked. The tarnished color, in circled area, on the break surface indicates the prior crack most visible on the wire side of the break on the left side. The other surfaces are clean and show a copper color indicating a fresh break.