Text

Submittal of Revised Root Cause Investigation - Diesel Generator Amphenol Connector
	ML082050232
Person / Time
Site:	Cooper
Issue date:	07/21/2008
From:	Vanderkamp D; Nebraska Public Power District (NPPD)
To:	; Document Control Desk, Office of Nuclear Reactor Regulation
References
	IR-08-002, NLS2008059
	Download: ML082050232 (44)
	v • d • e

Nebraska Public Power District "Always there when you need us" NLS2008059 July 21, 2008 U.S. Nuclear Regulatory Commission Attention: Document Control Desk Washington, DC 20555-0001

Subject:

Revised Root Cause Investigation - Diesel Generator Amphenol Connector Cooper Nuclear Station, Docket No. 50-298, DPR-46

Reference:

Letter to Nuclear Regulatory Commission (NRC) dated June 19, 2008, "Cooper Nuclear Station (CNS) - Reply to Preliminary White Finding Regarding NRC Inspection Report 05000298/2008002"

Dear Sir or Madam:

The purpose of this correspondence is to submit Nebraska Public Power District's (NPPD) revised root cause evaluation regarding the diesel generator amphenol connector at Cooper Nuclear Station (CNS).

The referenced letter stated that CNS was performing further investigations that may challenge the basis of our root cause evaluation of the diesel generator amphenol connector. This was an internal initiative to understand the effects of vibration on the amphenol connector through detailed vibration testing performed by an outside vendor.

This action was taken based on a review of our corrective actions during implementation of our root cause corrective actions.

Results of the laboratory testing on the connector identified that vibration is not a time-based failure mechanism for the loosening of the amphenol connector. The root cause has been revised to reflect a most probable cause of insufficient worker attention during connector reassembly. The revised root cause evaluation is enclosed.

COOPER NUCLEAR STATION P.O. Box 98 / Brownville, NE 68321-0098 Telephone: (402) 825-3811 / Fax: (402) 825-5211 www.nppd.com

NLS2008059 Page 2 of 2 If you have questions, you may contact me at (402) 825-2904.

Sincerely, David W. Van Der Kamp Licensing Manager

/jf Enclosure cc: Regional Administrator w/ enclosure USNRC - Region IV Cooper Project Manager w/enclosure USNRC - NRR Project Directorate IV-1 Senior Resident Inspector w/ enclosure USNRC - CNS NPG Distribution w/o enclosure CNS Records w/ enclosure

NLS2008059 Enclosure Enclosure Root Cause Investigation (Rev. 1)

Unexpected DG-2 Shutdown CR-CNS-2008-00304 Dated 7/11/2008

NLS2008059 Enclosure ROOT CAUSE INVESTIGATION (REV. 1)

UNEXPECTED DG-2 SHUTDOWN CR-CNS-2008-00304 Investigation Team Dan Buman, SED Manager Responsible Manager Dave Barker, Root Cause Supervisor Root Cause Team Leader Mark Metzger, SED Root Cause Team Member Ray Rexroad, SED Root Cause Team Member Jeff Beatty, Security Root Cause Team Member Chuck Behr, Maintenance Root Cause Team Member Mike England, Work Control Manager Root Cause Team Member Responsible Manager Dan Buman Date: 7-11-08 CARB Chairperson Mike Colomb Date: 7-11-08

Table of Contents Executive Sum m ary ........................................................................ ............................................... 3 A nalysis ............................................................................................................................................... 5 Extent of C ondition/C ause ......................................................................................................... 13 O perating Experience .................................................................................. ;.................................. 13 Safety Significance ...................................................................... ..................................................... 16 Appendix A : C ausal A nalysis Background Inform ation ........................................................ 21 I

Appendix B: C A PR Due-date Risk Assessm ent ....................................................................... 26 Appendix C : C orrective Actions ............................................................................................... 27 Appendix D: Laboratory Report on MPU Vibration Testing ................................................ 29 2

EXECUTIVE

SUMMARY

NOTE: This is a revision to the investigation approved on 2-12-08 due to receipt of additional information obtained through laboratory testing of the connector which refutes vibration as a failure mechanism, as well as additional analysis to further explore the human performance aspects associated with the event.

Event Description On 1-15-08, DG-2 was started from the Control Room per Procedure 2.2.20.1 for post maintenance testing for WO 4610296. Normal voltage and frequency levels were achieved and the unit ran for -60 seconds, and then shut down. No abnormal alarm conditions were received at the DG control panel or in the Control Room. The "run" and "cranking" lights on Panel C were noted to cycle several times during the shutdown.

Problem Statement What should be DG2 relay tachometer speed sensing circuit should maintain continuity when in service What is Intermittent connection in the DG2 relay tachometer speed sensing circuit What is wrong Intermittent connection in relay tachometer speed sensing circuit resulted in unexpected shutdown during post-maintenance testing on 1-15-08 Consequences Shutdown represents a load failure of DG2 Immediate and Interim Actions Taken

Troubleshooting performed 1-16-08 per WO 4610394 S0DG1 run to confirm no common mode failure

" Amphenol-type connection reassembled on DG2 relay tachometer speed sensing circuit perWO 4610394.

Performed inspections/tightness checks on other DG1 and DG2 amphenol-type connectors Note: The electrical connector discussed in this report is manufactured by ESC Electrical (ESC type MS3057-4A). However, the type of connector utilized is commonly referred to by another brand name (Amphenol) that has become synonymous with this type of connector. For clarity, when not addressing the specific manufacturer of the CNS connector, they will be referred to as "amphenol-type" connectors in this report.

3

Summary of Conclusions Table of Elements, Causal Factors, & Corrective Actions Element Description Action Most Insufficient worker attention A - (Action) Develop lessons learned Probable applied to correctly perform from this event as it relates to Cause reassembly of the DG2 Relay application of sufficient attention to Tachometer speed probe detail when removing & reassembling connector electrical connections in critical, applications. Present the lessons learned to appropriate Maintenance personnel.

Extent of Governor and relay tachometer Addressed in immediate actions taken Condition speed sensors for both DG1 and DG-2 Extent of Applies to workers responsible for Addressed in Corrective Action A Cause connector reassembly - I&C Maintenance CAER Corrective action effectiveness D - (CAER) Verify completion of above review corrective actions.

Other OE review of CR-CNS-2007-00480 CR being initiated to address outside of CA-8 indicate that scope of this report actions taken were too narrow Need to determine if CNS B - (ENHN) Benchmark best practices to practices for control of electrical establish controls for ensuring removed connections of this type reflect connectors in critical applications are industry standards properly reassembled. Create subsequent corrective action(s) to implement benchmark results.

C- (ENHN) Revise procedures 14.17.1 and 14.17.2 to provide the specific connector identifications 4

Analysis Summary of Techniques Utilized and Investigation Approach Data Gathering - Personnel statements/interviews, document reviews (CNS procedures, drawings, logs), OE documents, and communication with other plants.

Analysis Techniques - Failure Mode Analysis, Humdn Performance Analysis and Event & Causal Factor Charting.

Two primary areas of interest were pursued in this investigation. The first was to identify the failure mode and mechanism that resulted in the DG2 shutdown. The second was to determine how/why the componentwas in the state in which it was found.

Analysis Details Failure Mode After the unexpected DG2 shutdown, an FMEA was generated (see Appendix A) which was used to develop a troubleshooting plan. The resulting troubleshooting performed on 1-16-08 identified a loose amphenol-type connector to the relay tachometer speed sensor (DG-SE-3143). The as-found condition was as follows:

The coupling nut was observed to be engaged -1/4 turn (compared to an additional -5 turns to full engagement/tight). Note that there were differing recollections of the as-found connector ring engagement from the technicians that were present. No measurement was taken.

The connection was felt to be loose when manually manipulated

Resistance measurements showed instability when the connector was moved Y(e~

5

The loose relay tachometer speed sensor connection is deemed to be the failure mode that resulted in the unexpected DG2 shutdown based on the following:

A loose connector will result in the same symptoms and equipment responses as was noted during the DG2 runs.

The recorded information gathered during the 1-16-08 0327 DG2 run confirmed that the shutdown "command" came from the relay tachometer.

" Observation of the local RPM meter during the 1-16-08 0327 DG2 run indicated that the relay tachometer output did not increase beyond -400 RPM.

" After tightening the loose connector, a subsequent DG2 run was performed 1-16-08 at 1645 with a recorder connected to relay tachometer pick-up inputs. Resulting data showed a clean waveform with no noted abnormalities. Also noted during the run that the local RPM display indicated the proper value of 600 RPM.

To rule out other potential problems in the connector, the connector was disassembled and checked for.

Broken or bent pins,

Bad solder connections,

" Broken wires,

Strain points in the cable, and

Cleanliness of the connector and pick-up assembly.

No abnormal conditions were noted.

Failure Mechanism The next analytical question becomes, how did the connector become loose to the point of intermittent continuity? Four possibilities were considered:

Loose connector to DG2 relay tachometer speed sensor (1) (3)

(2) Connector (4)

Connector not Connector inadvertently Deliberate tigtenafte vibrated loose loosened by tampering maintenance human action

1) Connector not tightened after maintenance Supporting data:

Review of maintenance work history on the relay tachometer magnetic pick-up indicated the following:

o 12/2000- Replaced DG2 magnetic pick-up and relay tachometer (no work on connector, but would have to be removed to replace magnetic pick-up). Re.

WO 00-3915 6

o 12/1995 - Repaired connector for relay tachometer magnetic pick-up. Loctite 242 was applied to the connectors for both the governor magnetic pick-up and the relay tachometer magnetic pick-up due to Condition Report 1-19594 in which the author'noted the DG2 governor magnetic pick-up had "vibrated" loose during a run. Re. WO 95-04362 and WO 95-03959

Connector reassembly is considered a skill-of-the-craft evolution and is not a separate operation in work order instructions.

e No check is performed following connector reassembly

DG2 operated successfully the day prior during the normal monthly surveillance, and the failure was observed the following day (after performing maintenance) during post work testing. There were no signs of electrical instability or intermittent connection during or prior to the 1/14/08 surveillance run. After maintenance was performed on 1/15/08 (for governor oil sight glass repair), a post-maintenance run was performed on 1/15/08 which was when the problem resulted in the DG2 shutdown.

The discontinuity was repeatable on subsequent runs until the probe was restored.

See Safety Significance section for additional details.

Refuting data:

None

==

Conclusion:==

Likely failure mode

2) Connector vibrated loose Supporting data:

Previous Condition Report (CR 1-19594 written 12-11-95) whichstated "Amphenol connection to MPU for governor is vibrating loose during DG run". Condition noted to have been discovered 12-10-95 @ 1500 while "looking for reason for DG load spiking".

Associated Minor Maintenance History Index Sheet states the following under "work completed":

Loosen both MPU amphenol connectors and applied Loctite 242 sparingly ensuring Loctite did not come in contact with wire insulation. Tighten connectors firmly.

Document indicates that this was performed on both the magnetic pick-up connectors for both DG1 and DG2.

Refuting data:

Different connector than that found loose in this (the 2008) event

Testing performed by MPR of the actual connector with only minimal engagement at 10 times the level of vibration did not show any signs of backing off (see App. D)

No supporting data for the conclusion in the 1995 CR that vibration led to the loose connector.

Lack of internal or external operating experience that indicates vibration has led to Amphenol-type connectors loosening. There are a large number of these connectors used at Cooper and in the industry. The one anecdotal Cooper CR from 1995 and the 2003 Browns Ferry failure that was partially attributed to vibration (but was unsubstantiated by supporting facts), indicate that vibration has not been a failure mode that challenges the electrical integrity of the connection.

Conclusion:

Not a likely failure mode

3) Inadvertent loosening by human action 7

Supporting data:

The governor magnetic pick-up sensor/connector is located in close proximity to the relay tachometer pickup, sensor/connector that was found loose. That connector is periodically disconnected as part of procedure 14.17.1, DG2 Annual Calibration (and the sister procedure for DG1, 14.17.2) Step 9.8, which states, "At DG-EHO-EHOV2, disable electrical governor by disconnecting electrical connector (south side of the actuator)". Both connectors have labels attached at the connectors which read as follows:

o "DG-SE-3143" for the relay tachometer speed sensor, and o "DG-SE-DG2" for the governor speed sensor.

It is noteworthy that the procedural guidance does not provide the component identification for the specific connector. DG-EHO-EHOV2 is a designator for "DG2 Hydraulic Actuator" per SAP, and is not specific to the connector. The procedure does contain a specific step for restoring the connector, including independent verification.

The most recent performance of this procedure as a PM was on 3-14-07according to SAP data.

Refuting data:

The magnetic pickup connector is located on the side of the engine near the top at the southwest corner. Based on the location of the connector and the number of turns it was found from full-tight makes it unlikely that inadvertent contact with the connector coupling nut (e.g., bumping or stepping on) caused the connector to be in the as-found condition.

==

Conclusion:==

Possible failure mode

4) Deliberate tampering Supporting data:

Deliberate human action could result in the as-found condition Refuting data:

0 Investigation by Security under CR-CNS-2008-00329 found.no physical evidence of tampering (tool marks, damaged paint, missing parts) 0 The knowledge needed to pinpoint this instrument to disable DG2 is not common knowledge 0 Security Patrol Officers that toured'the diesel generator rooms reported no suspicious activity

==

Conclusion:==

Not a likely failure mode Failure Mode

Conclusions:

Review of industry operating experience has shown no industry experience with loosening of amphenol-type connectors due to vibration. Prior OE with this as a failure mode for loose connectors has not been substantiated either by internal or external testing. MPR Associates has performed vibration testing on the MPU/connector assembly at a test facility (see Appendix D) using parameters supplied by NPPD that were taken directly from DG2. The report identifies that after 250 hours0.00289 days 0.0694 hours 4.133598e-4 weeks 9.5125e-5 months of exposure to vibration levels well in excess of that seen on DG2, the connector has shown no discernable loosening throughout the test. This includes vibration testing with thread engagement similar to that in the as-found condition following the failure. In addition, this 8

testing demonstrated that it takes positive force to connect or disconnect the electrical connector from the MPU.

Based on this testing, it is concluded that the failure mechanism for the MPU connector becoming loose is not time-based vibration induced.

Deliberate tampering and vibration-induced connection loosening are refuted. This leaves the most likely premise that the relay tachometer speed sensor connector was loose due to,

1. Improper reassembly following previous maintenance or troubleshooting activities, or

2. Inadvertent disconnection of the relay tachometer speed sensor connector instead of the governor speed sensor connector, and failure to properly reassemble.

Although when the failure to properly re-assemble the connector occurred is not known it is more likely to have occurred in the recent past due to:

1. The testing has demonstrated the loosening is not a result of time based vibration.

2. The testing has demonstrated that pin connection alone is sufficient to maintain solid contact.

3. The testing has demonstrated that to disengage the pins requires positive action.

4. Monthly surveillances include intrusive checks (such as looking for oil leaks, etc.) that have the potential to disturb this connection.

5., Major maintenance includes activities that could have disturbed this connection.

6. Any electrical discontinuity of between the MPU and the relay tach (from disturbance of the connection) would have been self revealing during normal monthly surveillances.

Causal Analysis The ensuing question is why the connector was not reassembled.

A review of the work orders associated with the 1995 and 2000 work done on this connector and interviews with Work Control personnel indicate that separate operations for reassembly of the connector is not provided in Work Order instructions. Reassembling of the connector is considered a skill-of-the-craft evolution.

There are maintenance plans associated with the governor magnetic pickup and the relay tachometer. Applicable maintenance plans:

Maintenance Plan 800000014879 (PM 10445) checks and cleans the DG Governor magnetic pickup every 5 years, but is not performed on the Relay Tachometer magnetic pickup.

Maintenance Plan 800000005976 (PM 01446) calibrates the Relay Tachometer and Speed Indicator per Procedure 14.17.10, but the procedure does not direct disconnecting the magnetic pickup Amphenol connector.

In addition, other maintenance tasks can be performed which are done skill-of-the-craft and may result in disconnecting a connector without explicit documented guidance and/or checks for proper reassembly.

Human Performance The following are possibilities for the connector to have been left in the as-found condition:

. A human performance error associated with 9

o Inadvertently disconnecting the wrong probe (and not subsequently reconnecting it), or o Not applying sufficient attention following a skill-of-the-craft task to ensure the connector was properly reassembled The NUREG 6751 Human Performance Evaluation Process (HPEP) cause tree was utilized to assist in identifying the causes associated with human error.

10

Yes

8. Fitness for Duty I

Did they possess the Yes

9. KSAs Personnel required knowledge, skills Were they paying and abilities?

No I attention and motivated to perform correctly? 10. Attention/Motivation Were procedures and Yes 11. Procedures reference documents Were necessary tools, accurate, available, used?

Were the right equipment and PPE Yes I resources t 12. Tools/Eciuinment available and used?

Resources provided to Yes Yes Was the number of staff perform the task? 2 assigned appropriate for the task? "1 13. Staffmn Was the type and amount I

Yes of supervision appropriate for the task? *1I

14. Supervision I Work Could the task be Did the human-system 91 i W 15. HSI performed easily ? interface meet I Environment in the work information and control Were environmental environment? I conditions within Yes expected/comfortable

16. Task Environment limits?

I Communica-tion and Coordination Figure 7-1 The HPEP Cause Tree 10

Resources - The task did not involve special tools or oversight, therefore this path is not considered applicable.

Communication/Coordination - The task did not involve interaction with other personnel, therefore this path is not considered applicable.

Personnel -

There is no evidence or (due to the unknown time in which the error occurred) a way to establish whether the individual was unfit for duty. Therefore the presumptive conclusion, based on the CNS fitness-for-duty program, is that the individual was not unfit for duty.

Based on the lack of complexity, the error would involve insufficient application of attention to the task at hand.

Work Environment-

Although the work area is somewhat tight quarters, it is not deemed to pose an excessive risk to satisfactory performance of this task.

The human/system interface may pose a challenge to successful task completion for the following reasons:

o The MPU and Relay Tachometer probes are in close proximity to each other o Current CNS practicesallow performance of connector removal and reinstallation as a skill-of-the-craft task Conclusions Due to the inability to pin-point when or under what conditions the error occurred,.

determination of a definitive root cause is not possible. Therefore, the most probable cause is determined to be the following:

Insufficient worker attention applied to correctly perform reassembly of the DG2 Relay Tachometer speed probe connector Whether the CNS practice that allows performance of connector removal and reinstallation as a skill-of-the-craft task reflects a deviation from industry standards is unknown at this time. Actions will be provided to compare CNS practices to industry practices and to reconcile deviations in standards.

Although we are unable to pin-point when or under what conditions the MPU connector was not properly assembled, the safety significance section provides further discussion that demonstrates the actual electrical discontinuity did not occur prior to or during the successful monthly surveillance that was performed the previous day prior to the maintenance window. Also, the failure to improperly assemble the connector was more likely to have occurred in the recent past due to:

1. The testing has demonstrated the loosening is not a result of time based vibration.

2. The testing has demonstrated that pin connection alone is sufficient to maintain solid contact.

3. The testing has demonstrated that to disengage the pins requires positive force.

4. Monthly surveillances include intrusive checks (such as looking for oil leaks, etc.)

11

that have the potential to disturb this connection.

5. Major maintenance includes activities that could have disturb this connection.

6. Any electrical discontinuity of between the MPU and the relay tach (from disturbance of the connection) would have been self revealing during normal monthly surveillances 1211995 1212000 111512008 111512008 Barrier Analysis Key:

Fallure Occurrence Mode I Most Probable or Step Failure Cause Mechanism Failed B a rrier 12

Extent of condition/cause Extent of Condition Based on similarity of location and equipment type, the extent of the condition of loose amphenol-type connectors includes the governor and relay tachometer speed sensors for both DG1 and DG-2. These were verified to be tight as an immediate action.

Extent of Cause For the most probable cause associated with the worker attention to detail, the extent of cause is bound by the workers that perform probe reassembly - I&C Maintenance. A corrective action is provided to provide lessons learned to I&C Maintenance personnel.

Operating experience summary Internal OE The internal OE search included both work order history and CR searches.

12/2000 (WO 00-3915) - Most recent work on the DG2 relay tachometer where the DG2 magnetic pick-up and relay tachometer were replaced (no work on connector, but would have to be removed to replace magnetic pick-up).

12/1995 (CR 1-19594, WO 95-04362 and WO 95-03959) - CR noted that while looking for reason for DG2 load spiking, the governor magnetic pick-up connector was noted to be loose. CR states that connection is "vibrating loose during DG run". The CR was designated as a work item to apply Loctite sparingly to the connector coupling nuts for the DG1 and DG2 magnetic pick-up probes. Note that there is some ambiguity in the documentation on the scope of the work (i.e., was the work done on both MPUs for both DGs?). The work authorization was approved on 12-11-95 (Minor Maintenance #

95-03959) with the annotation that the work was authorized by the WCC Supervisor/WCC SRO for "#2 DG only". However, in the completion notes there are two sets of statements that indicate the Loctite was applied to both MPUs on both DGs.

"Loosen both MPU amphenol connectors and applied Locktite 242 sparingly ensuring Loctite did not come in contact with wire insulation. Tighten amphenol connectors firmly."

This was signed and dated on 12-11-95, and was followed by another entry:

"Loosened both MPU connectors on DG 1, applied Locktite 242 and reassembled per instructions, sat. Returned to service."

This was signed and dated on 12-15-95.

Discussion with the CR author indicated that the conclusion that the connection was loose due to vibration was based on a personal observation and was not derived from testing or physical evidence.

13

CR-CNS-2007-00480 documented an unexpected trip of DG2 during a monthly run in January 2007. Although the root cause was determined to be a manufacturing defect in a diode on the voltage regulator card, initial troubleshooting after the event did identify several loose connections (where individual wires were landed) which were subsequently determined to not have contributed to the event. However, due to the potential for one of those connections causing a VAR or voltage excursion under different conditions, a corrective action was assigned to establish PMs to periodically perform physical checks of electrical connections in the DG1 and DG2 control systems and to use thermography to detect "hot spots" in DG2 and DG2 electrical control sy stem connections.

The response to this corrective action (CR-CNS-2007-00480 CA-8) included an evaluation to establish the extent of the PMs (since the action was for DG control circuits and checking all terminations on the DG was deemed excessive). Three assumptions formed the basis for the ultimate scope of this effort:

1. Some terminations, if loose, would not adversely affect DG operation. An example is a switch that controls a function that is bypassed during emergency operations.

Also included is an assumption - the assumption that a wire with a loose fastener will not move from its location. This is due to the fact that the wire is "trained" or formed into a position, and the wire will not move on its own away from its termination.

2. It is assumed that all on-engine/skid components have other maintenance actions for calibration or periodic replacement, etc. Therefore, based on this assumption, none of these components will be separately inspected.

3. Vibration can cause terminations to loosen. The farther away from the engine, the less the vibration. For those components that are not on or adjacent to the engine vibration levels should not be sufficiently strong to warrant inspections (the OMAS (the switch found with loose terminations) is located remote from the engine (in the metering and relay panel), which is adjacent to the engine).

Based on the occurrence of this event, the above assumptions will need to be reanalyzed to identify additional scope.

Note that the comment for Assumption #3 that vibration can cause terminations to loosen was not applicable to this type of connector. Also note that with the completion of the laboratory analysis that refutes time-based vibration as a failure mode for this event, the above observations to not relate to this event. An additional CR is being initiated to address this.

External OE 12/2007 (San Onofre) - Event in which an emergency diesel generator failed during surveillance testing. San Onofre plant personnel were contacted and discussions yielded the following:

There was an intermittent speed probe signal

Trouble was initially thought to be grid induced equipment response, partly due to the fact that there plant information system onrly sampled at once per second, so that the resultant spikes were very small and of short duration. In addition, at "about the same time", the grid behavior was "abnormal".

14

Later in the month, the input signal "went away", and the DG went to maximum load.

, After the December failure, an instrumented run was performed. The channel for the magnetic pickup displayed signal dropouts.

Upon disassembly of the probe connector, found one of the pin terminations had an obviously inadequate solder connection. "It looked like one of the wires was tinned, and then inserted into the plug pin, and soldered without adding any additional solder". The'adjacent conductor was appropriately soldered.

The solder connections for the Cooper connector were examined during troubleshooting per WO 4610394 with no deficiencies noted (see Page 6 for details of inspection results). This OE is therefore considered to not be relevant to the event currently being investigated.

04/2004 (J.A. Fitzpatrick CR 2004-01590) - CR describing that during troubleshooting of a Containment Air Monitor problem, found and amphenol detector cable loose. When an I&C technician went to remove the connector the day before to remove high voltage from the detector, it was noticed the connection was tight, but the connector did not seem to be seated properly as it only took a few turns to remove the connector (this style connector has extensive thread engagement). The connector was disconnected and reconnected properly and another Work Order initiated to inspect the connector in the future.

05/2003 (Browns Ferry 2, LER 2003-002) - HPCI turbine issue where the speed sensor amphenol failed and the turbine kept sensing 0 speed, and the control system kept calling for more steam (opening the valve fully), resulting in high HPCI flow which in turn resulted in the system isolating the HPCI system and tripping the turbine. The amphenol was an Amphenol 97-3106 (similar to the Cooper DG2 relay tachometer connection, the primary difference being the Browns Ferry connector is an angled connection vs.

straight connection). Browns Ferry attributed the failure to "age, excessive wear, and vibration". Their corrective actions were to replace the connector, look at the similar connectors on the other HPCI and RCIC systems, revise procedures to visually inspect the connectors (doesn't specify when or how often), revise post-maintenance instructions to ensure connector continuity is demonstrated, and added PMs to periodically replace the HPCI/RCIC speed sensing amphenols.

04/2003 (J.A. Fitzpatrick CR 2003-01942) - CR describing that a Work Request documented a loose amphenol as the cause for corrective maintenance on a component. CR states that the connector was tightened and the CR was closed via "admin closure".

Repetitive Event Analysis A repetitive event is defined in Procedure 0.5 as "Any significant condition adverse to quality that resulted from the same identified root cause as a previous event or non-conformance for which CAPRs were established within the past 5 years."

Because the root cause is not the same as a previously identified root cause and no previous CAPRs were established to deal with this issue, this is not a repetitive event.

15

safety significance Actual Safety Consequences

DG2 unexpectedly shut down during the monthly run and resulted in declaring DG2 inoperable pending troubleshooting and repair. The shutdown represents a load failure in performance indicators. DG1 was available and operable for the duration of the event.

Potential Safety Consequences The primary question to answer is how the existing condition would or could have been worse under different plant operating conditions. Had DG1 been inoperable and/or unavailable, the condition rendering DG2 inoperable is clearly more severe as reflected in the plant Technical Specifications, until such time as the problem was diagnosed and repaired.

What if the event occurred during an emergency start/run demand?. Would the DG start and perform its function, and would the DG continue to run?

Would the DG start and perform its function?

Discussion The DG logic includes various local and remote annunciations. The control logic interfaces with the annunciation logic so that "run alarms" are not received when the DG is not running. An example is the CONTROL AIR PRESSURE LOW alarm. The pressure switch which is annunciated is pressurized when the DG is running. When the DG is not running, the pressure switch is not pressurized, and the annunciator system would "see" that the contacts of the pressure switch are open. An open contact would cause an alarm to be received.

However, when the DG is not running, an unpressurized pressure switch is a normal condition. In order to only receive an alarm when the pressure switch~contact is open AND the DG is running, a contact from the DG start/run/shutdown logic is placed in parallel with the pressure switch contact. Given this design, the start/run/shutdown logic provides a closed contact when the DG is not running, so that the annunciator will not be continuously received. When the DG is running, the start/run/shutdown logic contact opens so that a closed pressure switch contact would then be sensed, and alarmed, if a loss of air pressure in the control air system would occur.

For DG2, this annunciator point is 3622. This annunciation is normally received for every DG shutdown. When the DG is shutdown, the pressure switch contact closes first (and thus alarms), followed by the start/run/shutdown logic contact closing at 550 RPM, which cancels the sensed alarm condition. Once the start/run/shutdown logic contact closes (as the DG speed remains less than 550 RPM), the annunciator cannot again be "sensed" until the DG is again running (and speed rises above 550 RPM).

A review of annunciator point 3622 data history clearly indicated that point 3622 is 16

normally received one time for each DG shutdown (point 3612 for DG1 was also reviewed, and the equipment response is identical - one alarm for each DG shutdown). This same history also clearly demonstrates that the expected CONTROL AIR PRESSURE LOW alarm was not received on 1/15/2008, when the DG unexpectedly shutdown, and was not received on 1/16/2008, when the DG was operated for troubleshooting purposes. This alarm was received at the end of the operability run performed on 1/14/2008. This alarm has also been received at the end of the DG runs that had been performed for the prior six months.

What does this mean?

The data for the duration of each annunciation was collected and plotted. The plots indicate that the average duration for DG1 is approximately 4.8 seconds (and is steady),

and the duration for DG2 is approximately 5.3 seconds (and is marginally "noisier"). The duration for DG2 on 1/14/08 dropped to about 3.4 seconds.

This information indicates that the Relay Tachometer had been, for prior DG runs, providing a change-of-state output for each DG shutdown per its design. Since this change-of-state did occur for each shutdown, the Relay Tachometer (and it sensed input) had been operational in prior DG runs, including the 1/14/08 run. This information may or may not have been an indication of the connector became loose during engine coastdown.

This information indicates that for the runs prior to and including 1/14/08, the duration was "normal" and DG2 was capable of starting and performing its function. The earliest documented instance of anomalous magnetic pickup operation is during the engine coastdown on 1/14/08.

Would the DG continue to run?

Scenario One: The DG has started in response to an emergency LOCA start signal, the initiatingsignal for the DG start has not cleared, and the magnetic pickup/relay tachometer responds exactly as they did on 1/15/2008 and 1/16/2008. The following text is based on circuit analysis as follows (Re. drawing G5-262-743 SH 10):

1. DG has started and is running.

2. The relay tachometer momentarily senses less than 280 RPM.

3. Relay R102 momentarily de-energizes.

a. When Relay R 102 momentarily de-energizes, Relay 14RX3 momentarily de-energizes.

i. When contact 1-3 of Relay 14RX3 opens, Relays 14RY1dnd 14RY2 de-energize.

1. When contact CI-C2 bf Relay 14RY1 closes, a second power path is provided to the "normal run relays" (Relays 4MX, 4MX1, 4MX2, 4MX3, and 4MX4) as well as from the emergency start relay" (Relay 4EMX3).

2. When contact C3-C4 of Relay 14RY1 closes, nothing occurs (the annunciator circuit alarms on an open circuit).

3. When contact C7-C8 of Relay 14RY1 closes, the Control Room CRANKING light comes on.

17

4. When contact 1-3 of Relay 14RY2 opens, nothing occurs.

When the contact closes, however, the fuel oil booster pump will start and run for three seconds).

5. When contact 5-6 of Relay 14RY2 closes, the fuel oil booster pump will start/run.

6. When contacts 8-11/9-11 of Relay 14RY2 change state, room HVAC will cycle.

ii. When contact 6-7 of Relay 14RX3 opens, power is NOT removed from the "normal run relays" (Relays 4MX, 4MX 1, 4MX2, 4MX3, and 4MX4) as well as from the "emergency start relay" (Relay 4EMX3).

iii. When contact 8-11 of Relay 14RX3 closes, energizing Relay 62CLX.

b. When Relay R102 momentarily de-energizes, Relays 14RX4 and 14RX5 momentarily de-energize.

i. When contact 1-4 of Relay 14RX4 closes AND contact 1-4 of Relay 14RX5 closes, the right bank starting air admission valves open.

ii. When contact 5-6 of Relay 14RX4 closes AND contact 5-6 of Relay 14RX5 closes, the left bank start air admission valves open.

iii. When contact 8-11 of Relay 14RX4 closes, the engine start counter advances by one.

iv. When contact 9-11 of Relay 14RX5 closes, the RED light in the Control Room comes on.

4. Relay R103 momentarily de-energizes.

a. When Relay R103 momentarily de-energizes, Relay 14S1 momentarily de-energizes.

i. When contact 1-7 of Relay 14S1 opens, the breaker closure permissive for the DG output breaker is removed. This has no effect on the breaker.

ii. When contact 4-8 of Relay 14S1 closes, the CONTROL AIR PRESSURE LOW annunciator would alarm.

b. When Relay R 103 de-energizes, Relay 14S2 de-energizes.

i. When contact 3-7 of Relay 14S2 closes, Relay 14S3 energizes, and the Auxiliary Lube Oil Pump would start.

ii. When contact 4-8 of Relay 14S2 closes, the TURBO BEARING WEAR annunciator would alarm.

5. RESULT - DG2 CONTINUES TO RUN.

Scenario Two: The DG has started in response to a non-LOCA emergency start (i.e.,

an under voltage signal), and the magnetic pickup/relay tachometer responds exactly as they did on 1/15/2008 and 1/16/2008. The following text is based on circuit analysis as follows (Re. drawing G5-262-743 SH 10):

1I DG has started and is running.

2, The output breaker closes and reenergizes the bus and clears the emergency start signal.

3, The relay tachometer momentarily senses less than 280 RPM.

4, Relay R102 de-energizes. (NOTE: Other relays change state, but since this logic chain causes the DG to shut down, the other information is not included in this discussion).

18

5. When Relay R102 de-energizes, Relay 14RX3 de-energizes.

6. When contact 6-7 of Relay 14RX3 opens, power is removed from the "normal run relays" (Relays 4MX, 4MX1, 4MX2, 4MX3, and 4MX4) as well as from the "emergency start relay" (Relay 4EMX3).

7. DG2 SHUTS DOWN.

8. DG2 would automatically restart in response to the resultant under voltage start signal. Given the repeatable behavior of the magnetic pickup/relay tachometer, the DG would subsequently shutdown, and the process would repeat until the high drywell signal comes in from the loss of drywell FCUs.

9. After one or two cycles, operators would actuate the emergency stop to prevent the DG from. cycling.

10. Based on simulator, the high drywell pressure signal is actuated in 5 minutes.

11. One of the first steps of troubleshooting, operators reset the emergency stop.

12. DG2 automatically starts and now has same response as scenario 1.

13. RESULT DG2 CONTINUES TO RUN Safety Significance Conclusion This condition is judged to have resulted in only minor safety significance. This is based on:

1. This would not have resulted in a complete loss of DG2.

a. The as found condition of the probe demonstrates that the electrical discontinuity was only momentary and of short duration as the DG was restarted as part of troubleshooting and the condition did not repeat until 52 seconds later.

b. The logic would have automatically restarted DG2 as soon as troubleshooting began with only minimal interruption.

2. Based on PMIS data, the electrical discontinuity did not occur PRIOR to, or DURING the successful surveillance test conducted the day before.

a. Most likely the electrical discontinuity was a result of the maintenance window that occurred just prior to the post work testing where the DG unexpectedly shutdown.

3. The failure to properly assemble the connector is more probable to have been recent event as.

a. Testing demonstrates vibration is not a time based failure mode.

i. As found most likely same as left condition of connector

b. ANY electrical discontinuity would have been self revealing during monthly surveillances.

i. Monthly surveillances have high likelihood of disturbing the connector due to activities such as checking for oil leaks, etc.

ii. Maintenance activities performed on the engine have high likelihood for disturbing this connection.

19

Appendix A: causal analysis background information Failure Modes and Effects Analysis performed during event response:

Possible Cause Refuting Evidence Supporting Evidence Follow-Up Actions Comments Possible Cause Facts: Voltage and frequency came up to normal levels as expected and the unit ran for approximately 40 seconds, then shutdown. No abnormal alarm conditions were received at the DG control panel nor in the Control Room. The "run" and "cranking" lights on panel C were noted to cycle several times during the shutdown.

Overspeed Governor Overspeed gov did not actuate. Last component worked No mechanical NO Overspeed butterfly valves did on. Oil drained during shutdown signal not actuate. Overspeed fuel repair of sight glass. was present shutdown valve was not tripped.

Freq trace 599 RPM. Need 655 electrical trip. Mechanical trip 665.

2 Plugged fuel filter Voltage response would not have NO dropped out. Engine speed was steady until trip. Would expect decrease in engine speed or oscillations if fuel problem.

Subsequent DG loaded runs were acceptable without intervening fuel filter maintenance.

3 Fuel oil strainer clogged. Voltage response would not NO have dropped out. Engine speed was steady until trip. Would expect decrease in engine speed or oscillations if fuel problem.

Subsequent DG loaded runs were acceptable without intervening fuel filter maintenance.

20

4 Voltage regulator Voltage regulator given signal to NO shut down. OSIPI traces support no voltage regulator problem.

Startup trace same as previous start. S/D trace same as previous S/D. Voltage output at 0 will not cause DG shutdown. Voltage regulator output and engine speed reduced at same time, and .

comparable to a normal shutdown.

5 Components in run logic Took voltage measurements post Intermittent operation of NO run. As found-was SAT. "logic chain" components remains possible.

5a Relay 4MX Instrumented troubleshooting run Intermittent contact NO indicated that relay tachometer opening is electrically .

contact opened, causing the equal to a "shutdown equivalent of a "shutdown command".

command". Concurrent failure not credible.