IR 05000245/2014013
| ML15015A078 | |
| Person / Time | |
|---|---|
| Site: | Millstone  |
| Issue date: | 01/15/2015 |
| From: | Rogge J F Engineering Region 1 Branch 3 |
| To: | Heacock D A Dominion Resources |
| References | |
| IR-2014013 | |
| Download: ML15015A078 (31) | |
Text
January 15, 2015
Mr. David Heacock President and Chief Nuclear Officer Dominion Resources Millstone Power Station 5000 Dominion Boulevard Glen Allen, VA 23060-6711
SUBJECT: MILLSTONE POWER STATION UNIT 3 - NRC SPECIAL INSPECTION REPORT 05000423/2014013
Dear Mr. Heacock:
On December 9, 2014, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Millstone Power Station, Unit 3, in response to two failures of the turbine driven auxiliary feedwater (TDAFW) pump to start within the required time period during surveillance tests. The Special Inspection Team (SIT) Charter (Attachment 1 of the enclosed report), which the Team leader discussed on September 15, 2014, with Site Vice President Mr. Stephen Scace and other members of your staff, provides the basis and additional details concerning the scope of the inspection. The enclosed report documents the Team's inspection activities, observations and results, which were discussed on September 19 and December 9, 2014, with your Site Vice Presidents, Messrs. Scace and John Daugherty, respectively and other members of your staff. The Team examined activities conducted under the conditions of your license and NRC rules and regulations, related to safe operation of Unit 3, particularly with respect to the investigation and corrective actions taken by Dominion to address failures of the TDAFW pump to start in July 2014 and again in September 2014. The Team observed and reviewed complex troubleshooting activities, interviewed personnel concerning data collection and the use of test instrumentation and reviewed selected operations, test and maintenance procedures. The Team assessed causal factor analysis, relevant performance history, and extent-of-condition reviews to determine the significance and potential consequences of these failures. The Team also reviewed maintenance, modification activities, and testing conducted during the Fall 2014 refueling outage, which replaced TDAFW electrical control components and modified the overspeed shutdown circuitry. The enclosed chronology (Attachments 2 of the enclosed report) provides additional details on the sequence of events that the Team developed during the inspection. The Team concluded that overall Dominion responded acceptably to these TDAFW failures to start, taking appropriate actions to address and correct the potential causes through maintenance and modifications and demonstrated adequate system performance with surveillance testing and electrical circuit verifications. While no specific repeatable cause could be identified; the most likely cause was improper electrical and/or mechanical interactions with or within the TDAFW pump speed governor. If the TDAFW pump had been called upon in response to a plant transient, between May and September 2014, it likely would have performed as it did on July 15 and September 10, this protracted start would have permitted the pump to supply water to the steam generators within sufficient time to limit the significance of the initial failure. Further, refueling outage actions taken by Dominion identified and corrected additional governor to turbine control valve linkage issues, not attributable to the subject failures. The Team identified two violations of NRC requirements, as documented in the enclosed report which were of very low safety significance (Green). However, given their very low safety significance, and because they were entered into your corrective action program, the NRC is treating these findings as non-cited violations (NCV), consistent with Section 2.3.2.a of the NRC Enforcement Policy. If you contest the NCVs in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region I; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at Millstone. In addition, if you disagree with the cross-cutting aspect assigned to any finding, or a finding not associated with a regulatory requirement in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region I, and the NRC Resident Inspector at Millstone Power Station. In accordance with Title 10 of the Code of Federal Regulations (10 CFR) Part 2.390, NRC's "Rules of Practice," a copy of this letter, its enclosure and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room). We appreciate your cooperation. Please contact Wayne Schmidt of the Division of Reactor Safety staff at (610) 337-5315 if you have any questions regarding this letter or the enclosed report.
Sincerely,/RA/
John Rogge, Branch Chief Engineering Branch 3 Division of Reactor Safety cc: Distribution via ListServ supply water to the steam generators within sufficient time to limit the significance of the initial failure. Further, refueling outage actions taken by Dominion identified and corrected additional governor to turbine control valve linkage issues, not attributable to the subject failures. The Team identified two violations of NRC requirements, as documented in the enclosed report which were of very low safety significance (Green). However, given their very low safety significance, and because they were entered into your corrective action program, the NRC is treating these findings as non-cited violations (NCV), consistent with Section 2.3.2.a of the NRC Enforcement Policy. If you contest the NCVs in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region I; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at Millstone. In addition, if you disagree with the cross-cutting aspect assigned to any finding, or a finding not associated with a regulatory requirement in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region I, and the NRC Resident Inspector at Millstone Power Station. In accordance with Title 10 of the Code of Federal Regulations (10 CFR) Part 2.390, NRC's "Rules of Practice," a copy of this letter, its enclosure and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room). We appreciate your cooperation. Please contact Wayne Schmidt of the Division of Reactor Safety staff at (610) 337-5315 if you have any questions regarding this letter or the enclosed report.
Sincerely,/RA/
John Rogge, Branch Chief Engineering Branch 3 Division of Reactor Safety cc: Distribution via ListServ Distribution: See Next Page DOC. NAME: G:\DRS\DIRECTOR\Schmidt\MS3 SIT #3 TDAFW\MSSIT-Report FINALr4.docx ADAMS ACCESSION NUMBER: ML15015A078 SUNSI Review Non-Sensitive Sensitive Publicly Available Non-Publicly Available OFFICE RI/DRS RI/DRP RI/DRS RI/DRS NAME WSchmidt RMcKinley JTrapp JRogge DATE 1/12/15 1/12/15 1/12/15 1/15/15 OFFICIAL RECORD COPY Docket No: 50-423 License No: NPF-49
Enclosure:
Inspection Report 05000423/2014013 w/Attachments 1, 2, and 3 cc w/enclosure and
attachments:
Distribution via ListServ Distribution w/encl: (via email) D. Dorman, RA (R1ORAMAIL Resource) D. Lew, DRA (R1ORAMAIL Resource) H. Nieh, DRP (R1DRPMAIL RESOURCE M. Scott, DRP (R1DRPMAIL RESOURCE R. Lorson, DRS (R1DRSMAIL RESOURCE) J. Trapp, DRS (R1DRSMAIL RESOURCE) R. McKinley, DRP S. Shaffer, DRP J. DeBoer, DRP J. Ambrosini, DRP, SRI B. Haagensen, DRP, RI L. McKown, DRP, RI K. McKenzie, DRP, AA K. Morgan-Butler, RI OEDO RidsNrrPMMillstone Resource RidsNrrDorlLpl1-1 Resource ROPreports Resource i Enclosure U. S. NUCLEAR REGULATORY COMMISSION REGION I Docket No. 50-423 License No. NPF-49 Report No. 05000423/2014013 Licensee: Dominion Nuclear Connecticut, Inc. Facility: Millstone Power Station, Unit 3 Location: P.O. Box 128 Waterford, CT 06385 Dates: September 15 through December 9, 2014 Inspectors: Wayne Schmidt, Senior Reactor Analyst Daniel Orr, Senior Reactor Inspector Joseph DeBoer, Project Engineer Louis McKown, Resident Inspector, Millstone Unit 3 Approved By: John Rogge, Branch Chief Engineering Branch 3 Division of Reactor Safety ii Enclosure
SUMMARY OF FINDINGS
iii
REPORT DETAILS
1 1. Description of Equipment Condition Issues-------------------..1 2. Summary Assessment of Licensee Performance-----------------1 3. Pump Design and Operation - Background-------------------..1 4. Equipment Troubleshooting-------------------------....3 5. Control Valve to Governor Linkage Issues---------------...----9 6. Risk Significance of the Conditions----------------------..12 7. Exit Meetings--------------------------------13 ATTACHMENT 1 - SPECIAL INSPECTION TEAM CHARTER-------------..A1-1 ATTACHMENT 2 - DETAILED SEQUENCE OF EVENTS---------------..A2-1 ATTACHMENT 3 -
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
.A3-1 LIST OF DOCUMENTS REVIEWED-----------------------A3-2 LIST OF ACRONYMS-----------------------------.A3-3
iii Enclosure SUMMARY OF FINDINGS IR 05000423/2014013; 09/15/2014 - 12/09/2014; Millstone Power Station Unit 3; Special Inspection to review the September 10 and July 15, 2014, Repetitive Turbine Driven Auxiliary Feedwater Pump Failures to start and governor to control valve linkage issues; Inspection Procedure 93812, "Special Inspection."
A four-person NRC team, comprised of regional inspectors, a Millstone Power Station resident inspector and a regional senior reactor analyst conducted this Special Inspection, identifying two findings of very low risk significance (Green), each being treated as a non-cited violation (NCV). The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006. Mitigating Systems Green. The Team identified a Green NCV of Title 10 of the Code of Federal Regulations (10 CFR), Part 50, Appendix B, Criterion V, "Instructions, Procedures and Drawings," for the failure to follow procedure MA-AA-103, Conduct of Troubleshooting, Revision 11 following the failure of the TDAFW pump to properly start on September 10, 2014, during quarterly surveillance testing. Specifically, between September 10 and September 13, 2014, Dominion failed to identify, review and address conflicting troubleshooting results for the governor shutdown relay (3CR) and the electrical overspeed relay (1CON) in the TDAFW control circuit by failing to compare troubleshooting result with the expected results. Dominion entered this issue into their corrective action program as condition report 567073. This finding was more than minor because if left uncorrected, the failure to address anomalous conditions or inconsistent data in accordance with procedural requirements could result in degraded or deficient conditions. This issue was of very low safety significance because there was no loss of TDAFW operability or functionality. These troubleshooting issues, given the actions taken by Dominion to replace electrical components and the governor and subsequent satisfactory surveillance testing and electrical circuit checks, had no impact on TDAFW pump functionality. The Team concluded that this finding had a cross-cutting aspect in the Human Performance, Teamwork area because individuals and work groups failed to communicate and coordinate their activities within and across organizational boundaries to ensure nuclear safety is maintained. (H.4) (Section 4.b.2) Green. The Team identified a Green NCV of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Actions" associated with the failure to identify and correct adverse conditions related to the TDAFW pump governor to control valve linkage which potentially could have affected the reliability of the pump. Specifically, previously unidentified cam-plate pivot bushing wear and non-optimal linkage setup allowed degradation of the cam-follower spherical bearing and potential linkage sluggishness and binding from February 4 to October 29, 2014, when the unit entered a refueling outage. Dominion addressed these linkage issues during the refueling outage and entered this issue into their corrective action program as condition report 563885.
iv Enclosure This finding was more than minor because it represented a challenge to the equipment performance attribute of the Reactor Safety - Mitigating Systems cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating
events to prevent undesirable consequences. Leaving the control linkage misalignment issues uncorrected could have reduced the reliability of the TDAFW pump. This issue was of very low safety significance because there was no loss of TDAFW operability or functionality. The linkage issues had no impact on TDAFW pump functionality based on satisfactory surveillance test results. The Team determined that this issue had a cross-cutting aspect in the Human Performance cross-cutting area associated with Conservative Bias, in that Dominion failed to identify the potential importance of deficiencies in the control linkage configuration. (H.14) (Section 5.b.2)
REPORT DETAILS 1. Description of Equipment Condition Issues In accordance with the Special Inspection Team (SIT) Charter (Attachment 1), team members (the Team) conducted a detailed review of two failures of the Millstone Power Station Unit 3 (Unit 3) turbine driven auxiliary feedwater (TDAFW) pump to properly start. The Team reviewed the events leading up to, and Dominion Nuclear Connecticut, Inc.'s (Dominion) response following the September 10, 2014, failure during surveillance testing (ST) and a similar failure that occurred on July 15, 2014. 2. Summary Assessment of Licensee Performance Overall Dominion responded acceptably to the July and September failures of the TDAFW pump to start. While no specific cause could be identified; the most likely cause was improper electrical and/or mechanical interactions with or within the TDAFW pump speed governor. Detailed troubleshooting did not identify any repeatable electrical or mechanical equipment failures. Dominion took appropriate actions to address and correct the potential causes for these failures through maintenance and modifications and demonstrated adequate system performance with surveillance testing and electrical circuit voltage checks. However, aspects of the complex troubleshooting in September 2014 did not meet the expectations of Dominion procedures and NRC requirements, resulting in a non-cited violation (NCV). If the TDAFW pump had been called upon in response to a plant transient, between May and September 2014, it likely would have performed as it did on July 15 and September
10, failing initially to start, but starting successfully after approximately 15 to 20 minutes. This protracted start would have permitted the pump to supply water to the steam generators within sufficient time to limit the significance of the initial failure. Specifically, the Team verified that the delayed start would not have prevented the pump from meeting its safety function for a limiting case station blackout event. Refueling outage actions taken by Dominion identified and corrected additional governor to turbine control valve linkage issues, which while not attributable to the subject failures. However, these issues should have been identified and corrected earlier in accordance with NRC requirements, resulting in an NC
- V. 3. Pump Design and Operation - Background To assist in understanding the inspection details below, the Team developed the following information on the Unit 3 TDAFW pump through detailed review of operating and maintenance procedures, electrical circuit schematics, surveillance tests, and the updated final safety analysis report. The TDAFW pump, powered by steam from three of the four steam generators, provide make-up water to the four steam generators to account for steam released to atmosphere though the atmospheric dump valves to ensure that reactor decay heat is removed and reactor coolant system conditions are adequately maintained
Enclosure following a plant trip. This pump functions without alternating current electrical power, but does require direct current electrical power to control pump speed setpoint and to provide electrical overspeed protection. Steam from the steam generators flows through the admission valves, to the normally open trip throttle valve and then the turbine control valve. The control valve position and the associated steam flow is modulated by a Woodward PGG governor that maintains pump speed near its setpoint, in response to pump flow demand and steam pressure changes. When the pump is secured the governor is set hydraulically at the low speed setpoint of approximately 1000 rpm. As the pump starts, the speed setting is hydraulically increased to the normal running speed of approximately 4500 rpm as the internal shaft-driven gear oil pump increases pressure. The pump running speed can be changed using: 1) the internal direct current motor raise/lower switch in the control room or locally at control panel in the pump room; 2) by turning the speed setting knob locally on the governor itself; or 3) by locally manually throttling the trip throttle valve. The design includes a standard mechanical overspeed trip device that mechanically senses pump speed and trips closed the normally open trip throttle valve when an overspeed condition occurs. The pump also has a redundant electric overspeed device that will electrically release the trip throttle valve to a closed position when a
high speed signal is sensed. The electric overspeed device, which requires direct current power, is set at a lower speed than the mechanical overspeed device, so it is expected to trip first to prevent a mechanical overspeed. The PGG governor also has an internal governor shutdown solenoid valve that when energized, by either electrical or mechanical overspeed signals, ports off all oil pressure from the governor speed setting piston. In response the control valve fully closes, and the pump coasts down. For an electrical overspeed condition, once the trip throttle valve is closed and the speed is below 500 rpm, the governor shutdown solenoid valve should de-energize and close, allowing the governor to reposition the control valve to an open position, as the machine coasts down, ready for a subsequent restart. Following a mechanical overspeed trip, the governor shutdown solenoid valve remains energized until the mechanical overspeed device is reset. Pump speed is monitored by an Airpax 300 Series Control Tachometer. This electronic device monitors shaft speed providing local and remote speed indications
and operates the K1 and K2 contacts, discussed below, in the electric overspeed and speed control circuit. The electric overspeed function and governor shutdown solenoid are controlled by the overspeed and speed control direct current circuit (referred to hereafter as the governor control circuit or the circuit). The governor shutdown solenoid is energized by the governor shutdown relay (3CR). 3CR is energized by a contact associated with the electric overspeed relay (1CON) or by a limit switch (5LS) which closes when the mechanical overspeed trip device trips. Once 3CR energizes it remains energized by one of its own 3CR 6-10 contact, and a K1 permissive 20-21 contact. The K1 20-21 contact closes if the TDAFW is operating above 500 rpm. The 1CON relay is energized if the K2 overspeed contact 14-15 closes. The K2 14-15 contact closes at about 4752 rpm.
Enclosure The PGG governor is connected to the control valve through a linkage (referred to hereafter as the governor linkage or the linkage) that converts the rotational output of the governor to a horizontal force that positions the control valve. One main component is the cam-plate, a pivoting arm with a machined cam-slot in it. A spherical cam-follower bearing, connected to the control valve shaft, rolls in the cam-slot, such that as the cam-plate moves up and down, the control valve moves in and out. In January 2014, to help investigate previous overspeed trip conditions, Dominion installed strain gages on the governor output link and on the control valve stem. During system testing, a computer program monitored these instruments along with control valve position and turbine speed to produce a real-time thrust/tension data trace of governor and control valve operation. In the normal standby condition the trip throttle valve and the control valve are open and the governor is set at the hydraulic low speed setting of 1000 rpm. Additionally, the three solenoid operated steam admission valves are closed. When an actuation signal is generated the steam admission valves open and the pump starts to roll. As the speed gets above 1000 rpm the governor fully closes the control valve and as the speed decreases and gets below the 1000 rpm setting the governor opens the control valve. Then the governor hydraulically ramps the speed setting to the rated 4500 rpm. The normal quarterly surveillance test requires that the pump reach 4500 rpm in 90 seconds or less. When shutting down the pump, Dominion's practice is to push the mechanical overspeed trip device lever. This simulates a mechanical overspeed trip, releasing the trip throttle valve to a closed position. The 5LS limit switch closes, energizing and opening the governor shutdown solenoid valve and the control valve closes. To restore the machine to the standby condition, the operator resets the mechanical overspeed device and the trip throttle valve by closing its hand wheel fully, then the trip throttle valve is opened slightly and if there is any small amount of steam leakage past the control valve the turbine will start to spin and the governor will see a speed much lower than the low speed hydraulic setting and demand that the control valve open. Once the control valve is open, the operator closes the trip throttle valve and the steam admission valves, and then fully opens the trip throttle valve. While resetting the pump for restart following testing or during a start attempt, if the governor shutdown solenoid valve was open, either because it was energized or
mechanically bound, the governor would not open the control valve and would force it closed if it was open. Governor oil pressure would build up and with the governor shutdown solenoid energized the control valve would see a full close demand. 4. Equipment Troubleshooting a. Inspection Scope The Team evaluated Dominion's systematic approach to troubleshoot the TDAFW failure to start events on July 15 and September 10, 2014. To assist in causal analysis, and as required by the Charter, the Team developed a detailed timeline of the event (Attachment 2). The troubleshooting in response to each event included a complex
Enclosure troubleshooting plan and involved several workgroups, technicians, supervisors, engineers, and managers.
The Team gathered information through the review of troubleshooting plans, logs, and data collected and observation of complex troubleshooting and maintenance activities. The Team interviewed plant management, operators, maintenance technicians, supervisors, and engineers involved with the troubleshooting. To identify anomalous equipment behavior the Team reviewed data which included outputs from the plant process computer, temporarily installed multi-channel electrical circuit monitoring test equipment traces (Astro-Med) and governor and control valve strain gage outputs dating back to January 23, 2014. The Team evaluated Dominion's apparent cause evaluation for the July 15 failure to assess Dominion's opportunity to have identified component issues that should have been identified and corrected prior to the September 10 failure. The Team reviewed a plant modification, completed during the Unit 3 Fall (October to November) 2014 refuel outage (3R16) which replaced additional control relays in the governor control circuit and installed a supervisory and test circuit to provide status indication to operators for the governor shutdown solenoid and relay. b.1. Observations Failure Analysis and Corrective Actions
The Team could not determine a specific cause for the July and September failures, based upon detailed independent review of electrical troubleshooting; governor linkage strain gage data and discussions with plant staff. This was in part based on the non-repeatability of any electrical or mechanical issues during subsequent testing and troubleshooting. The symptoms observed during each event suggested an intermittent electrical issue in the governor control circuit and/or a mechanical issue within the governor. On both occasions the linkage strain data showed that the governor did not develop an opening force as it normally would during the starting sequence and that the machine appeared to have experienced the actuation of governor shutdown solenoid, with no occurrence of an overspeed condition. The Team found that Dominion's actions to replace electrical components following the July failure was based on inconclusive data. Based on the information available at the time, the actions taken by plant management to replace electrical components, reset
the control valve to the full open position, and monitor the state of the overspeed circuitry provided reasonable assurance that the problem was either corrected or would be detected. However, the failure recurred again in September. The Team found that Dominion's actions to replace electrical components following the September failure was based on inconclusive data and ineffective troubleshooting practices. The decisions by plant management to continue circuit monitoring and replace the governor were warranted.
Enclosure In both cases, completion of troubleshooting and maintenance activities were challenged by the lack of repeatability of any electrical or mechanical equipment failures and the Technical Specification requirement to restore the TDAFW pump to an operable condition within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> or shutdown the unit. The Team reviewed the Electric Power Research Institute (ERPI) Technical Report 100746, "Terry Turbine Maintenance Guide, AFW Application" finding that Dominion maintenance procedures appropriately used the information provided for a PGG governor. Additionally this technical report did not document any previous similar failures. The Team did not identify any issues with previously reviewed tests or system operation that would have provided any information to indicate that the TDAFW pump
was not always fully functional. A plant modification, MP3-14-01165, completed during 3R16 replaced and upgraded all the TDAFW circuit original plant equipment active electrical components (3CR, 4CR, and 1CON relays). The modification also installed a supervisory and test circuit for the governor shutdown solenoid and relay, providing a status light to determine if the shutdown solenoid was energized when not appropriate. July Troubleshooting On July 15, 2014, TDAFW failed to start during quarterly surveillance testing. When the steam admission valves opened the pump initially ramped up to approximately 1600 rpm, but then immediately coasted down to a standstill as the control valve remained closed. The trip throttle valve remained open because there was no mechanical or electric overspeed signal. Approximately 20 minutes later, the pump started and ramped up to rated speed, about 4500 rpm. The coast down to standstill and return to rated speed was an anomalous behavior of the turbine governor control system. Dominion declared the TDAFW pump inoperable, because of the failed surveillance test, at 5:07 a.m. on July 15, 2014.
Dominion promptly assembled a troubleshooting team using MA-AA-103, Conduct of Troubleshooting and developed a complex troubleshooting plan. Consistent with MA-AA-103, the troubleshooting team developed a failure mode/cause table that considered nine possible failure modes. Several failure modes were associated with the governor, the governor control circuit and components, the governor linkage, the mechanical overspeed device, or water from the main steam lines. Troubleshooting data collection identified that tachometer device K1 spare contact (the actual contact used in the control circuit, K1 20-21, could not be monitored because of physical constraints within the governor control circuit cabinet) did not cycle as expected and the power supply for the tachometer had excessive voltage ripple. The troubleshooting team postulated that recent prior maintenance on the 5LS limit switch energized the 3CR relay which was sealed in because the K1 20-21 relay contact was unexpectedly closed. The K1 20-21 relay contact being closed was postulated by Dominion based on the field observation that spare K1 contacts did not cycle as expected. Dominion further postulated if the 3CR relay was sealed in the governor shutdown solenoid valve would be open and the turbine would have behaved as
Enclosure observed during the failed start. While the TDAFW pump was at a standstill, the K1 20-21 contact was postulated to have opened dropping out the 3CR relay. Steam leakage past the control valve eventually allowed the pump to roll and because the shutdown solenoid was no longer energized the pump ramped up to rated speed. Dominion replaced the tachometer and its power supply given the potential anomalous behavior. After the tachometer was removed, during bench testing Dominion identified high resistance when closed on the K1 20-21 contact. Additionally, Dominion established compensatory measures in the form of measuring circuit voltages after the pump was in a ready to start configuration to verify the shutdown logic was not enabled. The TDAFW pump was declared operable at 8:14 p.m. on July 16, 2014, following a satisfactory quarterly surveillance test. Further, Dominion noted that the governor control circuit did not provide a status of shutdown logic to indicate that the equipment was ready to start and initiated corrective actions to modify the circuit to include governor shutdown solenoid status indication. Dominion modified the control circuit during the 3R16 outage as discussed in the Failure Analysis and Corrective Actions section above.
The tachometer and power supply were sent to an offsite independent laboratory for failure analysis. The laboratory analysis, FA Report 2626 from Reliant Labs completed on July 28, 2014, documented degradation for the K2 contacts and a failed output filter capacitor for the tachometer power supply. The laboratory report concluded the power supply output voltage was degraded to the point where the tachometer's relay could not remain in its "on" state.
Dominion concluded in its apparent cause evaluation (ACE) 019769 that these electronic components failed due to age related degradation. The devices were not identified as vulnerabilities and had no scheduled replacement frequency. Dominion initiated corrective actions in condition report (CR) 553896 to replace additional electronic components and to schedule periodic replacements. September Troubleshooting
On September 10, 2014, the operators started the TDAFW pump at 11:43 a.m. for the first quarterly surveillance test after the July 15 failure. The pump started and failed to reach rated speed with a response almost identical to the July failure. Dominion again declared the pump inoperable, assembled a troubleshooting team using MA-AA 103, Conduct of Troubleshooting and developed a complex troubleshooting plan. The Team validated that the pump had been in a standby condition between July 16 and September 10 and that Dominion had not completed any maintenance or surveillance testing that could have impacted the control circuit or the governor. The troubleshooting team determined during field investigation that the 3CR relay 9-5 contact was anomalously closed with the relay deenergized. The 3CR relay was replaced. However, the Team observed repeated shop testing and disassembly of the
Enclosure removed relay which indicated that the 9-5 contact was performing properly and revealed no apparent contact issues. The results of the troubleshooting were presented to a management challenge board. Senior plant management additionally directed replacement of the governor. Dominion completed a satisfactory full flow surveillance and post-maintenance test at 1:30 p.m. on September 13. As compensatory measures, Dominion established daily electrical circuit checks and ran the pump weekly prior to 3R16, which began in October 2014.
The removed PGG governor was expedited to the vendor for failure analysis. The vendor did not identify any anomalies with the governor and the hydraulic fluid did not contain any foreign material. A Team member observed portions of the vendor as-received testing and initial disassembly, noting that there were no anomalies found. This governor had a short service life as it had been installed in January 2014 following satisfactory vendor testing. b.2. Findings Failure to Resolve Anomalous Data During Complex Troubleshooting of the Turbine Driven Auxiliary Feedwater Pump Controls Introduction: The Team identified a Green NCV of Title 10 of the Code of Federal Regulations (10 CFR), Part 50, Appendix B, Criterion V, "Instructions, Procedures and Drawings," for the failure to follow procedure MA-AA-103, Conduct of Troubleshooting, Revision 11 following the failure of the TDAFW pump to properly start on September 10, 2014, during quarterly surveillance testing. Specifically, between September 10 and September 13, 2014, Dominion failed to identify, review and address conflicting
troubleshooting results for the governor shutdown relay (3CR) and the electrical overspeed relay (1CON) in the TDAFW control circuit by failing to compare troubleshooting result with the expected results. Description: The Team identified several anomalous conditions and inconsistencies in data and the observations of Dominion staff response. The Team also identified examples where the troubleshooting plan was not methodically executed. These observations included: There were no clear step by step instructions developed following the test failure, (i.e., 1) take initial voltages, 2) review, 3)reset the circuit and linkage, 4) take next set of voltages, 5) review-); There was no obvious revision control on initial troubleshooting forms that requested additional electrical circuit data and test equipment installation; There was an over reliance on verbal communications of results from the field rather
than systematically reviewing the recorded data; The troubleshooting on September 10 that measured the 3CR 9-5 contact closed also measured zero volts across the governor shutdown solenoid. These results were not electrically consistent and were neither identified, analyzed , nor explained; Astro-Med data showed the 1CON relay energized on September 13 during full flow testing but the trip throttle valve did not close. This unexpected response was
Enclosure observable in data taken the previous day, but the data had not been compiled or reviewed; Engineers did not resolve anomalous troubleshooting data. For recorded Astro-Med data, the engineers did not verify adequate test equipment installation. They assumed the technicians had incorrectly installed test equipment but the technicians, when interviewed by the Team, were adamant the test equipment was installed correctly and yielded accurate measurement; and, Astro-Med and governor linkage strain gage data contained on the test equipment or computer was not retrieved and cataloged after each specific troubleshooting step. The Team reviewed procedure MA-AA-103, Conduct of Troubleshooting, Revision 11, finding that Step 3.11.5 stated "if conditions or data that cannot be explained or are not clearly understood are identified, then ensure these conditions or data are specifically identified, reviewed, and addressed by the troubleshooting team." The Team determined that conflicting data and conflicting observations by technicians and engineers involved in troubleshooting were not specifically reviewed and addressed by the troubleshooting team and this procedure requirement was not satisfied. Dominion initiated CR567073 in response to this issue. Initial actions included issuing job aids to troubleshooting team members. Dominion intended long term actions to evaluate gaps for additional corrective actions to improve future troubleshooting. Analysis: The Team determined that a performance deficiency existed for the failure of Dominion to identify, explain or understand and address anomalous conditions and inconsistencies in data during the troubleshooting of the TDAFW failure to start in September 2014 as required by procedure MA-AA-103. This performance deficiency was associated with the Mitigating Systems cornerstone and was more than minor because if left uncorrected, the failure to address anomalous conditions or data in accordance with procedural requirements could result in degraded or deficient conditions. The Team determined that this issue was of very low safety significance (Green) in accordance with Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP), Attachment 0609.04, "Initial Screening and Characterization of Findings," for the Mitigating Systems Cornerstone, and IMC 0609, Appendix A, "The SDP for Findings At-Power." The very low safety significance resulted because there was no loss of TDAFW operability or functionality associated with the finding and each question provided in IMC 0609, Appendix A, Exhibit 2, Mitigating Systems Screening Questions, was answered No. These troubleshooting issues, given the actions taken by Dominion to replace electrical components and the governor and subsequent satisfactory surveillance testing and electrical circuit checks, had no impact on TDAFW pump functionality.
The Team concluded that this finding had a cross-cutting aspect in the Human Performance, Teamwork area because individuals and work groups failed to
communicate and coordinate their activities within and across organizational boundaries to ensure nuclear safety is maintained (H.4).
Enclosure Enforcement: 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, requires, in part, that activities affecting quality be prescribed by documented procedures of a type appropriate to the circumstance and be accomplished in accordance with these procedures. Dominion conducted complex troubleshooting of the Unit 3 TDAFW pump governor control circuit using procedure MA-AA-103, Conduct of Troubleshooting, Revision 11. Section 3.11 of the procedure requires that the troubleshooting team specifically identify, review, and address conflicting troubleshooting results by comparing troubleshooting results with the expected results. Contrary to the above, on September 10 and September 13, 2014, the troubleshooting team failed to identify, review, and address conflicting troubleshooting results for the 3CR and 1CON relays, respectively, by failing to compare troubleshooting result with the expected results. Specifically, electrical checks taken for the 3CR 9-5 contact provided electrically inconsistent data and the 1CON relay was instrumented and recorded as energized but
the circuit did not accordingly respond. Because this violation was of very low safety significance and was entered into Dominion's corrective action program as CR567073, this violation is being treated as a NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy. Corrective actions for this issue included issuing job aids to troubleshooting team members. Dominion intended to evaluate gaps for long term corrective actions to improve future troubleshooting. NCV 05000423/2014013-01: Failure to Resolve Anomalous Data During Complex Troubleshooting of the Turbine Driven Auxiliary Feedwater Pump Controls. 5. Control Valve to Governor Linkage Issues a. Inspection Scope The Team reviewed the operation of the governor linkage to determine if binding contributed to the TDAFW failures in July and September 2014. The Team identified sluggishness and potential binding of the linkage in strain gage data taken during testing following governor replacement in September 2014, and reviewed the corrective actions taken by Dominion in response to repeated turbine overspeed events between November 2013 and February 20141. The Team gathered inspection reports, causal evaluations, equipment performance data, work orders, maintenance procedures, licensee contracted external evaluations, and industry guidance as well as interviewed plant management, operators, maintenance technicians, engineers, and equipment vendor (Dresser-Rand) experts.
The Team evaluated the actions taken by Dominion during 3R16 to assess the condition of the linkage, assess the impact of the observed conditions on TDAFW pump operability, and verify that the linkage conformed to original manufacturer specifications. 1 As documented in Special Inspection Report 05000423/2014008.
Enclosure The Team observed the detailed disassembly and reassembly of the linkage by Dominion staff with a vendor expert on hand providing factory acceptance criteria (blueprinting). b.1. Observations The Team determined that the failures of the TDAFW pump to start in July and September 2014 were not attributable to governor linkage issues, based on detailed review of strain gage traces, as discussed in Section 4.b.1 above. The Team noted through review of the linkage strain traces following governor replacement in September 2014 that the control valve was not opening smoothly in response to governor opening signals. Dominion personnel also noted this adverse trend in control valve operation and complete a detailed linkage evaluation and maintenance activity during 3R16. The Team verified that Dominion appropriately evaluated linkage strain gage data to monitor the governor linkage performance for any adverse trends or potential binding. The 3R16 blueprinting and maintenance activates returned the linkage to within tolerances as determined by the vendor and the subsequent Mode 3 full flow system surveillance testing demonstrated improved linkage operation. b.2. Findings Governor Linkage Ineffective Corrective Action Introduction: The Team identified a Green NCV of 10 CFR Part 50, Appendix B, Criterion XVI "Corrective Actions" associated with the failure to identify and correct adverse conditions related to the TDAFW pump governor to control valve linkage which potentially could have affected the reliability of the pump. Specifically, previously unidentified cam-plate pivot bushing wear and non-optimal linkage setup allowed degradation of the cam-follower spherical bearing and potential linkage sluggishness and binding from February 4 to October 29, 2014, when 3R16 began. Description: The Team found that the conditions identified during 3R16, with the assistance of the pump vendor, included: excessive wear on and non-optimal alignment of the control valve cam-follower bearing, the wear patterns and the misalignment were very similar to that observed previously; cam-plate issues including non-vertical movement three to four times vendor acceptance criteria, because of pivot bushing wear; and rough areas in the cam-plate cam-slot near the closed position. The pump vendor also provided information concerning linkage setup and control valve movement that was different than previously provided in January 2014.
In review of past issues the Team found that on February 10, 2014, Dominion identified that the cam-follower bearing was not centered in the cam-slot (CR539341). This condition indicated linkage misalignment that had also been previously observed prior to replacement of the cam-follower bearing. At that point Dominion determined in consultation with the pump vendor that, while not optimal, the misalignment was within
Enclosure system design configuration. An enhancement action to improve the maintenance procedure was opened in the corrective action program but was closed, on March 3, 2014, by maintenance with no action taken based upon engineering's acceptance of the vendor disposition that the condition was acceptable. The Team found that Dominion failed to correct these issues as part of linkage evaluation and restoration activities associated with the overspeed trip events between November 2013 and February 2014. Moreover, the corrective actions taken at that time lacked specific and measureable alignment configuration acceptance criteria to correct contributing causes to the cam-follower bearing degradation. The Team verified that Dominion corrected these adverse conditions through maintenance activities, as demonstrated by review of linkage strain gage data collected during system full flow testing during startup from 3R16. Analysis: The Team determined that a performance deficiency existed for the failure of Dominion to promptly identify and correct control linkage misalignment issues, including cam-plate pivot bushing wear and non-optimal linkage setup, which were reasonably within their ability to foresee and correct. Specifically, these conditions allowed degradation of the cam-follower spherical bearing and potential linkage sluggishness and binding. This finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," dated September 7, 2012, as it represented a challenge to the equipment performance attribute of the Reactor Safety - Mitigating Systems cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable
consequences. Leaving the control linkage misalignment issues uncorrected could have reduced the reliability of the TDAFW pump. Specifically, Dominion failed to perform an adequate evaluation of the entire linkage given multiple indications of conditions adverse to quality and failed to establish adequate acceptance criteria for identified control linkage misalignment. This misalignment was a contributing cause to the degradation of the cam-follower bearing which resulted in past overspeed trip events. Dominion performed a detailed inspection and restoration of the linkage assembly to within vendor tolerances during 3R16 to enhance reliability. The Team determined that this issue was of very low safety significance (Green) in accordance with IMC 0609, SDP, Attachment 0609.04, "Initial Screening and Characterization of Findings," for the Mitigating Systems Cornerstone, and IMC 0609, Appendix A, "The SDP for Findings At-Power." The very low safety significance resulted because there was no loss of TDAFW operability or functionality associated with the finding and each question provided in IMC 0609, Appendix A, Exhibit 2, Mitigating Systems Screening Questions, was answered No. The linkage issues had no impact on TDAFW pump functionality based on satisfactory surveillance test results. The Team determined that this issue had a cross-cutting aspect in the Human Performance cross-cutting area associated with Conservative Bias, in that Dominion failed to identify the potential importance of deficiencies in the control linkage configuration. Specifically, upon identification of cam-follower bearing misalignment in
Enclosure the cam-slot race on February 10, 2014, Dominion management chose to accept the condition and permitted closure, without any corrective actions, of an associated maintenance enhancement action, based upon evaluation by engineering and the vendor that the condition was 'non-optimal' but acceptable. (H14) Enforcement: 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Actions" requires, in part, measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and non-conformances are promptly identified and corrected. Contrary to the above, from February 4 to October 29, 2014, conditions adverse to quality in the TDAFW pump
control linkage were not promptly identified and corrected by Dominion. Specifically, Dominion failed to identify and correct control linkage misalignment issues, including cam-plate pivot bushing wear and non-optimal linkage setup, which allowed degradation of the cam-follower spherical bearing and potential linkage sluggishness and binding. Further, Dominion failed to establish adequate acceptance criteria to correct a contributing cause to the degradation of the cam-follower bearing which resulted in past overspeed trip events. Because this issue is of very low safety significance (Green) and the licensee has taken actions to correct the linkage issues and entered this issue into their corrective action program (CR563885), this finding is being treated as an NCV consistent with the NRC Enforcement Policy Section 2.3.2. NCV 05000423/2014013-02: Failure to Promptly Identify and Correct Adverse Conditions related to the Turbine Driven Auxiliary Feedwater Pump Governor to Control Valve Linkage 6. Risk Significance of the Conditions a. Initial Assessment The initial risk assessment, as documented in the enclosed SIT Charter (Attachment 1) assumed that the TDAFW pump was not functional between July 17 and September 10, 2014, and estimated an internal and external initiating event increase in core damage in the one in 100,000 range over the 55 day exposure period. b. Final Assessment A Region I Senior Reactor Analyst (SRA) used the Millstone Unit 3 Standardized Plant Analysis Risk (SPAR) model, to complete a bounding above the baseline risk that included normal test and maintenance. The assumptions were the same as the initial assessment except: The exposure period was between June 2 and September 13, 2014, 100 days, which includes 6 days where the TDAFW pump was out-of-service, because of the July and September failures. Two bounding assumptions were used: o Best Case - The TDAFW pump would likely have behaved as it did on July 15 and September 10, with no loss of safety function. (see Section 2. above)
Enclosure o Worst Case - The TDAFW pump would have been recovered by operator action with a failure to recover of 0.1, using procedure Emergency Operating Procedure 35 GA-31, "Locally Restoring AFW Flow". The test and maintenance model was used to assess internal risk. The internal event CDP for the 100 day period was bounded by essentially no increase for the Best Case to an increase of approximately one in 625,000 for the Worst Case. The external (fire) events contribution was likewise bounded by essentially no increase for the Best Case to an increase of approximately one in four million (CDP control room evacuation fire = IEF x TDAFW FTS x 100/365 x 0.1 TDAFW recovery credit x 0.1 MDAFW pump credit) for a conservative quantitative upper-bound value. The total (internal and fire contribution) CDP bounding valves were no increase for the Best Case to an increase of approximately one in 700,000 for the Worst Case. 7. Exit Meetings On December 9, 2014, the Team leader presented the overall assessment and observations to members of Dominion's management led by Mr. John Daugherty, Site Vice President, and other members of his staff. The inspectors confirmed that any proprietary information reviewed during the inspection period was returned to Dominion.
ATTACHMENT 1 - SPECIAL INSPECTION TEAM CHARTER A1-1 Attachment 1 Special Inspection Charter Millstone Nuclear Power Station Unit 3 Multiple Turbine-Driven Auxiliary Feedwater (TDAFW) Pump failures on July 15, 2014 and September 10, 2014 Background: On July 15, 2014, the Millstone Unit 3 turbine-driven auxiliary feedwater pump (TDAFW) was started for a post maintenance test. The pump started and the turbine spun up to approximately 1700 rpm and then coasted down to zero. Approximately 15 minutes later the pump restarted and achieved rated speed and discharge pressure without any operator action. The operators tripped the pump upon discovering the pump running after the unexpected pump start.
Dominion initiated a troubleshooting team to evaluate the occurrence. The team determined that a digital tachometer had failed. The tachometer and its power supply were replaced. After further testing Dominion determined that the TDAFW was operable. On September 10, 2014, the TDAFW was started for a surveillance test. The pump started and the turbine spun up to approximately 2500 rpm and then spun down to zero. Approximately 15 minutes later the restarted and achieved rated speed and discharge pressure without any operator action. The resident inspectors continue to conduct baseline inspections with assistance from Division of Reactor Safety specialists. Basis for the Formation of the Special Inspection Team: The failure of the TDAFW involved a failure of safety-related equipment. There was a similar failure of a surveillance test on July 15, 2014. Additionally, the region issued a Special Inspection Report in the 3rd of 2014, addressing previous operability issues with the TDAF
- W. To assess the risk significance of this TDAFW failure to start (FTS) condition, the Region I Senior Reactor Analyst (SRA) used the Millstone Unit 3 SPAR model for assessing the internal risk contribution. Based upon the importance of the TDAFW in mitigation of the fire events that lead to control room abandonment (reference Dominion emergency operating procedure (EOP) 3509.1), the SRA used a developed event tree to assess the external events (fire) contribution. EOP 3509.1 was recently revised by Dominion to credit operation of the A motor-driven AFW pump. However, operation of the motor-driven pump relies heavily on operator actions to control make-up flow to the steam generators. A The SRA made the following assumptions: No recovery credit is assumed for internal events. For external events, recovery of the
'A' motor-driven AFW is assumed with a 0.1 failure probability. Millstone Unit 3 SPAR model event AFW-TDP-FS-P2 was changed from its nominal failure to start probability of 6.49E-3 to 1.0 (failure) Exposure period of assumed to be July 17 - September 10, 2014, 55 days.
ATTACHMENT 1 - SPECIAL INSPECTION TEAM CHARTER A1-2 Attachment 1 The initiating event frequencies (IEF) for fires involving a control room abandonment (using EOP 3509.1) are: control room fires - 4.48E-04; instrument rack room fires - 1.5E-04; and cable spreading room fires - 4.5E-05; or a total fire frequency of 6.43E-04/yr, if TDAFW is assumed to fail the fire frequency would equal the Conditional Core Damage Probability (CCDP). The zero test and maintenance model was used to assess internal risk Assuming no recovery, based upon the 55 day exposure time, the internal events CCDP contribution, using the Standardized Plant Analysis Risk (SPAR) model, was approximately 2.2E-6. The external (fire) events contribution was estimated at approximately 9.7E-6 (CCDP = IEF x TDAFWP FTS x 55/365 x 0.1 recovery credit) for a conservative quantitative upper-bound value; and approximately 5E-6 for a lower-bound value (assuming the nominal SPAR model fault tree solution for the A motor-driven auxiliary feedwater (AFW) pump with no human error probability adjustments). The total (internal and external contribution) CCDP value for this plant issue is estimated at between high E-6 to low E-5.
Based upon the preliminary CCDP estimate of high E-6 to low E-5, in accordance with Inspection Manual Chapter (IMC) 0309, this event falls mostly within the Special Inspection Team reactive inspection region. Objectives of the Special Inspection: 1. Generate a timeline of events which captures the failures, maintenance evolutions, and tests performed on the TDAFW system with regards to the July 15 and September 10, 2014 test failures. 2. Evaluate the adequacy and completeness of the maintenance on the TDAFW system, including preventive maintenance, maintenance practices, procedural guidance, post-maintenance testing, and supervisory oversight with regards to the July 15 and September 10, 2014, test failures. 3. Evaluate Dominion's application of pertinent industry operating experience and evaluation of potential precursors including the adequacy of any actions taken in
response to the operating experience or precursors. 4. Evaluate the timeliness and adequacy of Dominion's response to the July 15, 2014, and September 10, 2014, TDAFW system failures, including Dominion's cause analyses, extent of condition, corrective actions, and failure mode considerations. 5. Evaluate the adequacy of Dominion's current and past operability decisions with regards to the TDAFW system following the July 15, 2014, test failure.
6. Evaluate Dominion's assessment of the risk significance of the degraded condition,
including evaluation of input assumptions and independently evaluate the risk significance.
ATTACHMENT 1 - SPECIAL INSPECTION TEAM CHARTER A1-3 Attachment 1 Additionally, the team leader will review lessons learned from the Special Inspection and, if appropriate, prepare a feedback form on recommendations for revising the Reactor Oversight Process baseline inspection procedures in order to proactively identify the issues and causes involved with the event. Guidance:
Inspection Procedure 93812, "Special Inspection," provides additional guidance to be used by
the Special Inspection Team. Team duties will be as described in Inspection Procedure 93812. The inspection should emphasize fact-finding in its review of the circumstances surrounding the event. It is not the responsibility of the team to examine the regulatory process. Safety
concerns identified that are not directly related to the event should be reported to the Region I office for appropriate action. The team will conduct an entrance meeting and begin the initial onsite inspection on September 15, 2014. While on site, the Team Leader will provide daily briefings to Region I management, who will coordinate with the Office of Nuclear Reactor Regulation, to ensure that
all other parties are kept informed. The inspection is anticipated to complete in October 2014. A report documenting the results of the inspection should be issued within 45 days of the completion of the inspection. This Charter may be modified should the team develop significant
new information that warrants review.
ATTACHMENT 2 - DETAILED SEQUENCE OF EVENTS A2-1 Attachment 2 Detailed Sequence of Events and Integrated Timeline The Team developed the following detailed sequence of events and integrated timeline which includes the governor replacement dating back to January 2014 and the subsequent surveillances and failures to start on July 15 and September 10, 2014, through the completion of 3R16 restart testing. The Team independently reviewed data from: linkage trust instruments, any temporarily installed Astro-med test equipment available during the listed surveillance tests and as collected during troubleshooting. January 26, 2014 Governor Replaced - Post-maintenance test (PMT) - Satisfactory (SAT) Surveillance Test (ST) Mode 1 Full Flow Test. May 25, 2014 Loss of Offsite Power - TDAFW Ran for Approximately 4 Hours. June 2, 2014 SAT Mode 3 Full Flow ST as part of unit restart. July 14, 2014 WO 53102706007-Maintenance performed on TDAFW which consisted on centering two pins on the over-speed trip latch mechanism. July 15, 2014 @ 5:07 am TDAFW Declared Inoperable - Fail to Start - Quarterly ST 3622.3-1 - Reached 1680 RPM, shutdown and then restarted 20 minutes later - Starting governor rack position 6.5. July 15, 2014 SAT Maintenance Run of Quarterly S
- T. July 15/16, 2014 UNEXPECTED CONDITION-Identified a 365 mVAC RMS ripple on K1/K2 power supply. - power supply and Tachometer Module (K1/K2) Replaced. July 16, 2014 8:14 am SAT PMT Quarterly ST - TDAFW Declared Operable with Corrective Action implemented to perform voltage checks to verify machine was in a ready-to start electrical configuration. July 17, 2014 TDAFW control circuit fuses blown while technicians removed Astro-Med test equipment. They fuses were replaced shortly after. September 10, 2014 11:43 am TDAFW Declared Inoperable - Fail to Start - Quarterly ST-Reached 2480 RPM, shutdown and then restarted 16 minutes later - Starting governor rack position 10. September 10, 2014 SAT circuit voltage checks performed -Unsure if before or after restart. September 10, 2014 SAT Maintenance Run of Quarterly S
- T. September 10, 2014 After 5:30 pm I&C Performs checks on the circuit and finds the 3CR 9-5 Contact closed with the 3CR and Governor Shutdown solenoid De-energized. The Team subsequently determined that these conditions were anomalous, meaning that they both could not be true.
ATTACHMENT 2 - DETAILED SEQUENCE OF EVENTS A2-2 Attachment 2 September 10, 2014 7:00 pm Generation Test Services performs independent checks of the 3CR 9-5 contact and identifies that the 3CR 9-5 contact is OPEN with the 3CR De-energized. September 11, 2014 Maintenance Corrective Action-New Governor installed, 5LS Limit Switch and 3CR relay replaced- Additional Astro-Med data point added to monitor 1CON relay and electric Turbine Trip solenoid. September 12, 2014 Maintenance run performed with an electrical overspeed trip because the new governor was not connected to machine shaft- OCC Troubleshooting log entry state Overspeed trip run showed satisfactory circuit operation and Astro-Med data showed satisfactory operation of K2, 1CON and 3CR relay. The Team subsequently identified that this was not true the Astro-Med data showed potential improper operation of the 1CON relay. September 12, 2014 Maintenance Run performed with governor adjustments- All SA
- T. September 13, 2014 12:05 pm SAT PMT Mode 1 Full Flow ST - TDAFW Declared Operable. September 13, 2014 6:00 pm 1CON Relay discrepancy noted in the troubleshooting Logs - Troubleshooting commenced on 1CON relay. Engineering concluded that the circuit was operating properly and that the 1CON discrepancies were due improperly installed test equipment. September 13 through October 18, 2014 SAT - Five Compensatory Measure Quarterly STs with periodic circuit monitoring. October 2014 Linkage found in a degraded state during refueling outage and was re-installed to validate tolerances. November 2014 Circuit Modification completed - allows monitoring of the governor shutdown solenoid from control panel. November 15, 2014 SAT PMT Mode 3 Full Flow ST as part of unit restart.
ATTACHMENT 3 - SUPPLEMENTAL INFORMATION A3-1 Attachment 3 SUPPLEMENTAL INFORMATION
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened and Closed
- 05000423/2014013-01 NCV Failure to Resolve Anomalous Data During Complex
Troubleshooting of the Turbine Driven Auxiliary Feedwater Pump Controls (Section 4.b.2)
- 05000423/2014013-02 NCV Failure to Identify and Correct Adverse Conditions related to the Turbine Driven Auxiliary Feedwater Pump Governor to Control Valve Linkage (Section 5.b.2) KEY POINTS OF CONTACT Interviews Conducted Matt Adams, Plant Manager Sonny Stanley, Engineering - Director Lori Armstrong, Nuclear Safety and Licensing - Director Barbara Wilkens, Nuclear Excellence - Manager Clark Maxson, Nuclear Engineering - Manager Doug Scott, System Engineer Steve Cardoza, Generation Test Group - Supervisor Coby Wooten, Operations - Unit Supervisor David Carpenter, Instrument and Controls - Technician Paul Wynn, System Engineer Doug Kurtz, Instrument and Controls - Technician Steve Saulter, Mechanical Maintenance - Technician Jason La Montague, Instrument and Controls - Technician Harold Thompson, System Engineering - Contractor Albert Jaskot, Generation Test Group - Technician Ray Thompson, Mechanical Maintenance - Supervisor Don Sullivan, Mechanical Maintenance - Technician Jeff Weymouth, Mechanical Maintenance - Technician Ray Preston, Dresser-Rand Field Service Engineer
ATTACHMENT 3 - SUPPLEMENTAL INFORMATION A3-2 Attachment 3
LIST OF DOCUMENTS REVIEWED
In addition to the documents identified in the body of this report, the inspectors reviewed the following documents and records.
Procedures
- IC3485I02, Turbine Driven AFW Pump Airpax Model 300 Electronic Tachometer, Revision 1
- MA-AA-103, Conduct of Troubleshooting, Revision 11
- MP 3762AB, Terry Turbine Governor Control Valve Maintenance, Revision 002-04 C
- MP 711, Terry Turbine Governor Control Valve Maintenance, Revision 002-07
- MP 3762AC, Terry Turbine Trip Throttle Valve Maintenance, Revision 004-00
- Revision 009-01
- MP 3710AA, Lubrication Program, Revision 007-07
- MP 3720AC, Auxiliary Feedwater Pump Turbine Maintenance, Revision 005-14
- SP 3622.3, Auxiliary Feedwater Pump 3FWA*P2 Operational Readiness Test, Revision 017-21
- SP 3622.3, Auxiliary Feedwater Pump 3FWA*P2 Operational Readiness Test, Revision 017-24
- SP 3622.10, Auxiliary Feedwater Pump 3FWA*P2 Full Flow Test in Mode 1 (ICCE),
- Revision 000-00 C
- MP 711, Terry Turbine Governor Control Valve Maintenance, Revision 002-07
Documents
reviewed
- ACE 01769 Terry Turbine Fail to Start 7/15/2014 - Book 1
- Reliant Labs FA Report Number 2626
- MA-AA-103 Attachment 4, Complete Troubleshooting Failure Mode/Cause Table, 9/10/14 Troubleshooting Summary for Event
- CR558213
- SP 3622.10-.001, TDAFW Pump Full Flow Test in Mode 1 Surveillance Form, 9/12/14 EPRI Terry Turbine Maintenance Guide, AFW Application TR1007461,dated November 2002
- DMN-0-LI-012-0001-R0, Worley Parsons Millstone Turbine Driven Auxiliary Feedwater Pump
- Overspeed Events, dated March 11, 2014 MPR Associates Inc. Scoping Evaluation of Millstone Unit 3 Turbine Driven Auxiliary Feedwater Pump Turbine Governor Valve Pinning under Water Flow Conditions Letters
- 0282-0133-LTR-000#,1-5, dated March 22, 2014 through April 2, 2014 Root Cause Evaluation
- 001111, Millstone 3 Turbine Driven Auxiliary Feed Water Pump,
- 3FWA*P2, Trips on 11/4/13, 12/18/13, and 01/23/14, dated May 19, 2014 Dresser-Rand AFWP Turbine Governor Linkage Final Report 139992-003, dated 11/01/14
- Drawings: Elementary Diagram 125 VDC Turbine Driven Aux Feedwater Pump Motor Speed Changer, (3FWA*M7), NO. 12179-ESK-7RF
- Design Changes:
- MP3-14-01165, Millstone Unit 3 Turbine Driven Auxiliary Feedwater Pump Control Circuit Modifications, Revision 0
- ATTACHMENT 3 - SUPPLEMENTAL INFORMATION
- A3-3
- Attachment 3
- Vendor Manuals: 041-3C, Airpax 300 Series Control Tachometer 990-600-0006 Instruction Manual, Revision 1 Siemens Electromechanical Components, Potter & Brumfield Relays, KHA Series Datasheet, Revision 1
- Work Orders: 53102239919 53102411207 53102753335 53102689251 53102700183 53102631122
- Condition Reports:
- 539341
- 558213
- 558318
- 558385
- 558389
- 558478
- 558497
- 558524
- 558528
- 558539
- 558541
- 558542
- 558550
- 558700
- 558897
- 558909
- 558920
- 561963
- 562010*
- 562845*
- 563811*
- 563885*
- 567073*
- designates CRs generated based on NRC identified issues
- ATTACHMENT 3 - SUPPLEMENTAL INFORMATION
- A3-4
- Attachment 3
LIST OF ACRONYMS
3R16 Unit 3 Fall 2014 Refuel Outage