IR 05000348/2014010
| ML14066A285 | |
| Person / Time | |
|---|---|
| Site: | Farley  |
| Issue date: | 03/07/2014 |
| From: | Ehrhardt F J NRC/RGN-II/DRP/RPB2 |
| To: | Gayheart C A Southern Nuclear Operating Co |
| References | |
| IR-14-010 | |
| Download: ML14066A285 (22) | |
Text
March 7, 2014
Cheryl Vice President - Joseph M. Farley Nuclear Plant Southern Nuclear Operating Company, Inc. P.O. Drawer 470, BIN B500 Ashford, AL 36312
SUBJECT: JOSEPH M. FARLEY NUCLEAR PLANT - NRC SPECIAL INSPECTION REPORT 05000364/2014010
Dear Ms. Gayheart:
On January 17, 2014, the U.S. Nuclear Regulatory Commission (NRC) completed its initial assessment of the circumstances surrounding the unsatisfactory operability test of Unit 2 "B" train solid state protection system (SSPS) which occurred on January 10, 2014. Based on this initial assessment, the NRC sent an inspection team to your site on January 21, 2014. The team completed the onsite portion of a special inspection on January 24, 2014, and discussed the results of this inspection with you and other members of your staff. The inspection team documented the results of this inspection in the enclosed inspection report.
No findings were identified during this inspection.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's Agencywide Document Access and Management System (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,/RA/
Frank Ehrhardt, Branch Chief Reactor Projects Branch 2 Division of Reactor Projects Docket Nos.: 50-364 License No.: NPF-8
Enclosure:
Special Inspection Report 05000364/2014010
w/Attachment:
Supplemental Information cc distribution via Listserv Cheryl
SUMMARY OF FINDINGS
IR 0500364/2014010; 01/21/2014 - 01/24/2014; Joseph M. Farley Nuclear Plant, Unit 2; Special
Inspection.
This report documents special inspection activities performed onsite and in the Region II office by one senior resident inspector and one reactor inspector to review the circumstances surrounding the unsatisfactory operability test of Unit 2 "B" train solid state protection system (SSPS) on January 10, 2014. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006.
No findings were identified.
REPORT DETAILS
Summary of the Degraded Condition On January 10, 2014, the Unit 2 "B" train SSPS operability surveillance per FNP-2-STP-33.0B,
"Solid State Protection System Train B Operability Test" could not be completed satisfactorily due to unexpected indications during the logic testing portion of the surveillance. The licensee could not restore operability of the "B" train SSPS and shut down Unit 2 in accordance with the applicable technical specification (TS) action statements. The licensee placed the unit in Mode 5 on January 12, 2014. Troubleshooting revealed a "stray" termi-point clip electrically shorting across pins 16 and 17 of the A516 safeguards driver output card. In this condition, the "B" train of SSPS would not have actuated the master relay (K501) for a safety injection signal from "B" train SSPS, including sending start signals to the "B" train charging pump, residual heat removal pump, service water pump, component cooling water pump, containment fan coolers, and motor driven auxiliary feedwater pump. This event was reported to the NRC as Event Notification49715 dated January 11, 2014.
Special Inspection Charter Based on the deterministic and conditional risk criteria specified in Management Directive 8.3, "NRC Incident Investigation Program," a special inspection was initiated in accordance with NRC Inspection Procedure 93812, "Special Inspection Team." The inspection focus areas included the following special inspection charter items:
1. Develop a history of SSPS issues including a detailed sequence of events from time of surveillance failure until the system was declared operable (including a list of condition reports (CR) related to SSPS card issues for Unit 2.) 2. Review and evaluate the licensee's troubleshooting activities related to this event.
3. Review and assess the licensee's testing and maintenance practices of the SSPS cards and other similar type cards on both units. 4. Review and evaluate the licensee's compliance with vendor recommendations regarding maintenance of SSPS cards and other similar type cards and modifications implemented to the SSPS system.
system. 7. Assess the licensee's actions resulting from vendor technical bulletins and industry operating experience concerning SSPS and similar type cards.
8. Collect data necessary to support completion of the significance determination process, if applicable.
9. Identify any potential generic safety issues and make recommendations for appropriate follow-up action (e.g., Information Notices, Generic Letters, and
Bulletins).
OTHER ACTIVITIES
4OA5 Other Activities - Special Inspection (IP 93812)
a. Inspection Scope
The inspectors reviewed documents provided by the licensee to develop a detailed sequence of events associated with licensee activities from the point of time the licensee declared FNP-2-STP-33.0B, "Solid State Protection System Train B Operability Test" failed until Unit 2 train "B" SSPS was declared operable. Additionally, the inspectors interviewed station personnel in the development of this timeline. The inspectors also determined from the inspection activities associated with other charter items the need to develop a timeline associated with SSPS card failures that had occurred over the past ten years of plant operation. The inspectors reviewed station corrective action reports, including licensee corrective actions, to evaluate the potential for trends and insights related to the January 10, 2014, surveillance failure. The inspectors also reviewed station documents related to plant modifications associated with Unit 2 train "B" SSPS. The inspectors developed a timeline and evaluated the modification documents for potential causes of the January 10, 2014, surveillance failure.
b. Observations Failed FNP-2-STP-33.0B, "Solid State Protection System Train B Operability Test" Surveillance Timeline:
On January 10, 2014, at 9:19 AM (CST), the licensee implemented SNC 487319 and entered TS 3.3.1 and 3.3.2 to complete the 84 day surveillance requirement for Unit 2 train "B" SSPS. Fifty minutes later, the licensee placed logic switch "B" in switch position 22 and this position tested "bad". The system also failed to provide appropriate indications on switch positions associated with Phase "A" safety injection isolation, safety injection containment vent isolation, and feedwater control. The licensee implemented Appendix 8, "Determination of Logic OR Semi-Automatic Tester Failure,"
as required by FNP-2-STP-33.0B. At 1:21 PM (CST), the SSPS train "B" 15 volt DC power supply circuit breakers tripped following the manual input pushbutton being depressed to test the ability to actuate manual Phase A safety injection. During this activity, the licensee unexpectedly received indication of P-12 on the reactor protection system permissive monitor light boxes and all main control board first out annunciators illuminated. It was unknown by the licensee at this point, but the cause of these unexpected alarms was a direct short across logic output card A516 pins 16 and 17.
The licensee stopped their troubleshooting activities and at 3:45 PM (CST), engaged Westinghouse (vendor), SNC corporate, and plant Vogtle personnel for additional assistance in determining the cause of this issue.
The licensee explored possible causes of the abnormal SSPS indications, determined necessary troubleshooting activities, and developed SNC 544534 to implement the desired troubleshooting activities. On January 10, 2014, at 7:40 PM the licensee released SNC 544534 to perform troubleshooting activities. The licensee closed (reset)the 15 volt DC power supply circuit breakers, reset all alarms on the main control board, and replaced four SSPS cards (A108, A313, A516, and A517). At 8:10 PM (CST), the licensee reset SSPS train "B" per Appendix 1 of FNP-2-STP-33.0B and re-performed the surveillance test. On January 11, 2014 at 12:40 AM (CST), progress on the surveillance was stopped due to the "bad" indication at logic switch "B" position 22 and the surveillance was declared unsatisfactory. The licensee revised SNC 544534 and re-commenced troubleshooting activities. The inspectors were unable to establish the exact time the licensee replaced SSPS cards A108 (second replacement with a new card from warehouse stock) and A216 (first replacement of this card), but the timeline indicates these activities occurred between 5:18 AM (CST) and 11:14 AM (CST) on January 11, 2014. Each of the above replacements were task items in SNC 544534. At 11:14 AM (CST), the licensee also replaced SSPS card A313 a second time with another card from their warehouse stock. During the evening hours of January 11, 2014 [4:28 PM (CST) and 7:12 PM (CST)], the licensee performed voltage checks and simulated ground checks on SSPS cards A313 and A516. The licensee's vendor arrived onsite at 11:09 PM (CST) on January 11, 2014 and the licensee replaced SSPS card A313 a third time with a card from their warehouse stock. Throughout the morning hours of January 12, 2014 the licensee performed additional voltage checks on cards A313, A215, and A516. At 4:00 PM (CST) the licensee conducted an industry technical call which included SNC corporate, plant Hatch, and system experts at three similar plants.
The licensee replaced SSPS cards A313 (fourth replacement) and A108 (second replacement) with new cards at 4:54 PM (CST). The licensee discovered a stray termi-point clip shorting pin 16 to pin 17 on SSPS card A516 at 7:13 PM on January 12, 2014 and entered this adverse condition in their corrective action program (CAP) as CR 756274 during extent of condition evaluations. The licensee also discovered a second stray termi-point clip on pin 43 on SSPS card A407. This second stray termi-point clip was observed to be making contact with pin 43 and not shorted to any other component within the SSPS equipment. The licensee removed these stray termi-point clips from Unit 2 train "B" SSPS at 12:38 AM on January 13, 2014. Throughout the evening hours of January 13, 2014, the licensee performed additional troubleshooting for main control board annunciators which had been "locked in" related to the shorting of the termi-point clip. The licensee also performed a successful FNP-2-STP-33.0B surveillance test and declared Unit 2 train "B" SSPS operable at 6:45 PM (CST).
Technical Specification limited conditions of operation (LCO) 3.3.1 and 3.3.2 allowed outage times expired on January 11, 2014, at 9:19 AM (CST). The licensee commenced a Unit 2 power reduction at 11:02 AM in order to meet the 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> TS LCO requirements to be in Mode 3 within six hours. At 2:53 PM, the unit was tripped in accordance with the licensee's shutdown operating procedure at 15 percent rated thermal power. The licensee entered Mode 5 at 8:45 PM (CST) on January 12 and disabled all SSPS outputs at 12:38 AM (CST) on January 13, 2014. The licensee made the Unit 2 reactor critical at 5:07 PM (CST) on January 14 and achieved full power on January 15, 2014.
Unit 2 Train "B" SSPS Card Failures: September 8, 1994 CR 2000504114: High Containment Pressure Safety Injection position of logic switch "B" position 17 tested "bad", licensee determined card A317 had failed. January 29, 2003 CR 200300172: Logic switch "C" position 14 failed to test properly. The licensee determined card A317 had failed.
January 30, 2013 CR 200300173: Logic switch "D" position 14 failed to test properly. The licensee removed card A317, placed the card in slot A212, and placed original card removed during corrective actions for CR 200300172 in slot A317.
December 22, 2005 CR 2005112938: Logic switch "B" position failed to test properly. The licensee replaced cards A210, A313, and A517 and performed a root cause evaluation. January 12, 2006 CR 2006100318: The licensee discovered evidence of tin whiskers on eight universal logic boards on train A SSPS and 5 universal logic boards on train "B" SSPS. The licensee removed the tin whiskers. January 19, 2007 CR 2007100537: Logic switch "B" position 17 failed to test properly. The licensee replaced card A210.
February 2, 2007 CR 2007100962: Multiple main control board annunciator alarms. The licensee was unable to determine exact cause, but identified card A210 as the most likely cause.
April 16, 2007 CR 2007103509: During scheduled maintenance, the licensee performed discovery testing of universal logic boards with KIMKA tester and 15 of the 20 boards tested failed. The licensee also wrote CR 2007103727 for a driver board failing to properly KIMKA test.
December 21, 2007 CR 2007112795: Logic switch "B" position 23 failed to test properly (no indication). The licensee moved the switch back to position 22 and it failed to test properly ("bad" indication lit). The licensee performed a root cause evaluation and concluded that the most probable causes were intermittent component related failures of the semi-automatic tester board and clock counter boards. The licensee implemented policy of testing all boards prior to shipment offsite for refurbishment and testing prior to restocking in the warehouse and also changed test temperature from 70 degrees F to 100 degrees F.
October 26, 2008 No CR listed: Two instances of new cards failing KIMKA testing. These were decoder cards in warehouse inventory.
October 24, 2011 CR 365251: Logic switch "B" position 17 was selected and the amber test light remained illuminated. The licensee discovered a broken wire which created a ground to inhibit signal for position B-17. The licensee reconnected the wire.
October 26, 2011 CR 366069: The licensee discovered that wire A216-19 should be attached on deck "J" of logic switch "B" (S502) during post modification testing for reactor coolant pump circuit breaker anticipatory reactor trip modification. The licensee attached the wire.
April 25, 2013 CR 628483: All logic and permissive test positions for 1/1; 2/2; or 1/1 coincidences failed to indicate "good" with blocks inhibited.
The licensee discovered card A103 had failed. The licensee replaced failed card.
August 27, 2013 CR 694503: Main control board annunciator "Hi 3 CTMT press" alarmed and trip status light box (TSLB) 3.2 illuminated. The licensee suspected a card failure in the 7300 process system. Inspectors obtained no additional information.
August 29, 2013 CR 695348: Logic switch "B" position 17 failed to test properly during performance of surveillance FNP-2-STP-33.0B which was being used as one of the post maintenance tests to restore operability of containment pressure channel suspected in CR 694503 above. Additional indications included logic switch "D" positions 15 through 23. The licensee replaced SSPS cards A108, A209, A210, A313, A516 and A517. KIMKA testing identified out of tolerance conditions on cards A209, A313 and A108. The licensee performed an apparent cause determination and identified degraded components on card A108 as the cause.
January 10, 2014 CR 695815: Logic switch "B" position 22 failed to test properly. The licensee actions and cause of this condition are identified in the timeline related to the FNP-2-STP-33.0B surveillance timeline above.
Unit 2 Train "B" SSPS Modifications: March 9, 1995 DCP-95-2-8837 - resized power input fuses and replaced internal resistor in the system's 48 volt DC power supply. Wire modifications were only made to termination cabinets and involved no wiring installed by termi-point clips. March 25, 1995 DCP-B93-2-8626 - performed terminations for the steam generator narrow range level median signal selector. The modification was performed by licensee staff on Unit 1 and Unit 2 train "A" and train "B" SSPS and it removed and re-landed 27 wires with termi-point clip connectors. DCP-95-2-8951 - re-installed wiring within train "A" and train "B" SSPS cabinets to complete the signal path for the logic "A" test circuits and provide the third input to universal logic cards A303, A304 and A305. This modification corrected the bad test light received during performance of the logic "A" test for these circuits. Inspectors did not review the implementing work order (WO) for this design change.
October 12, 2011 SNC 73858 - implemented by WO 2082084113 which performed field wiring modification in SSPS train "B" input bays. The work was performed by licensee staff and involved the landing of wires on terminal in the termination racks. No termi-point clips were involved in this modification.
October 15, 2011 SNC 73860 - implemented by WO 2082084115 which performed internal wiring modifications in the SSPS train "B" logic cabinet to remove anticipatory reactor trip for an open reactor coolant pump circuit breaker. The work was performed by vendor technicians under observation of licensee staff. The work involved the removal and re-landing of 27 wires using termi-point clips to secure # 20 AWG wire to termination points associated with SSPS cards. Additional work orders implemented similar design modification to Unit 1 train "A" and train "B" SSPS and Unit 2 train
"A" SSPS.
October 20, 2011 SNC 73867 - contingency WO to correct any SSPS wiring issues resulting from SNC 73860 above. Inspectors reviewed WO 2082084122 to determine the scope of work and whether the work order was implemented. The work was performed by vendor technicians under observation of licensee staff. Wires were removed to address blown fuses and address lessons learned from Unit 1 and CR 2010114981. No termi-clips were involved in this work activity.
a. Inspection Scope
The inspectors reviewed the licensee's fault tree diagram, completed procedures, and initial work orders to evaluate the licensee's troubleshooting activities related to this event. The inspectors reviewed past work history and modifications performed on the SSPS train "B" logic cabinets to independently assess activities that may have been related to this event. The inspectors also conducted interviews with licensee personnel to verify that troubleshooting activities adequately identified potential causes for the failed SSPS operability surveillance test.
b. Findings and Observations
No findings were identified.
The inspectors concluded that the licensee's troubleshooting activities related to the failed SSPS operability surveillance test were adequate. In addition to replacing SSPS cards A108, A313, A516, and A517, the licensee's troubleshooting activities consisted of evaluating the semi-automatic tester board, SSPS card A213, and SSPS card A416.
The licensee also evaluated pin connections located on cards A515, A516, and A517 for potential causes of failure. Based on their troubleshooting efforts, the licensee determined that the cause of the failed SSPS "B" logic operability test was a loose termi-point clip located between pins 16 and 17 on safeguards card A516, which caused a short circuit between the two pins. The licensee conducted a visual inspection of approximately 80 percent of the "2B" SSPS cabinet and found another loose termi-point clip on pin 43 of card A407. The licensee determined that the loose termi-point clip found on card A407 was not a potential cause of failure due to the as-found location of the pin.
The licensee conducted a root cause evaluation following the restoration of the SSPS
"B" logic to an operable status. The root cause evaluation concluded the root cause to be a lack of foreign material exclusion (FME) accountability in SSPS cabinets. The root cause evaluation also identified two contributing causes leading up to the event. Most notably, that post-maintenance testing during previous Unit 2 "B" train SSPS failure troubleshooting sequences was not adequate to determine that the failure mechanism(s)had been correctly identified and the potential risk of future SSPS failures was not adequately managed or understood. The root cause evaluation also identified that ineffective causal analyses on previous Unit 2 "B" train SSPS failures contributed to the event. The inspectors independently concluded that the most likely cause of the loose termi-point clip was due to work performed during a modification inside the SSPS "B" logic cabinet which failed to identify the extra termi-point clips.
.3 Review and assess the licensee's testing and maintenance practices of the SSPS cards and other similar type cards on both units.
a. Inspection Scope
The inspectors reviewed maintenance and testing activities related to SSPS cards and other similar type cards to determine whether the licensee's preventative maintenance (PM) was reasonably effective in preventing component failures. The inspectors reviewed the licensee's PM program to verify procedures were in place to address age-related degradation for components installed beyond their vendor-recommended life and to address obsolescence of the SSPS and similar type cards. The inspectors also reviewed a sample of procedures to verify that testing ensured the SSPS met technical specifications and was maintained in an operable status. The inspectors conducted interviews with licensee personnel pertaining to preventative maintenance and testing activities. The inspectors also conducted a non-intrusive visual inspection of the Unit 2 SSPS input and logic cabinets to verify the material condition of the SSPS cards and to determine the adequacy of the licensee's inspection and maintenance procedures for identifying potential degradation the SSPS cards.
b. Findings and Observations
No findings were identified.
The inspectors concluded that a more rigorous inspection of the SSPS cards and of other similar type cards by the licensee would have been more effective in identifying loose or damaged termi-point clips and other foreign material. The inspectors noted procedure FNP-0-EMP-1908.01, "Electrical Cabinet Inspection and Cleaning," Version 8.0, contained sufficient detail for inspection of electrical connections in the SSPS logic cabinets if each wire was thoroughly inspected. Specifically, Step 7.2 of the procedure stated to remove dust and loose substances using a vacuum cleaner or clean dry air (30 psi or less.) Step 7.3 of the procedure stated to visually inspect components for corrosion, cracks, or breaks and any signs of overheating or deterioration. Step 7.4 of the procedure stated to ensure wiring connections are tight, not corroded, and insulation shows no sign of overheating or deterioration. However, the inspectors concluded that a visual inspection by use of a boroscope would have been more effective in identifying loose termi-point clips, degraded connections, and other foreign material. During troubleshooting of the shorted pins in the SSPS, the licensee conducted a visual inspection on approximately 80 percent (what could be inspected without moving wires)of the "2B" SSPS cabinet using a boroscope and found another loose termi-point clip on pin 43 of card A407. The loose termi-point clip found on A407 could have caused a reactor trip or have prevented a reactor trip depending on which adjacent pin was shorted. The inspectors noted that the licensee's concern with the potential risk associated with moving the wires inside the SSPS cabinets prevented this type of rigorous inspection of the SSPS system. The inspectors noted from interviews with station personnel that the licensee verbally prohibited craft from moving wires in the SSPS cabinets in the accomplishment of procedure FNP-0-EMP-1908.01. The inspectors also noted that the procedure was silent on moving wiring.
The inspectors also concluded that the licensee's PM program did not rigorously address age-related degradation and obsolescence of the SSPS cards in service. The inspectors identified that the licensee was not testing SSPS cards, when plant conditions allowed, to determine extent of card "drift" on cards installed in the plant. The inspectors also noted that the licensee discovered out of tolerance cards only after failed surveillance activities. Additionally, the inspectors noted that the licensee had Westinghouse test 22 spare SSPS cards and the 4 SSPS cards that were removed from the system during troubleshooting activities, using the Westinghouse Automated Board Tester System (WABTS) in January 2014. Of the 26 total cards tested by Westinghouse, 11 of the spare SSPS cards failed and had to be repaired. The inspectors noted that the 4 cards removed from the system (A313, A108, A516, and A517) passed testing. The inspectors noted that the licensee identified some PMs were not in place on the Unit 2 SSPS due to a pending design change to replace obsolete cards with a new design. The licensee established actions from CAR 207903 to re-create the PMs that were in place before the pending design change until the design change is completed. The licensee also submitted preventative maintenance change request (PMCR) 74654 to request PMs to functionally test obsolete SSPS cards installed in Unit 2.
.4 Review and evaluate the licensee's compliance with vendor recommendations regarding maintenance of SSPS cards and other similar type cards and modifications implemented to the SSPS system.
a. Inspection Scope
The inspectors reviewed the licensee's maintenance procedures and vendor recommendations contained in Westinghouse WCAP-17677-P, "Solid State Protection System Life Cycle Management," to verify that the licensee was appropriately implementing vendor recommendations. The inspectors conducted interviews with licensee personnel to verify that deviations from vendor recommendations were controlled. The inspectors also reviewed a sample of modifications pertaining to internal wiring of the SSPS train "A" and train "B" logic cabinet and main control board demultiplexer cabinet to verify that the licensee implemented vendor recommendations while performing these modifications.
b. Findings and Observations
No findings were identified.
The inspectors concluded that the licensee followed vendor recommendations in modifications implemented to the SSPS system. The inspectors also concluded that the licensee incorporated vendor recommendations into their maintenance procedures with some exceptions. The licensee identified three preventative maintenance activities related to the DC power supplies which were not completed in accordance with vendor recommendations. The licensee submitted PMCRs 74649 and 74651 to include these preventative maintenance activities as part of their PM program.
a. Inspection Scope
The inspectors reviewed licensee corrective action documents to assess the effectiveness of licensee corrective actions in addressing previous issues and abnormal indications with SSPS cards. The inspectors also evaluated potential trends and the inclusion of industry operating experience into corrective actions. The inspectors interviewed station personnel to assess the licensee's methodology for determining the cause of the card failures. The inspectors also reviewed applicable station procedures to evaluate the completeness of corrective actions and ensure that long term corrective actions remained in place.
b. Findings and Observations
No findings were identified.
The inspectors noted multiple instances of SSPS logic switch "B" position 17 indication showing a "bad" test. This logic test position tests the high containment pressure safety injection actuation circuit of SSPS. Over an approximately twenty year period from September, 1994 to January, 2014, the inspectors noted ten instances of "bad" indications as described above. The licensee's corrective actions consisted of replacing cards in the SSPS cabinet. The inspectors evaluated these card replacements for potential trends but did not see conclusive evidence of repeat failures of the same card.
The inspectors noted additional instances of SSPS logic switches being selected to a test position and receiving "bad" indication from the testing circuit. The corrective actions for these occurrences were to replace SSPS cards.
The inspectors noted that on December 21, 2007 random failures were observed by the licensee during the performance of SSPS logic testing. For test positions 22 and 23 of SSPS logic switch "B" the testing lamp remained illuminated without returning a "good" or "bad" test result. The licensee replaced semiautomatic tester and clock counter boards (SSPS cards A103 and A108) which failed to correct the condition. The licensee subsequently re-installed the original semi-automatic tester and clock counter boards and replaced universal logic boards (SSPS cards A210 and A313) and safeguards driver boards (SSPS cards A516 and A517). Additionally, the licensee performed a root cause determination which the inspectors reviewed. The inspectors noted that the licensee's root cause evaluated a previous long range plan of interim actions of using on-site card repair and KIMKA testing until Westinghouse (card vendor) provides new design boards.
The licensee determined that this corrective action had not prevented repetitive logic testing failures due to some cards being omitted from testing and KIMKA testing at ambient versus elevated temperatures. The root cause proposed testing of a larger population of SSPS cards including those current contained in warehouse stock, any card removed from the SSPS cabinets for inspection or troubleshooting, and any refurbished cards. Additionally, the root cause proposed testing of the cards at elevated temperatures (140 degrees F). The licensee implemented these proposed corrective actions and the inspectors noted that many additional abnormal indications were discovered with the SSPS cards as a result of the testing.
The inspectors also noted that on August 27, 2013, Unit 2 entered a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> LCO due to an inoperable containment pressure "Hi-3" alarm. While the licensee troubleshot this adverse condition, a 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> LCO was entered for failure to satisfactorily complete FNP-2-STP-33.0B, "Solid State Protection System Train B Operability Test." The inspectors noted the licensee replaced SSPS cards for the semi-automatic tester (A108), universal logic boards (A208, A210, and A313), and safeguards driver boards (A516 and A517).
The licensee performed KIMKA testing and discovered cards A209, A313, and A108 to be out of tolerance. These cards were removed from the system. The licensee performed a lower tier root cause (apparent cause) for this event and determined the cause was a degraded component on the A108 semi-automatic tester board and inadequate preventative maintenance. The licensee evaluated Westinghouse Technical Bulletin 10-2 and determined that no actions were required. The apparent cause stated that Technical Bulletin 10-2 conclusions were that the semi-automatic tester portion of the SSPS system detects hard circuit failures but may not detect degraded circuits. The bulletin recommended implementing preventative maintenance to perform component level board testing or perform full manual testing on the system, replacing original design boards with new design boards, and performing power supply maintenance to ensure proper operation. The apparent cause assigned a cause code of 8A3 "inadequate or no preventative maintenance," and identified that no preventative maintenance was in place to test SSPS boards on the component level as recommended in Technical Bulletin 10-2. The inspectors reviewed licensee corrective actions and determined the licensee had initiated TE 12715 (action item) to create planned maintenance to KIMKA test original design SSPS boards during refueling outages or add testing of boards to existing tin whisker planned maintenance. The due date assigned for the action item (TE 12715) is March 21, 2014.
The inspectors focused upon the above condition reports because they determined these events were significant in the history of cards failures at the site. The inspectors questioned the lack of current planned maintenance to identify potentially degraded components within the SSPS system (cards out of tolerance) prior to the existence of a failed card. However, the inspectors did not identify any violations of regulatory requirements.
a. Inspection Scope
The inspectors reviewed documents provided by the licensee to develop a detailed sequence of events associated with licensee activities from the point of time the licensee declared FNP-2-STP-33.0B, "Solid State Protection System Train B Operability Test" failed until Unit 2 train "B" SSPS was declared operable. The inspectors also interviewed station personnel in the development of this timeline (contained in Section 4OA5.1.) The inspectors evaluated licensee's immediate actions to return train "B" SSPS to an operable state, reviewed condition reports, and interviewed station personnel to assess the licensee's corrective actions. The inspectors also reviewed station work orders and procedures to determine the effectiveness of these corrective actions in restoring system operability.
b. Findings and Observations
No findings were identified.
The inspectors determined that licensee corrective actions were adequate to restore system operability. The inspectors noted the licensee replaced multiple SSPS cards, performed voltage checks of system circuitry, and performed testing of the manual safety injection circuitry to ensure operator actions would have been adequate to establish a train "B" safety injection actuation in the event one would have been needed.
The inspectors noted the licensee's troubleshooting steps followed a pre-established path mostly likely based upon previous site history with SSPS card failures. The inspectors noted that the licensee contacted industry experts to assist in troubleshooting activities. The inspectors determined that post maintenance testing adequately established that all circuits were working properly and adequately established system operability.
The licensee noted that a termi-point clip shorting across safeguards driver card A516 was the cause of this event. The inspectors evaluated the licensee's extent of condition activities regarding Unit 2 train "B" SSPS and determined it was adequate. The inspectors noted that the licensee visually inspected approximately 80 percent (what could be inspected without moving wires) of the Unit 2 train "B" SSPS cabinet using a boroscope. This inspection revealed an additional termi-point clip sitting atop pin 43 of SSPS card A407. The inspectors questioned the licensee regarding the potential for additional loose termi-point clips in Unit 1 train "A" and "B" SSPS cabinets. The licensee stated that this was very unlikely due to the lack of history of card issues in these cabinets and the use of new style digital cards which provide immediate indication of a circuit problem. The inspectors determined the licensee's extent of condition for Unit 1 was adequate. The inspectors questioned licensee personnel in regards to Unit 2 train "A" SSPS and were informed that loose termi-point clips most likely do not exist on this equipment. The licensee stated that lack of card failure history and multiple satisfactory surveillance tests support this conclusion. The inspectors determined that loose termi-point clips most likely were introduced into Unit 2 train "B" SSPS during modifications and similar modifications had been performed on Unit 2 train "A" SSPS. The inspectors noted that the licensee had already established corrective actions to inspect Unit 2 train
"A" SSPS and Unit 1 train "A" and "B" SSPS for loose termi-point clips in condition reports 757196 and 757198, respectively.
.7 Assess the licensee's actions resulting from vendor technical bulletins and industry operating experience concerning SSPS and similar type cards.
a. Inspection Scope
The inspectors reviewed applicable vendor technical bulletins and industry operating experience (OE) related to the SSPS and similar cards. The inspectors reviewed the licensee's response to technical bulletin 89-06, "Termi-Point Clips;" technical bulletin 05-4, "Potential Tin Whiskers on Printed Circuit Board Components;" and NRC Information Notice 2005-25, "Inadvertent Reactor Trip and Partial Safety Injection due to Tin Whisker;" to evaluate how the licensee reviewed and dispositioned each item. The inspectors reviewed maintenance procedures and work orders to acquire objective evidence that the operating experience for each item had been resolved to the extent practical. The inspectors also reviewed OE related to loose termi-point clips in SSPS and similar type cards.
b. Findings and Observations
No findings were identified.
The inspectors determined that the licensee appropriately incorporated the above operating experience into station procedures and planned maintenance. The inspectors also reviewed station design modification documents to assess the use of this information.
The inspectors reviewed industry OE provided by the licensee. The inspectors discovered three industry events related to loose termi-point clips that occurred during 2002. The inspectors evaluated this information to determine if the licensee had appropriately incorporated this information into station procedures, planned maintenance and system modifications. Because the three events were low level events, the inspectors determined that station procedures and planned maintenance would not have included this OE. However, the inspectors identified that station procedure NMP-ES-022, "DCP Site Approval, Implementation and Closure," Version 9.2 required a review of industry operating experience that is related to the design change for use in planning the implementation of associated work orders. This requirement is explicitly stated in step 5.6.13 of NMP-ES-022. The inspectors requested documentation of this search and were provided a single page from the design document which did not include these three industry events. The inspectors requested the licensee perform a search of the industry operating experience database using the keyword search "SSPS." The licensee informed the inspectors that this search did in fact capture these three events but also captured at least one hundred other events. The inspectors concluded that had the licensee included this information into station work orders, the loose termi-point clips may have been prevented. The inspectors also concluded that no performance deficiency existed as there was no requirement to capture specific events from OE searches.
The inspectors noted the licensee had incorporated Westinghouse technical bulletin 89-06, "Termi-Point Clips" and performed an inspection of Unit 1 and Unit 2 train "A" and "B" SSPS for potential loose wires prior to any plant modifications. The inspectors also noted the licensee inspected SSPS wires attached with termi-point clips and discovered no loose wires. The inspectors determined this work was performed prior to 2002 before the industry operating experience discussed above occurred.
.8 Collect data necessary to support completion of the significance determination process, if applicable.
The inspectors did not identify any performance deficiencies and therefore this charter item was not applicable.
.9 Identify any potential generic safety issues and make recommendations for appropriate follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).
a. Inspection Scope
The inspectors assessed information provided by the licensee in documents and interviews for potential generic safety issues. The inspectors reviewed plant history related to previous SSPS card issues. The inspectors also evaluated licensee corrective actions in an effort to discover potential causes and likely potential generic safety issues.
b. Findings and Observations
The inspectors did not identify any potential generic safety issues.
4OA6 Meetings, Including Exit
On January 24, 2014, the inspection team presented the inspection results to C. Gayheart, SVP and other members of the licensee staff. The inspection team confirmed that proprietary information was provided and examined during the inspection period. The inspection team left all proprietary information at the site.
ATTACHMENTS:
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee personnel
- B. Arens, Farley Licensing Supervisor
- H. Cooper, Engineering Programs Manager (Acting)
- C. Gayheart, FNP Site VP
- G. Gunn, Vogtle Licensing Supervisor
- J. Horn, Farley Regulatory Affairs Manager
- D. McKinney, SNC Fleet Regulatory Affairs Manager
- J. Seales, Engineering Systems Supervisor
- S. Tipps, Hatch Licensing Supervisor
NRC personnel
- N. Carte, Senior Electronics Engineer
- P. Niebaum, Senior Resident Inspector, Farley
- J. Sowa, Resident Inspector, Farley
- F. Ehrhardt, Chief, Region II Reactor Projects Branch 2
LIST OF ITEMS
OPENED AND CLOSED OPEN AND CLOSED
None
OPENED
None
CLOSED None
LIST OF DOCUMENTS REVIEWED
Condition Reports:
- 365251,
- 366069,
- 628483,
- 694503,
- 695348,
- 695815,
- 757196,
- 757198,
- 2000504114,
- 200300172,
- 200300173,
- 2005106070,
- 2005106109,
- 2005112938,
- 2006100318,
- 2007100537,
- 2007100962,
- 2007103509,
- 2007112795
- Design Changes:
- DCP-B93-2-8626, Perform Termination for the Steam Generator Narrow Range Level Median Signal Selector
- DCP-B95-2-8837, Resize Power Input Fuses and Replacement of Internal Resistor in 48 DC Power Supply
- DCP-B95-2-8951, Re-install Wiring Within Train A & B SSPS Cabinets to Complete Signal Path for the Logic A Test Circuits
- SNC-73858, Perform Field Wiring Modifications in SSPS Train B Input Channels
- SNC-73860, Perform Internal Wiring Modifications In SSPS Logic Cabinet Train B
- SNC-73867, Contigency WO for Westinghouse to Correct Any SSPS Wiring Issues After Completion of
- SNC-73860
Miscellaneous
- Documents:
- CAR 207903, Failure of SSPS B Logic Operability Test Apparent Cause Determination Extent of Condition Paper, 01/20/14
- Farley PMs vs. Sourcebook Recommendations JC, Rev.1 Fault Tree for
- STP-33.0B, 01/12/14 NRC Information Notice 2005-25, Inadvertent Reactor Trip and Partial Safety Injection Due to Tin Whisker
- NMP-AP-001-F02, Input Westinghouse Procedure
- NS-FSI-09-26 into SNC Procedure Database, 01/13/14
- PMCR 74651, Energize Spare SSPS Power Supplies
- PMCR 74654, Functional Test of Old Style SSPS Cards Termi-Point OE Reports Unit 2, B Train Solid State Protection System Operability Test Failure January 10, 2014
- Presentation, 01/21/14 WABTS Solid State Protection System Spare Board Testing Service, 01/13/14
- WCAP-17677-P, Solid State Protection System Life Cycle Management Planning Sourcebook (PA-SEE-0656), Rev.0
- Westinghouse Printed Circuit Card Test Reports, 01/13/14 Westinghouse Technical Bulletin 05-4, Potential Tin Whiskers on Printed Circuit Board
- Components Work Request
- 214443, SSPS Train B response to
- TB-89-06, 10/17/89
- Work Request
- 214490, SSPS Cont. Bd. Demux. Cab. response to
- TB-89-06, 10/25/89
- Work Request
- 226067, SSPS Train B response to
- TB-89-06, 10/09/90
- Work Request
- 226068, SSPS Train A response to
- TB-89-06, 10/09/90 Work Request
- 226071, Control Board Demultiplexer response to
- TB-89-06, 10/10/90
- Attachment Procedures:
- FNP-0-ACP-7.0, Foreign Material Exclusion Program, Version 10.0
- FNP-0-EMP-1908.01, Electrical Cabinet Inspection and Cleaning, Version 8.0
- FNP-2-STP-33.0B, Solid State Protection System Train B Operability Test, Version 53.1 (work completed 01/10/14)
- NMP-ES-022, DCP Site Approval, Implementation and Closure, Version 9.2
- NMP-MA-009-001, Foreign Material Exclusion Program Requirements, Version 5.1
- NMP-MA-015, 7300 Process Protection & Control System Printed Circuit Board Management,
- Version 1.0
- Work Orders:
- 1051778701, Inspect 1A SSPS cards for tin whiskers per
- TB-05-4, 04/11/06
- 1051778801, Inspect 1B SSPS cards for tin whiskers per
- TB-05-4, 04/14/06
- 2051776301, Inspect 2B SSPS cards for tin whiskers per
- TB-05-4, 10/08/05
- 2051776401, Inspect 2A SSPS cards for tin whiskers per
- TB-05-4, 10/18/05
- 2070003001, Perform 2A SSPS tin whisker card inspection, 10/29/08
- SNC394685, Provide low resistance path from Logic "L" ground to Channel II E Ground Bus,
- 05/17/12 SNC544534, Solid State Protection System Logic Cabinet Train B, 01/10/14