IR 05000483/2015009

IR 05000483/2015009, September 21, 2015 Through December 9, 2015, Callaway Plant - Special Report

ML16013A021
Person / Time
Site:	Callaway
Issue date:	01/13/2016
From:	Nick Taylor NRC/RGN-IV/DRP/RPB-B
To:	Diya F Union Electric Co
Taylor N
References
IR 2015009
Download: ML16013A021 (56)

Text

January 13, 2016

SUBJECT:

CALLAWAY PLANT - NRC SPECIAL INSPECTION REPORT 05000483/2015009

Dear Mr. Diya:

On December 9, 2015, the U.S. Nuclear Regulatory Commission (NRC) completed a Special Inspection at your Callaway Plant to evaluate the facts and circumstances surrounding the on-demand failures of three auxiliary feedwater system flow control valves. Based on the risk and deterministic criteria specified in NRC Management Directive 8.3, NRC Incident Investigation Program, the NRC initiated a special inspection in accordance with Inspection Procedure 93812, Special Inspection. The basis for initiating the special inspection and the focus areas for review are detailed in the Special Inspection Charter (Attachment 2). The NRC determined the need to perform a special inspection on September 8, 2015, and the onsite inspection started on September 21, 2015. The enclosed report documents the inspection findings that were discussed on September 25, December 9, and December 17 2015, with you and members of your staff. The team documented the results of this inspection in the enclosed inspection report.

NRC inspectors documented seven findings of very low safety significance (Green) in this report. All seven of these findings involved violations of NRC requirements. The NRC is treating these violations as non-cited violations (NCVs) consistent with Section 2.3.2.a of the NRC Enforcement Policy.

If you contest the violations or significance of these NCVs, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region IV; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC resident inspector at the Callaway Plant.

If you disagree with a cross-cutting aspect assignment in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV; and the NRC resident inspector at the Callaway Plant.

In accordance with Title 10 of the Code of Federal Regulations (10 CFR) 2.390, Public Inspections, Exemptions, Requests for Withholding, a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRCs Public Document Room or from the Publicly Available Records (PARS) component of the NRC's Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Nicholas H. Taylor, Branch Chief Project Branch B Division of Reactor Projects

Docket No. 50-483 License No. NPF-30

Enclosure:

Inspection Report 05000483/2015009 w/ Attachment 1: Supplemental Information

Attachment 2: Special Inspection Charter

Attachment 3: Detailed Risk Evaluation

ML16013A021 SUNSI Review By: DLP ADAMS Yes No Non-Sensitive Sensitive Publicly Available Non-Publicly Available Keyword:

OFFICE SPE:DRP/B SRI:EB1 SRI:NRO SRA:DRS C:DRP/B

NAME DProulx RKopriva JJacobson DLoveless NTaylor

SIGNATURE

/RA/

E-mail/RA/

E-mail/RA E-mail/RA/

/RA/

DATE 12/10/15 12/18/15 12/16/15 12/17/15 12/31/15

Letter to from N. Taylor dated January 13, 2016

SUBJECT: CALLAWAY PLANT - NRC SPECIAL INSPECTION REPORT 05000483/2015009

DISTRIBUTION:

Regional Administrator (Marc.Dapas@nrc.gov)

Deputy Regional Administrator (Kriss.Kennedy@nrc.gov)

DRP Director (Troy.Pruett@nrc.gov)

DRP Deputy Director (Ryan.Lantz@nrc.gov)

DRS Director (Anton.Vegel@nrc.gov)

DRS Deputy Director (Jeff.Clark@nrc.gov)

Senior Resident Inspector (Thomas.Hartman@nrc.gov)

Resident Inspector (Michael.Langelier@nrc.gov)

Branch Chief, DRP/B (Nick.Taylor@nrc.gov)

Senior Project Engineer, DRP/B (David.Proulx@nrc.gov)

Project Engineer, DRP/B (Shawn.Money@nrc.gov)

Project Engineer, DRP/B (Steven.Janicki@nrc.gov)

CWY Administrative Assistant (Dawn.Yancey@nrc.gov)

Public Affairs Officer (Victor.Dricks@nrc.gov)

Project Manager (John.Klos@nrc.gov)

Acting Team Leader, DRS/TSS (Eric.Ruesch@nrc.gov)

RITS Coordinator (Marisa.Herrera@nrc.gov)

ACES (R4Enforcement.Resource@nrc.gov)

Regional Counsel (Karla.Fuller@nrc.gov)

Technical Support Assistant (Loretta.Williams@nrc.gov)

Congressional Affairs Officer (Jenny.Weil@nrc.gov)

RIV Congressional Affairs Officer (Angel.Moreno@nrc.gov)

RIV/ETA: OEDO (Cindy.Rosales-Cooper@nrc.gov)

ROPreports (ROPreports.Resource@nrc.gov)

ROPAssessments (ROPassessment.Resource@nrc.gov)

Enclosure U.S. NUCLEAR REGULATORY COMMISSION

REGION IV

Docket:

05000483 License:

NPF-30 Report:

05000483/2015009 Licensee:

Union Electric Company Facility:

Callaway Plant Location:

Steedman, MO 65077 Dates:

September 21 through December 9, 2015 Inspectors:

D. Proulx, Senior Project Engineer, Project Branch B J. Jacobson, Senior Inspector, Vendor Inspection Branch, NRO R. Kopriva, Senior Reactor Inspector, Engineering Branch Approved By:

Nicholas H. Taylor Chief, Project Branch B Division of Reactor Projects

SUMMARY OF FINDINGS

IR 05000483/2015009; 09/21/2015 - 12/09/2015; Callaway Plant; Special Inspection to Evaluate

Causes of the Failure of Motor-Driven Auxiliary Feedwater Pump Flow Control Valves at the Callaway Plant

The special inspection activities described in this report were performed between September 21 and December 9, 2015, by two NRC region-based inspectors and one NRC headquarters inspector. Seven findings of very low safety significance (Green) are documented in this report.

All of these findings involved violations of NRC requirements. The significance of inspection findings are indicated by their color (Green, White, Yellow, or Red), which is determined using Inspection Manual Chapter 0609, Significance Determination Process, dated April 29, 2015.

Their cross-cutting aspects are determined using Inspection Manual Chapter 0310, Aspects within the Cross-Cutting Areas, dated December 4, 2014. Violations of NRC requirements are dispositioned in accordance with the NRC Enforcement Policy, dated February 4, 2015. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 5.

Cornerstone: Mitigating Systems

Green.

The team identified a non-cited violation of 10 CFR Part 50, Appendix B,

Criterion III, Design Control, for the licensees failure to assure that the design of the replacement reverse-engineered Modutronics controller cards for the auxiliary feedwater control valves were suitable for their application. Specifically, as of August 11, 2015, the licensee failed to establish suitable interface requirements in procurement documents to Nuclear Logistics Incorporated (the vendor) and verify the adequacy of the design by either design reviews or testing. Specifically, the team identified that neither the licensee nor the vendor had performed a design review sufficient to assure that the Modutronics controller cards were suitable for their application. In addition, the licensee had not provided the vendor with sufficient information to reverse-engineer the controller cards. Lastly, neither the licensee nor the vendor performed testing sufficient to verify the adequacy of the design of the new Modutronics controller cards. As a result, the replacement cards were supplied with motor field current rectifier bridges that were undersized and marginal for their application, such that two of them failed in service, rendering these auxiliary feedwater system valves inoperable. Following performance of a root cause analysis, the licensee replaced the deficient controller cards with those of a higher current rating. The licensee initiated Callaway Action Request 201505796 to place this item into the corrective action program.

The failure to ensure that the design of the replacement for the Modutronics cards was suitable for their application was a performance deficiency. This performance deficiency is more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Design Control and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, design deficiencies associated with these circuit cards resulted in the inoperability of auxiliary feedwater control valves and their ability to operate on demand. The team performed an initial screening of the finding in accordance with Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-Power. Using Inspection Manual Chapter 0609, Appendix A, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the team determined that the finding required a detail risk evaluation because it represented the potential loss of one train of safety-related equipment (auxiliary feedwater) for greater than the technical specification allowed outage time. A Region IV senior reactor analyst performed a detailed risk evaluation in accordance with Appendix A,

Section 6.0, Detailed Risk Evaluation, which determined that the finding was of very low safety significance (Green). The analyst determined that the importance of the failure of valves ALHV0005 and ALHV0007 was based on the postulated failure time of the turbine-driven auxiliary feedwater pump because this determined the position in which the valves failed. The internal events incremental conditional core damage probability was 8.17 x 10-7. The analyst also determined that the finding had only a minimal effect on external initiator risk and that the finding would not involve a significant increase in the risk of a large, early release of radiation.

This finding has a human performance cross-cutting aspect in the area of teamwork, because individuals in different work groups did not appropriately communicate across organizational boundaries. Specifically, licensee personnel did not adequately communicate the design and testing requirements for the reverse engineered cards (H.4).

(Section 4OA5.2.2.b)

Green.

The team reviewed a self-revealing non-cited violation of 10 CFR Part 50,

Appendix B, Criterion V, Instructions, Procedures, and Drawings, for failure to prescribe activities affecting quality using procedures appropriate to the circumstances. Specifically, on November 18, 2009, the licensee revised Procedure MTE-ZZ-QA033, MOVATS UDS

[motor operated valve actuator test system universal diagnostic system] Testing of Torque Controlled Modutronics Limitorque Motor Operated Rising Stem Valves, Revision 3, to incorporate a second method of valve testing, and introduced an error in bypassing a test of the Modutronics board setup feedback potentiometer. As a result, on July 23, 2015, the actuator misinterpreted the actual position of the valve, which subsequently failed to open when operators attempted to open the valve following a forced reactor shutdown. In response to this issue, the licensee has reviewed all maintenance and test activities that could affect the potentiometer and has revised the appropriate procedures. This finding was entered into the licensees corrective action program as Callaway Action Request 201505332.

The failure to provide a procedure appropriate to the circumstances for an auxiliary feedwater system flow control valve was a performance deficiency. This finding was more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Procedure Quality and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to provide a procedure appropriate to the circumstances to set up an auxiliary feedwater system flow control valve feedback potentiometer resulted in its inability to operate manually on demand. Using Inspection Manual Chapter 0609, Appendix A, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the finding was determined to be of very low safety significance, because it did not affect system design, did not result in a loss of system function, did not represent a loss of function of a single train for greater than its technical specifications allowed outage time, and did not cause the loss of function of one or more non-technical specification trains of equipment designated as high safety-significance. The valve would have automatically throttled auxiliary feedwater flow to approximately 300 gpm on demand. This finding did not have a cross-cutting aspect because the procedure revision resulting in the inadequate procedure was issued in 2009, and previous opportunities to correct the procedure occurred in 2010. Thus, this performance deficiency was not indicative of current licensee performance (Section 4OA5.2.3.b.1).

Green.

The team reviewed a self-revealing non-cited violation of 10 CFR Part 50,

Appendix B, Criterion XI, Test Control, for failure to ensure that testing demonstrated that structures, systems, and components will perform satisfactorily in service. Specifically, on October 24, 2014, the licensee failed to establish a suitable post-maintenance test program to demonstrate that the motor-driven auxiliary feedwater flow control valve Modutronics potentiometer had been set correctly after maintenance. The testing consisted of stroking the valve full open or full closed, and did not consider step changes in valve positioning and did not confirm the potentiometer feedback settings during valve positions that were not full open or full closed. In response to this issue, the licensee performed another calibration of the potentiometer, focusing on the potentiometer position during the valve stroke. The new post-maintenance test included opening the valve in discreet step changes to test the valve position feedback potentiometer. This finding was entered into the licensees corrective action program as Callaway Action Request 201505332.

The failure to establish a suitable post-maintenance test program to demonstrate that the motor-driven auxiliary feedwater flow control valve Modutronics potentiometer would be set correctly after maintenance or testing was a performance deficiency. This finding was more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to establish a post-maintenance testing program for the motor-driven auxiliary feedwater valve Modutronics potentiometer to verify that the potentiometer was set correctly, resulted in valve ALHV0011 failing to open when operators initiated a signal to place the valve in an open position. In accordance with Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings at Power, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the issue screened as having very low safety significance (Green) because it was a design or qualification deficiency that did not represent a loss of operability or functionality; did not represent an actual loss of safety function of the system or train; did not result in the loss of one or more trains of non-technical specification equipment; and did not screen as potentially risk significant due to seismic, flooding, or severe weather. The team determined that this finding did not have a cross-cutting aspect because the most significant contributor did not reflect current licensee performance. (Section 4OA5.2.3.b.2)

Green.

The team identified two examples of a non-cited violation of 10 CFR Part 50,

Appendix B, Criterion V, Instructions, Procedures, and Drawings, for the licensees failure to implement their corrective action program procedure. Specifically: (1) on November 20, 2014, the licensee designated the improper setting of the auxiliary feedwater flow control valve ALHV005 limit switches as Significance Level 5 (administrative close) instead of Significance Level 3 (lower tier cause evaluation) and (2) on December 9, 2014, the licensee downgraded the failure of the Modutronics card for valve ALHV0005 from Significance Level 1 (root cause analysis) to Significance Level 3 based on unverified assumptions of the failure mechanisms. Following failure of the Modutronics card for valve ALHV0005, the licensee assumed that the early failure was due to a manufacturing defect (infant mortality) without supporting data to prove this designation. The licensee entered these issues into the corrective action program as Callaway Action Requests 201506921 and 201507235.

The two failures to properly designate the Significance Level of Callaway action requests constitute a performance deficiency. This finding was more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failures to properly designate the significance of the conditions precluded determining the appropriate cause determinations and extent of conditions and resulted in failure to correct the conditions before they further manifested themselves following a trip. Using Inspection Manual Chapter 0609, Appendix A, Exhibit 2,

Mitigating Systems Screening Questions, dated June 19, 2012, the finding was determined to be of very low safety significance, because it did not affect system design, did not result in a loss of system function, did not represent a loss of function of a single train for greater than its technical specifications allowed outage time, and did not cause the loss of function of one or more non-technical specification trains of equipment designated as high safety-significance. This finding has a human performance cross-cutting aspect in the area of conservative bias in that the decision-making did not demonstrate a conservative/prudent choice in designating the significance level of the Callaway action requests based on two cases of unverified/incorrect information (H.14). (Section 4OA5.2.4.b.1)

Green.

The team identified a non-cited violation of 10 CFR Part 50, Appendix B,

Criterion XVI, Corrective Action, for the licensees failure to determine the cause and take corrective action to preclude repetition for a significant condition adverse to quality.

Specifically, on May 21, 2015, the licensee received new information that refuted the previously assumed failure mechanism for AFW flow control valve ALHV0005 documented in December 2014, but failed to initiate a new Callaway action request to document the new information and report it to appropriate levels of management. As a result, the licensee failed to identify the failure of the valve as a significant condition adverse to quality, determine the cause, initiate a prompt operability assessment, and identify corrective action to preclude repetition until valve ALHV0007 failed, for the same reason, following a reactor trip on August 11, 2015. The licensee entered this issue into the corrective action program as Callaway action request 201506846.

The failure to determine the cause and take corrective action to preclude repetition for a significant condition adverse to quality when failure analysis indicated that a significant defect existed on valves ALHV0005 and ALHV0007 was a performance deficiency. This finding was more than minor, and therefore, a finding, because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure of the licensee to determine the cause and take corrective action to preclude repetition for a significant condition adverse to quality when new information on the failure mechanism was received precluded determining the root cause and extent of condition and the performance of an operability determination, which resulted in failure to correct the condition before it further manifested itself following a reactor trip. Using Inspection Manual Chapter 0609,

Appendix AProperty "Inspection Manual Chapter" (as page type) with input value "NRC Inspection Manual 0609,</br></br>Appendix A" contains invalid characters or is incomplete and therefore can cause unexpected results during a query or annotation process., Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the finding was determined to be of very low safety significance, because it did not affect system design, did not result in a loss of system function, did not represent a loss of function of a single train for greater than its technical specifications allowed outage time, and did not cause the loss of function of one or more non-technical specification trains of equipment designated as high safety-significance. This finding has a human performance cross-cutting aspect in the area of consistent process in that the individuals that received the information concerning the failure mechanism of the Modutronics cards failed to use a systematic approach to documenting the information and communicating it to appropriate levels of management (H.13). (Section 4OA5.2.4.b.2)

Green.

The team reviewed a self-revealing non-cited violation of 10 CFR Part 50,

Appendix B, Criterion V, Instructions, Procedures, and Drawings, for failure to provide a procedure appropriate to the circumstances. Specifically, on March 4, 2014, the licensee performed Job 08505547, and had not correctly accounted for the differential pressure the valve would actually experience, and had incorrectly set and tested the close torque switch on valve ALHV0005. As a result, On November 15, 2015, during steam generator filling operations, Valve ALHV0005 failed to move in the closed direction when the torque switch opened. The incorrect close torque switch setting prevented the valve from going full closed. In response to this issue, the licensee, using Job 14005755, repaired the valve, and confirmed that the close torque switch settings were correct and successfully retested. This finding was entered into the licensees corrective action program as Callaway Action Report 201508399.

The failure to establish a procedure that included a suitable instructions to set the torque switch on a motor-driven AFW valve after maintenance or testing was a performance deficiency. This finding was more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to establish a post-maintenance testing program for the motor-driven auxiliary feedwater valve torque and thrust settings caused valve ALHV0005 not to close completely, causing the operators to take action and shut down motor-driven feedwater pump B. In accordance with Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings at Power, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the issue screened as having very low safety significance (Green) because it was a design or qualification deficiency that did not represent a loss of operability or functionality; did not represent an actual loss of safety function of the system or train; did not result in the loss of one or more trains of non-technical specification equipment; and did not screen as potentially risk significant due to seismic, flooding, or severe weather. The team determined that this finding had a cross-cutting aspect in the area of human performance, challenge the unknown, because the licensee did not stop and challenge that the tested differential pressure across valve ALHV0005 was significantly different than the other valves (H.11).

(Section 4OA5.2.4.b.3)

Green.

The team identified a non-cited violation of 10 CFR Part 50, Appendix B,

Criterion XVI, Corrective Action, for the licensees failure to identify and correct a condition adverse to quality. Specifically, as of September 23, 2015, the licensee had not taken corrective action, following previous identification of undersized field current rectifier bridges, to ensure that an independent review of the modified circuit design had been completed, or that the modified cards had been subjected to a sufficient testing and qualification program.

Thus, following questioning by the team, the licensee identified additional components (two other rectifier bridges) on the newly modified circuit cards that were also potentially undersized. The licensee performed an operability evaluation and concluded that the new cards were operable, based on additional circuit analysis that was performed. This issue was entered into the corrective action program as Callaway Action Request 201506874.

The failure to identify and correct a condition adverse to quality was a performance deficiency. This performance deficiency is more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences.

Specifically, the failure to identify and correct design deficiencies associated with these circuit cards could have resulted in the inoperability of auxiliary feedwater control valves and their inability to operate on demand. Using Inspection Manual Chapter 0609, Appendix A,

Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the finding was determined to be of very low safety significance, because it did not affect system design, did not result in a loss of system function, did not represent a loss of function of a single train for greater than its technical specifications allowed outage time, and did not cause the loss of function of one or more non-technical specification trains of equipment designated as high safety-significance. This finding has a human performance cross-cutting aspect in the area of Avoid Complacency, because the licensee did not thoroughly evaluate the issue to ensure that the resolutions address causes and extent of conditions. Specifically, the licensee had identified that the Modutronics cards failed because of improper design of the field current rectifier bridge, but did not plan for the possibility for other latent issues to determine if other components on the cards were adequately sized for their application (H.12). (Section 4OA5.2.6.b).

REPORT DETAILS

OTHER ACTIVITIES

Cornerstones: Mitigating Systems

4OA5 Other Activities

.1 Basis for Special Inspection

Background

The auxiliary feedwater (AFW) system at Callaway Plant includes three safety-related trains. One of these trains is served by a turbine-driven pump and is capable of providing flow to each steam generator through an air-operated flow control valve. The other two trains are served by motor-driven pumps, which provide flow to two steam generators each through motor-operated flow control valves. The motor-operated flow control valves (ALHV0005, ALHV0007, ALHV0009, and ALHV0011) provide feed flow to the steam generators as follows:

Motor-Driven AFW Pump A: ALHV0009, Steam Generator B

ALHV0011, Steam Generator C

Motor-Driven AFW PUMP B ALHV0005, Steam Generator D

ALHV0007, Steam Generator A

The AFW motor-operated flow control valves are operated in automatic and remote manual modes. The valves are normally open and in automatic control in standby mode. During automatic operation, the flow control valves are energized following a start of the associated motor-driven AFW pump. They then modulate to maintain approximately 300 gpm through each valve, using a Modutronics controller card that compares the flow setting to the actual flow. If sufficient flow to the steam generator is already provided by the turbine-driven AFW pump, the flow control valves will modulate closed. If desired, operators can also take remote manual control of the valves. In remote manual mode, the control system compares actual position (using a feedback potentiometer) to demanded position to position the valve.

The AFW flow control valves have a safety function to be open and modulate flow to provide adequate secondary heat removal. The valves also have a safety function to modulate in the closed direction to prevent AFW pump runout.

Details

On August 11, 2015, Callaway Plant experienced a main generator lock out, turbine trip, and subsequent reactor trip. All control rods fully inserted and decay heat was being removed through the steam dumps to the main condenser, with the steam generators fed via the turbine-driven AFW pump.

Failure of Valve ALHV0007: Approximately 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> following the trip, while attempting to transfer to the motor-driven AFW pumps, flow control valve ALHV0007, would not respond to open demand signals from the control room. Auxiliary feedwater train B was, therefore, declared inoperable.

The licensee's troubleshooting identified a failed Modutronics card in the actuator for valve ALHV0007. Failure analysis revealed a damaged rectifier bridge on the card. The licensee determined that the rectifier bridge might not have been sufficiently rated to accommodate all electrical loads. The Modutronics card had been modified from its original design in October 2014 by the vendor during refurbishment. The vendor, due to an inability to obtain replacement circuit cards, had reverse-engineered the existing card and redesigned the bridge rectifier. The newly provided bridge rectifier did not have the same current-carrying capacity as the original bridge rectifier.

The original rectifier was designed to carry currents up to 3.0 amps at 25°C, well in excess of the normal running current of 1.2 amps. The replacement rectifier was only capable of passing 1.5 amps at 25°C. The vendor failed to recognize that the normal operating temperature of the valve would further reduce the capacity of the rectifier to approximately 1.0 amps, setting the circuit card up for failures at normal operating temperature.

The licensee became aware of the design deficiency in December 2014. However, the licensee believed that the Modutronics cards with the redesigned rectifier bridges could remain in service until the next refueling outage in April 2016.

The licensee experienced the following failures of the Modutronics cards:

(a) Valve ALHV0005 - modified card placed in service November 15, 2014, subsequently failed on December 3, 2014. Replaced card with another card with the same modification. Replaced with new Modutronics card with higher rated rectifier bridge on August 11, 2015.

(b) Valve ALHV0007 - modified card placed in service October 23, 2014, and failed on August 11, 2015. Replaced with new Modutronics card with higher rated rectifier bridge.

The issue of concern was the inadequate modification of the Modutronics cards, which left them susceptible to short term failure, thus, rendering two AFW system flow control valves and the associated train of motor-driven AFW inoperable for an extended period.

The approximate exposure time for failure of the two valves is from November 15, 2014, to August 11, 2015, for valve ALHV0005, and from October 23, 2014, to August 11, 2015, for valve ALHV0007.

Failure of Valve ALHV0011: An additional failure, unrelated to the Modutronics cards, occurred with valve ALHV0011. This valve had an incorrect potentiometer setting for the positioner, which rendered the valve incapable of opening. This valve had maintenance performed in October 2014 and failed to respond while in service on July 23, 2015. The licensee declared valve ALHV0011 and the motor-driven AFW pump A inoperable at that time.

The exposure time for these two conditions, which together affect both motor-driven AFW pump trains, is from November 2014 to July 2015. The turbine-driven AFW pump was operable and maintained the safety function of the AFW system during this exposure time, except for planned maintenance and testing.

Management Directive 8.3, NRC Incident Investigation Program, was used to evaluate the level of NRC response for these events. In evaluating the deterministic criteria of Management Directive 8.3, it was determined that the event included multiple failures of the motor-driven AFW pump flow control valves which degraded the secondary heat removal mitigating system. The preliminary estimated conditional core damage probability was determined to be 2.4 E-5.

Based on the deterministic criteria and risk insights related to the multiple failures of the motor-driven AFW pump flow control valves, and the issues related to the licensees testing, maintenance, corrective action, design change, and operability programs, Region IV determined that the appropriate level of NRC response was to conduct a special inspection.

As directed by Charter Item B.1(a), the team determined that NRC Inspection Procedure 93812, Special Inspection Procedure, was appropriate for completing the inspection. The inspection included field walkdowns of equipment, interviews with station personnel, and reviews of procedures, corrective action documents, and design documentation. Activities required by Charter Items B.1(i) and B.1(j) were completed during the review of each inspection item described below. A list of documents reviewed is provided in Attachment 1 of this report; the Special Inspection Charter is included as

.2.1 Charter Item B.1(b):

Develop a complete sequence of events up to and including the failures of AFW system flow control valves ALHV0005, ALHV0007, and ALHV0011.

Inspection Scope The team developed and evaluated a timeline of the events leading up to, during, and after the AFW system flow control valve failures. The team developed the timeline, in part, through a review of Callaway action requests, work orders, self-assessments, station logs, vendor communications, and interviews with station personnel. The team created the following timeline during their review of the events leading to the failure of the AFW system flow control valves:

Date/Time Activity February 8, 2012 Licensee requested that Nuclear Logistics Incorporated (the vendor) reverse engineer the existing Modutronics controller cards, to provide for replacements and spare parts for the AFW system flow control valve actuators.

October 12, 2014 Licensee issued Request for Resolution 201100565 that approved use of the Revision 0 reverse-engineered replacement Modutronics cards from the vendor.

October 23, 2014 Modutronics card for valve ALHV0007 replaced with the reverse-engineered Revision 0 card. Valve stroke time tested satisfactorily.

October 24, 2014 Valve gasket and motor-operated valve actuator test system testing performed for valve ALHV0011. Work order did not include instructions for proper set up of the positioner feedback potentiometer.

October 28, 2014 Post-maintenance test of ALHV0011 performed. Test was a full stroke open and closed.

November 3, 2014 Modutronics card for valve ALHV0005 replaced with the reverse-engineered Revision 0 card. Valve stroke time tested satisfactorily.

November 15, 2014 Valve ALHV0005 failed to stroke closed during steam generator fill because of an improper torque switch setting.

December 3, 2014 Reactor trip because of fault on main generator exciter.

All three AFW pumps started to feed the steam generators. Valve ALHV0005 failed to modulate from full open position. Operators manually closed the valve and declared it inoperable. Licensee initiated Callaway Action Request 2014008899 to place this item into the corrective action program. The licensee found the field current rectifier bridge failed and burned.

December 8, 2014 Modutronics card for valve ALHV0005 replaced with the reverse-engineered Revision 0 card. Valve stroke time tested satisfactorily.

December 9, 2014 Callaway Action Request 2014008899 was downgraded from Significance Level 1 to Significance Level 3 based on the unverified conclusion that the failure was due to manufacturing defect, which caused early failure (infant mortality).

December 30, 2014 Licensee shipped Modutronics card from valve ALHV0005 that failed on December 3, 2014, to the vendor for failure analysis.

May 21, 2015 Vendor informed the licensee by letter that the failure of valve ALHV0005 was due to the rectifier bridge being undersized for the required field current. Though the rectifier bridge was ideally designed for 1.5 amps, increased temperatures, aging, current/voltage, and manufacturing variations rendered the rectifier bridges marginally qualified to meet the design requirements of approximately 1.0 amperes and susceptible to failure.

Vendor also informed the licensee that they had designed Revision 1 cards that had field current rectifier bridges with a higher current rating (4.0 amps versus 1.5 amps) to replace the existing Modutronics cards.

No action was taken by the licensee in response to this letter.

July 23, 2015

1:15 a.m.

The licensee initiated a forced shutdown because of unidentified reactor coolant leakage greater than the technical specification allowed 1 gpm.

1:46 p.m.

Main feedwater pump B was stopped because of unusual noise. Operators manually initiated the AFW system to control steam generator levels.

2:00 p.m.

When operators attempted to take remote manual control of AFW flow, Valve ALHV0011 failed to open from the main control board. Equipment operators used local manual control and moved the valve off its seat, allowing operators to control AFW flow. Operators inappropriately attributed the valve failure to hydraulic lock.

July 24, 2015 Licensee troubleshooting revealed that failure of valve ALHV0011 was due to an out-of-tolerance set up of the positioner feedback potentiometer. Maintenance personnel implemented the work order instructions for proper set up of the positioner feedback potentiometer.

Valve was stroke time tested satisfactorily.

August 11, 2015

1:39 a.m.

Turbine trip/reactor trip because of unrecognized electrical jumper in the main transformer. All AFW pumps started, the turbine-driven AFW pump provided adequate feed flow to all four-steam generators. In automatic response, all four AFW flow control valves serving the two-motor-driven pumps closed.

4:14 a.m.

Operators elected to secure the turbine-driven AFW pump and feed the steam generators with the motor-driven pumps. Valve ALHV0007 failed to open and was declared inoperable.

2:00 p.m.(approx.)

The licensee removed the Modutronics card from valve ALHV0007. The licensee found the rectifier bridge on the card failed and burned, similar to the failure of ALHV0005. The licensee replaced the failed card with a hybrid card (a Revision 0 Modutronics card with a rectifier bridge rated for 3.0 amps). Valve was stroke time tested satisfactorily.

August 12, 2015 Because the Modutronics card in valve ALHV0005 was susceptible to the same failure as valve ALHV0007, the licensee replaced the Revision 0 card with a hybrid card.

August 12, 2015 During an extent of condition review from the July 23 failure of ALHV0011, the licensee determined that valve ALHV0005 had an out-of-tolerance set up of the positioner feedback potentiometer. Maintenance personnel implemented the work order instructions for proper set up of the positioner feedback potentiometer.

Valve was stroke time tested satisfactorily.

September 23, 2015 Based on questions from the team, the licensee reviewed the complete design of the replacement Modutronic cards. The licensee identified additional undersized components. Operators performed an operability determination and determined that the Modutronics cards in valves ALHV0005 and ALHV0007 were degraded but operable.

.2.2 Charter Item B.1(c):

Review the manner in which configuration of smart valve controllers for valves ALHV0005 and AHLV0007 was changed (including design control and application of 10 CFR 50.59).

a. Inspection Scope

The team reviewed Purchase Order 569677 and other design documentation associated with the licensees procurement of eight printed circuit board controllers from the vendor.

These controllers were purchased from the vendor as a replacement for the original Modutronics controllers that were installed in the limit switch compartments of the AFW flow control valves. The controllers receive 120 VAC and DC control signals from a source external to the valve and then process the signal and apply power to both the field and armature windings of the flow control valve motors sufficient to stroke the valves and modulate AFW flow. Each controller consists primarily of one printed circuit card, associated discrete electronic components, and interfacing connections and wiring. The vendor-supplied controllers were custom designed for the licensee and were reverse-engineered from a licensee supplied schematic diagram and parts list from the original Modutronics controllers.

The NRC team also reviewed the corresponding vendor test plan and testing documentation, as well as other documentation associated with the vendors certification that the supplied controllers were qualified to IEEE 323-1974 and IEEE 344-1975.

b. Observations and Findings

The team identified that the licensee's purchase order to the vendor required certification to IEEE 323-1974 and IEEE 344-1975. The valves and controllers are located in an area with a maximum ambient temperature of 104°F and radiation less than 1000 rads; thus, they are classified as being in a mild environment. The vendor's qualification program consisted of thermal aging, seismic testing, burn-in, and functional testing.

No concerns were identified by the inspection team with the thermal aging or seismic portions of the testing program.

Failure to Verify the Suitability of the Design of the Reverse-Engineered Replacement Controller Cards for the Auxiliary Feedwater Flow Control Valves

Introduction.

The team identified a Green non-cited violation (NCV) of 10 CFR Part 50, Appendix B, Criterion III, Design Control, for the licensees failure to assure that the design of the replacement reverse-engineered Modutronics controller cards for the AFW control valves were suitable for their application, for failing to establish suitable interface requirements in procurement documents to Nuclear Logistics Incorporated (the vendor),and for failing to verify the adequacy of the design by either design reviews or testing.

Description.

The team identified that neither the licensee nor the vendor had performed a design review sufficient to assure that the reverse-engineered controllers were suitable for their application. In addition, the licensee had not provided the vendor with sufficient information to reverse-engineer the flow controller cards. Specifically, the licensee had not provided the vendor information regarding all input and output interfacing requirements for the new controllers. Most importantly, the licensee had not identified the motor field current as a critical characteristic in their Critical Characteristics Evaluation, which they had performed for the replacement controllers, nor had they provided the vendor with information regarding the motor field current requirements for the motors that were being supplied by these controllers. In addition, the licensee had not provided specific information regarding the environment where the controllers would be required to operate. The team identified that the licensee previously identified that the room environment where these controllers are installed can reach a maximum of 104°F, but that additional localized heating inside the limit switch compartment could occur which could further increase the temperature surrounding the electronic controller circuit cards. Without such information, the vendor was unable to verify that the design of the new cards was acceptable.

While the vendor attempted to mimic the design of the old Modutronics cards as closely as possible, in some instances the discrete electronic components installed on the Modutronics cards were no longer available or for other unstated reasons alternate parts were chosen. In one example, the controller circuit that supplied power to the motor field was modified from a circuit that used four individual diodes rated as being capable of supplying 3.0 amps to a circuit that used a single integrated bridge rectifier device that could only supply 1.5 amps of current. Furthermore, these values did not take into account any de-rating factors that would need to be applied due to temperature conditions inside the valve. Information obtained by the licensee concluded that the motor field could require up to 1.7 amps of current. As such, the newly designed circuit was not suitable for its application.

The team also identified that the licensees purchase order required the vendor to provide their test and qualification plans for review and approval. The vendor submitted their proposed qualification and test plan, Qualification Report for AMEREN - Callaway Plant for Electronic Positioner NLI-10A-LM-P to AMEREN, and the licensee subsequently approved this plan but failed to identify that the plan lacked testing of the portion of the circuit cards that supplied current to the motor field windings.

The vendor-testing program did not include the connection of the circuit cards to simulated loads that would mimic the as-installed application during either burn-in or functional testing. Specifically, the cards were not tested with an actual motor and while the armature circuits on the card were loaded utilizing a fixed resistor, the field winding circuits on the card were never loaded. Consequently, the suitability of the design was not fully proven through testing by the vendor. Furthermore, the licensees receipt inspection and post-maintenance testing procedures did not compensate for the inadequacies in the vendors testing, in that they also did not test the field circuits under design basis loading conditions. As such, the qualification and production testing that was ultimately performed by the vendor, and approved by the licensee, was insufficient and failed to identify that the rectifier design was undersized for its application. The inadequate design of the new cards, in conjunction with the inadequate testing protocol that was developed to verify their design and construction, led to repeated failures of the AFW flow control valves as described elsewhere in this report.

This improper design of the controller cards was originally documented in Callaway Action Request 201503968 by the licensee on June 2, 2015. At that time, the design of the cards was thought to be marginal but acceptable. The licensee placed this item into the corrective action program as Callaway Action Request 201505796 to perform a root cause analysis of the circuit card failures, which was completed on September 21, 2015.

Analysis.

The failure to ensure that the design of the replacement for the Modutronics cards was suitable for their application was a performance deficiency. This performance deficiency is more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Design Control and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, design deficiencies associated with these circuit cards resulted in the inoperability of AFW control valves and their ability to operate on demand. The team performed an initial screening of the finding in accordance with Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-Power. Using Inspection Manual Chapter 0609, Appendix A, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the team determined that the finding required a detail risk evaluation because it represented the potential loss of one train of safety-related equipment (AFW) for greater than the technical specification allowed outage time. A Region IV senior reactor analyst performed a detailed risk evaluation in accordance with Appendix A, Section 6.0, Detailed Risk Evaluation, which determined that the finding was of very low safety significance (Green). The analyst determined that the importance of the failure of valves ALHV0005 and ALHV0007 was based on the postulated failure time of the turbine-driven auxiliary feedwater pump because this determined the position in which the valves failed. The internal events incremental conditional core damage probability was 8.17 x 10-7. The analyst also determined that the finding had only a minimal effect on external initiator risk and that the finding would not involve a significant increase in the risk of a large, early release of radiation. The detailed risk assessment for this violation is included in this inspection report as Attachment 3.

This finding has a human performance cross-cutting aspect in the area of teamwork, because individuals in different work groups did not appropriately communicate across organizational boundaries. Specifically, licensee personnel did not adequately communicate the design and testing requirements for the reverse engineered cards (H.4).

Enforcement.

Title 10 CFR Part 50, Appendix B, Criterion III, Design Control, states, in part, that for those structures, systems and components (SSCs) to which this appendix applies, measures shall be established for the selection and review for suitability of application of materials, parts, equipment, and processes that are essential to the safety-related functions of the SSCs. Contrary to the above, from October 23, 2014, through August 11, 2015, for quality-related components associated with auxiliary feedwater system flow control valves to which 10 CFR Part 50, Appendix B applies, the licensee failed to select and review for suitability of application of materials, parts, equipment and processes that are essential to the safety-related function of the component. Specifically, the licensee failed to ensure that that the design of the replacement reverse-engineered cards for the AFW flow control valves were suitable for their application, failed to establish suitable interface requirements in procurement documents to the vendor, and failed to verify the adequacy of the design by either design reviews or testing, in that the replacement modutronics cards were supplied with undersized field current rectifier bridges that failed in service. This violation is being treated as an NCV, consistent with Section 2.3.2.a of the Enforcement Policy. The violation was entered into the licensees corrective action program as Callaway Action Request 201505796. (NCV 05000483/20150009-01, Failure to Verify the Suitability of the Design of the Reverse-Engineered Replacement Controller Cards for the Auxiliary Feedwater Flow Control Valves.)

.2.3 Charter Item B.1(d):

Inspect maintenance practices which led to failure of valve ALHV011, including post-maintenance tests.

a. Inspection Scope

The team reviewed the maintenance and testing practices associated with valve ALHV0011. This review includes work orders, surveillance tests, post-maintenance tests, and Callaway action requests up to and including the valves failure to respond from the control room on July 23, 2015.

.1 Failure to Have an Adequate Procedure for Calibration of the Auxiliary Feedwater Pump

Flow Control Valve Potentiometer

Introduction.

The team reviewed a self-revealing Green non-cited violation (NCV) of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, for failure to prescribe activities affecting quality by procedures appropriate to the circumstances. Specifically, on November 18, 2009, the licensee revised Procedure MTE-ZZ-QA033, MOVATS UDS [motor operated valve actuator test system universal diagnostic system] Testing of Torque Controlled Modutronics Limitorque Motor Operated Rising Stem Valves, Revision 3, and introduced an error in bypassing a test of the Modutronics board setup feedback potentiometer. As a result, on July 23, 2015, the actuator misinterpreted the actual position of the valve, which subsequently failed to open when operators attempted to open the valve following a forced reactor shutdown.

In response to this issue, the licensee has reviewed all maintenance and test activities that could affect the potentiometer and has revised the appropriate procedures.

Description.

On July, 23, 2015, the licensee was performing a controlled plant shutdown to Mode 5 to start Forced Outage 68 in response to an unidentified reactor coolant system leak. Main feedwater pump B was manually secured due to concerns with the pumps control and mechanical performance. In an effort to provide feedwater flow to the steam generators, operations entered Procedure OTN-AL-00001, Auxiliary Feedwater System, Revision 33, to manually start the motor-driven AFW pumps. With the motor-driven AFW pumps in service, operators attempted to establish approximately 50 percent demand to all steam generators through all four discharge flow control valves. Flow control valves ALHV0005, ALHV0007, and ALHV0009 opened and responded as expected, but flow control valve ALHV0011 remained closed based on indicator lights, and the lack of flow indicated to steam generator C. The operators then reduced the demand signal on flow control valve ALHV0011 to zero percent demand and an operations technician was dispatched to the valve location. The operations technician was instructed to manually open flow control valve ALHV0011 using the handwheel. When the operator opened the valve, the motor energized and drove the valve closed. The operating crew concluded that this was due to the zero percent demand signal and used main control board switch ALHK0011A to increase the demand signal to approximately 25 percent. The operations technician used the handwheel to open the valve, and then the valve started to respond as designed.

The initial cause for the failure of flow control valve ALHV0011 to open was believed be hydraulic locking of the valve. Based on the presumption of hydraulic locking, an operations technician manually opened the valve, and flow control valve ALHV0011 functioned as expected. The following day, the licensee investigated the issues leading to the valve not opening on a demand signal from the main control boards, and found the feedback potentiometer was off scale after the valve was closed electrically from the main control board. The potentiometer had been left after maintenance in a condition with the feedback potentiometer rotated past the zero point such that an open circuit condition existed. In this condition, the control circuit received a signal that the valve was 50 percent open when the valve was closed. In this condition, the valve would not respond to a remote manual open signal of less than 50 percent. The off-scale condition was corrected by adjusting the potentiometer coupling to the actuator gear set using Procedure MTE-ZZ-QA033, MOVATS UDS Testing of Torque Controlled Modutronics Limitorque Motor Operated Rising Stem Valves, Attachment 12.

The AFW motor-operated flow control valves use a Modutronics circuit board to provide the required electronic feedback to control the valve position. This circuitry also includes a potentiometer, which must be properly set to provide the necessary response for correct valve positioning. The licensee used Procedure MTE-ZZ-QA033, MOVATS UDS Testing of Torque Controlled Modutronics Limitorque Motor Operated Rising Stem Valves, Revision 2, to perform a diagnostic check of the valve. The procedure identifies the use of the Crane MOVATS Universal Diagnostic System Torque Thrust Cell (TTC)for checking the torque and thrust measurements of the valves. Step 6.3.10 of the procedure states, If the TTC was installed, perform Attachment 12, Modutronics Board Setup, to rescale the Modutronics Positioner due to the physical offset caused by the TTC. The use of Attachment 12 of the procedure verified that the feedback potentiometer was correctly set and/or adjusted.

In November 2009, the licensee added a second method of checking the torque and thrust of the valves into their procedures. This second method, the easy thrust torque testing method, was less invasive to the valve setup to complete the testing. Revision 3 of Procedure MTE-ZZ-QA033 was approved on November 18, 2009. Step 6.9.1 of Procedure MTE-ZZ-QA033, Revision 3, states, If the TTC was installed, perform the following steps. IF NOT, Proceed to Step 6:10. Step 6.9.5 states, After removal of TTC, perform Attachment 12. Per the procedure, if the licensee used the easy thrust torque method during the testing, they would proceed to Step 6.10, bypassing any requirement to use Attachment 12 to verify that the feedback potentiometer was adjusted or positioned correctly so that the AFW valves would function properly.

On October 24, 2014, the licensee performed maintenance work orders PM0825822 and associated Job 08512683.500 on valve ALHV0011. These work orders implemented routine valve gasket replacement as well as diagnostic testing of the valve, which required disassembly/reassembly of the actuator. The feedback potentiometer was removed and reinstalled to support this maintenance.

However, because Attachment 12 to Procedure MTE-ZZ-QA033 was no longer referenced in the body of Procedure MTE-ZZ-QA033, the set-up and calibration of the feedback potentiometer was not directed to be performed upon actuator disassembly/reassembly. Thus, Procedure MTE-ZZ-QA033 was not appropriate to the circumstances. Failure to properly set the feedback potentiometer following actuator disassembly/assembly resulted in a zero feedback signal from the potentiometer on valve ALHV0011, which indicated that the flow control valve was 50 percent open, when the valve was fully closed. Any remote manual demand signal of less than 50 percent open would not open the valve because the controller believed that the valve was already 50 percent open. This condition would not have affected automatic operation of the valve. The valve would have automatically throttled flow auxiliary feedwater flow to approximately 300 gpm on demand.

As part of the extent of condition review, on August 12, 2015, the licensee inspected the feedback potentiometers of flow control valves ALHV0005 and ALHV0007. During the inspection, flow control valve ALHV005 feedback potentiometer was found off scale.

The licensee concluded that the likely cause for the flow control valve ALHV0005 feedback potentiometer being off scale was the torque switch replacement, the replacement of the new Nuclear Logistics Incorporated electronic positioner boards, or the combination of the torque switch and positioner boards. The feedback potentiometer for valve ALHV0007 was in its proper position.

Therefore, on July 24, 2015, and again on August 12, 2015, the licensee identified that at least one of the four auxiliary feedwater system flow control valves had feedback positioners off-scale, because of the failure to set-up and calibrate the feedback potentiometer following valve testing or maintenance. The licensee determined that Procedure MTE-ZZ-QA033 had been revised in 2009. The licensee identified prior opportunities to correct the condition in 2010, when the valve position feedback potentiometers were set up and calibrated without procedure guidance. However, the procedure error was not corrected. The licensee initiated Callaway Action Request 201405332 to evaluate the condition.

Analysis.

The failure to provide a procedure appropriate to the circumstances for an auxiliary feedwater system flow control valve was a performance deficiency. This finding was more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Procedure Quality and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to provide a procedure appropriate to the circumstances to set an auxiliary feedwater system flow control valve feedback potentiometer resulted in its inability to operate manually on demand. Using Inspection Manual Chapter 0609, Appendix A, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the finding was determined to be of very low safety significance, because it did not affect system design, did not result in a loss of system function, did not represent a loss of function of a single train for greater than its technical specifications allowed outage time, and did not cause the loss of function of one or more non-technical specification trains of equipment designated as high safety-significance. The valve would have automatically throttled auxiliary feedwater flow to approximately 300 gpm on demand. This finding did not have a cross-cutting aspect because the procedure revision resulting in the inadequate procedure was issued in 2009, and previous opportunities to correct the procedure occurred in 2010. Thus, this performance deficiency was not indicative of current licensee performance.

Enforcement.

Title 10 CFR Part 50, Appendix B, Criterion V, Instruction, Procedures, &

Drawings, states, in part, that activities affecting quality shall be prescribed by procedures of a type appropriate to the circumstances. Contrary to the above, from November 18, 2009 to July 23, 2015, the procedure for motor-operated valve testing, an activity affecting quality, was not appropriate to the circumstances. Specifically, the licensee revised Procedure MTE-ZZ-QA033, Revision 2, to incorporate a second method of motor-operated valve testing, and introduced an error in bypassing a test of the Modutronics board setup feedback potentiometer, which checked and adjusted this function of the motor-driven AFW flow control valves. The procedure failed to check the feedback potentiometer, which was found off scale. As a result, on July 23, 2015, the actuator misinterpreted the actual position of the valve, which failed to open when operators attempted to open the valve during a reactor shut down. In response to this issue, the licensee has reviewed all maintenance and test activities that could affect the potentiometer and will be revising their procedures. This violation is being treated as an NCV, consistent with Section 2.3.2.a of the Enforcement Policy. The violation was entered into the licensees corrective action program as Callaway Action Request 201505332. (NCV 05000483/2015009-02, Failure to Have an Adequate Procedure for Calibration of the Auxiliary Feedwater Pump Flow Control Valve Potentiometer.)

.2 Failure to Have an Adequate Post-Maintenance Test for Setting the Motor-Driven

Auxiliary Feedwater Flow Control Valve Modutronics Potentiometer

Introduction.

The team reviewed a self-revealing Green non-cited violation (NCV) of 10 CFR Part 50, Appendix B, Criterion XI, Test Control, for the licensees failure to have an adequate testing program for the motor-operated AFW flow control valves.

Specifically, maintenance and testing procedures did not verify proper operation of the valves position feedback potentiometer. As a result, on July 23, 2015, during a controlled plant shut down to Mode 5 in response to an unidentified reactor coolant system leak, when motor-driven AFW pump A to steam generator C flow control valve ALHV0011 was given a signal to throttle open from the main control board hand switch, the valve failed to respond. This subsequently resulted in the inability of the operator to control level in steam generator C from the control room and an operator was dispatched to take local manual action to open the valve.

Description.

As discussed in section 4OA52.3.1, on July 23, 2015, when attempting to manually initiate the motor-driven trains of the auxiliary feedwater system, valve ALHV0011 did not respond to a remote manual open signal from the control room.

Operators locally cracked the valve open from its fully closed position, and then valve ALHV0011 was able to respond to a manual signal. The licensee determined that maintenance technicians failed to correctly set up and test the valves feedback potentiometer. The potentiometer had been left after maintenance in a condition such that feedback potentiometer was rotated past the zero point such that an open circuit condition existed, which is interpreted by the controller circuitry as the valve being 50 percent open. Thus, when given a remote manual signal to a position somewhat less than 50 percent, the valve did not move. The procedure used for performing maintenance on the actuator for valve ALHV0011 (Procedure MTE-ZZ-QA033)bypassed the attachment requiring set up and verification of the feedback potentiometer, as discussed above.

On October 24, 2014, the licensee performed maintenance work orders PM0825822 and associated Job 08512683.500 on valve ALHV0011. These work orders implemented routine valve gasket replacement as well as diagnostic testing of the valve, which required disassembly/reassembly of the actuator. The feedback potentiometer was removed and reinstalled to support this maintenance. The licensee work planners referenced Licensee Procedure APA-ZZ-0032, Appendix E, Post-Maintenance Test Program, Revision 010 to develop the post-maintenance test requirements following the maintenance on ALHV0011. Procedure APA-ZZ0032 further directed the planners to use the Valve Retest Manual, Revision 051, which was a licensee guidance document to aid the planners in designating post-maintenance tests for valves. Based on the available guidance, the planners designated a full-stroke open and close test for the maintenance on valve ALHV0011. The full stroke test was completed satisfactorily on October 28, 2014, and the valve was declared operable.

The Valve Retest Manual did not have additional instructions that distinguished the auxiliary feedwater smart valves from other standard motor operated valves. Therefore, the licensee did not consider uncalibrated feedback potentiometer as a potential failure mechanism. This condition had existed since inception of the manual.

During a full stroke test of the auxiliary feedwater pump flow control valves, the valve controllers compare demanded signal to actual valve position. If the feedback potentiometer is not set properly (i.e. open circuited with a 0 volt output) the controller receives a signal indicated that the valve is 50 percent open. During a full stroke, the valve would given a 100 percent open demand with a feedback signal of 50 percent open. Because the demand position is greater than the position feedback, the valve will open even with the feedback potentiometer improperly set, masking the degraded condition.

In order to ensure that the feedback potentiometer was set properly, in addition to the full stroke test the valve would have had to have been stepped on in smaller increments less than 50 percent to mimic auxiliary feedwater flow demand during periods of lower decay head removal, such as on July 23, 2014. Therefore, because the full stroke test was unable to reveal the degraded condition of the feedback potentiometer, the post-maintenance testing of valve ALHV0011 was inadequate to verify the work was performed properly. The team determined that the failure to establish a post-maintenance test that would verify the auxiliary feedwater system flow control valves would perform satisfactorily under all conditions while in service was a violation of 10 CFR 50, Appendix B, Criterion XI, Test Control.

The licensee initiated Callaway Action Request 201405332 to evaluate the concern with the flow control valve feedback potentiometers post-maintenance testing. In response to this issue, the licensee performed another calibration of the potentiometer, focusing on the potentiometer position during the valve stroke. During the investigation, the licensee discovered more work activities that could potentially impact the feedback potentiometer calibration, and added them to the list of work documents identified in Callaway Action Request 201405332 requiring revision. The additions include work that replaces or adjusts the torque switches, adjusts the limit switches, or adjusts the board potentiometers (including a board replacement). The licensee also initiated actions to revise the Valve Retest Manual for more specific post-maintenance test requirements of the auxiliary feedwater flow control valves, to include instructions to step open the valves at small increments during testing to verify proper operation of the valve position feedback potentiometers.

Analysis.

The failure to establish a suitable post-maintenance test program to demonstrate that the motor-driven AFW flow control valve Modutronics potentiometer would be set correctly after maintenance or testing was a performance deficiency. This finding was more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to establish a post-maintenance testing program for the motor-driven AFW valve Modutronics potentiometer to verify that the potentiometer was set correctly, resulted in valve ALHV0011 failing to open when operators initiated a signal to place the valve in an open position. In accordance with Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings at Power, dated June 19, 2012, Exhibit 2, Mitigating Systems Screening Questions, the issue screened as having very low safety significance (Green) because it was a design or qualification deficiency that did not represent a loss of operability or functionality; did not represent an actual loss of safety function of the system or train; did not result in the loss of one or more trains of non-technical specification equipment; and did not screen as potentially risk significant due to seismic, flooding, or severe weather. The team determined that this finding did not have a cross-cutting aspect because the most significant contributor did not reflect current licensee performance.

Enforcement.

Title 10 CFR Part 50, Appendix B, Criterion XI, Test Control, states, in part, that a test program shall be established to assure that all testing required to demonstrate that structures, systems, and components will perform satisfactorily in service is identified and performed in accordance with written test procedures which incorporate the requirements and acceptance limits contained in applicable design documents. Contrary to the above, on October 28, 2014, the licensee failed to assure that all testing required to demonstrate that structures, systems, and components would perform satisfactorily in service is identified and performed in accordance with written test procedures. Specifically, the licensee failed to establish a suitable post-maintenance test program to demonstrate that the motor-driven AFW flow control valve Modutronics potentiometer, a quality related component, had been set correctly after maintenance or testing. The testing consisted of stroking the valve full open or full closed, and did not consider step changes in valve positioning, and did not confirm the potentiometer feedback settings during valve positions that were not full open or full closed. In response to this issue, the licensee performed another calibration of the potentiometer, focusing on the potentiometer position during the valve stroke. The licensee also initiated actions to revise the Valve Retest Manual for more specific post-maintenance test requirements of the auxiliary feedwater flow control valves, to include instructions to step open the valves at small increments during testing to verify proper operation of the valve position feedback potentiometers. This violation is being treated as an NCV consistent with Section 2.3.2.a of the Enforcement Policy. The violation was entered into the licensees corrective action program as Callaway Action Request 201505332. (NCV 05000483/2015009-03, Failure to Have an Adequate Procedure for Setting the Motor-Driven Auxiliary Feedwater Flow Control Valve Modutronics Potentiometer.)

.2.4 Charter Item B.1(e):

Inspect the licensees response to failures of ALHV0005 and ALHV0011 and determine if corrective action and operability programs were implemented correctly.

a. Inspection Scope

The team reviewed the root cause analyses and the licensees program implementation of the corrective action and operability programs.

.1 Two Examples of a Failure to Properly Designate the Significance Level of Callaway

Action Requests

Introduction.

The team identified two examples of a Green non-cited violation (NCV) of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, for the licensees failure to implement their corrective action program procedure.

Specifically:

(1) on November 20, 2014 the licensee designated the improper setting of the AFW flow control valve ALHV0005 limit switches as Significance Level 5 (administrative close) instead of Significance Level 3 (lower tier cause evaluation) and

(2) on December 9, 2014, the licensee downgraded the failure of the Modutronics card for valve ALHV0005 from Significance Level 1 (requiring a root cause analysis and corrective action to prevent recurrence) to Significance Level 3 based on unverified assumptions of the failure mechanism. Following failure of the Modutronics card for valve ALHV0005, the licensee assumed that the early failure was due to a manufacturing defect (infant mortality) without supporting data to prove this designation.

Description.

(Example 1). On November 20, 2014, the licensee documented in Callaway Action Request 201408580 that the limit switches for AFW flow control valve ALHV0005 (that feeds steam generator D) were improperly set. This resulted in the continuous energization of the control valves motor field. This was screened by the licensee as a Significance Level 5 condition (the lowest level in the Callaway action request program, requiring only an administrative closure), thus, an apparent cause evaluation was not performed. The inspection team reviewed the guidance contained in Procedure APA-ZZ-00500, Corrective Action Program, Revision 62, for setting significance levels. Based on the guidance contained in that procedure, the issue was a condition adverse to quality, requiring a lower tier cause evaluation (i.e., an apparent cause evaluation). Callaway Action Request 20148580 should have been screened, at a minimum, as Significance Level 3.

The Significance Level 5 screening was based, at least partially, upon incorrect information documented in Callaway Action Request 201408580 that stated that the valve was initially designed with the motor field continuously energized. The documentation stated that it was not until a recent design change was performed that the motor field circuits were wired to be de-energized during the full open position (their normal position), and as such, this condition was not significant. Upon questioning by the team, the team learned that the statements in the Callaway action request were not correct, and that the valves had never been designed to have their motor fields continuously energized.

The initial corrective action that was taken in response to this issue was to tweak the valve positioner ZERO pot/limit switch contact. An assessment of how/why the switch was improperly set or why post-maintenance testing procedures failed to detect the condition was not originally performed by the licensee.

Additionally, in the root cause analysis that was ultimately performed for the failure of valve ALHV0005, as part of Callaway Action Request 201505796, the team noted that the licensee failed to identify this issue as a contributing cause to the failure of valve ALHV0005 to operate on demand on December 3, 2014. While the primary cause of this failure was ultimately attributed to an improper design of the motor field circuit (as described in Section 4OA5.2.7 of this inspection report), the fact that the motor field in this case was continuously energized for a significant period of time likely contributed to its failure. Consequently, as a result of the Significance Level 5 screening, no additional corrective actions were taken for the conditions documented in Callaway Action Request 201408580 to ensure that the maintenance and installation procedures utilized to set up these valves were adequate. This issue was entered into the licensees corrective action program as Callaway Action Request 201506921, on September 23, 2015.

(Example 2). On December 3, 2014, a turbine trip/reactor trip occurred because of a jumper improperly installed in the main transformer. In response to the trip, the AFW system automatically started and was fed by the turbine-driven AFW pump. However, valve ALHV0005, which is the flow control valve for steam generator D, failed to respond to an automatic signal and remained fully open. Operators unsuccessfully attempted to manually close the valve from the control room. An equipment operator was dispatched and closed the valve locally using the manual handwheel to prevent an overfill of steam generator D. The licensee initiated Callaway Action Request 2014008899 to enter this item into the corrective action program. The licensee removed the Modutronics card from the valve and found it failed and the motor field current rectifier bridge was burned.

On December 4, 2014, the Callaway action request screening committee (that designates the significance level of Callaway action requests) initially considered Callaway Action Request 2014008899 as a Significance Level 1, because it represented a significant operating abnormality, deviation from expected performance of plant equipment and of unanticipated deficiencies in the design or operation of structures, systems, or components which affect nuclear safety, as defined by Procedure APA-ZZ-00500, Corrective Action Program, Revision 62, Appendix 17. This would have resulted in a root cause analysis and corrective action to prevent recurrence.

Since a similar Revision 0 Modutronics card was installed on valve ALHV0007, the potential failure mechanism existed on valve ALHV0007 and needed to be addressed.

However, on December 9, 2014, engineering personnel returned Callaway Action Request 201408899 to the Callaway action request screening committee and requested that it be downgraded to Significance Level 3, Lower Tier Cause Evaluation. This request was based on the belief that the early failure of the Modutronics card in valve ALHV0005 was caused by a manufacturing defect (infant mortality). This assumption was not backed by data or operational/maintenance history of the cards, given that these cards were only supplied to the Callaway Plant, and the cards had a total of one month of installed service. Despite this lack of evidence of the assumed failure because of infant mortality, the screening committee accepted the recommendation to downgrade Callaway Action Request 201408899 to Significance Level 3. This downgrade was not in accordance with Procedure APA-ZZ-00500, Corrective Action Program, Revision 62, Appendix 17, and was a missed opportunity for the licensee to identify the root cause, corrective actions, and extent of condition of the failure of valve ALHV0005, prior to the subsequent failure of valve ALHV0007 on August 11, 2015. This issue was entered into the corrective action program as Callaway Action Request 201507235.

Analysis.

The two failures to properly designate the Significance Level of Callaway action requests, constitute a performance deficiency. This finding was more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failures properly to designate the significance of the conditions precluded determining the appropriate cause determinations and extent of conditions and resulted in failure to correct the conditions before they further manifested themselves following a reactor trip. In accordance with Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings at Power, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the issue screened as having very low safety significance (Green),because it did not affect system design, did not result in a loss of system function, did not represent a loss of function of a single train for greater than its technical specifications allowed outage time, and did not cause the loss of function of one or more non-technical specification trains of equipment designated as high safety-significance.

This finding has a human performance cross-cutting aspect in the area of conservative bias in that the decision making did not demonstrate a conservative/prudent choice in designating the significance level of the Callaway action requests based on two cases of unverified/incorrect information (H.14).

Enforcement.

Title 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, states, in part, that activities affecting quality shall be accomplished in accordance with procedures appropriate to the circumstances. Licensee procedure APA-ZZ-00500, Corrective Action Program, Revision 62, an Appendix B quality related procedure, provides instructions for implementing the corrective action program. Licensee procedure APA-ZZ-00500 states in Appendix 17 that

(1) the following shall be designated as Significance Level 3: A Significance Level 3 Adverse Condition is an event, defect, state or activity that prohibits or detracts from safe nuclear plant operation, where it is desired to perform a cause analysis, and

(2) the following shall be designated as Significance Level 1: A Significant Condition Adverse to Quality (SCAQ) is an event that prohibits or detracts from safe nuclear plant operation AND has caused a significant impact on unit capacity/reliability concern which impedes the effectiveness of the Operating Quality Assurance Program. Contrary to the above,

(1) on November 20, 2014, the licensee did not designate an event, defect, state or activity that prohibits or detracts from safe nuclear plant operation, where it is desired to perform a cause analysis as Significance Level 3, and

(2) on December 9, 2014, the licensee did not designate an event that prohibits or detracts from safe nuclear plant operation AND has caused a significant impact on unit capacity/reliability concern which impedes the effectiveness of the Operating Quality Assurance Program as Significance Level 1. Specifically,

(1) Callaway Action Request 201408580 described an adverse condition that constituted a state that detracted from safe nuclear operation, but was designated as Significance Level 5, and

(2) Callaway Action Request 201408899 described an event that prohibits or detracts from safe nuclear plant operation AND has caused a significant impact on unit capacity/reliability concern which impedes the effectiveness of the Operating Quality Assurance Program but was designated as Significance Level 3. The lower significance levels resulted in failure to determine the causes and correct the conditions, such that repeat failure of the auxiliary feedwater pump flow control valves occurred. This violation is being treated as an NCV, consistent with Section 2.3.2.a of the Enforcement Policy. The violation was entered into the licensees corrective action program as Callaway Action Request 201507235.

(NCV 05000483/2015009-04, Two Examples of a Failure to Properly Designate the Significance Level of Callaway Action Requests.)

.2 Failure to Determine the Cause and Take Corrective Action to Preclude Repetition for

the Inadequate Design of Auxiliary Feedwater Flow Control Valve Modutronics Cards

Introduction.

The team identified a Green non-cited violation (NCV) of 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, for the licensees failure to determine the cause and take corrective action to preclude repetition for a significant condition adverse to quality. On May 21, 2015, the licensee received new information that refuted the previously assumed failure mechanism for AFW flow control valve ALHV0005 documented in December 2014, but failed to initiate a new Callaway action request to document the new information and report it to appropriate levels of management. As a result, the licensee failed to identify the failure of the valve as a significant condition adverse to quality, determine the cause, initiate a prompt operability assessment, and identify corrective action to preclude repetition until valve ALHV0007 failed, for the same reason, following a reactor trip on August 11, 2015.

Description.

On December 3, 2014, following a reactor trip, AFW flow control valve ALHV0005 failed to move from its full open position when called upon. The licensee closed the valve locally by declutching the actuator and manually closing it, then declared the valve inoperable. The licensee initiated Callaway Action Request 201408899 to enter this item into the corrective action program.

As discussed in Section 4OA5.2.4.2 of this report, this condition was initially screened as Significance Level 1 (requiring a root cause and corrective action to prevent recurrence)then, was subsequently downgraded to Significance Level 3 (lower tier cause evaluation) based on unverified information that the Modutronics card rectifier bridge failed because of a manufacturing defect (infant mortality). On December 30, 2014, the licensee shipped the failed Modutronics card to the vendor for failure analysis.

On May 21, 2015, the vendor informed the licensee (by letter) that the Modutronics card failed because the rectifier bridge was undersized/marginal for the valve motor field current duty. The previous cards had rectifier bridges for the field current that were rated for 3.0 amps. The reverse-engineered cards were rated nominally at low temperatures for 1.5 amps. Taking into account temperature and other environmental conditions, this rating could have been degraded to about 1.0 amp. The vendor determined that the field current for the flow control valve motors could be up to 1.7 amps, thus the rectifier bridges were marginally qualified for service..

This vendor information was previously unevaluated information in the corrective action and operability process. This information demonstrated that a significant condition adverse to quality existed in more than one installed component in a safety related application. The failure to identify this information as a significant condition adverse to quality precluded the licensee from determining the cause and taking actions to preclude repetition, until valve ALHV0007 failed upon demand on August 11, 2015.

Analysis.

The licensees failure to determine the cause and take corrective action to preclude repetition for a significant condition adverse to quality when failure analysis indicated that a significant defect existed on valves ALHV0005 and ALHV0007, was a performance deficiency. This finding was more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure of the licensee to determine the cause and take corrective action to preclude repetition when new information on the failure mechanism was received resulted in the failure to correct the condition before it further manifested itself following a reactor trip. Using Inspection Manual Chapter 0609, Appendix A, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the finding was determined to be of very low safety significance (Green), because it did not affect system design, did not result in a loss of system function did not represent a loss of function of a single train for greater than its technical specifications allowed outage time, and did not cause the loss of function of one or more non-technical specification trains of equipment designated as high safety-significance. This finding has a human performance cross-cutting aspect in the area of consistent process in that the individuals that received the information concerning the failure mechanism of the Modutronics cards failed to use a systematic approach to documenting the information and reporting it to appropriate levels of management (H.13).

Enforcement.

Title 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, states, in part, that, in the case of significant conditions adverse to quality, measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition. Contrary to the above, for quality-related components associated with auxiliary feedwater system flow control valves, a significant condition adverse to quality existed, but the licensee did not assure that the cause of the condition was determined and corrective action taken to preclude repetition. Specifically, on May 21, 2015, the licensee received new information that valve ALHV0005 failed on December 3, 2014 because of an undersized rectifier bridge on the Modutronics card, a significant condition adverse to quality, but failed to initiate a Callaway action request such that the cause of the condition would be determined and corrective action taken to preclude repetition. As a result, the undersized rectifier bridges remained in service in valves ALHV0005 and ALHV0007 until a subsequent failure of the Modutronics card rectifier bridge in valve ALHV0007 on August 11, 2015. This violation is being treated as an NCV, consistent with Section 2.3.2.a of the Enforcement Policy. The violation was entered into the licensees corrective action program as Callaway Action Request 201506846. (NCV 050004832015009-05, Failure to Determine the Cause and Take Corrective Action to Preclude Repetition for the Inadequate Design of Auxiliary Feedwater Flow Control Valve Modutronics Cards)

.3 Failure to have an Adequate Procedure for Setting the Torque and Thrust Values for the

Auxiliary Feedwater Pump Flow Control Valves

Introduction.

The team identified a Green non-cited violation (NCV) of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, for the licensees failure to have an adequate procedure for testing the torque switch setting on AFW valve ALHV0005. Specifically, while performing Job 08505547, the licensee had not correctly accounted for the differential pressure the valve would actually experience at low flow rates and had incorrectly set and tested the close torque switch. As a result, On November 15, 2015, during steam generator filling operations, Valve ALHV0005 failed to move in the closed direction when the torque switch opened. The incorrect close torque switch setting prevented the valve from going fully closed upon demand.

Description.

On March 4, 2014, the train B AFW maintenance outage was started, in which valve ALHV0005 was scheduled for motor-driven operator actuator service and preventative maintenance inspection. Due to a wiring error during reassembly which caused motor to run backwards, the valve was over-thrusted in the closed direction.

New tasks were added to Job 08505547 to address the adverse impact of the error. The valve and actuator were disassembled with internal valve parts being replaced, and the actuator was inspected and reassembled. As part of the work, a motor-operated valve diagnostic test was completed. Because the valve was previously overthrusted, the technicians were directed to adjust the torque switch setting so that the valves closing thrust was left 9,724 pounds, which was significantly less than the other four AFW flow control valves settings, and less than its previous range of 11,029 to 14,188 pounds.

For a post-maintenance test of the torque switch setting, Job 08505547 directed use of Procedure OSP-AL-PV04B, Train B Motor Driven Auxiliary Feedwater Comprehensive Pump and Check Valve Test, Revision 19. This procedure was performed with the plant in Mode 1, with steam generator pressures at approximately 965 psig. Procedure OSP-AL-PV04B verified that the AFW flow control valves actuate to their safety position from full open, to approximately 50 percent open, to throttle AFW flow to each steam generator to approximately 300 gpm. Procedure OSP-AL-PV04B was completed satisfactorily for valve ALHV0005 with flow to Steam Generator D, to demonstrate the valve would properly throttle flow automatically, and not deengergize by the torque switch opening during operation. The differential pressure across valve ALHV0005 was approximately 710 psid during this test. This differential pressure across valve ALHV0005 was much less than could be expected when decay heat removal/steam generator level dictated that the valve be throttled in the closed direction to 10 percent or less, under normal and accident/post trip conditions. The differential pressures across the flow control valves could be over 1500 psid during these low flow conditions. Thus, using Procedure OSP-AL-PB04B was not adequate for verifying that the torque switch setting would ensure that the valve would perform satisfactorily while in service under all postulated conditions.

On November 15, 2014, during Refueling Outage 20, while filling the steam generators using Procedure OTN-AL-00001, Auxiliary Feedwater System, Revision 34, flow was established at approximately 150 klbm/hr to each steam generator using the motor-driven AFW pump B. Flow was then throttled to approximately 90 klbm/hr for steam generator D. Later, the reactor operator attempted to close valve ALHV0005 to stop filling steam generator D while continuing to fill steam generator A. With demand at zero percent to the valve, flow of 90 klbm/hr was still indicated on flow indicator ALFI0001A. The licensee determined that the closed torque switch had opened, deenergizing the valve slightly open. The torque switch was found to have opened prior to the valve fully closing under system differential pressure, causing valve movement to stop. The licensee determined that the cause of intermittent switch operation was the marginal as-left torque switch setting in March 2014. Further review of past motor-operated valve data determined the November 17, 2014, as-found closed thrust value of 9,724 pounds was lower than any of the previous settings, which ranged from 11,029 to 14,188 pounds. After switch replacement and setup, the as-left thrust at torque switch trip was 12,133 pounds. Post-maintenance testing was completed which confirmed that valve ALHV0005 was capable of closing against the system differential pressure for which it had previously failed of 1575 psid.

At the time the valve failed to fully close in Refueling Outage 20, the steam generator pressure was approximately 100 psig, resulting in a differential pressure across valve ALHV0005 of approximately 1,575 psid. As demonstrated during the troubleshooting in Refueling Outage 20, operation of the valve was intermittent; indicating the thrust setting (9,724 pounds) was marginal for the differential pressure across the valve. The incorrect torque switch setting in valve ALHV0005 impacted the ability of the valve to reduce mass/flow below approximately 90 klbm/hr (low flow conditions). Thus, the torque switch setting was not appropriate for all applicable plant conditions, and the post-maintenance test of the new torque switch setting as designated in Job 08505547 was not appropriate to verify that valve ALHV0005 would perform satisfactorily in service.

The licensee initiated Callaway Action Request 201408399 to evaluate the condition.

The licensee noted that the as-left torque switch thrust of ALVHV0005 was significantly less than the other three AFW motor-driven pump flow control valves of approximately 12,000 pounds. The team noted that the licensee did not challenge the low thrust setting for ALHV0005, despite all four of the AFW flow control valves having identical designs and system configurations, which was a missed opportunity to prevent valve ALHV0005 failure on November 15, 2015.

The licensee evaluated this condition in Callaway action request 20140899. The licensee revised the data sheets for valves ALHV0005, ALHV0007, ALHV0009, and ALHV0011 to require a minimum available thrust at torque switch trip value of 10,217 pounds, and the licensees motor-operated valve program commitments require a 25 percent margin be added to this value. The addition of this margin increased the target minimum available thrust at torque switch trip value to approximately 12,771 pounds for valves ALHV0005, ALHV0007, ALHV0009, and ALHV0011. This thrust value is greater than the satisfactorily set and tested as-left thrust after remedial actions were completed on Job 14005755 on November 20, 2014. The licensee reset each of these torque switch settings to align with the motor-operated valve program.

Analysis.

The failure to establish a procedure that included a suitable instructions to set the torque switch on a motor-driven AFW valve after maintenance or testing was a performance deficiency. This finding was more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to establish a suitable testing program for the motor-driven AFW valve torque and thrust settings caused valve ALHV0005 not to close completely, causing the operators to take action and shut down motor-driven auxiliary feedwater pump B. In accordance with Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings at Power, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the issue screened as having very low safety significance (Green) because it was a design or qualification deficiency that did not represent a loss of operability or functionality; did not represent an actual loss of safety function of the system or train; did not result in the loss of one or more trains of non-technical specification equipment; and did not screen as potentially risk significant due to seismic, flooding, or severe weather. The team determined that this finding had a cross-cutting aspect in the area of human performance, challenge the unknown, because the licensee did not stop and challenge that the tested differential pressure across valve ALHV0005 was significantly different than the other valves (H.11)

Enforcement.

Title 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures,

& Drawings, states, in part, that activities affecting quality shall be prescribed by procedures of a type appropriate to the circumstances. Contrary to the above, on March 4, through November 20, 2014, Job 08505547, which contained instructions for valve actuator work and post-maintenance testing, an activity affecting quality, was not appropriate to the circumstances. Specifically, Job 08505547 did not require the motor operated valve torque switch to be set under the most limiting differential pressure the valve would actually experience under all conditions, and had incorrectly set and tested the close torque switch setting on valve ALHV0005. The incorrect close torque switch setting prevented the valve from going full closed during an actual demand in low flow/high differential pressure conditions. In response to this issue, the licensee, using appropriate instructions in Job 14005755, repaired the valve, and confirmed that the close torque switch settings were correct and successfully retested. This violation is being treated as an NCV, consistent with Section 2.3.2.a of the Enforcement Policy. The violation was entered into the licensees corrective action program as Callaway Action Request 201508399. (NCV 05000483/2015009-06, Failure to Have an Adequate Procedure for Setting the Torque and Thrust Values for the Auxiliary Feedwater Pump Flow Control Valves.)

.2.5 Charter Item B.1(f):

Determine whether or not post-maintenance testing and/or surveillance testing should have discovered any of the degraded or non-conforming conditions.

a. Inspection Scope

The team reviewed surveillance and/or post-maintenance tests to determine if these tests should have discovered any of the degraded/non-conforming conditions.

b. Observations and Findings

The team determined that all surveillance testing was in accordance with the technical specifications and the applicable American Society of Mechanical Engineering (ASME)code for in service testing. Because the required surveillance and in service testing consisted of full stroke time testing, it was not of sufficient scope to identify the failure mechanisms as discussed in Section 4OA5.2.7 of this inspection report.

However, the team noted several instances in which post-modification and post-maintenance tests were not appropriate to the circumstances. These issues are discussed in detail in Sections 4OA5.2.2.b, 4OA5.2.3.b.1, 4OA5.2.3.b.2 and 4OA5.2.6.b of this inspection report.

.2.6 Charter Item B.1(g):

Review final modification to the valve controllers for ALHV0005 and ALHV0007 to assess compliance with 10 CFR Part 50, Appendix B and 10 CFR 50.59.

a. Inspection Scope

The NRC team reviewed documentation associated with the most recent design changes made to the AFW system flow valve control cards, including root cause analyses, failure analyses reports, and vendor qualification reports. Specifically, the team assessed what immediate corrective actions had been taken by the licensee to address the design deficiencies identified previously and discussed in 4OA5.2.7.1 of this report. During the time of this inspection these modified control cards were installed in the plant and were considered operable.

b. Observations and Findings

Failure to Identify and Correct Additional Undersized Components on Auxiliary Feedwater System Flow Control Valve Modified Modutronics Controller Cards

Introduction.

The team identified a Green non-cited violation (NCV) of 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, for the licensees failure to identify and correct a condition adverse to quality. Specifically, as of September 23, 2015, the licensee had not taken corrective action, following previous identification of undersized field current rectifier bridges, to ensure that an independent review of the modified circuit design had been completed, or that the modified cards had been subjected to a sufficient testing and qualification program, to ensure that no other undersized components were installed. Thus, following questioning by the team, the licensee identified additional components on the modified circuit cards that were potentially undersized, a condition adverse to quality. The licensee performed an operability evaluation and concluded that the new cards were operable, based on additional circuit analysis that was performed.

Description.

On September 23, 2015, during review of the licensees root cause analysis for the failures of the Modutronics cards, the team identified that the licensee had not taken sufficient actions to ensure that the design of the revised control cards was suitable for their application. The licensees extent of condition review for the failure of the Modutronics cards focused only on the extent of the self-revealing bridge rectifier failures to other valves, but not whether or not other components on these cards could have similar latent design issues. Nuclear Logistics Incorporated (the vendor) had addressed the concern with the undersized bridge rectifiers that were stated to have caused the previous circuit card failure by upgrading the bridge rectifier component, but sufficient testing or design reviews had not been completed of the entire circuit to ensure the design was suitable for its application. At the time of the inspection, the licensee had not completed an independent review of the circuit design and had not taken sufficient actions to ensure the circuit design had been properly reviewed by the vendor. In addition, the deficiencies associated with the lack of complete testing of control cards were not sufficiently addressed.

The licensee sent the control cards back to the vendor to be modified, but the testing that was done subsequent to their modification only mimicked the previous testing.

Again, the cards were not tested with the field circuit under load. This concern was identified as part of the licensees root cause analysis for Callaway Action Request 201505796, but adequate corrective actions to address the lack of sufficient testing of the as-installed control cards had not been taken. The team considered this to be a missed opportunity to identify additional under-rated components. Specifically, the licensee did not rigorously evaluate each of the newly designed components (by test or analysis) to determine if the new modified cards (post-corrective action) were suitable for their application, by looking for additional latent defects.

On September 23, 2015, following the teams questioning, the licensee determined that additional components on the modified cards might also be undersized. Rectifier bridges BR1 and BR3, components in the power supply and armature circuits, respectively, were sized to one-half of the current carrying capability of their similar components for the original Modutronics controller cards. The issue was entered into the licensees corrective action program as Callaway Action Request 201506874. The licensee performed an operability evaluation that concluded that the new cards were operable, based upon additional circuit analyses that was performed. Although the margin to failure was significantly decreased, the licensee determined that the replacement rectifier bridges (BR1 and BR3) would perform satisfactorily in service.

Analysis.

The failure to identify and correct a condition adverse to quality was a performance deficiency. This performance deficiency is more than minor, and therefore, a finding because it adversely affected the Mitigating Systems Cornerstone attribute of Equipment Performance and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to identify and correct design deficiencies associated with these circuit cards could have resulted in the inoperability of auxiliary feedwater control valves and their inability to operate on demand. Using Inspection Manual Chapter 0609, Appendix A, Exhibit 2, Mitigating Systems Screening Questions, dated June 19, 2012, the finding was determined to be of very low safety significance, because it did not affect system design, did not result in a loss of system function, did not represent a loss of function of a single train for greater than its technical specifications allowed outage time, and did not cause the loss of function of one or more non-technical specification trains of equipment designated as high safety-significance. This finding has a human performance cross-cutting aspect in the area of Avoid Complacency, because the licensee did not thoroughly evaluate the issue to ensure that the resolutions address causes and extent of conditions.

Specifically, the licensee had identified that the Modutronics cards failed because of improper design of the field current rectifier bridge, but did not plan for the possibility for other latent issues to determine if other components on the cards were adequately sized for their application (H.12).

Enforcement.

Title 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, states, in part, that, measures shall assure that conditions adverse to quality are promptly identified and corrected. Contrary to the above, for quality-related components associated with auxiliary feedwater system flow control valves, the licensee failed to promptly identify and correct a condition adverse to quality. Specifically, as of September 23, 2015, the licensee had not taken corrective action, following previous identification of undersized field current rectifier bridges, to ensure that an independent review of the modified circuit design had been completed, or that the modified cards had been subjected to a sufficient testing and qualification program, to ensure that no other undersized components were installed. Thus, following questioning by the team, the licensee identified additional components on the modified circuit cards that were potentially undersized, a condition adverse to quality. The licensee performed an operability evaluation that concluded that the new cards were operable, based on additional circuit analysis that was performed. This violation is being treated as an NCV, consistent with Section 2.3.2.a of the Enforcement Policy. The violation was entered into the licensees corrective action program as Callaway Action Request 201506874.

(NCV 05000483/2015009-07, Failure to Identify and Correct Additional Undersized Components on Auxiliary Feedwater System Flow Control Valve Modified Modutronics Controller Cards.)

.2.7 Charter Item B.1(h):

Review the licensees root cause analyses and determine if they were conducted at a level of detail commensurate with the significance of the problem.

a. Inspection Scope

The team reviewed the root cause analyses associated with Callaway Action Request 201505796, ALHV0007 - MDAFP B to SG A HV did not Open from MCB Switch, dated September 21, 2015, and Callaway Action Request 201505332, ALHV0011 did not Respond During MDAFP Operation, dated September 21, 2015.

The team evaluated these documents to determine if they appropriately described the root and contributing causes, extent of cause and condition, and corrective actions to prevent repetition.

b. Observations and Findings

The licensee assembled comprehensive teams to address the failures of AFW system flow control valves ALHV0005, ALHV0007, and ALHV0011. These analyses were reviewed as follows:

.1 Failure of Valves ALHV0005 and ALHV0007 (Callaway Action Request 201505796)

The licensee initiated a root cause team for the failures of valves ALHV0005 and ALHV0007 because the same subcomponent (the field current rectifier bridge on the Modutronics cards) was found failed and burned in the same manner, indicating a potential common cause.

The licensees initial cause for failure of valve ALHV0005, as documented in Callaway Action Request 201408899 was infant mortality (early failure because of a manufacturing defect). However, this assumption of cause was refuted by vendor information received on May 21, 2015, indicating that the rectifier bridges were undersized for their required current carrying capability in service. As discussed in Section 4OA5.2.4.3 of this report, this was a missed opportunity to elevate the issue to Significance Level 1, requiring a root cause analysis. Following the August 11, 2015, failure of valve ALHV0007, the licensee initiated Callaway Action Request 201505796 and elevated the issue to Significance Level 1.

The licensees root cause analysis identified the following root and contributing causes:

• Root

Cause:

The reverse-engineered design of the electronic positioner did not meet the original design specification for the bridge rectifier circuit.

• Contributing Cause 1: The responsible design engineer did not maintain sufficient vendor oversight to ensure adequate testing was performed.

• Contributing Cause 2: Vendor did not test the field rectifier bridge to challenge the design.

• Contributing Cause 3: Callaway guidance does not require proof testing for first-of-a-kind reverse-engineered products.

• Contributing Cause 4: The design engineer did not require the vendor to conduct a proof test to verify design.

• Contributing Cause 5: The responsible design engineer and qualified reviewer did not clearly understand the required scope of their review for the vendor qualification testing document.

• Contributing Cause 6: The lead responder for Callaway Action Request 201408899 used a manufacturer data sheet that contained misleading data to determine the capacity of the field bridge rectifier.

• Contributing Cause 7: Callaway Action Request 201408899 was rescreened from Significance Level 1 to 3.

The team determined that the licensee performed this root cause analysis at an appropriate level of detail to identify the issues, and the corrective actions to prevent recurrence. However, the team had the following observations concerning the root cause analysis in Callaway Action Request 201505796:

• The cause analysis did not discuss the effect of the misadjusted limit switches for the failure of valve ALHV0005, as discussed in Section 4OA5.2.4.b.1 of this report. The mis-adjustment of the limit switches resulted in the continuous energization of the field current for valve ALHV0005 for approximately 22 days.

This continuous energization and subsequent heat-up of the field rectifier bridge contributed to the valves failure.

• The cause analysis did not discuss the missed opportunities to rescreen the failure of valve ALHV0005 to Significance Level 1 and perform a prompt operability determination when the vendor failure analysis was received on May 21, 2015, as discussed in Section 4OA5.2.4.3 of this report. This contributed to the length of time that the degraded condition existed.

• Although the cause analysis identified the failures to verify the adequacy of the design, it did not contain corrective actions to completely verify the design of the reverse-engineered Modutronics cards currently installed in the plant. Following prompting by the team, the licensee performed this review and identified additional undersized components. This issue is further discussed in Section 4OA5.2.6.b of this inspection report.

The licensee initiated Callaway Action Request 201506919 to evaluate the teams observations, and make enhancements to the root cause analysis as needed.

No findings of significance were identified.

.2 Failure of Valve ALHV0011 (Callaway Action Request 201505332)

The licensee initiated a root cause team to determine the cause of the failure of valve ALHV0011 to respond from the main control room on July 23, 2015. Following a forced shutdown because of reactor coolant leakage in excess of the technical specification limits, the licensee initiated AFW for secondary heat removal. When transitioning to the motor-driven AFW pumps, the licensee attempted to open valve ALHV0011 from the control room, but it failed to move. An equipment operator was dispatched locally who manually cracked the valve open with the handwheel.

Subsequently, operators were able to operate the valve from the control room. The licensee initiated Callaway Action Request 201505332 to place this item into the corrective action program.

The licensee found that the valve did not operate from the main control room because the feedback potentiometer in the control system was off scale causing the actuator to misinterpret the actual position of the valve. With the feedback potentiometer off scale, the control system interpreted the feedback signal as 50 percent open while it was closed. Thus, any demand signal to less than 50 percent open would result in no valve movement. The licensee found that Procedure MTE-ZZ-QA033, MOVATS [Motor Operated Valve Actuator Test System] Testing of Torque Controlled Modutronic Limitorque Motor Operated Rising Stem Valves, did not contain the instructions for setting the feedback potentiometer. This issue is further discussed in Section 4OA5.2.3.b.2 of this inspection report. The licensee determined the root and contributing causes as follows:

• Root

Cause:

The implementation of programmatic controls and programmatic changes to valves ALHV0005, ALHV0007, ALHV0009 and ALHV0011 did not ensure that the written instructions were adequate.

• Contributing Cause 1: Roles and responsibilities for technical oversight, management, and monitoring of valves ALHV0005, ALHV0007, ALHV0009 and ALHV0011 were not clearly defined.

• Contributing Cause 2: The revision of the written instructions was performed without a technical/cross-disciplinary review by engineering.

• Contributing Cause3: Existing procedural guidance does not specifically test for an uncalibrated feedback potentiometer as a potential failure mechanism.

• Contributing Cause 4: Callaway personnel recognized that the valve setup and testing instructions were inadequate but did not take steps to correct the instructions after completion of the tasks.

The team determined that the licensee performed this root cause analysis at an appropriate level of detail to identify the issues and the corrective actions were adequate to prevent recurrence.

No findings of significance were identified.

4OA6 Meetings, Including Exit

Exit Meeting Summary

On September 25, 2015, following the onsite portion of the inspection, the inspectors provided a debrief of the preliminary results to Mr. Dave Neterer, Vice President, Nuclear Operations, and other members of the licensee staff. The licensee acknowledged the issues presented.

On December 9, 2015, the inspectors presented the final inspection results to Mr. Dave Neterer, Vice President, Nuclear Operations, and other members of the licensee staff. The licensee acknowledged the issues presented. The licensee confirmed that any proprietary information reviewed by the inspectors had been returned or destroyed.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

T. Herrmann, Vice President, Engineering

D. Hall, Director, Engineering Programs

J. Hiller, Supervising Engineer, Risk Management

M. Hudson, Principal Engineer

J. Hutchison, Consulting Engineer

S. Kovaleski, Director, Engineering Design

J. Little, Consulting Engineer

S. Maglio, Manager, Regulatory Affairs

M. McLachlan, Senior Director Engineering

J. Montgomery, Engineer

S. Taylor, Career Engineer

D. Turley, Supervising Engineer

NRC Personnel

T. Hartman, Senior Resident Inspector

D. Loveless, Senior Reactor Analyst

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

05000483/2015009-01 NCV Failure to Verify the Suitability of the Design of the Reverse-Engineered Replacement Controller Cards for the Auxiliary Feedwater Flow Control Valves (Section 4OA5.2.2.b)

05000483/2015009-02 NCV Failure to Have an Adequate Procedure for Calibration of the Auxiliary Feedwater Pump Flow Control Valve Potentiometer (Section 4OA5.2.3.b.1)

05000483/2015009-03 NCV Failure to Have an Adequate Post-Maintenance Test for Setting the Motor-Driven Auxiliary Feedwater Flow Control Valve Modutronics Potentiometer (Section 4OA5.2.3.b.2)

05000483/2015009-04 NCV Two Examples of a Failure to Properly Designate the Significance Level of Callaway Action Requests.

(Section 4OA5.2.4.b.1)

05000483/2015009-05 NCV Failure to Determine the Cause and Take Corrective Action to Preclude Repetition for the Inadequate Design of Auxiliary Feedwater Flow Control Valve Modutronics Cards.

(Section 4OA5.2.4.b.2)

05000483/2015009-06 NCV Failure to Have an Adequate Procedure for Testing the Torque and Thrust Values for the Auxiliary Feedwater Pump Flow Control Valves (Section 4OA5.2.4.b.3)

05000483/2015009-07 NCV Failure to Identify and Correct Additional Undersized Components on Auxiliary Feedwater System Flow Control Valve Modified Modutronics Controller Cards (Section 4OA5.2.6.b)