IR 05000458/2016010

From kanterella
(Redirected from ML16146A792)
Jump to navigation Jump to search
NRC Supplemental Inspection Report 05000458/2016010, March 21, 2016 Through April 13, 2016
ML16146A792
Person / Time
Site: River Bend Entergy icon.png
Issue date: 05/25/2016
From: Troy Pruett
NRC/RGN-IV/DRP
To: Maguire W
Entergy Operations
GREG WARNICK
References
EA-15-043 IR 2016010
Download: ML16146A792 (23)


Text

May 25, 2016

SUBJECT:

RIVER BEND STATION - NRC SUPPLEMENTAL INSPECTION REPORT 05000458/2016010

Dear Mr. Maguire:

Prior to January 30, 2015, your simulator failed to demonstrate expected plant response to operator input and to normal, transient, and accident conditions to which the simulator has been designed to respond. The simulator failed to accurately model feedwater flow and reactor vessel level response following a scram, failed to provide the correct alarm response for loss of a reactor protection system motor generator set, and failed to correctly model the operation of the startup feedwater regulating valve. These simulator modeling issues led to negative operator training, which subsequently complicated the operators response to a reactor scram at River Bend Station on December 25, 2014.

On June 29, 2015, the U.S. Nuclear Regulatory Commission (NRC) completed a Special Inspection Team inspection at the River Bend Station. Based on the results of this inspection, documented in NRC Inspection Report 05000458/2015009 dated July 7, 2015, (ML15188A532)

and the final significance determination in NRC Inspection Report 05000458/2015009 dated September 10, 2015, (ML15253A352) the NRC assigned a White finding Action Matrix input to the Mitigating Systems Cornerstone effective the second quarter of 2015.

In response to this Action Matrix input, the NRC informed you that a supplemental inspection using Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, would be required. On February 1, 2016, you informed the NRC that River Bend Station was ready for the supplemental inspection. On March 25, 2016, the NRC completed an on-site inspection and discussed the results with you and other members of your staff. On April 13, 2016, the NRC completed the supplemental inspection and discussed the results of this inspection with you and other members of your staff.

The results of this inspection are documented in the enclosed report.

The NRC performed this supplemental inspection to determine if: (1) the root and contributing causes for the significant issues were understood; (2) the extent of condition and extent of cause for the identified issues were understood; and (3) your completed or planned corrective actions were sufficient to address and prevent repetition of the root causes and contributing causes.

Your staff identified that the root cause for the White finding was ineffective simulation benchmarking, as an organizational to organizational programmatic breakdown, between the Operations organization and the Training organization. This resulted in a condition that allowed the simulator configuration to be misaligned with the design basis of River Bend Station.

The NRC determined that misalignment of the simulator configuration to the design basis of River Send Station led to negative operator training, which complicated the operators response to a reactor scram on December 25, 2014. The NRC concluded that your staff identified appropriate corrective actions to address the root cause, contributing cause, and extent of cause of the simulator configuration misalignment. During the on-site portion of the inspection, NRC inspectors determined that your staffs extent of condition evaluation was too limited in scope. The simulator testing activities used were not effective in identifying differences between the simulator and River Bend Station operating characteristics of components and systems that resulted in negative operator training. In response, your staff conducted an additional extent of condition evaluation, which was provided to the inspectors on March 29, 2016, after completion of the on-site portion of the inspection. The inspectors performed an in-office review of the information and, due to concerns regarding the adequacy of the sample selection, determined that the extent of condition evaluation was inadequate. Based on these determinations, the NRC concluded that the inspection objective involving the extent of condition was not met.

The NRC has determined that completed or planned corrective actions were insufficient to address this performance issue. Specifically, the extent of condition review was insufficient.

Therefore, the White finding will remain open and continue to receive consideration as an Action Matrix input until inspectors verify that all inspection objectives have been met. You should notify the NRC of your readiness for a re-inspection when corrective actions have been completed.

No findings were identified during this inspection.

In accordance with Title 10 of the Code of Federal Regulations (10 CFR) 2.390, Public Inspections, Exemptions, Requests for Withholding, a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRCs Public Document Room or from the Publicly Available Records (PARS) component of the NRC's Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/R Lantz for Troy W. Pruett, Director Division of Reactor Projects Docket No. 50-458 License No. NPF-47

Enclosure:

Inspection Report 05000458/2016010 w/ Attachment: Supplemental Information

REGION IV==

Docket: 05000458 License: NPF-47 Report: 05000458/2016010 Licensee: Entergy Operations, Inc.

Facility: River Bend Station Location: 5485 U.S. Highway 61N St. Francisville, LA 70775 Dates: March 21 through April 13, 2016 Inspectors: C. Henderson, Resident Inspector, Lead Inspector P. Presby, Senior Operations Engineer Approved By: Troy W. Pruett Director Division of Reactor Projects-1- Enclosure

SUMMARY

IR 05000458/2016010; 03/21/2016 - 04/13/2016; River Bend Station; Supplemental

Inspection - Inspection Procedure 95001 This supplemental inspection was conducted by a resident inspector from the Cooper Nuclear Station and a senior operations engineer from the NRCs Region I office. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process.

No findings were identified.

Cornerstone: Mitigating Systems

The NRC inspectors performed this supplemental inspection in accordance with Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, dated February 9, 2011, to assess the licensees evaluation associated with the failure to maintain the simulator modeling consistent with the reference unit (River Bend Station)response for normal and transient conditions. Specifically, the licensees failure to maintain the simulator consistent with the reference unit response for normal and transient conditions involved the failure to: (1) correctly model leakage flow rates across the feedwater regulating valves; (2) provide the correct alarm response for a loss of a reactor protection system motor generator set; and (3) correctly model the behavior of the startup feedwater regulating valve controller. These simulator modeling issues led to negative operator training, which subsequently complicated the operators response to a reactor scram at River Bend Station on December 25, 2014. The NRC staff previously characterized this issue as having low to moderate safety significance (White), as documented in NRC Inspection Report 05000458/2015009 dated September 10, 2015 (ML15253A352).

The inspectors determined that the licensee identified one root cause and one contributing cause for the White finding. The root cause was identified as ineffective simulation benchmarking, as an organizational to organizational programmatic breakdown, between the Operations organization and the Training organization. The contributing cause was the lack of implementation of a consistent process when personnel failed to recognize or correct the process weakness of Procedure EN-TQ-202, Simulator Configuration Control, until April 21, 2014. This resulted in a condition that allowed simulator configuration to be misaligned with the design basis of the reference unit, and prevented early detection, extended the condition, and resulted in additional consequences, in that negative operator training impacted operator response during an actual event.

The inspectors determined that the licensee completed corrective actions to ensure the three simulator fidelity issues were corrected and to provide procedural guidance for simulator benchmarking, simulator configuration control, and organizational programmatic interface.

However, the inspectors determined during the on-site portion of the inspection that the extent of condition evaluation was too limited in scope. The simulator testing activities used were not effective in identifying differences between the simulator and reference unit operating characteristics of components and systems that resulted in negative operator training. In response, the licensee conducted an additional extent of condition review, which was provided to the inspectors on March 29, 2016, after completion of the on-site portion of the inspection.

The inspectors performed an in-office review and, due to concerns regarding the adequacy of the sample selection, determined that the extent of condition evaluation was inadequate.

As a result of the inadequate evaluation of extent of condition, the White finding associated with the failure to maintain the simulator consistent with the reference unit response for normal and transient conditions will remain open. The licensee entered this deficiency into their corrective action program for resolution as Condition Reports CR-RBS-2016-02541, CR-RBS-2016-02550, and CR-RBS-2016-02896.

REPORT DETAILS

OTHER ACTIVITIES

4OA4 Supplemental Inspection

.01 Inspection Scope

This inspection was conducted in accordance with Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, to assess the licensees evaluation of a White finding, which affected the Mitigating Systems Cornerstone in the reactor safety strategic performance area. The inspection objectives were to:

  • provide assurance that the root causes and contributing causes of risk-significant performance issues are understood;
  • provide assurance that the extent of condition and extent of cause of risk significant performance issues are identified; and
  • provide assurance that the licensees corrective actions for risk-significant performance issues are sufficient to address the root and contributing causes and to prevent recurrence.

The licensee remained in the Regulatory Response Column of the NRCs Action Matrix in the second quarter of 2015 as a result of one inspection finding of low to moderate safety significance (White). The finding was associated with the licensees failure to maintain the simulator consistent with the reference unit (RBS) response for normal and transient conditions. Specifically, the licensees failure to maintain the simulator consistent with the reference unit response for normal and transient conditions involved the failure to:

(1) correctly model leakage flow rates across the feedwater regulating valves;
(2) provide the correct alarm response for a loss of a reactor protection system motor generator set; and
(3) correctly model the behavior of the startup feedwater regulating valve controller. These simulator modeling issues led to negative operator training, which subsequently complicated the operators response to a reactor scram on December 25, 2014. The finding was characterized as having a low to moderate (White)safety significance based on Inspection Manual Chapter 0609, Appendix I, License Operator Requalification Significance Determination Process, as discussed in NRC Inspection Report 05000458/2015009 (ML15188A532).

On February 1, 2016, the licensee informed the NRC that they were ready for the supplemental inspection. In preparation for the inspection, the licensee performed a root cause evaluation (RCE) under Condition Report CR-RBS-2015-04375. Revisions 3 and 3A of the RCE report, dated January 25, 2016, and March 28, 2016, respectively, were provided to the inspectors for review. The licensee also performed a Pre-NRC 95001 Inspection Snapshot Assessment and Mock Inspection, which was completed in January 2016.

The inspectors reviewed the licensees RCE in addition to other evaluations conducted in support of the RCE. The inspectors reviewed corrective actions that were taken or planned to address the identified causes. The inspectors also held discussions with licensee personnel to determine whether the root and contributing causes as well as the contribution of safety culture components were understood, and whether corrective actions taken or planned were appropriate to address the causes and preclude repetition.

.02 Evaluation of the Inspection Requirements

02.01 Problem Identification a.

Determine whether the evaluation documented who identified the issue (i.e., licensee identified, self-revealing, or NRC-identified) and under what conditions the issue was identified.

The licensees RCE documented the failure to maintain the simulator consistent with the reference unit response for normal and transient conditions related to feedwater flows, alarm response, and behavior of the startup feedwater regulating valve (FRV) controller.

These simulator modeling issues led to negative operator training. This subsequently complicated the operators response to a reactor scram in the actual plant on December 25, 2014. The inspectors determined that the RCE appropriately documented that the issue was identified by the NRC Special Inspection Teams review of reference unit response compared to the simulator response of the December 25, 2014, reactor scram. The issue was therefore NRC-identified.

The inspectors concluded that the licensees RCE adequately documented who identified the issue and under what conditions the issue was identified.

b.

Determine whether the evaluation documented how long the issue existed and prior opportunities for identification.

The licensees RCE included a determination of when the simulator fidelity issues had occurred associated with the failure to:

(1) correctly model leakage flow rates across the feedwater regulating valves;
(2) provide the correct alarm response for a loss of a reactor protection system motor generator set; and
(3) correctly model the behavior of the startup FRV controller.

The licensee determined the simulator was originally designed with a leakage variable in the FRV coding. However, the variable on each FRV was set at zero leakage with no instructor capability to manipulate the variable. The reference unit FRVs were not zero leakage components when new, and over time, they degraded to a higher leakage rate.

The magnitude of the deviation between the simulator and reference unit was only evident at low flow conditions such as reactor startup or shutdown conditions. The detection of this condition relies upon testing required by ANSI/ANS 3.5, Nuclear Power Plant Simulators for Use In Operator Training and Examination, post event simulator testing (PEST), and feedback from simulator users in the simulator discrepancy resolution process. This deviation was original to the simulator design, was of sufficient magnitude to be significant, and would be visible under duplicated operating conditions.

This deviation was not visible through the ANSI/ANS 3.5 testing, because testing conditions in the simulator (the reference scram scenario) did not represent the same operating conditions associated with the December 25, 2014, reactor scram. The PEST conducted to evaluate the October 17, 2014, scram did not reveal the deviation due to complications, and PEST was not historically required for all power transients and did not include the requirement to capture and compare alarms. The licensee was aware of the higher FRV leakage in the reference unit, but the condition was not identified through the simulator discrepancy resolution process. The incorrect FRV leakage flow rate modelling was not identified until the NRC Special Inspection Team reviewed the reference unit response compared to the simulator response of the December 25, 2014, reactor scram. Therefore, the simulator did not accurately represent FRV leakage behavior of the reference unit since simulator installation.

The licensee determined the simulator was originally designed where the reactor high pressure and drywell high pressure annunciators response was consistent with reference unit design documentation for a loss of reactor protection system (RPS) motor generator set, which results in a loss of RPS power. However, the reference unit experienced actuation of the reactor high pressure, drywell high pressure, and other annunciators when the loss of RPS motor generator set occurred. This condition was identified by the licensee on December 11, 2014, during a loss of offsite power training exercise in the simulator. During the training exercise, the licensee identified simulator testing for a loss of RPS power did not include any reference to the annunciators in question. The licensee initiated Simulator Deficiency Report DR 14-0155 to document the condition; however, the simulator was not modified to reflect actual reference unit alarm response until January 15, 2015, which was after the December 25, 2014, reactor scram. Therefore, the simulator did not provide the correct alarm response for a loss of an RPS motor generator set since simulator installation.

The licensee determined Simulator Deficiency Report DR 96-0071 revised the startup FRV simulator model in 1996. Deficiency Report 96-0071 changed the simulator startup FRV model to have linear and balanced operating characteristics, and require little setpoint deviations prior to initial opening. However, the reference unit startup FRV was a non-linear, non-balanced component requiring significant setpoint deviation prior to initial opening. The magnitude of this deviation related to the break-away opening characteristic of the startup FRV was not monitored by ANSI/ANS 3.5 testing or PEST, but should have been identified by operator feedback through the simulator discrepancy resolution process. This issue was not identified until the NRC Special Inspection Team reviewed the reference unit response compared to the simulator response of the December 25, 2014, reactor scram. Therefore, the simulator did not correctly model the startup FRV behavior of the reference unit since 1996.

The inspectors concluded that the licensees RCE was adequate with respect to identifying how long the issue existed and prior opportunities for identification.

c. Determine whether the evaluation documented the plant-specific risk consequences, as applicable, and compliance concerns associated with the issue.

The licensees RCE included a plant-specific risk-based safety significance evaluation of the issue. The licensee determined that they failed to maintain the simulator consistent with the reference unit response for normal and transient conditions related to feedwater flows, alarm response, and behavior of the startup FRV controller. These simulator modeling issues led to negative operator training. This subsequently complicated the operators response to a reactor scram in the actual plant on December 25, 2014. The licensees risk evaluation concluded that the overall risk significance of the issue was of low to moderate (White) significance, which was consistent with the result of the NRCs significance determination process for the White finding as discussed in NRC Inspection Report 05000458/2015009.

The inspectors concluded that the licensees RCE appropriately documented the plant-specific risk consequences and compliance concerns associated with the issue.

d. Findings

No findings were identified.

02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation a.

Determine whether the problem was evaluated using a systematic methodology to identify the root and contributing causes.

The inspectors determined that the licensee conducted an RCE in which three primary evaluation methods were used: Event and Causal Factors Chart (E&CF), Barrier Analysis, and Organization and Programmatic Evaluation (O&PE). The licensees evaluation identified one root cause and one contributing cause associated with this issue. The root cause was identified as ineffective simulation benchmarking, as an organizational to organizational programmatic breakdown, between the Operations organization and the Training organization. The contributing cause identified by the licensees evaluation was a lack of implementation of a consistent process when personnel failed to recognize or correct the process weakness of Procedure EN-TQ-202, Simulator Configuration Control, until April 14, 2014. These root and contributing causes resulted in allowing the simulator configuration to be misaligned with the design basis of the reference unit, until the simulator fidelity issues were corrected following identification by the NRC Special Inspection Teams review of reference unit response compared to the simulator response of the December 25, 2014, reactor scram. These issues are further discussed in Section 02.02.b below.

The inspectors concluded that the licensees RCE used a systematic methodology to identify the root and contributing causes of the issue.

b.

Determine whether the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

The licensees RCE provided a detailed history of the simulator. This historical evaluation included construction, major upgrades, various testing activities, and fidelity issues. The root and contributing causes were derived from an E&CF method correlation to the Safety Culture

Analysis.

This approach included consideration of Organizational and Programmatic aspects, as well as consideration of Failed Barriers.

The root cause was supported by nine key factors, and the contributing cause was supported by five causal factors. The nine key factors that supported the root cause were:

(1) The lack of teamwork between the Operations and Training organizations as demonstrated by the failure to communicate and coordinate activities within and across organizational boundaries. Personnel did not consistently initiate operator work arounds, training evaluation action requests (TEARs), or simulator deficiency reports when performance differences were noted, such as the FRV leakage;
(2) The lack of rigor and robust controls in simulator modification validation and verification practices.

This sometimes led to software revisions without clear bases and inadequate documentation. This contributed to the licensees failure to correctly model the behavior of the startup FRV in the simulator;

(3) Several instances of simulator documentation lacked clarity and were subject to interpretation. The lack of documented rationale for changes obscured the connection to the simulator design basis. This contributed to the licensees failure to correctly model the behavior of the startup FRV in the simulator;
(4) The failure to recognize, through the corrective action program process, the impact on operators ability to regulate reactor vessel water level while dealing with known degraded equipment reliability. This was a missed opportunity for the licensee to identify a trend of level eight trips and the impact on operators during response to a reactor scram;
(5) A limited process for capturing alarms received during a plant transient and submitting them to training for evaluation. This contributed to the licensee not identifying the simulator failure to provide the correct alarm response for a loss of a RPS motor generator set;
(6) Conditions were not identified as operator work arounds or burdens. Assessment performed per Procedure EN-FAP-OP-006, Operator Aggregate Impact Index Performance Indicator, did not recognize that the operator work around for FRV operation was not communicated to the Training organization or identified in the operations aggregate index. This was another missed opportunity by the licensee to identify that the simulator did not reflect FRV leakage for the reference unit;
(7) The checklist in Training Policy 97-02, Training Simulator Configuration Control, did not contain a review of the operator work around or burden list for impact on training; this would have provided an opportunity for the training organization to identify simulator fidelity issues for known equipment reliability issues, such as FRV leakage;
(8) A weakness in legacy documentation and process controls confused the bases and decision making process. This contributed to the licensees failure to correctly model the behavior of the startup FRV in the simulator; and
(9) Known equipment reliability issues, such as significant FRV leakage, that were not reflected in the simulator were not challenged nor corrected. In addition, testing routines were not critically evaluated to determine if there were inherent limitations, which contributed to the licensee not identifying the simulator fidelity issues that led to negative operator training.

The five causal factors that supported the contributing cause were:

(1) Procedure adherence; personnel did not consistently initiate operator work arounds, training evaluation action requests (TEARs), or simulator deficiency reports when performance differences were noted;
(2) Conservative bias; software modification and deficiency report acceptance process did not consistently apply a conservative approach to validate through positive and negative tests;
(3) Issue evaluation; simulator configuration control processes did not specifically address simulation extent of condition evaluation for modeling discrepancies;
(4) Issue resolution; weakness in extent of condition consideration contributed to some repetitive simulator deficiency reports; and
(5) Issue trending; the Training organization did not categorize and trend simulator deficiency reports. Simulator fidelity issues were not considered in the aggregate and in some cases contributed to the simulator fidelity issues associated with the White finding.

The combination of the root cause, contributing cause, supporting nine key factors, and causal factors resulted in a condition where the licensee was not effective in identifying and correcting differences between the simulator and the operating characteristics of the reference unit. This also prevented early detection, extended the condition, and resulted in negative operator training.

With the exception of the scope of the extent of condition evaluation discussed in Section 02.02.d below, the inspectors concluded the licensees RCE was conducted to a level of detail commensurate with the significance of the problem.

c. Determine whether the root cause evaluation included consideration of prior occurrences of the problem and knowledge of prior operating experience.

The licensees RCE included a review of internal and external operating experience.

The licensee conducted a fleet-wide search of the Entergy corrective action program and an industry-wide search for operating experience of previously documented conditions involving simulator fidelity issues. The licensee identified multiple examples of internal and external operating experience that indicated similar breakdowns in process, communication, and barriers. The licensee recognized there were multiple missed opportunities by the station to review the relevant internal and external operating experience. In each instance, the licensee dispositioned the similar operating experience without taking any actions, which contributed to the missed opportunities to identify the breakdowns in process, communication, and barriers.

The inspectors concluded that the licensees RCE included a consideration of prior occurrences of the problem involving simulator fidelity issues. The inspectors also concluded that the licensees evaluation included a consideration of prior operating experience.

d. Determine whether the root cause evaluation addressed the extent of condition and the extent of cause of the problem.

The licensees RCE included an extent of cause evaluation for the identified root cause and contributing cause. The root cause was identified as ineffective simulation benchmarking, as an organizational to organizational programmatic breakdown, between the Operations and Training organizations. The contributing cause was identified as a lack of implementation of a consistent process when personnel failed to recognize or correct the process weakness of Procedure EN-TQ-202, Simulator Configuration Control, until April 14, 2014. The object of the extent of cause review was to identify organization to organization programmatic breakdowns and lack of implementation of a consistent process. The listed defects were the failure to identify and communicate differences between the simulator and reference unit, and that personnel failed to recognize or correct process weaknesses. This review included the corrective action program, engineering change program, work management process, operating experience, department performance review meeting (DPRM), student feedback and training critiques, simulator review board, and assessment and benchmarking. Corrective actions were developed and completed to enhance these processes. For example, procedural guidance was developed for: post-transient alarm collection and transmittal to the Training organization; review of the Operations Aggregate Index; and requirements for training evaluation action request (TEAR)initiation. The inspectors concluded that the licensees RCE adequately addressed the extent of cause of the problem.

The licensees extent of condition review extended to simulator model performance in a simulated training environment. The object of the extent of condition review was to ensure reasonable and prudent actions are taken to provide confidence that there are no other simulator modelling issues that could contribute to negative operator training. The listed defect in the RCE for the extent of condition review was that the simulator failed to demonstrate the expected reference unit response to operator input, normal transients, and accident conditions. As part of this review, the licensee implemented two corrective actions to validate the extent of condition, thereby ensuring deviations between the simulator and reference unit met the tolerance criteria outlined in ANSI/ANS 3.5, Nuclear Power Plant Simulators For Use In Operator Training and Examination, Revision 2009. These actions were to perform:

(1) all simulator transient tests defined in Procedure EN-TQ-202, Simulator Configuration Control, Revision 9, Attachment 9.3, Transient Tests and Data Collection; and
(2) every malfunction test designated in Procedure EN-TQ-202, Attachment 9.6, Malfunctions. The completion of the tests were documented in Condition Report CR-RBS-2015-04375, Corrective Actions (CAs)46 and 47. The inspectors noted that these same tests are performed on an annual basis per the requirements of ANSI/ANS 3.5. The inspectors validated that the licensee had been performing these tests as required; however, the simulator transient and malfunction tests defined in Procedure EN-TQ-202 failed to identify the issues in the subject White finding (See Section 02.01.b above). Therefore, the inspectors questioned whether the testing conducted under CAs 46 and 47 would adequately evaluate the extent of condition of the listed defect, which was that the simulator model did not match the reference unit operation.

The ten simulator transient tests defined in Procedure EN-TQ-202, Attachment 9.3, are conducted per ANSI/ANS 3.5. These simulator transient tests are used to verify the fidelity of the integrated response of simulation to a set of major events, such as a manual scram or simultaneous closure of all main steam isolation valves (MSIVs). The transients are initiated and performed on the simulator without operator action, while monitoring approximately 20 key parameters indicative of overall reactor and drywell response. The transient tests utilize best estimated analyses in contrast to actual reference unit data as the benchmark for simulator comparison. The transient tests are not capable of identifying the unexpected plant response related to feedwater flows, alarm response, and behavior of the startup FRV controller. For example, the transient test would not have identified the reference unit behavior for the high drywell pressure alarm on a loss of RPS bus power, because the loss of RPS bus power was not evaluated by the transient tests.

The inspectors noted that the malfunction tests designated in Procedure EN-TQ-202, 9.6, had been performed as a corrective action in a previous apparent cause evaluation (ACE) extent of condition review documented in Condition Report CR-RBS-2014-03006. This ACE was conducted prior to the December 25, 2014, scram and failed to identify the three simulator fidelity issues (leakage flow modeling across the FRVs, alarm response for a loss of RPS motor generator set, and incorrect modelling of the startup FRV controller) that led to negative operator training. This was also identified by the licensees RCE as a missed opportunity because previous extent of condition reviews were limited in scope, and because the simulator testing activities were not effective in identifying and correcting differences from the operating characteristics of the reference unit components and systems. Therefore, the inspectors concluded that the re-performance of the malfunction tests designated in Procedure EN-TQ-202,

.6 , was not adequate to evaluate the extent of condition of the problems

identified in the subject White finding.

Based on the above, the inspectors determined the extent of condition review was narrowly focused and would not adequately encompass an extent of condition review of the listed defect, which was that the simulator modeling did not match the reference unit operation. In response, the licensee initiated Corrective Action CR-RBS-2015-04375-52 to conduct an additional extent of condition review and documented this review as CA 52 to the RCE. The licensee completed the additional extent of condition review and provided the information to the inspectors on March 28, 2016. The licensee initiated Condition Report CR-RBS-2016-02541 on March 29, 2016, to document the narrow focus of the RCE extent of condition review. The licensee initiated Condition Report CR-RBS-2016-02550 on March 30, 2016, as part of a post-inspection critique to evaluate the licensees supplemental inspection preparations.

The inspectors completed an in-office review of the licensees additional extent of condition evaluation documented in Condition Report CR-RBS-2015-04375, CA 52, on March 29, 2016. The objective of the licensees evaluation was to validate the simulator model by:

(1) validating simulator model fidelity for five components against the reference unit component operation. The licensees selection criteria were:
(a) one component subject to degrading conditions similar to the FRV leakage;
(b) two valve controller systems similar to the control system of the startup FRV; and
(c) a Group 6 isolation logic system similar to the logic of the RPS alarm response; and
(2) validating that historical simulator deficiency report resolution did not introduce new errors and was based on actual design. The licensees evaluation included a provision to increase the number of selected components by an additional 50 percent if the results of the extent of condition review revealed a level three training discrepancy, which is defined in the licensees training procedures as being a discrepancy which has a higher training impact due to its association with a safety significant system, impact on reactivity response, or relevance to an upcoming training topic. As an independent verification of the sample selection, the NRC resident inspector staff conducted a control room panel walkdown and identified additional valve controllers that were not selected by the licensee. The inspectors reviewed this additional controller population and did not identify any documentation in CA 52 for the exclusion of the valve controllers. As a result of the in-office review and independent verification, the inspectors questioned the adequacy of the extent of condition evaluation. The licensee entered this deficiency into their corrective action program as Condition Report CR-RBS-2016-02896.

In summary, the inspectors concluded that the licensees extent of cause evaluation was adequate, but the extent of condition evaluation was too limited in scope. Based on these determinations, the inspectors concluded that the inspection objective involving the extent of condition evaluation was not met.

e. Determine whether the root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in IMC 0310.

The licensees RCE included a review of whether a weakness in any of the safety culture components contributed to any causes of the issue. The licensee identified weaknesses in safety culture components that were related to the identified root cause and contributing cause. These safety culture components included change management (H.3), teamwork (H.4), documentation (H.7), avoid complacency (H.12),and consistent process (H.13) within the area of human performance.

Change management (H.3), documentation (H.7), and consistent process (H.13) were associated with the lack of rigor and robust controls of past simulator modification validation and verification. These led to software revisions without a clear basis and inadequate documentation as indicated by the startup FRV simulator modeling issue.

Teamwork (H.4) and avoid complacency (H.12) were associated with long-standing equipment reliability issues, such as the failure to correctly model leakage flow rates across the FRVs, that were not effectively communicated, challenged, or corrected to update the simulator model and prevent negative operator training.

The inspectors concluded that the licensees RCE included an appropriate consideration of safety culture components.

f. Findings

No findings were identified.

02.03 Corrective Actions a.

Determine whether appropriate corrective actions are specified for each root and contributing cause or that the licensee has an adequate evaluation for why no corrective actions are necessary.

The licensees RCE identified several corrective actions. The principal corrective actions to address the root and contributing causes included process changes to ensure changes in reference unit performance would be communicated to the training simulator support group to maintain simulator fidelity. Station Procedure OSP-0022, Operations General Administrative Guidelines, Revision 85, was revised to include guidance to:

(1) capture post-transient alarms;
(2) submit post-transient alarms to training for evaluation;
(3) submit every Procedure EN-OP-117, Operations Assessment, transient snapshot assessment to training for evaluation; and
(4) improve post event simulator testing (PEST). These guidelines were also included in revisions to the abnormal operating procedures and Procedure GOP-0005, Power Maneuvering, Revision 325.

The licensee revised the R-DAD-TQ series procedures to improve the simulator testing program, requirements for reference unit transient simulator benchmarking, simulator maintenance instructions, simulator modification resolution requirements, and simulator deficiency prioritization and to include adequate documentation to utilize the corrective action program to request and resolve reference unit operation data needs. The licensee conducted training with operations personnel on changes to the simulator model, and with the responsible managers on lessons learned for not identifying the excessive FRV leakage as a case study. Additionally, regulatory requirements for simulator fidelity were reviewed with appropriate licensee staff members.

The inspectors reviewed each of these corrective actions and determined they adequately addressed the identified root and contributing causes. The inspectors noted that an expanded extent of condition evaluation may result in the identification of additional corrective actions that are appropriate to address additional issues associated with the root and contributing causes. These actions will be evaluated in a future NRC supplemental inspection.

b. Determine whether corrective actions have been prioritized with consideration of risk significance and regulatory compliance.

The inspectors identified, as of the date of completion of the on-site inspection, that all except one corrective action for the simulator fidelity issues were completed. This action was associated with the failure to provide the correct alarm response for a loss of an RPS motor generator set. The licensee initiated Condition Report CR-RBS-2015-08800 to determine why certain control room annunciators actuated following the loss of RPS power and reactor scram that occurred on December 25, 2014. The licensee initiated simulator deficiency report DR 14-0155 to ensure that the simulator was modified to reflect the plant alarm response that was experienced during the event. This simulator modification was completed on January 15, 2015.

The licensee initiated simulator deficiency report DR 15-0124 to ensure that the simulator is modified as needed following resolution of the condition that is being evaluated under CR-RBS-2015-08800.

The inspectors determined that the licensee adequately prioritized the corrective actions with consideration of the risk significance of the simulator fidelity issues and regulatory compliance. This included appropriate actions to address the Notice of Violation and restore compliance (see Section 02.03.e below).

c. Determine whether a schedule has been established for implementing and completing the corrective actions.

The inspectors determined that the licensee established a schedule for implementing and completing the corrective actions. The inspectors identified, as of the date of completion of on-site inspection, that all except one corrective action for the simulator fidelity issue were completed. This final action was associated with the failure to provide the correct alarm response for a loss of an RPS motor generator set as discussed in Section 02.03.b. The inspectors concluded that a schedule had been established for completing this action.

d. Determine whether quantitative or qualitative measures of success have been developed for determining the effectiveness of the corrective actions to prevent recurrence.

The inspectors determined that the licensee had developed an effectiveness review plan that included the method, attributes, acceptance criteria, and schedule for effectiveness reviews of the corrective actions to prevent recurrence (CAPRs). The team reviewed this plan as captured in the corrective action program as RLO-2015-0188. The CAPRs were revisions to Procedure OSP-0022 and the R-DAD-TQ series procedures.

Additionally, RLO-2015-0188 included a collective effectiveness review of corrective actions per Procedure EN-LI-118, Attachment 9.13, Root and Apparent Cause Effectiveness Review. The measures for determining effectiveness are:

(1) All transient snapshot assessments and associated alarm data are submitted to training within the required timeframe and evaluated for implementation;
(2) All simulator change documents (design changes, modification, simulator deficiency reports) are clearly documented with reference to specific design data within the simulator design basis documentation supporting the subject change. All rejected, or on-hold, changes shall also include the reason for the rejection or on-hold status and document the corrective action program action requesting operational data; and
(3) All corrective actions/CAPRs are effective in correcting the root cause and contributing cause.

The inspectors concluded that adequate measures of success had been developed for determining the effectiveness of the corrective actions to prevent recurrence.

e. Determine whether the corrective actions planned or taken adequately address the Notice of Violation that was the basis for the supplemental inspection.

The NRC issued a Notice of Violation to the licensee on September 10, 2015, for the failure to maintain the simulator (NRC Inspection Report 05000458/2015009, ADAMS ML15253A352). Specifically, the licensee failed to maintain the simulator consistent with the reference unit response for normal and transient conditions by failing to:

(1) correctly model leakage flow rates across the FRVs;
(2) provide the correct alarm response for loss of an RPS motor generator set; and
(3) correctly model the behavior of the startup feedwater regulating valve controller. During the inspection, the inspectors determined that the licensee restored compliance by modifying the simulator to accurately model the reference unit.

The completed actions corrected the specific simulator issues associated with the Notice of Violation. These actions were documented in Simulator Deficiency Reports DRs 14-0155, 15-0002, and 15-0013. The inspectors conducted a detailed review of these corrective actions, including a demonstration of simulator operation under necessary simulator setup conditions to allow an effectiveness evaluation. The inspectors noted several weaknesses in simulator deficiency report resolution as listed below:

  • Simulator Deficiency Report DR 14-0155 was initiated to change the simulator model to ensure the alarm response for loss of an RPS motor generator set was consistent with the reference units alarm response. The inspectors noted that the simulator modification documented in DR 14-0155, while addressing the problem, did not ensure modeling was entirely accurate. Operators observed that certain alarms did clear at some time interval after the RPS motor generator set was lost in the reference unit, but before power was restored to RPS. The inspectors identified that the simulator did not accurately model the timeframe from when the alarms cleared in the reference unit until RPS power was restored. The licensee entered this deficiency into their corrective action program as Simulator Deficiency Report DR 16-0074.
  • Simulator Deficiency Report DR 15-0002 was initiated to change the simulator model variable to ensure the FRV leakage was consistent with reference unit leakage during the December 25, 2014, reactor scram. The licensee performed FRV maintenance during the forced outage completed in March 2016, which reduced the FRV leakage in the reference unit. The licensee initiated DR 15-0029 to adjust the simulator model variables to establish new FRV leakage consistent with the reference unit following the forced outage. The inspectors observed a simulator demonstration and identified that the FRV leakage was not consistent with reference unit leakage documented in DR 15-0029. The inspectors identified the simulator FRV leakage was approximately 15 percent higher than the reported reference unit FRV leakage. The licensee entered this deficiency into their corrective action program as Condition Report CR-RBS-2016-02322 and Simulator Deficiency Reports DRs 16-0069 and 16-0070.
  • Simulator Deficiency Report DR 15-0064 was initiated to create new remote functions to allow instructors to adjust FRV leakage for training purposes. The functions indicate the instructor is setting in a desired leakage value in units of pounds-mass per hour (lbm/hr). The inspectors reviewed the model coding and determined the function is actually adjusting percent valve open position. While the remote function variable labeling would be transparent to the operators in a simulator scenario, it could mislead a simulator instructor. The licensee entered this deficiency into their corrective action program as Condition Report CR-RBS-2016-02319 and Simulator Deficiency Report DR 16-0068.

These observations indicate:

(1) a potential need for additional rigor in the simulator modification process to ensure thorough validation of any simulator modifications by another software engineer; and
(2) the need for additional management oversight to ensure sufficient, detailed documentation of simulator deficiency report closeout testing such that the approval authority can assess the adequacy of testing and test results. In response to the observations, the licensee explained that the quality of simulator deficiency report documentation has improved, and provided examples of more recent simulator deficiency reports, where they have implemented the revised process that requires documentation of positive and negative testing. The licensee entered the observations into the corrective action program as Condition Reports CR-RBS-2016-02415 and CR-RBS-2016-02414.

f. Findings

No findings were identified.

02.04 Evaluation of IMC 0305 Criteria For Treatment Of Old Design Issues The licensee did not request credit for self-identification of an old design issue; therefore, the risk-significant issue was not evaluated against the IMC 0305 criteria for treatment of an old design issue.

4OA6 Meetings, Including Exit

Exit Meeting Summary

On April 13, 2016, the inspectors presented the inspection results to Mr. W. Maguire, Site Vice President, and other members of the licensee staff. The licensee acknowledged the issues presented. The licensee confirmed that any proprietary information reviewed by the inspectors had been returned or destroyed.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

M. Chase, Director Regulatory and Performance Improvement
J. Clark, Regulatory Assurance Manager
G. Dempsey, Simulator Specialist
K. Huffstatler, Senior Licensing Engineer
E. Frey, Simulator Specialist
G. Krause, Training Superintended
P. Lucky, Performance Improvement Manager
W. Maguire, Site Vice President
P. OConner, Training Manager
J. Reynolds, Operations Manager
H. Tesfay, Simulator Specialist
D. Looney, Simulator Specialist

NRC Personnel

J. Sowa, Senior Resident Inspector
B. Parks, Acting Resident Inspector

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Discussed

05000458/2015009-05 VIO Failure of the Plant-Reference Simulator to Demonstrate Expected Plant Response (Section 4OA4)

LIST OF DOCUMENTS REVIEWED