ML22032A002
| ML22032A002 | |
| Person / Time | |
|---|---|
| Site: | Turkey Point  |
| Issue date: | 02/02/2022 |
| From: | Marshall M NRC/NRR/DORL/LPL1 |
| To: | Moul D Florida Power & Light Co |
| Marshall M, 301-415-2871 | |
| References | |
| EPID L-72021-LRM-000 | |
| Download: ML22032A002 (45) | |
Text
Considerations for the potential use of a Multi-Stage Validation approach to support to nuclear power plant control room modifications Discussion of concepts and NRC licensing considerations Turkey Point pre-submittal meeting February 2, 2022
Outline:
- Background
- NUREG-0711 guidance on validation
- Turkey Point LAR considerations
- Multi-Stage Validation (MSV)
- Background
- Discussion of MSV
- Considering early-stage results from MSV testing
- Licensing Considerations for MSV
- Takeaways and discussion with licensee 2
=
Background===
3
- NUREG-0711, Human Factors Engineering Program Review Model
- Used by NRC staff to review the HFE aspects of licensing applications.
- Verifies that the applicants HFE program incorporates HFE practices and guidelines accepted by the staff, as described within the twelve HFE program elements 5
- Definition of validation:
- The set of activities to determine whether a system can accomplish its intended use, goals and objectives in the particular operational environment.
- In accordance with NUREG-0711, the NRC reviews an applicants integrated system validation (ISV) when reviewing a license amendment request (LAR)
- This review verifies that the applicant has validated, using performance-based tests, that the integrated system design supports the safe operation of the plant 7
Turkey Point LAR considerations
- Based on previous discussions with FPL regarding the scope of the planned control room modifications, there are expected to be significant human factors engineering (HFE) considerations associated with the proposed control room modifications at Turkey Point.
- The license amendment request (LAR) is expected to be submitted for review under the Alternative Review Process (ARP), discussed in the NRC digital Instrumentation and controls interim staff guidance (DI&C-ISG-06).
- As discussed in Section B.1.4 of DI&C-ISG-06, HFE is a review area outside the scope of the ARP interim staff guidance.
8
Turkey Point LAR considerations
- As discussed in DI&C-ISG-06, for digital I&C equipment modifications that involve HFE considerations, an NRC safety evaluation is expected to be performed in accordance with:
- NRC Standard Review Plan (SRP), Chapter 18, Human Factors Engineering
- NUREG-0711, Human Factors Engineering Program Review Model
- NUREG-1764, Guidance for the Review of Changes to Human Actions.
9
Turkey Point LAR considerations
- The NRC is considering whether ISV results for the planned Turkey Point modification will be available in time to support development of the NRC safety evaluation.
- This consideration is based on the timeline discussed at previous meetings for the design/implementation phases of the proposed modification 10
Turkey Point LAR considerations
- Moving forward, we will be looking for more information from FPL (at this or a future discussion) regarding the following:
- Updated scheduling information for HFE design and implementation activities, specifically the timing of planned ISV testing and reporting of results
- Alternative approaches the licensee is considering, if it is indeed the case that ISV test results are not expected to be provided within the timeframe of development for the NRCs safety evaluation 11
Turkey Point LAR considerations
- One approach that the NRC staff sees as being potentially feasible is the use of early-stage validation testing results obtained through an approach based on Multi-Stage Validation (MSV) 12
Multi-Stage Validation 13
Background on MSV
- Multi-stage validation (MSV) is an approach to enhancing confidence in validation results and conclusions.
- NEA technical report published in 2019
- IEEE standard published in 2021
- MSV entails a staged approach to validation testing, where results are obtained at stages throughout the design process 14
NEA Report No. 7466 Published by the Organisation for Economic Cooperation & Development Nuclear Energy Agency (NEA)
November 2019 https://read.oecd.org/10.1787/
fed221d4-en 15
NEA Report No. 7466
- Issued following NEA workshops on human factors validation of nuclear power plant control room designs and modifications held in 2015 and 2018
- Three objectives of the 2018 workshop were:
- Gaining common understanding and alignment on the defining and desirable characteristics of a multi-stage validation
- Identifying and discussing methods for conducting staged validations that optimize the building of cumulative evidence
- Identifying best practices for creating the MSV portfolio/safety case 16
17 IEEE Standard 2411-2021
- Recently-issued standard with state-of-the-art guidance on conducting validation activities
- Formalizes much of the discussion contained in NEA Report No. 7466
- This standard is Not officially endorsed by the NRC, to date; however, NRC staff were heavily involved in the development of the guidelines discussed throughout.
18
Discussion of MSV
- MSV entails a staged approach to validation testing, where results are obtained at stages throughout the design process.
- ISV testing represents the final stage within this approach
- At a conceptual level, MSV refers to the general notion of successive, coordinated validation efforts performed at multiple points/periods during the development of a control room design or design modification.
19
Discussion of MSV Defining characteristics of MSV:
- 1. An MSV is conducted as a series of validation activities, each with its own objectives, methods and results.
- 2. Each validation activity included within an MSV is designed to provide information that can be used as part of the basis for determining whether a system can accomplish its intended use, goals, and objectives.
- 3. Individual validation activities are conducted and grouped in time as stages that allow meaningful aggregation, summation, or comparison of results - both within and across stages - to support interim or final validation conclusions.
20
Discussion of MSV Guidelines for effective MSV testing (discussed in IEEE 2411-2021):
a) Validations are conducted from early (conceptual) to detailed (operational) stages of design development and operations.
b) The subjects of validation comprising an MSV include design concepts (e.g., operations, automation), system elements (e.g., subsystem designs) and the integrated design.
c) Results from each validation stage contribute to an accumulated body of evidence for validation of the final design.
d) Design changes made subsequent to a stage of validation are addressed through testing or analysis in the subsequent stage(s).
e) At each stage, validation methods, controls and rigor are commensurate with the intended use of the associated results and findings. (Annex A of IEEE 2411-2021 provides examples.)
f) Validation testing of design elements that are novel, complex, or critical to safety is initiated early in the design process and confirmed in integrated testing.
21
Discussion of MSV Example of a potential division of MSV stages
- 1. Concept Design Stage Validation testing, with
- 2. Subsystem Design Stage defined objectives and
- 3. Integrated System Design Stage outcomes, is performed at each stage
- 4. Deployment/Operations Stage Note that this is an example - The NEA and IEEE guidance do not place prescriptions or limitations on the number of stages or the specific stages to be used 22
Considering early-stage results
- Early-stage MSV tests can provide an opportunity to:
- confirm that design changes implemented since prior evaluations are effective and do not introduce new problems
- produce evidence addressing various high-level issues identified, which are related to the HSI system design
- identify remaining problems and opportunities to improve the HSI design 23
Considering early-stage results
- The use of early-stage MSV testing does not eliminate the need for testing the integrated system (i.e., via ISV testing)
- MSV subsumes ISV in a way that can improve the efficiency and effectiveness of ISV.
- The purpose of ISV is to show that the integrated system meets performance requirements and supports the plants safe operation.
So if ISV is still a necessary part of the process, then why are we discussing MSV as a potential approach?
24
Potential Licensing Considerations for MSV 25
Licensing considerations for MSV
- MSV was primarily developed for the purpose for enhancing confidence in validation results and conclusions by incorporating validation throughout the design and implementation process.
- Recognizing the constraint that ISV testing can place on the scheduling for significant/complex control room modifications, the NRC staff sees the potential to leverage early-stage MSV results to obtain the validation information needed to support the NRCs determination that a modified design still provides reasonable assurance for safe plant operation.
26
Licensing considerations for MSV
- It is conceivable that, with sufficient information, the NRC could reach a determination regarding adequate assurance of safety for a design based on early-stage MSV results
- The NRC could consider these early-stages results -
along with ISV implementation planning information provided by the applicant - but without the need to review the final ISV results 27
Considerations from IEEE 2411-2021
- Given the substantial cost of validation and implications of validation failure, there should be a sound technical basis for expecting the system to pass before validation testing is performed.
Grounds for such expectation may be given, in part, by the following:
- The use of proven design products where applicable
- The prior validation of innovative portions of the overall system 28
Considerations from IEEE 2411-2021
- Test methods should be considered and controlled in a way to ensure creditable results, for example:
- Validation testbeds should be of sufficient scope and fidelity to support the validation objectives for the applicable stage of validation testing.
- A sufficient variety of test scenarios should be considered to represent the conceivable scope of normal and emergency operations.
- Different sets of test scenarios may be justified as representative (i.e.,
having sufficient variety). However, if a test set cannot be claimed to be representative, then it is not acceptable as the basis for a validation conclusion.
29
Considerations from IEEE 2411-2021 Takeaway: If early-stage MSV results are used to support an application, then the following:
- design considerations,
- scope of early-stage testing,
- controls implemented while conducting testing,
- early-stage test results, and
- resolution of issues identified during early-stage testing should provide creditable support for the expectation that there will not be significant outstanding issues (HEDs) identified during eventual ISV testing 30
Considering MSV in conjunction with existing NRC guidance for reviews 31
32 NUREG-0711 ISV considerations
- NRC staff should review ISV testing and results for all modifications that may:
- 1) change personnel tasks;
- 2) change tasks demands, such as changing the tasks dynamics, complexity, or workload; or,
- 3) interact with or affect human system interfaces (HSIs) and procedures in ways that may degrade performance.
33
When MSV could be appropriate
- Based on the NUREG-0711 guidelines, NRC staff review of early-stage MSV test results (in lieu of final ISV results) may be justifiable in cases where:
- The licensee can demonstrate that the nature of operator tasks have not been significantly changed.
- Early-stage MSV testing results indicate that any changes to the dynamics, complexity, or workload associated with an affected task will not have an adverse effect on operator performance.
- Early-stage MSV testing results indicate that changes to HSIs and/or procedures will not degrade operator performance.
34
35 From SRP Chapter 18:
- In the case of operating reactors, the ISV must be complete prior to implementation of the proposed action or program.
- In the rare cases when this cannot be done, such as when the simulator configuration change is not yet complete, acceptance or approval may be based on a smaller scale preliminary ISV and a license condition that states that the full-scale ISV will be completed and provided to the NRC staff by a specific date.
Note that, although the license condition would require a report of final ISV results, these results would not necessarily be subject to NRC audit as part of the LAR process.
If the submittal relies on preliminary validation results, those results alone would have to be sufficient for the NRC to make its safety determination for the LAR, without reliance on future validation testing results.
The NRC may, however, elect to conduct follow-up inspections of final ISV results and resolution of any identified HEDs, but these activities would be conducted separately from the LAR process.
36
From SRP Chapter 18, Attachment A:
- Preliminary validation is a recommended practice for licensees implementing complex modifications and applicants for design certifications.
- The preliminary validation results should provide high confidence that the performance time criteria will be met in later stages of testing (e.g., the ISV).
- The preliminary validation should provide independent confirmation of the validity of estimates for task completion times determined during prior stages of analysis.
- The preliminary validation should be rigorous and conducted by operators, system technical experts, and human factors experts.
- Preliminary validation results should be such that there is high confidence that the time required for manual operator actions will satisfy the success criteria for the integrated system validation.
37
38 NUREG-1764
- Entails a four-step process for evaluating human actions (HAs) associated with an amendment request
- Provides guidance for determining the appropriate level of HFE review of changes to HAs that constitute new or modified actions, or involve modified task demands
- Plant modifications and associated HAs are categorized into regions of high, medium, and low risk (similar to Regulatory Guide 1.174).
- Appropriate level of graded HFE review (Level I, Level II, or Level III) is determined by the level of risk associated with the affected HA, along with the consideration of certain qualitative factors.
39
- A Level-I review (highest level) calls for an in-depth NRC staff review of the applicants verification and validation program, including a review of ISV testing and results.
- A Level-II review calls for verification that the licensee has demonstrated that the HAs can be successfully accomplished with the modified HSI, procedures, and training.
- Does not explicitly call for a review of ISV results.
40
- Level-II human action verification criteria:
41
NUREG-1764 Qualitative factors considered:
- Personnel Functions and Tasks If an applicant can demonstrate that
- Operating Experience qualitative factors are adequately
- New Actions considered and controlled, the NRC may
- Change in Automation be able to consider a reduction in the
- Change in Tasks level of HFE assessment required
- Change in Performance Context
- Design Support for Task Performance If an HA were initially screened as
- Change in Human-System Interfaces (HSIs) warranting a Level-I, based on the initial
- Change in Procedures risk estimates, but qualitative
- Change in Training considerations derived from design
- Performance Shaping Factors considerations and creditable early-stage
- Changes in Teamwork MSV results justified reduction a Level-II
- Changes in Skill Level of Individuals review, this could potentially support an Performing the Action NRC review based on early-stage MSV
- Change in Communication Demands results alone.
- Change in Environmental Conditions 42
Key Takeaways 43
- Based on existing NRC guidance, along with recent standard developments, NRC staff may be able to make a safety determination regarding HFE considerations prior to the completion of ISV testing.
- The applicant would need to demonstrate (through initial stages of MSV testing or other means) that their design adequately supports safe plant operation (i.e., operator performance is not significantly degraded).
- For significant control room modifications, regardless of the possibility that the NRC may be able to make a safety determination without ISV testing results, ISV testing should still be completed and documented (as discussed in SRP, Chapter 18, and NUREG-0711).
- NRC staff will still conduct a review of ISV implementation plans as part of their review, in accordance with NUREG-0711 guidance.
44
Considerations for the potential use of a Multi-Stage Validation approach to support to nuclear power plant control room modifications Discussion of concepts and NRC licensing considerations Turkey Point pre-submittal meeting February 2, 2022