ML24170A847

From kanterella
Revision as of 17:50, 24 November 2024 by StriderTol (talk | contribs) (StriderTol Bot change)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
Supporting Statement - Hptf
ML24170A847
Person / Time
Issue date: 06/18/2024
From: Watkins H
NRC/RES/DRA/HFRB
To:
References
Download: ML24170A847 (1)


Text

SUPPORTING STATEMENT FOR Human Performance Test Facility (HPTF)

(3150-XXXX)

NEW Abstract A. JUSTIFICATION

1. Need For the Collection of Information Much of the basis for current NRC Human Factors Engineering (HFE) guidance comes from data from research conducted in other domains (e.g., aviation, defense),

qualitative data from operational experience in nuclear power plants, and a limited amount from empirical studies in a nuclear environment. The former Office of New Reactors (NRO) issued a user need for the Office of Nuclear Regulatory Research (RES) to update its human factors (HF) review guidance with regards to emerging technologies (User Need NRO-2012-007) and more recently the Office of Nuclear Regulatory Regulation (NRR) issued a follow-on user need with the same purpose (User Need NRR-2019-008).

In the spring of 2012, the NRC sponsored a project to procure a low-cost simulator to empirically measure and study human performance aspects of control room operations to address the human performance concerns related to current as well as new and advanced control room designs and operations. Using this simulator, the Human Factors and Reliability Branch (HFRB) in the Office of Nuclear Regulatory Research (RES) Division of Risk Assessment (DRA) began a program of research known as the NRC Human Performance Test Facility (HPTF) to collect empirical human performance data with the purpose of measuring and ultimately better understanding the various cognitive and physical elements that support safe control room operation.

With the Commissions issue of SRM-M061020, directing the NRC to improve the state-of-the-art of human reliability analysis (HRA) and later the issue of SRM-M090204b to ensure that our data programs support these HRA initiatives, the HPTF was established with the dual HF and HRA data missions. The intent of the HPTF is to design experiments that balance domain realism and laboratory control sufficiently to collect systematic, yet meaningful, human performance data related to execution of common nuclear main control room (MCR) tasks. Recognizing the essential role of data to our HF and HRA programs, the HPTF aims to capture data from both novice and operational populations and conduct research specifically targeted to the nuclear domain.

In order to ensure that our laboratory is meeting the research needs discussed above, the HPTF seeks to obtain OMB approval of a generic clearance to collect empirical data to measure and contribute to our understanding of cognitive and physical elements supporting interactions with state-of-the-art technology and safe control room operation.

2. Agency Use and Practical Utility of Information The NRC Human Performance Test Facility (HPTF) was established to collect empirical human performance data to measure and better understand the cognitive and physical elements supporting interactions with technology and safe control room operation. The HPTF seeks generic clearance from OMB to collect data to address gaps in human factors (HF) and human reliability analysis (HRA) research in the nuclear domain. The HPTF enables the NRC to conduct responsive research to support the informational needs of our users (e.g., NRR HFE technical reviewers and HRA analysts). For example, these HF and HRA data are essential to ensure that our HFE guidance documents and HRA methods support the review and evaluation of state-of-the-art HF programs (as required by 10 Code of Federal Regulations (CFR) 50.34(f)(2)(iii)).

The HPTF will only submit a collection for approval under this generic clearance if it meets the following concerns:

The collection will contribute to our understanding of the various cognitive and physical elements that facilitate interactions with state-of-the-art technology and/or support safe control room operation, The collections are voluntary, The collections are low burden for the agency and the respondents based on estimated burden hours and number of respondents, Any collection is targeted to the solicitation of responses from NRC staff and, Information collected is anonymous and cannot be traced back to the respondent.

The types of collections this generic clearance covers includes but are not limited to:

Cognitive laboratory studies, such as those used to characterize system perceptions or assess usability and performance outcomes of system, and Online quantitative and/or qualitative surveys (e.g., assessing perceptions of technology or situation awareness and performance-based metrics), and In-person observation testing (e.g., website or software usability tests).

3. Reduction of Burden Through Information Technology Respondents will submit the requested information using fillable-fillable forms and/or computer-readable formatted forms. It is estimated that approximately 90% of the potential responses are filed electronically. In conjunction with the Government Paperwork Elimination Act (GPEA), HPTF collection of information will primarily

involve the use of electronic techniques such as web-based forms that will be shared, accessed, and completed online. This web-based method of collection and record management will also significantly minimize the likelihood of errors that may arise as the result of data transcription and transferring. Paper-based forms will not be used for data collection. Anticipated online tools to be used for information collection include survey platforms such as Microsoft Forms or Qualtrics.

4. Effort to Identify Duplication and Use Similar Information No sources of similar information are available. There is no duplication of requirements. Much of the basis for current NRC Human Factors Engineering (HFE) guidance comes from data from research conducted in other domains (e.g., aviation, defense), qualitative data from operational experience in nuclear power plants, and a limited amount from empirical studies in a nuclear environment. This research is required to address this dearth in the literature.
5. Effort to Reduce Small Business Burden Not applicable.
6. Consequences to Federal Program or Policy Activities if the Collection Is Not Conducted or Is Conducted Less Frequently If the collection is not conducted, HPTF will be unable to collect empirical human data with the purpose of measuring and ultimately better understanding the various cognitive and physical elements that support safe control room operation.

Additionally, NRC will be unable to produce research contributions in accordance with the Commissions issuances of SRM-M061020 and SRM-M090204b, which direct agency staff to improve the state-of-the-art of human reliability analysis (HRA) and data programs. The proposed schedule for collecting data is the minimum frequency necessary to ensure staff will continue to conduct research in a manner that will facilitate continuous monitoring, evaluation, and updates to HRA methodologies and license review guidance to stay on par with the state-of-the-art technology advancements being proposed for implementation in current and future nuclear power plant control rooms.

7. Circumstances Which Justify Variation from OMB Guidelines Not applicable.
8. Consultations Outside the NRC Opportunity for public comment on the information collection requirements for this clearance package has been published in the Federal Register.
9. Payment or Gift to Respondents Not applicable.
10. Confidentiality of Information Confidential and proprietary information is protected in accordance with NRC regulations at 10 CFR 9.17(a) and 10 CFR 2.390(b). However, no information normally considered confidential or proprietary is requested from respondents.

The only PII associated with any of the information collected will be part of the respondent scheduling. That information will never be stored or accessed from any of the machines where the research activities are conducted. It is not technologically possible to access the scheduling information on the data collection systems as the NRC policy is that the simulator and all associated computers are on a standalone network that is not connected to the NRC network or the internet more broadly.

Therefore, it is not technologically or practically feasible to associate any study data with any respondent PII. Additionally, any PII associated with this information collection used for scheduling purposes would be on the NRC network and would be covered by the NRCs existing SORN.

11. Justification for Sensitive Questions Not applicable
12. Estimated Burden and Burden Hour Cost TABLE 1: ESTIMATED BURDEN FOR DURATION OF GENERIC CLEARANCE (3 YEARS)

Requirement Description No. of Respondents Responses per Respondent Number of Responses Burden Hours per

Response

Total Burden Hours Cost at

$300 per hour Participate in 60-minute (Year 1)

Read informational letter Answer multiple choice questions regarding perceptions of system Read task instructions Practice task Complete task, such as interact with system interface (e.g., locate items on a set of digital panels)

Complete post experiment survey regarding interaction with system Debriefing and opportunity to ask 50 1

50 1

50

$15,000

questions to experimenter Participate in 60-minute study (Year 1)

Read informational letter Answer multiple choice questions regarding perceptions of system Read task instructions Practice task Complete task, such as interact with system interface (e.g., locate items on a set of digital panels)

Complete post experiment survey regarding interaction with system Debriefing and opportunity to ask questions to experimenter 50 1

50 1

50

$15,000 Participate in 60-minute study (Year 2)

Read informational letter Answer multiple choice questions regarding perceptions of system Read task instructions Practice task Complete task, such as interact with system interface (e.g., locate items on a set of digital panels)

Complete post experiment survey regarding interaction with system Debriefing and opportunity to ask questions to experimenter 50 1

50 1

50

$15,000 Participate in 60-minute study (Year 2)

Read informational letter Answer multiple choice questions regarding perceptions of system Read task instructions Practice task Complete task, such as interact with system interface (e.g., locate items on a set of digital panels)

Complete post experiment survey regarding interaction with system 50 1

50 1

50

$15,000

Debriefing and opportunity to ask questions to experimenter Participate in 60-minute study (Year 3)

Read informational letter Answer multiple choice questions regarding perceptions of system Read task instructions Practice task Complete task, such as interact with system interface (e.g., locate items on a set of digital panels)

Complete post experiment survey regarding interaction with system Debriefing and opportunity to ask questions to experimenter 50 1

50 1

50

$15,000 Participate in 30-minute study (Year 3)

Read informational letter Answer multiple choice questions regarding perceptions of system Read task instructions Practice task Complete task, such as interact with system interface (e.g., locate items on a set of digital panels)

Complete post experiment survey regarding interaction with system Debriefing and opportunity to ask questions to experimenter 50 1

50 1

50

$15,000 TOTAL 300 6

300 6

300

$90,000 It is estimated that the collection of information will occur twice annually. Estimates provided in Table 1 represent costs for the 3-year duration of collection covered by this generic clearance. The $300 hourly rate used in the burden estimates is based on the Nuclear Regulatory Commissions fee for hourly rates as noted in 10 CFR 170.20 Average cost per professional staff-hour. For more information on the basis of this rate, see the Revision of Fee Schedules, Fee Recovery for Fiscal Year 2023 (88 FR 39120, June 15, 2023).

13. Estimate of Other Additional Costs There are no additional costs.
14. Estimated Annualized Cost to the Federal Government No costs beyond normal labor costs for staff are anticipated.
15. Reasons for Change in Burden or Cost This is a new request for a generic information collection.
16. Publication for Statistical Use NRC plans on publishing a report summarizing findings. Studies will be completed within 1 to 3 years of approval by OMB. Data will be collected biannually, with each collection lasting no longer than 6 weeks, and the final report is expected to be published within one year after a data collection has been completed. Results will be made available in the form of Research Information Letters (RILs) and office/agency seminars and may be further shared as presentations at conferences and/or peer reviewed publications in scientific journals.
17. Reason for Not Displaying the Expiration Date Not applicable. Expiration date will be included in web-based form for collection of information.
18. Exceptions to the Certification Statement There are no exceptions to the certification statement.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Universe and respondent selection Sample The potential respondent universe consists of US NRC Staff (defined as full-time permanent staff and management level personnel, licensed plant operators, summer interns, and co-operative education students). Respondents may be selected to cover a broad sample or to include specific characteristics (i.e., individual differences such as operator experience) in order to facilitate research aims.

Recruitment method Participants will be recruited from agency announcements and emails. Participation is voluntary. It is expected that around 10% of all individuals receiving the announcement/email will respond to the recruitment message and complete the survey. While this response rate is lower than ideal, many methods are implemented in order to reduce survey dropout and non-response and to maximize the likelihood that individuals will participate in and complete in the survey. These methods are included below in section B3.

2. Procedures for collecting the information Statistical basis for sample estimation Statistical power analysis (using the G*Power software tool) will be conducted to determine the sample size required the number of participants required based on planned statistical testing such as F-tests (i.e., ANOVAs) or t-tests. This step will maximize the likelihood of detecting a significant effect with the desired level of power. Usability research standards will also inform minimum sample selection for generating meaningful results when the addition of qualitative data is collected. Once the pre-determined sample size is met, sample recruitment and data collection will be concluded. This will ensure that actual burden reflects the estimated burden detailed in section A12.

Access to specialized sample Given the lack of existing data due to domain specificity, it is often necessary for the HPTF to collect data from individuals with familiarity, current experience, or prior experience with nuclear power plants. Therefore, it is often crucial to ensure that our participants are from samples of individuals within the nuclear profession (i.e., NRC staff).

Data collection cycles Data collection cycles are anticipated to occur twice annually, with the purpose of measuring and understanding various cognitive and performance elements to address limitations and gaps in the current literature as well as emerging and urgent concerns as informed by changes to state-of-the-art technologies. This data

collection cycle is determined based on the ongoing HFE User Needs (i.e., User Need NRO-2012-00, User Need NRR-2019-008, and User Need NRR-2019-008) and based on the research goals established in the 5-Year Timeline (Development and Maintenance of Human Factors Engineering Review Guidance, Competencies, and Capabilities) which are jointly coordinated between the NRCs Office of Research and the Office of Nuclear Reactor Regulation. Each of these determining factors have been developed by NRC management and staff based upon their knowledge of state-of-the art HFE practices and principals, their experiences developing and using existing guidelines, and their knowledge of emerging HFE issues as they relate to human performance concerns regarding current, new, and advanced control room designs and operations.

Following standard OMB requirements, the HPTF will submit an individual request for each specific data collection activity under this generic clearance. Each request will include the individual instrument(s), a justification specific to the individual information collection, and any supplementary documents.

3. Methods to maximize response rates and to deal with statistical issues of non-response It is expected that around 10% of all individuals receiving the announcement/email will respond to the recruitment message and complete the survey. While this response rate is lower than ideal, many methods are implemented to reduce survey dropout and non-response, and to maximize the likelihood that individuals will participate in and complete in the survey. First, studies will be kept at the minimum possible length to increase the appeal of participating and minimize the burden of time required to complete the survey. Studies will be piloted to examine for participant fatigue and identify opportunities to streamline. Second, additional announcements and emails will be issued to follow-up with nonrespondents and maximize the number of individuals that complete the survey. Announcement emails will be drafted and tested among peers to ensure that wording is concise, informative, and engaging in tone. Finally, surveys will be formatted in a manner that reduces the need for scrolling. That is, when participants click to access the survey, they will not be required to scroll throughout a single page to access all questions in the survey. Additional questions will be placed on additional pages, which will follow the same format. This will minimize perceived workload of completing the survey and maximize the likelihood that respondents will complete the survey in full.
4. Tests or procedures Data collection instruments and research plans are reviewed by NRC personnel and often external research collaborators (e.g., Idaho National Labs, Brookhaven National Laboratory, and the University of Central Florida) to evaluate effectiveness of measurements and protocols for achieving research objectives. All experimental procedures and measures are developed and selected based on meeting an

established criteria (detailed in NUREG/CR-7190) that includes psychometric properties such as validity, reliability, sensitivity, diagnostic, and intrusiveness. Pilot testing is conducted with no more than 9 individuals to evaluate survey flow, instruction and measurement clarity, and completion time. Measures used will be representative of commonly used human factors questionnaires and metrics such as those detailed in NUREG/CR-7190 for assessing workload, situation awareness, and teamwork. Other measures will be used to provide insights as to how technologies following human-system interface design standards facilitate ease of use, conform to human factors engineering design guidelines, and how integrated systems (hardware design, software design, task procedures, and participant individual differences) support task outcomes. Evaluations such as these are discussed in NUREG-0711.

5. Contacts for Statistical Aspects and Data Collection Kelly Dickerson, Human Factors Psychologist Telephone: 301-415-3917 United States Nuclear Regulatory Commission Office of Research, Division of Risk Analysis, Human Factors and Reliability Branch