ML19007A360

Lessons Learned from the IRT Surveys and Focus Groups

ML19007A360
Person / Time
Issue date:	01/30/2019
From:	Michael Orenak Plant Licensing Branch II
To:
Orenak M, 415-3229
References
Download: ML19007A360 (5)

Text

Summary of Lessons Learned From IRT Surveys and Focus Groups The RIDM Phase 1 report recommended implementing the proposed IRT approach on licensing actions during a trial period from July through October 2018 to obtain feedback and capture lessons learned. Because reviewing and issuing typical licensing actions are often a year-long process, the team recognized that these licensing actions would not be completed during the trial period and acknowledged the results would be a snap-shot of the process, but would still yield useful lessons regarding the IRT process. In addition, a license amendment request (LAR) recently completed prior to the RIDM initiative was added to the list of case studies for feedback because it considered risk insights and had a team consistent with an IRT. About 35 staff members participated in implementing the IRT process during the trial period. After the trial period ended in October 2018, Team A sent online survey questions to staff to solicit feedback and recommendations from the IRT participants.

From the analysis of the survey responses, the survey revealed an overall positive experience with the IRT, although some effort at the beginning of the process is necessary to ensure the quality and timeliness of the final product. This observation is consistent with the focused group interviews with the participants of the four pilot licensing actions. The following comments were received from the focus group interviews:

• The IRT provided cross-training opportunities between the traditional deterministic reviewers and the PRA analyst to appreciate each others insights.

• SThe development of staff guidance to define scope and depth of review using risk insights would help the staff.

• Work management guidance may be needed for the risk analyst to charge time for deterministic licensing actions, especially if subsequently the team determines the risk analyst is not needed.

• The IRT should be considered for all complex submittals - even without a risk analyst. The IRT defined the scope and responsibility of the final safety evaluation product for the cross functional team in the beginning. The IRT members met occasionally to review together vs.

the traditional silo fashions. As a result, the individual branch inputs to DORL were more effective and efficient.

• TUsing a master safety evaluation (SE) file to receive draft inputs directly from individual technical reviewers, without going through branch chief, may remove the quality control and accountability opportunities. As a result, the project manager may require significant effort enduring package development to mitigate the deficiency of the draft SE input.

The IRT process was revised based on lessons learned based on the focus group interviews and surveys. The following are additional details of the survey and focused group interview results.

Survey Results The online survey contained the following questions:

1. Did the DORL project manager (PM) encounter any difficulty in determining whether a submittal was Type 1, Type 2, or Type 3?

2. Did the team have difficulties with determining whether to apply an IRT approach?

3. Were the prescribed meetings appropriate for identifying the staffing needs and review scope assignments?

4. Was the DRA checklist useful for determining the appropriate level of DRA engagement and how PRA insights can be used?

5. Was the IRT checklist useful for determining whether an IRT should be used?

6. Were there any difficulties with using a consolidated RAI approach (e.g., timeliness of each branchs RAI development)?

7. Did the review team use available IT tools (e.g., ADAMS, SharePoint, Office 365) to develop consolidated products? If yes, please describe any pros & cons.

8. Did the team encounter difficulties in determining whether a technical or DRA branch would lead the technical coordination of the consolidated RAI and SE development?

9. Were there any difficulties in determining how risk insights could be used to inform the regulatory decisions that had to be made?

10. Did you find the SE template language useful for determining how risk insights could be used to inform regulatory decisions?

11. Were there any difficulties with applying the consolidated RAI or SE concurrence processes?

12. Were there any difficulties in meeting RPS milestone dates or metrics as a result of the new process (not as a result of a complex review or licensee performance)?

13. Please provide any other suggestions or feedback.

A limited (five) number of licensing actions was selected for the short trial period, most of which were still ongoing. Of the estimated 35 NRR staff members, 20 who participated in the trial licensing actions responded to the online survey. Thus, the result may not be statistically significant for conclusions. There were many dont know, N/A, did not apply, did not get to that yet, and didnt use responses to the survey questions. Because of the small sample size and the many dont know, etc., responses, the staff identified only those questions where there was a significant trend in the response, which were Questions 2 and 10. For Question 2, most of the staff responded that the team had no difficulties with determining whether to apply an IRT approach. For Question 10, most of the staff responded that they found the SE template language useful for determining how risk insights could be used to inform regulatory decisions.

There were 14 written responses to Question 13 on other suggestions. The following are the general comments:

• The integrated review process can improve the quality of staff products.

• Staff guidance is needed on how to perform a RIDM Types 1 and 2 licensing review.

• The IRT process requires more work and the staff is already busy.

• The IRT process is too prescriptive and should provide flexibility, especially related to meetings.

• The IRT should not be limited to RIDM and may not include a risk analyst.

• Risk insights could inform the scope, complexity, and priority of staff review.

Though the responses may not be statistically significant, the survey revealed an overall positive experience with the IRT, although some effort is necessary to ensure the quality and timeliness of the final product.

Focus Group Interview Results Focus groups were held for four projects to engage staff in dialogue to obtain feedback and comments pertaining to the IRT process. The four projects were selected to cover the different types of submittals (i.e., Type 1, 2 or 3) and reviews that used other aspects of the IRT process such as the preparation of an integrated SE.

1. Type 1 Requests for FSAR and Surveillance Requirement Changes These licensing actions were Type 1 deterministic reviews, however the technical reviewers thought risk insights might help in the conduct of the review and invited a risk analyst from DRA to be part of the IRT. The risk analyst attended one or two meetings and discussed how risk insights could be used and provided a separate training session to explain the Standardized Plant Analysis Risk (SPAR) model. The risk analyst agreed with the technical reviewers that the license amendment request could be improved through the use of risk from generic qualitative insights. The risk analyst did not identify any other obvious risks that the licensee did not consider. At that point, the team decided no additional information was needed from the risk analyst. These was a good approach to involve a risk analyst in a Type 1 review.

The team suggested that (1) the IRT should be used for all submittals - risk or not, with or without a risk analyst, (2) staff guidance could help define scope and depth of review using risk insights, and (3) guidance may be needed for the risk analyst to charge time if subsequently the team determines the risk analyst is not needed.

2. Type 2 Emergency Amendment for an Emergency Diesel Generator The emergency amendment was a Type 2 review because the licensee provided risk information, but not to the extent of Regulatory Guide (RG) 1.174. An IRT with a risk analyst was formed from the beginning and the IRT worked well because the risk insights facilitated the staff conclusion to approve the emergency amendment. This was a good IRT approach for a Type 2 review that pre-dated the trial period in two aspects:

A. Included risk insight in review - While this was not a risk-informed LAR, the licensee did provide risk insights related to the proposed Technical Specification Completion Time. The licensee indicated that the NRC staff has previously reviewed the technical adequacy of the plants Probabilistic Risk Assessment (PRA) models. The PRA models were previously reviewed to the extent needed to support previous risk-informed LARs. Because this LAR was not risk-informed, the PRA models used to derive risk insights in this LAR were not reviewed by the staff to determine their technical acceptability to support this LAR. As a result, the staff did not rely on the numerical results provided by the licensee. However, the staff considered the licensee-provided risk insights to aid in the deterministic review of the proposed change. The staff also performed an independent assessment using the NRCs SPAR Models to evaluate the risk contribution from internal events, high winds, and seismic hazards. The licensee-provided risk insights and the NRC SPAR model insights and results both supported the engineering conclusions associated with the

appropriateness of the licensees proposed compensatory actions. The currently available risk insights and results did not challenge the engineering conclusion that the proposed change maintains defense-in-depth.

B. Integrated Review Team - Because of the urgency of the review of this multi-disciplined (electrical, mechanical, technical specification, risk, resident inspector, and Office of General Counsel (OGC)) LAR, an integrated review team was organized to review different aspects of the LAR concurrently. The team met at a kickoff meeting to align on interdependencies, the SE layout and responsible sections, and schedule. A simplified project management critical path chart was used to assign scope and responsibilities, as well as daily status updates, given the extremely short timeframe. The cross functional team met occasionally to review together vs. the traditional silo fashions. As a result, individual branch inputs were easier to integrate into the amendment package, resulting in approval of the emergency LAR before the Technical Specification Completion Time expired and the dual unit needed to be shutdown.

The follow up exigent request was a risk-informed (i.e., RG 1.174) LAR. A similar integrated review team and project management approach were used while DRA had the lead responsibility. The Division of Engineering electrical and mechanical branches provided the key deterministic evaluation (e.g. station blackout and common mode failure) to support DRAs risk-informed review. The team met and conducted interactive reviews as needed, including resolving OGCs comments on the draft SE in a team meeting with the responsible attorney. (Note: The licensee withdrew the exigent amendment request within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of the needed approval when the failed emergency diesel generators operability was restored.)

3. Type 2 Expedited Amendment to Extend Surveillance This Type 2 LAR had an IRT working with a risk analyst. The licensee performed a qualitative risk assessment to compare the risk level associated with certain alternative courses of action. Because this LAR was not a RG 1.174 submittal, the staff did not review the method used by the licensee to derive risk insights. The qualitative risk assessment was not used to support a risk-informed decision by the staff. However, the staff considered the insights provided, judged that they were reasonable, and determined that they did not challenge the conclusion that the proposed change maintains defense-in-depth.

This LAR was an effective application of a Type 2 review even though a formal risk evaluation was not performed to directly support a risk-informed decision. It provided insights to support the conclusion that the license proposed change maintained defense-in-depth that can be documented in the SE.

4. Type 3 Integrated Leak Rate Test This LAR was a routine Type 3 request that followed RG 1.174, and the PM opted to use an integrated SE approach from the IRT. The PM prepared an integrated SE outline in SharePoint, identifying specific sections for certain IRT reviewers to provide SE input, and negotiated with the reviewers to agree to the SE outline. The reviewers provided their SE input directly to SharePoint and the PM was able to assemble the SE without much duplication.

During the review of the SE, however, it was determined that technical branch chiefs should review their branchs individual inputs before they are assembled into the amendment package to provide assurances of the quality incoming evaluations.