ML17356A055

NRR E-mail Capture - Request to Make NEI Close-out of Facts & Observations Procedure (Appendix X to NEI 05-04, 07-12, and 12-13) Report Publicly Available

ML17356A055
Person / Time
Issue date:	12/20/2017
From:	Rosenberg S NRC/NRR/DRA
To:	Ed Miller Special Projects and Process Branch
References
Download: ML17356A055 (11)

Text

NRR-DMPSPEm Resource From: Rosenberg, Stacey Sent: Wednesday, December 20, 2017 1:15 PM To: Miller, Ed Cc: Dinsmore, Stephen

Subject:

Request to make F&O observations report publicly available Attachments: ML17265A812.pdf Hi Ed, Attached is the report on NRC staff observations of two on-site implementations of the NEI Close-out of Facts

& Observations procedure (Appendix X to NEI 05-04, 07-12, and 12-13). The document was originally marked as non-public, but, does not contain any sensitive information that would preclude it being publically available. Please make it publically available to support its inclusion in a public meeting summary where its contents were mentioned.

Thank you, Stacey Stacey Rosenberg, Chief PRA Licensing Branch A Division of Risk Assessment Office of Nuclear Reactor Regulation 301-415-2357 1

Hearing Identifier: NRR_DMPS Email Number: 68 Mail Envelope Properties (DM5PR0901MB245600B23F99116494273449950C0)

Subject:

Request to make F&O observations report publicly available Sent Date: 12/20/2017 1:15:19 PM Received Date: 12/20/2017 1:15:20 PM From: Rosenberg, Stacey Created By: Stacey.Rosenberg@nrc.gov Recipients:

"Dinsmore, Stephen" <Stephen.Dinsmore@nrc.gov>

Tracking Status: None "Miller, Ed" <Ed.Miller@nrc.gov>

Tracking Status: None Post Office: DM5PR0901MB2456.namprd09.prod.outlook.com Files Size Date & Time MESSAGE 646 12/20/2017 1:15:20 PM ML17265A812.pdf 107279 Options Priority: Standard Return Notification: No Reply Requested: No Sensitivity: Normal Expiration Date:

Recipients Received:

December 1, 2017 MEMORANDUM TO: Joseph G. Giitter, Director Division of Risk Assessment Office of Nuclear Reactor Regulation FROM: Stacey L. Rosenberg, Chief /RA/

PRA Licensing Branch A Division of Risk Assessment Office of Nuclear Reactor Regulation

SUBJECT:

NUCLEAR REGULATORY COMMISSION REPORT ON OBSERVATIONS OF IMPLEMENTATION OF THE INDUSTRY INDEPENDENT ASSESSMENT TEAM CLOSE-OUT OF FACTS AND OBSERVATIONS AFTER APPENDIX X ISSUED Regulatory Guide 1.200 (RG), An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment (PRA) Results for Risk-Informed Activities, describes one acceptable approach for determining whether the technical adequacy of the PRA can be used in regulatory decision-making for light-water reactors. RG 1.200 endorses, with clarifications, technical requirements described in the American Society of Mechanical Engineers (ASME) and the American Nuclear Society (ANS) ASME/ANS RA-Sa-2009, Standard for Probabilistic Risk Assessment for Nuclear Power Plant Applications (ASME/ANS PRA Standard).

Section 1-6 of the ASME/ANS PRA Standard provides requirements for a peer review of a PRA.

The industry peer review guidance in Nuclear Energy Institute (NEI) 05-04, NEI 07-12 and NEI 12-13 indicates that the peer review assessment is done against the technical requirements for Capability Category (CC) II in the ASME/ANS PRA Standard. The documentation of differences or deficiencies that do not allow a CC II to be assigned are generally labeled facts and observation (F&Os) in the industry peer review guidance documents.

By letter dated February 21, 2017 (Agencywide Documents Access and Management System Accession Number Package: ML17086A450), the NEI, submitted to the Nuclear Regulatory Commission (NRC) Appendix X to NEI 05-04, NEI 07-12 and NEI 12-13, Close-out of Facts and Observations (F&Os). By letter dated May 3, 2017 (ML17079A427), the NRC accepted the process described in Appendix X. In the acceptance letter, NRC stated that [i]n order for the NRC to consider the F&Os closed so that they need not be provided in submissions of future risk-informed licensing applications, the licensee should adhere to the guidance in Appendix X in its entirety. The letter also clarified that additional observation of Appendix X F&O closure reviews, and audits to support licensing actions may be performed, two of which are reported in this memo. The observations and audits are to provide continued monitoring and oversight of PRA acceptability.

CONTACT: Stephen C. Dinsmore, NRR/DRA 301-415-8482

J. Giitter The NRC observed the on-site implementation of this process on July 18-19, 2017, August 23-25, 2017, and one Webinar on August 31, 2017. The on-site observations were performed by different teams of NRC staff and one contractor. Several NRC team participants and the contractor observed the webinar. The two reports summarizing the observations are attached to this memorandum.

The observations confirmed that the teams were continuing to do a comprehensive evaluation about whether the licensee had properly implemented the reported resolution of each F&O into its PRA. However, as discussed in the attached reports, both the observations concluded that the review teams and the licensees are not fully applying vital steps in Appendix X. Specifically:

• Neither the licensee nor the independent assessment (IA) team described, discussed, or documented, whether each finding constitutes a PRA upgrade, maintenance update, or other, as defined in the ASME/ANS PRA (although one licensee added this during the on-site review);

• There was only a few instances observed of any, review [of] the system request (SR) to ensure that the aspects of the underlying SR that were previously not met, or met at CC I, are now met, or met at CC II;

• The July IA review included a remote review that did not include a web [] connection and therefore did not provide opportunities for effective remote interactions envisioned by the NRCs acceptance of the Appendix X guidance. In contrast, the August webinar did provide the opportunity to discuss and agree on the documents which were simultaneously viewed by all participants illustrating that the webinar is much more useful than a simple teleconference;

• During NRC review of the participants in the licensees analyses versus the IA team reviewers, it was not clear how the criteria for independence was determined and applied; and

• The August IA review included F&Os on supporting requirements from most of the different ASME/ANS PRA Standards internal events technical elements but was performed in a short time by a total of two reviewers. The ASME/ANS PRA Standard indicates that more than two reviewers would be expected when most of the technical elements are included in the review. This expectation is reflected in the Appendix X guidance.

Based on the two observations of the IA reviews performed after the final guidance was issued, the NRC concludes that the licensees and IA review teams are evaluating the implementation of proposed F&O resolutions, but are not fully implementing the guidance. This conclusion may result in more frequent audits of F&O closure reports and some requests for additional information to licensees referencing the F&O closure process to provide confidence that all steps in the guidance have been completed.

Enclosure:

Observation Trip Report

ML17265A812 OFFICE NRR/DRA/APLA NRR/DRA/APLA NRR/DRA/APLA NRR/DRA/APLA NRR/DRA/APLA NAME SDinsmore ADriver JEvans MReisifard SRosenberg DATE 09/25/2017 10/12/2017 10/03/2017 10/05/2017 12/01/2017 PRA Facts and Observations Technical Review Meeting Observation Trip Report NextEra Energy, July 2017 NRC Observation team:

Adrienne Driver, Reliability and Risk Analyst Jonathan Evans, Reliability and Risk Analyst Mehdi Reisi-Fard, Reliability and Risk Analyst Date: July 18-19, 2017 Place: NextEra Energy Headquarters, Juno Beach, FL The Nuclear Regulatory Commission (NRC) staff observed a licensees process for review and closure of Facts and Observations (F&Os) on July 18 and 19, 2017 at Juno Beach, FL. NextEra Energy hosted the independent assessment team for a few weeks to review the closure of F&Os for Duane Arnold Energy (12 IE/IF and 89 Fire F&Os), Point Beach (42 Fire F&Os), Plant St Lucie (66 IE/IF and 36 Fire F&Os), Plant Turkey Point (61 IE/IF and 61 Fire F&Os), and Seabrook Station (31 IE/IF F&Os). Six reviewers participated from Enercon (four reviewers participated remotely) and five reviewers from NextEra (one reviewer participated remotely).

Enercon provided the lead facilitator of the independent assessment team for closure of F&Os for Point Beach and St. Lucie that were observed by the NRC staff.

In the morning of July 18, 2017 (the first day of NRC staff observation), some independent assessment team members reviewed Point Beach F&O closures. The Point Beach and Duane Arnold independent assessments for F&O closure began the previous week on through the week of July 17, 2017 when the NRC observers were present. The St. Lucie independent assessment began the week of July 17, 2017. In the afternoon of July 18th, the team conducted a consensus meeting for approximately an hour (one reviewer participated remotely) to review about twenty Point Beach findings. The team continued reviewing closures in the afternoon for Point Beach and St Lucie. The team held a meeting with the licensee and NRC observers at the end of the day to discuss the progress. The NRC staff provided two comments pertaining to the independence of team members and evaluation of upgrades. These issues are discussed later in this report.

On July 19th, the independent review team members continued reviewing closures for Point Beach and St. Lucie. Two consensus meetings were held with remote participation from reviewers from Turkey Point, Point Beach and Duane Arnold. The remaining Point Beach F&Os with the exception of one were reviewed and discussed at the July 19th consensus meetings. At the end of the day a meeting was held between the licensee, independent review team members and the NRC observers. The NRC observers reiterated the potential issue with independence of the team members based on the discovery of new observations on the second day. The licensee briefly described their process for determining the independence of team members and stated that the process will be documented.

The following is a summary of the staffs observations. These observations do not reflect a final assessment by the staff and may be followed-up with upon review of the final report if used for a Licensing Action.

Enclosure

• Observation #1: Compared to other F&O closure assessments observed by NRC staff (ML17095A252), the two independent reviewers performed their individual assessment of findings and conducted consensus meetings without significant discussions. Although other reviewers were present in the room or on the phone during the consensus meetings, only those two reviewers who had been assigned the specific finding seemed to know about the finding and contributed in the consensus meeting discussions.

Earlier, the team lead stated that at least four or five reviewers have been involved in the consensus meetings.

• Observation #2: The NRC staff noted that the host utility did not provide a written assessment and justification of whether each finding constitutes a PRA upgrade, maintenance update, or other, as defined in the ASME/ANS PRA Standard. The staff communicated this issue with the licensee and the independent review team. Also, the NRC staff did not observe any discussion with respect to this issue by the independent review team during the reviews or consensus meeting discussions. The staff stated that the final report should include, for each F&O, an assessment of whether the resolution was determined to be a PRA upgrade or maintenance update and the basis for that determination. The team lead stated that the final document will include a discussion of the evaluation for upgrade vs maintenance, but it is not clear to the staff how this evaluation was performed.

• Observation #3: The NRC staff noted that a number of findings were included in the scope of the F&O closure review inappropriately. Such findings included those that had been superseded by subsequent peer-reviews and those that were identified to be open by the licensee. Including such items in the scope indicates lack of the licensees preparation.

• Observation #4: A number of people participated remotely as independent assessment team members. At least in one case, the staff observed that the reviewer who participated in the consensus meeting remotely did not seem to be adequately prepared for the discussions. His evaluations appeared inaccurate, when compared to the evaluations for the same F&O prepared by the other independent reviewer in multiple reviews for closure of F&Os. Overall, it was observed that in-person participation facilitates more efficient engagement during the review of F&Os and in the consensus meetings.

• Observation #5: In reviewing most of the findings, at least one and in some cases both reviewers were from NextEra Energy. The NRC observers asked whether information could be provided to better understand how the independence of the reviewers were established and how roles were defined for each review.

The staff further identified that a NextEra Energy PRA engineer who was assigned to independently review fire PRA findings had signed vendor analyses of some fire analyses. The licensee described that for each unit, there are PRA engineers who have not worked on the PRA model and each individuals independence and qualifications are verified.

• Observation #6: The NRC staff asked whether the team reviewed the SR to ensure that the aspects of the underlying SR that were previously not met, or met at CC I, are now met at CC II. The team considered and discussed the SR in just a few instances. It is not clear to the NRC staff whether the team systematically reviewed all the SRs associated with findings (especially in those cases where a number of SRs were associated with one finding) to evaluate whether the underlying aspects of the finding is met at CC II.

• Observation #7: The NRC staff did observe in one case where the finding identified errors. The team appropriately verified via sampling that the identified errors were fixed throughout the model.

• Observation #8: Independent review team stated that about half of the F&Os were determined to be closed. Also, the independent review team did not determine any resolution that would constitute a PRA upgrade. Finally, the team leader explained that the team has not encountered a situation that warrants documenting a dissenting opinion.

• Observation #9: In review of some F&Os related to the internal flooding PRA the staff identified that the licensee incorporated changes into the analyses, but there was no documented history of revision demonstrating that the document had been updated. For example, the original peer review team performed review of the revision 0 of the flooding analysis. For closure of F&Os the licensee incorporated the changes in revision 0, but did not appropriately revise the document to reflect the update and the individuals who performed the updated work. The staff discussed this observation with the licensee and how this could impede independence in reviews.

PRA Facts and Observations Technical Review Meeting Observation Trip Report Entergy Operations, Inc., August 2017 NRC Observation team:

Stephen Dinsmore, Senior Reliability and Risk Analyst Steven Short, PNNL Contact support to NRC Date: August 23-25, 2017 Place: Entergy Operations, Inc., Headquarters, Jackson, MS Date: August 31, 2017 Place: Teleconference The Nuclear Regulatory Commission (NRC) staff observed a licensees process for review and closure of Facts and Observations (F&Os) on August 23, 24, and 25 at Jackson MS. Energy Operations Inc. hosted the independent assessment team from the firm ENERCON for three days to review the closure of F&Os for the Grand Gulf Nuclear Station (GGNS). The BWROG conducted the peer review of the Grand Gulf PRA in September 2015, and issued the peer review report in February 2016 (Ref. 1). A total of 39 Findings were documented in the peer review report, and all were reported to be resolved by Entergy. There were one or more findings in each of the eight elements of the internal events (ASME/ANS Ra-Sb-2009, Section 2-2) elements. The resolutions to each Finding will be documented in an ENERCON report.

The review started Wednesday (8/23) morning with a brief kick-off meeting in which introductions were made and the objectives, logistics, and schedule of the review meetings were described. There were only two members of the review team, and so both members were tasked with reviewing all 39 Findings and therefore there was no matrix assigning reviewers to the different elements. There did not appear to be an assigned lead reviewer although one of the two reviewers maintained the working copy of the reviewers findings.

The review team and NRC/PNNL observers were located in one conference room; the licensee support team was located in a separate conference room. The PNNL and NRC staff observers were given one laptop computer and one thumb drive with the files that were provided to the review team that included the BWROG peer review report, the licensees dispositions to the F&Os, NEI 05-04/07-12/12-06 Appendix X, the internal/internal flooding events PRA (IE/IFPRA) notebooks for PRA Model Versions 4 and 4a, the CAFTA files for PRA Model 4a (model of record), an assessment of the impact of each Finding (Ref. 6), and the PRA Model Change Request (MCR) database. The PNNL and NRC staff observers reviewed the files as needed to understand and evaluate the review proceedings.

Prior to the end of the day on Wednesday (8/23) and Thursday (8/24), the review team held a consensus meeting on whether the licensees resolutions to those Findings that both team members had completed a review of that day had adequately addressed the Finding and were considered closed or open (the review team did not use the partially closed option in the Appendix X guidance). A consensus meeting was not held on Friday (8/25) because one of the review team members had to leave early (about 9 a.m.) to catch a flight home. First thing on Thursday (8/24) and Friday (8/25) mornings, the review team presented to the licensee the

results of the consensus meeting held at the end of the previous day. In some cases, the review team later changed open Findings to closed Findings based on further changes to the PRA model documentation made by the licensee during the day. One of the two reviewers had to leave on the early on August 25 and did not compete his review of all the F&Os before leaving. On August 31st, the two reviewers participated in a WebEx conference call to complete consensus on the remaining F&Os. The NRC staff and the PNNL contractor observed the conference call.

The following is a summary of the staffs observations. These observations do not reflect a final assessment by the staff and may be followed-up with upon review of the final report if used for a Licensing Action.

• Observation #1: There were only two independent reviewers so each reviewer had to be the lead or the support reviewer for every F&O that was closed. The NRC staff noted that there were SRs to be closed in most of the different elements of the internal events Standard and that, generally, multiple reviewers are used when more than several of the Standards elements have to be addressed. The Team responded that they had decades of experience between them. However, there were occasions when the discussions between the reviewers and/or between the reviewers and the licensee seemed somewhat confusing with respect to precisely what are the details of acceptable methods. For example it was unclear what conclusions were reached with respect to whether drains can be credited in flooding, and the what the generally acceptable process is for combining new plant data, generic data, and previous plant failure parameters in periodic data updates.

• Observation #2: The two reviewers were tasked with evaluating 39 F&Os within 2 1/2 days, with an unknown amount of work done off-site before and after these days. The review appeared rushed compared to previous reviews observed by the staff, with all consensus sessions and discussions with the licensee taking place very quickly, closed F&Os were not discussed and not (yet) closed F&Os discussed in a matter of minutes for each F&O.

• Observation #3: As in previous F&O resolution reviews (four prior to issuing the endorsed version of Appendix X, and two, including this review, after issuing Appendix X), the NRC staff noted that the host utility did not provide a written assessment and justification of whether each finding constitutes a PRA upgrade or maintenance update as defined in the ASME/ANS PRA Standard. The staff communicated this issue with the licensee and the independent review team. The licensee added this discussion for each F&O during the on-site review period. However, the staff did not observe any discussion on this issue by the review team during the reviews or consensus meeting discussions.

The staff noted the endorsed guidance states that the final report should include an assessment of whether the resolution of each F&O was determined to be PRA upgrade or maintenance and the basis for that determination. The team lead stated that the final document will include this discussion, but it is not clear to the staff when and how this evaluation will be performed.

• Observation #4: The observers noted that the F&O resolution report provided to the review team was inconsistent with the peer review report in terms of which F&Os were Findings and in which were Suggestions (e.g., F&O 1-4 was shown as a Suggestion in

the resolution report, but is reported as a Finding in the peer review report). The review team explained that they had noticed this also and that they were using the peer review report as the basis for their Finding closure assessment. This difference appeared to be the source of some confusion between the review team members, but which was resolved during the consensus meetings.

• Observation #5: The observers noted that the F&O resolution report did not contain a complete description of the F&Os as provided in the peer review report (e.g., the section titled Possible Resolution for each F&O in the peer review report was not provided in the F&O resolution report). In some cases, this additional information provided enhanced context for the F&O. Furthermore, the F&O resolution did not, in some cases, address the missing information. However, as noted in the previous bullet, the review team explained that they had noticed this also and that they were using the peer review report as the basis for their Finding closure assessment. This appeared to the observers to be the case.

• Observation #6: The NRC staff asked whether the team reviewed the SRs to ensure that the aspects of the underlying SR that were previously not met, or met at CCI, are now met, or met at CCII. The team response appeared to be, as with previous teams, that a determination that the F&O was satisfactorily closed automatically meant that the SR was a CC II, or met.

• Observation #7: The review team appeared to conclude that all the F&Os were satisfactorily closed. This was difficult for the observers to confirm because the closure discussions with the licensee were fairly rushed (see observation #2 above) and quite a few issues seemed to be resolved based on expected completion of work before the F&O closure report was finished. The NRC observed a final, web based consensus session between the two team members for the last set of F&Os but was not informed about and did not observe any review confirming the final closure of the F&Os, any review of the licensees determination that each F&O was maintenance and not an upgrade, nor any final summarizing presentation to the licensee.

In summary, the observers concluded that the licensee did not fully comply with the endorsed guidance in preparing for the F&O closure audit because the upgrade versus maintenance discussion had not been developed (which was fixed by the licensee during the review period) and because the F&O table provided did not completely characterize each F&O. Similarly, the review team also did not fully comply with the endorsed guidance because they also did not address the upgrade versus maintenance decision, and did not consistently check whether the result of resolving the F&O was a CC II, or met, status. Furthermore, it is not clear that any two reviewers have the depth of experience in every PRA method that is needed for the necessarily rapid review covering all the of Standards internal events elements. Finally, the very short review period led to some very rushed communication between all the participants. The NRC expects to observe additional reviews in the future.