ML19183A160
| ML19183A160 | |
| Person / Time | |
|---|---|
| Site: | 99902036 |
| Issue date: | 09/11/2019 |
| From: | Joseph Holonich NRC/NRR/DLP/PLPB |
| To: | Powers J -NeedNewValue |
| Holonich J, NRR/DLP/PLPB, 415-7297 | |
| References | |
| CAC ME9861, EPID L-2012-TOP-0003 | |
| Download: ML19183A160 (12) | |
Text
September 11, 2019 Mr. James Powers Vice President, Nuclear Island & Business Development Toshiba America Energy Systems Corporation 3735 Glen Lake Drive Suite 200 Charlotte, NC 28208
SUBJECT:
APRIL 2019 ELECTRONIC REGULATORY AUDIT REPORT FOR TOSHIBA LICENSING TOPICAL REPORT FOR TOSHIBA NONREWRITABLE-FIELD PROGRAMMABLE GATE ARRAY-BASED INSTRUMENTATION AND CONTROL SYSTEM FOR SAFETY-RELATED APPLICATION, UTLA 0020P, REVISION 2 (CAC NO. ME9861; EPID: L-2012-TOP-0003)
Dear Mr. Powers:
By letters dated October 9, 2012, March 13, 2013, February 23, 2015, and August 24, 2015 (Agencywide Documents Access and Management System (ADAMS) Accession Nos. ML12292A320, ML13080A206, ML15062A183, and ML15246A174), Toshiba Corporation (Toshiba) submitted its licensing topical report (LTR), Toshiba NRW-FPGA-Based Instrumentation and Control System for Safety Related Applications, and supporting documentation in accordance with Interim Staff Guidance-06 to the U.S. Nuclear Regulatory Commission (NRC) staff for review. The NRC staff is currently reviewing the LTR for use in safety system equipment at nuclear power plants.
On November 9, 2017, Toshiba representatives and the NRC staff discussed the quality of Toshiba documents. The meeting summary is available in ADAMS under Accession No. ML17334A013. Toshiba performed an extensive review of its documents and on June 19, 2018, submitted a report to the NRC (ADAMS Accession No. ML18176A034) summarizing its findings. Toshiba determined that several documents required modifications and by letter dated November 9, 2018, submitted the revised documents to the NRC (ADAMS Accession No. ML18320A026). In addition, Toshiba placed several supplemental documents and reports which were prepared to describe its findings in detail in Toshibas portal.
In April 2019, the NRC staff performed an electronic regulatory audit of the documents placed in Toshibas portal. The NRC staff efforts on the audit will support generation of a determination in the safety evaluation as to whether the power range monitoring system and oscillation power range monitoring unit are acceptable for use in safety related systems in domestic nuclear power plants. The audit report is enclosed.
J. Powers If you any questions or require any additional information, please feel free to contact me at 301-415-7297 or Joseph.Holonich@nrc.gov.
Sincerely,
/RA/
Joseph J. Holonich, Senior Project Manager Licensing Processes Branch Division of Policy and Rulemaking Office of Nuclear Reactor Regulation
Enclosure:
Audit Plan Docket No. 99902036
ML19183A160; *concurred via e-mail NRR-106 OFFICE DPR/PLPB/PM* DPR/PLPB/LA DE/EICB/ABC DPR/PLPB/BC DPR/PLPB/PM NAME JHolonich DHarrison RAlvarado DMorey JHolonich DATE 08/15/2019 08/15/2019 07/15/2019 09/10/2019 09/11/2019 APRIL 2019 ELECTRONIC REGULATORY AUDIT REPORT FOR TOSHIBA LICENSING TOPICAL REPORT FOR TOSHIBA NONRE-WRITABLE-FIELD PROGRAMMABLE GATE ARRAY-BASED INSTRUMENTATION AND CONTROL SYSTEM FOR SAFETY-RELATED APPLICATION UTLA 0020P, REVISION 2 TAC NO. ME9861; EPID: L-2012-TOP-0003
Background
The U.S. Nuclear Regulatory Commission (NRC) staff is currently evaluating Toshibas design and development processes for the power range monitoring (PRM) system and for the oscillation PRM (OPRM) unit. Over the course of its review, the NRC staff identified several areas where the quality of documents has not been at the level expected and consistent with the Toshiba Corporation (Toshiba) quality assurance program. These areas came from several different interactions with Toshiba and the details were discussed on the November 9, 2017, public telephone call. The meeting summary dated December 20, 2017, is available in the Agencywide Documents Access and Management System (ADAMS) under Accession No. ML17334A013.
Toshiba performed an extended review of its documents and submitted a report to the NRC summarizing its findings (ADAMS Accession No. ML18176A034). Toshiba determined that several documents required modifications and by letter dated November 9, 2018, submitted the revised documents to the NRC (ADAMS Accession No. ML18320A026). In addition, Toshiba placed in the Toshiba portal, several supplemental documents and reports which were prepared to describe its findings in detail. As part of its review, in April 2019, the NRC staff performed an electronic regulatory audit of the documents placed in Toshibas portal.
The purpose of the audit was to evaluate the assessment performed by Toshiba and the impact of the revisions to Toshibas documents to address the quality of its documents that support the safety basis of the proposed system. These documents are identified in Toshibas letters submitted on November 9, December 6, and December 27, 2018 (ADAMS Accession Nos. ML18319A305, ML18346A087, and ML19002A131, respectively).
The regulatory audit plan (ADAMS Accession No. ML19080A290) for the audit of the Toshiba documents detailed the plans and expectations for the electronic audit. The audit was in support of the NRC staff review of the Toshiba PRM system discussed in the licensing topical report (LTR). The NRC staff efforts on the audit will support generation of a determination in the safety evaluation (SE) as to whether the PRM system and OPRM unit are acceptable for use in safety related systems in domestic nuclear power plants.
Regulatory Audit Basis The NRC staff has identified the following information that is needed to complete its review, and verify compliance with Title 10 of the Code of Federal Regulations (10 CFR) 50.55(a), Codes and Standards, as applicable, and in conformance with implementing guidance such as Regulatory Guide (RG) 1.169, Configuration Management Plans For Digital Computer Software Used in Safety Systems Of Nuclear Power Plants, 1.171, Software Unit Testing For Digital Computer Software Used In Safety Systems Of Nuclear Power Plants, and RG 1.209, Guidelines For Environmental Qualification of Safety-Related Computer-Based Instrumentation And Control Systems in Nuclear Power Plants.
Audit Activities As part of the audit, the NRC staff reviewed the following documents placed in Toshibas portal during this electronic audit:
- a. FC51-3704-0007, Nuclear Energy Systems and Services Division (NED) Oscillation Power Range Monitoring (OPRM) Verification and Validation Report (VVR)
(Supplemental), Rev. 0 This document evaluates and documents the impacts of the changes made to the verification and validation (V&V) documents for the OPRM unit and describes how these changes relate to the sections of the existing NED VVR.
This document identifies the current Toshiba organization, which changed after the OPRM units information was docketed to the NRC. Toshiba determined that this change does not have any impact on the VVR. Before this reorganization, the Independent Verification and Validation (IV&V) team of the Instrumentation and Control Systems Design and Engineering Department (ICDD) of NED completed the V&V activities for the OPRM. These activities are summarized in the VVR.
With the reorganization, the ICDD of the NED was replaced by the Electrical System Design &
Engineering Department (ESDD). Toshiba Energy Systems & Solutions Corporation including ESDD, Nuclear Instrumentation and Control Systems Department (NICSD), and Power Platform Development Department (PPDD) split off from Toshiba Corporation. This information is included in every document reviewed during this electronic audit. Since this reorganization is not described in the revised LTR submitted by letter dated November 9, 2018. The NRC staff identified a request for additional information (RAI) #1 (ADAMS Accession No. ML19161A066) for Toshiba to provide a clear description of its updated organization.
FC51-3704-0007 also identifies the documents associated with each V&V phase that were impacted by the Toshiba Document Review Report (E2-2018-000531). This included the lifecycle from project planning and concept definition phase to system validation testing phase of the OPRM unit. Toshiba made modifications to the VVRs to include revisions to equipment requirement specification (ERS), unit equipment design specifications (EDSs), module validation specifications (MDSs), and field programmable gate array (FPGA) design specifications.
Toshiba concluded that there were no impacts on the past V&V activities. Toshiba evaluated the impacts on the Software Safety Analysis (SSA) Report (SSAR) and the VVR for the OPRM unit. Toshiba found that conclusions and recommendations described in the docketed SSAR and VVR remain applicable and valid.
- b. FC51-3704-0008, NED OPRM SSAR (Supplemental), Rev. 0 This document provides supplemental information associated with the SSA activities performed for the OPRM unit. The document describes the findings and resolutions after Toshiba performed an assessment of its documents quality. Assessment of certain activities previously performed were necessary because Toshiba updated its EDS for the power range neutron monitor, which includes the OPRM unit, the OPRM unit detailed design specification for the power range neutron monitor, and MDSs and FPGA design specifications.
Toshiba used the new design documents and compared them to the safety analysis performed in each lifecycle phase to determine if the modified design specification had an impact on the evaluation performed and technical conclusions. Toshiba found that the revised documents did not affect the technical evaluations and conclusions reported for each lifecycle phase.
Toshiba concluded that there were no impacts on the safety analysis performed during the system lifecycle activities. Further, the conclusions and recommendations described in FC51-3704-0004, Nuclear Energy Systems and Services Division Software Safety Analysis Report for Safety-Related Oscillation Power Range Monitor, remain applicable and valid.
In addition, the NED SSAR (Supplemental) stated that the risk of using this OPRM in the advanced boiling water reactor (ABWR) plants was considered minimal as long as the recommendations described in the NED SSAR were implemented. These recommendations were also identified in the LTR, in which Toshiba required licensees to follow the users manual for the OPRM unit.
- c. FC51-3704-1119, OPRM Requirements Traceability Matrix (RTM) (Supplemental),
Rev. 0 This document describes the assessment performed on the RTM for the safety-related OPRM, because of the evaluation of Toshiba documents quality. This assessment was performed by the Software Development (SD) team of the NICSD.
The RTM was used to track system and component requirements throughout the system development lifecycle. Thus, modifications made to the EDS for the power range neutron monitor, the OPRM unit detailed design specification for the power range neutron monitor, and module design specifications and FPGA design specifications, needed to be reflected in the RTM, if errors were found.
This document distinguishes between the modules developed for approval in the LTR and those developed only for Japanese domestic plants. Since this information was not included in the LTR, the NRC staff identified RAI #2 to request this information be provided by Toshiba.
Toshiba found that the new revisions for the design documents did not impact the RTMs created and used during the phases of the lifecycle. Toshiba made the necessary changes for consistency and addressed grammatical and typographical errors.
- d. FC51-3704-1120, OPRM VVR (Supplemental), Rev. 0 This document summarizes the assessment that Toshiba performed on the NICSD VVR for OPRM, as part of its self-assessment to evaluate the quality and consistency of its documents.
During this evaluation, Toshiba identified errors on the V&V activities performed and test reports that documented those activities. These included organization changes, editorial errors, and inconsistent descriptions among the documents of the OPRM system, the system design documents, and the revised RTMs.
Toshiba noted that it performed this assessment in accordance to NED V&V Plan and NICSD V&V Plan. These supplemental V&V activities were performed by NICSD.
Based on the results of the self-assessment, Toshiba revised its NED V&V Plan, NICSD V&V Plan, Software Quality Assurance (SQA) Plan, Software Test Plan, Software Configuration Management Plan, Preliminary Technical Evaluation Report (PTER), and Software Validation Test Plan. These revisions did not modify the technical purpose of these documents, and they did not affect the V&V activities performed previously. Using these new revisions, Toshiba evaluated the documents created for each lifecycle phase for the OPRM unit and revised them for consistency with other documents.
Toshiba found that these revisions did not affect the results obtained previously. In addition, Toshiba determined that the Master Configuration List (MCL), and the Project Control Document List (PCDL) needed to be updated to reflect the revisions and updated documents accordingly.
- e. FC51-3704-1121, OPRM RTM Report (Supplemental), Rev. 0 This document describes the assessment performed on the RTM for the safety-related OPRM, because of the evaluation of Toshiba documents quality. This assessment was performed by the IV&V team of the NICSD.
The NICSD SD team prepared the RTMs for the project planning and concept definition phase, requirements definition phase, and the system validation testing phase. The PPDD developed the RTMs for the design phase, implementation and integration phase, and module validation testing phase. The IV&V team was responsible for reviewing and accepting all the RTMs.
Because Toshiba identified and corrected errors in the EDS, SDD, OPRM unit detailed design specifications (Unit DDS), MDSs, and FPGA design specifications of the OPRM system, Toshiba needed to review the RTM to assess the impact of these revisions. The NICSD SD Team issued the RTM for OPRM (Supplemental) (FC51-3704-1119) to summarize these impacts. This supplemental report was reviewed by the IV&V team.
Toshiba found that the new revisions for the design documents did not impact the RTMs created and used during the phases of the system development lifecycle. Toshiba made the necessary changes for consistency and addressed grammatical and typographical errors.
- f. FC51-3704-1484, OPRM SSAR (Supplemental), Rev. 1 This document provides supplemental information associated with the SSA activities performed for the OPRM unit. For this effort, the Software Safety (SS) team of NICSD reviewed all SSARs prepared for each lifecycle phase. The SS team evaluated the impact the revised design documents (e.g., EDS) could have created in the SSARs previously prepared.
Toshiba found that the revisions to its design documents did not impact the results and conclusions obtained previously. Also, Toshiba determined that the SSARs did not need to be revised.
- g. FPG-DRT-C51-0017, NED V&V Final Report (Supplemental), Rev. 0 This document provides supplemental information for the V&V activities performed for the PRM system.
As part of the self-assessment that Toshiba performed to address document quality, Toshiba found that the VVR had editorial errors and descriptions that were inconsistent with other documents (e.g., EQR). As a result, Toshiba revised all documents prepared for the phases of the system development lifecycle. Toshiba prepared this supplemental to describe findings and modifications made, as well as, their impact on the final VVR. In addition, in this document, Toshiba described how it resolved the corrective actions identified during the regulatory audit performed in May 2016.
Toshiba listed the documents modified in this supplemental. The modifications made were to address inconsistencies and grammar errors; and they did not have any impacts on the results reported previously. Thus, Toshiba concluded that no further V&V activities were necessary for the PRM system.
- h. FPG-DRT-C51-0024, NED Hazard Analysis Report (Supplemental), Rev. 0 NED prepared this supplemental information to the Hazard Analysis Reports (HARs) for the PRM system. A HAR was prepared during each lifecycle phase.
This supplemental information covers the hazard analysis activities in the project planning and concept definition phase and the system validation testing phase for which NED was responsible. NICSD prepared a supplemental HAR for the other phases, which are described in FPG-DRT-C51-1002.
Because of the revisions to several design documents, Toshiba modified the ERS for the PRM system to address quality issues. Then Toshiba evaluated whether the revised ERS had any effect on the HARs for the project planning and concept definition phase and the system validation testing phase. NED was responsible for only these two phases of the PRM system lifecycle.
Toshiba concluded that the changes in the ERS did not impact the existing HARs, and they continue to be valid.
- i. FPG-DRT-C51-0027, NED PRM RTM (Supplemental), Rev. 0 This document provides supplemental information for the RTM prepared for the PRM system with the objective of confirming that all system requirements were traced to and from all design and testing documents. This document only covers the lifecycle phases for which NED was responsible, which were the project planning and concept definition phase and the system validation testing phase. NICSD was responsible for the other phases, and they prepared supplemental material FPG-DRT-C51-1004 to ensure the NED findings were properly implemented.
During the development lifecycle of the PRM system, Toshiba used the RTM to trace the system requirements. A RTM report was prepared for each lifecycle phase. Because of the quality assessments, Toshiba revised several design documents (e.g., ERS) for the PRM system. Then, Toshiba evaluated the impact of these changes in the RTM used for the PRM system. This supplemental information summarizes Toshibas findings and resolutions.
The findings included grammatical or syntax errors as well as missing descriptions. However, there were three significant findings, which required modifications to requirements that were not correctly described. These mistakes were only made in the ERS and not in the test procedures and reports. Toshiba modified the ERS and RTM to reflect the correct description and to establish consistency with the design and testing documents for the PRM system. Toshiba evaluated these findings in FPG-DRT-C51-1004, Nuclear Instrumentation & Control Systems Department Requirements Traceability Matrix (Supplemental), which is described below in item m.
In this document, Toshiba also briefly mentions how the failure of the watchdog timer in the RCV module was tested, and how this will be perceived by the APRM module. Since this information was not included in the LTR and it would support our safety evaluation, the NRC staff identified RAI #3.
Toshiba concluded that the changes made to the ERS were necessary for consistency with design documents. These changes were also made to the RTM. These changes did not impact the evaluation and conclusions documented in the System Validation Testing Phase for the PRM system. Also, Toshiba concluded that all requirements identified in the project planning and concept definition phase were traced for the PRM system.
- j. FPG-DRT-C51-1001, NICSD PRM Systems RTM Review Report (Supplemental).
This document provides supplemental information for the RTM prepared by NICSD for the PRM system. This includes the RTMs from the requirements definition phase to the unit/module validation testing phase.
This supplemental information evaluated that the modifications to the ERS were traced throughout the PRM system lifecycle. In this document, Toshiba noted that the Unit EDSs, the MDSs, FPGA design specifications, FPGA test specifications, and unit/module test specifications included grammatical errors and inconsistent descriptions. Toshiba evaluated these issues and summarized its results in this supplemental document. In addition, Toshiba prepared separate supplemental reports for each RTMs prepared for each phase of the PRM systems lifecycle.
Toshiba concluded that the documents modified did not have any impact on the design of the PRM system, and consequently the RTMs prepared for the lifecycle phases for which NICSD was responsible remain valid.
- k. FPG-DRT-C51-1002, NICSD Hazard Analysis Report (Supplemental), Rev. 0 NICSD prepared this document to supplement the NEDs HARs for the PRM system. Toshiba examined any possible impacts of the revisions of the design and testing documents described in E2-2018-000531. This assessment covered the lifecycle from the requirement definition phase through the unit/module validation testing phase of the PRM system. Because of this
review, Toshiba revised the ERS, Unit EDSs, MDSs, and FPGA design specifications of the PRM system. Therefore, the documents using this information (or using the RTM) required updates/revisions.
This NICSD HAR (supplemental) includes the lifecycle from the requirements definition phase to the unit/module validation testing phase. Impacts to the project planning and concept definition phase and system validation testing phase were evaluated by NED and were not part of this document.
Toshiba concluded that the changes in the design documents did not impact the existing HARs, and they continue to be valid. In addition, Toshiba noted that these changes did not have any impact in the PRM system design.
- l. FPG-DRT-C51-1003, PRM VVR (Supplemental 2), Rev. 0 This document describes the supplemental activities on the NICSD VVR that resulted from Toshiba quality assessment as well as Toshibas findings and resolutions.
Both NED and NICSD evaluated their V&V plans to address NRC comments identified during the regulatory audit performed in 2016 and as part of the quality assessment. Toshiba found that modifications were necessary to address inconsistencies and grammatical errors in the activities descriptions. Toshiba determined that these modifications did not affect the results of the V&V activities performed for the PRM system.
As part of the quality assessment reviews, NICSDs IV&V team reviewed the VVRs prepared for the requirements definition phase, design phase, the implementation and integration phase, unit/module validation testing phase, and NICSD final VVR. The IV&V team determined if new activities needed to be performed as part of the quality assessment. These activities were documented in the design verification reports prepared for each phase, which were described in this document. Toshiba found that several documents had inconsistencies and grammatical errors, but these findings did not change the results documented in the reports.
Also, NICSD IV&V team reviewed FPG-DRT-C51-1004, NICSD Requirements Traceability Matrix (Supplemental), and FPG-DRT-C51-1002, Nuclear Instrumentation & Control Systems Department Hazard Analysis Report for PRM System (Supplemental). The IV&V team found them acceptable.
Toshiba concluded that the V&V plans, VVRs, RTM, and hazard analysis were not technically affected by the quality assessment, and they remain valid.
- m. FPG-DRT-C51-1004, NICSD PRM RTM (Supplemental), Rev. 0 This document describes the supplemental information for the RTMs prepare by NICSD SD team as part of the quality assessment.
For this effort, NICSD SD team reviewed the RTM reports created from the requirements definition phase to the unit/module validation testing phase to determine if the requirements from the revised design documents (e.g., ERS) are traceable. This evaluation does not include the RTM for the project planning and concept definition phase and the system validation test phase because these were prepared by NED. These RTMs were reviewed in FPG-DRT-C51-0027, NED RTM Report (Supplemental).
This supplemental information describes Toshibas findings and resolutions to maintain traceability of the revised design documents with the RTMs prepared for the lifecycle phases listed above. This document also describes the design documents revised to address CAR-16-039, which was created during the 2016 regulatory audit. Toshiba made modifications to address editorial mistakes and inconsistencies. Toshiba did not find technical problems during this review, and therefore these modifications did not impact the RTMs.
Toshiba concluded that the modifications and revisions made to the design documents did not impact the PRM system design and were necessary to address issues identified in the CAR-16-039 and issues found during the quality assessment.
- n. MEM-JHA-000071, Confirmation of OPRM Qualification Testing Records, Rev. 0 This document describes NICSD evaluation of the OPRM qualification testing records as part of the quality assessment to determine if inconsistencies with design documents (i.e., EDS), test plans, test procedures, and test records and reports exist. For this effort, NICSD evaluated the system validation test, equipment qualification (EQ) test (environment test, seismic test), and an electromagnetic compatibility (EMC) test.
SVTR FC51-7513-1002 was revised to address inconsistencies with test plans, test specifications, test procedures, and test records.
Toshiba found that there were some differences in the descriptions between the EDS and EMC qualification test plan, the electromagnetic interference (EMI)/radio-frequency interference (RFI) test procedure and test records. Toshiba revised the EDS to correct these inconsistencies, which did not affect the test procedures, records, or test results.
Toshiba also evaluated the acceptance/operability test specification, test plans, test specifications, and test procedures for the setup & check-out test, the operability test, and the prudency test. Toshiba did not find inconsistencies in these documents.
Although inconsistencies were found, Toshiba concluded that these did not affect the results of the EQ/EMC qualification tests for the OPRM and for the SVTR.
- o. QAS-2018-000185, Evaluation among procedure revisions and record for the PRM system, Rev. 0 This document describes the evaluation performed to address inconsistencies between test records and test procedures identified in the qualification tests results for the PRM system and in the Qualification Test Summary Report. Also, this document summarizes the evaluation performed to address grammatical errors and unclear test methods and testing conditions. As a result of its evaluation, Toshiba revised its test procedures to establish test record consistency.
Toshiba concluded that these inconsistencies did not affect the results of the qualification tests for the PRM system.
- p. FPG-DRT-C51-0007, Critical Digital Review (Supplemental), Rev. 0 This document describes the finding and resolutions from the quality assessment performed on the Critical Digital Report (CDR) prepared for the PRM system. For the PRM system, Toshiba prepared the following CDRs:
(1) NRW-FPGA-Based PRM System Qualification Project FPG-DRT-C51-0005, NICSD's CDR Report (2) NRW-FPGA-Based PRM System Qualification Project FPG-DRT-C51-0006, Actels CDR Report (3) NRW-FPGA-Based PRM System Qualification Project FA32-3613-0001, Nuclear Energy Systems and Services Division Updated Critical Digital Review of Actel FPGA Software Tools Toshiba reviewed these documents in conjunction with the revised design documents to determine if the corrections made had any impacts on system design. Toshiba found that there were only grammatical errors, and the revised design documents did not affect these CDRs.
Toshiba concluded that the revisions that resulted from the quality assessment did not affect the CDRs.
- q. E2-2018-001141, Document Review Report (DRR) (Supplemental), Rev. 0 This document describes the supplemental information prepared for E2-2018-000531, Document Review Report, Rev. 3. The DRR describes Toshibas revision policies followed during the quality assessment to evaluate its documents. This DRR (Supplemental) was prepared to address some editorial problems in the Document Revision Policy A. This document includes tables describing the made changes.
Toshiba found that these corrections were only editorial and did not affect the information docketed in the DRR.
Conclusions The NRC staff successfully completed all audit activities outlined in the audit plan. Several Toshiba documents were reviewed to ensure quality requirements were properly addressed.
The NRC staff found that the information in the Toshibas portal was sufficient to assess the quality assessement performed by Toshiba. The NRC staff identified several open items that will require additional RAI questions. These RAI questions were discussed with Toshiba during a telephone call and are identified in a separate letter for Toshiba to provide its responses. In an email dated June 10, 2019, the NRC staff sent the RAI questions to Toshiba (ADAMS Accession No. ML19161A066).
Principal Contributor: Rossnyev Alvarado, NRR/DE/EICB