ML13232A263: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
 
(Created page by program invented by StriderTol)
 
(12 intermediate revisions by the same user not shown)
Line 2: Line 2:
| number = ML13232A263
| number = ML13232A263
| issue date = 10/08/2013
| issue date = 10/08/2013
| title = Diablo Canyon, Units 1 and 2 - Redacted Letter + Enclosure W/O Attachments, Audit Report for 2/11-14/13, Regulatory Audit at the CS Innovations Westinghouse Facility in Scottsdale, Az, to Support Digital Replacement of Process Protection Sy
| title = Redacted Letter + Enclosure W/O Attachments, Audit Report for 2/11-14/13, Regulatory Audit at the CS Innovations Westinghouse Facility in Scottsdale, Az, to Support Digital Replacement of Process Protection System LAR
| author name = Rankin J K
| author name = Rankin J
| author affiliation = NRC/NRR/DORL/LPLIV
| author affiliation = NRC/NRR/DORL/LPLIV
| addressee name = Halpin E D
| addressee name = Halpin E
| addressee affiliation = Pacific Gas & Electric Co
| addressee affiliation = Pacific Gas & Electric Co
| docket = 05000275, 05000323
| docket = 05000275, 05000323
| license number = DPR-080, DPR-082
| license number = DPR-080, DPR-082
| contact person = Rankin J K
| contact person = Rankin J
| case reference number = TAC ME7522, TAC ME7523
| case reference number = TAC ME7522, TAC ME7523
| document type = Letter
| document type = Letter
| page count = 14
| page count = 14
| project = TAC:ME7522, TAC:ME7523
| stage = Other
}}
}}
=Text=
{{#Wiki_filter:OffiCIAL USIi ONLY      PROPRIIiTARY INfORMATION UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 O:tober 8, 2013 Mr. Edward D. Halpin Senior Vice President and Chief Nuclear Officer Pacific Gas and Electric Company Diablo Canyon Power Plant P.O. Box 56, Mail Code 104/6 Avila Beach, CA 93424
==SUBJECT:==
DIABLO CANYON POWER PLANT, UNIT NOS. 1 AND 2 - REGULATORY AUDIT ON FEBRUARY 11-14, 2013, AT THE CS INNOVATIONS!
WESTINGHOUSE FACIUTY IN SCOTTSDALE, ARIZONA, FOR THE DIGITAL UPDATE TO THE PROCESS PROTECTION SYSTEM AMENDMENT (TAC NOS. ME7522 AND ME7523)
==Dear Mr. Halpin:==
By letter dated October 26, 2011, as supplemented by letters dated December 20, 2011, and April 2, April 30, June 6, August 2, September 11, November 27, and December 5,2012, and March 25, April 30, May 9, and May 30,2013 (Agencywide Documents Access and Management System (ADAMS) Accession Nos. ML113070457, ML113610541, ML12094A072, ML12131A513, ML12170A837, ML12222A094, ML12256A308, ML13004A468, ML12342A149, ML13093A311, ML13121A089, ML13130A059, and ML13154A049, respectively), Pacific Gas and Electric (PG&E, the licensee), submitted a license amendment request (LAR) to the U.S.
Nuclear Regulatory Commission (NRC) for the Diablo Canyon Power Plant, Unit Nos. 1 and 2 (DCPP). The proposed LAR would provide a digital replacement of the Process Protection System (PPS) portion of the Reactor Trip System and Engineered Safety Features Actuation System at DCPP.
To support its safety evaluation, the NRC's Instrumentation and Controls Branch conducted an audit at the CS Innovations (CSI)lWestinghouse facilities in Scottsdale, Arizona, from February 11-14, 2013. The purpose of this audit was to determine if the lifecycle processes used, and the outputs of those processes, will result in a PPS for use at DCPP, which will meet regulatory requirements. This audit provided information necessary to complete the NRC staffs evaluation of the proposed Advanced Logic System portion of the DCPP PPS. Enclosed is the report associated with this audit.
As noted in the enclosed audit report, the NRC staff addressed each of the planned audit activities outlined in the audit plan dated February 1, 2013 (ADAMS Accession No. ML13029A667). The audit reviewed multiple requirement threads of the PPS systems for compliance with the DCPP specific planning documents and included multiple interviews with CSllWestinghouse personnel from the Independent Verification and Validation, Design Attachments 1 and 2 to the Enclosure to this letter contain Proprietary Information. Upon separation from Attachments 1 and 2, this letter is DECONTROLLED.
OFfiCIAL USE ONLY        PROPRIETARY INFORMATION
OFFICIAL USE ONLY            PROPRIETARY INFORMATION E. Halpin                                        - 2 Engineering. Quality Control, and Configuration Management groups. Several Open Items were identified and noted in the audit report for future follow-up. Attachment 1. "Detailed Audit Notes," and Attachment 2, "Detailed Requirement Thread Notes." of the Enclosure contain proprietary information and have been withheld from public disclosure.
The cyber security review activities performed in conjunction with this audit are being documented in a separate report that is being written by the NRC's Office of Nuclear Security and Incident Response staff.
If you have any questions. please contact me at 301-415-1530 or via e-mail at Jennivine.Rankin@nrc.gQv.
Sincerely,
                                                \e¥\(~~
                                                ~~~~. Rankin, Project Manager Plant licensing Branch IV Division of Operating Reactor licensing Office of Nuclear Reactor Regulation Docket Nos. 50-275 and 50-323
==Enclosure:==
As stated cc w/Enclosure (no Attachments): Distribution via listserv OFFICIAL USE ONLY          PROPRIETARY INFORM~TION
UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON. D.C. 20555-0001 REPORT OF REGULATORY AUDIT ON FEBRUARY 11-14. 2013, IN SCOTTSDALE AZ OFFICE OF NUCLEAR REACTOR REGULATION INSTRUMENTATION AND CONTROLS BRANCH DIGITAL PROCESS PROTECTION SYSTEM PACIFIC GAS AND ELECTRIC COMPANY DIABLO CANYON POWER PLANT, UNITS 1 AND 2 DOCKET NOS. 50-275 AND 50-323 BACKGROUND The U.S. Nuclear Regulatory Commission (NRC) staff is currently engaged in a review of a replacement for the digital Process Protection System (PPS) at Diablo Canyon Power Plant, Unit Nos. 1 and 2 (DCPP). By letter dated October 26, 2011, Pacific Gas and Electric (PG&E, the licensee) submitted a license amendment request (LAR) to replace the DCPP Eagle 21 PPS with a new digital PPS (Agencywide Documents Access and Management System (ADAMS)
Accession No. ML11307A332). In addition, the licensee supplemented the LAR by letters dated December 20,2011, and April 2, April 30, June 6, August 2, September 11, November 27, and December 5,2012, and March 25, April 30, May 9, and May 30,2013 (ADAMS Accession Nos. ML113070457, ML113610541, ML12094A072, ML12131A513, ML12170A837, ML12222A094, ML12256A308, ML13004A468, ML12342A149, ML13093A311, ML13121A089, ML13130A059, and ML13154A049, respectively). The LAR requested NRC review and approval of the proposed design.
REGULATORY AUDIT BASIS To support its safety evaluation (SE), the NRC Office of Nuclear Reactor Regulation (NRR),
Division of Engineering (DE), Instrumentation and Controls Branch (EICB), conducted an audit at the Westinghouse/CS Innovations (CSI) facility in Scottsdale, Arizona. The purpose of this audit was to determine if the lifecyle processes used, and the outputs of those processes will result in a PPS system for use at DCPP which will meet regulatory requirements. This audit provided information necessary to complete the NRC staff's evaluation of the advanced logic system (ALS) portion of the proposed DCPP PPS. The scope of this audit was previously defined in the associated audit plan that was sent to the licensee on February 1, 2013 (ADAMS Accession No. ML13029A667).
Enclosure
                                                  - 2 AUDIT ACTIVITIES The NRC audit team, consisting of Richard Stattel, Samir Darbali, and Rossnyev Alvarado from EICB, and George Simons, Eric Lee, Stacy Smith and Christopher Chenoweth from the Office of Nuclear Security and Incident Response (NSIR), and Shiattin Makor from Region IV, visited the Westinghouse/CSI facility in Scottsdale, Arizona, from February 11-14, 2013, to perform this audit.
The audit was conducted for the following aspects of the DCPP PPS System Development life cycle:
* System Verification and Validation (V&V) - Verification that the V&V program used for the development of the DCPP ALS portion of the PPS meets the requirements of Institute for Electrical and Electronics Engineers (IEEE) Standard IEEE-1012, "Standard for Software Verification and Validation," and that the V&V program is implemented in a manner which reliably verifies and validates the design outputs of each stage of the design process.
* Configuration Management - Verification that the configuration management system includes the appropriate hardware and logic implementation within the configuration management program, and that the configuration management system is effectively controlling these items.
* Software Quality Assurance - Verification that the System Quality Assurance (SQA) program is effective in controlling the software development process to assure quality of the DCPP PPS application.
* System Safety - Verification that the system safety plans and procedures used during the safety analysis activities were adequate to determine that the logic implementation is safe to be used in the DCPP PPS.
* Cyber Security - Review of activities associated with addressing system and services acquisition controls as set forth in the licensee's NRC-approved Cyber Security Plan, and in accordance with Section 73.54, "Protection of digital computer and communication systems and networks," of Title 10 of the Code of Federal Regulations (10 CFR).
AUDIT
==SUMMARY==
Entrance Meeting (Monday, February 11, 2013)
At the entrance meeting, the audit team provided an overview of the audit plan and objectives for the audit. Facility logistics and a detailed audit schedule were discussed. During this meeting it was decided that NRR would coordinate audit activities with the NSIR audit team by having Samir Darbali and Shiattin Makor, who are working on the Secure Development and Operation Environment (SDOE) evaluation for the PPS, work directly with the NSIR audit team during the week.
                                                -3 Scott Roberts, the Director of Operations, introduced a number of Westinghouse/CSI staff members, including: Joe Basso, Project Manager, Marci Maher, V&V Manager, Brian Studaker, QA Manager, and Bill Irmen, Operations Manager. Marci Maher was asked to provide an overview of the AlS documentation to familiarize the staff with the organization of information to be reviewed during the audit.
At the end of the entrance meeting, CSI staff noted that the current status of the AlS subsystem is that both cores are operating in the prototype system with field programmable gate array (FPGA) version 1 installed. Design phase development is in progress and is proceeding to the FPGA version 2. The Design phase Independent V&V (IV&V) activities have not commenced at the time of this audit, the V&V team was working on performing the IV&V activities on the Requirements phase.
The following sections provide descriptions of the activities performed during this audit. In several areas, proprietary information was reviewed as a part of the audit. Detailed proprietary audit notes for each subject area are provided in Attachment 1 to this report. Attachment 2 provides detailed description of the requirements thread review performed during the audit.
1.0      CSI DCPP PPS Application V&V Program The objective of the IV&V portion of the audit was to confirm that the Westinghouse/CSIIV&V processes are implemented per its documentation, with a focus on record keeping, documentation, and management activities. Because the project has only completed the IV&V activities associated with the Requirements Phase, the audit team could not review all records related to the DCPP PPS replacement project AlS subsystem V& V activities. The audit team was, however, able to review the currently completed examples of AlS logic development.
Marci Maher, the AlS logic IV&V Manager, Secil Karaaslan, IV&V lead for the DCPP PPS project, and Jeff Vance, IV&V team member were the primary Westinghouse/CSI participants for this portion of the audit.
1.1      Independent V& V Organization and Processes The IV&V portion of the audit began with a thorough discussion of the IV&V processes used for the application development and its implementation for the AlS portion of the DCPP PPS. This discussion was consistent with the description provided in 6002-00003, "AlS V&V Plan," and 6116-00003, "Diablo Canyon PPS W Plan," which had previously been reviewed by the NRC staff. The audit team had an extensive question and answer session with Westinghouse IV&V team about the AlS system development processes, IV&V involvement and the IV&V organization.
During this discussion, the IV&V team members explained that they are currently reVising Westinghouse/CSI document 6116-00003, "DCPP PPS W Plan," to include V&Vactivities that were not previously described. Specifically, the IV&V team explained that the current AlS V&V Plan (Rev. 1) did not identify the IV&V team performing risk and hazard analysis required by IEEE Std. 1012. This discrepancy was identified by the NRC staff in the DCPP PPS Open Items Table as item 83. Further, the AlS logic IV&V Manager noted that this has been captured in Corrective Action Program (CAP) ticket #12-1 03-M041.34, inspection summary CAP, dated July 16, 2012, which stated that CSI should perform a software safety analysis to
                                                  - 4 meet the requirements of Branch Technical Position (BTP) 7-14. The ALS Logic IV&V Manager explained that the IV&V team performed a hazard analysis for the ALS system, and that this information is currently documented in Section 4.2 of 6116-0029 failure modes and effects analysis (FMEA).
The NRC staff and the IV&V team discussed the diversity requirements associated with ALS Core logic implementations. The audit team noted that the IV&V staff is knowledgeable of their roles in the DCPP PPS application development.
The IV& V team has completed the V& V activities for the planning phase. They are currently working on the V&V activities for the requirements phase. The audit team reviewed the IV&V Requirement Phase W Summary Report which was available in draft version, during the audit.
The IV&V team explained that for the requirement phase they are comparing the requirements supplied by PG&E (e.g., Functional Requirements Specification (FRS>> to the documents prepared by Westinghouse, WNA-DS-02442-PGE, "Diablo Canyon Units 1 and 2 Process Protection System Replacement Project, ALS System Requirements Specification," to identify discrepancies and/or missing information. The IV&V team has found that not all requirements in PG&E documents match the requirements in WNA-DS-02442-PGE, and the design teams are currently addressing this issue (see descriptions for OnTime' Tickets #4787 and #4800 below).
1.2    Independent V&V Documentation IV&V team members explained how identified problems are being documented and addressed using the corrective action processes. Specifically, the IV&V team explained that if errors, inconsistencies, or anomalies are encountered, the OnTime ' ticket process is used to identify and record them. Attachment 1 to this audit report provides additional information about this ticket process. The NRC staff reviewed the following OnTime 1M tickets that were issued by the IV&V team while performing V&V activities during the requirements phase:
* Ticket #4258 - describes an error in 6116-10201, "ALS PPS ALS-102 FPGA Requirement Specifications." A resolution for this issue was not provided in the ticket. This ticket will remain open until a new version of the document is formally released.
* Ticket #4787 - identifies that many PG&E requirements are not included in the Westinghouse ALS System Requirements Specification, WNA-DS-02442-PGE.
In particular requirements identified in the Interface Requirements Specification (IRS) are not included in the ALS System Requirements Specification. A resolution for this issue was not provided in the ticket. This ticket will remain open until a new version of the document is formally released.
* Ticket #4800 - identifies that the Requirements Traceability Matrix (RTM) does not clearly trace requirements in the FRS and WNA-DS-02442-PGE. In particular, there are requirements in the FRS that are not listed in the RTM. A resolution for this issue was not provided in the ticket. This ticket will remain open until a new version of the document is formally released.
                                              -5 The ALS Logic IV&V Manager explained that Westinghouse Review Action Items are also used to provide comments on draft version of DCPP PPS planning and process documents. Once these documents are formally released, the IV&V team uses the OnTime' ticket process, as described above.
When the ALS DCPP PPS system is released for the IV&V team to verify and validate the Verilog code, the IV&V team will use OnTime' tickets to identify any anomalies and/or inconsistencies. During this discussion, the IV&V team also explained how the logic revision scheme is used to identify the status of ALS products.
Observation: The NRC staff has added an open item to request that a description of the FPGA versions be included in the system management plan. This description should include a discussion of when and to whom products are released. This has been documented as Open Item 87.
1.3    Training Records Review The NRC staff performed a review of the qualification and training records for one of the IV&V engineers assigned to the DCPP PPS project. Note that because the project has only completed the V&V activities for the planning phase, the staff was not able to confirm that all of the requirements for Human Diversity described in Westinghouse Work Instruction 9006-00037, "Human Diversity Management for FPGA Based Development and Test Activities," were being met. The NRC staff conducted a confirmatory review of the activities associated with Westinghouse Work Instruction 9006-00037 pertaining to the subject IV&V Engineer.
The NRC staff also reviewed the work history of this individual to ensure that the requirements of the work instruction were being met. This form is approved by the Functional Manager, Operational Manager, Network Systems Administrator, and the OnTime TM Administrator.
1.4    ALS FPGA Design Process and Documents The ALS Logic IV&V Manager, described the ALS document hierarchy that established the requirements for the ALS platform and the ALS DCPP PPS. This information is provided in CSI document 6002-00000, "ALS Management Plan." The IV&V manager explained how these documents are identified and traced in the Requirements Traceability Matrix (RTM), 6116-0059.
The documents that define the ALS DCPP PPS are:
* PG&E Functional Requirements Specification
* WNA-DS-02442 System Requirement Specification
* 6116-00011 ALS Platform Design Specification
* 6116-10201 FPGA ALS-102 Requirements
* 6116-1 0203 DCPP ALS-102 Core A Design Specification
* 6116-10204 DCPP ALS-1 02 Core B Design Specification Note: The prefix number 6116 is a unique project identifier for the DCPP PPS project.
                                                -6 During this discussion, the NRC staff noted that the ALS platform document hierarchy to be prepared for the DCPP PPS project does not exactly match the documents that would be expected for a typical project. In particular, there was no 6116-10206 FPGA Design Specification was prepared for the DCPP PPS project. This is because FPGA ALS-102 Requirements document 6116-10201 includes this information, making it equivalent to a 6116 10206 document. Further, CSI personnel explained that ALS Platform document 6002-10206 includes the ALS-1 02 design specifications that would have otherwise been included in a 6116 10206 document.
Observation: The NRC staff has added an open item to request a brief explanation about this apparent document number misalignment between the ALS platform and the DCPP PPS project. This has been documented as Open Item 88.
CSI personnel explained that they prepared the RTM using Westinghouse document WNA-DS-02442 to trace PG&E requirements. The IV&V team found that Westinghouse document WNA-DS-02442 does not capture all PG&E requirements (see descriptions for Tickets #4787 and #4800, which currently address these). During the audit, the NRC staff and CSI discussed the fact that the current RTM does not trace requirements down to the core desjgn specifications.
Observation: The NRC staff added two open items to request a description of how all PG&E requirements will be captured in the RTM and an explanation of how the RTM will include traceability to the core design specifications. These have been documented as Open Items 93 and 92, respectively.
2.0    Quality Assurance Program This audit activity addressed the quality assurance program applicable to the ALS platform products and documentation associated with DCPP PPS project.
The Quality Assurance (QA) Lead for DCPP PPS, Brian Studaker, described the QA Westinghouse/CSI program. He explained that the CSI QA program is transitioning from CSI 9000-00000 Quality Assurance Manual to Westinghouse's Quality Management System (QMS). He also explained that Westinghouse 23.20 QA procedure establishes the interface agreement between Westinghouse and CSI, as well as the applicable procedures to be used at the Westinghouse/CSI facilities.
The NRC staff discussed QA activities for the DCPP PPS project with the QA team (B. Studaker, D. Harmon, and C. Bobbitt). The NRC staff confirmed that the quality assurance processes and procedures are subject to the same configuration management, corrective action, and change management activities that apply to ALS platform configuration items.
During this discussion, the QA team explained that they perform project specific audit assessments, reviews and inspections. These activities assess the QA aspects and efforts of the project during each phase of the lifecycle. The results of this audit are documented in the Quality Activity Report (QAR), which is updated after each phase of the lifecycle is completed.
During this NRC audit, the staff reviewed the QAR related to the DCPP PPS project, which are summarized in 6116-00300, "DCPP PPS QA Summary Report," Revision 0, dated October 1,
                                              -7 2012. Section 2 of this document identified all activities performed until 01/31/2013. For each activity this section identifies: date, QAR number, description, person responsible, and status.
If a non-conformance or finding is observed, this item is recorded in the Quality Activity Report and the standard corrective action processes would be followed (i.e. a CAP would be initiated).
The NRC staff reviewed several CAPs during the audit, which are described in Attachment 1.
One of these CAPs was related to implementation of the ALS core diversity. This CAP was initiated to record an issue related to the FPGA development for the PG&E ALS-102 application.
The proposed resolution for this CAP was reviewed by the staff to determine if adequate controls for the build process for the FPGA are being implemented. This CAP is still open and is currently awaiting further analysis.
During the audit, the NRC staff discovered a deficiency in a work instruction used for developing FPGA applications. To address this, Westinghouse/CSI opened a new CAP ticket. provides detailed information about this CAP.
3.0      Configuration Management This audit activity addressed the configuration management (CM) activities applied to the DCPP PPS products and documentation. This included a review of the configuration controls that govern the DCPP PPS application specific documentation and FPGA programming design. The Senior Network Administrator (B. Wheeler), and the Manager for Scottsdale Operations (W. Irmen) gave the NRC staff an overview of the process required to perform controlled changes. Design and planning documents are created and edited in the Westinghouse network computer. After a document is updated, a release record is created which requires digital signatures from the Author, Reviewer, and Approver.
The NRC staff also reviewed the processes used for controlling access, securely transferring files, and maintaining backups for configuration items.
The NRC staff discussed the process used to prevent parallel document checkout with the CM librarian (also the Network Administrator). The staff observed that measures have been implemented to prevent inadvertent parallel changes from being made to system configuration files. However, the NRC staff observed that these measures do not prevent access to the configuration file. These measures allow only the person who first opened the file to make changes to it. Additional users may open the file while it is being edited, but they are not able to apply any changes of their own.
The NRC staff reviewed two OnTime ' tickets related to configuration management of the DCPP PPS replacement project. The NRC staff also met with an FPGA Senior Engineer to discuss the processes used to make configuration changes associated with one of these OnTime ' tickets. The engineer used the Westinghouse computer to access the system requirements and verify the requirements were correct. He then accessed the FPGA development tool which required an additional login. The engineer guided the NRC staff to the specific lines of code and the process for making the appropriate changes to address the issue and close the ticket. The engineer showed the NRC staff the log of the changes made to the FPGA application design.
                                                  - 8 The NRC staff performed a review of Westinghouse/CSI document 6116-00400, "DCPP PPS Configuration Management Report," Rev. O. This document defines and tracks the Project Milestones, Configuration Items, Configuration Status Accounting (CSA) and Baselines for DCPP PPS configuration items. The CSA is comprised of the current revision configuration items in the release database. The CSA is updated periodically to maintain document control throughout the duration of the project. Working copies of the CSA documents are maintained in Westinghouse/CSl's CM tool, Concurrent Version System (CVS).
The DCPP PPS CM Report contains a list of open OnTime ' tickets. The report states that there should be no open tickets at the Manufacturing stage. All related design configuration items are in the Development Stage of the DCPP PPS Life-Cycle.
The NRC staff reviewed Westinghouse/CSI document 6116-00050, "DCPP PPS Configuration Status Accounting." This document provides the Identification Number, Status, Revision, and Date for DCPP PPS configuration items. The CSA also provides the Revision numbers for 3rd party tools used during the DCPP PPS project development, including the FPGA modification tool and CVS Suite.
4.0    Diversity CSI document 6002-00031, "ALS Diversity Analysis," describes the ALS platform's built-in diversity in terms of common-cause programming error considerations and in support of future licensee and application-specific diversity and defense-in-depth analyses. Westinghouse/CSI formalized implementation of the diversity claims by implementing procedures or work instructions to ensure ALS platform design diversity claims are supported. The NRC staff reviewed these work instructions and noted that they were not fully satisfying what was needed to establish the built-in diversity required for the DCPP PPS system. CSI has issued two CAPs to address this (CAP # 13-036-M051 and #13-042-M070).
During this review, the NRC staff noted that the work instructions used for one of the cores does not clearly identify how to implement core diversity. Attachment 1 provides additional information.
Observation: Westinghouse/CSI should consider adding clarification to the work instructions about the use of diverse techniques.
The NRC staff performed a targeted thread audit of the requirement associated with the platform's diversity attributes applicable to the DCPP PPS replacement project. The NRC staff observed how this requirement and design specifications are included in the PPS application.
Two of the Westinghouse/CSI FPGA Developers demonstrated the processes used for implementing diversity in the PPS system. Attachment 1 provides additional information.
4.1    Human Diversity The ALS Diversity Analysis also states that human diversity is another means to ensure diversity in the ALS platform. The NRC staff reviewed the Work Instruction used by ALSlWestinghouse personnel to ensure human diversity is maintained during FPGA development for the DCPP PPS project. Attachment 1 provides additional information.
                                                  -9 5.0      Thread Audit of the DCPP Application Software V&V Program The NRC staff selected specific requirements to audit based on Significance to the NRC staffs safety evaluation for the DCPP PPS replacement project.
The following threads were evaluated during this audit:
* FRS  3.2.1.16 detection of an internal rack failure
* FRS  3.2.1.10 and IRS 1.5.8 - time response
* FRS  3.2.2.7 - trip logic for reactor coolant flow low
* FRS  3.2.1.5.3 PPS - channel in Bypass Detailed notes for these requirements thread review audit activities are provided as . During the thread review, the NRC staff identified several Open Items, which are described in Attachment 2. The identified Open Items should be addressed by Westinghouse/CSI via its corrective action program during the upcoming design and implementation phases of the project. Each of these areas will be further evaluated during the next NRC audit which is to be performed when the PPS design implementation is complete.
6.0      Secure Development Environment This audit activity addressed the secure development environment (SDE) applied to the ALS platform and the DCPP PPS products and design documentation. This included a review of the activities and documentation incorporated by Westinghouse/CSI to prevent the inclusion of errors and unintended functionality.
7.0      ALS Service Unit (ASU) Demonstration The NRC staff was able to observe a demonstration of the ASU functionality while connected to the DCPP PPS prototype. Both the ASU and the DCPP PPS are still in the development stage.
The staff observed the indications and alarms that occur when the test ALS bus (TAB) is connected to the maintenance workstation (MWS) and when a board is pulled from the rack.
The staff also observed how calibration and testing modes are accessed, and how setpoint changes will be made. The staff was informed that a custom cable was being developed to connect the TAB to the ALS-102 board.
8.0      Exit Meeting (Thursday - February 14, 2013)
During the exit meeting, Westinghouse/CSI was provided with a summary of the Open Items identified during the audit. In addition, a list of documents was provided with a request to provide the NRC staff access to support its ongoing review activities. This list is included in Open Item 91.
                                              - 10 CONCLUSION The NRC staff addressed each of the planned audit activities outlined in the audit plan. Several requirements threads were selected and evaluated for compliance with the DCPP specific planning documents. Interviews were conducted with Westinghouse/CSI personnel from the IV&V, Design Engineering, Quality Control, and Configuration Management groups.
The following Open Items were identified during this audit.
87    FPGA versions 1, 2, 3, descriptions were explained to the NRC staff during the ALS audit but these release processes are not captured in the system development plan or system management plan.
88    Please describe why there is a misalignment of document numbers between the platform 6002-xxx01, 6002-xxx06 and application specific documents 6116 10201. For example, please explain why is there no 6116-10206.
89    Ensure that the audit schedule issues (Pennatronics) identified during the cyber security review portion of the ALS audit is resolved prior to issuance of the Diablo PPS safety evaluation. The NRC will be reviewing the responses to the CAP's that Westinghouse has written on this issue to assess if there are any implications on the DCPP PPS system.
90    Once CSI has completed the SDOE evaluation to show conformance to RG 1.152 requirements, the results will have to be docketed.
91    Please provide the NRC access to the following documents via SharePoint:
* Work instruction for Human Diversity Management for FPGA Based Development and Test Activities, Document number 9006-00037, Rev. 0
* ALS Core A FPGA Build Procedure, Document number 9006-00043, Rev. 3
* ALS Core B FPGA Build Procedure, Document number 9006-00071, Rev. 1
* 6116-10203/4 Core A and Core B Design Specifications
* RTM sorted by FRS.
92    The Requirements Traceability Matrix (RTM) does not trace to CSI documents 6116-10203/4 Core A and Core B<Design Specifications. CSI must include this traceability to the RTM once the 6116-1020314 Core A and Core B Design Specifications are finalized.
                                            - 11 93    The RTM for the ALS subsystem was prepared using Westinghouse document WNA-DS-02442 to trace PG&E requirements. The IV&V team found that Westinghouse document WNA-DS-02442 does not capture all PG&E requirements (see descriptions for Tickets #4787 and #4800).
Date: October 8,2013 Attachments:
: 1. Detailed Audit Notes (Proprietary)
: 2. Detailed Requirement Thread Notes (Proprietary)
OFFICIAL USE ONLY          PROPRIETARY INFORMATION E. Halpin                                        - 2 Engineering, Quality Control, and Configuration Management groups. Several Open Items were identified and noted in the audit report for future follow-up. Attachment 1, "Detailed Audit Notes," and Attachment 2, "Detailed Requirement Thread Notes," of the Enclosure contain proprietary information and have been withheld from public disclosure.
The cyber security review activities performed in conjunction with this audit are being documented in a separate report that is being written by the NRC's Office of Nuclear Security and Incident Response staff.
If you have any questions, please contact me at 301-415-1530 or via e-mail at Jennivine.Rankin@nrc.gov.
Sincerely, IRA!
Jennie K. Rankin, Project Manager Plant Licensing Branch IV Division of Operating Reactor Licensing Office of Nuclear Reactor Regulation Docket Nos. 50-275 and 50-323
==Enclosure:==
As stated cc w/Enclosure (no Attachments): Distribution via Listserv DISTRIBUTION:
PUBLIC                                                RidsNrrPMDiabloCanyon Resource LPL4 RlF                                            RidsRgn4MailCenter Resource RidsAcrsAcnw_MailCTR Resource                        RStattel, NRRlDE/EICB RidsNrrDeEicb Resource                              RAlvarado, NRRlDE/EICB RidsNrrDorlLpl4 Resource                            SMakor, RIV/DRS/EB2 RidsNrrLAJBurkhardt Resource                        ELee, NSIR ADAMS Accession Nos.:          ML13232A261 (Ltr + Encl + PI Attachments);
ML13232A263 (Redacted Ltr + Encl w/o Attachments)
OFFICE      NRRlDORULPL4/PM              NRRlDORLlLPL4/LA            NRRlDORULPL4/LA
. NAME        MBartlett                    JRankin                      JBurkhardt DATE        8/19/13                      10/7/13                      8123/13                I OFFICE      NRRlDE/EICB/BC                NRRlDORLlLPL4/BC            NRRlDORLlLPL4/PM NAME        JThorp                        MMarkley                    JRankin DATE        5/17/13                      1018/13                      10/8113              \:
OFFICIAL RECORD COpy OFFICIAL USE ONLY          PROPRIETARY INFORMATION}}

Latest revision as of 02:21, 6 February 2020

Redacted Letter + Enclosure W/O Attachments, Audit Report for 2/11-14/13, Regulatory Audit at the CS Innovations Westinghouse Facility in Scottsdale, Az, to Support Digital Replacement of Process Protection System LAR
ML13232A263
Person / Time
Site: Diablo Canyon  Pacific Gas & Electric icon.png
Issue date: 10/08/2013
From: Jennivine Rankin
Plant Licensing Branch IV
To: Halpin E
Pacific Gas & Electric Co
Rankin J
References
TAC ME7522, TAC ME7523
Download: ML13232A263 (14)


Text

OffiCIAL USIi ONLY PROPRIIiTARY INfORMATION UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 O:tober 8, 2013 Mr. Edward D. Halpin Senior Vice President and Chief Nuclear Officer Pacific Gas and Electric Company Diablo Canyon Power Plant P.O. Box 56, Mail Code 104/6 Avila Beach, CA 93424

SUBJECT:

DIABLO CANYON POWER PLANT, UNIT NOS. 1 AND 2 - REGULATORY AUDIT ON FEBRUARY 11-14, 2013, AT THE CS INNOVATIONS!

WESTINGHOUSE FACIUTY IN SCOTTSDALE, ARIZONA, FOR THE DIGITAL UPDATE TO THE PROCESS PROTECTION SYSTEM AMENDMENT (TAC NOS. ME7522 AND ME7523)

Dear Mr. Halpin:

By letter dated October 26, 2011, as supplemented by letters dated December 20, 2011, and April 2, April 30, June 6, August 2, September 11, November 27, and December 5,2012, and March 25, April 30, May 9, and May 30,2013 (Agencywide Documents Access and Management System (ADAMS) Accession Nos. ML113070457, ML113610541, ML12094A072, ML12131A513, ML12170A837, ML12222A094, ML12256A308, ML13004A468, ML12342A149, ML13093A311, ML13121A089, ML13130A059, and ML13154A049, respectively), Pacific Gas and Electric (PG&E, the licensee), submitted a license amendment request (LAR) to the U.S.

Nuclear Regulatory Commission (NRC) for the Diablo Canyon Power Plant, Unit Nos. 1 and 2 (DCPP). The proposed LAR would provide a digital replacement of the Process Protection System (PPS) portion of the Reactor Trip System and Engineered Safety Features Actuation System at DCPP.

To support its safety evaluation, the NRC's Instrumentation and Controls Branch conducted an audit at the CS Innovations (CSI)lWestinghouse facilities in Scottsdale, Arizona, from February 11-14, 2013. The purpose of this audit was to determine if the lifecycle processes used, and the outputs of those processes, will result in a PPS for use at DCPP, which will meet regulatory requirements. This audit provided information necessary to complete the NRC staffs evaluation of the proposed Advanced Logic System portion of the DCPP PPS. Enclosed is the report associated with this audit.

As noted in the enclosed audit report, the NRC staff addressed each of the planned audit activities outlined in the audit plan dated February 1, 2013 (ADAMS Accession No. ML13029A667). The audit reviewed multiple requirement threads of the PPS systems for compliance with the DCPP specific planning documents and included multiple interviews with CSllWestinghouse personnel from the Independent Verification and Validation, Design Attachments 1 and 2 to the Enclosure to this letter contain Proprietary Information. Upon separation from Attachments 1 and 2, this letter is DECONTROLLED.

OFfiCIAL USE ONLY PROPRIETARY INFORMATION

OFFICIAL USE ONLY PROPRIETARY INFORMATION E. Halpin - 2 Engineering. Quality Control, and Configuration Management groups. Several Open Items were identified and noted in the audit report for future follow-up. Attachment 1. "Detailed Audit Notes," and Attachment 2, "Detailed Requirement Thread Notes." of the Enclosure contain proprietary information and have been withheld from public disclosure.

The cyber security review activities performed in conjunction with this audit are being documented in a separate report that is being written by the NRC's Office of Nuclear Security and Incident Response staff.

If you have any questions. please contact me at 301-415-1530 or via e-mail at Jennivine.Rankin@nrc.gQv.

Sincerely,

\e¥\(~~

~~~~. Rankin, Project Manager Plant licensing Branch IV Division of Operating Reactor licensing Office of Nuclear Reactor Regulation Docket Nos. 50-275 and 50-323

Enclosure:

As stated cc w/Enclosure (no Attachments): Distribution via listserv OFFICIAL USE ONLY PROPRIETARY INFORM~TION

UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON. D.C. 20555-0001 REPORT OF REGULATORY AUDIT ON FEBRUARY 11-14. 2013, IN SCOTTSDALE AZ OFFICE OF NUCLEAR REACTOR REGULATION INSTRUMENTATION AND CONTROLS BRANCH DIGITAL PROCESS PROTECTION SYSTEM PACIFIC GAS AND ELECTRIC COMPANY DIABLO CANYON POWER PLANT, UNITS 1 AND 2 DOCKET NOS. 50-275 AND 50-323 BACKGROUND The U.S. Nuclear Regulatory Commission (NRC) staff is currently engaged in a review of a replacement for the digital Process Protection System (PPS) at Diablo Canyon Power Plant, Unit Nos. 1 and 2 (DCPP). By letter dated October 26, 2011, Pacific Gas and Electric (PG&E, the licensee) submitted a license amendment request (LAR) to replace the DCPP Eagle 21 PPS with a new digital PPS (Agencywide Documents Access and Management System (ADAMS)

Accession No. ML11307A332). In addition, the licensee supplemented the LAR by letters dated December 20,2011, and April 2, April 30, June 6, August 2, September 11, November 27, and December 5,2012, and March 25, April 30, May 9, and May 30,2013 (ADAMS Accession Nos. ML113070457, ML113610541, ML12094A072, ML12131A513, ML12170A837, ML12222A094, ML12256A308, ML13004A468, ML12342A149, ML13093A311, ML13121A089, ML13130A059, and ML13154A049, respectively). The LAR requested NRC review and approval of the proposed design.

REGULATORY AUDIT BASIS To support its safety evaluation (SE), the NRC Office of Nuclear Reactor Regulation (NRR),

Division of Engineering (DE), Instrumentation and Controls Branch (EICB), conducted an audit at the Westinghouse/CS Innovations (CSI) facility in Scottsdale, Arizona. The purpose of this audit was to determine if the lifecyle processes used, and the outputs of those processes will result in a PPS system for use at DCPP which will meet regulatory requirements. This audit provided information necessary to complete the NRC staff's evaluation of the advanced logic system (ALS) portion of the proposed DCPP PPS. The scope of this audit was previously defined in the associated audit plan that was sent to the licensee on February 1, 2013 (ADAMS Accession No. ML13029A667).

Enclosure

- 2 AUDIT ACTIVITIES The NRC audit team, consisting of Richard Stattel, Samir Darbali, and Rossnyev Alvarado from EICB, and George Simons, Eric Lee, Stacy Smith and Christopher Chenoweth from the Office of Nuclear Security and Incident Response (NSIR), and Shiattin Makor from Region IV, visited the Westinghouse/CSI facility in Scottsdale, Arizona, from February 11-14, 2013, to perform this audit.

The audit was conducted for the following aspects of the DCPP PPS System Development life cycle:

  • System Verification and Validation (V&V) - Verification that the V&V program used for the development of the DCPP ALS portion of the PPS meets the requirements of Institute for Electrical and Electronics Engineers (IEEE) Standard IEEE-1012, "Standard for Software Verification and Validation," and that the V&V program is implemented in a manner which reliably verifies and validates the design outputs of each stage of the design process.
  • Configuration Management - Verification that the configuration management system includes the appropriate hardware and logic implementation within the configuration management program, and that the configuration management system is effectively controlling these items.
  • Software Quality Assurance - Verification that the System Quality Assurance (SQA) program is effective in controlling the software development process to assure quality of the DCPP PPS application.
  • System Safety - Verification that the system safety plans and procedures used during the safety analysis activities were adequate to determine that the logic implementation is safe to be used in the DCPP PPS.
  • Cyber Security - Review of activities associated with addressing system and services acquisition controls as set forth in the licensee's NRC-approved Cyber Security Plan, and in accordance with Section 73.54, "Protection of digital computer and communication systems and networks," of Title 10 of the Code of Federal Regulations (10 CFR).

AUDIT

SUMMARY

Entrance Meeting (Monday, February 11, 2013)

At the entrance meeting, the audit team provided an overview of the audit plan and objectives for the audit. Facility logistics and a detailed audit schedule were discussed. During this meeting it was decided that NRR would coordinate audit activities with the NSIR audit team by having Samir Darbali and Shiattin Makor, who are working on the Secure Development and Operation Environment (SDOE) evaluation for the PPS, work directly with the NSIR audit team during the week.

-3 Scott Roberts, the Director of Operations, introduced a number of Westinghouse/CSI staff members, including: Joe Basso, Project Manager, Marci Maher, V&V Manager, Brian Studaker, QA Manager, and Bill Irmen, Operations Manager. Marci Maher was asked to provide an overview of the AlS documentation to familiarize the staff with the organization of information to be reviewed during the audit.

At the end of the entrance meeting, CSI staff noted that the current status of the AlS subsystem is that both cores are operating in the prototype system with field programmable gate array (FPGA) version 1 installed. Design phase development is in progress and is proceeding to the FPGA version 2. The Design phase Independent V&V (IV&V) activities have not commenced at the time of this audit, the V&V team was working on performing the IV&V activities on the Requirements phase.

The following sections provide descriptions of the activities performed during this audit. In several areas, proprietary information was reviewed as a part of the audit. Detailed proprietary audit notes for each subject area are provided in Attachment 1 to this report. Attachment 2 provides detailed description of the requirements thread review performed during the audit.

1.0 CSI DCPP PPS Application V&V Program The objective of the IV&V portion of the audit was to confirm that the Westinghouse/CSIIV&V processes are implemented per its documentation, with a focus on record keeping, documentation, and management activities. Because the project has only completed the IV&V activities associated with the Requirements Phase, the audit team could not review all records related to the DCPP PPS replacement project AlS subsystem V& V activities. The audit team was, however, able to review the currently completed examples of AlS logic development.

Marci Maher, the AlS logic IV&V Manager, Secil Karaaslan, IV&V lead for the DCPP PPS project, and Jeff Vance, IV&V team member were the primary Westinghouse/CSI participants for this portion of the audit.

1.1 Independent V& V Organization and Processes The IV&V portion of the audit began with a thorough discussion of the IV&V processes used for the application development and its implementation for the AlS portion of the DCPP PPS. This discussion was consistent with the description provided in 6002-00003, "AlS V&V Plan," and 6116-00003, "Diablo Canyon PPS W Plan," which had previously been reviewed by the NRC staff. The audit team had an extensive question and answer session with Westinghouse IV&V team about the AlS system development processes, IV&V involvement and the IV&V organization.

During this discussion, the IV&V team members explained that they are currently reVising Westinghouse/CSI document 6116-00003, "DCPP PPS W Plan," to include V&Vactivities that were not previously described. Specifically, the IV&V team explained that the current AlS V&V Plan (Rev. 1) did not identify the IV&V team performing risk and hazard analysis required by IEEE Std. 1012. This discrepancy was identified by the NRC staff in the DCPP PPS Open Items Table as item 83. Further, the AlS logic IV&V Manager noted that this has been captured in Corrective Action Program (CAP) ticket #12-1 03-M041.34, inspection summary CAP, dated July 16, 2012, which stated that CSI should perform a software safety analysis to

- 4 meet the requirements of Branch Technical Position (BTP) 7-14. The ALS Logic IV&V Manager explained that the IV&V team performed a hazard analysis for the ALS system, and that this information is currently documented in Section 4.2 of 6116-0029 failure modes and effects analysis (FMEA).

The NRC staff and the IV&V team discussed the diversity requirements associated with ALS Core logic implementations. The audit team noted that the IV&V staff is knowledgeable of their roles in the DCPP PPS application development.

The IV& V team has completed the V& V activities for the planning phase. They are currently working on the V&V activities for the requirements phase. The audit team reviewed the IV&V Requirement Phase W Summary Report which was available in draft version, during the audit.

The IV&V team explained that for the requirement phase they are comparing the requirements supplied by PG&E (e.g., Functional Requirements Specification (FRS>> to the documents prepared by Westinghouse, WNA-DS-02442-PGE, "Diablo Canyon Units 1 and 2 Process Protection System Replacement Project, ALS System Requirements Specification," to identify discrepancies and/or missing information. The IV&V team has found that not all requirements in PG&E documents match the requirements in WNA-DS-02442-PGE, and the design teams are currently addressing this issue (see descriptions for OnTime' Tickets #4787 and #4800 below).

1.2 Independent V&V Documentation IV&V team members explained how identified problems are being documented and addressed using the corrective action processes. Specifically, the IV&V team explained that if errors, inconsistencies, or anomalies are encountered, the OnTime ' ticket process is used to identify and record them. Attachment 1 to this audit report provides additional information about this ticket process. The NRC staff reviewed the following OnTime 1M tickets that were issued by the IV&V team while performing V&V activities during the requirements phase:

  • Ticket #4258 - describes an error in 6116-10201, "ALS PPS ALS-102 FPGA Requirement Specifications." A resolution for this issue was not provided in the ticket. This ticket will remain open until a new version of the document is formally released.
  • Ticket #4787 - identifies that many PG&E requirements are not included in the Westinghouse ALS System Requirements Specification, WNA-DS-02442-PGE.

In particular requirements identified in the Interface Requirements Specification (IRS) are not included in the ALS System Requirements Specification. A resolution for this issue was not provided in the ticket. This ticket will remain open until a new version of the document is formally released.

  • Ticket #4800 - identifies that the Requirements Traceability Matrix (RTM) does not clearly trace requirements in the FRS and WNA-DS-02442-PGE. In particular, there are requirements in the FRS that are not listed in the RTM. A resolution for this issue was not provided in the ticket. This ticket will remain open until a new version of the document is formally released.

-5 The ALS Logic IV&V Manager explained that Westinghouse Review Action Items are also used to provide comments on draft version of DCPP PPS planning and process documents. Once these documents are formally released, the IV&V team uses the OnTime' ticket process, as described above.

When the ALS DCPP PPS system is released for the IV&V team to verify and validate the Verilog code, the IV&V team will use OnTime' tickets to identify any anomalies and/or inconsistencies. During this discussion, the IV&V team also explained how the logic revision scheme is used to identify the status of ALS products.

Observation: The NRC staff has added an open item to request that a description of the FPGA versions be included in the system management plan. This description should include a discussion of when and to whom products are released. This has been documented as Open Item 87.

1.3 Training Records Review The NRC staff performed a review of the qualification and training records for one of the IV&V engineers assigned to the DCPP PPS project. Note that because the project has only completed the V&V activities for the planning phase, the staff was not able to confirm that all of the requirements for Human Diversity described in Westinghouse Work Instruction 9006-00037, "Human Diversity Management for FPGA Based Development and Test Activities," were being met. The NRC staff conducted a confirmatory review of the activities associated with Westinghouse Work Instruction 9006-00037 pertaining to the subject IV&V Engineer.

The NRC staff also reviewed the work history of this individual to ensure that the requirements of the work instruction were being met. This form is approved by the Functional Manager, Operational Manager, Network Systems Administrator, and the OnTime TM Administrator.

1.4 ALS FPGA Design Process and Documents The ALS Logic IV&V Manager, described the ALS document hierarchy that established the requirements for the ALS platform and the ALS DCPP PPS. This information is provided in CSI document 6002-00000, "ALS Management Plan." The IV&V manager explained how these documents are identified and traced in the Requirements Traceability Matrix (RTM), 6116-0059.

The documents that define the ALS DCPP PPS are:

  • PG&E Functional Requirements Specification
  • WNA-DS-02442 System Requirement Specification
  • 6116-00011 ALS Platform Design Specification
  • 6116-10201 FPGA ALS-102 Requirements
  • 6116-1 0203 DCPP ALS-102 Core A Design Specification
  • 6116-10204 DCPP ALS-1 02 Core B Design Specification Note: The prefix number 6116 is a unique project identifier for the DCPP PPS project.

-6 During this discussion, the NRC staff noted that the ALS platform document hierarchy to be prepared for the DCPP PPS project does not exactly match the documents that would be expected for a typical project. In particular, there was no 6116-10206 FPGA Design Specification was prepared for the DCPP PPS project. This is because FPGA ALS-102 Requirements document 6116-10201 includes this information, making it equivalent to a 6116 10206 document. Further, CSI personnel explained that ALS Platform document 6002-10206 includes the ALS-1 02 design specifications that would have otherwise been included in a 6116 10206 document.

Observation: The NRC staff has added an open item to request a brief explanation about this apparent document number misalignment between the ALS platform and the DCPP PPS project. This has been documented as Open Item 88.

CSI personnel explained that they prepared the RTM using Westinghouse document WNA-DS-02442 to trace PG&E requirements. The IV&V team found that Westinghouse document WNA-DS-02442 does not capture all PG&E requirements (see descriptions for Tickets #4787 and #4800, which currently address these). During the audit, the NRC staff and CSI discussed the fact that the current RTM does not trace requirements down to the core desjgn specifications.

Observation: The NRC staff added two open items to request a description of how all PG&E requirements will be captured in the RTM and an explanation of how the RTM will include traceability to the core design specifications. These have been documented as Open Items 93 and 92, respectively.

2.0 Quality Assurance Program This audit activity addressed the quality assurance program applicable to the ALS platform products and documentation associated with DCPP PPS project.

The Quality Assurance (QA) Lead for DCPP PPS, Brian Studaker, described the QA Westinghouse/CSI program. He explained that the CSI QA program is transitioning from CSI 9000-00000 Quality Assurance Manual to Westinghouse's Quality Management System (QMS). He also explained that Westinghouse 23.20 QA procedure establishes the interface agreement between Westinghouse and CSI, as well as the applicable procedures to be used at the Westinghouse/CSI facilities.

The NRC staff discussed QA activities for the DCPP PPS project with the QA team (B. Studaker, D. Harmon, and C. Bobbitt). The NRC staff confirmed that the quality assurance processes and procedures are subject to the same configuration management, corrective action, and change management activities that apply to ALS platform configuration items.

During this discussion, the QA team explained that they perform project specific audit assessments, reviews and inspections. These activities assess the QA aspects and efforts of the project during each phase of the lifecycle. The results of this audit are documented in the Quality Activity Report (QAR), which is updated after each phase of the lifecycle is completed.

During this NRC audit, the staff reviewed the QAR related to the DCPP PPS project, which are summarized in 6116-00300, "DCPP PPS QA Summary Report," Revision 0, dated October 1,

-7 2012. Section 2 of this document identified all activities performed until 01/31/2013. For each activity this section identifies: date, QAR number, description, person responsible, and status.

If a non-conformance or finding is observed, this item is recorded in the Quality Activity Report and the standard corrective action processes would be followed (i.e. a CAP would be initiated).

The NRC staff reviewed several CAPs during the audit, which are described in Attachment 1.

One of these CAPs was related to implementation of the ALS core diversity. This CAP was initiated to record an issue related to the FPGA development for the PG&E ALS-102 application.

The proposed resolution for this CAP was reviewed by the staff to determine if adequate controls for the build process for the FPGA are being implemented. This CAP is still open and is currently awaiting further analysis.

During the audit, the NRC staff discovered a deficiency in a work instruction used for developing FPGA applications. To address this, Westinghouse/CSI opened a new CAP ticket. provides detailed information about this CAP.

3.0 Configuration Management This audit activity addressed the configuration management (CM) activities applied to the DCPP PPS products and documentation. This included a review of the configuration controls that govern the DCPP PPS application specific documentation and FPGA programming design. The Senior Network Administrator (B. Wheeler), and the Manager for Scottsdale Operations (W. Irmen) gave the NRC staff an overview of the process required to perform controlled changes. Design and planning documents are created and edited in the Westinghouse network computer. After a document is updated, a release record is created which requires digital signatures from the Author, Reviewer, and Approver.

The NRC staff also reviewed the processes used for controlling access, securely transferring files, and maintaining backups for configuration items.

The NRC staff discussed the process used to prevent parallel document checkout with the CM librarian (also the Network Administrator). The staff observed that measures have been implemented to prevent inadvertent parallel changes from being made to system configuration files. However, the NRC staff observed that these measures do not prevent access to the configuration file. These measures allow only the person who first opened the file to make changes to it. Additional users may open the file while it is being edited, but they are not able to apply any changes of their own.

The NRC staff reviewed two OnTime ' tickets related to configuration management of the DCPP PPS replacement project. The NRC staff also met with an FPGA Senior Engineer to discuss the processes used to make configuration changes associated with one of these OnTime ' tickets. The engineer used the Westinghouse computer to access the system requirements and verify the requirements were correct. He then accessed the FPGA development tool which required an additional login. The engineer guided the NRC staff to the specific lines of code and the process for making the appropriate changes to address the issue and close the ticket. The engineer showed the NRC staff the log of the changes made to the FPGA application design.

- 8 The NRC staff performed a review of Westinghouse/CSI document 6116-00400, "DCPP PPS Configuration Management Report," Rev. O. This document defines and tracks the Project Milestones, Configuration Items, Configuration Status Accounting (CSA) and Baselines for DCPP PPS configuration items. The CSA is comprised of the current revision configuration items in the release database. The CSA is updated periodically to maintain document control throughout the duration of the project. Working copies of the CSA documents are maintained in Westinghouse/CSl's CM tool, Concurrent Version System (CVS).

The DCPP PPS CM Report contains a list of open OnTime ' tickets. The report states that there should be no open tickets at the Manufacturing stage. All related design configuration items are in the Development Stage of the DCPP PPS Life-Cycle.

The NRC staff reviewed Westinghouse/CSI document 6116-00050, "DCPP PPS Configuration Status Accounting." This document provides the Identification Number, Status, Revision, and Date for DCPP PPS configuration items. The CSA also provides the Revision numbers for 3rd party tools used during the DCPP PPS project development, including the FPGA modification tool and CVS Suite.

4.0 Diversity CSI document 6002-00031, "ALS Diversity Analysis," describes the ALS platform's built-in diversity in terms of common-cause programming error considerations and in support of future licensee and application-specific diversity and defense-in-depth analyses. Westinghouse/CSI formalized implementation of the diversity claims by implementing procedures or work instructions to ensure ALS platform design diversity claims are supported. The NRC staff reviewed these work instructions and noted that they were not fully satisfying what was needed to establish the built-in diversity required for the DCPP PPS system. CSI has issued two CAPs to address this (CAP # 13-036-M051 and #13-042-M070).

During this review, the NRC staff noted that the work instructions used for one of the cores does not clearly identify how to implement core diversity. Attachment 1 provides additional information.

Observation: Westinghouse/CSI should consider adding clarification to the work instructions about the use of diverse techniques.

The NRC staff performed a targeted thread audit of the requirement associated with the platform's diversity attributes applicable to the DCPP PPS replacement project. The NRC staff observed how this requirement and design specifications are included in the PPS application.

Two of the Westinghouse/CSI FPGA Developers demonstrated the processes used for implementing diversity in the PPS system. Attachment 1 provides additional information.

4.1 Human Diversity The ALS Diversity Analysis also states that human diversity is another means to ensure diversity in the ALS platform. The NRC staff reviewed the Work Instruction used by ALSlWestinghouse personnel to ensure human diversity is maintained during FPGA development for the DCPP PPS project. Attachment 1 provides additional information.

-9 5.0 Thread Audit of the DCPP Application Software V&V Program The NRC staff selected specific requirements to audit based on Significance to the NRC staffs safety evaluation for the DCPP PPS replacement project.

The following threads were evaluated during this audit:

  • FRS 3.2.1.16 detection of an internal rack failure
  • FRS 3.2.1.10 and IRS 1.5.8 - time response
  • FRS 3.2.1.5.3 PPS - channel in Bypass Detailed notes for these requirements thread review audit activities are provided as . During the thread review, the NRC staff identified several Open Items, which are described in Attachment 2. The identified Open Items should be addressed by Westinghouse/CSI via its corrective action program during the upcoming design and implementation phases of the project. Each of these areas will be further evaluated during the next NRC audit which is to be performed when the PPS design implementation is complete.

6.0 Secure Development Environment This audit activity addressed the secure development environment (SDE) applied to the ALS platform and the DCPP PPS products and design documentation. This included a review of the activities and documentation incorporated by Westinghouse/CSI to prevent the inclusion of errors and unintended functionality.

7.0 ALS Service Unit (ASU) Demonstration The NRC staff was able to observe a demonstration of the ASU functionality while connected to the DCPP PPS prototype. Both the ASU and the DCPP PPS are still in the development stage.

The staff observed the indications and alarms that occur when the test ALS bus (TAB) is connected to the maintenance workstation (MWS) and when a board is pulled from the rack.

The staff also observed how calibration and testing modes are accessed, and how setpoint changes will be made. The staff was informed that a custom cable was being developed to connect the TAB to the ALS-102 board.

8.0 Exit Meeting (Thursday - February 14, 2013)

During the exit meeting, Westinghouse/CSI was provided with a summary of the Open Items identified during the audit. In addition, a list of documents was provided with a request to provide the NRC staff access to support its ongoing review activities. This list is included in Open Item 91.

- 10 CONCLUSION The NRC staff addressed each of the planned audit activities outlined in the audit plan. Several requirements threads were selected and evaluated for compliance with the DCPP specific planning documents. Interviews were conducted with Westinghouse/CSI personnel from the IV&V, Design Engineering, Quality Control, and Configuration Management groups.

The following Open Items were identified during this audit.

87 FPGA versions 1, 2, 3, descriptions were explained to the NRC staff during the ALS audit but these release processes are not captured in the system development plan or system management plan.

88 Please describe why there is a misalignment of document numbers between the platform 6002-xxx01, 6002-xxx06 and application specific documents 6116 10201. For example, please explain why is there no 6116-10206.

89 Ensure that the audit schedule issues (Pennatronics) identified during the cyber security review portion of the ALS audit is resolved prior to issuance of the Diablo PPS safety evaluation. The NRC will be reviewing the responses to the CAP's that Westinghouse has written on this issue to assess if there are any implications on the DCPP PPS system.

90 Once CSI has completed the SDOE evaluation to show conformance to RG 1.152 requirements, the results will have to be docketed.

91 Please provide the NRC access to the following documents via SharePoint:

  • Work instruction for Human Diversity Management for FPGA Based Development and Test Activities, Document number 9006-00037, Rev. 0
  • ALS Core A FPGA Build Procedure, Document number 9006-00043, Rev. 3
  • ALS Core B FPGA Build Procedure, Document number 9006-00071, Rev. 1
  • 6116-10203/4 Core A and Core B Design Specifications
  • RTM sorted by FRS.

92 The Requirements Traceability Matrix (RTM) does not trace to CSI documents 6116-10203/4 Core A and Core B<Design Specifications. CSI must include this traceability to the RTM once the 6116-1020314 Core A and Core B Design Specifications are finalized.

- 11 93 The RTM for the ALS subsystem was prepared using Westinghouse document WNA-DS-02442 to trace PG&E requirements. The IV&V team found that Westinghouse document WNA-DS-02442 does not capture all PG&E requirements (see descriptions for Tickets #4787 and #4800).

Date: October 8,2013 Attachments:

1. Detailed Audit Notes (Proprietary)
2. Detailed Requirement Thread Notes (Proprietary)

OFFICIAL USE ONLY PROPRIETARY INFORMATION E. Halpin - 2 Engineering, Quality Control, and Configuration Management groups. Several Open Items were identified and noted in the audit report for future follow-up. Attachment 1, "Detailed Audit Notes," and Attachment 2, "Detailed Requirement Thread Notes," of the Enclosure contain proprietary information and have been withheld from public disclosure.

The cyber security review activities performed in conjunction with this audit are being documented in a separate report that is being written by the NRC's Office of Nuclear Security and Incident Response staff.

If you have any questions, please contact me at 301-415-1530 or via e-mail at Jennivine.Rankin@nrc.gov.

Sincerely, IRA!

Jennie K. Rankin, Project Manager Plant Licensing Branch IV Division of Operating Reactor Licensing Office of Nuclear Reactor Regulation Docket Nos. 50-275 and 50-323

Enclosure:

As stated cc w/Enclosure (no Attachments): Distribution via Listserv DISTRIBUTION:

PUBLIC RidsNrrPMDiabloCanyon Resource LPL4 RlF RidsRgn4MailCenter Resource RidsAcrsAcnw_MailCTR Resource RStattel, NRRlDE/EICB RidsNrrDeEicb Resource RAlvarado, NRRlDE/EICB RidsNrrDorlLpl4 Resource SMakor, RIV/DRS/EB2 RidsNrrLAJBurkhardt Resource ELee, NSIR ADAMS Accession Nos.: ML13232A261 (Ltr + Encl + PI Attachments);

ML13232A263 (Redacted Ltr + Encl w/o Attachments)

OFFICE NRRlDORULPL4/PM NRRlDORLlLPL4/LA NRRlDORULPL4/LA

. NAME MBartlett JRankin JBurkhardt DATE 8/19/13 10/7/13 8123/13 I OFFICE NRRlDE/EICB/BC NRRlDORLlLPL4/BC NRRlDORLlLPL4/PM NAME JThorp MMarkley JRankin DATE 5/17/13 1018/13 10/8113 \:

OFFICIAL RECORD COpy OFFICIAL USE ONLY PROPRIETARY INFORMATION