ML15103A010
| ML15103A010 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 05/19/2015 |
| From: | Siva Lingam Plant Licensing Branch IV |
| To: | Halpin E Pacific Gas & Electric Co |
| Haskell R | |
| References | |
| TAC ME7522, TAC ME7523 | |
| Download: ML15103A010 (13) | |
Text
OFFICIAL USE ONLY PROPRIETARY INFORMATION UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 May 19, 2015 Mr. Edward D. Halpin Senior Vice President and Chief Nuclear Officer Pacific Gas and Electric Company Diablo Canyon Power Plant P.O. Box 56, Mail Code 104/6 Avila Beach, CA 93424
SUBJECT:
DIABLO CANYON POWER PLANT, UNITS 1 AND 2 - REGULATORY AUDIT REPORT FOR THE JUNE 3-5, 2014 AUDIT AT THE INVENSYS OPERATIONS MANAGEMENT FACILITY IN LAKE FOREST, CALIFORNIA, FOR THE DIGITAL UPDATE TO THE PROCESS PROTECTION SYSTEM LICENSE AMENDMENT REQUEST (TAC NOS. ME7522 AND ME7523)
Dear Mr. Halpin:
By letter dated October 26, 2011, as supplemented by letters dated December 20, 2011, and April 2, April 30, June 6, August 2, September 11, November 27 and December 5, 2012, and March 7, March 25, April 30, May 9, May 30, and September 17, 2013, and April 24 and April 30, 2014 (Agencywide Documents Access and Management System (ADAMS) Accession Nos. ML113070457, ML113610541, ML12094A072, ML12131A513, ML12170A837, ML12222A094, ML12256A308, ML13004A468, ML12342A149, ML13267A129, ML13093A311, ML13121A089, ML13130A059, ML13154A049, ML13261A354, ML14205A031, and ML14121A002, respectively), Pacific Gas and Electric (PG&E, the licensee), requested the U.S. Nuclear Regulatory Commission (NRC) staff's approval of an amendment for the Diablo Canyon Power Plant, Units 1 and 2 (DCPP). The proposed license amendment request would provide a digital replacement of the Process Protection System (PPS) portion of the Reactor Trip System and Engineered Safety Features Actuation System at DCPP.
The NRC Instrumentation & Controls Branch (EICB) conducted an initial audit of the Invensys Operations Management (IOM) facilities in Lake Forest, California, from November 13-16, 2012 (ADAMS Accession No. ML 1301SA149). During that audit, the NRC staff was unable to observe how the design phase outputs of the Tricon system are subject to the verification and validation process for the IOM portion of the PPS design, and anticipated to perform a follow-up audit. In addition, the NRC staff identified documents required to be submitted on the docket to facilitate completion of its assessment of the Tricon V10 platform changes/software revisions. As a result, IOM submitted these documents on February 11, 2013 (ADAMS Accession No. ML130920558).
Attachments 1 through 5 of the Enclosure to this letter contain Proprietary Information. Upon separation from Attachments 1 through 5, this letter and the Enclosure are DECONTROLLED.
OFFICIAL USE ONLY PROPRIETARY INFORMATION
OFFICIAL USE ONLY PROPRIETARY INFORMATION E. Halpin The EICB staff conducted a second follow-up audit at the IOM facilities from June 3-5, 2014. The audit was performed in accordance with the regulatory audit plan dated May 20, 2014 (ADAMS Accession No. ML14126A377). The purpose of this second audit was to gain information necessary to determine if the lifecycle processes used, and the outputs of those processes, have produced a PPS system for use at DCPP which meets regulatory requirements.
This audit provided information necessary to complete the NRC staff's evaluation of the proposed Tricon portion of the DCPP PPS.
The Enclosure to this letter details the results of the audit; Attachments 1 through 5 of the Enclosure contain Proprietary Information and will be withheld from public disclosure pursuant to Section 2.390 of Title 10 of the Code of Federal Regulations.
If you have any questions, please contact me at 301-415-1564 or via e-mail at Siva.Lingam@nrc.gov.
Sincerely,
~<j*~
Siva P. Lingam, Project Manager Plant Licensing Branch IV-1 Division of Operating Reactor Licensing Office of Nuclear Reactor Regulation Docket Nos. 50-275 and 50-323
Enclosure:
Audit Report cc w/encl (no attachments): Distribution via Listserv OFFICIAL USE ONLY PROPRIETARY INFORMATION
UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 REGULA TORY AUDIT REPORT PERFORMED AT INVENSYS OPERATIONS MANAGEMENT FACILITY ON JUNE 3-5, 2014 IN SUPPORT OF THE DIGITAL PROCESS PROTECTION SYSTEM PACIFIC GAS AND ELECTRIC COMPANY DIABLO CANYON POWER PLANT, UNITS 1 AND 2 DOCKET NOS. 50-275 AND 50-323
Background
By letter dated October 26, 2011, as supplemented by letters dated December 20, 2011, and April 2, April 30, June 6, August 2, September 11, November 27 and December 5, 2012, and March 7, March 25, April 30, May 9, May 30, and September 17, 2013, and April 24, and April 30, 2014 (Agencywide Documents Access and Management System (ADAMS) Accession Nos. ML113070457, ML113610541, ML12094A072, ML12131A513, ML12170A837.
ML12222A094, ML12256A308, ML13004A468, ML12342A149, ML13267A129, ML13093A311, ML13121A089, ML13130A059, ML13154A049, ML13261A354, ML14205A031, and ML14121A002, respectively), Pacific Gas and Electric (PG&E, the licensee), requested the U.S. Nuclear Regulatory Commission (NRC) staff's approval of an amendment for the Diablo Canyon Power Plant, Units 1 and 2 (DCPP). The proposed license amendment request (LAR) would provide a digital replacement of the Process Protection System (PPS) portion of the Reactor Trip System and Engineered Safety Features Actuation System at DCPP. The LAR requested U.S. Nuclear Regulatory Commission (NRC) review and approval of the proposed design.
From November 13-16, 2012, the NRC Instrumentation & Controls Branch (EICB) conducted the first audit of the Tricon PPS subsystem at the Invensys Operations Management (IOM) facilities in Lake Forest, California (see audit report dated March 4, 2013; ADAMS Accession No. ML 1301BA149). During that audit, the NRC staff was unable to observe how the design or implementation phase outputs were subject to the verification and validation processes. The staff also identified several open items and noted that a follow-up audit would be required to evaluate resolution activities for each of these open items. To support its safety evaluation, EICB staff conducted a second audit at the IOM facilities in Lake Forest, California.
Regulatory Audit Basis The purpose of this second regulatory audit was to gain information needed to determine if the lifecycle processes used, and the outputs of those processes have resulted in a PPS system for use at DCPP, which meets regulatory requirements. This audit provided information necessary Enclosure
for completion of the NRC staff's evaluation of the proposed Tricon portion of the DCPP PPS.
The audit was performed in accordance with the regulatory audit plan dated May 20, 2014 (ADAMS Accession No. ML14126A377).
Audit Activities The NRC audit team, consisting of Richard Stattel, Rossnyev Alvarado, and Samir Darbali from EICB, and Shiattin Maker from Region IV, visited the IOM facility in Lake Forest, California, from June 3-5, 2014, to perform the regulatory audit. The following activities were performed during this audit:
- 1. Entrance Meeting At the entrance meeting, the audit team provided an overview of the audit plan and objectives for the audit. Facility logistics and a detailed audit schedule were discussed.
The Project Manager (PM) for the PPS project introduced a number of IOM staff members, including the Project Director, the Quality Manager - Nuclear, Project Engineer (PE), Application Engineer (AE), and the Independent Verification and Validation (IV&V) Manager for the DCPP PPS replacement project.
- 2. Factory Facility Tour Following the entrance meeting, IOM staff provided a factory facility tour for the audit team. The audit team was able to view the DCPP PPS safety system, as it was being prepared for the Factory Acceptance Test (FAT).
- 3. Follow-up Thread Requirement Reviews The NRC staff was able to complete this audit activity successfully. The audit team selected requirements from the first audit to perform follow-up reviews of the associated design and implementation phase activities (see March 4, 2013, IOM Tricon Audit Report):
- TCold
- Reactor Coolant System (RCS) Temperature Signal Conditioning
- Safety Analysis Process Thread For each thread, the audit team traced requirements to specific test cases performed during system verification testing and to test cases to be performed during the PPS FAT.
Detailed notes for these requirements thread reviews are provided in Attachment 1.
The NRC staff used the PPS Project Traceability Matrix (PTM) to facilitate performance of thread requirements evaluations. The NRC staff made several observations regarding the PTM, which are summarized in Attachment 2.
- 4. Maintenance Workstation Institute of Electrical and Electronics Engineers (IEEE) 7-4.3.2, "IEEE Standard Criteria for Digital Computers in Safety Systems of Nuclear Power Generating Stations,"
Clause 5.11, Identification, defines identification requirements for software systems. To assist in the evaluation of this item, IOM personnel demonstrated how the Tri Station 1131 (TS 1131) Maintenance Workstation (MWS) can be used to verify that the correct software is installed into the DCPP PPS system hardware using plant equipment, which was available in the FAT test area. The NRC staff observed the MWS application software provides an additional indication of the current Tricon Software Application Program (TSAP) version and that a Tricon software verification activity can be performed with the PPS system operable. The PG&E representative also confirmed periodic tests will be performed to verify that installed software is correct during plant operation. This audit activity was used to support development of inspection item 5 (see Attachment 5). The NRC staff was able to complete this audit activity successfully.
- 5. Time Response The NRC staff reviewed the relation between the specified time response requirements for PPS and the safety analysis response time assumptions listed in the DCPP Updated Final Safety Analysis Report (UFSAR) Table 15.1-2. The NRC staff verified Tricon application scan times to be correctly set. The staff also reviewed the software verification scan time test results, which were completed in May of 2014 and confirmed satisfactory scan time performance results. Detailed notes for this review are provided in Attachment 1. The staff also reviewed the calculation for determining the delay time for the Steam Generator (SG) Water Level Trip Time Delay (TTD). Detailed notes for this review are also provided in Attachment 1. The NRC staff was able to complete this audit activity successfully.
- 6. Independent Verification and Validation The purpose of the IV&V portion of the audit was to confirm that the IOM IV&V activities are performed and documented per its approved processes, with a focus on record keeping, documentation, and management activities. The NRC staff was able to complete this audit activity successfully.
The NRC staff performed a follow-up thread audit of selected requirements, during which IOM staff was asked to track the implementation of various system and software requirements through each phase of the design process using the PTM. They were also asked to show how the design phase outputs were subject to the V&V process activities.
At the time of the audit, IOM was performing activities associated with the test phase of the software development lifecycle, including development of test procedures and test cases to be used during the FAT. Descriptions of the requirements threads are provided in Attachment 1.
The NRC staff reviewed the software development process and V&V activities to confirm that IOM IV&V processes are implemented per its documentation and that the general
requirements of current industry standards are being followed. Detailed notes for this review are provided in Attachment 2.
The NRC staff also evaluated if the V&V team was sufficiently independent in terms of cost, schedule, and management. The V&V summary reports were a guide for this activity.
- 7. Configuration Management The NRC staff conducted a review the configuration management (CM) system used for the DCPP PPS software development and interviewed IOM personnel responsible for performing CM activities. This portion of the audit focused on IOM's record keeping, documentation, management activities, and use of CM tools. The NRC audit team reviewed plans, procedures, and guides used for CM and observed how IOM staff used tools to control and manage software and documentation, as well as track non-conformities within its processes. The NRC staff was able to complete this audit activity successfully.
As previously described in the first audit report dated March 4, 2013, when documents are released, they are stored in the Nuclear Integration Records. Each document is accompanied by a Design Review Checklist (DRC), a Design Review Comment Sheet (DRCS), and a Document Review Release (DRR). The DRC, DRCS, and DRR identify comments, modifications, reviewers, and approvals for all project documents. During the audit, the NRC staff reviewed several DRRs to observe how CM of project documents was implemented. Attachment 2 contains descriptions of the DRRs reviewed during the audit.
Regarding software modifications, during the first audit, IOM explained the process for managing and controlling the TSAP software application for the DCPP PPS replacement project. Once the TSAP programs are created, a Software Development Checklist (SDC) is generated for that version of the application. If a new revision of the TSAP is necessary, a new SDC will be created. Attachment 2 contains a brief description of the SDC reviewed during the audit.
IOM uses a Master Configuration List (MCL) for configuration control of all configuration items, project documents, TSAP versions, and documentation of final system configuration. During the audit, the NRC staff reviewed the MCL, and observed how IOM personnel used the MCL to identify configurable items tracked, stored, and controlled.
In addition, the NRC staff discussed the use of the System Hierarchy Automated File Transfer (SHAFT) program with IOM personnel. The Tricon V10 Topical Report (TR)
Safety Evaluation Report (SER) states that after testing is completed, IOM uses the SHAFT program to track configuration of the system (e.g., version of the system, modules requested, etc.) to be installed in a plant-specific application. There are two portions of the program: 'Shaft Design' is used to build systems (like a configuration list for what parts/modules are needed for the Tricon V10), and 'Shaft Floor,' which is used
in the test floor to track and replace serial numbers in modules. Both use one common database.
IOM provided the Nuclear Order Checklists for all four PPS protection sets from SHAFT.
These checklists contain the Tricon High Density System Configuration Sheets, dated November 2, 2012. These sheets are developed by Manufacturing and contain the chassis number, module type, serial number, revision, chassis key, cable key and slot position/location. The NRC staff compared the serial numbers found in these sheets to the serial numbers in the MCL and reviewed System Integration Deficiency Report (SIDR)-1058, which identifies discrepancies between High Density System Configuration Sheets and the MCL (see Attachment 2).
- 8. Software Quality Assurance The audit team reviewed the software quality assurance (SQA) processes with the IOM quality assurance (QA) and V&V managers. Both of these managers are responsible for evaluating the effectiveness of the SQA program in assuring quality of the DCPP PPS application software. The NRC staff was able to complete this audit activity successfully.
Quality Surveillance Reports:
The NRC staff reviewed Quality Surveillance Reports (QSRs) to assess QA activities performed for the PPS system application by IOM. Quality Program Manual (QPM) 17.2, "Quality Surveillance," states that QA surveillances are performed at the end of each lifecycle. The results of these activities are documented in QSRs, which are updated after each phase of the lifecycle is completed. For example, QSR 14-31 describes QA activities performed to ensure PPS protection set I project-specific documentation had been prepared or performed in accordance with the QPM, project management plan, project quality plan, and licensee requirements. The report includes a checklist which documents the status and acceptability of implementation phase activities. This report makes a recommendation for the project to enter the test phase for PPS protection set I.
QSR 14-27 was also reviewed by the audit team. This "Readiness Review Surveillance" was performed in preparation for the audit. This is a type of surveillance which can be performed at the discretion of the quality manager (QM) or IOM Director for any purpose.
QPM 17.2 describes how the readiness review surveillance should be performed. This report includes a discussion of corrective action report (CAR). As a result of this review surveillance, IOM personnel prepared CAR-2492 to document an error found in the PTM. Specifically, this CAR was initiated to identify that the PTM incorrectly traced requirement R-3001 to Section 3.3.5.8.1 of the software requirements specification (SRS). To resolve this error, IOM personnel revised the PTM to trace R-3001 to the correct SRS section which is Section 3.3.2.8. The NRC staff reviewed CAR-2492 and the modified PTM and observed that this correction was made.
Problems and Anomalies:
The IOM Software Quality Assurance Plan (SyQAP), 993754-1-900, states that when problems or anomalies are identified during the design, implementation, and test phases, they should be documented and resolved in accordance with IOM Project Procedures Manual (PPM) 10.0, Nonconformance & Corrective Action. If resolution of the anomaly requires the preparation of an anomaly report, IOM personnel will file an SIDR. The SIDR would accompany the SOC for the new version of the application program. When unexpected test conditions and/or deviations from procedural requirements are identified, the problems are also documented and disposition is determined on a CAR. The NRC staff reviewed several SIDRs and CARs that were created to track non-conformances.
Attachment 2 contains description of the SIDRs and CARs reviewed during the audit.
Licensee Quality Assurance Activity Assessment:
An interview with the PG&E QA manager was held to discuss the role of the PG&E QA organization as defined in the SyQAP. The NRC staff also discussed the transition of QA responsibilities from IOM to the licensee. At the current stage of the system development, the licensee has performed some document reviews and is planning to perform onsite QA inspections during the upcoming FAT tests. The licensee will have plant representatives present during all of the system FATs to oversee the vendor testing activities.
The NRC staff discussed its comments on the auditability of the PTM with the licensee.
The NRC's expectation is that licensee will need to provide oversight and assess completeness of requirements implementation activities prior system installation and testing. The NRC staff also discussed the design and implementation phase V&V summary reports with the licensee. The licensee QA organization had not reviewed these IOM reports. The NRC staff pointed out to the licensee that these reports provide project information on development activities and suggested that PGE QA begin performing IOM V&V document reviews and should consider participating in the project conference calls.
The NRC staff discussed the inspection items list in order to make the licensee's QA organization aware that several of the QA activities defined in 3.3 of the SyQAP will be inspected by NRC during site inspections. The NRC staff also referred the Licensee QA manager to the inspection items list which is provided in Attachment 5 of this report.
- 9. Software Safety The IOM software safety plan (SSP) identifies processes and activities for improving software safety throughout the software development lifecycle. The audit team reviewed IOM software safety processes including the SSP and the procedures used during PPS software safety analysis activities with representatives of the IOM QA and IV&V organizations to assess the effectiveness of these programs in achieving this objective.
The audit team obser\Jed that the IV&V activities being performed on the PPS included safety analysis activities commensurate with the requirements for software integrity level 4 software as defined by Section 4.3 of the Software V&V Plan (993754-1-802).
The safety goals identified in the SSP, which include mitigation of software related hazards, appear to have been achieved in the implemented PPS application. The NRC staff was able to complete this audit activity successfully.
As stated in the SSP, software program activities are performed by personnel from both the design engineering and IV&V organizations. It was apparent to the audit team that the engineers from these organizations were effectively communicating software safety issues with each other effectively and were communicating on a regular basis.
Personnel interviewed from each organization were found to be knowledgeable and well informed of current and past safety issues pertaining to the PPS application.
- 10. Tricon Communications (Alternative to Interim Staff Guidance (ISG)-04, Position 10)
The audit team reviewed the function blocks in the application program associated with operation of the Tricon keyswitch and the Channel Out-of-Service (OOS) switches to evaluate the alternative proposed for ISG-04, position 10. The NRC staff also reviewed required actions to be performed on a change of keyswitch or Channel OOS switch position. In addition, the audit team reviewed the function (i.e., gated access) that enables the maintenance workstation to write to internal tag names while the Tricon keyswitch is in the RUN position.
Attachment 3 provides a detailed description of the Tricon keyswitch and the application program that allows modifications of certain parameters when the keyswitch is in RUN mode, and the parameter changes are only made to the system when associated channels are inoperable.
The NRC staff found the software documentation for the DCPP PPS includes necessary information regarding the keyswitch voting logic, fault analysis, command execution, and diagnostic status to allow modifications of certain parameters when the keyswitch is in the RUN position.
- 11. Secure Development.Environment This audit activity addressed the secure development environment applied to the DCPP PPS products and design documentation. This included a review of the activities and documentation incorporated by IOM at the Lake Forest, California, facility to prevent the inclusion of errors and unintended functionality within the PPS application. A summary of the NRC staff observations is provided in Attachment 4.
- 12. Inspection Items During the audit, the NRC staff discussed inspection items that will be included in the SER. Several changes and clarifications were made to the proposed list. The revised inspection items list is included as Attachment 5 to this report.
- 13. Exit Meeting During the exit meeting, IOM was provided with a summary of the open items identified during the audit to be included in the project Open Items List. Additionally, a list of documents was provided with a request to provide the NRC staff access in order to complete review activities. This list is provided as Item 120 in the project Open Items List below.
Conclusions The NRC staff addressed each of the planned audit activities outlined in the audit plan. Several requirements threads were selected and evaluated for compliance with the DCPP PPS specific planning documents and with regulatory requirements. Interviews were conducted with IOM personnel from the IV&V, Design Engineering, Quality Assurance, and Configuration Management groups.
The following Open Items and Requests for Additional Information were identified during this audit.
118. The NRC staff requests resolution of the SIDR-1057 and CAR-2531 be placed on SharePoint for follow-up review. The NRC staff is particularly interested in reviewing the extent of condition analysis performed for each of the process deficiencies identified.
119. IOM is using laptops to develop the V1 O Tricon protection set application code for the DCPP PPS Replacement project. Each of these laptop computers has the TS1131 application installed. During the IOM audit on June 3-5, 2014, PG&E informed the staff that it does not plan to use these laptops, and only plans use the TriStation installed in the MWSs. Please clarify what PG&E will do with the development laptops once they are delivered to them. Also, please clarify what controls will be placed on the TriStation software installed in these laptop computers.
120. The NRC requests the following documents be placed on SharePoint when available.
- Completed FAT Test Case for tests relating to Turbine Impulse Pressure, Time Response Verification, and SG Trip Time Delay TTD function.
- 993754-11-902-1, FAT Procedure, Appendix 1-18, Turbine Impulse Chamber Pressure (TICP)
- Current version of the Hardware Design Description (HOD)
- Updated Safety Analysis Report
- PTM Revision 8.0 Including traceability to Validation Test Cases
- Design of Controller Transfer Function Design Input Specification SAP Specification No. 110000000552 Revision 4
121 Implementation Phase V&V Summary Report - The NRC staff reviewed the IV& V tasks summarized in this report and confirmed correlation to several of the activities prescribed in the Software V&V Plan (SVVP) Section 5.2.4. There were two prescribed activities in the SVVP that did not have corresponding activities in Section 3.2 of the Implementation phase IV&V activity summary report. These were:
(2) Verify that the Input/Output (1/0) List is correct and ensure implementation requirements are adequately incorporated, and (9) Nuclear IV&V shall be responsible for system test equipment staging to perform the Validation Test.
The NRC staff noted that the protection set one 1/0 list is listed as an Implementation phase input document (993754-11-806), however, the summary report makes no mention of a specific IV&V activity to ensure correctness of this list. It can be inferred that performance of test equipment staging for the Validation test was completed prior to performance of the informal dry-run of the FAT as described in task number 13, however, there is no specific activity listed to ensure accomplishment of this task.
122. Implementation Phase IV&V Summary Report- Several of the input documents pertain only to protection set I, however, the summary report scope is intended to include all four protection sets. Section 2.0 of the summary report provides a reference to a project conditional release (CR 993754-06), but this does not describe a reduction in scope to only protection set one.
Please explain how IV&V activities for the entire scope of the PPS Tricon system (i.e. all four protection sets) will be reported.
123. Implementation Phase IV&V Summary Report- One of the Deficiency Reports remained unresolved at the completion of the Implementation phase. This was CAR 2507 which is an issue with the Emulation Test Driver used for system verification tests. No evaluation of risks associated with this unresolved issue was provided in the Technical and Management Risks section of the summary report and the report recommended exiting the implementation phase with this condition present. Additionally, the report states that "all deficiencies are resolved," and that "all output Documents are issued." These statements seem to conflict with the information in Table 5-3, and with the conditions of the provisional release in that only protection set I implementation is complete.
The NRG staff requested the licensee to provide responses to these open items either through RAI responses or through future correspondence to support completion of the PPS system safety evaluation.
Principal Contributors: Richard Stattel, NRR/DE/EICB Rossnyev Alvarado, NRR/DE/EICB Samir Darbali, NRR/DE/EICB Date:
Attachments
- 1. Requirements Thread (proprietary)
- 2. Software Activities (proprietary)
- 3. Description of Tricon Communications regarding Alternative to ISG-04, Position 1O {proprietary)
- 4. Secure Development Environment (proprietary)
- 5. Revised Inspection Items List (proprietary)
ML15103A011 (OUO-PI version); ML15103A010 (Redacted Ltr +Encl)
OFFICE NRR/DORL/LPL4-1/PM NRR/DORL/LPL4-1 /PM NRR/DORL/LPL4-1/LA NAME RHaskell Slingam JBurkhardt DATE 4/27/15 4/29/15 4/20/15 OFFICE NRR/DE/EICB/BC NRR/DORL/LP L4-1 /BC NRR/DORL/LPL4-1/PM NAME JThorp MMarkley Slingam DATE 4/30/15 5/19/15 5/19/15