ML20196F799

Forwards Rept Representing Results of 981026-29 Audit of Year 2000 Pragram at Hope Creek Generating Station.Audit Conducted as Followup to NRC GL 98-01, Year 2000 Readiness of Computer Sys at Npps, Issued 980511

ML20196F799
Person / Time
Site:	Hope Creek
Issue date:	11/30/1998
From:	Richard Ennis NRC (Affiliation Not Assigned)
To:	Keiser H Public Service Enterprise Group
References
GL-98-01, GL-98-1, TAC-MA1845, NUDOCS 9812070124
Download: ML20196F799 (25)

Text

_. . . _ _ . . . - . _ _ _ . _ _ ~ . . . _ _ . _ _ _ _ _ _ _ _ . _ _ . _ . . . . _ _ . _ _ _ _ _

November 30, 1998

!. Mr. Harold W. Keiser

! Chief Nuclear Officer & President-

! Nuclear Busir.ess Unit Public Service Electric & Gas Company ,

j - Post Office Box 236 l l Hancocks Bridge, NJ 08038  ;

SUBJECT:

AUDIT REPORT ON THE YEAR 2000 PROGRAM, HOPE CREEK GENERATING  ;

l STATION (TAC NO. MA1845) i -

Dear Mr. Keiser:

On October 26-29,1998, the NRC staff conducted an audit of the Year 2000 (Y2K) program at I the Hope Creek Generating Station as a followup to NRC Generic Letter (GL) 98-01, " Year 2000 Readiness of Computer Systems at Nuclear Power Plants," issued on May 11,1998. The enclosed report presents the results of the audit. The results of this audit and subsequent audits at other selected nuclear power plants will be used by the staff to determine the need for  !

additional action, if any, on Y2K readiness for nuclear power plants.

In accordance with 10 CFR 2.790 of the NRC's " Rules of Practice," a copy of this letter and its i enclosure will be placed in the NRC Public Document Room. Please contact me at (301) 415-  !

1420 if you have any questions or comments on the enclosed audit report. l Sincerely, ORIGINAL SIGNED BY:

Richard B. Ennis, Project Manager -

t Project Directorate 1-2 Division of Reactor Projects - 1/11 Office of Nuclear Reactor Regulation i

Docket No. 50-354 l

Enclosure:

As stated i

cc w/ encl: See next page DISTRIBUTION: See attached list

?

OFFICE PDI-2/PM PDI-3/LA A HIC 6/BC - PDI-2/D NAME REnnis mw TClark NI JW RCapra

\

DATE Ii /3 */98 Il /3I)98 Il 170/98 11/So/EB _.

OFFICIAL RECORD COPY DOCUMENT NAME: HCMA1845.AUD-j 9812070124 981130 *

, PDR ADOCK 05000354 i

P PDR

&fo hh,k k

p ***?u g *, UNITED STATES g j NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. unaan ang

,,,,,8 November 30, 1998 Mr. Harold W. Keiser Chief Nuclear Officer & President -

Nuclear Business Unit Public Service Electric & Gas Company Post Office Box 236 Hancocks Bridge, NJ 08038

SUBJECT:

AUDIT REPORT ON THE YEAR 2000 PROGRAM, HOPE CREEK GENERATING STATION (TAC NO. MA1845)

Dear Mr. Keiser On October 26-29,1998, the NRC staff conducted an audit of the Year 2000 (Y2K) program at i the Hope Creek Generating Station as a fo!!owup to NRC Generic Letter (GL) 68-01, " Year 2000 l Readiness of Computer Systems at Nuclear Power Plants," issued on May 11,1998. The  ;

enclosed report presents the results of the audit. The results of this audit and subsequent audits at other selected nuclear power plants will be used by the staff to determine the need for i additional action, if any, on Y2K readiness for nuclear power plants. l In accordance with 10 CFR 2.790 of the NRC's " Rules of Practice," a copy of this letter and its enclosure will be placed in the NRC Public Document Room. Please contact me at (301) 415-1420 if you have any questions or comments on the enclosed audit report.

Sincerely, l

Richard B. Ennis, Project Manager  ;

Project Directorate 1-2 l Division of Reactor Projects - 1/11 l Office of Nuclear Reactor Regulation Docket No. 50 354 I

Enclosure:

As stated cc w/ encl: See next page l

t

. .____ _-_m. _ . . _ _ _ _ . . _ . _ . _ . . _ _ . . . - _. . _ . . _ . _ . .. ._ _ _. . -_ __ _. ...__.__.

t l

DISTRIBUTION: r Docket File (50-354)

PUBLIC  ;

I PDl-2 Reading File l l OGC' ACRS i J. Zwolinski <

R. Capra R. Ennis

' P. Milano T. Clark '

J. Wermiel M. Chiramal .

M. Waterman l M. Gareri A. Della Greca, Rl  ;

A. Hansen  !

J. Linville, RI I i

f 1

(

I t

I i

i 1

i t

i 1

2 Mr. Harold W. Keiser Hope Creek Generating Station Public Service Electric & Gas Company cc: ,

Jeffrie J. Keenan, Esquire Manager- Joint Generation Nuclear Business Unit- N21 Atlantic Energy P.O. Box 236 6801 Black Horse Pike Hancocks Bndge, NJ 08038 Egg Harbor Twp., NJ 08234-4130 Hope Creek Resident inspector Richard Hartung U.S. Nuclear Regulatory Commission Electric Service Evaluation Drawer 050g Board of Regulatory Commissioners Hancocks Bridge, NJ 08038 2 Gateway Center, Tenth Floor Newark, NJ 07102 Mr. Louis Storz Sr. Vice President - Nuclear Operations Lower Alloways Creek Township Nuclear Department c/o Mary O. Henderson, Clerk P.O. Box 236 Municipal Building, P.O. Box 157 Hancocks Bridge, NJ 08038 Hancocks Bridge, NJ 08038 G%f al Manager- Hope Creek Operations Mr. Elbert Simpson Hc,ps Creek Generating Station Senior Vice President-P.O. Box 236 Nuclear Engineering Hancocks Bridge, NJ 08038 Nuclear Department P.O. Box 236 Director- Ucensing Regulation & Fuels Hancocks Bridge, NJ 08038 i Nuclear Business Unit- N21 l P.O. Box 236  !

Hancocks Bridge, NJ 08038 Regional Administrator, Region i U.S. Nuclear Regulatory Commission  !

475 Allendale Road  !

King of Prussia, PA 19406 Dr. Jill Lipoti, Asst. Director Radiation Protection Programs NJ Department of Environmental Protection and Energy CN 415  !

Trenton, NJ 08625-0415

a uru p 'e UNITED STATES g j NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 2066H001 p j

%*****/g  !

l 1

l U.S. NUCLEAR REGULATORY COMMISSION l

OFFICE OF NUCLEAR REACTOR REGULATION (NRR)

AUDIT REPORT ON l IMPLEMENTATION OF GENERIC LETTER (GL) 98-01 1

" YEAR 2000 READINESS OF COMPUTER SYSTEMS AT NUCLEAR POWER PLANTE" Docket Nos: 50-354 License No: NPF-57 Licensee: Public Service Electric & Gas Company (PSE&G)

Facility: Hope Creek Generating Station Location: Hancocks Bridge, NJ Dates: October 26 - 29,1998 Audit Team Members: Michael Waterman, NRR Mario Gareri, NRR Aniello Della Greca, DRS, Region i Approved by: Jared Wermiel, Chief Instrumentation and Controls Branch Office of Nuclear Reactor Regulation 5

Enclosure

EXECUTIVE

SUMMARY

From October 26 through 29,1998, the NRC staff conducted an audit of the Year 2000 (Y2K) readiness program at the Hope Creek Generating Station (Hope Creek) in accordance with the audit plan (Attachment 1) for this activity. The purpose of the audit was to (1) assess the effectiveness of the Public Service Electric and Gas Company (PSE&G, the licensee) programs for achieving Y2K readiness, including continued safe operation of the plant as well as compliance with applicable NRC regulations and license conditions with respect to potential Y2K problems, (2) evaluate Y2K program implementation to assure that the licensee's' schedule is in accordance with NRC Generic Letter (GL) 98-01 guidelines for achieving Y2K readiness by i 1

July 1,1999, and (3) assess the licensee's contingency plans for addressing risks associated with potential events resulting from Y2K problems. The audit team reviewed selected licensee documentation regarding the Hope Creek Y2K readiness program and conducted interviews with the cognizant licensee personnel. The results of this audit and subsequent audits at other selected plants will be used by the staff to determine the need for additional action, if any, on l Y2K readiness for nuclear power plants.

l Based on the staff's assessment and evaluation of the Hope Creek Y2K readiness program, the following observations were made:

1. PSE&G has a common Y2K project implementation plan which establishes the scope and control of the Nuclear Business Unit (NBU) Y2K Project Plan at the Hope Creek plant. The Y2K Project Plan is comprehensive and incorporates the major elements of the nuclear power industry Y2K problem guidance contained in Nuclear Energy institute (NEI)/ Nuclear Utilities Software Management Group (NUSMG) 97-07, " Nuclear Utility Year 2000 Readiness."

2. The Hope Creek Y2K program is receiving appropriate management support and oversight.

3. The licensee began the formal Hope Creek Y2K readiness program on August 1,1997, and finished the plant inventory and initial assessment phase on April 15,1998. The  ;

detailed assessment phase was begun January 19,1998. The detailed assessment phase for mission critical information systems and information technology was ,

completed September 15,1998. The detailed assessment phase for the mission l critical digital systems (embedded components) is scheduled to be completed February 24,1999. The licensee has established a tightly controlled schedule for completing the Y2K readiness implementation phase for mission critical systems and components by September 30,1999. The Y2K readiness schedule appears to be achievable because of the dedicated effort at this site, the fact that the licensee has already begun modification or replacement of major critical computer systems, and the licensee has received support via information sharing with the Boiling Water Reactor Owners Group, '

Westinghouse Owners Group, EPRI, PECO, the Northeast Energy Alliance (NEA), and PJM Interconnection, L.L.C.

i

4. The licensee has started the PSE&G NBU Y2K contingency planning. The licensee is using the nuclear industry guidance in NEl/NUSMG 98-07, " Nuclear Utility Year 2000

' Readiness Contingency Planning," for this effort and is in the process of integrating 4

contingency planning in the PSE&G NBU Year 2000 Project Plan. With proper integration of this effort in the project implementation plan and schedule, the licensee should be able to complete this effort by September 30,1999.

5. The licensee is conducting confirmatory testing of mission critical system Y2K compliance at the plant site, regardless of vendor certifications. The licensee stated that this was necessary because some vendor certifications of Y2K compliance have not been reliable.

6. The licensee will be implementing a program in which Y2K project team members will routinely evaluate documentation packages prepared by other team members. For example, Y2K team members involved in digital systems Y2K compliance will review packages prepared by information systems /information technology (IS/IT) Y2K team members. This approach is commendable and appropriate.

I

7. The licensee's independent assessment of the PSE&G NBU Y2K project plan was I thorough. Recommendations from that assessment are being addressed by the 4

PSE&G NBU Y2K project team for subsequent revisions of the PSE&G NBU Y2K project plan.

8. Materials management is being addressed aggressively at the PSE&G Enterprise level.

The licensee recognizes that the Y2K checks and balances now in place should be i

proceduralized for future material control.

I 1

9. Contingency planning for reactor operations is critical. Senior reactor operators (SROs) from Hope Creek and Salem have been assigned to help with Y2K contingency planning; however, a dedicated full-time lead for intemal contingency planning i apparently is not on board. This could impact contingency planning schedules. The l licensee is pursuing this issue.

I ii e- e r -- - e- q - ---

REPORT DETAILS

1.0 INTRODUCTION

The objectives of the Hope Creek Y2K readiness program audit were to:

1. Assess the effectiveness of the PSE&G NBU (the licensee) program for achieving Y2K l readiness, including continued safe operation of the Hope Creek plant as well as l compliance with applicable NRC regulations and license conditions with respect to potential Y2K problems,

2. Evaluate Y2K program implementation to assure that the licensee's schedu!e is in accordance with NRC GL 98-01 guidelines for achieving Y2K readiness by July 1 ,

1999,and

3. Assess the licensee's contingency plans for addressing risks ass =iated with potential i

events resulting from Y2K problems.

l The audit was conducted in accordance with the established audit plan outline (Attachment 1)  ;

which was based in part on the guidance and requirements contained in the following i documents:

l i

e GL 98-01, " Year 2000 Readiness of Computer Systems at Nuclear Power Plants" e Licensee Response (s) to GL 98-01 l

• Plant technical specifications and license terms and conditions l e Applicable NRC regulations j e NEl/NUSMG 97-07, " Nuclear Utility Year 2000 Readiness"

• NEl/NUSMG 98-07, " Nuclear Utility Year 2000 Readiness Contingency Planning" l Prior to the audit at the plant site, the audit team obtained and reviewed the PSE&G NBU Y2K e

Project Plan, Rev. 3, dated June 1998. (Document Number 2 listed in Attachment 2),

The audit process staried with an entrance meeting attended by the PSE&G NBU Year 2000 Project Manager (PM), Y2K project team members, senior and other site personnel, and members of the audit team. Attachment 3 is a list of the attendees. The PM and members of the project team described project organization, the project plan and its implementation, project status, and ongoing activities.

Following the meeting, the audit team spent the rest of the audit reviewing the project plan and its associated procedures the plan implementation products (documents and data bases) and interacting with the project manager and project team members The documents reviewed and

referenced in this audit are listed in Attachment 2.

l

b 2

2.0 HOPE CREEK Y2K PROJECT DESCRIPTION i

2.1 Project Oraanization '

The Salem-Hope Creek project manager (PM) for the PSE&G NBU Year 2000 Project is Mr. R.

Moore. The PM reports to the NBU Director of Special Projects (Mr. G. Overbeck) and to the NBU Senior Vice President, Nuclear Engineering (Mr. E. Simpson), who reports to the NBU Chief Nuclear Officer, who reports in turn to the PSE&G Enterprise Chief Executive Officer (CEO), who is the Enterprise Y2K sponsor. The PM stated that aggressive sponsoring efforts by Mr. Simpson and Mr. Overbeck have been critical to the success of the NBU Y2K program.

The PSE&G NBU Y2K readiness program has 30 full-time equivalent persons and an l additional .9 part time persons supporting selected portions of the Y2K project. The licensee is also contracting with General Electric and Westinghouse for support. The Y2K project scope l

includes both Salem Nuclear Generating Station, Units 1&2 and Hope Creek.

l The licensee is currently merging many of the separate programs and processes at Hope l Creek and Salem into single organizations to serve both plants. Additionally, the Y2K project l

at Hope Creek includes software and embedded systems at Salem. Consequently, many of

} the Hope Creek Y2K efforts reviewed by the staff also apply to Salem.

The licensee participates with other organizations that are addressing the Y2K problem. The licensee has been involved with the Boiling Water Reactor Owners Group, Westinghouse l

' Owners Group, EPRI, PECO, the Northeast Energy Alliance (NEA), and PJM Interconnection, l L.L.C. to share information on Y2K issues. l l 2.2 Proiect Plan The PSE&G NBU Y2K project plan (Item 2 of Documents Reviewed) was developed by the i licensee to establish the scope and control of the Y2K program in the licensee's NBU, and is l applied uniformly at both PSE&G nuclear power plants. The PSE&G NBU Y2K project plan is based on the guidance provided in NEl/NUSMG 97-07, which was accepted by the NRC in NRC Generic Letter 98-01 as guidance that presents one approach for achieving Y2K readiness. The audit team's review confirmed that the PSE&G NBU Y2K project plan is based l on the guidance contained in NEl/NUSMG 97-07.

The PSE&G NBU Y2K project plan consists of six phases. The identification (Inventory and l

Initial Assessment) Phase comprises the elements of awareness, inventory, categorization, and classification. The Detailed Assessment Phase comprises the elements of prioritization, analysis of initial assessment, vendor evaluations, utility owned or supported software i evaluations, interface evaluations, and remediation planning. The Resolution Phase comprises the elements of remediation, verification, and contingency planning. The Validation 3

Phase comprises the elements of Y2K validation testing, and contingency planning for systems failing Y2K tests. The implementstion Phase comprises the elements of planning and l performing final implementation in the plant, initiating contingency plans and compensatory

actions for failed or deferred acceptance tests, and decommissioning or retiring obsolete

l l

3 software and systems. The Closeout Phase comprises the activities that document completion of the Y2K activities for each system. The Y2K project plan also includes requirements for quality assurance, regulatory considerations, and documentation. Appendix K of the NBU Y2K project plan provides a cross reference to NEl/NUSMG 97-07. The PSE&G NSU Y2K project implementation schedule is provided in Table 1.

2.2.1 Awareness The NBU Y2K program, which addresses both Hope Creek and Salem, was initiated in August 1997. Part of this program phase included the NBU Communication Plan: Y2K, Rev. 2, dated June 6,1998. The communication plan (Appendix I to the Y2K project plan) describes the actions necessary to inform NBU management and employees of project status, and interfaces with the PSE&G corporate Y2K compliance effort and extemal organizations. The communication plan prescribes actions for briefing NBU management, educating the general j population of NBU personnel via departmental newsletters, training personnel who are to l

perform inventory and assessment activities, coordinating Y2K team communications, and reporting team progress to NBU management.

Based on communication samples reviewed by the audit team, the staff concluded that the PSE&G NBU has an effective Y2K awareness program.

2.2.2 Initial Assessment The initial assessment part of the Identification (Inventory and initial Assessment) Phase started on August 1,1997, and was completed on April 15,1998. The completed initial assessment resulted in the identification of an inventory of the software applications and embedded system components at the Hope Creek and Salem plants. The tasks of initial assessment included: (1) awareness, (2) inventory, (3) categorization, and (4) classification.

The inventory of potentially impacted applications, computer systems and hardware, and embedded plant systems / components was developed by plant personnel familiar with and responsible for each plant system / functional area using the NBU Y2K project plan. For software applications, the licensee used the inventory developed for the plant software quality assurance manual as a means for gaining additional confidence in the completeness of the inventory. Some software systems are being evaluated by the PSE&G Enterprise [ corporate]

Y2K project team, which supports the NBU Y2K project team through a formal memorandum of understanding.

In the identification of embedded systems, the licensee reviewed the procedures and documentation for occurrences of phrases that would indicate the existence of an intemal clock or processor, surveyed the vendors for information on their equipment, performed system walk-downs, and reviewed schematics, program listings, and reference manuals on

various instrumentation and control systems. The results of the initial assessment of the software applications and embedded items were placed in the NBU and Enterprise Y2K data bases.

The total inventory of potentially affected mission critical systems and components at the Hope i

i

4 i

4 Creek and Salem plants is 3010, including - 34 information system /information technology (IS/IT) applications, 33 stand-alone IS/IT applications, 666 infrastructure IS/IT applications, 40 digital systems, and 2237 digital components. Table 2 provides the results of the inventory of software, and Table 4 lists the mission critical digital systems. Of the 67 IS/IT applications,32 have been verified to be Y2K compliant or ready, or are being retired. Of the 666 infrastructure IS/IT applications,434 have been verified to be Y2K compliant or ready, or are  ;

being retired. Of the 40 mission critical digital systems, 5 have completed the resolution l phase. Of the 2237 mission critical digital components,1387 have been verified to be Y2K l compliant, ready, or are being retired. The balance of the systems and components in these categories is undergoing continued detailed assessment. Table 5 lists the embedded systems and components that were reviewed by the audit team.

Prioritization The inventory includes a prioritization of the identified items. The priority is based on mission criticality and importance of the functions being performed. The licensee defines mission critical systems or components as those systems or components that directly impact nuclear safety, power production, or current licensing bases. The licensee used risk assessment methods to prioritize each inventoried item as high, medium, or low. High priorO ,lems are items that must be verified to be Y2K compliant or ready by the deadline date. Iwedium priority items are those items that should be verified to be Y2K compliant or ready by the deadline date. Low priority items are discretionary items that may be verified to be Y2K compliant or ready by the deadline date. The licensee's prioritization process encompasses the criteria described in NEl/NUSMG 97-07.

Table 2 lists the prioritization of the inventoried IS/IT applications. The high priority items in Table 2 are mission critical. Table 3 provides a list of IS/IT mission critical systems and components reviewed by the audit team. Table 4 lists the inventory of mission critical digital systems in Salem Unit 1, Salem Unit 2, systems common to Salem Units 1 and 2, and Hope Creek. A total listing of digital components is too extensive, and is therefore not provided in this report. Table 5 lists the digital systems and :omponents that were reviewed by the audit team.

Analysis of identification Phase Activities The licensee completed the identification phase on April 15,1998. During the analysis of the initial assessment, the licensee evaluated the failure risk of each item as the basis for assigning the priority; recommended the approach / plan for detailed assessment, testing, and remediation; and estimated the detailed assessment /remediation cost. Unless specifically noted otherwise, the licensee did not formally assess and remediate low priority items.

Remediation of these items will be done as time and resources permit. Items that are to be replaced or made obsolete before January 1,2000, were not assessed and no remediation actions will be taken.

NRC Audit Team Assessment The audit team reviewed in detail 21 of the 67 mission critical IS/IT applications identified in m - . .-- ,e ,- , , - . . ,

. . - - - . ~ - - _ - . .. - - - - .__- - .. . - . . . - - - -

l 5

Table 2 (see Table 3). Of the 40 mission critical digital systems (see Table 4) and 2237 mission critical digital components identified by the licensee, the audit team reviewed 23 (see Table 5).

The audit team found the NBU Application Assessment questionnaire was only partially  !

completed in several IS/IT packages. Three packages have progressed into at least the '

Detailed Assessment Phase, were prioritized as high priority mission critical, yet the documentation in the packages did not explicitly identify the packages as such. Additionally, Failure Modes and Effects Criticality Analysis documentation was not included in these three i

packages. The project manager stated that these forms were not used, and the project plan l was to be updated to reflect this change.

l In assessing determinations of risk, the audit team used as audit criteria the licensee's basic Y2K methodology, which was described in Appendix B to the NBU Y2K project plan. The IS/IT Y2K processes were specified in Appendix C. The audit team determined that Appendix C to NBU Y2K Project Plan (Rev. 3), "Information Systems and Technology (IS/IT)" was not consistent with the processes or focus used for assessing digital systems Y2K risks. The IS/IT processes address risk assessment as a management tool for providing standard, periodic high-level reporting on the level of risk for a number of risk drivers and gives a snapshot of the progress made in resolving specific system Y2K problems. Appendix B, " Basic Y2K l Methodology," uses risk assessment in terms of the impact on power production, current licensing basis, or nuclear safety. The licensee stated that this difference in defining risk has been noted, and Appendix C is to be changed to be consistent with the processes used for the digital systems risk evaluations per Appendix 8.

The licensee is reviewing the impact of the above discrepancy and will correct the Y2K system assessment packages as necessary to reflect the documentation requirements of Appendix B.

Nevertheless, the licensee's documentation was sufficient forjustifying the results of the initial assessments of prioritization/ categorization and Y2K compliance /non-compliance.

2.2.3. Detailed Assessment Detailed assessment is performed for all high and medium priority items consisting of vendor evaluation, interface ovaluation, spare parts evaluation, training system evaluation, test plans /results, subject matter experts review, and results. Vendor evaluation encompasses evaluation of available manufacturer /developerinformation-(such as contracts, correspondence, vendor manuals, Intemet listings, and vendor owners groups),

communication with vendors using the corporate vendor management program standard vendor questionnaire, and direct communication with vendors. Interface evaluation includes both intemal and extemal system interfaces for passing date- and time-related data. Test plans /results involve the development of test procedures and acceptance criteria to determine if a Y2K date problem exists. Spare parts evaluation involves the review of items to ensure that Y2K problems are not recurrent after remediation efforts are completed. Training systems evaluations address training systems that duplicate plant systems, such as the plant simulator and considers the impact of Y2K problems on simulator existing equipment and upgrades.

Subject matter expert review addresses the use of any other relevant information obtained i

through walkdowns, review of embedded chips, and review of source code. Results of the l assessments provide remediation recommendations and justifications.

- . - . - - . - - . - .~. - - -_ - . - ~ - - - - - -. - - - - _ -

1 .

i 6 4

l The licensee is scheduled to complete its detailed assessment in December 1998. To date,

detailed assessment has been completed on all mission criticalIS/IT applications, stand alone j

applications, and infrastructure systems and components. Detailed assessments have been completed on 21 of 40 mission critical digital systems, and 1387 of 2237 mission critical digital

} components (Salem and Hope Creek). Based on the audit team's review of completed items, the audit team found that the detailed assessments performed to date follow the project j

implementation plan. As with the inventory and Initial Assessment Phase, the audit team i

found some documentation deficiencies; however, the licensee's documentation was sufficient

forjustifying the results of the detailed assessments, which established Y2K compliance /non-

compliance.

2.2.4. Y2K Testina and Validation 1

~ The licensee is testing all mission critical IS/IT systems, digital systems, and digital  !

components, regardless of vendor Y2K compliance certifications. This approach is based upon the licensee's experience with changing vendor certifications over the past year. The l audit team noted that the decision on whether to perform additional embedded component Y2K problem susceptibility testing is based on the licensee's determination of importance of j the affected system and knowledge of the item, prior experience with the vendor, and other i relevant information obtained.

1 The licensee's Y2K project team and Y2K contractors are performing all testing to ensure

consistency in the implementation of Y2K susceptibility testing. In reviewing some of the i testing that has been completed, the audit team found inconsistencies in test acceptance i criteria. On Page 25 of Phase 11, Environmental Computer Based Training User Acceptance i Test Package for Sequencing 99,00,01, the reason for test failure stated, "Since Access 97 windows from 00-29 and 30-99, the application fails to recognize the proper date sequence l past the year 2029." The audit team found that the person performing the test interpreted '99' 4

to be 2099 instead of 1999, which placed the '99' date outside the Access 97 windowing, and hence caused the test to fail. In this case, testing for 2099 was not applicable for Hope Creek or Salem because these plants are not licensed to operate beyond 2026 (2046 with a license extension for Hope Creek) or 2008 (2028 with license extension for Salem), respectively.

! Consequently, testing for a rollover from 2099 to 2100 was not appropriate. Similar test discrepancies were noted in other test packages.

i As another example, in the Data integrity testing, the Y2K test failed. The test report stated p

that the Access 97 windowing problem caused the failure because the proper century is not j represented beyond the year 2030 or before the year 1929. The year 2000 falls between

these dates, and should not have failed this test. The licensee reviewed this test and j

determined that the test for data integrity tested the year date for all years in the range 2000 to j 2099. Again, this range of dates was not appropriate for the Hope Creek and Salem plants.

4 The licensee is reviewing all test documentation to ensure that the testing for rollover dates is within the time and date domain of the applications. The staff concludes that it is important for licensees to ensure that, when using contractors to perform Y2K testing, the contractors are

familiar with the domain in which the applications will operate.

l, 4

I

7 2.2.5, Remediation Remediation is the process of retiring, replacing or modifying software or embedded software  !

devices that are to be retained in sarvice, but have been determined to be affected by the Y2K problem. The program implementation plan provided Y2K compliance criteria for replacement or modification. After remediation is completed, validation testing is required. The licensee is performing the required Y2K remediation validation testing using existing plant procedures for l

digital system upgrades. The final documentation of the detailed assessment and remediation is the Y2K certification package.

2.2.6. Reaulatory Considerations The PSE&G NBU Y2K project plan and associated documents include references to existing plant procedures that have guidance on regulatory considerations, such as applicability of 10 CFR 50.59 for plant modification reviews, reportability evaluations per 10 CFR 50.72,10 CFR 50.73, and 10 CFR Part 21, and operability determinations as required by plant technical specifications.

2.2.7. Continaency Plannina The licensee has begun contingency planning using a framework similar to that described in NEl/NUSMG 98-07, " Nuclear Utility Year 2000 Readiness Contingency Planning." The contingency plan for Hope Creek mission critical digital systems and components is scheduled for completion on March 24,1999. Contingency planning for mission critical IS/IT systems is scheduled for completion on January 19,1999. The proposed contingency planning activities are consistent with the guidance of NEl/NUSMG 98-07. If at the completion of the detailed assessment phase the integrated plan is implemented with the same rigor as presently applied, the audit team believes that the licensee's schedule for completion of the project -

contingency plans can be achieved.

2.2.8. Y2K Proaram Manaaement The licensee's Y2K program schedule is aggressively tracked on a continuous basis by corporate and site management. The Y2K program progress is summarized in a format that defines the progress of each Y2K system being evaluated. Additionally, the licensee summarizes the status of each phase for corporate management review. From discussions with licensee personnel, schedule slippages have resulted in immediate corporate management attention. At present, there are no schedule slippages in the Y2K program in the NBU.

2.2.9. Electric Grid issues The audit team discussed electric grid issues with the licensee. The licensee stated that the PJM Interconnection, L.L.C. (PJM), to which Hope Creek and Salem provide their power, is a limited liability company govemed by a Board of Managers. The PJM has initiated activities to address grid reliability with respect to the Y2K problem. PJM is the largest centrally-dispatched

electric control area in North America. The PJM electrical grid includes all or part of

8 I

Pennsylvania, New Jersey, Maryland, Delaware, Virginia, and the District of Columbia. There are approximately 200 power producers on the grid, with a total capacity of 56,000 MW, of '

which approximately 13,000 MW are produced by nuclear power plants. PJM centrally coordinates the operation of these power producers by monitoring in 3-second intervals the operating status of each power producer on the grid.

)

The PJM also has tie lines into other grids, such as the grid serving New York City. These tie lines are closely monitored to ensure large increases or decreases in load demand do not affect the PJM grid.

The PJM contingency plans are based on the impact of changes in power production and the probability of changes in power production or demand. During the Y2K changeover period, PJM will have, online, contingency response procedures for the 20 most significant power reduction scenarios. The contingency plans are intended to be consistent between utilities in the region to ensure one utility will not adversely impact others. The contingency plans will consider probable and credible worst-case scenarios for extemal as well as intemal events.

To ensure sufficient electrical energy resources are available, the PJM utilities and power producers will operate at 75% capacity during the Y2K changeover. This will provide the PJM grid with 25% reserve load to handle load changes on the PJM grid and at the tie lines to other grids.

3.0 AUDIT TEAM OBSERVATIONS The following observations were made by the team auditing the PSE&G NBU Y2K readiness program:

1. PSE&G has a common Y2K project implementation plan which establishes the scope and controls of the NBU Year 2000 Program at the licensee's two nuclear power plants (Hope Creek and Salem). The NBU Year 2000 Readiness Program is comprehensive and incorporates the major elements of the nuclear power industry Y2K problem readiness guidance contained in Nuclear Energy Institute (NEI)/ Nuclear Utilities Software Management Group (NUSMG) 97-07, " Nuclear Utility Year 2000 Readiness."

2. The NBU Y2K readiness program is receiving appropriate management support and oversight. Effective communication of issues and program status has been established at alllevels of management.

3. The licensee began the formal Hope Creek Y2K readiness program on August 1,1997, and finished the plant inventory and initial assessment phase on April 15,1998. The detailed assessment phase was begun January 19,1998. The detailed assessment phase for mission critical information systems and information technology was completed September 15,1998. The detailed assessment phase for the mission critical digital systems (embedded components) is scheduled to be completed February 24,1999. The licensee has established a tightly controlled schedule for completing the Y2K readiness implementation phase for mission critical systems and components by September 30,1999. The Y2K readiness schedule appears to be achievable because of the dedicated effort at this site, the fact that the licensee has already begun

! 9 l modification or replacement of major critical computer systems, and the licensee has received support via information sharing with the Boiling Water Reactor Owners Group, Westinghouse Owners Group, EPRI, PECO, the Northeast Energy Alliance (NEA), and PJM Interconnection, L.L.C.

i

4. The licensee has started the PSE&G NBU Y2K contingency planning. The licensee is using the nuclear industry guidance in NEl/NUSMG 98-07, " Nuclear Utility Year 2000 Readiness Contingency Planning," for this effort and is in the process of integrating contingency planning in the PSE&G NBU Year 2000 Project Plan. With proper integration of this effort in the project implementation plan and schedule, the licensee should be able to complete this effort by September 30,1999.

5. The licensee is conducting confirmatory testing of mission critical system Y2K compliance at the plant site, regardless of vendor certifications. The licensee stated that this was necessary because some vendor certifications of Y2K compliance have not been reliable.

6. The licensee will be implementing a program in which Y2K project team members will routinely evaluate documentation packages prepared by other team members. For example, Y2K team members involved in digital systems Y2K compliance will review packages prepared by IS/IT Y2K team members. This approach is commendable and appropriate.

7. The licensee's independent assessment of the PSE&G NBU Y2K project plan was thorough. Recommendations frcm that assessment are being addressed by the PSE&G NBU Y2K project team for subsequent revisions of the PSE&G NBU Y2K project plan.

8. Materials management is being addressed aggressively at the PSE&G Enterprise level.

The licensee recognizes that the Y2K checks and balances now in place should be proceduralized for future material control.

9. Contingency planning for reactor operations is critical. Senior reactor operators (SROs) l from Hope Creek and Salem have been assigned to help with Y2K contingency planning; however, a dedicated full-time lead for intemal contingency planning apparently is not on board. This could impact contingency planning schedules. The licensee is pursuing this issue.

Table 1 PSE&G NBU Y2K Project Implementation Schedule Table 2 PSE&G NBU IS/IT Inventory Table 3 IS/IT Systems and Components Reviewed by Audit Team Table 4 Inventory of Mission Critical Digital Systems Table 5 Digital Systems and Components Reviewed by the Audit Team Attachment i Hope Creek Y2K Audit Plan Outline Attachment 2 Documents Reviewed

Attachment 3 Entrance Meeting - Attendees

l Table 1 - PSE&G NBU Y2K Project Implementation Schedule l

Activity Startina Date Finishina Date identification 8/1/97 4/15/98 (Inventory and initial Assessment)

Detailed Assessment 4/20/98 2/24/99 Resolution and Contingency Planning 5/18/98 5/24/99 Validation and Change Planning 8/21/98 7/21/99 Outage - 2/21/99 4/11/99*

Implementation 10/1/98 9/30/99 Closeout 10/1/99 2/10/00" l

Some system implementations will be scheduled during the 2/21/99 outage.

The PSE&G NBU Closeout activities involve only completion of required documentation.

j i

i 1

l

~

l Table 2 - PSE&G NBU IS/IT Inventory Priority Line of Business High Medium Low Total Business Process Reengineering 6 0 0 6 Communications 0 0 1 1 i l Design Engineering 11 26 216 253 Eng Assurance / Spec Projects 0 2 6 8 Hope Creek - Operations 4 10 48 62 Human Resources 0 0 2 2

information Technology 27 23 48 98 Licensing / Reg & Fuels 5 60 29 94 Maplewood Labs 1 6 2 9 Medical 0 1 0 1 Nuc Business Support 1 10 8 19 QA/Nuc Trng/Emerg Prep 4 14 39 57 Salem - Operations 5 11 9 25 Site Maintenance 2 13 28 43 System Engineering 1 11 10 22 Total 67 187 446 700 I l

l 1

I l

I i

1 l

l l

- ,- ._ . . . ~ _ . _ _ . _ _ ... _ _ . _ _ _ . _ _ _

Table 3 -IS/IT Systems and Components Reviewed by Audit Team Software Applications File Number Name Phase Completed ,

100076 Cable and Raceway Tracking System ll 100263 Incore ll 100353 Nuclide Identification 11

! 100364 Outage Risk Assessment and Management il  !

100372 P1Back ll 100449 RCS Mass Leak Rate Correction Calculation ll i: 100662 Whole Body Count 11 100295 Material Access Control Center IV >

100029 AirCEt Easytest Diagnostics V 100118 Critical Software Media for ME101 Baseplate Program V 100124 Critical Software Media fotME035 Baseplate Program V 100127 Critical Software Media for ME150 Baseplate Program V 100138 Cymcoord V 100183 Engineering Work Management Systems (including SIDS V Online 100186 Environmental Computer Based Training V 100259 InterPlot V l 100443 Radman V 100458 Respirator Fit System V 100473 Ryansoft V 101344 HX-BAL V

, 102003 Fitness for Duty Computer Based Training V l

l-- Phase 11 - Detailed Assessment Phase IV - Validation l Phase V-Implementation i

l 4

~

2 Table 4 - Inventory of Mission Critical Digital Syse ms System Description Salem 1 Salem 2 Co mon Cre k Digital Feedwater Control System X Advanced Digital Feedwater Control System X X Oscination Power Range Monitor X Salem Switchyard Data Acquisition and Monitoring X X Meteorological Data Acquisition and y

Monitoring Nuclear Security System X X Overhead Annunciator System X X Fire Detection and Alarm System X GE Transient Analysis and Recording System X Plant Event and Parameter Archiving y-System Service Water PLC Monitoring Systern X X Moisture Separator and Reheater PLC Indicating System X X Leading Edge Flowmeter X X Radiation Monitoring System X j pdiatiari Monitoring System X l Nuclear Steam Supply System Process Coriiputer X Emergency Responte Data System X X Dose Assessment Dats faterface System X X Control Room integrated Display System X ,

Safety Parameters Display System X X l Plant Computer Monitoring and Alarm System X X l

Auxiliary Annunciator System X X l Waste Gas Monitoring and Contro!

X X Recorder Measuring and Test Equipment X X Training Center Simulator Computer Systems X X Total 9 10 7 14 l

Table 5 - Digital Systems and Components Reviewed by the Audit Team Digital Components and Systems File Number Name Phase Completed D00062B1 Radiatien Monitoring / Leak Detection 11 D0013281 EmergEdy Response Data System. ERDS ll D00133B1 Dose Assessment Data Interface System ll D00145B1 M&TE Astro-Med Recorder il D0014781 M&TE Astro-Med Recorder 11 D00148B1 M&TE Astro-Med Recorrier 11 D0015081 M&TE: Dranetz; Power Quality Analyzer il D0015181 M&TE: Dranetz; Power Quality Analyzer, Mdl 8000-2 ll D00159B1 M&TE: HEISE, Pressure Calibrator ll D0016081 Safety Parameter Display System ll D00164B1 M&TE: Datron/Wavetek, Digital Multimeter il D0016581 M&TE: Datron/Wavetek, Digital Multimeter il D00171B1 M&TE: Panmetrics, Ultrasonic Flowmeter, Mdl PT868 ll D00172B1 M&TE: FLUKE, Calibrator, Model 5500A 11 D00173B1 M&TE: FLUKE, AC Measurement Standard, Model 5790A 11 D0017381 M&TE: FLUKE, Calibrator, Model 5790A 11 D00177B1 M&TE:OPTALIGN, Model AL2-610-LR 11 D0018881 Simulator System 11 D00194B1 M&TE: Yokogawa, Vortex Flowmeter, Mdl YF105 ll D00196B1 Digital Feedwater Control System 11 D00200B2 Alert Notification System 11 D0046B1 NSSS Process Computer il D0047B1 Control Room Integrated Display System (CRIDS) 11 Phase 11 - Detailed Assessment l

I

Hooe Creek Y2K Audit Plan Outline A. Project organization B Project Manager-C. Project Sponsor-

1. Participation in BWR Owners Group, CDSV group activities related to the Y2K effort, (EPRI, NEI). Peer review efforts thru CDSV group. EPRI participation for embedded systems.

2. Corporate activities

3. Schedule of activities for Y2K readiness Activity Starting Date Finishing Date Communication /

Awareness Project Plan inventory Detailed analysis / testing Remediation Validation / testing Contingency Planning

4. Inventory (Review the Information Database.)

Classification:

5. Analysis -

Number of items identified as Y2K compliant. Review how this was determined - Vendor data; any additional testing.

Number ofitems not Y2K compliant -

Accept As is: (Review how this was determined. require validation testing. Check vendor data, Owners Group data, any testing by vendor?)

ATTACHMENT 1

l Eliminate Fix Replace a) Vendor evaluation - validation testing based on criticality of item, prior experience with vendor, extent of documentation, or plant knowledge of the item b) Plant owned or supported software (including tools) evaluation - knowledge based decisions, scanning, testing. When testing proposed, need test specifications and  ;

procedures. '

) Interface evaluation - Part of corporate plan (?) Grid, substation, communication, d) Embedded components evaluation - knowledge based decisions and testing. When ,

sufficient vendor and plant information is available tb support a knowledge-based i decision, no additional testing is required. (Review the documents when this is the '

case.) -

6. Remediation - Use of existing software procedures (?). Verify long-term commitments for l maintaining Y2K readiness.

7. Y2K Testing and Validation l

l Assessment testing - Per Computer problem / change reports (PCRs) and associated V&V plans and test procedures.

l Testing subsequent to remediation - unit testing; integration testing; system testing.

8. Regulatory Considerations - 10 CFR 50.59 reviews; reportability evaluations per 10 CFR l 50.72,50.73 and part 21; operability determinations.

t

9. Contingency Planning - NEl/NUSMG 98-07, GAO/AIMD -10.1.19 Intemal Risks Extemal Risks Remediation Risks (Vendor support, resource limitations, etc.)

10. Y2K Management Plan-Tracking against milestones of the project. Management awcreness. Status reporting

! Extemal resources l Use of existing procedures for software QA, configuration management, V&V.,

t Documentation j Audits (any audits done/ reports issued).

i l

i

}

1 a

1

l l

Documents _Bayjeggd i

1. NBU Y2K Project Team (Roster of participants)

2. "PSE&G NBU Y2K Project Plan," Rev. 3, June 1998

3. "NBU Y2K Project Self-Assessment Report," October 22,1998

4. PSE&G Status Reporting Package, Period Covering September 1998

5. Appendix C to NBU Y2K Project Plan, Information Systems and Technology (IS/IT) (Draft)

6. NEl/NUSMG 97-07, " Nuclear Utility Year 2000 Readiness," dated October 1997.

7. NEl/NUSMG 98-07, " Nuclear Utility Year 2000 Readiness Contingency Planning," dated August 1998.

8. NRC Generic Letter No. 98-01, " Year 2000 Readiness of Computer Systems at Nuclear Power Plants," dated May 11,1998.

9. "PSE&G NBU integrated Planning and Prioritization," NC.NA-AP.ZZ-0082(Z), Rev.1, dated June 24,1998 i

10. "PSE&G NBU General Y2K Test Plan," NC.Y2-AP.ZZ-0101(Q), Rev.1, dated August 25, 1998

11. "PSE&G NBU General Y2K Test Procedure," NC.Y2-GP.ZZ-0100(Q), Rev.1, dated  !

August 25,1998 l 12. NBU Y2K Audit Entrance Meeting Slide Handouts

13. Elide Handouts, " Operating Plan, Preparations for Y2K" 1

4 1

ATTACHMENT 2

7 l- .

Entrance Meetina - Attendees l October 26.1998 i'

Joanne Carter NBU Y2K Project, IS/IT  ;

Don Crouch Materials Manager Bob Green PSE&G Y2K Project Manager j- Ashok Hasija NBU Y2K Project Engineer l Kathy Master NBU Y2K Project, Admin Support l ' Jim Metro NBU Y2K Project, Digital Lead Peter Moeller Licensing Ray _ Moore NBU Y2K Project Manager i Gary Overbeck Director- Engineering Assessments /Special t

Projects

' Dennis Pease QA/NSR I Becky Recchione Digital '

Glen Rogers Vice President - Information Technology -

PSEG Dave Scull NBU Y2K, IS/IT Jim Shank Design Engineering - Digital Bert Simpson Sr. Vice President - Nuclear Engineering Skip Sindoni Communication Lou Storz Sr. Vice President- Operations Dave West Purchasing

! Anne Williams IS/IT Corp.

Ron Dowdney Y2K Project Scheduling David Garchow Director, Design Engineering

- Tom Gordon IS/IT Jack Gibson PECO Energy Phil Duca Salem Licensing Mike Waterman NRC Mario Gareri NRC Neil Della Greca NRC 9

i i

i i

4 ATTACHMENT 3 d

a b , s , - n - r , - - -