ML17292B598

Audit Rept on Implementation of GL 98-01, YK2 Readiness of Computer Sys at Npps

ML17292B598
Person / Time
Site:	Columbia
Issue date:	03/12/1999
From:	NRC (Affiliation Not Assigned)
To:
Shared Package
ML17292B597	List: ML17292B596 ML17292B598
References
GL-98-01, GL-98-1, NUDOCS 9903250016
Download: ML17292B598 (38)

Text

U.S. NUCLEAR REGULATORYCOMMISSION OFFICE OF NUCLEAR REACTOR REGULATION AUDITREPORT ON IMPLEMENTATIONOF GENERIC LETTER 98-01 "YEAR2000 READINESS OF COMPUTER SYSTEMS AT NUCLEAR POWER PLANTS" Docket No:

, License No:

Licensee:

Facility:

Location:

Dates:

Audit Team Members:

Approved by:

50-397 NPF-21 Washington Public Power Supply System (WPPSS)

WPPSS Nuclear Project No. 2 (WNP-2)

Richland, Washington January 26 - January 28, 1999 Matthew Chiramal, NRR W. Keith Mortensen, NRR Robert Brill, RES Jerry Mauck, Chief (Acting)

Instrumentation and Controls Branch Office of Nuclear Reactor Regulation 99032500th 9903i2 PDR ADOCK 05000397 P

PDR EXECUTIVE

SUMMARY

From January 26 to January 28, 1999, the staff of the U. S. Nuclear Regulatory Commission (NRC) conducted an audit of the Year 2000 (Y2K) program at the Washington Public Power Supply System (WPPSS) Nuclear Project No. 2 (WNP-2). The purpose of the audit was to (1) assess the effectiveness of the programs of WPPSS (the licensee) for achieving Y2K readiness, including continued safe operation of the plant as well as compliance with applicable NRC regulations and license conditions with respect to potential Y2K problems, (2) evaluate Y2K program implementation to ensure that the licensee's schedule follows the NRC Generic Letter (GL) 98-01 guidelines for achieving Y2K readiness by July 1, 1999, and (3) assess the licensee's contingency plans for addressing risks associated with potential events resulting from Y2K problems.

The audit team reviewed selected items from the licensee's documentation regarding WNP-2 and conducted interviews with cognizant licensee personnel.

The results of this audit and subsequent audits at other selected plants will be used by the staff to determine the need for additional action, ifany, on Y2K readiness for nuclear power plants.

On the basis of its assessment and evaluation of the WNP-2 Y2K readiness program, the audit team concluded as follows:

The WNP-2 Year 2000 Project Plan and associated detailed procedures for implementing the plan at WNP-2 are considered to be excellent. They are comprehensive and detailed documents which incorporate the guidance of GL 98-01, "Year 2000 Readiness of Computer Systems at Nuclear Power Plants," and NEI/NUSMG 97-07, "Nuclear UtilityYear 2000 Readiness."

The plan includes the outline of Contingency Planning and associated instructions that are based on the guidance of NEI/NUSMG 98-07, "Nuclear UtilityYear 2000 Readiness Contingency Planning."

WNP-2 is using existing quality assurance (QA) and modification program procedures as well as Y2K-specific plant procedures to achieve Y2K readiness.

The applicable considerations for regulatory requirements and criteria are addressed in the Year 2000 Project Plan implementation activities. The licensee has been sharing data and work experience through activities in the BWR Owners Group Y2K subcommittee; United Service Alliance; Litton Enterprise Systems Readiness database; EPRI conferences, workshops, and database; and NEI and NUSMG workshops.

3.

The project is well organized and adequately staffed with plant and contractor staff.

The Project organization includes a Contingency Planning Lead as the single-point-of-contact for the contingency planning process.

Based on the audit team's interaction with the project staff, the audit team considers the Y2K project staff to be very competent and knowledgeable in the activities they perform.

4.

WNP-2 Y2K readiness including contingency planning is scheduled to be achieved by July 1,1999. The audit team noted that the project emphasis has been on addressing the embedded systems; and the successful completion of activities to achieve their Y2K readiness by July 1, 1999 appears to be on track. However, regarding the Y2K readiness activities of software assets, the audit team echoed the concern on their progress raised by the October 1998 self assessment and the January 1999 quality

audit. The audit team noted the corrective actions planned and taken in response to the recommendation of the assessment teams, particularly the reorganization of the Y2K project in October 1998, its prominence in the organization, the addition of responsibility for software asset activities, and the increased resources provided.

These actions will help in meeting the project schedule of having all critical, high, and medium priority assets Y2K ready by July 1999.

5.

The WNP-2 ficensee has'not identified any Y2K problems with systems needed for safe shutdown of the plant.

6.

Critical to the completion of WNP-2 Y2K project is the completion, on schedule, of the four "related projects" that are separate from the Y2K project. The audit team identified that the Y2K program schedule has no flexibilityto account for unforseen problems in

~

the Y2K readiness activities of the four related projects, and the remaining work on software assets and embedded systems. The Y2K project manager (PM) and project sponsor acknowledged this and stated their intent to address the issue.

Based on the audit team's review of several project work packages of the assets the audit team observed that, although incomplete as certification packages, most of the documentation and contents are consistent and followthe guidance provided by the WNP-2 Y2K Program Plan.

Some inconsistencies were noted and identified to the PM.

The PM intends to feedback as "lessons learned" some of the examples of inconsistencies and, additionally, willrequire thorough reviews during certifications to resolve the inconsistencies.

The PM also plans to issue completed system-level packages for certification starting in March 1999, along with additional guidance to ensure consistency.

4

1.0 INTRODUCTION

The objectives of the audit of the WNP-2 Y2K Program were to:

1.

Assess the effectiveness of the WNP-2 programs for achieving Y2K readiness including continued safe operation of the plant as well as compliance with applicable NRC regulations and license conditions with respect to potential Y2K problems 2.

Evaluate Y2K program implementation to ensure that the licensee's schedule is in accordance with NRC GL 98-01 guidelines for achieving Y2K readiness by July 1, 1999 3.

Assess the licensee's contingency plans for addressing risks associated with potential events resulting from Y2K problems The audit was conducted in accordance with the established audit plan (http://www.nrc.gov/NRC/Y2K/y2kaudit.html), which was based in part on the guidance and requirements contained in the following documents:

GL 98-01, "Year 2000 Readiness of Computer Systems at Nuclear Power Plants" Licensee Responses to GL 98-01

~

Plant technical specifications and license terms and conditions Applicable NRC regulations NEI/NUSMG 97-07, "Nuclear UtilityYear 2000 Readiness" NEI/NUSMG 98-07, "Nuclear UtilityYear 2000 Readiness Contingency Planning" Before the audit at the plant site, the audit team reviewed the WNP-2 Y2K Project information that the licensee had sent in advance to NRC Headquarters.

This information included the WNP-2 Year 2000 Project Plan, Revision 1, dated December 1, 1998; the licensee's response to GL 98-01; an organizational chart; selected Y2K procedures; the Y2K Project schedule; a Y2k status report; samples of documentation packages; and other items of interest to the audit.

team.

The audit process began with an entrance meeting attended by the WNP-2 Y2K Project Manager and other plant personnel and members of the NRC audit team. The WNP-2 Y2K Sponsor was not able to attend the entrance meeting because of a presentation being made to the Washington State legislature regarding the status of Y2K preparations at WNP-2.

The plant Engineering Manager acted for the sponsor at the entrance meeting.

Members of the WNP-2 Y2K organization described the project organization, the project plan, the implementation of the project, and the project's current status.

Attachment 1 is a list of the attendees at the entrance meeting.

For the rest of the audit activity, the audit team reviewed the WNP-2 Year 2000 Project Plan and associated project documentation and interacted with the WNP-2 Y2Kteam personnel.

The documents reviewed by the audit team are listed in Attachment 2.

The audit activity concluded in an exit meeting in which the audit team summarized the results of the audit. Attachment 3 is a list of the attendees at the exit meeting.

The WNP-2 Y2K Project schedule is provided in Table 1.

2.0 WNP-2 PROJECT DESCRIPTION 2.1

'Pro ect Or anization The corporate senior level sponsor for the WNP-2 Y2K Project is the Vice President of Operations Support, R. L. Webring. As a result of a self-assessment activity the project team was reorganized in November 1998.

In the resulting organization G. L. Gelhaus was assigned to be the WNP-2 Y2K Project Manager.

The Y2K Project Manager is responsible for the overall management of the Year 2000 Project.

His responsibilities include; (1) developing the Year 2000 Project Plan, (2) tracking all phases of the project, (3) keeping the senior sponsor current on project progress, (4) developing a Y2K project budget and (5) successful completion of the project.

In addition, the Y2K Project Manager will be the single point of contact for the NRC as identified in GL 98-01. The Y2K Project Manager is supported by the core team. The core team consists of 16 dedicated full-time WNP-2 and contractor employees and the full-time equivalent of approximately.10 additional WNP-2 and contractor contributors.

The core team is responsible for conducting detailed assessments, remediation planning and remediations.

The Y2K core team is supported by the Y2K extended team.

The extended team consists of representatives from affected WPPSS organizations.

The extended team assists the WNP-2 Y2Keffort by:

Acting as a focal point for communications between the Y2K core team and site organizations for which the extended team member is responsible; Coordinating the inventory (hardware, software, and embedded devices), testing and Y2K issue resolution within their organization; Bringing Y2K issues to the attention of the Y2K Project core team; Participating in briefings and training sessions for various Y2K activities; and Aiding in the general awareness of the Y2K problem.

Points of interaction have been designated with line organizations so that help from supporting organizations may be obtained in such areas as remediation planning and contingency planning. A Remediation Review Panel, consisting of the Manager of Operations, Quality Supervisor, Manager of Engineering, and Chief Information Officer has been established to review and concur on any impacts on WNP-2 assets, including recommended modifications, work-arounds, "use-as-is" dispositions, and contingency plans. This concurrence by Site Management of the impact of Y2K on the important assets is sought in addition to the approvals of the asset owner and the Y2K Project Manager.

The objective of the panel is to ensure that site management agrees to the risks and expenditures of resources so that the Y2K Project team does not make the risk and resource decisions without the knowledge of the appropriate management.

In addition four major computer system (replacements) projects titled "Related Projects,"

although managed independently from the Y2Kproject, are tracked by the project manager for Y2K readiness because their completion on time is critical to the success of the Y2K project.

These are: (1) the Control Room Plant Data Information System (PDIS) Project, (2) the PeopleSoft Project, (3) the Client Server Project and (4) the Passport Project.

2.2

~Prc'ect Pian The WNP-2 Year 2000 Project Plan, Revision 1, dated December 1, 1998, is the plant-specific plan that was developed by the licensee.

It is based on the guidance provided in NEI/NUSMG 97-07, which was endorsed by the NRC in NRC GL 98-01 as guidance that when properly implemented presents one approach for achieving Y2K readiness.

The plan was organized around the inventory and assessment processes developed by Litton Enterprise Solutions (LES). The audit team's review confirmed that the WNP-2 Year 2000 Project Plan is based on the guidance contained in GL 98-01 and NEI/NUSMG 97-07.

The WNP-2 Year 2000 Project Plan, Revision 1, includes the following phases:

awareness, initial assessment, detailed assessment, remediation, and notification. It also includes requirements for Y2Ktesting and validation (Y2K investigative testing during detailed assessment and confirmatory testing after remediation), regulatory considerations, contingency planning and risk management, QA, and documentation and retention of records.

WNP-2 has used several contractors in the building of the inventory of Y2K-susceptible assets and to perform detailed assessments, training, and independent evaluations. These contractors include personnel from LES (procedures, training and initial inventory assessment),

Raytheon (detailed assessments of embedded systems), Bechtel (embedded system testing), CANUS (test training and independent evaluation), and Lockheed Martin (detailed assessments and remediation of software assets).

2.2.1 Awareness An early significant event in the Y2K awareness phase of the Y2K program was G. L. Gelhaus's presentation of the Year 2000 Project Overview for System Engineers on April30, 1998. This presentation was followed soon after by the formal Y2K kickoffwith plant management on May 5, 1998. The licensee attempted to involve all employees in the effort to identify assets that might be sensitive to the Y2K problem. The purpose of the initial communications was to raise general awareness of the issue and to communicate its importance to the organization.

The audience included the following personnel:

Management Operations Subject matter experts System engineers Software or system sponsors General employees Support organizations such as procurement and engineering programs.

The awareness effort is ongoing. Recently a site-wide challenge was issued to all employees to identify items not already in the Y2Kdatabase, and a reward of a free lunch was offered those who were successful.

As a result, 23 entries were submitted, of which 12 were assets not previously included in the inventory of digital assets (for example: digital camera and label maker).

On the basis of its review of the licensee's communications, the audit team concluded that the licensee's Y2K awareness program is effective.

The initial assessment phase of the WNP-2 Y2K Project began in January 1998. The purpose of the initial assessment was to identify all software applications and embedded system components at WNP-2, an effort that required input from the Y2K support team.

The tasks of initial assessment included (1) inventory, (2) categorization, (3) classification, (4) prioritization, and (5) analysis of the initial assessment.

In the identification of embedded systems, the Y2K support team reviewed procedures and documentation for phrases that would indicate the existence of an internal clock or processor; surveyed the vendors for information on their equipment; performed system walk-down inspections; and reviewed schematics, program listings, and reference manuals.

2.2.2.1 Inventornt The initial inventory consisted of a list of over 90,000 assets.

The list was derived from the following sources:

PASSPORT (master equipment list) database Procedures and documentation in which certain phrases might indicate the existence of an internal clock or processor Information from vendors about their equipment System walkdown inspections Schematics, programming listings, and reference manuals Miscellaneous lists of departmental assets The licensee's team reviewed this list and removed from the inventory obvious nonsusceptible items such as snubbers, ieechanical equipment, pipe spools, and non-digital valves. This reduced the list to about 3000 assets with potential for Y2K susceptibility. These assets were then sorted according to system and referred to the system engineers for review.

The embedded systems inventory was performed with the assistance of LES resources in accordance with WPPSS Instruction Number ENG-DES-26, "Year 2000 Readiness Assessment Methodology." This information was used to prepare advance information packets for interviews with System Engineers and I8C technicians.

The information gathered from these processes defined the initial embedded system inventory.

The embedded system inventory, kept in a database proc'ured from LES, typically contained the following information:

Unique identification of software or component Description Manufacturer Version number (software)

Model number (hardware)

Responsible organization Initial compliance status Contact name Classification Categorization Prioritization The inventory process for software assets is not as direct as the inventory of embedded assets.

Software applications do not have formally tracked asset numbers ("EPN" numbers) and, therefore, do not all appear in one central database.

The inventory of the software assets began with the Software Quality Assurance (SQA) listing of controlled software. The list was then expanded to include individual department listings of software.

The non-SQA software assets are generally work-saving programs that are not used directly in safety-related plant applications.

However, the loss of these programs would result in significant inconvenience.

The Y2Kteam is taking additional steps to ensure that all the non-SQA programs are included in the inventory.

2,2, Afterthe inventory was made, the Y2Kteam categorized the inventoried items.

Categorization is the process that groups applications, systems, and components to allow management to efficiently assign resources to the classification and prioritization activities. Each item included in the initial inventory was assigned to a category.

The categories used at WNP-2 were similar to those identified in NEI/NUSMG 97-07, as follows:

Telecommunication equipment Embedded devices Test equipment Software interfaces, Facilities Software Applications Chemistry 8 Environmental Emergency Planning Health Physics Mainframe hardware Network Simulator

, Training 2.2.2.3 Classification Classification is the process by which the Y2K project team ev'aluated potential risk factors to obtain criteria for assigning resources to evaluate and resolve potential Y2K problems.

The Y2K project team assigned risk categories to facilitate the process of determining the order in which to perform the detailed assessment of the inventoried items. Two types of risk were

defined:

(1) date usage risk (DUR), and (2) safety and regulatory risk (SRR). The Y2K project team evaluated DUR assignments for information systems (IS) infrastructure and for limited use hardware and software applications and SRR assignments for plant embedded and telecommunication items and IS infrastructure and limited use items.

Each of the two types of risk has three defined category levels, as follows:

The DUR categories indicate the level of urgency that the date or time function has on the application, system, or component.

The three categories of DUR are as follows:

CatecaoCr I

Date or time stamped data or data for long-term averaging, integrating, trending, scheduling, or reporting.

~Cate o

II Data used for short-term averaging, integrating, trending, scheduling, or reporting.

~Cate o

III Data used for time-independent calculations or operations.

The SRR categories indicate the level of impact on plant and personnel safety, continued generating capability, and other facilityoperating capabilities that the application, system, or component may have ifthe potential Y2K problem is not resolved and/or mitigated. The three categories of SRR are as follows:

CatecoaCr I

The application, system, or component:

ls, of itself, nuclear-safety related equipment Contiols a nuclear-safety related application, system, or component May force an immediate or near-immediate plant shutdown Is used for nuclear-safety related activities or calculations Provides automatic control of critical plant functions May require entry into a technical specification limiting condition for operation (LCO) with a limit of 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> or less May degrade the ability to protect the health and safety of the general public

~Cate o

II The application, system, or component:

May degrade the ability to protect the health and safety of plant personnel May degrade the control of plant habitability systems May require entry into an LCO with a limitof more than 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> May affect the control or tracking of other critical plant information or operations Is required by regulations or license commitments

~Cate o

The application, system, or component:

Controls other plant systems May affect the control or tracking of other plant information or operations Is required to meet guidelines from the Institute of Nuclear Power Operations (INPO) or other quasi-regulatory requirements

'ay affect other non-plant applications or systems not covered by Categories I or II 2.2.2.4 Prioritization The Y2K project team prioritized the inventory to ensure that resources were effectively allocated to high-risk, high-priority items as follows:

PRIORITY Critical Critical High High Medium Medium Low Low Low Safety and Regulatory Risk (SRR)

Date Usage Risk (DUR)

Other factors beyond the above risk categories also determine prioritization of the inventory results.

These other factors included the following:

Items known to have Y2Kfailures or vulnerabilities.

Complexity of testing or remediation strategy.

Ifa system or component might require a plant, system, or component outage, the Y2K project team assigned a higher priority to this item over non-outage items with higher risk ratings.

Items that required engineering design packages for remediation also received a higher priority.

The number of items requiring testing/remediation.

When the Y2K project team identified a common component and the number of the items requiring attention was large, it assigned a

higher priority to this inventory item.

'l Long lead time items. The Y2K project team assigned a higher priority to these items, which included items that required testing and remediation by outside agencies whose services might become increasingly difficultto obtain as the year 2000 nears.

Availabilityof replacement parts and components.

Input from WNP-2 management.

The Y2K project team assigns all assets having priorities of "Critical"or "High" as "Mission Critical."

2.2.2.5 Anal sis of InitialAssessment The final step of the initial assessment was to determine the scope, schedule, and estimated resources required for the detailed assessment on the basis of initial prioritization and categorization.

This step also involved business considerations and decisions. The licensee completed the initial assessment in July 1998.

Of a total 3084 assets (2373 embedded systems and 711 software items), the licensee has assigned 810 as Mission Critical (of which 753 are embedded systems), 411 as Medium and 1863 as Low.

2.2.3 Detailed Assessment The licensee has not yet completed the detailed assessment phase.

This is scheduled for completion by May 30, 1999. The purpose of the detailed assessment is to obtain sufficient information about each inventoried item to determine its expected Y2K performance.

Detailed assessments are performed in accordance with one of two WNP-2 procedures, i.e., ENG-DES-26 or SWP-CSW-03.

The Y2K project team uses assessment results to make decisions regarding actions required for any remediation.

The detailed assessment phase includes, as

'ppropriate, the consideration of information from other databases, vendor evaluations, investigative testing, or the conducting of engineering evaluations to identify any deficiencies and their failure modes and effects. The licensee states that as of February 1999, 89% of Mission Critical detailed assessments and 82% of the total detailed assessments have been completed.

The remaining asset assessments, including the necessary remediation, are scheduled to be completed by June 1999.

The testing of embedded systems and software systems is performed using test plans based on PPM 8.3.402, "Testing of Year 2000 (Y2K) Readiness."

The test plans include a 10 CFR 50.59 screening, ifrequired, to ensure that the test does not result in an unreviewed safety question.

Whenever testing is performed, expected results are identified to the extent possible before testing.

(See Section 2.2.4.1 below for additional discussion on testing.)

Deficiencies identified during the detailed assessments are reported on a Problem Evaluation Request (PER) form. Each PER is assigned to a responsible person for remediation, assigned a completion date, and entered into the plant tracking log. The remediation plan is reviewed and approved by the "owner" of the system, application, facility, or equipment and recorded into

the Y2K readiness database.

In addition, the remediation plans are also reviewed by the WNP-2 Y2K Remediation Review Panel.

2.2.3.1 Vendor Evaluation As a part of detailed assessment activity, the Y2K project team determines whether the Y2K susceptible items are vendor supported so that responsibility for subsequent activities can be established.

For assets supported by vendors, a determination is made for the appropriate commercial instrument (contract, license agreement, interface plan, etc.) to be used for detailed evaluation and remediation activities. These activities may include remediation by the vendor, cooperative efforts with the vendor, or the issuance of a request for Y2K information and certification from the vendor. For items determined to be vendor supplied, but for which no vendor support is available or forthcoming, the item is evaluated using WPPSS resources or cooperative agreements with other utilities, using the Y2K process defined in NEI/NUSMG 97-

07. For the majority of the items, the detailed evaluation is performed or verified by the WNP-2 Y2K project team.

For vendor responses that indicate that an application or device'is Y2K ready or compliant the Y2K project team decides whether to perform confirmatory testing. The decision is based on the criticality of the item, prior experience with the vendor, the extent of documentation provided, or WPPSS knowledge of the item.

Based on the review of some "Low"priority asset documentation, the audit team noted that the Y2K project team appeared to give less attention to the non-safety-related and non-operating-plant software and personal computers (PCs).

For example, they were grouping their PCs by type, and testing one of each type. There may be differences in the BIOS of two PCs having the same part number but different dates of manufacture.

Also there could be a problem with stand-alone software.

Many suppliers do not upgrade the version number for minor changes.

This was identified as a potential implementation problem.

2.2.3.2 Evaluation of Licensee Owned or Su orted Software The Y2K project team evaluates WPPSS owned or supported software using source codes to identify Y2K issues.and/or testing of the software for Y2K problems.

This evaluation involves the writing of test plans and/or procedures for critical applications.

The result of the evaluations is tracked in the Y2K readiness database.

2.2.3.3 Remediation Plannin After an application has been determined to be deficient regarding a Y2K problem, a business decision is made as to whether the application willcontinue to be used as-is or whether it willbe retired, replaced, upgraded, or modified. In instances in which the asset is of low significance, this business decision may precede any detailed evaluation or testing. This business decision is documented to include the options evaluated and their costs, schedules, benefits, and risks.

These business decisions involve management and the "owner" of the system, component, or asset.

2.2.4 Remediation or Re lacemen Remediation involves retiring, replacing, modifying, upgrading, leaving as is, or testing before use an application as determined in the remediation planning phase.

Allitems in this phase, including warehouse items, are tracked for replacement projects, purchases, conversions, deletions, retirements, and vendor efforts in the Y2K readiness database.

Plant software and embedded system changes are approved by the appropriate system engineer or other designated "owner" and reviewed by the Remediation Review Panel.

The licensee has chosen to undertake major upgrading or replacement of computer systems and business software rather than perform extensive evaluation and testing.

Many of the computers and computer applications are being retired. The remaining scope is undergoing detailed assessment, testing, and remediation as appropriate. This effort has lagged and the licensee has contracted for additional resources from Lockheed Martin to assist the in-house staff in completing the software applications.

The WNP-2 Y2K Project schedule calls for the completion of all remaining detailed assessments of mission critical software and embedded systems by March 31, 1999. (As stated earlier, by the end of February 1999, 89% of Mission Critical detailed assessment has been completed.)

Upgrading of the Reactor Building Elevated Release Monitor Computer, the Transient Data Acquisition System Computer, and the Security System Computer are the significant remediations identified and their implementation is scheduled for the Spring 1999 outage. Allbut one of the required remediations identified to date are scheduled to be completed by July 1, 1999. One computer remediation, deactivation of the control room "Prime" computer, willbe completed during the Fall 1999 refueling outage.

This last computer upgrade interfaces with the Emergency Response Data System (ERDS) and Graphic Display System (GDS). The upgraded computer system will be running in parallel with the "Prime" and willbe Y2K Ready by July 1, 1999, but the exchange of the two computers is scheduled to occur during the Fall 1999 refueling outage.

The audit team concludes that the completion of each of these replacements and upgrades is critical to the success of the WNP-2 Y2K effort. These projects are being closely monitored by the Y2K Project Manager.

2.2.4.1 Testin and Validation The WNP-2 Y2K Project Plan includes investigative testing in support of evaluation efforts during the detailed assessment phase, ifneeded, to determine ifa Y2K problem is present.

Similarly, Y2K confirmatory testing to validate conclusions reached by the detailed assessment is also an option. For Mission Critical assets, Y2K confirmatory testing following remediation is a requirement in the plan to determine ifthe remediation activities have eliminated the Y2K problem and no unintended functions are introduced.

2.2.4.2 Certification Certification involves returning remediated and tested components and systems back into operation and ensuring all documentation is complete.

This phase also includes preparation of the documentation required to satisfy GL 98-01.

Certification willinvolve sign-offs by the system engineer or asset owner and the forwarding of the certification package to the Y2K project team who will prepare and file the certification documentation.

2.2.5 Re ulato Considerations The licensee's Regulatory Affairs Department is responsible for assisting the Y2K project team with regulatory matters related to the Y2K project. This assistance includes maintaining a current awareness of the relevant regulatory climate, advising the team on regulatory strategies including the licensee's response to GL 98-01, interfacing with the NRC on such items as impacts upon the Technical Specification or Final Safety Analysis Report, preparing correspondence to the NRC, assisting with NRC briefings when necessary, and participating in contingency plans.

2.2.6 Notification The Y2K Project Team willnotify the users of the remediated equipment of changes to the software, hardware and systems.

This is to include changes to documentation and training due to the Y2K project activities.

2.2.7 Contin enc Plannin andRisk Mana ement The licensee began contingency planning activities using NEI/NUSMG 98-07 in November 1998. The project organization includes a Contingency Planning Team with a Contingency Planning Lead as the single-point-of-contact responsible for all contingency planning. The team's mission is to develop an integrated, comprehensive, risk-sensitive contingency plan for Y2K-induced events for WNP-2 and WPPSS functions. The specific areas to be considered for contingency planning include equipment and software remediation readiness, and internally and externally initiated Y2K events.

The audit team evaluated the licensee's contingency planning and found that the licensee has a good organization working on contingency planning issues.

One of the concerns that the licensee is addressing is that although they have found that one of the two telephone companies is Y2K compliant, there are not enough trunk lines to handle the expected traffic.

The licensee feels that the traffic on January 1, 2000, may exceed a "Mother's Day" load, thus tying up the system.

The proposed remediation is to instruct those personnel who will be assigned to be on standby for the entry into year 2000 to report for duty unless they have received a notification to not report. This way, ifthe phones fail, the needed people willstill be available.

The audit team asked the licensee about the local radio broadcast station that is part of the emergency response provision, ifthey are known to have a Y2K plan to ensure that they willbe broadcasting at the turn of the millennium. This had not been covered by the plan as yet, so the licensee called the station and asked. The immediate response was from the station was that they have a Y2K program to assure operation.

The licensee willfollowup on this issue.

The WNP-2 contingency planning activities for external events is scheduled for completion in February 1999 and the internal events plus remediation are scheduled for completion in March 1999. The integrated plan is scheduled to be completed in April 1999, with the final approved plan scheduled for mid-May 1999. An assessment of the Y2Kcontingency plan is planned to be performed to determine the adequacy of the plan. The assessment willbe scheduled to allow time for correcting problems before Y2K sensitive dates.

P il h

The audit team concluded that the proposed contingency planning activities are consistent with the guidance of NEI/NUSMG 98-07 and that the licensee's schedule for completion of the project contingency plans is realistic.

2.2.8 Y2K Pro ram Mana emen The WNP-2 Y2K program activities and schedules are tracked by the Project Manager and the Project Sponsor.

There are routine weekly project meetings involving the team members and monthly project reviews involving management.

Briefings of Corporate Nuclear Safety Review Board and Executive Board are also used as means for management oversight.

The licensee participated with three other utilities in a meeting from July 26 to July 30, 1998 to assess the effectiveness of each of the utilities'2K Plans.

Each of the four utilities sent two or three participants to participate on. the assessment team. The team used guidance provided in NEI/NUSMG 97-07 to compare and contrast the Y2K project plans of each utilitywith those of the other assessment participants.

Several recommendations to the WNP-2 plan were developed and subsequently implerriented as a result of this assessment.

WNP-2 conducted its own self-assessment on October 19-24, 1998. The WNP-2 self-assessment team consisted of plant staff as well as industry peers. The purpose of this self-assessment was to evaluate the effectiveness of the WNP-2 Y2K project plan.

Aiiindependent consultant review of embedded systems activities in September 1998, and a quality audit in January 1999 are other assessments performed.

Additional assessments including one on contingency planning activities is planned.

The recommendations provided by these assessments and audits are being implemented at WNP-2.

In November 1998, the Y2K project was reorganized as follows. The leadership was changed; the reporting structure was changed, Lockheed Martin was contracted to augment plant staff dealing with software assets, and an integrated project schedule was developed.

The licensee has augmented the language in procurement contracts to include a standard clause to require Y2K compliance.

In addition, a Y2K procurement guide has been developed that is intended to control the existing inventory and new purchases.

2.2.9 Electric Grid Issues WNP-2 is located within the Northwest electrical grid, which is under the jurisdiction of the Bonneville Power Administration (BPA) and dominated by hydroelectric generation. The BPA grid encompasses a network of major hydroelectric generation facilities operated by the Corps of Engineers, the Bureau of Land Reclamation, and local utilitydistricts. WNP-2 does not own or maintain any transmission lines. WNP-2's main generator and transformers interface directly with BPA's Ashe and Benton substations.

Because of their dependence on BPA for grid interfacing, the WNP-2 Y2K Project established a formal agreement with BPA that BPA would followthe NEI/NUSMG 97-07 guidance for Y2K activities of key substations.

Contingency planning for Y2K critical dates is underway and is being developed in coordination with BPA. The initial "best. posture" for WNP-2 has been approved by senior management and forwarded to BPA for integration with the Western States Coordinating Council contingency

planning. This "best posture" recommendation involves WNP-2 operating at 80% of full power for the entry into the year 2000.

3.0 AUDITTEAM FINDINGS The audit team reviewed in detail 13 Mission Critical embedded system packages (see Table 2) and 3 Mission Critical software assets (see Table 3). The team met and interacted with WNP-2 Y2K staff during the review process.

The audit team was briefed by the Contingency Plarining Lead on the ongoing activities which are presently focused on external interfaces and events.

Based on the audit team's review of WNP-2 Y2K program activities, the audit team has the following observations:

The WNP-2 Year 2000 Project Plan and associated detailed procedures for implementing the plan at WNP-2 are considered to be excellent. They are comprehensive and detailed documents which incorporate the guidance of GL 98-01, "Year 2000 Readiness of Computer Systems at Nuclear Power Plants," and NEI/NUSMG 97-07, "Nuclear UtilityYear 2000 Readiness."

The plan includes the outline of Contingency Planning and associated instructions that are based on the guidance of NEI/NUSMG 98-07, "Nuclear UtilityYear 2000 Readiness Contingency Planning."

WNP-2 is using existing quality assurance (QA) and modification program procedures as well as Y2K-specific plant procedures to achieve Y2K readiness.

The applicable considerations for regulatory requirements and criteria are addressed in the Year 2000 Project Plan implementation activities. The licensee has been sharing data and work experience through activities in the BWR Owners Group Y2Ksubcommittee; United Service Alliance; Litton Enterprise Systems Readiness database; EPRI conferences, workshops and database; and NEI and NUSMG workshops.

3.

The project is well organized and adequately staffed with plant and contractor staff.

The Project organization includes a Contingency Planning Lead as the single-point-of-contact for the contingency planning process.

Based on the audit team's interaction with the project staff, the audit team considers the Y2Kproject staff to be very competent and knowledgeable in the activities they perform.

WNP-2 Y2K readiness including contingency planning is scheduled to be achieved by July 1,1999.

The audit team noted that the project emphasis has been on addressing the embedded systems; and the successful completion of activities to achieve their Y2K readiness by July 1, 1999 appears to be on track. However, regarding the Y2K readiness activities of software assets, the audit team echoed the concern on their progress raised by the October 1998 self assessment and the January 1999 quality audit. The audit team noted the corrective actions planned and taken in response to the recommendation of the assessment teams, particularly the reorganization of the Y2K project in October 1998, its prominence in the organization, the addition of responsibility

for software asset activities, and the increased resources provided.

These actions will help in meeting the project schedule of having all critical, high, and medium priority assets Y2K ready by July 1999.

5.

The WNP-2 licensee has not identified any Y2Kproblems with systems needed for safe shutdown of the plant.

Critical to the completion of WNP-2 Y2K,project is the completion, on schedule, of the four "related projects" that are separate from the Y2K project. The audit team identified that the Y2K program schedule has no flexibilityto account for unforseen problems in the Y2K readiness activities of the four related projects, and in the remaining work on software assets and embedded systems. The Y2K project manager (PM) and project sponsor acknowledged this and stated their intent to address the issue.

Based on the audit team's review of several project work packages of the assets the audit team observed that, although incomplete as certification packages, most of the documentation and contents are consistent and followthe guidance provided by the WNP-2 Y2K Program Plan.

Some inconsistencies were noted and identified to the PM.

The PM intends to feedback as "lessons learned" some of the examples of inconsistencies and, additionally, willrequire thorough reviews during certifications to resolve the inconsistencies.

The PM also plans to issue completed system-level packages for certification starting in March 1999, along with additional guidance to ensure consistency.

Attachments:

1. Table 1

2. Table2

3. Table 3

4. List of Attendees

5. List of Documents Reviewed

6. List of Attendees - Exit Meeting

Table 1 WNP-2 Project Plan Schedule Activity Awareness InitialAssessment Embedded Systems

'Software Systems Detailed Assessments Starting Date April30, 1998 April25, 1998 January 1998 Finishing Date Ongoing July 8, 1998 Dec. 31, 1998 Other Mission-Critical Embedded and Software Systems July 9, 1998 Plant Mission-Critical Embedded Systems July 9, 1998 March 31, 1999 March 31, 1999 AllMedium-and Low-Priority Embedded Systems

• 'AllMedium-and Low-Priority Software Systems September 1998 May 30, 1999 September 1998

• • May30, 1999 Remediation of AllAssets Confirmatory Testing Certification Contingency Planning External Events Internal Events Plus Remediation Schedules Integrated Plan Manage'ment Approval of Plan Contingency Training Complete July 1998 Dec. 21,1998 March 1999 Oct. 15, 1998 July 1, 1999 June 30, 1999 July 1, 1999 July 1, 1999 February 1999 March 1999 April 1999 May 15, 1999 Sept. 1, 1999

• Assessments of software assets were formally included in the LES2000 data base in October 1998.

• "The licensee may not complete the detailed assessment for some of the low-prioritysoftware.

~

~

Table 2 Embedded Items Reviewed by the Audit Team EPN NUMBER Asset Description, System, Manufacturer

,Priority Y2K Intended Status Approach COND-CR-1 Condenser Leak Detection Conductivity Recorder, Yokogawa Model 4178-500-32/BU/AK-06/REM, 180-mm Dot Printing Micro Recorder Critical Ready Use as is; adjust dates manually EMPREP-040-01 PRM-XAY-1C Monitor, Ikegami Model C/N-20A Stack Monitor Multichannel Buffer-Low Range, EG&G Model 921 Critical Critical Comp-To be replaced liant by new PDIS in Sept. 1999 Comp-Scheduled for liant replacement by new computer system PRM-COMP-3 Stack Monitor Computer System, EG8G Critical Sus-pect Replace ENV-057 TSC-RR-1 (ARM-RR-32)

MS-LR/PR-623A (OG-RR-601)

DG-'SC-GOV/DGI/A14

& DG-SC-GOV/DG2/A14 Data Recording System with Software, Telog Instruments, Inc.,

Model 2400 RX BLDG High Range Area RAD Mon Recorder H13-P614, Yokogawa Model 4263 Series OG Post Treatment RAD Recorder H13-P600, Yokogawa Model 4263 Series UR100 100T Diesel Generator Governor Speed Control, Woodward Governor Company, Model 9903-303b Series 2301a High Critical Critical High Non-Software up-compl-grade willmake iant the system Y2K compliant.

Comp-None liant Comp-None liant Inert 'one RFT-GOV-1A 8 RFT-GOV-1 B RFW-DT-1A 8 RFW-DT-1B Governor Lovejoy Controls Corporation Model P&DU High Inert None COND-LIC OG-RR-601 RFW-CPU-L010B RRC-CPU-L04B Condenser.Hotwell Level Control, Foxboro, MODEL760CSA-3A, 760CSA-AT, 8 760CNA, OG Post Treatment Rad Recorder H13-P600 (*SC2B)

RFW GE FANUC 90-70 CPU, General Electric Central Processing Unit for PLC B High Critical Critical High Inert None Comp-None liant Comp-None liant Comp-None liant

Table 3, Software Assets Reviewed by the Audit Team Asset Name Modcomp OS Security System Software TADS Asset Description MAXIVModcomp Y2K OS Software (SEC-20.0)

WNP-2 Site Security System Software (SEC-20.1)

Transient Data Acquisition System Compliance Non-compliant Non-compliant Non-compliant Priority High High Critical

LIST OF ATTENDEES ENTRANCE MEETING-January 26, 1999 G. L. Gelhaus D. W. Coleman T. W. Myers J. A. McDonald F. K. Butz D. Maley Sal Ghbein G. W. Ash J. D. Carpenter Scott Wood Robert Brill Keith Mortensen Matthew Chiramal Fred Schill Amanda Barber Tom Morales Douglas L. Williams Joyce A. Caldwell Eddie E. Tubbs Robert Quay Dale Atkinson Jim Engbarth Paul Inserra Scott Oxenford Tim Messersmith WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS NRC/RES NRC/NRR NRC/NRR WPPSS WPPSS WPPSS BPA LMSI WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS

/

Y2K Project Manager Manager, Regulatory Affairs Y2K Engineer Plant Production Manager CIO Embedded Systems Test Lead Plant Modifications Business Systems Team Leader Network Operations Acting Manager, Technical Services Office of Nuclear Regulatory Research Instrumentation and Controls Branch Instrumentation and Controls Branch Licensing Acting Quality Manager-Embedded Systems Lead, Y2K Nuclear Engineer Senior Software Engineer Senior System Analyst Y2K Project Lead Engineering Manager Quality Supervisor Licensing Manager Operations Manager Contingency Planning Lead ATTACHMENT4

LIST OF DOCUMENTS REVIEWED NEI/NUSMG 97-07, "Nuclear UtilityYear 2000 Readiness," dated October 1997 NEI/NUSMG 98-07, "Nuclear UtilityYear 2000 Readiness Contingency Planning," dated August 1998 WPPSS, WNP-2 Year 2000 Project Plan, Revision 1, dated December 1, 1998 WPPSS, WNP-2 Y2K Project, Detailed Assessment Team for Embedded Systems, Desktop Guidelines for Detailed Assessment

Packages, Revision 1, Dated November 5, 1998 WPPSS, Procedure Number 8.3.402, Testing of Year 2000 P2K) Readiness, Revision 2, Dated December 30, 1998 WPPSS, Instruction Number ENG-DES-26, Year 2000 Readiness Assessment Methodology, Revision 1, Dated October 20, 1998 WPPSS, Procedure Number SWP-CWS-03, Preparing Software Year 2000 (Y2K) Detailed Assessments, Revision 0, Dated December 9, 1998 WPPSS, Y2K Software Assessment Guideline, Revision 1, Dated December 8, 1998 ATTACHMENT5

'1

LIST OF ATTENDEES EXIT MEETING - January 28, 1999 G. L. Gelhaus D. W. Coleman T. W. Myers D. Maley S. F. Ghbein G. W. Ash J. D. Carpenter Robert Brill Keith Mortensen Matthew Chiramal Fred Schill A. S. Barber, Tom Morales Douglas L. Williams Joyce A. Caldwell Eddie E.-Tubbs Robert Quay Dale Atkinson Jim Engbarth Scott Oxenford Tim Messersmith B. J. Van Erem K. Davenport A. E. Mouncer R. L. Webring WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS NRC/RES NRC/NRR NRC/NRR WPPSS WPPSS WPPSS BPA LMSI WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS WPPSS Y2KProject Manager Manager, Regulatory Affairs Y2K Engineer Embedded Systems Test Lead Plant Modifications Business Systems Team Leader Network Operations Office of Nuclear Regulatory Research Instrumentation and Controls Branch Instrumentation and Controls Branch Licensing Acting Quality Manager Embedded Systems Lead, Y2K Nuclear Engineer Senior Software Engineer'enior System Analyst Y2K Project Lead Engineering Manager Quality Supervisor Operations Manager Contingency Planning Lead Procurement Engineering Supervisor Y2KAdmin. Support General Counsel VP, Operations Support/PIO ATTACHMENT6

1/

,1