ML20203C226

Audit Rept on Implementation of GL 98-01, Year 2000 Readiness of Computer Systems at Nuclear Power Plants, for Waterford Steam Electric Station,Unit 3 on 981214-17

ML20203C226
Person / Time
Site:	Waterford
Issue date:	02/09/1999
From:	NRC (Affiliation Not Assigned)
To:
Shared Package
ML20203C218	List: ML20203C213 ML20203C226
References
GL-98-01, GL-98-1, NUDOCS 9902110281
Download: ML20203C226 (21)

Text

. .- -.- - - - - - - -. .- .-. . .

s U.S. NUCLEAR REGULATORY COMMISSION OFFICE OF NUCLEAR REACTOR REGULATION (NRR)

AUDIT REPORT ON IMPLEMENTATION OF GENERIC LETTER (GL) 98-01

" YEAR 2000 READINESS OF COMPUTER SYSTEMS AT NUCLEAR POWER PLANTS' Docket No: 50-382 License No: NPF-38 Licensee: Entergy Operations, Inc.

Facility: Waterford Steam Electric Station, Unit 3 Location: Killona, LA Dates: December 14-17,1998 Audit Team Members: Michael E. Waterman, NRR Matthew Chiramal, NRR Approved by: Jared S. Wermiel, Chief l Instrumentation and Controls Branch l Office of Nuclear Reactor Regulation i

{-

i t

ENCLOSURE 9902110291 990209 PDR ADOCK 05000382 P PDR

s EXECUTIVE

SUMMARY

From December 14 through 17,1998, the NRC staff conducted an audit of the Year 2000 (Y2K) t readiness program at the Waterford Steam Electric Station, Unit 3 (W3) in accordance with the audit plan for this activity. The purpose of the audit was to (1) assess the effectiveness of the Entergy Operations, Inc. (EOl) program for achieving Y2K readiness, including continued safe operation of the plant as well as compliance with applicable NRC regulations and license

, conditions with respect to potential Y2K problems, (2) evaluate Y2K program implementation to assure that the licensee's schedule is in accordance with NRC Generic Letter (GL) 98-01 guidelines for achieving Y2K readiness by July 1,1999, and (3) assess the licensee's t contingency plans for addressing risks associated with potential events resulting from Y2K  ;

problems. The audit team reviewed selected licensee documentation regarding the W3 Y2K readiness program and conducted interviews with the cognizant licensee personnel. The results of this audit and subsequent audits at other selected plants will be used by the staff to determine the need for additional action, if any, on Y2K readiness for nuclear power plants.

Based on the staff's assessment and evaluation of the W3 Y2K readiness program, the following observations were made:

1. EOl has a common Y2K project implementation plan for all its nuclear facilities including W3. The W3 Y2K project plan entitled " Year 2000 Project Guide Y2K-001," establishes the scope and control of the Y2K Project at the W3 plant. The Y2K project plan is comprehensive and incorporates the major elements of the nuclear power industry Y2K problem guidance contained in Nuclear Energy Institute (NEI)/ Nuclear Utilities Software I Management Group (NUSMG) 97-07, " Nuclear Utility Year 2000 Readiness, " and NRC J GL 98-01. '

2. The licensee was preparing for a refueling outage scheduled in February 1999 at the time of the NRC audit. Consequently, the systems and operations staff and particularly the mid-level management, whose support is necessary for addressing Y2K issues at W3, were not as available as at other plants. Directed support from the W3 Y2K project sponsor and upper management will continue to be necessary to expedite Y2K program progress at W3. The licensee agreed with this observation and has committed to l provide more aggressive management oversight of the Y2K project.

3. The licensee began the formal W3 Y2K readiness program in June 1997, and finished l the plant inventory and initial assessment phase in May 1998. The detailed assessment i phase for systems and components that are mission critical is scheduled to be ,

completed by December 31,1998. Confirmatory testing of mission critical systems for  !

Y2K readiness is conducted by the licensee at the plant site. Remediation of mission  ;

critical systems has begun. No Y2K problems have been identified in safety-related l systems to date.

[ The Entergy Global Year 2000 policy requires that all Mission Critical asset work )

!' projects be completed by December 31,1998, and the work projects for important j assets be completed by June 30,1999. However, the audit team noted that, of the 8

. Mission Critical application software items,4 will be completed in April 1999, and of the l 275 Mission Critical embedded items in 25 work projects,8 work projects are scheduled

. I

- . - . -- - --- -- . . - - - . - -. .- ---~.--- - .-

s.

l I

for completion by June 1999. Thus, the initially established deadline for the Mission Critical assets will not be met at W3, and this may impact the deadline for completing ,

the important asset work projects. Additionally, there does not appear to be sufficient ]

flexibility in the schedules to account for unforseen delays in completing the work projects. l To address the above schedule conflict, the licensee established a revised schedule for Y2K readiness of mission critical and important system projects by June 30,1999. The ,

audit team believes that, with the additional resources the licensee plans to obtain in i the immediate future, the above Y2K readiness schedule appears to be achievable. This  ;

conclusion is based on the expertise and experience of the present W3 Y2K project j team members, support from management, and support available via information 1 sharing and interactions with EPRI, other licensees, the Combustion Engineering Owners Group (CEOG), plant vendors, the Southeast Electricity Reliability Council, and the North American Electric Reliability Council.

4. The licensee has just started Y2K contingency planning, and is using the nuclear industly guidance in NEl/NUSMG 98-07, " Nuclear Utility Year 2000 Readiness Contingency Planning," for this effort. The W3 contingency plan is scheduled for completion on June 30,1999. The W3 plan will be incorporated into the EOl plan format with a single corporate plan for Y2K contingencies and site-specific plans for each facility connected to the EOl grid. EOl will be providing the milestones for identifying the risks, analyzing the risks, and creating the plans. At present, the licensee has assigned one person part-time as lead for W3 Y2K contingency planning with support from two Y2K project team members. The audit team considers that the schedule for completion of detailed contingency planning may be adversely impacted by ,

the refueling outage activities and other Y2K readiness activities. The audit team ]

believes that additional resources are needed to support W3 Y2K contingency planning  !

activities in order to meet the established W3 Y2K project schedules. The licensee has )

committed to provide additional resources on contingency planning in order to meet the >

above schedule.

5. ' An independent assessment of the W3 Y2K project plan was performed by the licensee on October 19-21,1998. Recommendations from that assessment are being addressed by the W3 Y2K project team in subsequent revisions of the W3 Y2K project plan. The audit team reviewed the W3 Y2K Project team draft response to the recommendations of the assessment report and believes the proposed corrective actions should address the recommendations adequately.

6. The Y2K project documentation system used at W3 is well-organized. Related appi cations and embedded systems are combined into work project documentation t packages. The documentation is maintained in an electronic database. Documents not produced on the system are scanned into the database. The database is backed-up on a weekly basis. Upon completion of the project, the documentation is printed and inserted into a hard-copy file, with a summary of the activities that were performed. The documentation in the completed work project includes a summary of the work performed and permits a rapid review of the Y2K activities status and conclusions for each project.

11

e

1.0 INTRODUCTION

The objectives of the W3 Y2K readiness program audit were to:

1. Assess the effectiveness of the W3 program for achieving Y2K readiness, including continued safe operation of the W3 plant as well as compliance with applicable NRC regulations and license conditions with respect to potential Y2K problems,

2. Evaluate Y2K program implementation to assure that the licensee's schedule is in accordance with NRC Generic Letter (GL) 98-01 guidelines for achieving Y2K readiness by July 1,1999, and,

3. Assess the licensee's contirigency plans for addressing risks associated with potential events resulting from Y2K problems.

The audit was conducted in accordance with the established audit plan outline (Attachment 1) which was based in part on the guidance and requirements contained in the following documents:

e GL 98-01," Year 2000 Readiness of Computer Systems at Nuclear Power Plants"

• Licensee Response (s) to GL 98-01

• Plant technical specifications and license terms and conditions e Applicable NRC regulations e NEl/NUSMG 97-07, " Nuclear Utility Year 2000 Readiness"

• NEl/NUSMG 98-07, " Nuclear Utility Year 2000 Readiness Contingency Planning" Prior to the audit at the plant site, the audit team obtained and reviewed the W3 Y2K Project Plan, Rev. 3, dated November 23,1998. The team also reviewed the W3 Readiness Assessment Report dated November 6,1998, which provided the conclusions and recommendations of an assessment of the W3 Y2K Program by an independent team of auditors.

The audit process started with an entrance meeting attended by the W3 Y2K Project Manager (PM), Y2K project team members, senior and other site personnel, and members of the audit team. Attachment 3 is a list of the attendees. Margit Triggs, the W3 Y2K PM, described the project organization, the project plan and its implementation, project status, and ongoing activities.

Following the meeting, the audit team spent the rest of the audit reviewing the W3 project plan and its associated procedures, the plan implementation products (documents and data bases),

and interacted with the Y2K PM, W3 Y2K project team members, and members of the EOl Y2K project team. The documents reviewed and referenced in this audit are listed in Attachment 2.

1 .

l 2.0 WATERFORD 3 Y2K PROJECT DESCRIPTION 2.1 Project Oraanization The EOl Y2K readiness program at W3 has approximately15 full-time equivalent persons supporting the W3 part of the EOl Y2K project, with these resources distributed among the W3 Y2K project team, management, and system engineers. The W3 Y2K PM reports to the W3 Y2K project sponsor (Mr. R. Burski), who is the Director, Site Support. Mr. Burski reports to the W3 Vice President of Operations (Mr. C. Dugger). The PM stated that the sponsoring efforts have been important to the W3 Y2K program.

The licensee participates with other organizations that are addressing the Y2K problem. The licensee has received support via participation in the EPRI Y2K program, interfacing and sharing information with other licensees, the Combustion Engineering Owners Group, NEl/NUSMG, system vendors, the Southeast Electricity Reliability Council, and the North American Electricity ReliaFty Council.

2.2 Project Plan The W3 Y2K project plan, entitled " Year 2000 Project Guide Y2K-001," was developed by the licensee to establish the scope and control of the Y2K program, and is applied uniformly at every EOl nuclear power plant. The W3 Y2K project plan is based on the guidance provided in NEl/NUSMG 97-07, which was accepted by the NRC in NRC Generic Letter 98-01 as guidance that presents one approach for achieving Y2K readiness. The audit team's review confirmed that the W3 Y2K project plan is based on the guidance contained in NEl/NUSMG 97-07.

The W3 Y2K project plan consists of several phases gathered into the following groups:

Asset identification, Categorization, Screening and Inventory; Asset Classification and Prioritization; Detailed Assessment; Asset Testing and Post Remediation Testing; Asset l Remediation; Asset Validation and Certification; and Contingency Planning.

The schedule of W3 Y2K readiness efforts is provided in Table 1.

An independent assessment of the W3 Y2K project plan was performed October 19-21,1998.

Recommendations from that assessment are being addressed by the W3 Y2K project team in i subsequent revisions of the W3 Y2K project plan.

i

~ . - .-

l

• l 2.2.1 Awareness The EOl Y2K program, which addresses all EOl nuclear plants, began in June 1997. The licensee has undertaken efforts to brief W3 management, educate the general populatien of W3 personnel via departmental newsletters, train personnel who are to perform inventory and assessment activities, and coordinate Y2K team communications. The audit team reviewed samples of "Inside Entergy" (a W3 employees' newsletter) and presentation material used at meetings with management and all-site personnel and at continued training sessions for site personnel. The audit team also reviewed several project status reports issued by the Y2K PM to keep management informed of the schedule and cost of the project and significant activities and issues.

2.2.2 Asset Identification / Inventory The Y2K Project Team, with assistance from plant personnel, identified W3 assets that are potentially vulnerabic to Y2K date problems and includeo them in the inventory database, Y2K Asset index (Al). The inventory was screened and categorized as either Active or inactive based on whether the asset was considered Y2K vulnerable.

The inventory of potentially affected applications and embedded plant systems / components was developed by plant personnel familiar with each plant system / functional area. In the identification of embedded systems, the licensee reviewed the procedures and documentation for the existence of an internal clock or processor, surveyed the vendors for information on their equipment, performed system walk-downs, and reviewed schematics, program listings, and reference manuals on various instrumentation and control systems.

2.2.3 Asset Classification and Prioritization An Asset Classification Che:klist was used to ascertain the level of mission criticality of each asset on the basis of four broad categories of consideration, as follows:

• Regulatory - Addresses any concerns of the impact of the failure of the asset on regulatory requirements or commitments.

• Power Operations - Addresses any concerns related to the assets potential impact on continued power operations.

• . Cost Concerns - Addresses any concerns related to increased burden on W3 department staffing or budgets.

-* Industrial Safety - Addresses any concerns related to personnel safety and potential plant equipment degradation.

L Each question on the Asset Classification Checklist must be answered "Yes" or "No." Based on the responses to the questions, a classification is made, as follows:

j

• Mission Critical - An asset that, if lost and for which there is no timely remedy, would

disable or otherwise adversely impact the ability of the Entergy system to

(a) operate in i

I l

l -

_.m . _.. n_ . _ - - .

+

4 4

a safe manner, (b) provide service to customers, (c) generate revenue, or (d) avoid legal exposure.

j Important - An asset is considered important if it is not Mission Critical but places an ]

unanticipated operational requirement on plant operators, causes some degree of regulatory impact, has moderate cost impact, or has a wide scope of usage. -

Desirable - An asset is considered Desirable if it is not Mission Critical or important.

I If the asset is Mission Critical or important, a Y2K Documentation Package is required for certification of the asset. A Y2K Documentation Package Number is assigne1 by the Y2K Project for tracking purposes. The classification of each asset is also enterad into the Y2K Al.

The Y2K Project Team prioritizes the certification of the asset based on its c:assification and availability, and determines the scope and schedule for certification. The Y2K team also confirms the priority, classification, and Y2K vulnerability of the asset prior to detailed assessment activities.

1 During the analysis of the initial assessment, the licensee evaluated the failure risk of each item as the basis for assigning the priority; recommended the approach / plan for detailed ,

assessment, testing, and remediation; and estimated the detailed assessment /remediation cost. Unless specifically noted otherwise, the licensee did not formally assess and remediate desirable items. Remediation of these items may be done as time and resources permit.

Mission critical and important items will be verified to be Y2K compliant or ready by the deadline date of June 30,1999. Desirable items will not be verified to be Y2K compliant or ready by the deadline date.

The licenses identified 8 mission critical and 35 important applications that are potentially affected by the Y2K problem. The licensee also identified 272 mission critical and 365 important embedded systems that are potentially affected by Y2K problems.

Assessment / testing has been completed for nearly all of the mission critical software applications, embedded devices, and spare parts identified by the licensee.

Assessment / testing has not been completed for the software applications, digital devices and spare components important to operations. Table 2 provides a list of mission critical software systems reviewed by the audit team. Table 3 lists the embedded devices that were reviewed by the audit team.

2.2.3. Detailed Assessment Detailed assessments consist of source code examinations, review of vendor certifications and tests, testing systems in-house, and review of industry group certifications and tests. Vendor evaluations encompass evaluation of available manufacturer / developer information (such as contracts, correspondence, vendor manuals, Intemet listings, and vendor owners groups),

communications with vendors using the corporate vendor management program standard  !

vendor questionnaire, and direct communication with vendors. Test evaluations involve the development of test procedures and acceptance criteria to determine whether a Y2K problem exists.

5-The licensee was scheduled to complete its detailed assessment and certification of mission critical items by December 31,1998. This schedule has not been met because the licensee is installing new equipment during the in February 1999 refueling outage, and testing and remediation cannot be completed without this equipment. Of the 8 Mission Critical application software items,4 will not be completed until April 1999; and, of the 275 Mission Critical embedded items in 25 work projects,8 work projects are scheduled for completion by June 1999.

The detailed assessments performed to date follow the project implementation plan. The audit team found the licensee's documentation sufficient for justifying the results of the detailed assessments, which established Y2K compliance /non-compliance and readiness of systems.

The licensee is scheduled to complete Y2K detailed assessment, remediation, and testing of systems and equipment important to operations by June 30,1999.

2.2.4. Y2K Testina Y2K problem susceptibility testing is based on the licensee's determination of the importance of the affected system and knowledge of the item, prior experience with the vendor, and other relevant information. Mission critical assets classified as compliant, ready, or accept-as-is are validated by testing. The licensee's Y2K project team developed the onsite testing procedures to ensure consistency in the implementation of Y2K susceptibility testing.

Y2K testing and remediation testing is conducted in accordance with the licensee's Y2K test plan which includes, as appropriate, date setting, date roll-over of all high risk dates in powered up and powered down states, date calculations, date comparison and sorting during and following high risk dates, and date interfaces. If an asset passes all portions of the testing, the asset is categorized as compliant, ready, or use-as-is.

In certain cases, when plant equipment is also present in the plant simulator, the simulator equipment is used as the base test equipment, and the plant equipment is then certified Y2K ready by its equivalence to the simulator equipment. Equipment and firmware technical manuals and data under configuration control is the most frequently used source of information for determining equivalence between equipment. The audit team found the licensee's use of available technical manuals and data to be acceptable.

The auditors found only minor documentation discrepancies in some equipment test packages.

These discrepancies did not indicate an adverse trend in Y2K certification practices or failure to properly identify Y2K problem susceptibility.

2.2.5 Remediation Remediation is the process of retiring, replacing or modifying software or embedded software devices that are to be retained in service, but have been determined to be affected by the Y2K problem. The program implementation plan provided Y2K compliance criteria for replacement or modifical:on. After remediation is completed, validation testing is required. The licensee is performing the required Y2K remediation validation testing using the test procedures and test plans developed by the W3 Y2K project. The resulting certifications are acceptable.

l

w i

j.

If an asset is to be replaced or modified due to Y2K non-compliance or if it fails during testing in  !

L an unacceptable failure mode, then a remediation plan is prepared and the remediation effort is implemented using existing plant processes (e.g., engineering requests, software change requests, or condition identification forms). A mission critical or important asset that has been determined to be Y2K non-compliant but maintains an acceptable failure mode and effect, known to have an acceptable failure mode and effect, or remediation still has some areas of non-compliance but maintains an acceptable failure mode and effect, can be certified to be Y2K ready with the development of an acceptable-as-is justification per the Y2K plan.

2.2.6. Validation and Certification Certification is the completion and documentation of Y2K readiness activities for individual systems and components. This includes documentation of validated system modifications and development of required contingency plans.

Of the 275 Mission Critical embedded items in 25 work packages,57 embedded items have been completed in 15 packages. Two packages were to be completed by December 31,1998, and the remaining eight packages will be completed by June 1999.

2.2.7. Reaulatory Considerations The W3 Y2K project plan and associated documents included references to existing plant procedures that have guidance on regulatory considerations, such as applicability of 10 CFR50.59 for plant modification reviews, reportability evaluations per 10 CFR 50.72,10 CFR 50.73, and 10 CFR Part 21, and operability determinations as required by plant technical specifications.

2.2.8. Continaency Plannina .

The licensee has begun contingency planning using a framework similar to that described in NEl/NUSMG 98-07," Nuclear Utility Year 2000 Readiness Contingency Planning." The EOl corporate project will be providing the milestones for identifying the risks, analyzing the risks

. and creating the plans. A single corporate plan for Y2K contingencies will be developed and each plant will add site specific plans. Development of risk statements are the current high priority at W3. The contingency plan for W3 mission critical systems and components is scheduled for completion on June 30,1999. The licensee has committed 0.5 FTE from the Emergency Planning Group to contingency planning, with additional support from two of the Y2K team members. The licensee's schedule for completion of the Y2K project contingency plans is likely to be impacted by the refueling outage scheduled for February 1999 and other Y2K readiness activities for the remaining mission critical and important assets.

The licensee will use the mission critical assets to identify the high priority items periodically needed for contingency planning. W3 has significant emergency response experience because of the preparations for hurricanes and toxic chemical accidents from neighboring plants and river spills. The W3 contingency plan will provide for the staff necessary to undertake the identified actions.

1 Communications within the EOl grid are through commercial communications systems. These networks are undergoing Y2K readiness evaluation by the companies that own the systems, and are projected to be ready before the end of 1998. The licensee is also planning contingency actions for the unlikely event that communications within the EOl grid are disrupted by Y2K failures. For example, EOl has satellite phones for the transmission operations centers j and all nuclear plant emergency response facilities.  !

2.2.9. Y2K Proaram Manaaement The licensee's Y2K program schedule is aggressively tracked on a continuous basis by corporate and site management. The Y2K program progress is summarized in a format that l defines the progress of each Y2K system being evaluated. There have been schedule slippages in the W3 Y2K program, but these slippages have been addressed and accommodated in the program schedule.

I The documentation system used at W3 is well organized. Related applications and embedded  ;

systems are combined into work project documentation packages. The documentation is l maintained in an electronic database. Documents not produced on the system are scanned l into the database. The database is backed up on a weekly basis. Upon completion of the project, the documentation is printed and inserted into a hardcopy file, with a summary of the activities that were performed. This summary provides a review of the Y2K activities and conclusions for each project.

EOl has a common Y2K project implementation plan for all its nuclear facilities. The W3 Y2K project plan establishes the scope and control of the Y2K Project Plan at the W3 plant.

The licensee is preparing for a W3 rfueling outage in February 1999. Consequently, the systems and operations staff, whose support is necessary for addressing Y2K issues at W3, are not available to review Y2K work packages at this time. Directed support from the W3 Y2K Project sponsor will continue to be necessary to expedite Y2K progress at W3. The licensee has committed to provide additional resources on the Y2K project to meet the established schedules.

The Entergy Global Year 2000 policy requires that all mission critical asset work projects be completed by December 31,1998, and the work projects for important assets be completed by June 30,1999. The deadline for the Mission Critical assets will not be met at W3, and this will impact the deadline for completing the important asset work projects. There does not appear to be sufficient flexibility in the schedules to account for unforseen delays in completing the work projects. The licensee has addressed this issue by obtaining additional Y2K program resources and has set a June 30,1999 date for completing mission critical and important i assets. This revised schedule appears to be achievable with the additione.! resources.

2.2.10. Electric Grid issues  !

The licensee is interacting with the North America Electric Reliability Council (NERC) and the Southeast Electric Reliability Council (SERC) to develop grid stability plans for generation and transmission.

1

8-EOl mission critical grid assets are scheduled to be tested and certified by the end of 1998.

There are more than 1000 components in the EOl grid, which will be reviewed on a manufacture model/ version approach. The licensee will test and certify one component, and extrapolate the results of the testing to the remainder of the inventory. The licensee considered testing more than one component, but factored in the risk of damaging equipment when pulling the boards, and queried the vendors about the differences in " identical" components. The licensee has not found any problems with variations in components that have the same model/ version numbers. 1 EOl grid activities in the first half of 1999 will involve important assets and contingency planning. There are five regional transmission operation centers, which perform the switching and control of their own region. EOl is working closely with the mainframe manufacturer in the upgrading of the transmission operation centers. All but one center is being upgraded. The other center is to be certified in early 1999.

The Global Positioning System (GPS) receiver at the Beaumont center is being replaced with a Y2K compliant receiver. The GPS receiver provides the standard time, which is used as the l base time for monitoring grid frequency, i

The licensee has started reviewing blackstart procedures. Coordination portions of the blackstart procedures need to be upgraded. All blackstart procedures will be revised by May 1, )

1999. The licensee is currently developing the activities necessary for restoring the grid. An outline of the SERC contingency plan for grid recovery is due to NERC by January 26,1999.

The EOl grid capacity is approximately 24,000MW, and EOl anticipates a load of 12,000MW on

. Scturday, January 1,2000, and 16,000MW on the following Monday. EOI is conducting system

studies to look at overall grid stability, and is taking snapshots of grid performance on critical dates.. NERC requires four snapshots - 6pm Central time (Greenwich mean time is midnight),

midnight Central time, midnight Eastem time, then 2am Eastem time. NERC requires this snapshot for all utilities. This snapshot was scheduled for January 1,1999. The snapshot will d

look at frequency, voltage, generation patterns, load patterns, line flows, tie-line flows. The

purpose is to develop the models for the system availability studies. The licensee is also j studying the dynamic nature of load losses. In April 1999, the licensee will conduct a drill to test i the grid communication and backup capabilities. The black start drills will be done in the September 1999 time frame.

]-

i 2.2.11. Critical Sucoliers The licensee has prepared a list of critical suppliers, and has compared their list to the corporate list of suppliers. Letters have been sent to critical suppliers, but W3 has gotten only a 10-25% response. EOl is following up on the responses. The licensee is planning on stockpiling a 30-day supply of critical supplies as a contingency.

4 3.0 - AUDIT TEAM OBSERVATIONS

- The audit team reviewed in detail 17 of the 43 mission critical and important to operations software applications (see Table 2). The audit team also reviewed 40 of the 637 mission critical embedded systems and components identified by the licensee (see Table 3). The licensee's documentation was sufficient for justifying the results of the assessments of Y2K compliance /non-compliance.

d 9

The following observations were made by the team auditing the EOl Y2K readiness program, as it was applied at W3:

1. EOl has a common Y2K project implementation plan for all its nuclear facilities including W3. The W3 Y2K project plan entitled " Year 2000 Project Guide Y2K-001," establishes the scope and control of the Y2K Project at the W3 plant. The Y2K project plan is comprehensive and incorporates the major elements of the nuclear power industry Y2K .

problem guidance contained in Nuclear Energy Institute (NEI)/ Nuclear Utilities Software l Management Group (NUSMG) 97-07, ' Nuclear Utility Year 2000 Readiness, " and NRC ,

GL 98-01. I l

2. The licensee was preparing for a refueling outage scheduled in February 1999 at the time of the NRC audit. Consequently, the systems and operations staff and particularly the mid-level management, whose support is necessary for addressing Y2K issues at W3, were not as available as at other plants. Directed support from the W3 Y2K project sponsor and upper management will continue to be necessary to expedite Y2K program progress at W3. The licensee agreed with this observation and has committed to ,

provide more aggressive management oversight of the Y2K project. l

3. The licensee began the formal W3 Y2K readiness program in June 1997, and finished the plant inventory and initial assessment phase in May 1998. The detailed assessment phase for systems and components that are mission criticalis scheduled to be completed by December 31,1998. Confirmatory testing of mission critical systems for Y2K readiness is conducted by the licensee at the plant site. Remediation of mission critical systems has begun. No Y2K problems have been identified in safety-related systems to date.

The Entergy Global Year 2000 policy requires that all Mission Critical asset work projects be completed by December 31,1998, and the work projects for Important assets be completed by June 30,1999. However, the audit team noted that, of the 8 Mission Critical application software items,4 will be completed in April 1999, and of the 275 Mission Critical embedded items in 25 work projects,8 work projects are scheduled for completion by June 1999. Thus, the initially established deadline for the Mission Critical assets will not be met at W3, and may impact the deadline for completing the

~

Important asset work projects. Addit onally, there does not appear to be sufficient flexibility in the schedules to account for unforseen delays in completing the work projects.

To address the above schedule conflict, the licensee established a revised schedule for Y2K readiness of mission critical and important system projects by June 30,1999. The audit team believes that, with the additional resources the licensee plans to obtain in the immediate future, the above Y2K readiness schedule appears to be achievable. This conclusion is based on the expertise and experience of the present W3 Y2K project L team members, support from management, and support available via information i sharing and interactions with EPRI, other licensees, the Combustion Engineering Owners Group (CEOG), plant vendors, the Southeast Electricity Reliability Council, and the North American Electric Reliability Council.

I l

l

I

! l l I l

l

- 10- ,

l l

4. The licensee has just started Y2K contingency planning, and is using the nuclear industry guidance in NEl/NUSMG 98-07, " Nuclear Utility Year 2000 Readiness Contingency Planning," for this effort. The W3 contingency plan is scheduled for ,

completion on June 30,1999. The W3 plan will be incorporated into the EOl plan i format with a single corporate plan for Y2K contingencies and site-specific plans for i each facility connected to the EOl grid. EOl will be providing the milestones for identifying the risks, analyzing the risks, and creating the plans. At present, the licensee ,

has assigned one person part-time as lead for W3 Y2K contingency planning with i support from two Y2K project team members. The audit team considers that the schedule for completion of detailed contingency planning may be adversely impacted by the refueling outage activities and other Y2K readiness activities. The audit team believes that additional resources are needed to support W3 Y2K contingency planning ,

activities in order to meet the established W3 Y2K project schedules. The licensee has I committed to provide additional resources on contingency planning in order to meet the above schedule.

S. An independent assessment of the W3 Y2K project plan was performed by the licensee on October 19-21,1998. Recommendations from that assessment are being addressed by the W3 Y2K project team in subsequent revisions of the W3 Y2K project plan. The audit team reviewed the W3 Y2K Project team draft response to the recommendations of the assessment report and believes the proposed corrective actions should address the recommendations adequately.

6. The Y2K project documentation system used at W3 is well-organized. Related ,

applications and embedded systems are combined into work project documentation l packages. The documentation is maintained in an electronic database. Documents not produced on the system are scanned into the database. The database is backed-up on a weekly basis. Upon completion of the project, the documentation is printed and  !

inserted into a hard-copy file, with a summary of the activities that were performed. The documentation in the completed work project includes a summary of the work performed ,

and permits a rapid review of the Y2K activities status and conclusions for each project. '

Table 1 W3 Y2K Project implementation Schedule l Table 2 W3 Software Applications Reviewed by the Audit Team Table 3 W3 Embedded Systems and Components Reviewed by the Audit Team Attachment 1 Waterford 3 Y2K Audit Plan Outline Attachment 2 Documents Reviewed Attachment 3 Entrance Meeting Attendees I

l i

,mw

i i

Table 1 - W3 Y2K Project implementation Schedule i

Activity Startina Date Finishina Date  !

I' Developed inventory 6/97 5/98 Classified and Prioritized Inventory late 1997 5/98 Detailed Assessment of Mission early 1998 12/31/98 Critical Assets Readiness Certification of Mission mid 1998 6/30/99 Critical and Important Assets Contingency Planning late 1998 6/30/99 1

l l

)

l T.1 - 1

.w ,

- - . - . . ~ .. .--- - -. .- - . - . . . - . - - - . . . . . . . . .

^'

'l

• l l

, Table 2 - W3 Software Applications Reviewed by the Audit Team I

!Project ID Package ID Application Crit. Completed )

Rank Remediation WPO2O WF3-OO96 TRANSACT Code I WPO27 WF3-0193 Voice Notification System I WPO43 WF3-OO31 Risk Management Query System i ,

WPOO6 WF3-OOO6 C. R. Trending I W' CORP WF3 2237 Licensing Research System "NEW" M WPOO3 WF3-OO84 Maintenance Rule System Structures and M Components WPOO4 WF3-0205 TSC/ EOF Dose Assessment Program M 11/02/98 W'NUC WF3-0128 Bechtel Pipe Stress SW (46000007) M WPO49 WF3 2240 10CFR50.59 Database M

'WPO48 WF3-2239 FSAR/LDCR Change Log M WPO47 WF3-2238 ' Licensing Research System "OLD" M i W' CORP WF3-0194 Work Management System M W' CORP WF3-0195 Station Information Management System M l W'NUC WF3 OOO2 Project View M l

WPO26 WF3-OO48 A-Fault: version 3.5, Level 2.1 M

WPOO4 WF3-0206 Control Room Dose Assessment Program M 11/02/98 i WPO26 .WF3-OO50 DAPPER; version 3.5, Level 2.1 -

M l l

1 - Important M - Mission Critical T.2 - 1 i

Table 3 - W3 Embedded Systems and Components Reviewed by the Audit Team Project ID Package ID Embedded System / Component Crit. Completed Rank Remediation WP151 WF3-0388 Switchyard Controls i WP163 WF3-0320 TLD Equipment & Software /ERIMS - Effluents i System WP163 WF3-0321 TLD Equipment & Software ERIMS/ MET Data i WP131 WF3-0934 I (WP159 WF3-1073 CE NSSS Transient Simulator (CENTS) 1

'WP131 WF3-0935 l WP145 WF3-0309 Rosemount Transmitters I (WP100 WF3-0382 Site Security M

[WP100

WF3-0774 M iWP101 WF3-0625 Radiation Monitoring System M

[WP101 WF3-0644 M l lWP101 WF3-0645 M

{WP101 IWF3-0646 M IWP101 WF3-0678 M UP101 WF3-0686 M

[WP101 WF3-0688 M lWP101 WF3-0689 M i_WP101 WF3-0702 M

'WP101 WF3-0709 M WP101 WF3-0728 M WP101 WF3-0730 M lWP102 WF3-0404 Distributed G2 40 Multiplexer (quantity 7) M iWP102 WF3-0416 ' OpenVMS 6.1 M

[WP102  ;WF3-0423 DECSet (VaxSet) M iWP102 WF3-0810 Plant Monitoring Computer M WP103 WF3-0567 MXL Custom Software Generator, Ver 5.0E M 09/23/98 (CSG-M)

WP103 WF3-0793 Fire Detection / Control Room - Main C7ntrol M 09/23/98 Panel WP103 WF3-0800 Fire Detection Main Control Panel- MSB M 09/23/98

'WP105 WF3-0951 Valve & Loose Parts Monitoring M WP108 WF3-2195 Temperature Monitoring Panel M 09/21/98 WP112 WF3-0116 Core Element Assembly Position Display M 11/19/98 System i WP112 WF3-2232 Addressable Constants Loader Disk M 11/19/98 T.3 - 1

• l l

Table 3 - W3 Embedded Systems and Components Reviewed by the Audit Team Project ID PackageID Embedded System / Component Crit. Completed j Rank Remediation WP113 WF3-0870 ACTM Cards M 11/13/98 WP114 WF3-2244 VOTES Test Set, Version 2.5 M WP115 WF3-0600 Recorders M WP118 WF3-0350 Terry-Turbine speed control M 09/04/98 I WP120 WF3-0356 FWPT controls M WP120 WF3-0358 FW Flow Transmitters M i WP135 WF3-0386 Plant Monitors (Various) M 10/08/98 JWP162 WF3-0346 ; COLSS M l - Important M - Mission Critical I

i T.3 - 2 I

- -..- ~ -. . - . ~ . . ..- - -. . - - - - . - . . - - . . - . . . - . - - - - . - - . .

9 Waterford 3 Y2K Audit Plan Outline

- A. Project organization

'B Project Manager -

C. Project Sponsor -

1. Participation in Owners Group, Group activities related to the Y2K effort, (EPRI, NEI).  !

Peer review efforts.  !

2. Corporate activities  ;

3. Schedule of activities for Y2K readiness l Activity Starting Date Finishing Date Communication /-

. Awareness -

Project Plan Inventory ,

' Detailed analysis / testing ,

Remediation Validation / testing l l  !

Contingency Planning

4. Inventory (Review the information Database.)

Classification:

. 5. Analysis -

Number of items identified as Y2K compliant. ; Review how this was determined -

- Vendor data:any additional testing.

. Number of items not Y2K compliant -

Accept As 1s: (Review how this was determined. ___ require validation testing.

Check vendor data, Owners Group data,~any testing by vendor?)

ATTACHMENT 1

• +"y "

s a) Vendor evaluation - validation testing based on criticality of item, prior experience with vendor, extent of documentation, or plant knowledge of the item l

l b) Plant owned or supported software (including tools) evaluation - knowledge based

! decisions, scanning, testing. When testing proposed, need test specifications and procedures. t c)' interface evaluation - Part of corporate plan (?) Grid, substation, communication, I

d) Embedded components evaluation - knowledge based decisions and testing.

When sufficient vendor and plant information is available to support a knowledge-based decision, no additional testing is required. (Review the documents when this is the case.)

6. Remediation - Use of existing software procedures (?). Verify long term commitments for maintaining Y2K readiness.

7. Y2K Testing and Validation 1

Assessment testing - Per Computer problem / change reports (PCRs) and associated V&V plans and test procedures.

Testing subsequent to remediation - unit testing; integration testing; system I testing.

8. Regulator / Considerations - 10 CFR 50.59 reviews; reportability evaluations per 10 l l

CFR 50.72, 50.73 and part 21; operability determinations.  !

9. Contingency Planning - NEl/NUSMG 98-07 InWnal Risks External Risks l Remediation Risks (Vendor support, resource limitations, etc.)

l  !

10. Y2K Management Plan-Tracking against milestones of the project. Management awareness. Status reporting External resources Use of existing procedures for software QA, configuration management, V&V., i Documentation Audits (any audits done/ reports issued).

l 1

0 l

l-

e E

6 Documents Reviewed

1. W3 Y2K Project Team (Roster of Participants)

2. " Year 2000 Project Guide," Y2K-001, Rev. 3, dated November 23,1998

3. " Year 2000 Project," Procedure No. W2.115, Rev. O, dated August 26,1998.

4. Newsletter and associated W3 communications

5. Application and Embedded Systems Asset index

6. W3 Y2K Audit Entrance Meeting Slide Handouts 1

1 i

ATTACHMENT 2 1

'G Entrance Meetina - Attendees December 14.1998 Randy Douett Maintenance Manager /EOl Early C. Ewing Dir, Nuclear Safety and Regulatory Affairs  ;

Greg Pierce Director, Quality /EOl/W3

. Theodore Leonard General Manager, Plant Operations Craig Zeringue Project Coordinator / Project Management Marie Bishop EOl Y2K Project Manager Fugate Chester Operations Support /EOl Jay O'Hearn Dir, Training & Emergency Preparedness

' Andre Dejdie Supervisor, IT Mike Brandon Lic. Sup./ Licensing l Paul Gropp Manager DE/EIC  !

Tom Farnholtz NRC-SRI W3 David A. Young Lic Eng./ Licensing Dean Hertzog IT Consultant /ESI Office of Year 2000 l Chuck Bryan Y2K Team, W3

- Margit Triggs Y2K PM, W3 Maurice Rieffel Office of the Year 2000  ;

Raymond Burski Director, Site Support i Odie Tucker Y2K Embedded Lead, Project Management Chuck Dugger . Vice President, Operations, W3 j Mike Waterman NRC Matt Chiramal NRC i A.J. Wrape Director / Design Engineering I l

l b

l l

l ATTACHMENT 3 L

_ -