ML18064A878

LER 95-008-00:on 950728,discovered During Design Change Testing That None of Four Containment High Pressure Channels Would Initiate Rt.Caused by Programmatic Deficiencies. Reviewed Selected Tests & Mods from Recent Refueling Outage

ML18064A878
Person / Time
Site:	Palisades
Issue date:	08/28/1995
From:	Gire P CONSUMERS ENERGY CO. (FORMERLY CONSUMERS POWER CO.)
To:
Shared Package
ML18064A877	List: ML18064A876 ML18064A878
References
LER-95-008, LER-95-8, NUDOCS 9509050154
Download: ML18064A878 (10)

Text

NRC Form 388 U.S. NUCLEAR REGULATORY COMMISSION (9*831 APPROVED OMB NO. 3160-0104 EXPIRES: 8/31 /86 LICENSEE EVENT REPORT (LERI FACILITY NAME 111 DOCKET NUMBER 121 PAGE 131 Palisades Plant 015101010121515 1 I OF 0 19 TITLE 141 LICENSEE EVENT REPORT SS-008 - BYPASSED CONTAINMENT HIGH PRESSURE TRIPS ON REACTOR PROTECTION SYSTEM EVENT DATE 161 LEA NUMBER 161 REPORT DATE 181 OTHER FACILITIES INVOLVED 181 SEQUENTIAL REVISION FACILITY NAMES MONTH DAY YEAR YEAR NUMBER NUMBER MONTH DAY YEAR

• • • • • • • • • • • • • ************* N/A ol5lololol I 011 2la s s sis - olola - olo ola 2la sis N/A 0151010101 I THIS REPORT IS SUBMITTED PURSUANT TO THE REOUIREM ENTS OF 10 CFR I: fCh<<Jc OM or more of tha following} 1111 OPERATING N 20.402(b) 20.4061cl 60.731*11211ivl 73.711bl POWER MODE 191 20.4061*111 Hi) - 60.361cll1l x 60.731*1121M - 73.711cl LEVEL 20.4061*111 lliil 60.361cll2l 60.731all211viil OTHER (Specify in Abstract (10) o I o lo 20.4061*111 lliii) 60.731*112Hil 60. 7 311112llviii11Al below and in Text,

- 20.4061*111 )(iv) 20.4061*1111M 60.731*112llii) 60.731*112lliiil LICENSEE CONTACT FOR THIS LER 1121

- 60.731all211viii11Bl 60.731*11211xl NRC Form 366Al NAME TELEPHONE NUMBER Paul J Gire, Staff Licensing Engineer sARrA,Cr~ 1 7 1 6 1 4 1 - 18 1s 1, 13

• CAUSE SYSTEM COMPONENT COMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT 1131 MANUFAC*

TUR ER REPORTABLE TO NPRDS

~

CAUSE SYSTEM COMPONENT MANUFAC*

TUR ER REPORT ABLE TO NPRDS w.

I I

I I I I I I r I I I I I SUPPLEMENTAL REPORT EXPECTED 1141 I

I I

I I I I I I I I I EXPECTE.0 I I MONTH DAY I YEAR

'Xl YES Uf YH, com,Wr. EXPECTED SUBMISSION DA TEI hNO SUBMISSION DATE 1161 ,I o 210 s *1 s ABSTRACT IJ.jmit to 1400 spacoa, i.e., approximor.ly fifteen llingkt-spactt typttwrimn lintt*I 116)

On July 28, with the plant in cold shutdown condition, all control rods inserted, and at refueling boron, it was discovered during design change testing. that none of the four containment high pressure channels would initiate a reactor trip. An undesired connection across the containment high pressure trip contacts was discovered in the printed circuit boards of the interconnection modules. The connection was introduced as a result of a modification installed in 1992 which replaced all four channels of reactor protection because of high maintenance and obsolescence of spare parts. This connection provided a circuit path that bypassed the containment high pressure trip to the matrix logic of the reactor protection system, thereby disabling the containment high pressure trip circuit.

The problem was caused by programmatic deficiencies in the design change program, testing program, and industry experience review program.

Corrective actions completed include a review of selected tests and modifications from the recently completed refueling outage to ensure system and component performance was verified properly. Corrective actions to be completed include the establishment of a modifications teaming approach to enhance oversight of design changes, and the formation of a testing authority to oversee all design and surveillance testing.

9509050154 950828

~DR ADOCK 05000255

--*- -** *----**-****-**-**-------- ~.OR ..

NRC Form 388A U.S. NUCLEAR REGULATORY COMMISSION (9-83) APPROVED OMB NO. 3160-0104 EXPIRES: 9/31/86 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER 121 LER NUMBER (31 PAGE (4)

SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 - 0 I0 I8 - 0 I0 0 I2 OF 0 I9 EVENT DESCRIPTION - ~

On July 28, with the plant in cold shutdown condition, all control rods inserted, and at refueling boron, it was discovered during design change testing that none of the four containment high pressure (CHP) channels of the reactor protection system (RPS),[JD], would initiate a reactor trip.

The design change testing followed recently completed minor upgrades to the RPS trip unit connector blocks. An undesired connection across the CHP trip contacts was discovered in the printed circuit boards of the interconnection modules. The connection was introduced as a result of a separate 1992 modification which replaced all four channels of the RPS because of high maintenance and obsolescence of spare parts. This connection provided a circuit path that bypassed the CHP trip to the matrix logic of the RPS, thereby disabling the CHP trip circuit on all four independent safety channels.

The Palisades RPS design contains twelve available trip units in each of the four separate safety channels. The initial plant design contained ten active trip functions, with the two remaining unused trip functions bypassed to complete the systern logic. Prior to initial criticality, a CHP trip function was added to the plant design, and the eleven active trips remained essentially untouched until a modification was completed in March of 1992. The 1992 modification included the replacement of all twelve trip units for the four channels of the RPS system because of high maintenance and obsolescence of spare parts. Facility Change FC-888, 11 Upgrade RPS Bistable Trip Units And Power Supplies, 11 was designed by the original equipment supplier, Combustion Engineering (CE). The original wiring list showing two bypassed trip units was inadvertently implemented without adequate validation and verification as a design input into the planned upgrade. The bypassed configuration of the CHP trip units was not identified during post modification testing in the 1992 refueling outage nor during monthly TS surveillance testing due to inadequate test procedures.

Upon discovery of the disabled CHP trip functions, a multidisciplinary root cause analysis team was established. Insight on the use of Event and Causal Factor Analysis was provided by an individual from the Institute of Nuclear Power Operations (INPO) during the first week of the root cause evaluation. Concurrent with the Palisades evaluation, a root cause evaluation was initiated by CE to assess the adequacy of the quality controls and processes utilized in the CE Nuclear Services Engineering Department. The circuit boards were modified to re-enable the CHP trip function and an operability test was developed and completed for all trip functions on the four channels of the RPS.

CAUSES OF THIS EVENT The root cause evaluation team identified numerous inappropriate actions and program deficiencies that are categorized into three primary program areas with five identified primary root causes. Th~ three primary program areas are;

NRC Form 3BBA U.S. NUCLEAR REGULATORY COMMISSION 19-83) APPROVED OMB NO. 3160-0104 EXPIRES: Bl31186 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE 141 SEQUENTIAL . REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 - 0 I0 I8 - 0 I0 0 I3 OF 0 I9 A. Inadequate design control program, B. Inadequate testing control program, C. Inadequate industry experience review program.

The five primary root causes are:

1. Inadequate program scope

2. Inadequate prioritization of work

3. Inadequate program monitoring or management

4. Inadequate attention to emerging problems

5. Lack of commitment to program implementation Tt:ie significant findings with their associated root causes are categorized below within the program area associated with the problem:

A. INADEQUATE DESIGN CONTROL PROGRAM Root Cause #1 Inadequate Program Scope, (omission of necessary functions or guidance in procedures):

1. Plant procedures did not alert users that vendor information was uncontrolled and COL!ld be inaccurate. Vendor documents (manuals and drawings) were not always maintained up to date. This became an event precursor when the FC-888 project engineer removed the outdated wire list from the vendor file and used it as a design input without adequate validation and verification. The wiring list still had a circuit bypass for the CHP channel since the initial plant design did not include a CHP trip.

2. Plant procedures did not establish clear roles and responsibilities for the implementation of modifications. Besides the Prime Design Reviewer, no other personnel on site felt direct ownership for the successful implementation of the modification.

Root Cause #2 Inadequate Prioritization Of Work, (inappropriate application of resources due to a misunderstanding of task significance or complexity or availability of resources):

1. The plant Prime Design Reviewer (PDR) and the Combustion Engineering (CE) designer, who acted as the Respons!ble Engineer, were both inexperienced. The assignment of the inexperienced engineers to this modification was due to a lack of available experienced personnel and the mindset of management that the RPS modification was one of limited complexity and scope. It was believed that CE as a whole had the knowledge and skills to balance out this inexperience. The new RPS design was a clone of the early 1980's RPS System designs that were used for St. Lucie and Millstone nuclear plants.

NRC Form 366A U.S. NUCLEAR REGULATORY COMMISSION 19-831 APPROVED OMS NO. 3160-0104 EXPIRES: 8/31/86 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 5 0 0 0 2 5 5 9 5 - .0 0 8 - 0 0 0 4 OF Q 9

2. There were very few modification team meetings conducted. All of the technical reviews were completed by CE. Palisades personnel only provided cursory reviews of portions of the design. At that time, there was no programmatic guidance to provide system engineer involvement or ownership in the modification planning and reviews.

Root Cause #3 Inadequate Program Monitoring or Management, (insufficient oversight and self assessment):

1. There was an over reliance on CE due to their being on the approved suppliers list and being the original equipment manufacturer. CE's design reviews and testing were less than adequate and the plant oversight was not effective.

Root Cause #4 Inadequate Attention To Emerging Problems, (ineffective problem identification and root cause analysis):

1. During modification preparations, CE was behind schedule and several problems were identified during design reviews, testing and installation. With the lack of experience and attention associated with the oversight of this modification, Palisades was unable to recognize the generic implications.

2. During the installation of the RPS modification in 1992, two design problems were identified. The problems could have been linked to the use of the same incorrect wiring list. If these two conditions had been sufficiently evaluated in 1992 for generic implications, the precursor (inaccurate wiring list) could have been identified and the inadvertent CHP bypass could have been discovered at that time. Corrective action documents were not written for the two problems noted during the modification installation. The corrective action process initiated in 1994 is more intrusive and has resulted in Condition Reports (CRs) being generated and evaluated in a more timely manner for similar issues associated with modification problems.

B. INADEQUATE TESTING CONTROL PROGRAM Root Cause #1 Inadequate Program Scope, (omission of necessary functions or guidance in procedures):

Root Cause #5 Lack of Commitment to Program Implementation

1. The plant procedures and organization lacked a single point of contact for testing to ensure consistent application of proper testing techniques and overlap.

NRC Form 388A U.S. NUCLEAR REGULATORY COMMISSION 18-831 APPROVED OMB NO. 3160-0104 EXPIRES: 8/31 /86 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 - 0 I0 I8 - 0 I0 0 I5 OF 0 I9

2. There was an inappropriate reliance on the existing TS Surveillance Test M0-3, "Reactor Protection Matrix Logic Tests," which provided a portion of the post modification verification. The planned approach for the post modification testing consisted of two separate tests. A test procedure written by CE tested the channel relays for proper operation. The second test was the normal monthly Palisades TS test M0-3. This test was used to verify functional operability of the channel relay output to the RPS matrix logic. The TS test M0-3 was used without verifying that it would functionally test all the matrix logic that needed to be tested after the modification.

A simplified block diagram of the RPS testing from FC-888 is shown as Attachment 1. It was determined that M0-3 was not fully meeting the intent of Tech Spec Table 4.17 .1 item 13, RPS matrix logic testing every 31 days. M0-3 did verify that the High Containment Pressure (CHP) relay *contact was opening (single pole double throw contact) but it failed to determine if the contact was being bypassed. The TS surveillance procedure was revised to assure adequate testing overlap to verify the RPS matrix logic performance. This adverse condition pointed out the causal factor that the surveillance test program lacked direction in functional testing of Tech Spec equipment including assurance of adequate overlap of all functions being tested.

Root Cause #2 Inadequate Prioritization Of Work, (inappropriate application of resources due to a misunderstanding of task significance or complexity or availability of resources):

1. The existing roles and responsibilities of the system engineers allows other priorities to take precedence over a thorough review of existing TS Surveillance Procedures or new modification test procedures. This situation led to cursory reviews of the modification test procedures.

Root Cause #3 Inadequate Program Monitoring Or Management, (insufficient oversight and self assessment):

1. The CE Factory Acceptance Test (FAT) scope should have identified the undesired bypass for the CHP trip. However, there were problems with the test procedure being written incorrectly, identifying that CHP trip test lamps should be lit when they should have been off and, conversely, off when they should have been lit. Currently it is being assumed that the test rig for the FAT was wired incorrectly, which resulted in a failure to identify the CHP bypass. Plant oversight of the FAT procedure and performance was inadequate. The Palisades PDR and Test Engineer had other commitments and were not present to oversee the FAT. The System Engineer and two instrument and control technicians observed a portion of the FAT solely to familiarize themselves with the new system hardware. The FAT failure to locate the CHP bypass was a causal factor indicating that CE's test program was less than adequate.

. NRC Form 388A U.S. NUCLEAR REGULATORY.COMMISSION (9-831 APPROVED OMB NO. 3160-0104 EXPIRES: 8/31/86 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER 121 LER NUMBER 131 PAGE (4)

SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant Q 5 Q Q Q 2 5 5 9 5 - Q Q 8 - Q Q Q 6 OF Q 9 C. INADEQUATE INDUSTRY EXPERIENCE REVIEW PROGRAM Root Cause #3 Inadequate Program Monitoring Or Management, (insufficient oversight and self assessment):

1. The documented evaluations of several similar Industry Experience Events (INs) were reviewed and found to be inadequate. Four of the INs point out inadequate post modification testing, inadequate overlap of testing, and inadequate testing of safety related circuits. There was a tendency to provide only a cursory review of the implications of the industry events with a general assumption that the present testing methods are inherently valid.

SAFETY SIGNIFICANCE Preliminary reviews indicate that the safety significance of not tripping the reactor on CHP would be minor if not negligible. Palisades is working with ABB~CE and our fuel vendor, Siemens Power Co. to quantify the final predicted impact. Final results should be available in September and a supplement to this LER will be provided by 10/20/95.

The failure to trip the reactor on CHP impacts two sections of the FSAR. FSAR section 14. 14,,

"Steam Line Break Inside Containment," takes credit for the reactor trip on CHP at 2.55 seconds into the event. Preliminary review indicates that without CHP the next trip that would occur would be the Variable High Power (VHP) trip at approximately 7 seconds. This would present a slightly higher challenge to the Primary Coolant System (PCS) in the first 20 seconds of the main steam line break (MSLB), event but this condition is bounded by the consequences of the rod withdrawal event. Delaying the reactor trip will actually make the overall cool down less severe, which would reduce the return to power predicted for the MSLB. Therefore, the present .analysis prediction for fuel failure should be bounded by the MSLB inside containment with the CHP trip of the reactor disabled.

The second FSAR section impacted is 14. 18, "Containment Pressure and Temperature Response." The mass and energy release rate analysis for both the Loss of Coolant Accident (LOCA) and Main Steam Line Break (MSLB) took credit for CHP trip of the reactor. Initial indications are that there would be no impact for the LOCA because the reactor would shut down from voids for this event. The MSLB would have additional energy released from the PCS because the reactor would not trip until approximately 7 seconds from VHP. The additional energy will have a minimal impact on the predicted containment pressure and temperature response following a MSLB. Reanalysis is being performed to quantify the impact.

NRC Form 388A U.S. NUCLEAR REGULATORY COMMISSION (8-831 APPROVED OMB NO. 3160-0104 EXPIRES: B/31 /86 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME (1 I DOCKET NUMBER 121 LER NUMBER 131 PAGE (41 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant Q 5 Q 0 0 2 5 5 9 5 - 0 0 8 - 0 Q 0 7 OF 0 9 CORRECTIVE ACTIONS TAKEN AND RESULTS ACHIEVED The following corrective actions were completed and are described in detail in the associated plant condition report C-PAL-95-1117:

1. The RPS matrix channels have been modified to restore the CHP trip function.

2. The Technical Specifications Surveillance Test (M0-3) for the RPS matrix logic was revised to provide adequate overlap in testing. This will assure that the requirements of Technical Specification Table 4. 17 .1 item 13 are being adequately verified during the monthly test.

3. A review of 24 Technical Specifications Surveillance Procedures, post modification testing completed during the 1995 REFOUT, and Special Tests completed during the 1995 REFOUT was completed. This review was performed to verify that procedures used for testing safety related circuits completely test the design functions of the equipment. As part of this review, condition reports were written and procedures were revised. A number of procedur enhancements were recommended to the procedure sponsors. None of these deficien~ies resulted in any of the equipment being declared inoperable.

4. An evaluation was completed for all of the functional equivalent substitu.tion (FES) and specification change (SC) packages installed in 1995 REFOUT. If the FES or SC changed the logic matrix of either the RPS or Engineering Safeguards initiation, the acceptance test was reviewed to verify that the logic matrix functioned properly. This evaluation resulted in two SC' s and two FES' s being reviewed.

The testing which verified each project was found acceptable. There was a concern about the testing for SC-95-033 because the testing was not comprehensive in that the test only verified proper operation of the modified contact and did not verify that the remainder of the logic had not been disturbed. Additional surveillance testing fully verified the adequacy of the entire circuit. *

5. An independent evaluation of the testing adequacy for FES-95-032, "RPS Trip Unit Connector Block Replacement" was completed. The evaluation consisted of a review of trip module inputs and outputs affected by the modification to determine the extent to which all functions were checked as part of the post modification test. The scope of the work affected more than 2500 pins in the connector blocks. The results of the assessment indicate that functions using 2% of the pins were not verified and 9% of the pins were not thoroughly tested. The evaluation determined that the post modification testing was less than adequate. Operability of the RPS was subsequently verified by a new test.

NRC Form 388A U.S. NUCLEAR REGULATORY COMMISSION 19-83) APPROVED OMS NO. 3160-0104 EXPIRES: 8/31 /86 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 11) DOCKET NUMBER 121 LER NUMBER 13) PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0600026693 0 8 OF 0 9 The test plan was developed to declare the RPS operable. The plan objective was to assure full operability of the RPS through the use of overlap and/or full functional testing.

The strategy assured that every RPS input and output functioned properly through functional tests and finally, assured total operability of the RPS safety and non-safety related functions. This evaluation found six data logger outputs not functioning. A condition report (C-PAL-95-1284) and work orders were initiated to adequately resolve these .problems. It was also determined that time response testing of the RPS had not been performed as part of the Thermal Margin Monitor (TMM) modification performed in 1992. Calculations were performed to verify adequate safety margins. RPS time response testing is currently scheduled for the 1997 REFOUT.

The testing plan was reviewed by ABB-CE. The review was formalized and documented testing recommendations and enhancements to assure full RPS operability.

Corrective Actions to Avoid Recurrence

1. Provide enhancements to the Design *control Program by implementing the following changes:

a. Add a requirement to Administrative Procedure (AP) 10.44, "Engineering Records Center Distribution and Control of Design Documents," to attach a notice to any second generation print made to a vendor document, manual or drawing being issued by ERC to indicate that they have not been maintained as living documents.

b. Revise AP 9.44, "Design Document Control," to indicate vendor manuals and vendor drawings have not been maintained as living documents.

c. Revise AP 9.45, "Vendor Manual Control," to indicate vendor manuals and vendor drawings have not been maintained as living documents.

d. Revise AP 9.03, "Facility Change," to add the requirement to initiate condition reports when potentially generic, or significant issues are discovered during Facility Change project work, and eliminate the option of using an inexperienced Prime Design Reviewer (PDR).

e. Revise AP 9.04, "Specification Changes," to add the requirement to initiate condition reports when potentially generic, or significant issues are discovered during Specification Change project work.

NRC Form 388A U.S. NUCLEAR REGULATORY COMMISSION 19-831 APPROVED OMS NO. 3160-0104 EXPIRES: B/31/86 LICENSEE EVENT REPORT ILER) TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0500025593 Q 9 . OF Q 9

f. Provide continuing training to engineering support personnel using a case study methodology to include as a minimum: testing program improvements, management expectations for testing and consultant/vendor scrutiny, and industry experience report evaluations. *

g. Develop a guidance document on the formation of teams to perform modifications. The team should take the lead for the modification and have roles an~ responsibilities established.

2. Provide enhancements to the Test Control Program by implementing the following changes:

a. Assign responsibility for all modification testing (FC, SC, FES, TM) to Systems Engineering which will act as the testing authority. Surveillance tests within Systems Engineering shall meet the standards established by the testing authority.

b. Revise applicable administrative procedures to emphasize functional testing requirements and accountability for test adequacy/completeness. Ensure procedures include the expectation of 100% functional testing of anything changed or affected directly or indirectly by the work done for a project. Include a description of overlapping when relying on multiple tests to meet requirements.

3. Provide enhancements to the Industry Experience Program by implementing the following changes:

a. Re-evaluate the documented responses to similar Industry Experience Reports (IN) pertaining to inadequc;1te circuit modifications and testing, (IN-88-83, IN-92-65, and IN-93-38).

b. Develop a plan to implement reviews to comply with NRC intended actions for the n*ew (currently in draft) Generic Letter No. 95-XX: "Testing of Safety-Related Logic Circuits" (IN 95-15) published in the Federal Register May 22, 1995, page 27143.

c. Implement an independent technical review and approval of industry experience evaluations by a Safety and Design Reviewer. Re-emphasize the importance of management expectations for industry experience reviews.

ATTACHMENT 1 CONSUMERS POWER COMPANY PALISADES PLANT DOCKET 50-255 BLOCK DIAGRAM OF RPS MODIFICATION TESTING I Page