ML18065A205: Difference between revisions
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
||
Line 24: | Line 24: | ||
REVISION NUMBER MONTH DAY YEAR FACILITY NAMES N/A ojsjojojol I 017 2la 9 5 915 .. | REVISION NUMBER MONTH DAY YEAR FACILITY NAMES N/A ojsjojojol I 017 2la 9 5 915 .. | ||
olola - 0 I1 1 10 210 915 N/A oI s I o I oI o I I THIS REPORT IS SUBMITTED PURSUANT JO THE REQUIREMENTS OF 10 CFR ! : !Check one or more of rhe following/ {11) | olola - 0 I1 1 10 210 915 N/A oI s I o I oI o I I THIS REPORT IS SUBMITTED PURSUANT JO THE REQUIREMENTS OF 10 CFR ! : !Check one or more of rhe following/ {11) | ||
OPERATING N 20.402{b) 20.40S{c) SO. 73{a){2){iv) 73.71{b) | OPERATING N 20.402{b) 20.40S{c) SO. 73{a){2){iv) 73.71{b) | ||
Io I o lo - | Io I o lo - | ||
MODE {9) | MODE {9) | ||
POWER LEVEL 1101 | POWER LEVEL 1101 | ||
--- _-*..* ) > | --- _-*..* ) > | ||
20.40S{a){1 ){i) 20.40S{a){1){ii) - | 20.40S{a){1 ){i) 20.40S{a){1){ii) - | ||
S0.36{c){1) | S0.36{c){1) | ||
S0.36{c){2) | S0.36{c){2) x SO. 73{a){2){v) | ||
x SO. 73{a){2){v) | |||
SO. 73{a){2){vii) * - | SO. 73{a){2){vii) * - | ||
73.71{c) | 73.71{c) | ||
OTHER {Specify in Abstract | OTHER {Specify in Abstract | ||
< 20.40S{a){ 1){iii) | < 20.40S{a){ 1){iii) | ||
- S0.73{a){2){i) | - S0.73{a){2){i) | ||
....__ SO. 73{a){2){viii){A) below and in Text, | ....__ SO. 73{a){2){viii){A) below and in Text, 20.40S{a){1 ){iv) 20.40S{a){1 ){v) - so. 73{a){2){ii) | ||
20.40S{a){1 ){iv) 20.40S{a){1 ){v) - so. 73{a){2){ii) | |||
S0.73{a){2){iii) | S0.73{a){2){iii) | ||
LICENSEE CONTACT FOR THIS LER {12) | LICENSEE CONTACT FOR THIS LER {12) | ||
SO. 73{a){2){viii){B) | SO. 73{a){2){viii){B) | ||
Line 77: | Line 62: | ||
** | ** | ||
* i u<: | * i u<: | ||
./\/*<:\ | ./\/*<:\ | ||
.. ) | .. ) | ||
Line 108: | Line 92: | ||
: 5. Lack of commitment to program implementation The significant findings with their associated root causes are categorized below within the program area associated with the problem: | : 5. Lack of commitment to program implementation The significant findings with their associated root causes are categorized below within the program area associated with the problem: | ||
A. INADEQUATE DESIGN CONTROL PROGRAM Root Cause #1 Inadequate Program Scope, (omission of necessary functions or guidance in procedures): | A. INADEQUATE DESIGN CONTROL PROGRAM Root Cause #1 Inadequate Program Scope, (omission of necessary functions or guidance in procedures): | ||
1*. Plant procedures did not alert users that vendor information was uncontrolled C!nd_ could be inaccurate. Vendor documents (manuals and drawings) were not always maintained | 1*. Plant procedures did not alert users that vendor information was uncontrolled C!nd_ could be inaccurate. Vendor documents (manuals and drawings) were not always maintained up to date. This became an event precursor when the FC-888 project engineer removed the outdated wire list from the vendor file and used it as a design input without adequate validation and verification. The wiring list still had a circuit bypass for the CHP channel since the initial plant design did not include a CHP trip. | ||
up to date. This became an event precursor when the FC-888 project engineer removed the outdated wire list from the vendor file and used it as a design input without adequate validation and verification. The wiring list still had a circuit bypass for the CHP channel since the initial plant design did not include a CHP trip. | |||
: 2. Plant procedures did not establish clear roles and responsibilities for the implementation of modifications. Besides the Prime Design Reviewer, no other personnel on site felt direct ownership for the successful implementation of the modification. | : 2. Plant procedures did not establish clear roles and responsibilities for the implementation of modifications. Besides the Prime Design Reviewer, no other personnel on site felt direct ownership for the successful implementation of the modification. | ||
Root Cause #2 Inadequate Prioritization Of Work, (inappropriate application of resources due to a misunderstanding of task significance or complexity or availability of resources): | Root Cause #2 Inadequate Prioritization Of Work, (inappropriate application of resources due to a misunderstanding of task significance or complexity or availability of resources): | ||
Line 140: | Line 122: | ||
* NRC Form 366A | * NRC Form 366A | ||
{9-83) | {9-83) | ||
LICENSEE EVENT REPORT (LERI TEXT CONTINUATION U.S. NUCLEAR REGULATORY COMMISSION APPROVED OMB NO. 3150-0104 EXPIRES: 8/31/85 FACILITY NAME {1 J DOCKET NUMBER 121 LER NUMBER {3) PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 -- 0 I0 I8 - 0 I1 0 I6 OF 0 I9 C. INADEQUATE INDUSTRY EXPERIENCE REVIEW PROGRAM Root Cause #3 Inadequate Program Monitoring Or Management, (insufficient oversight and self assessment): | |||
: 1. The documented evaluations of several similar Industry Experience Events (INs) were reviewed and found to be inadequate. Four of the INs point out inadequate post modification testing, inadequate overlap of testing, and inadequate testing of safety related circuits. There was a tendency to provide only a cursory review of the implications of the "' | : 1. The documented evaluations of several similar Industry Experience Events (INs) were reviewed and found to be inadequate. Four of the INs point out inadequate post modification testing, inadequate overlap of testing, and inadequate testing of safety related circuits. There was a tendency to provide only a cursory review of the implications of the "' | ||
industry events with a general assumption that the present testing methods are inherently valid. | industry events with a general assumption that the present testing methods are inherently valid. | ||
Line 150: | Line 132: | ||
The FSAR analysis for fuel failure during a MSLB remains bounding even if the reactor trip is delayed due to the loss of the CHP trip function. In the short term, a postulated MSLB would produce an overpower transient which would terminate in a reactor trip on VHP. This initial power ramp would not result in a more severe challenge to the specified acceptable fuel design limits than the existing FSAR analysis results for a postulated fast withdrawal of a control rod bank. In the long term, the effect of delaying the reactor trip by - 3.5 seconds would result in a reduction in the cooling of the affected primary loop. The reduction in cooling would be equal to the amount of power produced in the 3.5 second delay period. Operation at power for several seconds after a MSLB will result in a modest increase in both the secondary side pressure and the break flow rate during the initial portion of the steam generator blowdown. These increases would partially offset the additional heat added by the reactor core. The net effect would be to decrease the cooldown, to reduce the return to power, and to make the transient less severe and less of a challenge to DNB or fuel failure. There would be no significant effect on plant operators' responses to the event. | The FSAR analysis for fuel failure during a MSLB remains bounding even if the reactor trip is delayed due to the loss of the CHP trip function. In the short term, a postulated MSLB would produce an overpower transient which would terminate in a reactor trip on VHP. This initial power ramp would not result in a more severe challenge to the specified acceptable fuel design limits than the existing FSAR analysis results for a postulated fast withdrawal of a control rod bank. In the long term, the effect of delaying the reactor trip by - 3.5 seconds would result in a reduction in the cooling of the affected primary loop. The reduction in cooling would be equal to the amount of power produced in the 3.5 second delay period. Operation at power for several seconds after a MSLB will result in a modest increase in both the secondary side pressure and the break flow rate during the initial portion of the steam generator blowdown. These increases would partially offset the additional heat added by the reactor core. The net effect would be to decrease the cooldown, to reduce the return to power, and to make the transient less severe and less of a challenge to DNB or fuel failure. There would be no significant effect on plant operators' responses to the event. | ||
NRC Form 366A (9-831 | NRC Form 366A (9-831 | ||
* U.S. NUCLEAR REGULATORY COMMISSION APPROVED OMB NO. 3150-0104 EXPIRES: 8/31/85 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE (41 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 - 0 I0 I8 -- 0 I1 0 I7 OF 0 I9 The second FSAR section impacted is 14.18, "Containment Pressure and Temperature Response." The mass and energy release rates for both the Loss of Coolant Accident (LOCA) and a MSLB are used to predict the containment pressure and temperature response. The LOCA analysis did not take credit for the CHP trip function. As discussed in the previous paragraphs, the MSLB analysis did take credit for the CHP trip function. The impact from the delayed reactor trip would have been additional energy addition from the primary system to the secondary system during the delay period of - 3.5 seconds. The additional energy addition results in a minor increase in the predicted containment pressure and temperature response following the MSLB. | * U.S. NUCLEAR REGULATORY COMMISSION APPROVED OMB NO. 3150-0104 EXPIRES: 8/31/85 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE (41 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 - 0 I0 I8 -- 0 I1 0 I7 OF 0 I9 The second FSAR section impacted is 14.18, "Containment Pressure and Temperature Response." The mass and energy release rates for both the Loss of Coolant Accident (LOCA) and a MSLB are used to predict the containment pressure and temperature response. The LOCA analysis did not take credit for the CHP trip function. As discussed in the previous paragraphs, the MSLB analysis did take credit for the CHP trip function. The impact from the delayed reactor trip would have been additional energy addition from the primary system to the secondary system during the delay period of - 3.5 seconds. The additional energy addition results in a minor increase in the predicted containment pressure and temperature response following the MSLB. | ||
Line 178: | Line 159: | ||
; | ; | ||
NRC Form 366A (9-831 | NRC Form 366A (9-831 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION U.S. NUCLEAR REGULATORY COMMISSION APPROVED OMB ND. 3150-0104 EXPIRES: B/31 /85 FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER (31 PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant o Is IoIoIoI 2 I s I s s Is - o lo I a - o I, o Is OF o Is | ||
: c. Revise AP 9.03, "Facility Change," to add the requirement to initiate condition reports when potentially generic, or significant issues are discovered during Facility Change Project work, and eliminate the option of using an inexperienced Prime Design Reviewer. | : c. Revise AP 9.03, "Facility Change," to add the requirement to initiate condition reports when potentially generic, or significant issues are discovered during Facility Change Project work, and eliminate the option of using an inexperienced Prime Design Reviewer. | ||
: d. Revise AP 9.04, "Specification Changes," to add the requirement to initiate condition reports when potentially generic, or significant issues are discovered during Specification Change Project work . | : d. Revise AP 9.04, "Specification Changes," to add the requirement to initiate condition reports when potentially generic, or significant issues are discovered during Specification Change Project work . | ||
Line 192: | Line 172: | ||
{IN 95-15) published in the Federal Register May 22, 1995, page 27143. | {IN 95-15) published in the Federal Register May 22, 1995, page 27143. | ||
: c. Implement a second level critical review of industry experience evaluations by Systems Engineering. Re-emphasize the importance of management expectations for industry experience reviews. | : c. Implement a second level critical review of industry experience evaluations by Systems Engineering. Re-emphasize the importance of management expectations for industry experience reviews. | ||
* ATIACHMENT 1 CONSUMERS POWER COMPANY* | * ATIACHMENT 1 CONSUMERS POWER COMPANY* | ||
PALISADES PLANT DOCKET 50-255 BLOCK DIAGRAM OF RPS MODIFICATION TESTING 1 Page | PALISADES PLANT DOCKET 50-255 BLOCK DIAGRAM OF RPS MODIFICATION TESTING 1 Page |
Revision as of 10:34, 3 February 2020
ML18065A205 | |
Person / Time | |
---|---|
Site: | Palisades |
Issue date: | 10/20/1995 |
From: | Gire P CONSUMERS ENERGY CO. (FORMERLY CONSUMERS POWER CO.) |
To: | |
Shared Package | |
ML18065A203 | List: |
References | |
LER-95-008, LER-95-8, NUDOCS 9510300219 | |
Download: ML18065A205 (11) | |
Text
NRC F~*m 366 U.S. NUCLEAR REGULATORY COMMISSION
{9-83) APPROVED OMB NO. 31S0-0104 EXPIRES: 8/31 /8S LICENSEE EVENT REPORT (LERI FACILITY NAME {1) DOCKET NUMBER {2) PAGE 13)
Palisades Plant o I Io Io Io I I I 5 2 5 5 1 I OF o I9 TITLE {4)
LICENSEE EVENT REPORT 95-008- BYPASSED CONTAINMENT HIGH PRESSURE TRIPS ON REACTOR PROTECTION SYSTEM EVENT DATE {S) LER NUMBER {6) REPORT DATE {6) OTHER FACILITIES INVOLVED {8)
MONTH DAY YEAR YEAR II SEQUENTIAL NUMBER I
REVISION NUMBER MONTH DAY YEAR FACILITY NAMES N/A ojsjojojol I 017 2la 9 5 915 ..
olola - 0 I1 1 10 210 915 N/A oI s I o I oI o I I THIS REPORT IS SUBMITTED PURSUANT JO THE REQUIREMENTS OF 10 CFR ! : !Check one or more of rhe following/ {11)
OPERATING N 20.402{b) 20.40S{c) SO. 73{a){2){iv) 73.71{b)
Io I o lo -
MODE {9)
POWER LEVEL 1101
--- _-*..* ) >
20.40S{a){1 ){i) 20.40S{a){1){ii) -
S0.36{c){1)
S0.36{c){2) x SO. 73{a){2){v)
SO. 73{a){2){vii) * -
73.71{c)
OTHER {Specify in Abstract
< 20.40S{a){ 1){iii)
- S0.73{a){2){i)
....__ SO. 73{a){2){viii){A) below and in Text, 20.40S{a){1 ){iv) 20.40S{a){1 ){v) - so. 73{a){2){ii)
S0.73{a){2){iii)
LICENSEE CONTACT FOR THIS LER {12)
SO. 73{a){2){viii){B)
SO. 73{a){2){x)
NAME TELEPHONE NUMBER Paul J Gire; Staff Licensing Engineer 6AjEA,Cr~ 1 7 1 6 14 1 _18 19 1, 13 COMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT {13)
+****~
MANUFAC* REPORTABLE MANUFAC* REPORTABLE CAUSE SYSTEM COMPONENT TURER TO NPRDS In CAUSE SYSTEM COMPONENT TURER TO NPRDS I
I I
I I
I I I I I
I I
I I
SUPPLEMENTAL REPORT.EXPECTED {14)
I I I
I I
I I
I I
I I
EXPECTED I
I I
I MONTH DAY
- i u<:
./\/*<:\
.. )
YEAR
~ YES (Jf yes, comp/ere EXPECTED SUBMISSION DATE) h--1 NO SU8MIS_SION DATE {1S)
I I. I.
ABSTRACT {Umir ro 1400 spaces, i.e .. approximately fifteen single-space typewritten lines) {16)
On July 28, 1995 with the plant in cold shutdown condition, all control rods inserted, and at _
refueling boron, it was discovered during design change testing that none of the four containment high pressure channels would initiate a reactor trip. An undesired connection across the containment high pressure trip contacts was discovered in the printed circuit boards of the interconnection modules. The connection was introduced as a result of a modification installed in 1992 which replaced all four channels of reactor protection be.cause of high maintenance and obsolescence of spare parts. This connection provided a circuit path that bypassed the containment high pressure trip to the matrix logic of the reactor protection system, thereby disabling the containment high pressure trip circuit.
The problem was caused by programmatic deficiencies in the design change program, testing program, and industry experience review program.
Corrective actions completed include a review of selected tests and modifications from the recently completed refueling outage to ensure system and component performance was verified properly. Corrective actions to be completed include the establishment of a modifications teaming approach to enhance oversight of design changes, and the formation of a testing authority to oversee all design and surveillance testing.
9510300219 951020 PDR ADOCK 05000255 S PDR
NRC Form 366A U.S. NUCLEAR REGULATORY COMMISSION (9-B3) APPROVED OMB NO. 3150-0104 EXPIRES: B/31 /85 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (21 LER NUMBER (3) PAGE (41 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 5 0 0 0 2 5 5 9 5 _ 0 0 8 _ 0 1 0 2 OF 0 9 EVENT DESCRIPTION On July 28, 1995 with the plant in cold shutdown condition, all control rods inserted, and at refueling boron, it was discovered during design change testing that none of the four containment*
high pressure (CHP) channels of the reactor protection system (RPS),[JD], would initiate a reactor trip. The design change testing followed recently completed minor upgrades to the RPS trip unit connector blocks. An undesired connection across the CHP trip contacts was discovered in the printed circuit boards of the interconnection modules. The connection was introduced as a result of a separate 1992 modification which replaced all four channels of the RPS because of high maintenance and obsolescence of spare parts. This connection provided a circuit path that bypassed the CHP trip to the matrix logic of the RPS, thereby disabling the CHP trip circuit on all four independent safety channels.
The Palisades RPS design contains twelve available trip units in each of the four separate safety channels. The initial plant design contained ten active trip functions, with the two remaining
. unused trip functions bypassed to complete the system logic. Prior to initial criticality, a CHP trip function was added to the plant design, and the eleven active trips remained essentially
- untouched until a modification was completed in March of 1992. The 1992 modification included the replacement of all twelve trip units for the four channels of the RPS system because of high maintenance and obsolescence of spare parts. Facility Change FC-888, " Upgrade RPS Bistable Trip Units And Power Supplies," was designed by the original equipment supplier.,
Combustion Engineering (CE). The original wiring list showing two bypassed trip units was inadvertently implemented without adequate validation and verification as a design input ihto the planned upgrade. The bypassed configuration of the CHP trip units was not identified during post modification testing in the 1992 refueling outage nor during monthly TS surveillance testing due to inadequate test procedures.
Upon discovery of the disabled CHP trip functions, a multidisciplinary root cause analysis team was established. Insight on the use of Event and Causal Factor Analysis was provided by an individual from the Institute of Nuclear Power Operations (INPO) during the first week of the root cause evaluation. Concurrent with the Palisades evaluation, a root cause evaluation was initiated by CE to assess the adequacy of the quality controls and processes utilized in the CE Nuclear Services Engineering Department. The circuit boards were modified to re-enable the CHP trip function and an operability test was developed and completed for all trip functions on the four channels of the RPS.
CAUSES OF THIS EVENT The root cause evaluation team identified numerous inappropriate actions and program deficiencies that are categorized into three primary program areas with five identified primary root causes. The three primary program areas are;
NRC Form 366A U.S. NUCLEAR REGULATORY COMMISSION (9-83) APPROVED OMB NO. 3150-0104 EXPIRES: 8/31/85 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER 12) LER NUMBER (3) PAGE (4)
SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 5 0 0 0 2 5 5 9 5 - 0 0 8 - 0 0 3 OF 0 9 A. Inadequate design control program, B. Inadequate testing control program, C. Inadequate industry experience review program.
The five primary root causes are:
- 1. Inadequate program scope
- 2. Inadequate prioritization of work
- 3. Inadequate program monitoring or management
- 4. Inadequate attention to emerging problems
- 5. Lack of commitment to program implementation The significant findings with their associated root causes are categorized below within the program area associated with the problem:
A. INADEQUATE DESIGN CONTROL PROGRAM Root Cause #1 Inadequate Program Scope, (omission of necessary functions or guidance in procedures):
1*. Plant procedures did not alert users that vendor information was uncontrolled C!nd_ could be inaccurate. Vendor documents (manuals and drawings) were not always maintained up to date. This became an event precursor when the FC-888 project engineer removed the outdated wire list from the vendor file and used it as a design input without adequate validation and verification. The wiring list still had a circuit bypass for the CHP channel since the initial plant design did not include a CHP trip.
- 2. Plant procedures did not establish clear roles and responsibilities for the implementation of modifications. Besides the Prime Design Reviewer, no other personnel on site felt direct ownership for the successful implementation of the modification.
Root Cause #2 Inadequate Prioritization Of Work, (inappropriate application of resources due to a misunderstanding of task significance or complexity or availability of resources):
- 1. The plant Prime Design Reviewer (PDR) and the Combustion Engineering (CE) designer, who acted as the Responsible Engineer, were both inexperienced. The assignment of the inexperienced engineers to this modification was due to a lack of available experienced personnel and the mindset of management that the RPS modification was one of limited complexity and scope. It was believed that CE as a whole had the knowledge and skills to balance out this inexperience. The new RPS design was a clone of the early 1980's RPS System designs that were used for St. Lucie and Millstone
- nuclear plants.
NRC Form 366A U.S. NUCLEAR REGULATORY COMMISSION (9-831 APPROVED OMB NO. 3150-0104 EXPIRES: 8/31/85
- LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME (11 DOCKET NUMBER (21 LER NUMBER (31 PAGE (41 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 5 0 0 0 2 5 5 9 5 - 0 0 8 - 0 0 4 OF 0 9
- 2. There were very few modification team meetings conducted. All of the technical reviews were completed by CE. Palisades personnel only provided cursory reviews of portions of the design. At that time, there was no programmatic guidance to provide system engineer involvement or ownership in the modification planning and reviews.
Root Cause #3 Inadequate Program Monitoring or Management, (insufficient oversight and self assessment):
- 1. There was an over reliance on CE due to their being on the approved suppliers list and being the ori.ginal equipment manufacturer. CE's design reviews and testing were less than adequate and the plant oversight was not effective.
Root Cause #4 Inadequate Attention To Emerging Problems, (ineffective problem .identification and root cause analysis):
- 1. During modification preparations, CE was behind schedule and several problems were identified during design reviews, testing and installation. With the lack of experience and attention associated with the oversight of this modification, Palisades was unable to recognize the generic implications.
- 2. During the installation of the RPS modification in 1 992, two design problems. were identified. The problems could have been linked to the use of the same incorrect wiring list. If these two conditions had been sufficiently evaluated in 1 992 for generic
. implications, the precursor (inaccurate wiring list) could have been identified and the inadvertent CHP bypass could have been discovered at that time. Corrective action documents were not written for the two problems noted during the modification installation. The corrective action process initiated in 1994 is more intrusive and has resulted in Condition Reports (CRs) being generated and evaluated in a more timely manner for similar issues associated with modification problems.
B. INADEQUATE TESTING CONTROL PROGRAM Root Cause #1 Inadequate Program Scope, (omission of necessary functions or guidance in procedures):
Root Cause #5 Lack of Commitment to Program Implementation
- 1. The plant procedures and organization lacked a single point of contact for testing to ensure consistent application of proper testing techniques and overlap.
NRC Form 366A 0 U.S. NUCLEAR REGULATORY COMMISSION (9-83) APPROVEO OMB NO. 3150-0104 EXPIRES: 8/31/85 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAM~ (1) DOCKET NUMBER 12) LEA NUMBER (3) PAGE (4)
SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 5 0 0 0 2 5 5 9 5 - 0 0 8 - 0 0 5 OF 0 9
- 2. There was an inappropriate reliance on the existing TS Surveillance Test M0-3, "Reactor Protection Matrix Logic Tests," which provided a portion of the post modification verification. The planned approach for the post modification testing consisted of two separate tests. A test procedure written by CE tested the channel relays for proper operation. The second test was the normal monthly Palisades TS test M0-3. This test was used to verify functional operability of the channel relay output to the RPS matrix logic. The TS test M0-3 was used without verifying that it would functionally test all the matrix logic that needed to be tested after the modification.
A simplified block diagram of the RPS testing from FC-888 is shown as Attachment 1. It was determined that M0-3 was not fully meeting the intent of Tech Spec Table 4.17 .1 item 13, RPS matrix logic testing every 31 days. M0-3 did verify that the High Containment Pressure (CHP) relay contact was opening (single pole double throw ,,
contact) but it failed to determine if the contact was being bypassed. The TS surveillance procedure was revised to assure adequate testing overlap to verify the RPS matrix logic performance. This adverse condition pointed out the causal factor that the surveillance test program lacked direction in functional testing of Tech Spec equipment including assurance of adequate overlap of all functions being tested.
Root Cause #2 Inadequate Prioritization Of Work, (inappropriate application of resources due to a misunderstanding of task significance or complexity or availability of resources):
- 1. The existing roles and responsibilities of the system engineers allows other priorities to take precedence over a thorough review of existing TS Surveillance Procedures or new modification test procedures. This situation led to cursory reviews of the modification test procedures.
Root Cause #3 Inadequate Program Monitoring Or Management, (insufficient oversight and self assessment):
- 1. The CE Factory Acceptance Test (FAT) scope should have identified the undesired bypass for the CHP trip. However, there were problems with the test procedure being written incorrectly, identifying that CHP trip test lamps should be lit when they should have been off and, conversely, off when they should have been lit. Currently it is being assumed that the test rig for the FAT was wired incorrectly, which resulted in a failure to identify the CHP bypass. Plant oversight of the FAT procedure and performance was inadequate. The Palisades PDR and Test Engineer had other commitments and were not present to oversee the FAT. The System Engineer and two instrument and control technicians observed a portion of the FAT solely to familiarize themselves with the new system hardware. The FAT failure to locate the CHP bypass was a causal factor indicating that CE's test program was less than adequate.
{9-83)
LICENSEE EVENT REPORT (LERI TEXT CONTINUATION U.S. NUCLEAR REGULATORY COMMISSION APPROVED OMB NO. 3150-0104 EXPIRES: 8/31/85 FACILITY NAME {1 J DOCKET NUMBER 121 LER NUMBER {3) PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 -- 0 I0 I8 - 0 I1 0 I6 OF 0 I9 C. INADEQUATE INDUSTRY EXPERIENCE REVIEW PROGRAM Root Cause #3 Inadequate Program Monitoring Or Management, (insufficient oversight and self assessment):
- 1. The documented evaluations of several similar Industry Experience Events (INs) were reviewed and found to be inadequate. Four of the INs point out inadequate post modification testing, inadequate overlap of testing, and inadequate testing of safety related circuits. There was a tendency to provide only a cursory review of the implications of the "'
industry events with a general assumption that the present testing methods are inherently valid.
SAFETY SIGNIFICANCE The failure to trip the reactor on CHP impacts two sections of the Final Safety Analysis Report (FSAR). The two sections are section 14.14, "Steam Line Break Inside Containment," and section 14.18, "Containment Pressure and Temperature Response." The safety significance of not tripping the reactor on CHP would have been minor.
With regard to FSAR section .
- 14. 14, "Steam Line Break Inside. Containment," the present limiting.
case is a MSLB with loss of offsite power and a loss of a DIG. The calculation predicts 2% of the core fails due to penetrating departure from nucleate boiling (DNB). This is due to the return to a power level of - 13% at 240 seconds into the event. The analysis did not take. credit for a
- reactor trip on CHP in the limiting case. Assuming the reactor did not trip on CHP at 2.5 seconds after the event, the next trip that would occur would be the variable high power (VHP) trip at no later than 6 seconds after the event.
The FSAR analysis for fuel failure during a MSLB remains bounding even if the reactor trip is delayed due to the loss of the CHP trip function. In the short term, a postulated MSLB would produce an overpower transient which would terminate in a reactor trip on VHP. This initial power ramp would not result in a more severe challenge to the specified acceptable fuel design limits than the existing FSAR analysis results for a postulated fast withdrawal of a control rod bank. In the long term, the effect of delaying the reactor trip by - 3.5 seconds would result in a reduction in the cooling of the affected primary loop. The reduction in cooling would be equal to the amount of power produced in the 3.5 second delay period. Operation at power for several seconds after a MSLB will result in a modest increase in both the secondary side pressure and the break flow rate during the initial portion of the steam generator blowdown. These increases would partially offset the additional heat added by the reactor core. The net effect would be to decrease the cooldown, to reduce the return to power, and to make the transient less severe and less of a challenge to DNB or fuel failure. There would be no significant effect on plant operators' responses to the event.
NRC Form 366A (9-831
- U.S. NUCLEAR REGULATORY COMMISSION APPROVED OMB NO. 3150-0104 EXPIRES: 8/31/85 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE (41 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 - 0 I0 I8 -- 0 I1 0 I7 OF 0 I9 The second FSAR section impacted is 14.18, "Containment Pressure and Temperature Response." The mass and energy release rates for both the Loss of Coolant Accident (LOCA) and a MSLB are used to predict the containment pressure and temperature response. The LOCA analysis did not take credit for the CHP trip function. As discussed in the previous paragraphs, the MSLB analysis did take credit for the CHP trip function. The impact from the delayed reactor trip would have been additional energy addition from the primary system to the secondary system during the delay period of - 3.5 seconds. The additional energy addition results in a minor increase in the predicted containment pressure and temperature response following the MSLB.
The increases in the predicted pressure and temperature are 0.6 psi and 1.4 degrees fahrenheit, respectively. No design limits would have been exceeded with a disabled CHP trip and a postulated MSLB.
CORRECTIVE ACTIONS TAKEN AND RESULTS ACHIEVED The following corrective actions were completed and are described in detail in the associated
- plant condition report C-PAL-95-1117:
- 1. The RPS matrix channels have been modified to restore the CHP trip function.
- 2. The Technical Specifications Surveillance Test (M0-3) for the RPS matrix logic was revised
- to provide adequate overlap in testing. This will assure that the requirements of Technical Specification Table 4.17 .1 item 13 are being adequately verified during the monthly test.
- 3. A review of 24 Technical Specifications Surveillance Procedures, post modification testing completed during the 1995 REFOUT, and Special Tests completed during the 1995 REFOUT was completed. This review was performed to verify that procedures used for testing safety related circuits completely test design functions of the equipment. As part of this review, condition reports were written and procedures were revised. A number of procedure enhancements were recommended to the procedure sponsors. None of these deficiencies resulted in any of the equipment being declared inoperable.
- 4. An evaluation was completed for all of the Functional Equivalent Substitution (FES) and Specification Change (SC) packages installed in 1995 REFOUT. If the FES or SC changed the logic matrix of either the RPS or Engineering Safeguards initiation, the acceptance test was reviewed to verify that the logic matrix functioned properly. This evaluation resulted in two SC's and two FES's being reviewed.
The testing which verified each project was found acceptable. There was a concern about the testing for SC-95-033 because the testing was not comprehensive in that the test only verified proper operation of the modified contact and did not verify that the remainder of the logic had not been disturbed. Additional surveillance testing fully verified the adequacy of the entire circuit.
NRC Form 366A (9-83)
- LICENSEE EVENT REPORT (LERI TEXT CONTINUATION U.S. NUCLEAR REGULATORY COMMISSION APPROVED OMB NO. 3150-0104 EXPIRES: 8/31 /85 FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (3) PAGE(4)
SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 - 0 I0 I 8 - 0 I1 0 I8 OF 0 I9
- 5. An independent evaluation of the testing adequacy for FES-95-032, "RPS Trip Unit Connector Block Replacement" was completed. The evaluation consisted of a review of trip module inputs and outputs affected by the modification to determine the extent to which all functions were ct}ecked as part of the post modification test. The scope of the work affected more than 2500 pins in the connector blocks. The results of the assessment indicate that functions using 2% of the pins were not verified and 9% of the pins were not thoroughly tested. The evaluation determined that the post modification testing was less than adequate. Operability of the RPS was subsequently verified by a new test.
The test plan was developed to declare the RPS operable. The plan objective was to assure full operability of the RPS through the use of overlap and/or full functional testing. The strategy assured that every RPS input and output functioned properly through functional tests and finally, assured total operability of the RPS safety and non-safety related functions. This evaluation found six data logger outputs not functioning. A condition report (C-PAL-95-1284) and work orders were initiated to adequately resolve these problems. It was also determined that time response testing of the RPS had not been performed as part of the Thermal Margin Monitor (TMM) modification performed in 1 992.
Calculations were performed to verify adequate safety margins. RPS time response testing is currently scheduled for the 1996 REFOUT.
The testing plan was reviewed by ABB-CE. The review was formalized and documented testing recommendations and enhancements to assure full RPS operability.
Corrective Actions to A void Recurrence
- 1. Provide enhancements to the Design Control Program by implementing the following -
changes:
- a. Revise Administrative Procedure (AP) 9.44, "Design Document Control," AP 9.45, "Vendor Manual Control," and AP 10.44, "Engineering Records Center Distribution and Control of Design Documents," to indicate vendor manuals and vendor drawings have not been maintained as living documents. Also, add a requirement to AP 10.44 to attach a notice of that condition to any second generation print issued by ERC which contains information obtained from a vendor document, manual or drawing.
- b. Revise Project Management Construction and Testing procedure 1-3, "Project Team Organization and Responsibilities," to better describe roles, responsibilities, and requirements for team meetings.*
I
NRC Form 366A (9-831 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION U.S. NUCLEAR REGULATORY COMMISSION APPROVED OMB ND. 3150-0104 EXPIRES: B/31 /85 FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER (31 PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant o Is IoIoIoI 2 I s I s s Is - o lo I a - o I, o Is OF o Is
- c. Revise AP 9.03, "Facility Change," to add the requirement to initiate condition reports when potentially generic, or significant issues are discovered during Facility Change Project work, and eliminate the option of using an inexperienced Prime Design Reviewer.
- d. Revise AP 9.04, "Specification Changes," to add the requirement to initiate condition reports when potentially generic, or significant issues are discovered during Specification Change Project work .
. e. Provide quarterly continuing .training to Engineering Support personnel using a case study methodology to include as a minimum; testing program improvements, management expectations for testing, and consultant/vendor scrutiny, and Industry Experience Report evaluations.
- 2. Provide enhancements to the Test Control Program by implementing the following *changes:
- a. Assign responsibility for review of all modification testing to Systems Engineering who will act as the testing authority. Surveillance tests assigned to Systems Engineering shall meet the standards established by the testing authority.
- b. Revise applicable Administrative Procedures to emphasize functional testing requirements and accountability for test adequacy/completeness. Ensure procedures include the expectation of 100% functional testing of anything changed or affected directly or indirectly by the work done for a project. Include a description of overlapping when relying on multiple tests to meet requirements.
- 3. Provide enhancements to the Industry Experience Program by implementing the following changes:
- a. Re-evaluate the documented responses to similar Industry Experience Reports {IN) pertaining to inadequate circuit modifications and testing, {IN-88-83, IN-92-65, and IN-93-38).
- b. Develop a plan to implement reviews to comply with NRC intended actions for the new
{currently in draft) Generic Letter No. 95-XX: "Testing of Safety-Related Logic Circuits"
{IN 95-15) published in the Federal Register May 22, 1995, page 27143.
- c. Implement a second level critical review of industry experience evaluations by Systems Engineering. Re-emphasize the importance of management expectations for industry experience reviews.
- ATIACHMENT 1 CONSUMERS POWER COMPANY*
PALISADES PLANT DOCKET 50-255 BLOCK DIAGRAM OF RPS MODIFICATION TESTING 1 Page
BLOCK DIAGRAM OE RPS MODIFICATION TESTING Eleven Different Reactor Trip Indicating Lights fed Functional Inputs from one of the contacts of the trip unit ,
relays. These do not
-- function to trip the reactor, but they can be used to verify that the ladder logic relays actuated.
Trip Unit Matrix Trip Indicating Ughts. Can be Bistables and Output Relays used to show the ladder /ogle for
. Auxiliary Trip reactor trip has been actuated.
Units Ladder logic matrix contacts.* These
-- actually trip the reactor
~
Clutch Power Supplies (Trip the Reactor) via the clutch power supplies.
PostModificati~o~n~T~e=s~t~~~~~~~~~~~~~~~~~~~~_,
Post Mod Test only tested to this point, It did not test the ladder logic matrix contacts.
Factory Acceptance Te$t ------
Factory Acceptance Test, as written, should have caught problem, but did not * * * * *
.. ~-*~
Surveillance Test --------.~ .. . ....
Surveillance Test goes far enough In the logic, but only covers one out of eleven functions * * * * *
*** *-*****-*********- ***--------------------~