IR 05000255/1995012

From kanterella
Jump to navigation Jump to search
Enforcement Conference Rept 50-255/95-12 on 950801-04.No Violations Noted.Major Areas Discussed:Apparent Violations, Root Causes,Contributing Factors & Licensee Corrective Actions
ML18065A089
Person / Time
Site: Palisades Entergy icon.png
Issue date: 09/13/1995
From: Gardner R
NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:
Shared Package
ML18065A088 List:
References
50-255-95-12-EC, EA-95-169, NUDOCS 9509210300
Download: ML18065A089 (37)
v • d • e


Text

I

.*

U.S. NUCLEAR REGULATORY COMMISSION REGION II I REPORT NO. 50-255/95012 CDRSl EA NO.95-169 FACILITY Palisades Nuclear Generating Plant License No. DPR-20 LICENSEE Palisades Nuclear Generating Plant 27780 Blue Star Memorial Highway Covert, MI 49043-9530 MEETING Predecisional Enforcement Conference September 7, 1995 Ronald N~ Gardner, Chief Plant Systems Section Region Ill Office 801 Warrenville Road Lisle, 11 60532-4351 DATES OF ORIGINAL INSPECTION August 1 through 4, 1995 INSPECTORS D. Butler, Reactor Inspector R. Winter, Reactor Inspector APPROVED BY MEET ING SUMMARY Date Apparent violations identified during the inspection were discussed, along with the corrective actions taken or planned by the liceDsee. The apparent violations involved a failure to (1) use an updated, controlled wiring list as a design input for Facility Change (FC) No. 888; (2) perform an appropriate post-modification test for FC-888; and (3) demonstrate the operability of the reactor protection system (RPS) containment high pressure trip function in accordance with Technical Specification 3.17.1 from April 1992 to May 199 '*

PDR ADOCK 05000255 G

PDR

  • INSPECTION DETAILS Persons Present at Conference R. Fenech, Vice President, Nuclear Operations T. Palmisano, Plant General Manager K. Powers, General Manager, Nuclear Services D. Smedley, Manager, Licensing D. Fadel, Manager, Systems Engineering G. Szczotka, Manager, Nuclear Assessments R. Swanson, Manager, Design Engineering R.. Gerling, Manager, Nuclear Fuels R. Vincent, Licensing Administrator M. Genrick, Operations Technical Superintendent T. Duffy, Safety Analysis Supervisor G. Aikire, Control Room Supervisor M. Nordin, Design Engineering L. Seamans, Engineering R. Westerhof, I&C Engineering B. Meredith, Systems Engineering D. Bemis, Systems Engineering C. Mathews, Plant Support Engineering F. Ruell, Nuclear Control Operator R. Bradshaw, Manager, I&C, ABB-Combustion Engineering D. Menard, RPS Engineer, ABB-Combustion Engineering U. S. Nuclear Regulatory Commission H. Miller, Regional Administrator, RIII G. Grant, Director, Division of Reactor Safety (DRS), Rill J. Lieberman, Director, Office of Enforcement, OE J. Grobe, Acting Deputy Director, DRS, RIII B. Burgess, Director, Enforcement Investigation Coordination Staff~ Rill B. Berson, Regional Counsel, RIII R. Gardner, Chief, Plant Systems Section, RIII M. Dapas, Acting Section Chief, Division of Reactor Projects (DRP), Rill M. Parker, Senior Resident Inspector, Palisades, RIII P. Pelke, Enforcement Specialist, Rill D. Butler, Reactor Inspector, Riii E. Cobey, Reactor Inspector, Riii*

J. Lennartz, Reactor Engineer, Rill J. Hannon, Director, Project Directorate 111-1, NRR, participated via telephone

  • Predecisional Enforcement Conference A Predecisional Enforcement Conference was ~eld in the NRC Region III Office on September 7, 199 Three apparent violations of NRC regulations were discusse The inspection findings are documented in Inspection Report No. 50-255/95010(DRS) which was transmitted to the licensee by letter dated August 23, 1995.
  • The purpose of this conference was to discuss the apparent violations, root causes, contributing factors, and the licensee's corrective action The licensee's presentation included an acknowledgement of the apparent violations which had occurred, a discussion of the event's safety significance, a discussion of the circumstances which caused the event, and an outline of corrective actions taken or planne The NRC representatives questioned the licensee to clarify the extent of the licensee's investigation and corrective action A copy of the licensee's handouts used during the presentation are attached to this repor Attachments:

As Stated

NUCLEAR PLANT

  • .
....
.**-:*.::::******:-*..

Bypassed Containment High Pressure (CHP) Trips To The Reactor Protection System (RPS)

September 7, 1995

'.

September 7, 1995

~/t.121*s~'l:l:i§*****~*~~mee'l$li}*~~*1****§*~*~j*~~§NIEN*~::~:~~*~*~*llE'~*C*E*'

Bypassed CHP Inputs To The RPS AGENDA CPCo INTRODUCTION RA FENECH

. EVENT DESCRIPTION................... TJ PALMISANO INITIAL MANAGEMENT RESPONSE......... TJ PALMISANO RPS PROJECT TEAM.................... GB SZCZOTKA

  • RPS INVESTIGATION/REPAIR/OPERABILITY TESTING......................... RS WESTERHOF
  • GENERIC EVALUATION OF TEST PROCEDURE ADEQUACY................... MT NORDIN
  • SAFETY SIGNIFICANCE OF DELAYED TRIP........ RJ GERLING *
  • ROOT CAUSES AND CORRECTIVE ACTIONS..... GB SZCZOTKA MITIGATING FACTORS................... RW SMEDLEY MANAGEMENT LESSONS LEARNED......... TJ PALMISANO CLOSING REMARKS............. *......... RA FENECH

I

.

September 7, 1995

.

e~l4IS1Al:liS*-***P~l;Q_J:*c*1slC)*~~*~****i*-i~*~~eMiN,T*.**~mNFEREf;A*~e*********************

Bypassed CHP Inputs To The RPS EVENT DESCRIPTION:

During Post Modification Testing of the Reactor Protection System (RPS) on July 28, 1995, it was discovered that none of the containment high pressure (CHP) channels would initiate a reactor tri *

Further investigation revealed that this condition resulted from an inadequate design change which was installed in April of 199 *

Plant operated two cycles in this conditio *

At the time of discovery Palisades was in a refueling outage with all control rods fully inserted and the primary coolant system at refueling boron concentration. RPS operability was not required for this plant mod Page 1

'.

September 7, 1995 P:i.(.lilSA.Qt;*~***:*e*~~DECISJ~*~1*1:***i*N*~~*ll§t,,1EN*"t****g'w*e*ij*~EN¢*1:** ************

Bypassed CHP Inputs To The RPS INITIAL PLANT MANAGEMENT RESPONSE: (4 to -72 hours) Restriction Imposed to Maintain Refueling Boron Concentration Quarantined RPS Pending Development of Investigation Plan Established RPS Response Team

RPS Recovery (Investigation/Repair)

RPS Operability Testing

Generic Test Procedure Adequacy

Multi-Disciplinary Root Cause Analysis

Safety Significance

Licensing/NRC Interface Notified NRC Region Ill and NRR Personnel

  • *

4 Hour Report

Information Packet

Conference Call with Region Ill and NRR Conducted Special PRC Mobilized ABB-CENO Resources

Assist with RPS Repair Plan Development

Perform RPS Repairs

Assist with Comprehensive RPS Test Plan Development Arranged INPO Assistance to Critique Root Cause Analysis Effort Page2

'

Bypassed CHP Inputs To The RPS RPS PROJECT TEAM:

I if RPS RE COVE RY NER KA TO (a.. n ote 1)

  • lnvHtigatlon Plan
  • Apparent Cause
  • Repair Plan
  • Poat Maintan1nca Functional Cheek

. (1) CE A11istanee (2) CE/Siemens A11istance (3) INPO A11istanee

~

RPS OPERABILITY TESTING RS w_ESTERHOF (SH note 1)

  • Operability THI Plan
  • RPS lnpuVOutpu Analy1i1
  • Overtap Analyala
  • RPS MOcl Revl-PROJECT TEAM PLANNING LEADER

,...,

SUPPORT

~

GB SZCZOTKA JE LANKES BACKSHIFT MA FERENS I

~

~

~

SAFETY ROOT CAUSE GENERIC TEST SIGNIFICANCE ANALYSIS ADEQUACY TC DUFFY LO SEAMANS MT NORDIN (SH not* 2)

(SH note 3)

  • PMT/Surveillancel
  • Safety Analysis Impact
  • EvenVCausal Factor Spacial THI Procedure Chart Review
    • Root Cau1e1
  • FES/SC Revi-s
  • Corrective Actions
  • NPAO Independent Review ol FES*95-032 Page 3 September 7, 1995 i "

LICEN SING RA VIN CENT

  • NRC liaison

I

.

September 7, 1995 P.~t:l$ADE$*****-~*R~J:>ee*1s1*EJ.*f.1*~*~****E*N*~i*a~*~*IVl*eNil*ttl8F:e*Re*N*c*&*************:****

Bypassed CHP Inputs To The RPS RPS DESCRIPTION*: (Reference Figure 1) RPS Provides Reactor Trips in Response to 11 Input Signals SIG Low Level (2)

S/G Low Pressure (2)

High Reactor Power Thermal Margin/Low Pressure High PZR Pressure RPS Employs a 2 out of 4 Trip Logic Low PCS Flow Loss of Load High Startup Rate Containment High Pressure A Trip Unit Takes a Signal Input and Outputs a Trip Signal in the Form of Open Relay Contacts These Relay Contacts (eg, A1-1, 81-1) Form the Legs of the Logic Ladder (Six Total) Two Parallel Contact Openings in a Logic Ladder will Cause the Ladder's Matrix Relays (eg, AB1, AB2) to Drop Out Opening a Contact on a Matrix Relay De-energizes the Power Supplies for the Control Rod Drive Clutches De-energizing all the Clutches Allows all the Rods to Drop into the Core by Gravity (ie, Reactor Trip) RPS Initiation Logic Test

Simulates tripping of two channels of same parameter and assures trip unit relays actuate

Overlap with RPS Actuation Logic Test Page4

... '

September 7, 1995 Bypassed CHP Inputs To The RPS TECHNICAL DISCUSSION: (Detection/Investigation/Repair)

CHP BYPASS DEFICIENCY

Printed circuit board jumpe *

95 REFOUT trip unit connector block replacement project testin *

With CHP selected, RPS actuation logic test failed (could not de-energize clutch power supplie *

CE supplied "Black Box".

CHP REPAIRS

Compared notes with ABB-C *

ABB-CE (original manufacturer) repaire *

Verified repairs by comparing A-B to C-D matri DEFICIENCIES:

1992 PMT inadequat *

92-95 surveillance test inadequat *

1995 connector block testing inadequate.

Pages

I

.

  • .,

September 7, 1995

- *... i:rAuSAD~i?leoECISf<lll~Q'llii~eMEN!l\\:lC'~~ERENCE.\\*

Bypassed CHP Inputs To The RPS

1995 REFOUT RPS.OPERABILITY TEST PLAN:

. TEST PHILOSOPHY:

  • Assure full operability of the RPS through the use of overlap and/or functional testing PHILOSOPHY ASSURED BY:
  • All wires touched by CHP repairs and connector block modification were teste * Satisfactory testing of all safety and non-safety function * Existing testing review to verify proper overlap. and technical specification requirements me * Reviewing all past RPS modifications and envelope any deficiencie * Developing test matrix for RPS operability (Figure 3 and 4)

TESTING COMPLETION:

  • All testing performed satisfactorily, except datalogger outputs from 6 of 44 trip units.

Pages

j

. ' '

..

Bypassed CHP Inputs To The RPS GENERIC TEST PROCEDURE ADEQUACY:

'

REVIEW OF TEST PROCEDURES OBJECTIVE September 7, 1995 Verify that procedures used for testing safety related circuits completely test the design functions of the circuit SCOPE Procedures which verified the logic of safety-related circuits to accomplish their design functions were reviewe These included Technical Specification Surveillance Procedures (excluding M0-3), 95 REFOUT Special Test Procedures, Permanent Maintenance Procedures, and 95 REFOUT Post Modification Test Procedure METHODOLOGY

Review the purpose and acceptance criteria pf.the procedur *

Compare purpose/acceptance criteria to the design functions stated in the Design Basis Documen * *

Verify complete test of function from sensing parameter to completion of expected action. If a logic diagram is available, indicate on the diagram the portions of the circuit that are tested. If a logic diagram is not available, use a block diagram to show the portions of the circuit that are teste *

Look for steps that give the performer option Are the options sufficiently restricted such that all available paths are tested? Is it possible that a logic path could be left untested?

Document results of revie CONCLUSIONS

24 test procedures were reviewed (see Figure 5).

  • No testing deficiencies were found that caused equipment to be declared inoperable or revealed a lack of verification of a safety related design function.

3 condition reports were written, 2 procedures were revised prior to use and enhancements were recommended for 7 procedure Page 7

't

September 7, 1995 Bypassed CHP Inputs To The RPS GENERIC TEST PROCEDURE.ADEQUACY:

REVIEW OF FUNCTIONALLY EQUIVALENT SUBSTITUTIONS AND SPECIFICATION CHANGES

All Functionally Equivalent Substitutions (FES) and Specification Changes (SC) installed during the 95 REFOUT were reviewed to identify those that could impact the logic matrix of either the RPS or Engineered Safeguards initiatio FESs and 2 SCs met the above criteria and were reviewed using the previously discussed methodolog One *problem was noted with testing following SC-95-033, "Rewire SIS Contact As Alternative for Replacement of Non EQ Cable." * The entire logic scheme was not tested as result of the Specification Change, but was covered shortly thereafter via additional surveillance testin *

FES 95~032 (RPS Connector Block Replacement) was independently reviewed by the Nuclear Performance Assessment Departmen Out of 2596 pins, 56 were not tested. Not all outputs to the events recorder were verifie The trip matrix logic was not 100% teste auxiliary functions were not teste OVERALL CONCLUSION

There is adequate assurance that critical design characteristics and system functions are being tested.

Page B

I..

September 7, 1995 Bypassed CHP Inputs To The RPS SAFETY SIGNIFICANCE:

EVENTS AFFECTED FSAR 14.14 - Steam Line Break Inside Containment

The present limiting case is an MSLB with loss of offsite power and a loss of a DIG. The calculation predicts 2% of the core fails due to penetrating DNB. This. is due tc the return to power of -13% at 240 seconds. The analysis did take credit for a reactor trip on CHP in the limiting case. The trip was credited 2.55 seconds into the event. Assuming the reactor did not*

trip on CHP, the next trip that would occur would be the Variable High Power (VHP) trip at -6 seconds*.

There would be no significant effect on operator *

The FSAR analysis remains bounding even if reactor trip on CHP does not occur. In the short term, the failure of. the reactor to trip on CHP would produce an overpower transient which would terminate in a reactor trip on VHP. This initial power ramp would not result in a more severe challenge to the Specified Acceptable Fuel Design Limits (SAFDLs) than the* FSAR analysis for the fast withdrawal of a control rod bank. In the long term, the effect of delaying the reactor trip by = 3.5 seconds would be to reduce the cooling of the primary loop by the amount of power produced in the seconds. Operation at power for several seconds after a steam line break will result in a modest increase in the secondary pressure and in break flow rates during the initial portion of the steam generator blowdow This increase would partially offset the additional heat added by the reactor cor The net effect would be to decrease the cooldown, to reduce the return to power, and to make the transient less severe and less of a challenge to DNB or fuel mel * *

Therefore, the present analysis prediction of 2% fuel failure is bounding for the MSLB inside containment with the reactor trip from CHP disabled.

Page 9

I.

~

I September 7, 1995

.

  • .*... PA[(si(oes P.REDECISISNl121MellGEMENsGGiNfERENCE**.*** /.

..

..*.

~...

.... *...

.....

. *:*.

.

..

........ *.... *.. *.*.*....... *.... ;.. *... *.,*....... *.*.*. *.*. *. *... *... *...... *, *.*.. *.. *. *... *.*,. *. *. *....... :*

.. *..

.. *. *... *...... *. *...;. :.*..... *,*....*..........

Bypassed CHP Inputs To The RPS SAFETY SIGNIFICANCE:

FSAR 14.18 - Containment Pressure & Temperature Response

The mass and energy release rate calculations for both the LOCA and MSLB are used to predict the containment pressure and temperature respons The LOCA did not rely on CHP to trip the reactor. The MSLB did take credit for the CHP signal tripping the reacto *

The impact on the MSLB containment calculation will be some additional energy addition from the primary to the secondary between the time the reactor was tripped on CHP and the next trip. As discussed above, that trip is the VHP trip and.it would occur within 6 seconds from event initiatio The extra energy addition results in an increase to predicted containment.

pressure and temperature response following a MSLB, but the design pressure and temperature limits are not exceede Overall Conclusion The safety significance of not tripping the reactor on CHP would have been minor. The MSLB fuel analysis would actually have shown slightly improved results. The MSLB containment analysis would have resulted in slightly higher pressures and temperatures, but no design limits would have been exceede ;

Page 10

..

.-

September 7, 1995 el==**-=e=.1>..==12==1s=_-~==[)==s===s=i P==8==E==*o=e==$==1s=1a===N==:11==: '==:Hs==]"==: 1r:==:<D==!~==i==&==***r.==E:=N==m==c==a==N==i==e==F.l==e==N.==c=E=-*

  • * -

Bypassed CHP Inputs To The RPS ROOT CAUSE ANALYSIS: Root Cause Analysis Team

Plant Engineering* and Modifications Manager - Sponsor

Engineering Root Cause Analysis Expert

  • Maintenance Root Cause Analysis Expert

Training Representative

INPO Representative - Oversight

  • Summary Event/Causal Factor Chart
  • *

Event Sequence

Contributing Factors

Broken Barriers

. Issues/Causes/Corrective Actions

Inadequate Design Control Program

Inadequate Test Control Program

Inadequate Industry Experience Review Program

Page 11

  • ***<Y5 \\.:\\>**/..* RALlSADE.S*RB.EQEGISlQNALENRQRGEMENJf: CONFERENCE:*.*

.*.*.**.*... **.* *.*.*.. *.*.*.*.*..,-...;-:-:::-:-:.;-*.. *

Inaccurate Information Used to Develop f----)

FC-888 1990-1991

  • CHP Trip ln1talled In 1969 - Wire List Not Updated
  • Vendor Documents Not Maintained aa "Living*

Documents Bypassed CHP Inputs To The RPS SUMMARY LEVEL EVENT AND CAUSAL FACTOR CHART Design Input to FC-888 Not Adequately Verified 1990-1992

  • Inadequate Site Tech Review*
  • Poor Contractor Performance
  • Inadequate Management Oversight
  • Mlased Opportunities (No CRs Written)

- Ground Fault

- CHP Trip Alarm

~,.

CE Factory Post Surveillance Acceptance Modification Testing Does Test Did Not Testing Does Not Detect Detect Problem -) Not Detect

-*)

Problem

---)

Problem 3/92 3/92

  • CE Teat Program Less
  • Inadequate Post Then Adequate Modification Teat Guidance
  • Inadequate Management Oversight
  • Inappropriate Reliance on Surveillance Teat For Modification*
  • Post Modification Teat Review Not Performed (31 Testa)

4/92-Present

  • Surveillance Test Guidance Lacking
  • Inadequate Review of Industry Operating Experience B B Q tJ B B tJ B CONFIGURATION DESIGN CORRECTIVE FACTORY POST SURVEILLANCE INDUSTRY CONTROL C.ONTROL ACTION TEST MODIFICATION TESTING EXPERIENCE PROGRAM TESTING REVIEW Page 12 RPS Connector Post Mod Testing Detects CHP Jumper 7/95
  • lnadequat.e Post Modification Test Guidance
  • Inappropriate Reliance on Surveillance Teat*

For Modifications B

POST MODIFICATION TESTING r 7, 1995*

Bypassed CHP Inputs To The RPS DESIGN CONTROL DEFICIENCIES ROOT CAUSE #1:

Inadequate Program Scope (Omission of Necessary Functions/Guidance in Procedures)

CAUSAL FACTORS/ INAPPROPRIATE ACTIONS CORRECTIVE ACTIONS

  • Outdated wiring list used - Design control procedures did not alert users that vendor information was uncontrolle * Inadequate plant ownership for modifications - Procedure did not establish clear roles and responsibilities for the implementation of modification * Administrative procedures will be revised to alert users that vendor manuals and vendor drawings have not been maintained as living documents. Notices will be attached to all second generation vendor prints issued by the ER * The clarity of engineering roles and respon*sibilities for modifications has improved through prior procedure changes resulting from the implementation of the P2E * Procedural revisions will be made to describe how the design team will specify and verify that design requirements are satisfie Page 13

Septe

  • .***********************************************************************************e,.,,~A*a.§§*.**.eR.~*'1§,:1*§lQ.NA.L.. ***~*NEQBC.eM*l;.. ~*1**.QQ.N.FJ;.Rl;NC.E..

Bypassed CHP Inputs To Th~ RPS DESIGN CONTROL DEFICIENCIES ROOT CAUSE #2:

Inadequate Prioritization of Work (Inappropriate Application of Resources Due to a Misunderstanding of Task Significance or Complexity or Availability of Resources)

CAUSAL FACTORS/ INAPPROPRIATE ACTIONS CORRECTIVE ACTIONS

  • An inexperienced plant prime design reviewer was used due to the following management mindsets:

-

That the RPS mod was of limited scope

-

That CE was the Original Equipment Manufacturer (OEM) with the necessary knowledge and skills

-

It was necessary to get by with available resources

  • Few modification team meetings were conducted and only cursory design reviews were performed by plant personne * Training and.Qualification requirements have been established for prime design reviewer * Management sensitivity regarding the proper application of experienced resources for complex jobs has improved (eg, use of multi-disciplinary design reviews for facility changes and speci~cation changes).
  • The facility change procedure will be revised to eliminate the option of using an inexperienced prime design reviewe Page 14 *

..... *.**************************************************;****************************************1*:***************el*t,1§.~*g*~*&***t?8§*m*1;*g*1*§*1QNl,.k***.l.;.N*i*~.RQ.~.M*J;*N::t.**QQ~FJ;REN.CE.

.

. *.*._.*;

Bypassed CHP Inputs To The RPS DESIGN CONTROL DEFICIENCIES ROOT CAUSE #3:

Inadequate Program Monitoring and Management (Insufficient Oversight and Self Assessment)

CAUSAL FACTORS/ INAPPROPRIATE ACTIONS

  • Plant oversight was not effective in detecting the inadequacies in CE's design and testing activities. There was an over reliance on CE due to being the OEM and being on the Approved Suppliers Lis CORRECTIVE ACTIONS
  • Continuing training will be provided to engineering support personnel regarding management expectations for consultant/vendor scrutiny and input to the Approved Suppliers Lis ROOT CAUSE #4:

Inadequate Attention to Emerging Problems (Ineffective Problem Identification/Root Cause Analysis)

  • CAUSAL FACTORS I INAPPROPRIATE ACTIONS.
  • Evaluations of the precursors to this event were less than adequate - Numerous problems were encountered during the design, installation and testing of this modification which if documented and evaluated properly could have prevented this conditio CORRECTIVE ACTIONS
  • The requirements for documenting design problems on Condition Reports will be strengthened through revisions to the Facility Change and Specification Change Procedure Page 15

....*. :.. \\ *.. \\::::::;:1::***************************::*****************.. ********eA*~*tsA.~.es*****e*B§P*ta~*l.s.t.~N'L.***EN.~*(!JB.c.e.Me*Mm**.. ~.. f:=ERefiCE*...

Bypassed CHP Inputs To The RPS TEST CONTROL DEFICIENCIES ROOT CAUSE #1:

Inadequate Program Scope (Omission of Necessary Functions/Guidance in Procedures)

CAUSAL FACTORS/ INAPPROPRIATE ACTIONS

  • Procedures provided inadequate guidance for conducting surveillance and post modification testin a. There are no definitions for the various kinds of tests to be performed, eg

-

End-to-end testing

-

Overlap

-

Device

-

Vertical

-

Horizontal b. There is no formal PMT philosophy on the extent of

. horizontal and vertical testing to be applied for various modification * There is an inappropriate reliance being placed on surveillance testing for modification * Collectively, the above items led to the inconsistent application of test requirement CORRECTIVE ACTIONS

  • The administrative procedures for Design Control, Temporary Modification Control and Surveillance Test Program Control will be revised to emphasize functional testing requirements. The procedures will include:

a. The expectation of 100% functional testing of anything affected by a design chang b. A description of test overlap when relying on multiple tests to meet requirement Page 16 *

................ ************************* **1*********:*.********.:*******************i**:i.*i***i*:*:***i1*u:1s~*a.~s*****~*.a§*i*i~*l§.llN"A.~****.§*N.fiQ.R.g.§M.E;*N.ro*****~*~*N.f l;*RE:NCI:

Bypassed CHP Inputs To The RPS TEST CONTROL DEFICIENCIES ROOT CAUSE #2:

Lack of Commitment to Program Implementation CAUSAL FACTORS / INAPPROPRIATE ACTIONS

  • There was no single point accountability for test control and there were no dedicated testing "experts" associated with the Modification and Surveillance Test Program CORRECTIVE ACTIONS
  • The responsibility for the review of all modification testing (FC, SC, FES, TM) will be assigned to System Engineering who will act as the testing authorit Surveillance tests within System Engineering shall meet the standards established by the test authorit ROOT CAUSE #3:

Inadequate Prioritization of Work (Inappropriate Application of Resources Due to a

  • Misunderstanding of Task Significance or Complexity or Availability of Resources)

CAUSAL FACTORS / INAPPROPRIATE ACTIONS

  • There was a lack of ownership for the FC-888 modification by the Test Engineer and System Engineer due to over-reliance* on CE as the OE * For System Engineers, many other problems take precedence over Surveillance Test Procedure review * Both of the above situations have led to cursory rather than detailed reviews of Post Modification and Surveillance Test procedure CORRECTIVE ACTIONS
  • The above noted procedure and organization changes will address the issue of accountability and application of resources to assure Post Modification and Surveillance Test adequacy. (See Root Cause #1 and #2)
  • Continuing Training will be provided to engineering support personnel on the test program improvements and managements expectations for testin Page 17
  • Bypassed CHP Inputs To The RPS TEST CONTROL DEFICIENCIES Septe 7, 1995*

ROOT CAUSE #4:

Inadequate Program Monitoring and Management (Insufficient Oversight and Self Assessment)

CAUSAL FACTORS/ INAPPROPRIATE ACTIONS CORRECTIVE ACTIONS

  • There was inadequate monitoring of CE's test activities for
  • As mentioned previously, Continuing Training will be FC-888 (1992 RPS mod) by CPCo personne provided to engineering support personnel regarding management's expectations for consultant/vendor scrutin *

Page 18 *

7, 199 Bypassed CHP Inputs To The RPS INDUSTRY EXPERIENCE PROGRAM DEFICIENCIES ROOT CAUSE #1:

Inadequate Program Monitoring and Management (Insufficient Management Oversight and Self Assessment)

CAUSAL FACTORS I INAPPROPRIATE ACTIONS CORRECTIVE ACTIONS

  • The evaluations and responses to the following Information Notices were inadequate:

IN-88-83 IN-92-65 IN-93-38 IN-95-15

  • The review of industry operating expe.rience has not been thorough due to:

a. A predisposition to use earlier work without further evaluation. (eg, 1986 SFE evaluation)

b. A mindset that the purpose of the reviews is to make things go awa c. Cursory rather than detailed review d. Oversight/Assessment of responses was less than adequat e. Ownership by evaluators was less than adequat Responsibilities as defined for evaluators are not clea Focus is presently is on schedule, not qualit * A second level critical review and approval of Industry Experience evaluations by System Engineering will be implemente * * The responses to the evaluations of Industry Experience Reports IN-88-83,* IN-92-65, and IN-93-38 which pertain to inadequate circuit modifications and testing will be re-evaluate * A plan to implement reviews to comply with NRG intended actions for the draft Generic Letter #95-XX:

"Testing of Safety-Related Logic Circuits" (IN 95-15)

published in the Federal Register May 22, 1995, will be develope * Continuing Training will be provided to engineering support personnel regarding managements expectations (or Industry Experience Report evaluation Page 19 * September 7, 1995

.*.********.*****PiA*Ll$@*QeS.****i~*BE*~.l:~ISl~*~~*~****e.*N*~~*ti1~*MENJr****§*~*~*1r~*~EN*~*e..

Bypassed CHP Inputs To The RPS CORRECTIVE ACTION SUMMARY:

Design Control Program

Implement a core Multi-Disciplinary Design Review Group

Revise procedures to

-

Better control the use of vendor documents

-

Initiate condition reports for modification problems.

  • -

Require use of experienced Prime Design Reviewers

-

Describe the role and responsibilities of the Design Team

Provide Continuing Training regarding vendor scrutiny Test Control Program

Establish a Test Authority for PMT and Surveillance Testing

Revise procedures to clarify functional testing requirements

Provide Continuing Training to emphasize management expectations for testing adequacy Industry Experience Program

Implement a second level System Engineering review of Industry Experience Evaluations

Re-evaluate INs 88-83, 92-65, and 93-38

  • *Develop a plan to implement the recommendations of the draft Testing Generic Letter

Revise Administrative Procedure to clarify. responsibilities for IE Evaluators and Reviewers

Provide Continuing Training to emphasize management expectations for performing/reviewing *industry Experience Evaluations Page 20

.

'..:

September 7, 1995

... ********:*******e~*~rs~ees*:***R~~DECISlcl:*N*~**~****j*fi*~<:>*B.~e.:i!i*EN:F**¢e*~*~*ERENC~:**************

Bypassed CHP Inputs To The RPS MITIGATING FACTORS:

Self-Identified

  • The loss of the CHP inputs to the RPS was self identifie * The RPS Monthly Surveillance Test (M0-3) was revised in early 1995 to remove the random selection of the channel to be teste Aggressive Corrective Actions
  • A significant pre-startup review of selected test* procedures was performe * An extensive test plan to re-establish RPS Operability was develope * An in depth Root Cause Evaluation was conducted and comprehensive Corrective Actions were identifie Historical
  • The faulty modification occurred prior to implementing the Palisades Performance Enhancement Pla Minimal Safety Significance
  • Margins of safety were not reduced by the even * There. was no adverse impact on the health and safety of the Public.

Page 21

  • ..

September 7, 1995 Bypassed CHP Inputs To The RPS MANAGEMENT LESSONS LEARNED:

  • Management involvement in modification process must continue to be stresse * Past Corrective Actions have fostered improvement in the.testing area,
  • but additional focus, alignment, and guidance is neede * Use of industry experience must be improve *. Corrective Action System changes are workin * Must continue to promote critical self assessment throughout the organization.

Page 22

,

L, *

RPS INITIATION LOGIC Y-10 CHP Jumpe~

HOLD DROP OU't'

Figure 1,

Y-20 Channel Function

VHP

HI RATE

LO PCS Flow

S/G #1 Level

S/G #2 Level

S/G #1 Pressure

S/G #2 Pressure

Pzr Pressure

TM/LP

Loss of Load

CHP Matrix Relays (4)

....

.

.

~ ~

.

~ ~

.

.

.

.

~ ~

....

....

....

....

.

.

.

.

.

~ *~

~* ~ ~ ~ ~ ~ ~.

.

.

.

.

.

.

.

.

.

.

.

.

.~ ~ ~ ~ ~ ~ ~ ~ ~

  • ~

.

.

.

.

.

.

.

.

.

.

.

.

.

.

~ ~ ~ ~* ~

.....

SOLDER SIDE

.

~

....

.

.

.

~ ~ ~

REMOVE TRAC MIN. -

~* --.-i TYP. <3> PLACES

....

CENTRALLY LOCATED AS SHOW TRIP UNIT INTERCONNECTION MODULE ASSEMBLY DETAIL/ARTWORK NOT TO SCALE

,:;*.,* **

,.*

1 *

...

.

N

~

.... V1G,2Q30,40 TUT PANEL swrrau [

--- - *-*. --.. --**

-*-*.. -

11 *

RPS

.....

1U1'

I POWER SUPP\\.Y DRAWER Ml-3A

... ---

u~---** -- -*... - *-

Ul-2A P0-1 RPS AW1, AW10, FRONT PANEL IHOICATIOltS (Vlltllua&.W*)

AW11, I..

Ul-5A AW12

Ml *3A *

Ml-3° *

Rl-68 APS-1-9 RPS-l-8°

ANALOG LOOP TESTS*

MATRIX AUCTIOMCEAING TEST*

PORV 2/4 MAIAIX lESl HI-l 0 HI *2*

Ml*2A Ml*SA Rl-47

. *

ZPM TEST*

PO-I Rl-94 RPS.OPERAS x

COMPLETE TEST Of RPS INITIATION AHO ACIUAllOll LOGI ICST REVISED 10 INCLUDE ASSURING CHANNEL BYPASS IS FULLY f UllC T I OIAl. *.

PERFORM REVISED CHANNEL IAIP TEST PORTION Of lESl WHICH INCLUOCS VERIFICATION THAI IH[ TRIP MATRIX IS NOi GROUNDED BY IOllTOillll6 MATRIX LIGHT eAIGHINCS CHP AHO SIS LEFT AND RIGHT CHANNEL PRESSURE SWITCH CALIBRATIO TEST INPUTS PRESSURE INTO SWITCH ANO VERIFIES TEST ll6HTS AID CHAIUICL RPS TRI TEST PERFORMED foi *rnw.* CAL IBAATIOlt DAT.

.

.

LOSS-Of *LOAD CHAIUIEL TRIP FUNCTIONAL T[S TRIPS OllC CHANNEL AT A TIME TO ASSURE PROPER INDICATIONS ANO OUIPUTS:

ATVS flllCTIOIAL TEST Of 2/4 MATRIX LOGIC WHICH ASSURES RPS TRI TEST TO VERIFY LOW FLOW. S/G LEVELS AHO PRESSURES. ANO PRESSURIZER PRESSURE CHANNELS ARE PROPERLY COllNECIEO TO COAllCCT TRIP lllll Ill RP UST 10 llCLUOC SIGllAL RUfHJP/IXMI TO VERIFY POLARITY Of CONNCCTIOl VERIFY EACH MATRIX PCMR SUPPLY IS PROPERLY CONNCCTCO TO MATRIX BY REMOVING REDUNDANT SUPPLY ANO VERIFYHIG MATRIX IS SllLL POWERE '

FUNCTIONALLY VERIFY PORV 2/4 MATRIX BY SIMULATING VARIOUS COMBINATIONS Of HIGH PRESSURIZER PRESSURE lAI THIS TEST IS FOR PAV-1042B ONLYI IN~IVIDUALLY TESTS VHP TRIP ~NO PAC-TRIP CHANNELS FROM NI DRAWER T~OUGH lO RPS TRIP UNI VERIFIES TRIP UfllT RECEIVES TRIP/PRETAI PAOCEDuRE MOOIFIED TO INCLUDE VEAIFICATl<>>I OF TRIP UltlT MATRIX LIGHTS, ALARMS ANO DAT ALOG6£1l.

VERIFIES RPS TRIP UNIT POWER SUPPLY VOLTAGE VERIFIES LOW FLOW, LOW S/G LEVELS AND PRESSURES, ANO PRESSURIZER PRESSURE CHANNEL TRIPS/PAE-TRIPS OCCtlt T~OUGH RPS TRIP TEST CIRCUITR TEST REVISED TO RECORD TRIP VOLIAGC AND PROVIDE WARNING Of POSSIBLE PORV OPUlllG _LOGIC IRI VERIFIES RPS TH/LP CHANNEL TRIPS T~OUGH THH ANO RPS TRIP TEST CIRCUITR VERIFIES CHP ANO SIS LOGIC FROM PRCSSlllC SWITCH TO RPS AHO SIS MATRI VERIFIES RPS TRIP UNIT TRIP INDICATION ON RP...

FUMCTIONALLY VERIFIES HIGH-RATE TRIP Of RP INITIATES 2.6 OPH RAT( WHEN ABO~( 10*4l POWER ANO ASSl.'aES RPS TRIP TEST ALSO VERIFIES RWP LOGIC ON ANY OME Of FOUR HIGH RATE PAE-TRIPS OR 2/4 VHP rttC*IRIPS.

VI TO TEST EACH CHANNEL AUIO*BYPASS Of ZPM VHCN ABOVE l0*4 VERIFIES REACTOR TRIP ON TURBINE TRIP ANO BOTH MANUAL TRIP PUSHBUTTONS TRIP TH[ RP TEST RUN AT HOT SHUTDOWN TO ALLOW OCTCAHINATION Of PCS LOW FLOW TRIP SETPOINT AFTER DETERMINATION, THC NEW SCTPOINTS ARC INSTALLED I.MIO THC LOW FLOW TRIP UNITS USING A PARTIAL Ml- *

NEV TEST OR ACQUIRES REVISION BEFORE PERFORMANC ~-

FIGURE 5 TEST PROCEDURES REVIEWED Technical Specification Surveillance Procedures Ml-39, Auxiliary Feed Water Actuation System Logic Test MO-7 A-1, Emergency Diesel Generator 1-1 (K-6A) MO-7 A-2, Emergency Diesel Generator 1-2 (K-68) Q0-1, Safety Injection System Q0-2, Recirculation Actuation System RI-7, Low Pressure SIS Initiation Logic Rl-14, SIRW Tank Level Switch Interlocks Test Rl-17, Main Steam Isolation Valve Circuits Test and Valve Closure Testing R0-11, Containment High Radiation Test 10. R0-12, Containment High Pressure and Spray System Tests 11. R0-97, Auxiliary Feed Water System Automatic Initiation Test 12. RT-SC, Engineered Safeguards System - Left Channel 13. RT-SD, Engineered Safeguards System - Right Channel 14. RT-13A, Normal Shutdown Sequencer Test - Left Channel 15. RT-138, Normal Shutdown Sequencer Test - Right Channel REFOUT Special Test Procedures T-275, Switchgear Bus 1 C Fast Transfer Test T-314, Functional Test of Bus 1 C Undervoltage Relays T-315, Functional Test of Bus 1 D Undervoltage Relays Permanent Maintenance Procedures RPS-1-8, Anticipated Transient Without Scram (A TWS) End to End Function.al Test REFOUT Post Modification Test Procedures T-FC-940-01, DG 1-1 Breaker and Protective Trip Logic Upgrade T-FC-940-02, DG i-2 Breaker and Protective Trip Logic Upgrade T-FC-940-03, DG 1-1 Bus 1 C Power Source Logic T-FC-940-04, DG 1-2 Bus 1 D Power Source Logic Wl-FC-954-01, Change P-8B Control from CV-0521 to CV-0522A.

  • Page 1 Rev 4 CONTAINMENT HIGH PRESSURE BYPASSED-EVENT & CAUSAL FACTOR (CF) CHART I

I FORMAT Issue relating to even Significant issue relating to even Facts/conditions or inappropriate actions relating to the issu Causal Factors relating to the issu Causal Factor - A factor that shapes the outcome of the situation relating to the issu Barrier broken or faile C-

-95-1117.-~

LDS*95-045 Attachment A

  • CONTAINMENT HIGH PRESSURE BYPASSED - EVENT & CAUSAL FACTOR (Cf) CHART

-

. ~

C-5-111; -.

LDS*95--045 Attachment A System Functional Evaluation (SFE) Review of Testing for Safety Channels (1988)

Wire List, Tech Manual Inaccurate (1989)

Applicable Mamo Not in RPS File (1989)

Page 2 Rev 4 Inaccurate Information Used to Develop FC-888 (1990-1991)

CF #1 No Priority To Update Vendor File Documents Configuration Management Design Control Design Input Not Verified, eg, Physical Walkdown or Drawing Review (1990-1992)

To Page 3 To Other Pages Inadequate Walkdowns CE on Approved Supplier List

  • ---
  • From Page 2 Page 3 Rev 4

CONTAINMENT HIGH PRESSURE BYPASSED - EVENT & CAUSAL FACTOR (Cf) CHART CE Factory Acceptance Teat Did Not Detect (December 1991)

Factory Test From Page 1 Procedure Not Correct CF 13 FC-888 Inadvertently Bypasaea CHP Trip by Designing & Installing a Circuit Board Connection (March 1992)

Inadequate Oversight, etc From Page 1 Similar Issues Found (Missed Opportunities)

- Ground Found During Installation

- CHP Trip Alarm Not Wired (March 1992)

No Evaluation of Root Cause and Action to Prevent *

Recurrence Correcltve Action Program C-

-95-1117 '

LDS*95-045 Attachment A To Page 4

  • CONTAINMENT HIGH PRESSURE BYPASSED* EVENT & CAUSAL FACTOR (CF) CHART Poat Modification Testing Doea From ---1.-

Not Detect Problem Page 3 (March 1992)

Page 4 Rev 4

. Post Modification Teating Industry Experience Surveillance Teat M0-3 Ooe Not Detect Problem During 31 Teats (April 1992 - Present)

Surveillance Testing Operator* Never Noticed that Channel 11 Matrix Relay Light Stayed On c~

-ss-1111~.-'

~!

Los*9s--04s Attachment A To Page 5

,.

From Page 4 Page 5 Rev 4 CONTAINMENT HIGH PRESSURE BYPASSED - EVENT & CAUSAL FACTOR (CF) CHART Fortunately Channel 11 Selected Clone of Surveillance Test M0-3 Relied on for Post Mod Test

__..___ CF#8 Inappropriate Reliance On*

Surveillance Test for Post Mod Testing RPS Connector Post Modification Testing Detects CHPJumper (July 28, 1995)

Post Modification Tea ting Not Ali Functions That Were Effected Were Tested CF #8 EDCs Not Evaluated for Generic Implications

-

    • 1"' _.._

,-:.>.*

~*

c-95-1117.:

LDS*95-045.

Attachment A

Retrieved from "https://kanterella.com/w/index.php?title=IR_05000255/1995012&oldid=5017924"