ML18065A231

From kanterella
Jump to navigation Jump to search
Responds to NRC 950929 Ltr Re Violations Noted in Insp Rept 50-255/95-10.Corrective Actions:Reactor Protection Sys Matrix Channels Modified to Restore Containment High Pressure Trip Function
ML18065A231
Person / Time
Site: Palisades Entergy icon.png
Issue date: 10/30/1995
From: Smedley R
CONSUMERS ENERGY CO. (FORMERLY CONSUMERS POWER CO.)
To:
NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM)
References
EA-95-169, NUDOCS 9511030385
Download: ML18065A231 (15)
v • d • e


Text

-.

  • consumers Power POWERIN&

lllllCHl&AN"S PRO&RESS Palisades Nuclear Plant: 27780 Blue Star Memorial Highway, Covert, Ml 49043 October 30, 1995 U S Nuclear Regulatory Co11111ission Document Control Desk Washington, DC 20555 DOCKET 50-255 - LICENSE DPR PALISADES PLANT REPLY TO NOTICE OF VIOLATION - NRC INSPECTION REPORT 95010 - INOPERABLE CONTAINMENT HIGH PRESSURE TRIP ON REACTOR PROTECTION SYSTEM (EA 95-169)

  • - NRC Inspection Report No. 95010(DRS) dated August 23, 1995, and NRC Notice of Violation letter dated September 29, 1995, documented the results of an inspection that reviewed the inoperability of the Palisades Reactor Protection System (RPS) containment high pressure trip function. The inspection report and letter identified three violations which were classified as a Severity
  • Level III Problem. They involved: *
1. Use of an outdated, uncontrolled wire list as a design input to a modification;
2. Conduct of an inadequate post-modification test;
3. Operaiion of the facility with the containment high pressure reactor trip function bejng inoperable when required.

Because the Palisades Plant had been the subject of escalated enforcement within the last 2 years, self identification of the problem and implementation of comprehensive corrective actions resulted in no civil penalty being

  • proposed.

Attachment 1 provides our response to the Notice of Violation.

SUMMARY

OF COMMITMENTS The convnitments for necessary corrective actions have been previously documented under licensee Event Report 95-008-01. These co11111itments are

  • restated here for comnleteness:

9511030385 951030

.PDR ADOCK 05000255 Q PDR A CMS~ COMPANY

  • 1. Provide enhancements to the Design Control Program by implementing the following changes:

2

a. Revise Administrative Procedures (AP) 9.44, *Design Document Control," AP 9.45, "Vendor Manual Control,* and AP 10.44,
  • Engineering Records Center Distribution and Control of Design Documents" to indicate vendor manuals and vendor drawings have not been maintained as living documents. Also, add a requirement to AP 10.44 to attach a notice of that condition to any second generation print issued by ERC which contains information obtained from a vendor document, manual or drawing.
b. Revise Project Management Construction and Testing procedure 1-3, "Project Team Organization and Responsibilities," to better describe roles, responsibilities, and requirements for team meetings.
c. 'Revise AP 9.03, "Facility Change,* to add the requirement to*

initiate condition reports when potentially generic, or significant issues are discovered during Facility Change Project work, and to eliminate the option of using an inexperienced Prime Design Reviewer.

d. Revise AP 9.04*, "Spedfication Changes," to add the requirement t.o.

initiate condition reports when potentially generic or significant issues are discovered during Specification Change Project work.

e. Provide quarterly continuing training to Engineering Support personnel using a case study methodology to include as a minimum:

testing program improvements, 'management expectations for testing and consultant/vendor scrutiny, and Industry Experience Report evaluations.

2. Provide enhancements to the Test Control Program by implementing the *.

following changes:

a. Assign responsibility for review of all modification testing to Systems Engineering who will act as the testing authority.
  • Surveillance tests assigned to Systems Engineering shall meet the standards established by the testing authority.
b. Revise applicable administrative procedures to emphasize functional testing requirements and accountability for test adequacy and completeness. Ensure procedures include the expectation of 100%

functional testing of anything changed or affected directly or indirectly by the work done for a project. Include a description of .

overlapping when relying on multiple tests to.*meet requirements .

  • 3. Provide enhancements to the Industry Experience Program by implementing the following changes:

3

a. Re-evaluate the documented responses to similar Industry Experience Reports (IN) pertaining to inadequate circuit modifications and testing, ,IN_-88-83, IN-92-65, and IN-93-38.
b. Develop a plan to implement reviews to comply with NRC intended actions for the new (currently in draft) Generic Letter No. 95-XX:
  • resting of Safetj-Related Logic Circuits" published on May 22, 1995, in the Federal Register.
c. Implement a second level critical review and approval of Industry Experience evaluations by Systems Engineering. Re-emphasize the importance of management expectations for industry experience reviews.

~13:

Richard W. Smedley Manager, Licensing CC: Administrator, Region III, USNRC NRC Resident Inspector - Palis~des

  • CONSUMERS POWER COMPANY To the best of my knowledge, information and beljef, the contents of this submittal are truthful and complete.

By ;I J -;*.:.L__:_~~

Thomas7J. Palmisano Plant General Manager Sworn and subscribed to before me this <?fJ;J._ day of *&!,cbtt.>,. 1995.

Alora H. Davis, Notary Public Berrien County, Michigan (Actin~ in V~n Buren County, Michigan)

Hy convnission expires August 26, 1999

\

ATTACHMENT 1 CONSUMERS POWER COMPANY PALISADES PLANT DOCKET.50-255 REPLY TO NOTICE OF VIOLATION NRC INSPECTION REPORT 95010 10 Pages

/

  • 'ATTACHMENT 1 REPLY TO NOTICE OF VIOLATION BACKGROUND On July 28, 1995, with the plant in cold shutdown condition, all control rods ins~rted, and at refueling boron, it was discovered during design change testing that none of the four containment high pressure channels would initiate a reactor trip. An undesired connection across the containment high pressure trip contacts was discovered in the printed circuit boards of the interconnection modules. The connection was introduced as a result of a modification installed in 1992 which replaced all four channels of reactor protection because of high maintenance and obsolescence of spare parts. This connection provided a circuit path that bypassed the containment high pressure trip output to the matrix logic of the reactor protection system, thereby disabling the containment high pressure trip circuit.

The NRC Notice of Violation letter dated September 29, 1995, identified three violations, 95010-01013, 95010-01023, and 95010-01033. Each violation will be replied to individually. However, since some actions are applicable to more than one violation, all three violation responses should be considered together. This reply restates information (including actions) that were discussed during the September 7, 1995 predecisional enforcement conference.

~~ I. VIOLATION 95010-01013.

~A. 10 CFR 50, Appendix B. Criterion Ill, "Design Control," requires in part, that measures be established to assure that applicable regulatory requirements and the design basis are correctly translated into ipecifications, drawings, procedures, and instructions. The design control measures shall provide for verifying or checking the adequacy of design by the performance of a suitable testing program.

1. Contrary to the above, the design basis was not correctly translated into specifications, drawings, procedures, and instructions in that uncontrolled Wire list No. 87789-ICE-3912, Revision 2, was used as a design input for Facility Design Change No. FC-888 which was installed in March 1992.

Specifically, the wire list included twelve incorrect interconnection module circuit connections which bypassed the reacto~ protection system containment high pressure trip function. (01013)"

  • cpco RESPONSE REASON FOR THE VIOLATION The violation occurred because of prograD111atic deficiencies in the design control program with the following primary root causes:

I. Inadequate Program Scope, (omission of necessary functions or guidance in procedures):

a. Plant procedures did not alert users that vendor information was uncontrolled and could be inaccurate. Vendor documents (manuals and drawings) were not always maintained up to date.

This became an event precursor when the FC-888 project engineer removed the outdated wire list from the vendor file and used it as a design input without adequate validation and verification.

The wiring list still had a circuit bypass for the CHP channel since the initial plant design did not include a CHP trip.

b. Plant procedures did not establish clear roles and responsibilities for the implementation of modifications.

Besides the Prime Design Reviewer, no other personnel on site felt direct ownership for the successful implementation of the modification.

2. Inadequate Prioritization Of Work, (inappropriate application of resources due to a misunderstanding of task significance or complexity or availability of resources):
a. The plant Prime Design Reviewer (PDR} and the .Combustion Engineering (CE) designer, who acted as the Responsible Engineer, were both inexperienced. The assignment of the inexperienced engineers to this modification was due to a lack of available experienced personnel and the mindset of management that the RPS modification was one of limited complexity and scope. It was believed that CE, as a whole, had the knowledge and skills to balance out this inexperience. The new RPS design was a clone of the early 1980's RPS System designs that were used for St. Lucie and Millstone nuclear plants.
b. There were very few modification team meetings conducted. All of the technical reviews were completed by CE. Palisades personnel only provided cursory reviews of portions of the design. At that time, there was no prograD111atic guidance to provide system engineer involvement or ownership in the modification planning and reviews.
3. Inadequate Program Monitoring and .Management, (insufficient oversight and self-assessment):
a. There was an over reliance on CE due to their being on the approved suppliers list and being the origi~al equipment
  • 4.

manufacturer. CE's design reviews and testing were less than adequate and the plant oversight was not effective.

Inadequate Attention To Emerging Problems, (ineffective problem 3

identification and root cause analysis):

a. During modification preparations, CE was behind schedule and several problems were identified during design reviews, testing and installation. With the lack of experience and attention associated with the oversight of this modification, Palisades was unable to recognize the generic implications.
b. During the installation of the RPS modification in 1992, two design problems were identified. The problems could have been linked to the use of the same incorrect wiring list. If these two conditions had been sufficiently evaluated in 1992 for generic implications, the precursor (inaccurate wiring list) could have been identified and the inadvertent CHP bypass could have been discovered at that time. Corrective action documents were not written for the two problems noted during the modification installation. The corrective action process initiated in 1994 is more intrusive and has resulted in Condition Reports (CRs) being generated and evaluated in a more timely manner for similar issues ~ssociated with modification problems .

CORRECTIVE ACTIONS TAKEN AND RESULTS ACHIEVED The following corrective actions have been taken:

1. The RPS matrix channels have been modified to restore the CHP trip function.
2. The Technical Specifications Surveillance Test (M0-3) for the RPS matrix logic was revised to provide adequate overlap in testing.

This will assure that the requirements of Technical Specification Table 4.17.1 Item 13 are being adequately verified during the monthly test.

3. A review of 24 Technical Specifications Surveillance Procedures, post modification testing completed during the 1995 REFOUT, and Special Tests completed during the 1995 REFOUT was completed. This review was performed to verify that procedures used for testing safety related circuits completely test the design functions of the equipment. As part of this review, condition reports were written and procedures were revised. A number of procedure enhancements were reconvnended to the procedure sponsors; None .of these deficiencies resulted in any of the equipment bEing declared inoper~ble.

4.. An evaluation was completed for all of the Functional Equivalent Substitution (FES) and Specification Change ~SC) packages installed

  • in the 1995 REFOUT. If the FES or SC changed the logic matrix of either the RPS or Engineering Safeguards initiation, the acceptance test was reviewed to verify that the logic matrix functioned properly. This evaluation resulted in two SC's and two FES's being reviewed. The testing which verified each project was found 4

acceptable. There was a concern about the testing for SC-95-033 because the testing was not comprehensive in that the test only verified proper operation of the modified contact and did not verify that the remainder of the logic had not been disturbed. Additional surveillance testing fully verified the adequacy of the entire circuit.

5. A core multi-disciplinary design review group has been implemented for technical review of modifications.

CORRECTIVE ACTIONS TO BE TAKEN TO AVOID FURTHER NONCOMPLIANCE The following additional actions are planned to enhance the Design Control Program:

1. Revise Administrative Procedures (AP) 9.44, *Design Document Control,* AP 9.45, *vendor Manual Control,* and AP 10.44,
  • Engineering Records Center Distribution and Control of Design Documents,* to indicate vendor manuals and vendor drawings have not
    • been maintained as living documents. Therefore, the information in these documents shall be verified and validated before it is used to make decisions or prior to being used for design input. Also, add a requirement to AP 10.44 to attach a notice to any second generation print made to a vendor document, manual or drawing being issued by ERC. Personnel receiving the second generation copy will thus be notified that vendor documents (manual or drawings) have not been maintained as living documents.
2. Revise Project Management Construction and-Testing procedure 1-3,
  • project Team Organization and Responsibilities,* to better describe roles, responsibilities, and requirements for team meetings. *
3. Revise AP 9.03, *Facility Change,* to add the r~quirement tci initiate condition reports when potentially generic, or significant issues are discovered during Facility .Change Project work, and eliminate the option of using an inexperienced Prime Design Reviewer.
4. Revise AP 9.04, *specification Changes,* to add the requirement to initiate condition reports when potentially generic, or significant issues are discovered during Specification Change Project work~

5,. Provide quarterly continuing training to Engineering Support personnel using a case study methodology to include as a minimum; testing program improvements, management expectations for testing and consultant/vendor scrutiny, and Industry Experience Report evaluations. *

  • DATE WHEN FULL COMPLIANCE WILL BE ACHIEVED Full compliance has been achieved.

5 II. VIOLATION 95010-01023.

"A. 10 CFR 50, Appendix B. Criterion III, *Design Control,* requires in part, that measures be established to assure that applicable regulatory requirements and the design basis are correctly translated into specifications, drawings, .procedures, and instructions. The design control measures shall provide for verifying or checking the adequacy of design by the performance of a suitable testing program.

2. Contrary to the above, the post-modification test (Procedure No. M0-3, Revision 12) conducted on April '3, 1992, for Facility Design Change No. FC-888 was not suitable to v.erify or check the adequacy of design. Specifically, the test did not
  • identify that the reactor protection system containment high pressure trip function was bypassed by the installation of interconnection modules containing twelve incorrect printed circuit board connections. (01023)"

CPCO RESPONSE REASON FOR THE VIOLATION The violation occurred because of programmatic deficiencies in the testing control program with the following primary root causes:

1. Inadequate Program Scope, (omission of necessary functions or guidance in procedures) and Lack of Commitment to Program Implementation:
a. The plant procedures and organization lacked .a single point of contact for testing to ensure consistent application of proper testing techniques and overlap.
b. There was an inappropriate reliance on the existing Technical Specifications (TS) Surveillance Test M0-3, *Reactor Protection Matrix Logic Tests," which provided a porti-0n of the post modification verification. The planned approach for the post modification testing consisted of two separate tests. A test procedure written by CE tested the channel relays for proper operation. The second test was the normal monthly Palisades TS test M0-3. This test was used to verify functional operability of the channel relay output to the RPS matrix logic. The TS test M0-3 was used without verifying that it would functionally test all the matrix logic that needed to be tested after the modification .
  • M0-3 was determined to not fully meet the intent of Technical Specifications Table 4.17.1 Item 13, RPS matrix logic testing every 31 days. H0-3 did verify that the High Containment Pressure (CHP) relay contact was opening (single pole double 6

throw contact) but it failed to determine if the contact was being bypassed. The TS surveillance procedure was revised to assure adequate testing overlap to verify the RPS matrix logic performance. T~is adverse condition pointed out the causal factor that the surveillance test program lacked direction in functional testing of Technical Specifications equipment including assurance of adequate overlap of all functions being tested.

2. Inadequate Prioritization Of Work, (inappropriate application of resources due to a misunderstanding of task significance ~r complexity or availability of resources):
a. The then existing roles and responsibilities of the system engineers allowed other priorities to take precedence over a thorough review of existing TS Surveillance Procedures or new modification test procedures. This situation led .to cursory reviews of the modification test procedures.
3. Inadequate Program Monitoring and Management, (insufficient oversight and self-assessment):
a. The CE Factory Acceptance Test (FAT) scope should have identified the undesired bypass for the CHP trip. However, there were problems with the test procedure being writte~

incorrectly, identifying that CHP trip test lamps should be lit when they should have been off and, conversely, off when they should have been lit. Currently it is being assumed that the test rig for the FAT was wired in~orrectly, which resulted in a failure to identify the CHP bypass. Plant oversight of the FAT procedure and performance was inadequate. The Palisades PDR and Test Engineer had other convnitments and were not present to oversee the FAT. The System Engineer and two Instrument and Control technicians observed a portion of the*FAT solely to familiarize themselves with the new system hardware. The FAT failure to locate the CHP bypass was a causal factor indicating that CE's test program was less than adequate.

CORRECTIVE ACTIONS TAKEN AND RESULTS ACHIEVED The following corrective actions have been taken and results achieved:

1. The RPS matrix channels have been modified to restore the CHP trip function.
2. The Technical Specifications Surveillance Test (H0-3) for the RPS matrix logic was revised to provide adequate overlap in testing .

This will a_~ure that the requirements of Tecrnical Specifications

    • 3.

Table 4.17.1 Item 13 are being adequately verified during the monthly test.

A review of 24 Technical Specifications Surveillance Procedures, 7

post modification testing completed during the 1995 REFOUT, and Special Tests completed during the 1995 REFOUT was completed. This review was performed to verify that procedures used for testing safety related circuits completely test the design functions of the equipment. As part of this review, condition reports were written and procedures were revised. A number of procedure enhancements were reconvnended to the procedure sponsors. None of these deficiencies resulted in any of the equipment being declared inoperable.

4. An evaluation was completed for all of the functional equivalent substitution (FES} and specification change (SC} packages installed in 1995 REFOUT. If the FES or SC changed the logic matrix of either the RPS or Engineering Safeguards initiation, the acceptance test

-.* was reviewed to verify that the logic matrix functioned properly.

Thi.s evaluation resulted in two SC's and two FES's being reviewed.

The testing which verified each project was found acceptable. There was a concern about the testing for SC-95-033 because the testing was not comprehensive in that the test only verified proper operation of the modified contact and did not verify that the remainder of the logic had not been disturbed. Additional surveillance testing fully verified the adequacy of the entire circuit.

5. An independent evaluation of the testing adequacy for FES-95-032;
  • RPS Trip Unit Connector Block Replacement* was completed. The evaluation consisted of a review of trip module inputs and outputs affected by the modification to determine the extent to which all functions were checked as part of the post modification test. The scope of the work affected more than 2500 pins in the connector blocks. .The results of the assessment indicate that functions using 2% of the pins were not verified and 9% of the pins were not thoroughly tested. The evaluation determined that the post modification testing was less than adequate. Operability of .the RPS was subsequently verified by a new test.
6. A test plan was developed to declare the RPS operable. The plan objective was to assure full operability of the RPS through the use of overlap and/or full functional testing. The strategy assured that every RPS input and output functioned properly through functional tests and finally, assured total operability of the RPS safety and non-safety related functions. The testing plan was reviewed by ABB-CE .

8 CORRECTIVE ACTIONS TO BE TAKEN TO AVOID FURTHER NONCOMPLIANCE Provide enhancements to the Test Control Program by implementing the following changes:

1. Assign responsibility for review of all modification testing to Systems Engineering who will act as the testing authority.

Surveillance tests within Systems Engineering shall meet the standards established by the testing authority.

2. Revise applicable administrative procedures for Design Control, Temporary Modification Control, and Surveillance Test Program Control to emphasize functional testing requirements and accountability for test adequacy/completeness. Ensure procedures include the expectation of 100% functional testing of anything changed or affected directly or indirectly by the work done for a design change. Include a description of overlapping when relying on multiple tests to meet requirements.

DATE WHEN FULL COMPLIANCE WILL BE ACHIEVED Full compliance has been achieved.

III. VIOLATION 95010-01033, wB. Technical Specification 3.17.1 requires that four Reactor Protective System (RPS) trip unit channels and the associated instrumentation for the functions listed in Table 3.17.1, and 6 matrix logic channels and 4 initiation logic channels be operable except as allowed by the permissible operational bypasses. Specification 3.17.1 applies when there is fuel in the reactor, more than one control rod is capable of being withdrawn,- and the PCS is less than refueling boron concentration.

Contrary to the above, from April 11, 1992, until May 22, 1995, the six matrix logic channels for high containment pressure were not operable when there was fuel in the reactor, more than one control rod was capable of being withdrawn, and the PCS was less the refueling boron concentration. (01033)w CPCO RESPONSE REASON FOR THE VIOLATION The reasons for the above violation are the same as those reasons listed for violations 95010-01013 and 95010-01023. There were also opportunities to detect the problem, however, that we missed. Therefore, an additional reason also applies to this violation:

  • 1. Inadequate Program Monitoring or Management (insufficient oversight and self-assessment)
a. The documented evaluations of several similar Industry 9

Experience Events (INs) were.reviewed and found to be inadequate. Four of the INs point out inadequate post modification testing, inadequate overlap of testing, and inadequate testing of safety related circuits. There was a tendency to provide only a cursory review of the implications of the industry events with a general assumption that the present testing methods are inherently valid.

CORRECTIVE ACTIONS TAKEN AND RESULTS ACHIEVED The following corrective actions were completed:

1. The RPS matrix channels have been modified to restore the CHP trip

. function.

2. The Technical Specifications Surveillance Test (M0-3) for the RPS

. matrix logic was revised to provide adequate overlap in testing.

This will assure that the requirements of Technical Specifications Table 4.17.1 Item 13 are being adequately verified during the monthly test .

  • 3. The test plan was developed to declare ~he RPS operable~ The plan objective ~as to assure full operability of the RPS through the use of overlap and/or full functional testing. The strategy assured that every RPS input and output functioned properly through functional tests, and assured total operability of the RPS safety and*non-safety related functions.

CORRECTIVE ACTIONS TO BE TAKEN TO AVOID FURTHER NONCQMPLIANCE Provide enhancements by implementing the following changes:

1.
  • Assign responsibility for review of all modification testing to Systems Engineering who will act as the testing authority.

Surveillance tests assigned to Systems Engineering shall meet the standards established by the testing authority.

2. Revise applicable administrative procedures to emphasize functional testing requirements and accountability for test adequacy and completeness. Ensure procedures include the expectation of 100%

functional testing of anything changed or affected directly or indirectly by the work done for a project. Include a description of overlapping when relying on multiple tests to meet requirements.

3. Provide enhancements to the Industry Experience Program by implementing the following changes:

-* a. Re-evaluate the documented responses to similar Industry Experience Reports (IN) pertaining to inadequate circuit modifications and testing, IN-88-83, IN-92-65, and IN-93-38.

10

b. Develop a plan to implement reviews to comply with NRC intended actions for the new (currently in draft) Generic Letter No.

95-XX: *resting of Safety-Related Logic Circuits* (IN 95-15) published in the Federal Register Hay 22, 1995.

c. Implement a second level critical review and approval of Industry Experience evaluations by Systems Engineering. Re-emphasize the importance ~f management expectations for industry experience reviews.

DATE WHEN FULL COMPLIANCE WILL BE ACHIEVED Full compliance has been achieved .

Retrieved from "https://kanterella.com/w/index.php?title=ML18065A231&oldid=2413676"