ML20136E679
Text
. . . .. . -
^[
s '. ,
- 4- ;
1 1
4:
- From: ' Mark Miller, Elf .-
To: ATP1.JWY STripp TLorIc,(('ll I
Date: 11/18/9610:47am
Subject:
- Here it is .
i l '
9 I
1 1
1 i
N
'R -9703130281.970306 7 PDR FOIA BINDER 96-485- .PDR- z,,,
r .
'V l
.i (Closed)URI 335/96-11-06, " Unit l'N! Wiring Errors" Background on Unit 1 Nuclear Instrumentation
.The;St. Lucie Unit 1 Nuclear Instrumentation (N1) System is designed to employ a total of 10 detectors, all located external to the reactor vessel, as described below:
Detector Detector Types Channel Purpose Associated RPS Trips Numbers .
- 1. 2. 3, 4 Fission Chamber- Wide Range High Rate of Change Logarithmic Detection
- 5. 6. 7. 8 Uncompensated Linear Safety
- Variable High Power lon Chamber Related e, Thermal Margin /Lcw Pressure Detection e Local Power Density
- 9. 10- Uncompensated Linear None -
lon Chamber Non-Safety Detection '
(control channels)
. Detectors 5, 6. 7, 8. 9. and 10 above are designed with 2 axially distinct detectors each, forming single detector assemblies to allow for the detection of power fractions in the upper and lower axial halves of the core. The signals from the upper and lower halves of each of these detectors are summed to form a total power signal from each detector. The upper and lower halves are also combined
. electronically to develop an Axial Shape Index (ASI) signal consistent with the
- equation
- ASI.- (Lower Detector Power - Vooer Detector Power)
(Lower Detector Power + Upper Detector Power) i 4
ASI is used in the control of axial core power distribution by operators and to provide inputs to Thermal Margin / Low Pressye (TM/LP) and Local Power Density (LPD)
Reactor Protection System (RPS) trips y Each of the 4 linear safety related detectors i
feed an individual RPS cnannel and the RPS trips the reactor on a 2-out-of 4 coincidence for a given trip (e.g..TM/LP).
The purpose of the TM/LP ' trip is to prevent exceeding Departure from Nucelate Boiling (DNB)Llimits and to provide a low pressure reactor trip in the event of a loss of coolant accident. The trip setpoint itself is designed to.be a reactor coolant
. system (RCS) pressure limit, set at the higher of 1750 psia or a pressure setpoint calculated by the core' protection calculator based upon RCS temperature conditions, reactor power, and power distribution (expressed by the ASI signals generated by detectors 5; 6.-7._and 8). The TM/LP.. trip is required per Technical. Specifications (TS) Table 2.2-1, with setpoints adjusted to agree with Figures 2.2-3 and 2.2-4 of-
', .TS; ~TS Table 3.3 1 requires that at least.3 operable RPS channels provide 'this trip
- function at or above 1% thermal ~ power.
.N
4 The purpose of the LPD trip is to. prevent peak local power density in the fuel from exceeding 21 kw/ft, thus assuring that the melting point of the fuel will not be reached during anticipated operational occurances. The trip is designed to be initiated whenever axial power offset, as indicated by AS1, exceeds either a high or a low calculated setpoint. The LPD trip is required per TS Table 2.2-1, with setpoints adjusted to agree with Figures 2.2-3 and 2.2-4 of TS. TS Table 3.3 1 requires that at least 3 operable RPS channels provide this trip function at or above 15% thermal power.
Each linear NI detector assembly is cylindrical in design, with the'two detectors located axially adjacent to one another. The signal cables for each detector assembly (which transmit output from both detectors) exit the assembly from one end of the cylinder. Each detector is installed in a cylindrical housing external to the reactor vessel in positions spaced radjally about the vessel to ensure that a'll quadrants of the core are monitored.
The signal cables for the detectors installed in Unit I had been labeled by the vendor as "TO,P SIG" and B0T SIG." indicating output for each detector in the assembly. This labeling scheme assumed that each assembly was installed in its housing oriented such that the signal cables exited the top of the assembly.
However, the St. Lucie design is such that the detector assemblies are installed with the signal cabling. exiting the assemblies from the bottom, making the " TOP SIG" signal cable correspond to the detector monitoring the bottom half of the core, and vice versa.
During the most recent Unit 1 outage, the licensee elected to replace their existing NI circuitry with a new systen developed cooperatively with Gamma-Metrics. The scope of the change included new NI drawers for all four RPS channels and new wide range detectors. During the outage, unrelated to this modification, uncompensated ion chambers (U!Cs) were replaced in RPS channel B (detector 6) and in the control channel detector 9 location.
Detection of Problem / Licensee Action 1
On July 30, with Unit 1 at 100% power. Reactor Engineering (RE) personnel were analyzing power ascension testing data (from low power physics testing completed on July 25) for the determination of Shape Annealing Factors (SAFs - which correlate axial power distribution as determined by incore detectors to ASI as indicated by excore detector channels). During the data review, it was determined that RPS channels A. C and D and control channel 9 indicated ASI values which were trending in a manner which was opposite to that expected. In pursuing the issue, the licensee identified a discrepancy involving wiring errors in the safety related linear NI channels. The errors resulted in channels A. C. and D reporting ASI values which were opposite of true values; that is, an apparent wiring error had reversed upper and lower dotector inputs to the NI drawers such that ASI was misciculated.
At 1:00 p.m. the same day. Unit 1 operators were informed of these conditions and immediately declared the A. C and D channels out-of-service (005), which placed the unit in TS 3.0.3 due to 3 of 4 RPS channels for TMLP and LPD being inoperable. The inspector responded to the control room and found that leads were being reversed on the A and C channels in an attempt to restore the channels to' operability. Reactor
. Engineering support was available, with new NI gain values being calculated in I
. support of 1&C as the leads for the affected channels were properly aligned. At 1:50 l
j .
l and'2:00 p.m.. work was completed on the A and C channels, respectively, and.
operatbrs drove CEAs into the core to verify proper A$l response, The inspector verified that proper channel response occurred, and.the A and C channels were declared operable at 2:00 p.m.. leading to an exit from TS 3.0.3.
Overview of Deficiencies Leading to the Event in reviewing the issue, the licensee determined that all four safety related NI channels had been connected to the NI drawers in the RPS cabinets with the upper and lower detector inputs reversed. These field errors resulted from errors .in the associated Control Wiring Diagrams (CWDs): that is. the detectors were connected at the NI drawers in conformance with approved engineering drawings. The drawings were prepared and approved with errors in the designation of connection points to the NI drawers. The errant drawings were a part of the Plant Change / Modification (PC/M) package which replaced the NI drawers during the most recent Unit 1 outage.
In addition to the wiring errors described above..the licensee determined that NI detectors 6 (associated with RPS channel B) and 9 (one of two control channels) were miswired at the detector ends (in containment), which resulted in RPS NI channel B indicating correctly (as the error at the detector end cancelled out with the error at the Ni drawer) and the control channel indicating incorrectly. The miswiring at I the detectors was.the result of errors made by maintenance personnel during installation.
Details of Deficiencies As a result'of the deficiencies described above the' licensee performed a root cause evaluation. The. inspectors reviewed the evalaution. discussed the events with the licensee and performed independent reviews of documentation associated with the Ni modification. The results of these activities are described below.
Errors Associated with CWD's
- a. Scope The licensee's root cause effort determined that, during the preparation of PC/M 96-009-195. *RPS N1 Drawer Replacement." a lack of available' vendor information (in the form of approved technical manuals and drawings) forced
. design personnel to develop drawings based on assumptions generally derived from a similar modification made to Unit 2 during the most recent Unit 2 outage. As a result. CWDs JPN-009-195 002 (pro'viding wiring instructions for '
RPS channel A), JPN-009-195-004 (providing wiring instructions for RPS channel B)1 JPN-009-195-006 (providing wiring instructions for RPS channel C). and )i JPN-009 195-008 (providing wiring instructions for RPS channel D) were -I modified showing upper and lower detector ca.ble connections at each respective drawer which were in conflict with the' internal wiring of the drawers which were to be installed (i.e, field cables for the upper detectors were fed to the drawers' lower inputs and vice versa).
'Beyond the assumption of similarity between units the licensee found that the -
verification process for the drawings in question contained weaknesses that j
resulted in a. failure to identify the condition. Specifically. the licensee '
.found that the drawings were prepared by_a Lead Designer (draftsman) based on
. inputs from the Lead Engineer (having responsibility for developing the
+ -
3 i s ,
1 e 3
, . modification package). 'The drawings were then checked by a second Designer.
constitutir.g (essentially) a drafting check. The drawings were then reviewed !
and approved by the Lead Engineer and approved by the Lead Engineer's ;
l Supervisor. -
The licensee found that this process differed _from the approval process for $
the modification package, which was independently reviewed by a second [
engineer. However, the licensee pointed out that the package review did not include a point to-point wiring check of the affected drawirgs. The licensee l found that the method and sequence of veri,fication described appeared to meet ,
the requirements of QI 1.7. Revision _.. " Design Input / Verification." but .
that the method revealed a weakness of the process. !
While the subject drawings were approved without having approved vendor technical manuals WTMs) available to support the review, the licensee had
-placed a hold on the PC/M completion pending receipt of the vendor documents.
^
.Upon receipt. the VTM and other vendor documents were to be incorporated into the PC/M via Change Request Notification (CRN). However, when the documents were received, the licensee stated that they were not reviewed against other pac,kage drawings; thus. the wiring designation discrepancy was not identified.
The licensee identified an additional weakness in the amount of time available
' to verifying engineers. Specifically, the licensee stated: ;
"The total U1 project man-hours through May 1996 was approximately 1150 man-hours. For scoping and scheduling purposes on a critical complex project.
30% of a projects [ sic] total. man-hours should be allocated for verification activities. This should have accounted for a few hundred man-hours. The verifying Engineer on the Engineering Package charged a total of 40 man hours to the project. 15% were on overtime. The verifying Engineer on the CRN charged a total of 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> to the project. This relatively small amount of i time spent on verification is due to project delays due to supporting l unanticipated outages. the number of large projects being worked coincidently I
[ sic] in PEG 1&C...and the available resources... 1 I
i The inspectors reviewed these licensee-generatea findings for correctness. ,
i
- b. Findings j i
The inspectors reviewed CWDs prepared with the PC/M package and found the l licensee's conclusions on the reasons for the field miswiring to be correct.
The inspectors noted that the licensee's conclusion that the drawings had received an inadequate independent verification to be c < rect; however, the inspectors disagreed with the conclusion that the vern ation process was performed met the requirements of the applicable Quality Instruction. The
+ inspectors reviewed Instruction 01 1.7, Revision . " Design l Input / Verification." and found the following: )
I Step _ , " Design Verification is the process whereby a competent
, individual, who has remained independent of the design process.
reviews the design inputs. .and design outpu,t to verify design adequacy..."
Step i " REQUIREMENT FOR ENGINEER TO PERFORM REVIEW" j
l l
, i
- i !
2a +
LStep _ " REQUIREMENT TO SAVE DESIGN INPUTS AS 0A RECORDS" I The inspectors concluded that the subject 01 was of sufficient clarity to .
ensure an appropriate review. ' Spec 1'fically, the 01. when taken as a whole, ,
required that an engineer (not a Designer) perform an independent review of '
i
- .. design output documents (e.g. drawings) to ensure agreement with design inputs l l '(in the case of the subject PC/M the Gamma-Metrics VTM). Contrary to these '
requirements, the licensee, chose to approve the subject CWDs when no approved design input document was available aga1nst which the. output could be reviewed ;
and allowed the review to be performed without the required level,of. ;
independence. Further. when the approved design input was available the '
. 11censee failed to perform necessary reviews to ensure agreement between vendor-generated documents and existing licensee-generated drawings.
U '
As a result of the licensee's findings relative to a lack of independence jn I. ,
the verification process. the licensee performed a review of open PC/Ms to e determine whether other examples existed. The licensee found that 3 out of.8 !
open PC/Ms included drawings which had been reviewed by Lead Engineers.
i 10 CFR 50, Appendix B, Criterion 111. Design Control. requires, in part, that l measures be established to ensure that the design basis is correctly translated into drawings and that design control measures provide for l verifying the adequacy of the design by individuals other than those who i performed the original design. The inspectors concluded that the licensee's !
- failure to. adequately implement the salient aspects of instruction 01 1.7 in this issue collectively represented a failure to satisfy Appendix B ]
requirements and, as such, represent an apparent violation (EE! ,
' 50 335/96-22-01
- Failure to Control Design Processes for Nuclear l Instrumentation Modifications").
FOLLOW PROCEDURES WHICH.' COLLECTIVELY. REPRESEtlT A BREAKDOWN IN THE 0A PROGRAM J OR IN THE DESIGN CONTROL PROCESS, The inspector concluded that the licensee's review of the hours spent (or lack thereof) in the review of the subject modification showed thoroughness in the self-assessment process. However, the inspector found that, in focusing on
"...the amount of time available [ emphasis added) to the verifying Engineers to perform verifications. ." the licensee failed to identify the larger issue of ensuring that verifications are not curtailed due to schedular constraints.
Installation Errors PUT WRITEUP FOR DETECTOR MISWIRINGS HERE Beacon Errors / Safety Significance Prior Opportunities to identify
- a. Scope-
. 15e inspectors. reviewed the subject event and activities preceeding its !
identification to determine whether the licensee had opportunities to identify j
2 the miswiring of NI drawers sooner.
- b. Findings The inspector reviewed 0A Audit Report OSL-PM 96-17, which included In PMON 96-052. a review of the licensee's performance with respect to design control for the subject PC/M. The 0A review concluded that the licensee's engineering, maintenance and QA organizations were not sufficiently sensitive to the warning signs of a breakdown of the quality program. Indicators of problems cited in the OA report included:
. Vendor difficulties in meeting fabrication schedules
. Loss of the project's Lead Engineer four months prior to implementation
. Installation difficulties traceable to erroneous design information provided to the vendor from the licensee
. Loss of the Instrument & Control (I&C) Supervisor responsible for implementation midway through the installation process
. Numerous noise problems encountered in wide range NI channels after installation
. At least two cases in which the new design provided improper output to the RPS
. A large volume of change paperwork wr'itten against the installation package (32 CRNs), a large number of scope changes to the installation work order (20), and a large number of deviations against the pre-operational test procedure (46).
The^ inspector reviewed the CRNs generated against the subject PC/M and found the following:
. The CRNs addressed problems in the areas of structural / installation.
vendor recommendations for perfonnance enhancement, correction of vendor document errors, design errors, and field wiring difficulties.
. A large number of the CRNs addressed themselves to the installation of the wide range detectors and noise problems associated with system startup.
. A number of the CRNs clearly indicated inaccuracies in the new design.
to include:
. Incorrect statements of the sizes of cables already installed in the plant which were to be employed with the new design
. Failures to change the ranges of control panel meters and strip chart recorders to accomodate the new range of the wide range
. detectors
. A failure to properly treat the RCS Low flow trip within the
e J
modification package.' which resulted in a constant trip signal from the RPS after installation The inspector also reviewed 13 Condition Reports (CRs) associated with the installation of this modification. As in the case of the CRNs. the CRs covered a broad area and addressed both wide range and linear range installation issues. The inspector identified two CRs of particular note:
- CR 96-1818, written on July 23 by I&C. questioced the operability status of Ni detectors'6 (RPS channel B) and 9. given their installation during the outage and the need to have the reactor at power to calibrate them. The resolution to the CR stated that the detectors were operable, but stated that a check for ASI agreement would be performed at 13% power to compare RPS channel B with A. C, and D. This resolution was performed by RE and the results of the' test will be deccribed below.
. CR 96-1358, written on June 11 by 0A, reported a possible loss of design control in the installation. The CR reported that multiple problems had been encountered during the installation of the modification, including the loss of key personnel, a work package which had become " voluminous and unwieldy" with *13 work package scope changes... approx 40 ' deviations
- to the Pre-Op pr'ocedure and also numerous changes to the Vendor Tech manual specifications.* The CR Condition Description concluded that * .with so many changes to the .
Tech Manual and Pre-Op procedure ano also the lack of personnel that are experienced in this particular modification. the possibility exists that the Design Control process could fail." The resolution to this CR prepared by I&C and Engineering, concluded that a loss of design control had not occurred, as a review of all associated paperwork indicated that appropriate reviews had been performed and approvals fo,r actions taken had been obtained correctly.
The inspector noted that the licensee's . root caus,e evaluation identified that several opportunities existed for identification of the miswired conditions during power ascension testing. On July 25, with'the reactor below 5%.
operators noted channel B to be indicating more bottom-peaked than the other channels.' This was explained by reactor engineering as expected behavior and the B channel was calibrated (forced to agree with incore data which resulted in B channel indicating similarly to channels A. C and D. Later the same day, with the reactor at 5.5% and Contiol Element Assemblies (CEAs) being withdrawn from 90 to 103 inches. ASI on channels A. C. and D were noted to trend toward the top of the core; however, channel B indicated ASI moving toward the bottom of the core. Operators questioned the indication and were told that B chammel, being a new detector. would require calibration at a higher power level.
WHAT ABOUT 13% COMPARISON?????
On July 28 while increasing power from 70% to 98%. operators again questioned a mismatch between channel B and A. C and D. RE responded that ASI was within TS limits and that a shape annealing factor (SAF) test was being .
performed on channel B. as the detector was new. to bring it into agreement with the balance of the channels. The disagreement was again identified on
o ,
5 July 30 by operators. That same day. RE identified the miswiring while analyzing SAF data.
The inspector found that there existed a clear opportunity to identify the miswired channels auring power ascension testing. Operators were successful in identifying anomolous behavior days before the ultimate identification, of the condition. While the licensee's root cause evaluation concluded that
- [ prior] to significant power levels it is difficult to detect the discrepancies in RPS ASI trends. " operational experience in this case indicates that the trend could have been (and was, although not in an integrated fashion) identified at relatively low power levels.
The inspector further concluded that a failure to resolve the first indications of channel disagreement in a formal, technically defensible way (i.e. testing performed specifically with the goal of establishing a basis for the channel-to-channel differences) delayed the identification. .
POSSIBLE ANGLES THE TOTALITY OF THE ISSUE - INDICAT10NS OF QUALITY PROBLEMS. FAILURE TOSHUT THE JOB DOWN. FAILURE TO ACT.ON OPERATOR IDENTIFICATION -
COLLEC11VELY REPRESENT A FAILURE TO IDENTIFY AND CORRECT A SIGNIFICANT CONDITION ADVERSE TO QUALITY (A WEAK VIOLATION - SAYS THEY SHOULD HAVE BEEN CLARV0YANT)
=
B CHANNEL CONSISTENTLY FAILED A CHANNEL CHECK FROM THE FIRST TIME OPERATORS IDENTIFIED THE CONDITION AND S0 WAS INOP PER TS. FAILURE TO PURSUE THE INOPERABILITY (AND POSSIBLY TO IDENTIFY THE REAL PROBLEM)
WAS A FAILURE TO IDENTIFY AND CORRECT (A WEIRD VIOLATION - SAYING THAT THE ONLY GOOD CHANNEL THEY HAD SHOULD HAVE BEEN DISCOUNTED AS BROKEN).
- CHANGE FROM CRITER10N XVI TO CRITERION 1. SAY THAT. SINCE 0A WAS FULLY AWARE OF THE PROBLEMS ASSOCIATED WITH THIS MOD. THAT THEY SHOULD HAVE SHUT THE JOB DOWN. CRITERION 1 SAYS THEY SHALL HAVE THE AUTHORITY T0
. IDENTIFY OUALITY PROBLEMS. TO INITIATE. 0R PROVIDE SOLUTIONS." AS SUCH. 0A SHOULD HAVE INITIATED THE SOLUTION OF A FULL DESIGN REVIEW AFTER THE CRs THEY WR0TE ( A WILD VIOLATION. BUT PROBABLY CLOSER TO THE REAL PROBLEM). 1 l
l'VE IDENTIFIED 8 CRs WRITTEN AGAINST ASPECTS OF THE NI JOB THATl IN MY l ESTIMATION. CONSTITUTED NONCONFORMANCES AS DEFINED IN THE 0A PLAN, BUT l
WHICH WEREN'T FLAGGED AS SUCH. FAILING TO DO SO RESULTED IN THEM i GETTING CLOSED OUT WITHOUT A FRG REVIEW. PERHAPS IF THE FRG HAD SEEN !
ALL THESE CRs. THEY MIGHT HAVE BEGUN TO QUESTION THE QUALITY OF THE JOB (ALTHOUGH I DOUBT IT). WHICH MIGHT HAVE LED TO DISCOVERY.
I THINK CRITER10N XVI 15 A REALLY WEAK APPROACH 10 THIS ISSUE. LIKE IT OR
. NOT, ALL THE BARRIERS THAT COULD HAVE PREVENTED IT WERE IN ENGINEERING *S HOUSE IN THE REVIEW AND POST-IMPLEMENTATION TESTING PROGRAM. WE COULD DING OA FOR l
. NOT SHUTTING THE JOB DOWN (AND I WOULDN'T MIND THAT'NECESSARILY, AS THEY STAND TO DEVELOP SOME CHARACTER). OR FIND SOME WAY TO CITE THE TREATMENT OF THE ,
OPERATOR CONCERNS. BUT I THINK WRITING T00 WILD A VIOLATION IS JUST GOING TO l DIVERT THEIR ATTENTION FROM WHAT'S IMPORTANT. AND WRITING ONE .WE EAT (LIKE THE l CEDMCSROOM50.59)JUSTCUZTHEYPANELSAYSTOGETSUSNOTHINGBUTTROUBLE. 1
SUGGEST GOING TO MANAGEMENT AND TELLING THEM XVI IS NOT THE WAY.
P 9
l l
i l
l i
1 l
.