ML20136E688
Text
r _ __ .
.g .
i N '
A From: Mark Millere 67 1 To: ATP1.CAC1 Date: 11/18/96 3:13am MASTOIE[ j
Subject:
St. Lucie NI inspection i
Attached is a writeup I started this weekend. It's got some background infor and more ,
detail on the issue, but if you want to use it, it'll require some editorial work. I've spent !
two days now pouring over old paper, and the bottom line is I think we're gonna look I unrealistic going in with Criterion XVI. Please give this a read. My opinions are at the j end. j Again, if you want to use this, John'll have to do some cut & pasting. ,
i 1
. . i 1
1
. l l
i i
As 8
9703130285 970306 PDR FOIA B INDER.9 6- 48,5 PDR ,
(_____
d 8 k
(Closed)URI 335/96-11-06 " Unit 1 NI Wiring Errors" Background on Unit'1 Nuclear Instrumentation The St. Lucie Unit 1 Nuclear Instrumentation (NI) System is designed to employ a total of 10 detectors, all located external to the reactor vessel, as described below:
Detector Detector Types Channel Purpose Associated RPS Trips Numbers 1.2,3.4 F1ssion Chamber W1de Range High Rate of Change Logarithmic Detection
- 5. 6. 7. 8 Uncompensated Linear Safety . Variable High Power lon Chamber Related e. Thermal Margin / Low Pressure Detection e Local Power Density 9, 10 Uncompensated Linear None lon Chamber Non-Safety Detection (control channels)
Detectors 5. 6. 7. 8. 9. ar.; 10. above. are designed with 2 axially distinct detectors each, forming sin 31e detector assemblies to allow for the detection of power fractions in the upper and lower axial halves of the core. The signals from the upper and lower halves of each of these detectors are summed to form a total power signal from each detector. The upper and lower halves are also combined electronically to develop an Axial Shape Index (ASI) signal consistent with the equation ASI - (lower Detector Power -' Uoner Detector Power)
(Lower Detector Power + Upper Detector Power)
ASI is used in the control of axial core power distribution by operators and to provide inputs to Thermal Margin / Low Pressure (TM/LP) and Local Power Density (LPD)
Reactor Protection System (RPS) trips. Each of the 4 linear safety related detectors )
feed an individual RPS channel and the RPS trips the reactor on a 2-out-of-4 l coincidence for a given trip (e.g. TM/LP). '
The purpose of the TM/LP trip is to prevent exceeding Departure from Nucelate Boiling ;
(DNB) limits and to provide a low pressure reactor trip in the event of a loss of l coolant accident. The trip setpoint itself is designed to be a reactor coolant I system (RCS) pressure limit, set at the higher of 1750 psia or a pressure setpoint calculated by the core protection calculator based upon RCS temperature conditions.
reactor power. and power distribution (expressed by the ASI signals generated by l detectors 5. 6. 7. and 8). The TM/LP trip is required per Technical Specifications (TS) Table 2.2-1. with setpoints adjusted to agree with Figures 2.2-3 and 2.2-4 of TS. TS Table 3.3-1 requires that at least 3 operable RPS channels provide this trip I function at or above 1% thermal power. '
1 The purpose of the LPD trip is to prevent peak local power density in the fuel from exceeding 21 kw/ft, thus assuring that the melting point of the fuel will not be reached during anticipated operational occurances. The trip is designed to be initiated whenever axial power offset, as indicated by ASI. exceeds either a high or a low calculated setpoint. The LPD trip is required per TS Table 2.2-1 with setpoints adjusted to agree with Figures 2.2-3 and 2.2-4 of TS. TS Table 3.3-1 requires that at least 3 operable RPS channels provide this trip function at or above 15% thermal power. '
Each linear NI detector assembly is cylindrical in design, with the two detectors located axially adjacent to one another. The signal cables for each detector assembly (which transmit output from both detectors) exit the assembly from one end of the cylinder. Each detector is installed in a cylindrical housing external to the reactor vessel in positions spaced radially about the vessel to' ensure that all quadrants of the core are monitored.
The signal cables for the detectors installed in Unit I had been labeled by the vendor as " TOP SIG" and BOT SIG." indicating output for each detector in the assembly. Th'is labeling scheme assumed that each a'ssembly was in' stalled in its housing oriented such that the signal cables exited the top of the assembly.
However, the St. Lucie design is such that the detector assemblies are installed with the signal cabling, exiting the assemblies from the bottom, making the " TOP SIG" signal cable correspond to the detector monitoring the bottom half of the core, and vice versa.
During the most recent Unit 1 outage the licensee elected to replace their existing I Ni circuitry With a new system developed cooperatively with Gamma-Metrics. The scope of the change included new NI drawers for all four RPS channels and new wide range detectors. During the outage, unrelated to this modification, uncompensated 1on :
chambers (UlCs) were replaced in RPS channel B (detector 6) and in the control I channel detector 9 location.
l Detection of Problem / Licensee Action On July 30. with Unit 1 at 100% power Reactor Engineering (RE) personnel were '
analyzing power ascension testing. data (from low power physics testing completed on j July 25) for the determination of Shape Annealing Factors (SAFs - which correlate axial power distribution as determined by incore detectors to ASI as indicated by '
excore detector channels). During the data review, it was determined that RPS channels A. C, and D and control channel 9 indicated ASI values which were trending in a manner which was opposite to that expected. In pursuing the issue. the licensee 1dentified a discrepancy involving wiring errors in the safety related linear NI. ,
channels. The errors resulted in channels A. C. and D reporting ASI values which were opposite of true values; that is, an apparent wiring error had reve.rsed upper and lower detector inputs to the NI drawers such that' ASI was misciculated.
At 1:00 p.m. the same day. Unit 1 operators were informed of these conditions and immediately declared the A. C. and D channels out-of-service (005), which placed the l unit in TS 3.0.3 due to 3 of 4.RPS channels for TMLP and LPD being inoper3ble. The l inspector responded to the control room and found thdt leads were being reversed on the A and C channels in an attempt to restore the channels to operability. Reactor
, Engineering support was available. With new NI gain values being calculated in
. support of I&C as the leads for the affected channels were properly aligned. At 1:50
.,; O .
I A , .
l and 2:00 p.m.. work was completed on the A and C channels, respectively, and, operators drcve CEAS into the core to verify proper ASI response. The inspector
~
verified that proper channel response occurred. and the A and C channels were - ;
declared operable at 2:00 p.m.. leading to an ex1t' from TS 3.0.3. '
. Overview of Deficiencies Leading to the Event 6 i
.In reviewing the issue, the licensee determined that all four safety related NI channels had been connected to the NI drawers in the RPS cabinets with the upper and ;
lower detector inputs reversed. These field errors resulted from errors in the associated Control Wiring Diagrams (CWDs); that is 'the detectors wer:e connected at the NI drawers in conformance with approved engineering drawings. The drawings were !
- prepared and approved with errors in the designation of connection points to the N!'
. drawers. The errant drawings were a part of the Plant Change / Modification (PC/M) i
. package which replaced the NI drawers during the most recer.t Unit 1 ' outage.
In addition to the wiring errors described above. the licensee determined that NI
+
detectors 6 (associated with RPS channel B) and 9 (one of two control channels) were-miswired at the detector ends (in containment), which resulted in RPS NI channel B indicating correctly (as the error at the detector end cancelled out with the error at the N1 drawer) and the control channel indicating incorrectly. The miswiring at !
the detectors was.the result of errors made by mairdenance personnel during installation. ,
Details of Deficiencies I As a result of the deficiencies described above, the licensee performed a root cause evaluation. The. inspectors reviewed the evalautionl discussed the events with the licensee and performed independent reviews of documentation associated with the NI modification. -The results of these activities are described below. ,
Errors Associated with CWD's
- a. Scope >
The licensee's root cause effort determined that, during the preparation of l t
PC/M 96-009 195. "RPS NI Drawer Replacement." a lack of available' vendor i information (in the form of-approved technical manuals and drawings) forced design personnel to develop drawings based on assumptions generally derived
- from a similar modification made to Unit 2 during the most recent Unit 2 outage. As'a result. CWDs JPN-009-195 002 (pro'iding v wiring instructions for 7 RPS channel A). JPN-009-195-004 (providing wiring instructions for RPS channel ;
B). JPN 009-195-006 (providing wiring instructions for RPS channel C), and JPN-009195-008 (providing wiring instructions for RPS channel D) were
' modified showing upper and lower' detector cable connections at each respective
] ' drawer which were in conflict with the internal wiring of the drawers which j were to be installed (i.e. field cables for the upper detectors were fed to the' drawers' lower inputs and vice versa).
- .Beyond the assumption of similarity between units. the licensee round that the
. verification process for the drawings in question contained weaknesses that resulted in a. failure to identify the condition. Specifically, the licensee ,
found that the drawings were prepared by a lead Designer (drafttman) based on
- inputs from the Lead Engineer (having responsibility for developing the ;
e i _
l
a J
modification package). The drawings were then checked by a second Designer, cons,tituting (essentiall; a drafting check. The drawings were then reviewed and approved by the Lead Engineer and approved by the Lead Engineer's Supervisor. -
The licensee found that this process differed from the approval process for the modification package. which was independently reviewed by a second engineer. However, the licensee pointed out that the package review did not include a point to-point, wiring check of the affected drawings. The licensee found that the method and sequence of verification described appeared to meet the requirements of 0! 1.7. Revision _ , " Design input /Verifi. cation." but that the method revealed a weakness of the process. .
While the subject drawings were approved without having approved vendor technical manuals (VTMs) available to support the review, the licensee had placed a hold on the PC/M completion pending receipt of the vendor documents Upon receipt, the VTH and other vendor documents were to be incorporated into the PC/M via Change Request Notification (CRN). However, when the documents were received, the licensee stated that they were not reviewed against other package drawings; thus, the wiring designation discrepancy was not, identified.
The licensee identified an additional weakness in the amount of time available to verifying engineers. Specifically, the licensee stated:
"The total U1 project man-hours through May 1996 was approximately 1150 man-hours. For scoping and scheduling purposes on a critical complex project, 30% of a projects [ sic) total man-hours should be allocated for verification :
activities. This should have accounted for a few hundred man-hours. The verifying Engineer on the Engineering Package charged a total of 40 man-hours to the project. 15% were on overtime. The verifying Engineer on the CRN j charged a total of 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> to the project. This relatively small amount of !
time spent on verification is due to project delays due to supporting unanticipated outages, the number of large projects being worked coincidently
[ sic] in PEG I&C. .and the available r sources.. "
The inspectors reviewed these licensee-generated findings for correctness.
- b. Findings The inspectors reviewed CWDs prepared with the PC/M package and found the licensee's con'clusions on the reasons for the field miswiring to be correct.
The inspectors noted that the licensee's conclusion that the drawings had received an inadequate independent verification to be correct; however, the ;
inspectors disagreed with the concicsion that the verification process was l performed met the requirements of the applicable Quality Instruction. Th.e !
inspectors reviewed Instruction 01 1.7, Revision _, " Design input / Verification." and found the following:
Step ,_ " Design Verification is the process whereby a competent !
individual, who has remained indeoendent of the design process, reviews the" design inputs. .and design output to verify design adequacy.
Step _ " REQUIREMENT FOR ENGINEER TO PERFORM REVIEW" i
4 Step _ " REQUIREMENT TO SAVE DESIGN INPUTS AS 0A RECORDS" The inspectors concluded that the subject 01 was of sufficient clarity to ensure an appropriate review. Specifically, the 01, when taken as a whole.
required that an engineer (not a Designer) perform an independent review o,f
, design output documents (e.g. drawings) to ensure agreement with design inputs (in the case of the subject PC/M. the Gamma-Metrics VTM). Contrary to these requirements, the licensee chose to approve the subject CWDs when no approved design input document was available against which the output could be reviewed and allowed the review to be performed without the required level of independence, further, when the approved design input was available. the licensee failed to perform necessary reviews to ensure agreement between vendor generated documsnts and existing licensee-generated drawings.
As a result of the 11censee's findings relative to a lack of independence in the verification process, the licensee performed a review of open PC/Ms to determine whether other examples existed. The licensee found that 3 out of 8 open PC/Ms included drawings which had been reviewed by Lead Engineers.
10 CFR 50. Appendix B. Criterion 111. Design Control, requires, in part, that measures be established to ensure that the design basis is correctly translated into drawings and that design control measures provide for verifying the adequacy of the design by individuals other than those who performed the original design. The inspectors concluded.that the licensee's failure to adequately implement the salient aspects of Instruction 01 1.7 in this issue collectively represented a failure to satisfy Appendix B requirements and, as such, represent an apparent violation (eel 50-335/96-22 01. " Failure to Control Design Processes for Nuclear Instrumentation Modifications").
MILLER'S COMMENTS - WE.SHOULD REALLY CONSIDER THIS AS'A SERIES OF FAILING TO FOLLOR PROCEDURES WHICH. COLLECTIVELY. REPRESENT A BREAKDOWN IN THE OA PROGRAM OR IN THE DESIGN CONTROL PROCESS.
The inspector, concluded that the licensee's review of the hours spent (or leck thereof) in the review of the subject modification showed thoroughness in the self-assessment process. However, the inspector found that, in focusing on
.the amount of time avaflable [ emphasis added] to the verifying Engineers 1 to perform verifications. " the licensee failed to identify the larger issue of ensuring that verifications are not curtailed due to schedular constraints.
Installation Errors PUT WRITEUP FOR DETECTOR MISWIRINGS HERE Beacon Errors / Safety Significance Prior Opportunities to identify
- a. Scope The inspectors reviewed the subject event and activities preceeding its identification to determine whether the licensee had opportunities to identify l
I 1
I
)
3 the' miswiring of NI drawers sooner. .
3
- b. Findings i
The inspector reviewed 0A Audit Report 05L-PM-96-17. which included in PHON ,
96 052, a review of the licensee *s performance with respect to' design c'ontrol !
for the subject PC/M. The 0A review concluded that the licensee's i engineering, maintenance and 0A organizations were not sufficiently sensitive ,
- to the warning signs of a break? vn of the quality program. Indicators of l problems cited in'the QA report.1ncluded:
e Vendor difficulties in meeting fabrication schedules i e Loss of the project's Lead Engineer four months prior to implementation
{
e Installation difficulties traceable to erroneous design information provided to the vendor from the licensee i e Loss of the Instrument .& Control (l&C) Supervisor responsible for ;
implementation midway through the installation process 't
- Numerous' noise problems encountered in wide range NI channel's after i installation . '!
I e At least two cases in which the new des'ign provided improper output to !
the RPS !
e A .large volume of change paperwork written against the installation !
package (32 CRNs), a large number of scope changes to the installation .
work order (20), and a large number of deviations against the [
pre operatio,nal test procedure (46).
The' inspector reviewed the CRNs generated agairist the subject PC/M and found ,
the following:
e The CRNs addressed problems in the areas of structural / installation. ;
vendor recommendations'for performance enhancement correction of vendor document errors,- design errors, and field wiring difficulties.
I e A large number of the CRNs addressed themselves. to the installation of {
the wide range detectors and noise probl'ms e associated with system j startup.
l 1
e A number of the CRNs clearly indicated inaccuracies in the new design
, to include e incorrect statements of the sizes of cables already installed in the plant which were to be employed with the new design
. - Failures to change the ranges of control panel meters and strip chart recorders to accomodate the new range of the wide range
. detectors e A failure to properly treat the RCS Low Flow trip within the 4.
w u-v < r >vm. w s-- - -,n... + ,e4 - e .-e.
, s ,
l modification package.'which resulted in a constant trip signal I from the RPS after installation The inspector also reviewed 13 Condition Reports (CRs) associated with the ,
installation of this modification. As in the case of the CRNs the CRs I covered a broad area and addressed both wide range and linear range installation issues. The inspector identified two CRs of particular note:
- CR 96-1818, written on July 23 by I&C. Questioned the operability l status of N1 detectors 6 (RPS channel B) and 9 given their l installation during the outage and the need to have the reactor at power to calibrate them. The resolution to the CR stated that the i detectors were operable, but stated that a check for ASI agreement would be performed at 13% power to compare RPS channel B with A. C. and D. This resolution was performed by RE and the results of the test will be described below.
e CR 96-1358, written on June 11 by 0A reported a possible loss of design control in the installation. The CR reported that multiple '
problems had been encountered during the installation of the modification. including the loss of key personnel, a work package which l had become " voluminous and unwieldy" with "13 work package scope i changes. . approx 40 ' deviations' to the Pre-Op prdcedure, and also j numerous changes to.the Vendor Tech manual specifications." The CR Condition Description concluded that "...with so many changes to the Tech Manual and Pre-Op procedure and also the lack of personnel that are experienced in this particular modification, the possibility exists that the Design Control' process could fail." The resolution to this l CR prepared by I&C and Engineering, concluded that a loss of design control had not occurred, as a review of all associated paperwork indicated that appropriate reviews had been performed and approvals for actions taken had been obtained correctly.
The inspector noted that the licensee's root cause evaluation identified that several opportunities existed for identification of the miswired conditions during power ascension testing. On July 25. with the reactor below 5%.
operators noted channel B to be indicating more bottom-peaked than the other channels. .This was explained by reactor engineering as expected behavior and the B channel was calibrated (forced to agree with incore data. which resulted in B channel indicating similarly to channels A. C. and D. Later the same day, with the reactor at 5.5% and Control Element Assemblies (CEAs) being ,
withdrawn from 90 to 103 inches. ASI on channels A. C. and D were noted to trend toward the top of the core: however, channei B indicated ASI moving toward the bottom of the core. Operators questioned the indication and were told that B chammel, being a new detector, would require calibration at a higher power level.
WHAT ABOUT 13% COMPARISON?????
On July 28, while increasing power from 70% to 98%.' operators again questioned a mismatch between channel B and A. C. and D. RE responded that ASI was within TS limits and that a shape annealing factor (SAF) test was being .
performed on channel.B. as the detector was new. to bring it into agreement w1th the balance of the channels. The disagreement was again identified on
_ - . _ , ~ _ _ _ .- _ _ _ _ . _ _ . - -. .
July 30 by operators. That same day. RE identified the misviring while j analyzing SAF data.
The. inspector.found that there existed a clear opportunity to identify the l
, miswired channels during power ascension testing. Operators were successful L in identifying anomolous behavior days before the ultimate identification of i
the condition. While the licensee's root cause evaluation concluded that' i "[ prior] to significant power levels it is difficult to detect the 1
discrepancies in RPS ASI trends.. " operational experience in this case l indicates that the trend could have been,(and was, although not in an' ;
j integrated fashion) identified at relatively low power levels.
The inspector further concluded that a failure to resolve the first [
indications of channel disagreement in a formal. technically defensible way i (i.e. testing performed specifically with the goal of establishing a basis .for
. the channel-to-channel differences) delayed the identification. . !
F POSSIBLE ANGLES i e THE TOTALITY OF THE ISSUE - INDICATIONS OF QUALITY PROBLEMS, FAILURE
- TOSHUT THE JOB DOWN. FAILURE TO ACT ON OPERATOR IDENTIFICATION -
COLLECTIVELY REPRESENT A FAILURE TO IDENTIFY AND CORRECT A SIGNIFICANT
] CONDITION ADVERSE TO QUALITY (A WEAK VIOLATION - SAYS THEY SHOULD HAVE BEEN CLARVOYANT)
- B CHANNEL CONSISTENTLY FAILED A CHANNEL CHECK FR'OM THE FIRST TIME OPERATORS IDENTIFIED THE CONDITION AND 50 WAS INOP'PER TS. FAILURE TO
- PURSUE THE INOPERABILITY.(AND POSSIBLY TO IDENTIFY THE REAL PROBLEM)
WAS A FAILURE TO IDENTIFY AND CORRECT (A WEIRD VIOLATION - SAYING THAT i l THE ONLY GOOD CHANNEL THEY HAD SHOULD HAVE BEEN DISCOUNTED AS BROKEN). ;
. CHANGE FROM CRITERION XVI TO CRITERION 1. 'SAY THAT, SINCE 0A WAS FULLY AWARE OF THE PROBLEMS ASSOCIATED WITH THIS M00. THAT THEY SHOULD HAVE I l
SHUT THE. JOB DOWN. CRITERION I SAYS THEY SHALL HAVE.THE AUTHORITY T0 i
" ..!DENTIFY QUALITY PROBLEMS: T0 INITIATE. .OR PROVIDE SOLUTIONS." AS l SUCH, 0A SHOULD HAVE INITIATED THE SOLUTION OF A FULL DESIGN REVIEW AFTER THE CRs THEY WROTE f A WILD VIOLATION, BUT PROBABLY CLOSER TO THE REAL PROBLEM).
I'VE IDENTIFIED B CRs WRITTEN AGAINST ASPECTS OF THE NI JOB THAT IN MY ESTIMATION. CONSTITUTED NONCONFORMANCES AS DEFINED IN THE 0A PLAN, BUT WHICH WEREN'T FLAGGED AS SUCH. FAILING TO DO SO RESULTED IN THEM GETTING CLOSED OUT WITHOUT A FRG REVIEW. PERHAPS IF THE FRG HAD SEEN ALL THESE CRs, THEY MIGHT HAVE BEGUN TO QUESTION THE QUAllTY OF THE JOB (ALTHOUGH I DOUBT IT) WHICH MIGHT HAVE LED TO DISCOVERY.
I THINK CRITERION XVI IS A REALLY WEAK APPROACH TO THIS ISSUE. LIKE IT OR NOT. ALL THE BARRIERS THAT COULD HAVE PREVENTED IT WERE IN ENGINEERING'S HOUSE IN THE REVIEW AND POST-IMPLEMENTATION TESTING PROGRAM. WE COULD DING OA FOR NOT SHUTTING THE JOB DOWN (AND 1 WOULDN'T MIND THAT NECESSARILY. AS THEY STAND i TO DEVELOP SOME CHARACTER) OR FIND SOME WAY TO CITE THE TREATM.ENT OF THE l OPERATOR CONCERNS. BUT I-THINK WRITING T00 WILD A VIOLATION IS JUST GOING TO ;
, _ DIVERT THEIR ATTENTION FROM WHAT'S IMPORTANT. AND WRITING ONE,WE EAT (LIKE THE I CEDMCS ROOM 50.59) JUST CUZ THEY, PANEL SAYS TO GETS US NOTHING BUT TROUBLE. I 1
)
I
~
F"""T
~
<h' h
h' SUGGEST G0ING TO MANAGEMENT AND TELLING THEM XVI IS NOT THE WAY.
P b
1
, e e
s
.I i
l i
l i
i 8
i I
'I I
l j
)