ML20057C747
| ML20057C747 | |
| Person / Time | |
|---|---|
| Site: | Millstone  |
| Issue date: | 09/22/1993 |
| From: | Doerflein L NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML20057C746 | List: |
| References | |
| 50-336-93-18, FACA, NUDOCS 9309290338 | |
| Download: ML20057C747 (18) | |
See also: IR 05000336/1993018
Text
.
.
i
U.S. NUCLEAR REGULATORY COMMISSION
.
REGION I
i
i
E
Docket No.:
50-336
5
Report No.:
93-18
i
License No.:
Licensee:
Northeast Nuclear Energy Company
P. O. Box 270
Hartford, CT 06141-0270
Facility:
Millstone Nuclear Power Station, Unit 2
Inspection at:
Waterford, CT
Dates:
August 2,1993 - September 17, 1993
Inspectors:
P. D. Swetland, Senior Resident Inspector
D. A. Dempsey, Resident Inspector, Unit 2
Approved by:
W /.
Oerv
E
S !R4 !9 3
lawrence T. Doerflein, Chief /
Date
Reactor Projects Section No. 4A, DRP
Scope: Special inspection of the repair activities associated with a body-to-bonnet leak on a
letdown system manualisolation valve.
Results: See Executive Summary.
930%290338 930922
E
ADOCK 05000336 i
<
G
iCL
j
l
. -
-
.
.
.
Executive Summary
Plant Ooerations
On August 5,1993, Unit 2 was shutdown and depressurized due to reactor coolant system
(RCS) leakage greater than the technical specification limit of one gallon per minute. The
RCS leakage increased suddenly during maintenance efforts to seal a body-to-bonnet leak on
a nonisolable letdown system isolation valve. The repair activity was suspended and
,
operators effectively implemented the abnormal operating procedures to promptly reach and
maintain stable shutdown conditions.
'
Maintenance
The leak sealing activities were not conducted with sufficient control and oversight
commensurate with the safety consequences of a failure of this RCS boundary valve.
Numerous instances of inadequate instructions and failure to follow procedures and
instructions occurred. As a result, the letdown valve was subjected to a continuing series of
improper drilling and excessive peening activities which degraded the overall integrity of the
valve. After more than 30 sealant injections, one of four bonnet studs failed causing the
RCS leakage to increase to nearly five gallons per minute.
Engineering
Engineering evaluation of the ongoing leak sealing activities was limited to the requested
justification of nonconforming conditions and of followup leak sealing strategies. The effects
of the continuing valve degradation were not fully known or considered in these evaluations.
The licensee failed to conduct a written safety evaluation addressing the changes to the valve
which were implemented by the leak sealing strategy. The precise cause of the valve stud
failure remained unresolved at the end of the inspection.
Safety Assessment /Ouality Verification
,
,
The continuing leak sealing activities on this nonisolable valve, based on narrowly focused
evaluations and poorly supervised implementation, revealed a nonconservative safety
approach to this repair. In particular, management tolerated the necessity for repeated
j
injections and the escalating valve degradation, and did not consider sufficiently the shutdown
l
alternative. Management failed to effectively use the quality assurance program to assure the
l
correct implementation of the critical repair activities.
l
ii
i
l
'
)
. . _ _ - _
>
l
.
,
i
TABLE OF CONTENTS
'
r
Page
1.0
INTROD UCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I
2.0
BACKGROUND....................................
I
i
3.0
SYSTEM / COMPONENT DESCRIPTION . . . . . . . . . . . . . . . . . . . .
1
4.0
MAINTENANCE ACTIVITIES
3
..........................
1
5.0
ENGINEERING AND TECHNICAL SUPPORT . . . . . . . . . . . . . . . . 6
,
,
6.0
QUALITY VERIFICATION
8
............................
7.0
MANAGEMENT INVOLVEMENT . . . . . . . . . . . . . . . . . . . . . . . . 9
8.0
OVERALL CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
9.0
M AN A G EM ENT M EETING . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
l
l
l
l
l
!
i
1
iii
<
.
DETAILS
1.0
INTRODUCTION
A special inspection was performed of the circumstances regarding licensee repair activities
for a nonisolable reactor coolant leak from letdown system isolation valve 2-CH-442.
Excessive leakage from this valve resulted in a forced shutdown of Millstone Unit 2 on
August 5,1993. The inspection consisted of observations of repair activities, interviews with
licensee personnel, and review of documents, and focused on the following licensee
activities:
- Implementation of the licensee's repair plan
- Adequacy of engineering and technical support
l
Management oversight of repair activities
- Licensee oversight of vendor activities
Quality Assurance Program performance
A detailed chronology of events is provided in Attachment I to this report.
j
2.0
BACKGROUND
-
l
On August 5,1993, in an ongoing attempt to stop reactor coolant system (RCS) leakage from
i
the body-to-bonnet flange joint (split line) of letdown system isolation valve 2-CH-442, the
licensee injected sealant into the valve gasket area. (The licensee repeatedly had been
peening the split line area and injecting scalant into the gasket area since June 1993.
l
Leakage would stop for some period of time and then return as sealant was expelled from the
'
split line area.) This injection was performed in preparation for welding a clamp across the
split line of the valve in order to retain the sealant within the split line area. While injecting
sealant and peening the flange area, the valve bonnet appeared to lift slightly from the valve
body, significantly increasing the leakage rate and forcing abandonment of the repair. The
licensee determined that the one gallon per minute limit for unidentified reactor coolant
system leakage (Technical Specification 3.4.6.2) had been exceeded, and at 2:39 p.m., a
l
reactor shutdown was commenced. The licensee declared an Unusual Event in accordance
with its emergency plan and notified the NRC. The unit was placed in the cold shutdown
l
condition (Mode 5) at 9:33 a.m., on August 6, reducing RCS leakage to less than one gallon
per minute. The licensee terminated the Unusual Event at 9:52 a.m.
3.0
SYSTEM / COMPONENT DESCRIIrrlON
Valve 2-CH-442 is a 2-inch, normally-open, manual gate valve used to isolate the letdown
portion of the chemical volume and control system from RCS loop "2B" for maintenance and
for local leak rate testing of system containment isolation valves. It is the first valve down
stream of the RCS penetration, and is located in the containment building at an elevation
approximately four inches below the centerline of the RCS hot leg. The valve is listed in
Table 4.3-12, " Active and inactive Valves In The Reactor Coolant System Boundary," of the
l
Millstone 2 Final Safety Analysis Report (FSAR). It is a Velan 1500#, model MO8-3543B-
l
{
!
I
'
,
,
2
13MS, with design and operating pressure and temperature ratings of 2485 psig/650 degrees
Fahrenneit ( F) and 2235 psig/550"F, respectively. The body and bonnet material is ASTM
A-182 Type 316 stainless steel. The four,3/4"-10UNC-2A body-to-bonnet studs are ASTM
A-461 Type 630, and the nuts are ASTM A-194 Grade 8M stainless steel.
Valve 2-CH-442 was disassembled for sett leakage repair in November 1992. During
reassembly, the bonnet was installed 90 degrees from its normal position on the valve body.
While having no functional impact on valve performance, this error subsequently complicated
the task of fabricating and installing a sealant injection clamp around the valve. On
May 24,1992, when the very small body-to-bonnet leak was identified by the licensee, the
unit was in the hot standby condition (Mode 3) following a reactor trip during main
condenser thermal backwashing. The inspector learned that the licensee had considered a
'
plant cooldown to repair the leak at this time, but had decided to perform a temporary leak
repair instead due, in part, to concerns regarding the effect of the cooldown on the
performance of the mechanical seal on the "D" reactor coolant pump. Plant startup
commenced on May 25 and full power was achieved on May 27. The licensee planned to
perform the temporary repair by injecting sealant directly into the valve gasket area through
fittings drilled and threaded into the split line; to design, fabricate, and install an external
sealant-retaining clamp; and, in the long term, either to permanently repair or replace the
valve during the next regularly scheduled outage. From June 4 to August 5 the valve was
injected with sealant with varying degrees of success. Two leak repair vendors were
involved in this activity under the direction of unit maintenance engineering. The valve was
injected approximately 30 times using an estimated 0.63 gallons of sealant before the
shutdown on August 5.
Valve 2-CH-442 was cut out of the system on August 8 and moved to the reactor building
,
'
for disassembly and inspection under the direction of a licensee investigation team. The
inspector witnessed portions of this activity, examined the disassembled valve, and reviewed
the licensee's findings. The valve body and bonnet had been peened excessively. The entire
body-to-bonnet split line of the valve had upset metal. There was aui errant chisel groove
j
starting one-half inch below the split line at the southwest corner stud which extended
j
approximately 1-1/2 inches across the south face of the body. The groove had been peened
'
partially, and was 1/8 to 3/16 inches deep. Three of the four injection fittings adjacent to the
studs on the east and west sides of the valve were drilled past the outer dia.neter of the
i
gasket and well into the gasket area. Sealant had penetrated all four stud holes. The valve
showed no signs of damage from steam cutting, or injection of sealant into the process How
areas (i.e., no mainlining). All four of the studs had thread damage from peemng of the
split line. The southwest corner stud *o had been damaged slightly when an injection
fitting was drilled too close to the
aole. This stud had completely sheared in a brittle
manner two threads below the surfaw of the b* flange. The southeast corner stud also
appeared to be bent slightly and had been dm M by drilling during the first attempt to
install a valve clamp.
.
.
3
'
The licensee sent the valve and studs to an independent laboratory for detailed examination
i
and failure analysis. Pending the results of this analysis, the stud failure mechanism is
unresolved (336/93-18-01).
4.0
MAINTENANCE ACTIVITIES
i
The leak repairs performed on valve 2-CH-442 were performed under automated work orders
(AWOs) issued on June 4 (M2-93-07225), June 10 (M2-93-07864), and June 11
(M2-93-07939). The repairs were to be implemented in accordance with instructions
contained in nonconformance reports 293-090,293-091 and 293-094, respectively. Each
AWO package contained four documents providing guidance for the repair activities: a
nonconformance report (NCR) disposition, Maintenance Procedure MP-2721M, " Leak
Sealing Procedure," Station Form SF-365, " Checklist For ' Injection' Repair Engineering
Evaluation," and engineering memoranda referenced by the preceding documents. The
inspector reviewed the AWO packages to identify the quality attributes needed to assure the
valve's structural integrity and to prevent injecting sealant into the RCS All of the AWOs
included the following guidance:
Injection fittings to be drilled between stud holes to assure structural integrity of
e
flange load carrying sections (two AWOs contained . etches showing load path areas
to be avoided)
Injection fittings to be drilled in non-pressure retaining section of the bonnet flange;
)
i.e., do not drill past the outer gasket diameter
Injected sealant volume to be limited to avoid mainlining into the RCS
lejection pressure to be selected such that ASME Code allowable stress limits are not
e
exceeded for the limiting valve component
Peening of entire circumference of the valve is prohibited
The inspector noted that no limits were placed upon the number of times the valve could be
injected; that no quantitative guidance was provided regarding proximity of the fittings to the
valve studs; and that no cautt
or prohibition regarding peening of the valve near the studs
existed. The inspector cono
%t these deficiencies were contrary to Step 6.3.1 of
Procedure ACP-QA-2.02C, '
-
f rders," which requires that all procedures and forms
,
referenced in AWO packages be reviewed to ensure that adequate guidance is provided for
work to be performed. As a result, the valve was injected about 30 times; the fittings were
installed very close to the studs, damaging one stud which ultimately sheared; and all four
studs were damaged by peening. In addition, contrary to Procedure MP-2721M, the entire
circumference of the valve was peened, and three out of four of the injection holes were
drilled past the outer diameter of the gasket and into the pressure retaining sectica of the
bonnet flange. The inspector also noted tha; the peening of valve surfaces was excessive and
not controlled by licensee personnel as evidenced their inability to determine when and how
the chisel groove on the body surface occurred.
l
l
!
l
t
_-
.
.
i
4
l
The inspector determined that from June 4 to July 23, the only quality control (QC) hold
point listed in the inspection plans (Form SF-207) was to verify sealant injection pressure.
The inspector reviewed licensee administrative procedures to determine requirements for QC
hold points. Procedure ACP-QA-2.02C, Step 6.3.4, requires the identification of quality
attribums from design documents, procedures, codes, specifications, and standards; and
acceptance criteria against which success / failure of the inspection may bejudged. Procedure
ACP-QA-3.33, " Performance of Reviews For The Use Of ' Epoxy / Rubber' and ' Injection'
Type Repair Materials," Section 5.3, requires the job supervisor to ensure that special
conditions specified in checklist SF-365 are implemented in the AWO inspection plan. Three
of the five attributes (highlighted items) listed above were contained in Form SF-365, but
,
were not included in the AWO inspection plans.
l
The inspector concluded that instructions to assure proper iustallation of the injection fittings
and to prevent damage to the valve studs had been inadequate; that the procedure and
i
instruction steps contained in the AWO packages necessary to assure a quality repair of valve
2-CH-442 had not been implemented; and that the inspection plans did not contain sufficient
'
hold points to assure proper implementation of the repair activity. This is an apparent
violation of licensee procedures and of 10 CFR 50, Ap,endix B, Criterion V, " Instructions,
Procedures, and Drawings," which states, in part, that activities affecting quality shall be
prescribed by documented instructions and procedures; that they shall be accomplished in
accordance with these instructions and procedures; and that appropriate acceptance criteria
shall be provided for determining that important activities have been satisfactorily
accomplished (336/93-18-02).
The inspector determined through interviews that maintenance personnel had been aware on
June 4, that one injection fitting may have been drilled past the outer diameter of the gasket.
This nonconforming condition was not formally documented or dispositioned. The inspector
also found that on June 10, licensee personnel identified that all four injection valves had
been installed into the valve load transfer path contrary to the guidance contained in the
AWO package. The nonconforming condition was resolved as acceptable per a three-way
memorandum, dated June 10. Nonetheless, in both cases, failure to initiate an NCR when
nonconforming conditions were identified during the performance of work is contrary to
Step 6.6.1.10 of Procedure ACP-QA-2.02C and Step 6.1.1 of Procedure ACP-QA-5.01,
" Nonconforming Materials and Parts," and is also an apparent violation of 10 CFR 50,
Appendix B, Criterion V, which requires that activities be conducted in accordance with the
documentea procedures.
On June 12, an attempt was made to install an injection clamp onto the valve. This activity
was controlled by AWO M2-93-07940 which referenced the disposition to NCR 293-094,
dated June 11. The clamp installation included drilling of two holes per valve stud to inject
sealant into the stud holes. Two holes were drilled through the bonnet in the vicinity of the
southeast stud before it was determined that the clamp did not fit the valve. The inspector
found that the drilling was not documented in the AWO, and that the maintenance engineer
had not been aware that the holes had been drilled. In addition, the inspector learned that
- _ _
-
_
.-
_
._
_ _ _ _
__ __.
_.
'i
,
.
,
.
.
.
5
!
the AWO had not been authorized by the operations department prior to being performed.
l
Believing that no work actually had been performed under the AWO, it subsequently was
cancelled. The licensee subsequently daermined that the holes had been drilled too deeply,
in violation of the NCR disposition, and that the southeast stud had been damaged by the
drilling. Performance of work without authorization by operations and cancellation of an -
AWO under which work has been performed are violations, respectively, of Steps 6.5.8.2
and 6.18.1 of Procedure ACP-QA-2.02C, " Work Orders." The inspector concluded that this
activity was symptomatic of a breakdown in the licensee program for' control of work on
quality components, and was an additional example of the apparent violation of 10 CFR 50,
.
Appendix B, Criterion V.
{
The inspector noted during review of AWOs ano through discussions with licensee personnel
that reconstruction of work performed on valve 2-CH-442 was difficult because the AWOs
i
did not indicate clearly which fittings were injected (and how much sealant was used) on any
j
,
given occasion. Also, when and by how much the valve split line had been peened was not
l
always documented. In addition, the inspector noted several discrepancies regarding
i
implementation of Procedure ACP-QA-2.02C:
Purchase order and vendor names were not identified on two AWOs, contrary to
l
Steps 6.2.5 and 6.16.2.1
.
.
l
The " Surveillance Requested" block on three AWOs was not completed for vendor
i
services, contrary to Step 6.2.7.3
_
l
Light grinding, buffing, and a dye penetrant test were performed on the valve without
i
an operations authorization signature, contrary to Step 6.5.8.2
,
Commercial grade sealant was dedicated for use on the valve using an NCR, contrary
!
to Step 6.6.1.29
The department approval line of three AWOs was not signed by a vendor
!
l
representative, contrary to Step 6.16.3.6
_
The inspector concluded that while these items individually had little safety significance, they
l
indicated a significant pattern oflack of attention to detail by licensee personnel, and were
!
additional examples of the apparent violation of 10 CFR 50, Appendix B, Criterion V.
I
l
On June 15, the unit requested NUSCO to evaluate welding a sealant-retaining clamp onto
l
the valve. Between June 18 and July 7, the valve did not leak. The valve was reinjected on
l
July 8 and 9. On July 13, AWO M2-93-08812 was initiated to take dimensions on the valve
and to fabricate the clamp. The AWO was assigned a priority of "3",_which, per
ACP-QA-2.02C, is routine work to improve unit performance, availability, and operability
which should be scheduled and completed in a timely manner. The inspector noted that the
sealant injection AWO's had been assigned a priority of "2" and considered that the lower
priority assigned to the clamp work indicated the licensee's sense that the leakage was under .
.
.
.-
.-
.
1
l
6
control. The NUSCO evaluation was received by the unit on July 28, and reflected in the
disposition to NCR 293-113, dated August 3. The AWO to install the clamp was released on
August 5. The inspector concluded that the licensee did not approach installation of the
l
clamp with a sense of urgency commensurate with the consequences of a valve failure.
Through interviews with licensee personnel, the inspector determined that maintenance
supervisors often deferred to the expertise of the vendors and did not exercise direct technical
oversight of vendor activities at the job site. This resulted in the conditions adverse to
quality listed above such that the integrity of the valve was degraded severely, forcing a plant
shutdown. The inspector also noted that maintenance supervision and management failed to
initiate NCRs for nonconformances which occurred due to the failure to follow procedures
and instructions. The inspector acknowledged that a self-initiated investigation into the
performance of the first vendor resulted in their removal from the job on June 11. However,
a vendor NCR was not generated, and at the end of the inspection period the vendor
remained conditionally approved to perform safety-related work at Millstone. The inspector
I
also noted that maintenance management initined a debriefing of the job in early July to
determine whether the leak sealing activities could be enhanced. The recommendations
!
focused primarily on radiological and industrial safety considerations, as well as, personnel
'
protection and comfort. The nuclear safety consequences, nonisolable RCS leak, were not
addressed. Nevertheless, further injection of the valve continued and a comprehensive
reevaluation of the repair technique and the nuclear safety implications of the degrading
j
condition of the valve was not performed. The inspector noted that maintenance management
did not inspect the valve directly in the containment until August 2, at which time shutdown
l
of the unit was recommended. Although the degraded overall condition of the valve was
noted, licensee response to the recommendation focused on the engineering evaluation of an
apparent crack in the body of the valve near the errant chisel mark. Upon successful
disposition of that indication, licensee attention refocused on the repair activities, rather than
the relative safety implications of the degraded condition of, and continuing leakage from the
unisolable valve and the prudency of plant shutdown. The inspector concluded that
maintenance supervision and management were ineffective in ensuring quality repair activities
and appropriately considering the safety implications of continued sealant injection of an
unisolable RCS boundary valve.
1
5.0
ENGINEERING AND TECIINICAL SUPPORT
The inspector reviewed Form SF-365, the licensee checklist completed in accordance with
Procedure ACP-QA-3.33 to assure that evaluation of design considerations relevant to the
performance of leak sealing activities is performed. The form considered the physical
condition of the valve studs and nuts, chemical compatibility of sealant with system materials
I
and the reactor coolant, valve stress analysis, system evaluation to limit volume of sealant
injected, seismic evaluation of the valve including weight of sealant and injection fittings,
industrial and personnel safety concerns, radiation exposure, and retest requirements.
,
l
However, the inspector noted that some of the conclusions reached in the evaluation were
based on engineering judgement rather than rigorous analysis, and were not well documented
i
l
l
7
before the activity commenced. The inspwtor considered that this informality was not
commensurate with the safety signi6cance of a potential failure of this valve (small break
loss of coolant accident). The inspector also identified the following weaknesses in the
licensee's evaluation:
No root cause evaluation of the body-to-bonnet leak was performed prior to
perfcrming the leak repair
Stress calculations did not consider effects of peening the split line or the thermal
effects caused by repetitive isolation and restoration of letdown flow
No limits were established on the number of times the valve could be injected
Detailed guidance for the implementation of on-line leak sealing at nuclear power plants was
promulgated to the industry in July 1989 (EPRI NP-6523-D). This document provides
specific details of leak sealing techniques and specifies precautions needed to assure that leak
scaling activities are performed safely. The inspector found that the licensee was not
familiar with this document and that certain precautions listed were not incorporated in the
licensee's leak sea!ing procedures. In particular, the fact that improper peening can result in
excessive loading of flange bolts; that the leak sealing process constitutes a modification to
the plant; that reinjections should be limited and carefully evaluated; and that clamp-type
(enclosure) leak sealing requires careful assessment of the condition of the closure studs were
not adequately covered in Procedures MP 2721M or ACP-QA-3.33.
The inspector also identified engineering performance weaknesses during the leak repair
activities. Lack of engineering presence at the job site may have contributed to a lack of
appreciation for the degrading condition of the valve as repair activities continued to be
performed. The discrepancies which resulted from the lack of control over the repair
activities required engineering resources to be focused on justification of the repair
nonconformances rather than the accumulated effects of the repair and the continued
appropriateness of further injections.
Poor communication / support between the unit and NUSCO engineering staffs occurred on
several occasions. At site request, a calculation was performed which justified a significant
increase in the maximum sealant injection pressure. The calculation assumed that the
injection fittings would not be hr.talled at certain areas of the split line, near the studs. By
the time the calculation was forwarded to the site, the four fittings already had been installed
in the proscribed areas. Also, the fact that some of the injection fittings may have been
drilled past the outer diameter of the gasket was not communicated to NUSCO. On June 15,
the unit requested NUSCO to evaluate welding a sealant clamp onto the valve. Despite the
fact that direct injection of the valve continued to be unsuccessful, NUSCO's formal response
was not expedited and, therefore, not completed until late July.
,
i
__
.
.
.
8
The inspector questioned the licensee's position that the repair was a maintenance activity
rather than a modification to the valve, despite the istallation of injection fittings into the
split line, the proposed addition of an external clamp, peening of the split line, drilling into
j
the gasket area, and changes in stresses to the valve as a result of the leak sealing process.
The inspector considered that the changes to the valve required a safety evaluation pursuant
'
to 10 CFR 50.59. The licensee's failure to perform a safety evaluation of the changes to
valve 2-CH-442 is an apparent violation. (336/93-18-03).
The inspector concluded that lack of effective engineering involvement in field activities and
inadequate scope of engineering evaluations ofleak sealing activities contributed to an
ongoing decrease in the margin of safety designed into valve 2-CH-442.
6.0
QUALITY VERIFICATION
,
The inspector reviewed the Quality Services Department's (QSD) role with respect to the
valve 2-CH-442 repairs to assess program effectiveness. The inspector found that QSD
involvement consisted of inspection and surveillance activities. Regarding inspections,
Procedure ACP-QA-2.02C, Step 6.4, required quality control (QC) personnel to review and
approve AWO packages per Procedure QSD-4.05, " Review of Work Orders, Millstone
Station." Section 6.4 of Procedure QSD-4.05 states that inspection plans are a place to
document inspection attributes " agreed" to by QSD and the lead department head. The QSD
reviewer is directed to consider where deviations from expected results may occur and,
where independent verification would be desired to ensure that work performed on safety-
related systems meets code, technical specification, procedure, or good workmanship
practices. The step further indicates that hold points are negotiated by the parties involved in
the activity to be inspected.
As noted in Section 4.0 of this report, the inspection plans associated with valve 2-CH-442
were very limited and failed to include several important quality attributes. Lack of critical
technical review of the work activity was evident in the QSD reviewer's approval of the
inspection plan. Based on interviews, the inspector determined that the ongoing poor
,
maintenance practices were not focused on by the QC inspectors who attended these repairs.
This occurred because the assigned coverage was limited in scope and because unit
management had, in the past, discouraged inspectors from exceeding their assigned
inspection scope. The NRC concluded that the QC inspectors complied with the
requirements of the inspection plans and performed the tasks which they were expected to
perform. But, in so doing, the QSD failed to function effectively in assuring a quality repair
of the valve. The inspector also concluded that Procedure QSD-4.05 establishes a possible
,
j
framework for loss ofindependence of the quality assurance function in its apparent emphasis
j
on negotiation of hold points with the inspected department.
1
'
l
9
l
QSD surveillance of leak repair vendor services was required by Step 6.2.7.3 of Procedure
ACP-QA-2.02C. In addition, the purchase orders applicable to both vendors required
'
performance of surveillances to verify compliance with approved procedures. Procedure
MP-2721M, " Leak Sealing Procedure," was listed explicitly in the purchase orders. The
l
inspector reviewed the QSD surveillance reports associated with the valve work. Numerous
examples of procedure and instruction noncompliances were identified by the NRC in
Section 4.0 of this report. Nonetheless, the surveillance reports concluded that no procedural
j
I
or programmatic noncompliances were observed. Through discussions with QSD personnel
the inspector found that the surveillances were limited to inspection plan hold point
verification and general observation of radiological and industrial safety practices. Based on
I
the limited scope of observed activities, the inspector questioned the broad conclusions
contained in the surveillance reports, and was concerned that the reports could result in
j
unconditional approval by the licensee of marginal vendors. Also, to the extent that the
scope of the surveillances was driven by the AWO inspection plan developed by the lead
department, the independence of this important quality assurance function was potentially
diluted.
j
i
The inspector concluded that the QSD coverage of the work on valve 2-CH-442 was
l
ineffective in pieventing conditions adverse to quality in apparent violation of 10 CFR 50,
Appendix B, Criterion X, Inspections, which requires, in part, that a program for inspection
of activities affecting quality shall be executed to verify conformance with the documented
instructions / procedures, for accomplishing the activity. This is an apparent violation
i
(336/93-18-04). Based on the discussions with maintenance management and QSD
,
l
personnel, the inspector concluded that the level of QSD involvement in this repair had met
the QA program expectations. Therefore, the inspector concluded that this event represented
'
a potential breakdown in the licensee's quality assurance program.
7.0
MANAGEMENT INVOLVEMENT
,
1
The body-to-bonnet leak from valve 2-CH-442 was discovered on May 24 while the plant
was in the hot standby condition (Mode 3) following a reactor trip. Unit 2 management
considered cooling down to repair the valve, but decided to startup on May 25 based, in part,
on a concern to minimize perturbing the "D" reactor coolant pump seal and the confidence
that the sealant injection process had been used successfully at the unit in the past.
However, the specific application of leak seal techniques to this nonisolable RCS penetration
did not receive appropriate management focus and oversight commensurate with the
consequences of a valve failure. Fnm discovery of the leak until August 2, when a possible
l
through-wall leak was discovered o i the valve body, plant management relied primarily on
the ability to fabricate and install a sealant injection clamp to stop the leak. The number of
times that the valve was being injected and the deteriorating condition of the valve appeared
not to be a significant safety concern. The inspector noted that management oversight and
control of in-field activities were inadequate, and the plant operations review committee was
I
not significantly engaged in this process. As evidenced by the period between June 15 and
July 28, during which time the plant awaited NUSCO engineering evaluation of welding the
l
I
l
l
l
.
-
10
clamp to the valve body and bonnet, the inspector concluded that there was little sense of
urgency regarding fabrication / installation of the clamp. Not until a potential through-wall
leak was discovered near the split line of the valve was fabrication of the clamp placed on an
expedited schedule. The inspector concluded that the engineering calculations demonstrated
,
margins of safety for the valve. However, undermining the calculations was a lack of
j
appreciation regarding the degraded physical condition of the valve. The inspector concluded
that there was a lack of questioning attitude within the unit staff regarding the
j
appropriateness of continuing with the leak repair activities. Site and corporate management
showed an apparent willingness to tolerate the ongoing inability to stop the leak and the lack
of urgency in implementing the clamp installation. In addition, senior managers were
generally familiar with the ongoing difficulty in repairing this leak, but did not cause the unit
j
to appropriately reassess the safety significance of continuing the repair activities on this
'
nonisolable RCS penetration. The inspector concluded that the licensee's management
activities regarding this issue were not commensurate with the potential safety consequences
l
of a valve failure.
8.0
OVERALL CONCLUSION
unsuccessfully to perform a temporary leak seal repair to letdown system isolation valve
'
For over two months, with Unit 2 operating at full power, the licensee attempted
2-CH-442. The valve was not isolable from the reactor coolant system. Over the course of
the repairs, the valve was injected with sealant material over 30 times; the valve was
exclusively peened; the bonnet was incorrectly drilled in areas close to the studs; injection
holes were improperly drilled into the valve gasket area; and one or more of the studs were
damaged by the uncontrolled drilling and/or peening operations. On August 5,1993, one of
the four valve body-to-bonnet studs failed, forcing a shutdown of the plant. The final cause
of the stud failure remains unresolved.
Three apparent violations of licensee and NRC requirements were identified in the areas of
work implementation and control; safety review / engineering support of maintenance; and
quality verification activities. In addition, licensee management's willingness to tolerate
continuing failure to stop the leak; the apparent lack of urgency in fabrication of an injection
i
clamp; and the lack of direct oversight and control of field activities indicated a non-
conservative approach to this activity which was not commensurate with the potential safety
consequences of a valve failure.
l
1
l
1
I
1
.
.
.
11
9.0
MANAGEMENT MEETINGS
Following the inspection, an exit meeting was held on September 17, 1993, to discuss the
inspection findings and observations with station rnanagement. Licensee commerts
concerning the issues in this report were documented in the applicable report section. No
proprietary information was covered within the scope of the inspection. No written material
regarding the inspection findings was given to the licensee during the inspection.
.-
.
.
-
..
-
- - - - - . . - .
. - _ _
-__ -
._-
l
'
l
.
.
.
(
'
ATTACHMENT I
Valve 2-CII-442 Event Chronology
Reactor tripped during main condenser thermal
May 24
backwashing
Light wisp of steam identified at valve 2-CH-442 body-to-bonnet joint
(split line)
May 25
Reactor startup commenced
May 27
Unit 2 at full power
Maintenance engineering contacted valve vendor (Velan) regarding
availability of replacement valve
Small steam wisps from valve split line at the east and west sides of the
June 4
valve
Automated work order (AWO) M2-93-07225 released to perform
i
sealant injection repair - Leak Repair, Inc.
Installed first injection fitting and injected valve at the northwest (NW)
position - leak did not stop
Installed second injection fitting and injected valve at the southeast (SE)
l
position - leakage present at SE fitting
l
June 11
. AWO M2-93-07864 released to install additional fittings and to perform
,
sealant injection
l
Installed third and fourth injection fittings at the northeast (NE) and
l
southwest (SW) positions and injected valve - leak not stopped
Injected first two fittings - leak not stopped
'
Injected valve - leak slowed
June 12
c
Removed injection fittings to install injection clamp fabricated by 12ak
Repair, Inc. - clamp did not fit
Two holes drilled through bonnet at southeast (SE) stud per AWO
M2-93-07940.
Reinstalled fittings, peened split line, and injected valve - leak not
stopped
,
Leak Repair, Inc. replaced by Furmanite, Inc.
AWO M2-93-07939 released to perform leak repair of valve -
Furmanite, Inc.
Valve injected, but sealant extruded from split line
Attempted to inject valve, but sealant would not travel - leak rate 0.63
gallons per minute
Peened valve split line and injected - leak not stopped
Injected valve - leak stopped
Leak Repair, Inc. fitting at SE position fell out - leak restarted
j
June 13
Installed adapter and new fitting, and injected valve - leak rate 1 - 2
drops per minute from an injection fittmg
Injected valve - leak stopped
j
.
_
- .
.
-
-.
.
.
_- ._
.
9
.
.
Attachment 1
2
Velan provided quotation for replacement valve - 24 week lead time
June 14
Peened split line and injected valve - leak stopped
June 15
Closed circuit television installed in control room to monitor valve
leakage
Unit 2 engineering requested Northeast Utilities Service Company
(NUSCO) to perform evaluation of ASME Code and practicality of
proposal to weld bar stock on two sides of valve for clamping strength,
and to " seal weld" split line of remaining sides
Work commenced on design / fabrication of injection clamp - day shift
coverage
Injected valve - leak not stopped
June 18
Injected three fittings (NE, SE, SW positions) - leak stopped
Leakage restarted two hours later at SE fitting
Injected all four fittings - leak stopped
Velan informed maintenance engineering that one-fcr-one replacement
June 22
valve not available
Maintenance engineering initiated inquiries with valve vendors
June 24
regarding replacement valve
July 3
Leak restarts
Redrilled and injected four fittings - leak stopped
July 7
Tapped and installed adapter and new fitting at SW position and
July 8
,
injected valve - leak stopped
Leak restarted three hours later
Injected valve - leak stopped
i
e
leak restarted three hours later
Injected valve - leak not stopped
Reactor coolant system unidentified leakage 0.58 gallons per minute
July 9
Peened and injected valve - leak stopped
Conference call between licensee and NRC Region I regarding status of
valve
AWO M2-93-08812 released to obtain valve dimensions and fabricate
July 13
injection clamp
Maintenance engineering contacted Divesco for one-for-one
July 23
replacement valve from " cancelled plant" spares - no valve available -
alternate valves considered
Injected valve - leak stopped
Leak restarted - reinjected valve - leak stopped
Maintenance engineering contacted another valve vendor and obtained
July 26
list of replacement valves available
Licensee searched NPRDS for one-for-one spares at other utilities - no
butt weld Velan valves availabic
NUSCO in-stock inventory searched - no QA Class 1 replacement
l
valves available
.
l
.
i
!
.
.
Attachment 1
3
l
Peened and injected valve - leak stopped
August 1
Injected valve - leak stopped, then restarted
August 2
Injected valve - slight leakage
Furmanite personnel observed apparent linear indication / surface defect
and through wall leakage below split line on south face of valve body -
peened area and informed maintenance personnel
Maintenance manager, engineering, and quality services personnel
inspected valve and noted a chiseled groove (1 -1/2 inches long by 3/16
inch deep) on the valve body near the southwest body-to-bonnet stud,
1/2 inch below the split line - confirmed apparent through-wall leakage
'
Maintenance manager contacted Unit 2 DSEO and unit director and
recommended shutdown of the unit
Fabrication of valve injection clamp expedited -24-hour coverage
NUSCO stress analysis and materials engineers arrive on-site to inspect
August 3
and evaluate condition of valve - concluded that leakage originated
from pressurized stud hole rather than through-wall leakage
Started stress calculations to evaluate structural integrity of valve
assuming failure of one body-to-bonnet stud
Conference call conducted between licensee, NRC Office of Nuclear
Reactor Regulation (NRR), and NRC Region I regarding condition of
valve
Peened and injected valve through fittings at SW and NW positions -
leak stopped momentarily, then restarted at the SE position
Performed " informational" dye penetrant examination of suspect area of
valve body - results inconclusive
Second conference call conducted between licensee and NRC Region I
regarding dye penetrant test results
NUSCO stress analysis engineering completed reviews to allow
installation of socket welded replacement valve August 4
4
Peened and injected valve - leakage stopped
Furmanite maintained constant injection pressure on fitting at SW
position to facilitate formal dye penetrant examination - after light
grinding, examination showed no cracks in valve body
Fitting at SW position developed steam plume
Injected fittings at SW and NW positions
Replacement valve vendor forwards quotation for new valve
August 5
AWO M2-93-09431 released to weld injection clamp bar stock along
north and south split lines of valve
Peened and injected valve six times - Furmanite personnel observed
valve bonnet lift slightly from body flange and sudden increase in valve
leakage - repairs abandoned - reactor coolant system unidentified
leakage greater than 1.0 gallons per minute
Reactor shutdown commenced at 2:39 p.m.
-
..
-
-
-
.
_
. _ _ _ _ _ -
-
_ _ _
t
.
-
l
Fedmet Resister / val sr No sas_[_mder, Jgutrsass_/ weelene
as
,
-
,
l
'
L CWeede Far =ammentOpme
times estegates of NeessessesEbe[ . 'embfset is gamesmael amassing, that
"
a
t
Emmenanceme- e
-summai
me=a.
does. bema me.p
se.e me n erger
i
Enforceanut comissemose wanet be
"""r"*" 8"d other - wheek
then rhe puedeled,and ens
open to the nehmeif theadeseement
"8 e**'h8 d 8h* ****hhs 8Pym of
disruptisepemens msgr be memd.
'
I
asenoa ' W.a
licensees.
Each soglemet eEse estiloestimme to
i
in.1) w
be tahamassinet as
n. Anmeemme opas shamesoment
osedmet toadmessentesasereno.
(
widual,erif theaemen,theaghmet
Censmesses
pressedhylaamendemos weekrepenal
!
l
taken against an individual, turas en
precusa mendassementasederemoe
As soma amitle determined thatan
wGlcondone tobe a messiesbetween
whetheramindividualhas commented
enforcement seafesumoe we be open to
b NRCand he hommeos.Whus ik
m
l
.32) Involves eigdficant personnel
"8*'*'" ""8"****i' " I
dy b
&
public observetles, it le met apes for
failures where the NRChas requested
4
eram
eopen kpuhuc
puhuc participablem,
th:t theladividual(s) involved be
obeavenen se part of es agWe trial
Perseas altamengopen amlerosament
i
present at the analerance;
i
(3)le bened on the Andings of en NRC
g'g,F"
",,*", g'*,g*E8 coudereness ase samladed that (1) the
P'
""
g
appenet v6elettene diesessed at apen
i
OfBoe of Investigneans (OI) soport; or
(4) Involves enfagsamie saformaales.
e payan,
w
esbdm
esfemament osmimemens are esb6ect to
i
88 8 8" "
l
Priv cy Amtinformaties,orother
furtherspleerand may be emb6est to
i.
inforseatiam erhich could be considered
changeprter to anyseashing
so that the NRC can schedule an
enferoementacessand(2)ee
!
Enforesament onnierences levolvdag
no NECwelalas now appropr6ste
oplaica made byMRC employees at
appropnately sised confamme room.
,ing,,,,,, ,g ,g,,, ,,,,,,,,,g,,, og
,
i
medicalmisedministradone or
b
open enforcement am.miere.mose or the
overos.peserse wGIbe open asemains
!
has been
is* *me.< -e m.t to. ed .
. re.se .e be oo.dect.d
O
pan 2 @
represam.t. Anal deterad.astione er behefs.
!
wi.tho.ut dieciosing the = paa=d
obem v,dem. ,s de ,o ..._. .
m ad
m.
the esmacy's , grevi e.o.mmes e on
i
mv ai . .e.e. , ed.-
os
Cin
propenla essantance
j
enlareementosedmuseos wtB mot be
enforcessmed =8======= m es pWie
i
open to the peblie if the conferemme ws nonmally atleast to woridesdays k
wl* es endemoe k ein aseos,
e
j
be aaanhan *=d by telephone or the
advance of the enforceement confersnoe
- U'8 *P** *"I"""""
8
conference wGlbe oseducted et a
g, gag,,,,, %
wiu beprovided en opportasity to
2
i
relatively eenelllicemese's facihty.
(1)
poseedm es puhus
esbelt writtaa commente sammymonely
i
Finauy with theappreselof the
%%
to the reglemal alboa.These comments
l
Execettre Direeterfor Operations-
(2)Tou.4eeenlaphans messagnet and
will =
==r===atre be forwenied to the
j
colorcement conferenceswillnotbe
(3) Toll-free electronic beMeem heard
Director of the OSes of Emlermement for
j
open to the peblicto specialcasse
tweew and consideretloa.
where good ammes hee basa ahoses aber
establishesentof thetoR4ee
Deled at Seahalle MEL this rik day of July
i
balanctog the besattof pebBc
message eyetems.the pubtle sesy can
tema.
2
observates ognimet the pasandmilmpest (301) 4ED erst to obtain a recordlag of
For the NedserItaquineery e--
}
on the agency's salmenement actios ta a upconhgspea enderoament
seminei).mies.
perticular case.
conferesses.DeNRCwElissueeastbar secreeary (ge --
e
!
The NRC wGl strive to condect opes
Fedemi Regleter modes after es tou4ee gyR Dos,es seeme Fund y-o et aies W
enforcement conferences dartas the
mesessesyneemsare.=emaasa a
,
two year armipseyesin nocordamse
To eastet esISC h maldag
with the fousenes thme goals:
sporapr6ste arrangements to support
(1)A,
Mas percent of aR
pubhe observesenofemissammet
II754
.
i
elipble s'alorc==ame conferences
cominnassa.ineviduais =====nedta
a
!
conducted by theNRC wiu be open for
attamens e pareenter endercoment
j
pubue m o m en:
. m rumes m e id .o e e s m ov e st
Correctl0n3. pm.i n#m,
(2) Atleast oss open maleroomset
identISedlato moodus motlee
corference will be condmetod ia ensk of ammemadog he spea enforcement
~ y ,L sy* N.,23e
the reponal othces: and
. conferesse ao letar them Eve busiases
Pr6 der. July v. tes:
(3) Open enfora====t conferemens
days petar w the enioressment
will be conducted with a veristy at the
comieremon.
I
types of bconesa.
naaras samt fuggut.ATOftY
i
To evoid potentialbiae la aba,
15.Cememe of Open h
C00m
Camiumsw
I
siloccan process and to attempt to meet
j
tha three goals stated ebeve.evet,v
In eonardamse with eestent proctles.
Thee Year Titd proyesa ter
,
i
fourth elephie enfesamment comiereme
enforcement conferences will contemme
Censlucibis Open baseeroament
to aa== manl be held at the NRC. "'
Confomneces poesy statement
i
involving one of three categories d
F
]
licensees wGiamreemDyhe open to the
ofBees. Members of the pubbe wilbe
Carroesias
pubhc dertug the ertal preyes.
allowed ecomes to the NRC regional
4
!
liowever,in cases where there is en
ofBees to amend open amieressenet
la notice h es-1sts beginning
j
ongting edtabcatory proceeding with
conferences in accordemos with the
on pass 3Gret in the issue of Friday,
i
one or more intervanore, enforessment
" Standard Operettag proosderes For
July,1st test, on page 307e2. In the
conferenose tavolving tesues related to
Provishag Seaertty Sopport For NRC
second ochman, onder maTen beginning
the subject matterof the ongoeg
He '
And Meetmse" published
la tbs fifth line " July 11.19er" should
-
edfudication may also be opened.For
No
,1,teet (se FR 56 51).These
reed "}ely 11.190e".
the purposes of tide trialproyam, the
procedures prov6de that visitors may be
enuessansissee e
<
i
4
I
!
4
. _ . . - - - , _
m
.
-m.
.-- . _ - - - . ,
,
.
. n
._
- -- _ - - . - - - .
.-
. - _ - - .- - __
f
~
s
.O_
j
x,>
gX ;' x L1..(-g
z.y 3 , a y.
~
e
c
.,
i
-
J
~
w.w
.
g
Soaretary of due f%--a==a== U
-
U
a#
j
Nodier Reedstory r%==a==a==
J
Wasidenten.DCases. AT!N: , *7
j
Dedeems and Servies Bremsk.
j
Hand dekseresaments toc Oss White
Filot North, usee ReakviMe 70s.
j
Rockvius,he betereasy:45 a.sa.to 4:15
p.m. Federal weddays.
!
Copies of esamente may be examined .
l
at the NRC PuhusDocument Room.212e
- .
LStmet.NW.(lawer Level).
W.shington.Dc
!
nonnemassessommunemoonraer:
.
'
1
James Lishesses. Director. OfRee of '
f
Enferoement. U.S. Nonlear Regelstory
1
j
Comsdanism.Wa=W=gn== DC30655
!
(301-406 aret).
4
i
sissensamuraseseemaavisen
r
'
!
,
no NRC's ennentpoucy on
,
enforossist osoferences is addressed in
Secties V of the latest revision to the
l
"GeneralStatessent of " and
,
precedme forEsionament cuons."
-
(Referoament puber)toCFR part 2.
,
!
r.
Cthatusespubhehadon
F
1a,133B(SF FR SFW1).no
Esferossent
states that.
"eeforcement
willnot
nonostly be open to the public."
I
However,theComenseios has decided
.
toimplement a tetet proyam to
i
j
deterates arbeterto malatain the
i
j
current poterorth regard to
i
j
enfore====* emedessenes or to adopt a
7tse-YearTrid Prepass9er
new policy that weeklallow most
enforcenset osoferences to be open to
4
CondueWigOpen Gefereement
attendemos by au meenbers of the public.
!
CentemneensPoesyStatement
Peter Statenest
!
AGeseW5Nunteer Regalatory
i
Casemiesten,
pg,,ues
!
no NRCle =r3====*aag a two-yeat
a
- """
trialproyen to anow public
stamaanetHeNasiner Regalatory
observetten of selected enforcement
l
Commissism OSCHeissuingele poker coulemaans.He NRC willmonitor the
i
statement en the 5=y6-==a=*a== et a
pngma anddeteredse whether to
i
twi> year trial proyees to anser selected estabheb a peemeneet poucy for
'
esforcement acaderences to be open to
condeeling
seterosenant
attendeses by au messebers of the
confersease
en an assosoment of
generet pubhc.nle pokey statement
tim fouewing aherle:
describes the twi> year trial pseynes
(1)Whater the fact that the
4
andteienes the pubhc of hour to get
confamoseoneopenimpacted the .. . ,
)
infonestlesenupoeming opes
NRC's abEtty to condmet a meaningful ,
j
enforcassent conferences.
confuseos and/or =pI===at the NRC's
n
l
paTeetnis that proyenis e5setive os
7"8""
July 10.1983.w!dle comumente en the
(2)Wheder the spea conference
!
propean ese beise received.Sebedt
impacted the housese's participation in
i{
commente en er betere to compiedes
the conduescu
of the trial prepas scheduled for July
(3)Whether the NRC expended e
I
11.1988. Comments received after tide
l
dets will be caseidered if it le practical
- Wa==* emeestof resources m
i
to do ee.but the ra==== ion is able to
making the osederence public: and
i
i
ensure comenderation only for c8nunents
(4)The ame==e of public interese d'
received on or before this date.
opening the enforcement conference.
'
'
b
.
4
.
-
_
.
.
. .
. . . .