IR 05000293/1997002
| ML20217Q021 | |
| Person / Time | |
|---|---|
| Site: | Pilgrim |
| Issue date: | 08/15/1997 |
| From: | Conte R NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | Boulette E BOSTON EDISON CO. |
| References | |
| 50-293-97-02, 50-293-97-2, NUDOCS 9708290112 | |
| Download: ML20217Q021 (3) | |
Text
9 k
,
August 15,1997 E. Thomas Boulette, PhD Senior Vice President - Nuclear Boston Edison Company Pilgrim Nuclear Power Station 600 Rocky Hill Road Plymouth, Massachusetts 02360-5599 SUBJECT: INSPECTION REPORT 50-293/97-02
Dear Mr. Boulette:
This letter refers to your June 20,1997 and July 11,1997 letters which responded to a Notice of Violation concerning two examples of inadequate electrical design control as documented :n NRC Inspection Report 50 293/97-02, dated May 22,1997.
Thank you for informing us of your corrective actions. We have reviewed this matter in accordance with NRC Inspection Manual Chapter 92903, " Follow-Up - Engineering." Your corrective actions taken and planned seem appropriate to address the design control inadequacies experienced with the 480/120 voltage regulating transformers and MO-130153, RCIC r.ystem full flow test valve. The corrective actions will be inspected during future inspections. We do note that the unexpected shutdown of the voltage regulating transformers, which were installed in 1992, occurred previously due to improper control of breaker trip settings. Both inadvertent shutdowns of the voltage regulating transformers placed the plant outside of the plant's design basis and revaaled not only inadequacies with electrical design control, but also inadequacies with pre-operational testing.
In light of your response and corrective actions on programmatic control of vendor interfaces being weak, the NRC plans to conduct an increased inspection focus at your facility. This increase would include a review of the licensing and design bases information.
Your continued cooperation with us is appreciated.
Sincerely, Original Signed By: i Richard J. Conte, Chief Projects Branch 8
Division of Reactor Projects i i
Docket No. 50-293 License No. DPR-35 oGq0E8
.v ffh.h.lf lf 9708290112 970815 PDR G
ADOCK 05000293 PDR l f'- Ol
)
.-. __ _ _ __. _. ._ _ _ _ _ __ __ _ _ _ __ _ _ _ .. _ __
l
.
, l E. Thomas' Boulette, PhD
.
, I cc w/o cv of Licensee's Resoonse Letter L. Olivier, Vice President - Nuclear and Station Director T. Sullivan, Plant Department Manager N. Desmond, Regulatory Relations D. Tarantino, Nuclear Information Manager
< cc w/cv of Licensee's Resoonse Letter:
R. Hallisey,' Department of Public Health, Commonwealth of Massachusetts The Honorable Therese Murray The Honorable Joseph Gallitano B. Abbanat, Department of Public Utilities Chairman, Plymouth Board of Selectmen Chairman, Duxbury Board of Selectmen Chairman, Nuclear Matters Committee Plymouth Civil Defense Director P. Gromer, Massachusetts Secretary of Energy Resources J. Miller, Senior issues Manager i J. Fleming A. Nogee, MASSPIRG Office of the Commissioner, Massachusetts Department of Environmental Quality Engineering Office of the Attorney General, Commonwealth of Massachusetts
- T. Rapone, Massachusetts Executive Office of Public Safety Chairman, Citizens Urging Responsible Energy Commonwealth of Massachusetts, SLO Designee
,
$
e e
. - .
- e-
'. _
,
E. Thomas Boulette, PhD 3
'
Distribution w/cv of Licensee's Resoonse:
Region i Docket Room (with concurrences)
PUBLIC Nuclear Safety Information Center (NSIC) '
NRC Resident Inspector R. Conte, DRP M. Conner, DRP C. O'Daniell, DRP P. Milano, NRR A. Wang, NRR W.' Dean, OEDO R. Correia, NRR F. Talbot, NRR
'
DOCDESK
' Inspection Program Branch, NRR (IPAS)
DOCUMENT NAME: g:\ branch 8\pg\nov.res T2 vece4ve e sepy of this document,inacate in the boa: 'C' = Copy without attachinent/ enclosure *E' = Copy with attachment / enclosure 'N' = No copy
'
OFFICE. Al/DRP , , /. / l NAME RConte . /ff)/
DATE 08/14/97 7- 08/ /97 08/ /97 08/ /97 08/ 17 OFFICIAL RECORD COPY
!
l
______________ _ __ _ _ _ -.
,-4 .
,
'
.
e so = -~.
Pdgren Nuclear Power station
. Rocky Hdi Road Ptymouth. Massachusetts 02360 July 11,- 1997 BECo Ltr. 2.97.073 Senior Vice PresKlent - Nuclear
- U.S. Nuclear Regulatory Commission Document Control Desk Washington, D.C. 20555 License DRP-35 Docket 50 293 SUBJECT: SUPPLEMENTAL REPLY TO NOTICE OF VIOLATION (REFERENCE NRC INSPECTION REPORT NO. 50-293/97-02)
Enclosed is Boston Edison Cornpany's supplemental reply to the Notice of Violation contained in the subject inspectior report.
In the initial reply to the Notice of Violation (97-02-02), documented in BECo letter 2.97.065 dated June 20,1997, we committed to submit a supplemental response to address vendor interface issues by July 11,1997. This letter describes the status of actions taken or ongoing to address ver. dor interface issues.
The following commitment is made in this letter:
. Develop criteria that identifies categories of vendor interface (e.g., specialty skills, proprietary information used for analyses or calculations, atypical or prototype design) that will require
'
additional controis. Create a checklist or matrix for determining additional controls when special vendor oversight / interface is required. This commitment will be completed by ,
December 31,1997.
Please do not hesitate to contact me if there are any questions regarding the enclosed reply, tLL E.T. Boulette, PhD KRD/dmc/297073 Enclosure 1: Supplemental Reply to Notice of Violation i
Co203eay aCy._ l
. _ - - . _ - . . . . . - . . -. . . ... . . - - -._. . - .
.,
4.-.
,
.
cc: Mr. Alan B. Wang, Project Manager ,.
Project Directorate 1-3 Office of Nuclear Reactor Regulation
- Mail Stop: - OWF1482 -
U. S. Nuclear Regulatory Commission 1 White Flint North 11555 Rockville Pike -
Rockville, MD 20852 U.S. Nuclear Regulatory Commission Region l '
.475 Allendale Road King of Prussia, PA 19406 -
Senior Resident inspector d
.
1
%
.
, - - - - . , , , . . - , , , ,
. ,-r- , _ _ - ___ _____ -
- - *
. i
-
l
Boston Edison Docket No. 50-293 l Pilgrim License No. DPR-35 Enclosure 1 Supplemental Reply to Notice of Violation The cover letter of inspection Report 97-02 states, in part, "... additional problems with the interface / oversight of vendor activities were observed during RFO #11 including rework of the ECCS suction strainer slip joint clearances and the physical damage to the main steam line plugs that occurred when lowering the steam separator into the vessel. Both examples involved over-reliance on vendor communications. Also, continuing problems were experienced with RWCU pump mechanical seals. The adequacy of the interface with vendors was documented in NRC Inspection Report No. 50-293/95-22."
Adeauacy of Interface with Vendors Documented in IR 95-22/lFI 95-22-02 In NRC Inspection Report 95-22, an inspector follow item, IFl 95-22-02, was opened conceming vendor services. The inspector considered the area of root cause analysis of supplier / vendor-related problem reports to be a potential program weakness, and ownership of related corrective action items (related to vendor controls) regarding measures to preclude problem recurrence was not clear.
Since inspection Report 95-22 was issued, significant changes were made in the Problem Report (PR) program to strengthen the root cause analysis process. Fundamental to the change is the classification of a problem report into one of two levels, significant conditions adverse to quality (SCAQ) or non-SCAQ.
A SCAO level problem undergoes a detailed root cause analysis. The group-level manager responsible for the analysis approves the cause and corrective action to preclude recurrence.
Corrective actions to preclude recurrence must meet 3 criteria to be considered valid:
. The problem would not have occurred had the noted cause not been present.
. The problem will not recur due to the same cause if the noted cause is corrected or eliminated.
- Correction or elimination of the noted cause will prevent recurrence of similar conditions which could occur due to the same cause.
In addition to group-level manager approval, a root cause analysis and its corrective actions must be approved by the Corrective Action Review Board (CARB). The CARB reviews all PR root cause analyses for technical content, accuracy, adequacy of all corrective actions, and administrative completeness / acceptability.
If a vendor-related problem is designated as a SCAQ, then the cause and corrective action undergoes the analysis and approval process. Therefore, ownership of corrective action items related to vendor controls regarding measures to preclude recurrence is clear and within the framework of the corrective action process.
. - . .- -- . - ---- . -- - - . - . . - - -.._...._.-.- - - - --
. O_-4 , 3
-
.
. Discussion !
Historical vendor interface issues including these examples served as a basis for developing -j
'
preliminary criteria regarding vendor oversight / interface:
-* - ECCS Suction Strainer Slip Joint . t e Main Steam Line Plugs e~ - Regulating Transformers -
-
. . Fuel Shutdown Margin
There are two general types of vendors / contractor support: augmentation and specialty. The . -
. L work force must be augmented for maintenance, projects and engineering because of the '
increased workload and shift coverage during outages. - In these cases, we are able to monitor -
performance and its effectiveness because we possess the expertise ourselves.- This has not -
proved to be an area of significant concem relative to vendor interface. .
There is, also, heavy reliance on contractors who possess, for example: specialty skills and
,
. knowledge, proprietary information used for analyses or calculations, unique modifications and/or
~
,
' new designs (prototype). j
~
-
.
_.
ln some cases, our oversight did nM identify the critical design and implementation criteria. As a - ;
result of this insight, preliminary attributes have been identified regarding the issue of vendor .
' I oversight and vendor interface.- These attributes, although preliminary, serve to determine -
whether or not a vendor is a specialty contractor. The attributes include: -
i
- . - - Is the task routine or a one time task i. . . Does the vendor possess specialty knowledge and skills ?
e is the vendor's information proprietary ?
. Is time a factor ?
Once these attributes are developed, we plan to create a matrix that flags extra measures or .
precautions to be taken depending on the attributes of the specialty service being considered.
,
Preliminary examples include:
e Require third party review e- Independent verification .
. .
Enhanced management oversight of the implementation
. -- Augmented quality oversight
,
e identify critical design and implementation criteria-e . Augmented quality inspection of vendor design process n(
- Also, a common cause analysis is being conducted for problems identified during RFO #11.
When this letter was prepared,' the common cause analysis had not been completed. The -
results of the common cause analysis will be reviewed regarding vendor oversight / interface to detemiine if additional adjustments to the identified corrective actions are necessary.
I Summarv ,
'
-
- In recognition of vendor interface / oversight as an area for improvement, we are planning the
'
following action independent of on-going root cause or common cause analyses:
c i Develop criteria that identifies categories of vendor interface _(e.g., specialty skills, proprietary
, information used for analyses or calculations, atypical or prototype design) that will require
. additional controls.- Create a checklist or matrix of additional controls when special vendor -
< oversight / interface is required; .
P
"
,
-..
A- $ w- ee ,,my-tr Y Y 1 * -
_ _ - _ _ _ _ _ . _
..
..
.
.
e (. .
In addition,-we will complete the following as part of our established corrective action process: i e Review the results of the RFO #11 common cause analysis regarding vendor oversight / interface issues. The focus of the review is to determine if additional actions or adjustments are necessary for vendor oversight / interface.
. Complete the root cause analysis for the ECCS suction strainer slip joint clearances and implement corrective actions identified.
,
.
. . _ - - _ _ _ _ _
. _ . _ _ - _ - _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ __.
.. ..
..
.
. ,- -
1.- . . .
..
somon sm Pdgern Nuclear Power station Rocky Hdt Road -
Plyrnouth, Massachusetts 02360 L T. M ue, M June 20,1997 senior vce President - Nuclear g U. S. Nuclear Regulatory Commission Document Control Desk Washington, DC 20555 License DPR 35 Docket 50-293 SUBJECT: REPLY TO NOTICE OF VIOLATION (REFERENCE NRC INSPECTION REPORT NO,50-293/97 02)
Enclosed is Bosten Edison Company's reply to the Notice of Violation contained in the subject inspection report.
The following commitments are made in this letter:
. Specification E-15A and regulating transformers documentation (vendor manual V-1184) will be revised to reflect PDC 97-11 as part of the routine modification close-out process by August 31,1997,
. The electrical engineering department will update the design guide to enhance the Equipment Codes and Standards section by June 30,1997,
. The specification procedure will be revised to require identification of all new component automatic functions by July 31,1997,
. The. human performance aspects of the cause of the deficiency in specification E15A which were addressed via the corrective action process (PR 97.9245) will be discussed in the electrical engineering department training by August 31,1997.
. The vendor interface issues associated with this violation will be addressed in our -
supplemental response due by July 11,1997.
The NRC requested inspection report (IR 97-02) that Boston Edison staff specifically address
. actions planned to strengthen the oversight and interface of vendor activities. This response will be included in a supplement to this violation response by July 11,1997.
4D%2(,03l$ ?lyp
. . . .
- _ - _ . . . _ ._ _. _ _ . _ . . __ _. _._._ .
. .
,
. .
Please do not hesitate to contact me if there are any questions regarding th enclosed reply.
!
.
.
,A J --
.' T. Boulette, PhD KRD/dmc/vio97-02 Enclosure 1: Reply to Notice of Violation cc: Mr. Alan Wang, Project Manager Project Directorate 1-3 Office of Nuclear Reactor '
Mall Stop: OWF 1482 U. S. Nuclear Regulatory Commission 1 White Flint North 11555 Rockville Pike Rockville, MD 20852 U. S. Nuclear Regulatory Commission Region 475 Allendale Road King of Prussia, PA 19406 Senior NRC Resident inspector Pilgrim Nuclear Power Station
.
. , _ . , -
.n.m --
. .
,
4. 4 e
Boston Edison Docket No. 50-293 Pilgrim License No. DPR-35 Enclosure 1 Reply to Notice of Violation
'As a result of an inspection conducted from March 3 to April 28,1997, the following violation of NRC requirements was identified (VIO 50-293/97-02-01). In accorda'nce with the NRC Enforcement Policy (60 FR 34381; June 30,1995), the violation is described below:
ViolatioA 10CFR Part 50, Appendix B, Criterion Ill, Design Control, requires that applicable regulatory requirements and design basis for those structures, systems, and components to which this appendix applies are correctly translated into specifications, drawings, procedures, and instructions.
1. Contrary to the above, the function of the 480/120 voltage regulating transformers was not correctly translated into specifications which became evident during a significant storm on April 1,1997, when the running SSW and RBCCW pumps tripped and isolations of the RWCU system and RBIS occurred with the resultant start of the SBGT system. The transformers were procured and installed by engineering specification E-51 A and PDC 91-51 A and automatically tumed off due to an unknown vendor installed undervoltage design feature.
2. Contrary to the above, the function of MO-130153 (RCIC system full flow test valve) was not correctly translated into drawings when the valve functioned in the seal in close mode on April 17,1997, causing an inadvertent overspeed/ trip and increased unavailability time of the RCIC system turbine. Electrical design drawing mig 27 did not reflect an earlier modification made at an indeterminate date which converted MO-1301-53 to jog open/ jog close.
The above failures adversely affected the performance of safety related equipment and unnecessarily challenged the operators.
This is a Severity Level IV violation (Supplement IV)."
.
.
,
. .
,
.
REASON FOR THE VIOLATION Example No. 1 - 480/120 Voltaae Reoulatina Transformers The reason for the violation cited as example No.1 was that purchase specification (E15A) did not specify operation during voltage transients. A contributing cause was a vendor-installed undervoltage trip which came with the delivered transformers without the knowledge of Boston Edison. This feature in the delivered product was not captured in the equipment documentation.
-
Discussion:
The regulating transformers were installed in 1992 via design modifications per PDC 9159A and PDC 9159B and associated safety evaluation nos. 2664,2685, and 2706. Transformers X55, X56, X57, and X58 were designed te regulate input voltages of 480 volts +/- 20% (i.e.,576 384 volts) and provide regulated output voltages of 120VAC +/ 4%. Each regulating transformer contains a programmable microprr,cessor control unit (MCU) that senses input voltage and selects the proper voltago t::p to provide the regulated 120 volt output voltage. The MCUs were programmed by the manufacturer to shut down the regulating transformer if loput voltage was outside the transformer's input voltage range of 576 - 384 volts. The input voltage range of +/-
20% was based on the worst case voltage dips that would be seen during design basis accidents concurrent with a loss of offsite power. Transient voltages below 384VAC due to 345KV grid disturbances were not considered.
During a severe storm on April 1,1997, safeguard panels Y3 and Y4 :ost power due to the automatic shut down of voltage regulating transformers X55 and X56. After each shut down, the operators took manual action to open and reclose the input breakers to the transformers that reset the transformers and restored power to Y3 and Y4, At the same time, regulating transformer X58, which powers the "B' train of the post accident sampling system (PASS), shut down and was reset in the same fashion. Transformer X57 was tagged out of service for maintenance during the storm.
The transformers shut down due to a transient undervoltage condition. This condition was identified when a review of the 480VAC voltage level showed a voltage drop to approximately 350VAC during the storm.
The design documentation (vendor manual, etc.) provided by the manufacturer and supplier of the transfomiers did not identify an automatic shut down feature if input voltage was less than 380 volts. The lack of the identification of the feature is significant because an automatic shut down due to input voltages from zero volta to 384 volts would require a manual reset of the transfomier versus a designed automatic reset if input voltage was zero volts. Specification E15A did not request that the transformers continue operation or prohibit shutdown during voltages outside the design regulation range.
The cause for the automatic shut downs of the regulating transformers was a deficiency in specification E15A which did not address the effects of 480 volt transients of less than 384 volts.
The apparent cause of the deficiency was an unintentional cognitive error made by the utility electrical engineer who prepared the specification. Nuclear engineering procedure 3.08,
' Specifications and Reports", govems the preparation and issuance of specifications. There were no unusual electrical engineering work location characteristics that contributed to the error. The
. - - - - - .
- ,
,
,- .
,
.
human performance aspects of the cause of the deficiency in specification E15A was addressed via the corrective action process (PR 97.9245) which concluded that the problem report would be discussed in the electrical engineering department training. ,
Example No.2 - MO 130153 wirino discrepancy
. The reason for the violation cited as example No. 2 was lack of design control at a previous time in plant operation due to elementary and connection tirawings not reflecting a change that made MO-130153 a jog close valve.
.
Discussion:
During RCIC testing on April 17,1997, MO 1301-53 operated as a seal-in close valve instead of operating as a jog valve in both open/close directions. A new bucket for MO-130153 was installed under PDC 93-38 during RFO #11. The new bucket was wired in accordance with the existing connection and elementary drawings per the design basis. These documents referenced MO-130153 as a jog open/ seal-in close valve. The wiring in the old bucket was then investigated. It was found that a wiring change made to an auxiliary relay in the bucket changed the function from a seal in close valve to a jog close valve. This change was not captured in previous drawing revisions. A search of the maintenance request (MR) database also revealed this change was not captured on a MR. It is, therefore, concluded that the change was made at cn indeterminate date possibly dating back to pre 1972 startup testing.
The undocumented wiring change did not result in a failure of valve MO-130153.
CORRECTIVE STEPS TAKEN AND RESULTS ACHIEVED Example No.1 The microprocessor control units (MCUs) for the four regulating transformers were modified via a design change (PDC 97-11) on April 12,1997. The modification disabled the undervoltage and cvervoltage shut down functions. This allows the transformers to operate in the unregulated mode when the input voltage is outside the design range. Post work tests were performed to ensure the regulating transformers would continue to operate during simulated undervoltage and overvoltage conditions. During the tests, the highest and lowest transformer taps were selected without a shut down of the respective regulating transformer. Based on the satisfactory results of this testing, the regulating transformers were declared operable.
Problem report, PR 97.1658, was written to document that the safety evaluation 2664 performed to allow the installation of the regulating transformers did not evaluate the consequences of an undervoltage transient shut down. The problem report evaluation concluded that the undervoltage shutdown was not considered because the vendor had not notified Boston Edison about the transformer's undervoltage trip, and specification E15A did not ensure conti.1ued operation of the transformers during voltage transients. The evaluation also concluded that the procedures to write specifications and safety evaluations are adequate; however, the specification procedure will be enhanced to require identification of all new component automatic functions. The electrical engineering department design guide will be also be updated to enhance the Equipment Codes and Standards section.
e 5
_ . . . --
_ ._ _ . . _ _ _ . _ _ _ _ _ . _ _ _ _ _ . . _ _ _ _ _ _ _ _ . _ _ .
.. .
,
.- .
..
Example No. 2 An engineering design change (FRN 93-38-21) was issued on April 18,1997, to change valve
-
'
- MO 130153 to a jog open/close valve. The valve was then stroked (jogged in the open and close directions) with satisfactory results. The document changed the closing circuit from a seal-in type circuit to a jog type circuit. The affected drawings will be revised as part of the routine modification close-out process. With this change, the control circuitry for MO 130153 functions as a jog circuit in the open and close directions. The change did not affect the automatic closing
'
function of valve MO 130153.
A review was conducted of the other breakers replaced under PDC 93 38 focusing on those that power motor operated valves that contain a jog control switch circuit. The review concluded the buckets were wired in accordance with design, and the undocumented wiring change was an isolated instance that could date back to initial start-up testing (circa 1972). Current procedures
,
and work practices require approved design changes for wiring changes and documenting wiring discrepancies in accordance with the problem report process.
- CORRECTIVE STEPS THAT WILL BE TAKEN TO AVOID FURTHER VIOLATIONS Example No.1 Specification E-15A and regulating transformers documentation (vendor manual V-1184) will be
,
revised to reflect PDC 97-11 as part of the routine modification close-out process by August 31, d
1997.
! The specification procedure will be enhanced to require identification of all new component automatic functions by July 31,1997.
The electrical engineering department will update the design guide to enhance the Electrical Codes and Standards and section by June 30,1997.
.
l The human performance aspects of the cause of the deficiency in specification E15A, which were addressed via the corrective action process (PR 97,9245), will be discussed in the electrical engineering department training by August 31,1997.
The vendor interface issues associated with this violation will be addressed in our supplemental
response due by July 11,1997.
Example No. 2 Our current procedures and work practices require approved design changes for wiring changes and documenting wiring discrepancies in accordance with the problem report process. No further changes are required to prevent recurrence.
. - . . . - - _ - . . - - . . _ -- - - - - - ,-
_ .. ._. . _ _ _ . . _ _
,
.- .
,
- . .s_-
-
+
..
. DATE WHEN FULL COMPLIANCE WILL BE ACHIEVED Example No.-1-
. Full compliance will be achieved by August 31,1997, following implementation of the last open '
corrective action. These commitments are being tracked under RC 97.0023.
Example No. 2 ,
Full compliance was achieved when MO-130153 was rewired and tested on April 18,1997.
REFERENCES 1) LER 97-007 * Safeguards Buses De energized and Losses of Off site Power during -
. Severe Storm while Shut Down", Boston Edison Letter #2,97.050 dated May 1,1997.
,
2) LER 97 009 "RCIC System inoperable due toTurbine Overspeed Trip during Surveillance",
Boston Edison Letter #2.97.056 dated May 19,1997.
.-,
,
!
l
,
!
!
l i
,
7 l