IR 05000387/1994023
| ML17164A460 | |
| Person / Time | |
|---|---|
| Site: | Susquehanna  |
| Issue date: | 11/30/1994 |
| From: | Greca A, Ruland W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML17164A459 | List: |
| References | |
| 50-387-94-23, 50-388-94-24, NUDOCS 9412090269 | |
| Download: ML17164A460 (17) | |
Text
0 DOCKET/REPORT NOS:
LICENSEE:
FACILITY:
DATES:
U. S.
NUCLEAR REGULATORY COMMISSION
REGION I
50-387/94-23 50-388/94-24 Pennsylvania Power and Light Company Allentown, Pennsylvania Susquehanna Steam Electric Station October 24-28, 1994 INSPECTOR:
nie o
e a
reca, r.
eactor ngineer Electrical Section Division of Reactor Safety a e APPROVED BY:
xaam
.
u an
,
ie Electrical Section Division of Reactor Safety
// '$Q ate d:
A di p ti,t dt i
tl pttility f actions taken by Pennsylvania Power and Light Company (PPSL) in response to NRC Bulletin 90-01, Supplement 1, "Loss of Fill-Oil in Transmitters Manufactured by Rosemount."
This inspection was performed in accordance with the guidance of NRC Inspection Manual Temporary Instruction 2515/122,
"Evaluation of Rosemount Pressure Transmitter Performance and Licensee-Enhanced Surveillance Programs."
The scope of this inspection included a
review of the enhanced monitoring program for Model 1153B/D Rosemount transmitters manufactured prior to July 11, 1989. Other areas reviewed included:
the disposition of stored transmitters, the Rosemount transmitter calibration procedure, the criteria used to identify transmitter failures caused by oil loss, and PPEL's actions to address failed transmitters.
Data on the performance of Rosemount transmitter models outside the scope of Bulletin 90-01, Supplement 1 was also obtained.
These included all Model 1151, 1152, and 1153A transmitters and the Model 1153B/D and 1154 transmitters manufactured after July ll, 1989.
9412090269 941202 PDR ADQCK 05000387 PDR
Results:
Most Rosemount transmitters designated as safety-related by PP8L had indication function only.
The inspector concluded that, for the transmitters within the scope of surveillance, the actions taken by PPEL to address the loss of fill-oilconcerns described in Bulletin 90-01 and Bulletin 90-01, Supplement 1, were adequate.
- However, weaknesses were also identified in the transmitter monitoring program, as indicated by some of the inspection results below:
All pre-July 11, 1989, Model 1153 transmitters in the spare parts inventory were modified or replaced with transmitters equipped with sensing modules manufactured after July 11, 1989.
An enhanced monitoring program had been established for transmitters within the scope of NRC Bulletin 90-01 and its supplement.
However, no procedure had been prepared to define responsibilities and authorities.
This item was unresolved, pending preparation of a PPRL specific program.
The calibration trending program was adequate in identifying transmitters that indicated potential fill-oilloss.
However, the assessment of the trended data was informal and without supporting documentation.
- As a result, no justification was available to support the acceptability of several transmitters with abnormal trend.
This item was unresolved, pending the licensee reevaluation of transmitter trends and, if necessary, appropriate corrective actions.
Monitoring methods were limited to calibration data trending and awareness of transmitter response time.
No noise analysis or trending of operational data was being done.
Station personnel had been trained to recognize the symptoms associated with the loss of fill-oil and the training program had provisions for training newly-hired personnel.
In addition, the instrument calibration procedures contained adequate direction for the technicians on how to test for and identify a loss of fill-oil.
The licensee identified two pre-July 1989, Model 1153B transmitter failures and another was being investigated.
The licensee identiFied no failures of Model 1151, 1152, or post-July 1989 Model 1153 in safety-related application DETAILS 1.0 BACKGROUND (TI 2515/122)
On December 22, 1992, the NRC issued NRC Bulletin 90-01, Supplement 1,'Loss Of Fill-Oil in Transmitters Manufactured by Rosemount," to inform licensees of actions taken by the NRC staff and the industry in evaluating loss of fill-oil in Rosemount transmitters and to request licensees to take actions to resolve the issue.
Licensees were requested to develop an enhanced surveillance program for Model 1153, Series B and D, and Model 1154 transmitters manufactured before July ll, 1989.
The purpose of the surveillance program was to ensure that installed Rosemount transmitters meet current design criteria as highly reliable components for which failures can be readily detected.
Model 1151, 1152, and 1153A transmitters were excluded from the actions requested in the supplement due primarily to few confirmed oil loss failures and differences in the oil sensor design, as compared to Model 1153B/D and 1154 transmitters.
Similarly, due to transmitter design and process improvements and few confirmed failures, Model 1153B/D and 1154 transmitters, that were manufactured after July ll, 1989, were also excluded from the supplement actions.
2.0 PURPOSE OF THE INSPECTION The objectives of the inspection were to:
(1) verify that the actions taken by Pennsylvania Power and Light Company (PP&L) in response to Bulletin 90-01, Supplement 1 were acceptable; (2) evaluate the performance of safety-related model 1151, 1152 and 1153A transmitter at the Susquehanna facility; (3) obtain information on PP&L experience with failure of model 1153B/D and 1154 transmitters manufactured after July ll, 1989; and (4) determine PP&L reporting practices for failed transmitters.
The additional data collected on transmitters that are outside of the scope of the bulletin supplement will be used to verify failure reports, determine to what extent licensees notify Rosemount of transmitter failures, and to confirm that actions requested by the bulletin supplement were sufficient.
3.0 DISPOSITION OF STORED TRANSMITTERS The inspector reviewed the actions taken by PP&L to ensure that Rosemount Model 1153B and D and Model 1154 transmitters manufactured before July ll, 1989, and held in storage as stock spares would not be used in safety-related systems or systems installed in accordance with 10 CFR 50.62.
The inspector's review of the latest device list determined that most of the transmitters currently in use at the Susquehanna Steam Electric Station (SSES)
had been manufactured prior to July ll, 1989.
However, all transmitters were being monitored with an enhanced surveillance program, as requested by Bulletin 90-01.
Discussions with the licensee also determined that the spare transmitters held in stock had been returned to Rosemount so that their sensing modules could be replaced with others manufactured after July ll, 1989.
All transmitters had been modified and returned to stock soon after the issuance of the bulletin. Verification of several stocked spare transmitters found that their serial number either ended in "A" or was in the 500,000 range.
These serial numbers indicated, respectively, that the
transmitters had been either modified or manufactured after July 11, 1989.
PPSL also provided records of current requirements for new transmitter purchase orders.
According to these records,
"the vendors shall not supply material fabricated prior to 7/ll/89."
The inspector concluded that PP&L had taken appropriate actions to prevent inadvertent use of transmitters susceptible to oil leakage in safety-related
'r ATWS applications.
4.0 SURVEILLANCE PROGRAN 4. 1 Scope of Surveillance The inspector reviewed PPSL's instrument database and their response to Bulletin 90-01 and its supplement to determine their scope of surveillance and compliance with the NRC requirements.
In response to NRC Bulletin 90-01, PPKL searched the database of both SSES units and determined that the combined number of transmitters within the scope of review and surveillance described in the bulletin was 117.
A list was
~ provided with the response.
The list identified the model, serial, and tag numbers of installed transmitters as well as spare transmitters currently held in storage.
All transmitters were Model 1153B.
All transmitters were placed in an enhanced surveillance program, as requested by the NRC bulletin.
The licensee's review of the data revealed that two installed transmitters and thirteen spares were from manufacturing lots identified by Rosemount as having a high failure fraction due to loss of fill-oil. As indicated previously, the spares were returned to Rosemount for sensor module replacement.
The installed transmitters, one measuring inlet flow of the reactor water cleanup system and the other pressure (leakage)
between the inboard and the outboard main steam isolation valves of main steam line A, were evaluated by the licensee and found acceptable for service in the particular application.
The licensee placed these transmitters, like the others, in their enhanced surveillance program.
In response to NRC Bulletin 90-01, Supplement 1, the licensee again reviewed their database and found that several transmitters had reached
"maturity," as defined in the Supplement and in Rosemount Bulletin No. 4.
Although these transmitters could be dropped from the enhanced surveillance program, PPKL indicated that they would maintain them in the program until they determined that no benefit would result in continuing the enhanced monitoring.
During the current inspection, the inspector's review of safety-related transmitters in the enhanced monitoring program determined that the list had been expanded to 135 units and that the increase was due to the replacement of Model 1151 and 1152 transmitters with environmentally-qualified Model 1153s.
The inspector also determined that 28 of the 135 transmitters had a safety classification gP, PPEL's designation for components having a safety passive
function.
The remainder of the transmitters, although safety-related, did not perform a reactor protection system (RPS) or engineered safety feature (ESF)
actuation function and were not part of a system installed in accordance with
CFR 50.62.
Based on the above review, the inspector concluded that PPIIL's current scope of surveillance extended beyond the requirements of Bulletin 89-01, Supplement l.
4.2 Loss of Fill-Oil Failures and Reporting Practices The inspector reviewed available data to determine PPEL's experience regarding loss of fill-oilfailures in Rosemount Model 1153B and D and Hodel 1154 transmitters manufactured after July ll, 1989, and the licensee reporting practices regarding these failures.
In their letter of September 13, 1990, to the NRC, the licensee indicated that their investigation of the Rosemount transmitters at SSES had found that only one was suspect of having experienced a loss of fill-oil. The transmitter had been replaced approximately one year prior to the issuance of the NRC Bulletin and was not available for testing.
Therefore, its failure mode could not be confirmed.
Following the issuance of the bulletin, several other transmitters were returned to Rosemount for failure analysis.
Three were confirmed as having suffered a loss of fill-oil. None of these transmitters had been manufactured or included a sensing cell manufactured after July ll, 1989.
As indicated above, past transmitter failures were reported to Rosemount and failure modes were analyzed.
Discussions with the licensee indicated that no formal policy had been established regarding reportability.of failure and that no plans had been made to change past reporting practices.
The inspector concluded that PPhL's actions and reporting practices regarding failed transmitters were acceptable.
4.3 Surveillance Program To verify the acceptability of the actions taken by PPSL in response to NRC Bulletin 90-01, Supplement 1, the inspector reviewed the administration of the enhanced surveillance program, the monitoring techniques, the testing intervals, training of operations and technical support personnel, applicable plant procedures, and transmitter calibration records.
Pro ram Administration The administration of the Rosemount transmitters enhanced surveillance program at PP8L was the responsibility of the Instrumentation and Controls (I&C)
Production Engineering group, a branch of the site maintenance organization.
The inspector's review of responsibilities within the group determined that no formal procedure had been prepared.
Therefore, the authorities, responsibilities, and the methods to implement the program were not clearly define Discussions with the licensee indicated that the surveillance program had been established using the guidelines of Rosemount Technical Bulletin No. 4, dated December 22, 1989.
Based on the Technical Bulletin recommendations, the licensee had collected the applicable calibration data and developed appropriate graphs.
An I&C Production Engineer had been assigned to the program to receive, maintain, and evaluate the calibration data and to identify trends that might be indicative of fill-oilloss.
If a transmitter calibration data showed evidence of potential fill-oilloss, steps would be taken to establish the operability of the transmitter and to determine whether further actions, such as increased calibration and monitoring frequency or replacement of the transmitter, were warranted.
Although the inspector found the basic principles of the program acceptable, he considered the lack of a formal program a weakness in that it did not ensure that all transmitter test results would be forwarded to the responsible engineer, that the calibration data would be evaluated in a timely manner, and that proper actions would be taken in the case of suspect trends.
PP&L engineering indicated that, on occasion, calibration results had to be retrieved from files to update the graphs, and no documentation existed indicating when individual calibration trends had been reviewed.
As described later in this section, the inspector's review of the calibration data graphs found that some transmitters showed a questionable trend.
To address the inspector's observations in this area, the licensee provided a
copy of a calibration procedure (No. SI-116-301) revision that had been initiated via the procedure change approval process on October 21, 1994.
Step 7.2 of the revised procedure required that completed data sheets be forwarded to I&C Production Engineering and to initial that the step was completed.
This change, when approved, will be added to all safety-related Rosemount transmitter calibration procedures.
In addition, PP&L Engineering indicated that a procedure would be prepared by the end of December 1994 to formalize the enhanced surveillance program process.
The adequacy of the licensee's program is unresolved pending completion of a procedure that meets the criteria of NRC Bulletin 90-01, Supplement 1.
(50-387/94-23-01 and 50-388/94-24-01)
Monitorin Techni ues Test Intervals The elements of PP&L's program for monitoring the performance of the Rosemount transmitters manufactured prior to July ll, 1989, were outlined in their response to the original bulletin.
The inspector's review of the September 13, 1990, letter revealed that the surveillance program consisted largely in the trending of output drift derived from transmitter calibration records.
The program was based on Rosemount Technical Bulletin No. 4, as were the acceptance criteria for transmitter operability.
Specifically, the letter stated that the cumulative, one-sided zero drift, calculated and adjusted in accordance with the Technical Bulletin, would be compared to the allowable cumulative drift value specified in the Bulletin. If trending indicated that, for a particular transmitter, this value would be exceeded before the next calibration period, the transmitter would be suspect of fill-oilloss and replace The above letter also stated that training had been performed to ensure that licensed and non-licensed operators and I&C technicians were aware of the loss of fill-oilsymptoms, e.g.,
sluggish response during calibration and zero drift.
Regarding process noise.
the licensee stated that "Decrease in noise level in the transmitter output signal will be evaluated only in conjunction with other symptoms."
Honitoring frequency was not specifically addressed by the letter.
The inspector's review of monitoring program found that the licensee had done comprehensive research of the calibration data and evaluated it in accordance with the recommendations of Rosemount's Technical Bulletin No. 4.
Graphs had been prepared for ease of review.
Following the original historical trending, the licensee continued the collection of calibration records, calculated the zero and span drifts and revised the graphs.
When a safety-related transmitter was. replaced with a new one, or a new safety-related transmitter
'as added to the list, a new trending graph was developed.
Also, as indicated previously, the output drift trending of many transmitters was continued, despite having reached maturity.
All graphs appeared to be up-to-date.
Calibration periods were in accordance with the criteria of Bulletin 90-01 and its Supplement.
~Trainin The inspector reviewed PP&L's training program and conducted interviews to determine the capability of the technical personnel to identify loss of fill-oil failures.
The review revealed that both the ISC technical staff and the reactor operators had undergone training and that training sessions had been conducted between June and October 1990.
The program was based on Rosemount Technical Bulletin No.4 and consisted of slide presentations and discussions addressing industry experience, transmitter failure modes, and failure symptoms.
Training of I&C technical staff was conducted during several I&C shop and safety meetings.
For reactor operators, the loss of fill-oil awareness was imparted in classrooms as part of the requalification program.
A video tape of presentation to the operators was provided to the inspector for his review.
A review of the video tape showed that the basic concerns regarding loss of fill-oil had been addressed and that the methods of detections, e.g.,
sluggish response, output drift, output differences between, redundant instruments, had been adequately emphasized.
Although no additional lesson plans had been prepared and no other training sessions had been scheduled following the original training, the loss of fill-oilphenomenon and its symptoms were concisely discussed in the I&C training lesson, Unit IC007C, Principles of Process Instrumentation.
Interviews of I&C personnel indicated familiarity with the oil issue and awareness of symptoms.
Calibration Records The inspector reviewed the calibration records of the safety-related Hodel 11536 transmitters to verify the proper implementation of PP&L's enhanced surveillance program, to evaluate the scope and quality of their trending
methods, and to determine the adequacy of their transmitter performance evaluations.
The records reviewed included tables showing actual calibration results, applicable transmitter data and calculated zero and span shifts.
For each transmitter the records also included a graph of the cumulative zero and span shifts versus time with low and high limits shown, as applicable.
As indicated previously, the inspector's review of the calibration data graphs found that some transmitters showed a questionable trend and in some cases the limits specified in Rosemount Bulletin No.
4 had been exceeded.
For instance, for flow transmitter FT-E41-2N008, five of the six data points taken between 1987 and 1994 showed a positive span shift, with the span low limit being reached in October 1991 and slightly exceeded in March 1994.
Similarly, the drift graph of FT-15121A showed a negative zero shift trend, with the zero low limit being exceeded in September 1991.
In neither of the above examples, nor in the other cases reviewed by the inspector, did the zero or span shift trend clearly indicated that the transmitter had suffered a loss of fill-oil. For instance, in the first example the 1993 data point did not result in a span shift in the positive direction and the zero shift was relatively stable throughout the observation period.
In the second example the graph showed that most of the zero shift had occurred between 1985 and 1991, with no other data points being taken during the same period.
However, the suspect trend of the shift and the fact that a limit had been exceeded warranted a more in-depth evaluation of the calibration records.
No documents existed that showed that this had been done.
Discussions with the licensee I&C engineering personnel indicated reasonable assurance that the trends were not indicative of transmitter failur'e.
Their confidence was based primarily on the recent performance of the instruments in question.
Also, most of the transmitters, including those. with questionable trends, had indication function only.
Therefore, the inspector did not view them as a cause for an immediate operability concern.
The licensee indicated that they would review all graphs and address each abnormal trend by the end of December 1994.
The adequacy of the transmitters with abnormal trend is unresolved pending PP&L's review of applicable calibration records and their implementation of appropriate corrective actions, where warranted.
(50-387/94-23-02 and 50-388/94-24-02)
Transmitters Calibration Procedures The inspector reviewed several calibration procedures to determine the extent to which they provided guidance in identifying potential loss of fill-oil.
Examples of procedures reviewed included SI-162-302, Revision 8, "Calibration of Reactor Vessel Pressure Channels PT-14201A&B (Accident Monitoring)" and SI-162-303, Revision 6, "Calibration of Reactor Vessel Water Level Channels LT-14201A&B (Accident Monitoring)."
Section 5.4 of all procedures required that the technicians inform I&C Production Engineering if they observed certain symptoms.
The symptoms specified by the procedure were typical of loss of fill-oil and included sluggish response, transmitter failure to respond over entire calibrated range, and transmitter drift in excess of IX.
The inspector concluded that the procedures contained adequate instructions to help the technician recognize a potential oil loss, if one existe Until recently the loss of fill-oilconcern affecting certain Rosemount transmitters was not addressed by the licensee's corrective and preventive maintenance procedures.
On October 24, 1994, PP&L strengthened their monitoring process by revising Instruction No. IC-IE-05, "Maintenance of Equipment gualification."
Revision 6 of this procedure, applicable to all safety-related instruments, included a form requiring the technician to observe loss of oil symptoms, record the transmitters performance, and report abnormal conditions to I&C Production Engineering.
The inspector also witnessed the calibration of two transmitters that were scheduled to be performed during the inspection period.
The calibration results indicated acceptable transmitter performance.
However, the inspector observed that the method used might not reveal transmitter sluggishness, if it existed, for the following reasons:
The required input pressure was applied to the system using a hand pump.
Therefore, a relatively long time was needed to reach and maintain stable the desired input pressure.
In addition, the input pressure and output current/voltage were recorded by two technicians, one at the transmitter and the other in the control room.
Although the technicians were in continuous radio contact with each other, neither could effectively judge the speed of response of the transmitter.
Rosemount Technical Bulletin No.
4 states that the response time of the range code 6 transmitters tested is about 1 second.
In comparison, several seconds were required to achieve a
stable input pressure and almost ten more seconds to announce and record the transmitter output.
The inspector concluded that, for the transmitters tested, sluggishness would be a loss of fill-oilindicator only in the case of a large loss.
Conclusions Host Rosemount transmitters, designated as safety-related by PP&L, had indication function only.
The inspector concluded that, for the transmitters within the scope of PP8L's enhanced surveillance, the program was generally acceptable.
Adequate training had been provided to the technical staff and to the reactor operators to identify degrading transmitters due to loss of fill-oil.
This was evidenced by their identification of several failed transmitters.
Also, the failure symptoms were clearly described in the calibration procedures, ensuring that technicians were reminded of the concerns associated with the failures.
However, the lack of a formal program procedure, clearly defining authorities and responsibilities, hindered the timely and proper review of the calibration results.
This was evident in the lack of documentation addressing the acceptability of transmitters with suspect trends and/or with cumulative drift in excess of the limits described in Rosemount Technical Bulletin No. 4.
PP&L's primary monitoring techniques were limited to the trending of zero and span drift.
For the transmitter calibrations observed, the inspector considered any monitoring of response time to be ineffective in detecting oil loss.
No trending of operational data or noise analysis was being performed at SSES.
However, operators had been advised to be attentive at differences in redundant monitoring instruments.
The adequacy of the program and of several transmitters was unresolved pending further review and corrective action by the license.0 REVIEW OF MODEL 1151, 1152, AND 1153A TRANSMITTERS The inspector reviewed the licensee's records to identify all Model 1151, 1152, and 1153A Rosemount transmitters in safety-related (excluding pressure boundary) applications, to evaluate potential calibration failures, and to verify the results of PP&L's analysis to identify the cause of the failure.
The inspector found 22 Rosemount transmitters in this group were installed at Susquehanna, 19 Model 115ls, and 3 Model 1152s.
These model transmitters were not addressed by Bulletin 90-01 and its Supplement and were not included in the enhanced surveillance monitoring program.
The licensee had not recorded and not reported any failures from loss of fill-oilin this group of transmitters.
The licensee utilized a computerized tracking program to record equipment failures or abnormal conditions observed during the surveillance and maintenance activities performed on all instruments, including the transmitters within this group.
The program, governed by PPKL Instruction No.
IC-IA-07, "Installed Plant Instrumentation (IPI) Corrective Action Worksheet Process,"
involved the codification of observed instrument performance and entering the codes, along with pertinent data, in the computer.
These codes, including symptoms, root causes, and corrective action, were used by the licensee to further evaluate identified problems with specific instruments.
The inspector reviewed the summary sheet for all Model 1151 and 1152 transmitters and requested the calibration records for several transmitters showing an abnormal amount of "Out-of-Tolerance" and "Instrument Drift" codes.
None of the instruments evaluated indicated the potential for loss of fill-Ol1.
Conclusion The operation of the Model 1151 and 1152 transmitters at Susquehanna has been reliable, with no failures attributed to a loss of fill-oil. Adequate means existed to identify such failures.
6.0 NODEL 1153B/D AND 1154 TRANSMITTERS MANUFACTURED AFTER JULY 11, 1989
, The inspector reviewed the licensee's records to identify all Model 1153B/D and 1154 Rosemount transmitters manufactured after July ll, 1989, and used in safety-related (excluding pressure boundary) applications.
He also reviewed the calibration failures and evaluated the results of PPSL's review to identify the cause of the failure The inspector determined that eight Rosemount transmitters from this group, all Model 1153B, were installed at Susquehanna.
Of these, five were replacements for Model 1151 transmitters and three were replacements for failed Model 1153 transmitters.
All transmitters were included in the enhanced surveillance monitoring program, and the licensee has not reported any failures due to loss of fill-oil.
The inspector's review of the applicable drift graphs determined that six of the eight transmitters showed a stable output.
The graphs of two transmitters, LT-14201B and PT-12644, displayed large zero and span shifts during the first calibration period, following installation.
The shift of neither transmitter was attributable to loss of fill-oil.
Conclusion The operation of post-July ll, 1989, Model 1153B SSES transmitters was reliable with no failures attributed to a loss of fill-oil.
7.0 EXIT MEETING At the conclusion of the inspection, on October 28, 1994, the inspector met with the PP&L representatives denoted in Attachment l.
At that time, the inspector summarized the scope of the inspection and the results.
In particular, the inspector discussed in detail the concerns and the bases for two unresolved issues.
The licensee concurred with the inspection results and indicated that a procedure to better control the monitoring of the transmitter performance and the evaluation of the transmitters with abnormal performance would be completed by the end of December 1994.
No proprietary information was reviewed during the course of this inspection.
Attachment:
Persons Contacted
ATTACHMENT Persons Contacted Penns lvania Power and Li ht Com an T. Dalpiaz R. J.
Dixon J.
Finnegan C. Galbraith D. Gandenberger G. J.
Gukzynski S.
Kuhn R. J.
Pregg G. Stanley G. J. Treven R. Wehry Manager, Nuclear Naintenance I8C Production Engineer Supervisor, Nuclear Compliance ILC Engineer ISC Production Supervisor Hanager, Nuclear Plant Services Supervisor, Electrical/IEC Systems Engineering Supervisor, Site equality Verification Vice President, Nuclear Operations IKC Production Engineering Group Lead Compliance Engineer U. S. Nuclear Re ulator Commission
D. J.
Hannai Resident Inspector W. H. Ruland Chief, Electrical Section, Division Reactor Safety
Denotes those present at exit meetin ~ '