IR 05000327/1997002
| ML20137F514 | |
| Person / Time | |
|---|---|
| Site: | Sequoyah  |
| Issue date: | 03/24/1997 |
| From: | NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II) |
| To: | |
| Shared Package | |
| ML20137F493 | List: |
| References | |
| 50-327-97-02, 50-327-97-2, 50-328-97-02, 50-328-97-2, NUDOCS 9704010121 | |
| Download: ML20137F514 (24) | |
Text
.
.
B i
D. S. NUCLEAR REGULATORY COMMISSION REGION 11
Docket Nos:
50-327, 50-328
License Nos:
-
Report No:
50-327/97-02, 50-328/97-02 Licensee:
TVA Facility:
Sequoyah Units 1 & 2 Location:
Sequoyah Access Road Hamilton County, TN. 37379 Dates:
February 3-14,1997 Inspectors:
Caswell Smith, Reactor inspector i
Engineering Branch
Division of Reactor Safety, Region ll i
D. Davis, NRC Contractor
,
J. Malanda, NRC Contractor Approved By:
Harold Christensen, Chief Engineering Branch Division of Reactor Safety 9704010121 970324 ADC' K 05000327 PDR c
G PDR j
l
.J
<
,
.
.
EXECUTIVE SUMMARY Sequoyah Nuclear Plant, Units 1 & 2
)
NRC Inspection Report 50-327,328/97-02 Sequoyah Nuclear Plant (SONP)-1996 Reliability Report finding number 4 stated that tha
technicalinvnivement of the engineering organizations in maintenance and operational
'
activities has been ineffective in improving plant reliability and performance. This inspection included detail reviews of corrective actions 14-1,14-2,14-3,15, and 16 delineated in Problem Evaluation Report (PER) SQ960393PER which was developed and implemented to correct the deficiencies identified in SONP-1996 Reliability Report finding No.4.
Corrective.
14 was intended to establish a more effective engineering / maintenance interface af s ;9 organization of the Site Engineering and Material organization on October 1,
-
1996. Baseo c objective evidence reviewed this corrective action was not demonstrated as having been adequately implemented. Similarly, corrective action 15 which involved the
,
establishment of a Unit 2 cycle 7 High Impact Team (HIT) to evaluate knowledge based j
errors specific to reliability improvement systems, was inconclusive as to whether or not this corrective action had ever been performed. PER No. SO960393PER had been rejected twice by Nuclear Assurance & Licensing (NA&L) for failure of the PER documentation to provide a stand alone auditable record of appropriate corrective actions and/or recurrence controls. On February 6,1997, NA&L initiated a first level escalation of the PER to ensure compliance with the requirements of the 10 CFR 50 Appendix B, Criterion 16, corrective action program.
Overall the inspectors concluded that the licensee was adequately implementing corrective action 16 which involved hardware reliability issues. Objective evidence reviewed demonstrated that the licensee was following a sound technical approach in the disposition of degraded and nonconforming equipment.
!
Enclosure
.
.
Y
,'
.
.
Report Details
.
lli Engineering Backaround Sequoyah Nuclear Plant (SONP) performed a self assessment to evaluate why the reliability of the station had not improved even though there has been a large investment in personnelimprovements and material condition. The results of this assessment was documented in the Sequoyah Nuclear Plant Reliability Study, dated March 5,1996. The assessment was structured to have four teams develop data from different sources and cross validate information between teams. The reliability report documented the results of the common themes that run through the individual i
findings, documents specific findings, recommends corrective actions, and itemizes
-
the hardware reliability issues. Report Finding No. 4 stated that the technical
'
involvement of the engineering organizations in maintenance and operational activities was not effective in improving plant reliability and performance. Problem Evaluation Report (PER) No. SQ960393PER was prepared to perform root cause analysis, extent of condition evaluations and to develop corrective actions for adequate recurrence control of the identified deficiencies.
PER No. SQ960393PER corrective actions 14-1,14-2,14-3,15 and 16 were i
developed to address the deficiencies of Finding No.4. Corrective actions 14 included a reassessment of engineering involvement in and support of maintenance planning, maintenance performance, maintenance problem analysis, trending and data analysis to adjust the preventive and predictive maintenance program. Corrective action 15 required the establishment of a Unit 2 cycle 7 High Impact Team (HIT), to evaluate knowledge based errors specific to reliability improvement systems; and corrective action 16 required performance of additional reviews of trip sensitive systems to ensure that any components or sub-components that were at or near end of life, or had been exposed to adverse operating conditions would be replaced.
The focus of this inspection was to evaluate the adequacy of the licensee's corrective actions implemented for equipment reliability problems. Required improvements in the conduct of engineering / maintenance interface operations implemented by corrective actions 14 and 15 were reviewed by the inspectors. Additionally, corrective action 16 related to hardware reliability issues listed below were inspected by the team:
The licensee identified 12 Balance of Plant Systems that should be
-
considered. The licensee indicated that the Action Plan would be completed in 4 phases. These phases were as follows:
"16a, The first phase and initial scope will be set at (12) of BOP
-
U2 systems that Operations and TS mutually agree have the highest risk of trips or runbacks. The review will identify trip / runback vulnerabilities (preferred method is via fault tree analysis).....
j
I
.
.
16b, The second phase will be a continuation of activities on the
-
~
same set of systems to include further review of the identified vulnerabilities to: identify whether the components involved are obsolete, have adequate spare parts, or have industry identified weaknesses....
16c, The third phase will be to complete a Phase 1 and 2 review
-
of the comparable U1 systems...
16d, The fourth phase will be additional BOP systems..... "
-
The team chose specific problem evaluation reports (PERs), design change notices
. DCNs), work orders (WOs), operator work arounds (WAs), issue identification forms, (
and work requests (WRs) identified in the above phases to evaluate the effectiveness of the licensee controls in identifying, resolving, and preventing recurrence ofissues that degrade the quality of plant operations or safety. The team also evaluated the communication and cooperation between the engineering and maintenance organizations. The team performed walkdowns, conducted interviews with system engineers, design engineers, and maintenance personnel, and reviewed specific document types noted above for selected systems from the group of Balance of Plant systems identified by the licensee as having the most impact on plant reliability through their self assessment.
E1 Conduct of Engineering E1.1 Sequoyah Nuclear Plant-1996 Reliability Report Corrective Actions:. Engineering Maintenance interface Deficiencies a.
Inspection Scope The inspector reviewed the results of the licensee's reassessment of engineering involvement in and support of maintenance planning, maintenance performance, i
maintenance problem analysis, trending and data analysis. Problem Evaluation Report
(PER) No. SQ960393PER, corrective actions number 14-1,14-2,14-3, and 15 were developed for implementating the recommendations delineated in the above report for correcting the identified deficiencies in engineering / maintenance interface.
Responsibilities for completion of corrective actions 14-1,14-2, and 14-3 were-l assigned to the Component Engineering Manager, the Maintenance Manager and the l
Engineering and Materials Manager respec ively. Responsibility for implementing corrective action number 15 was assigned to the Outage Director. The inspector
performed an independent evaluation of the corrective actions to verify the adequacy
)
of root cause analysis, extent of condition evaluations, recurrence controls, and procedural compliance with corrective action program requirements delineated in
procedure SSP-3.4, Corrective Action, Revision 19.
i b.
Observations and Findinos PER No. SQ960393PER describes the corrective actions completed for closure of
'
corrective action 14,,2,3, and 15 as a reassessment having been performed by the j
i
.
-
.
Engineering and Materials Manager, Maintenance Manager, and the Component Engineering Manag'er. This reassessment of engineering involvement in and support of plant maintenance activities resulted in engineering roles, responsibilities, and expectations being newly defined and agreed upon by involved plant management.
Implementation of the new division of engineering responsibilities were scheduled to become effective upon reorganization of the site engineering organization. This reorganization became effective on October 1,1996.
The licensee's " Engineering Reorganization Plan" developed and implemented for the recent site engineering reorganization identified action iterns that required completion to facilitate an orderly restructuring of the site engineering organization. Action items identified by subject heading included staffing; program transfers; project transfers; procedure changes; administrative issues; workforce issues and communications.
The Engineering Reorganization Plan also identified a lead mechanical nuclear procedure list, a lead electrical procedure list, a lead civil procedure list, a lead engineering support procedure list, and a lead technical support procedure list that would require revision because of the reorganization. Newly defined division of engineering responsibilities were also described for equipment qualification, preventive maintenance, component support, leak repair, maintenance rule and NPRDS trending, and spare parts.
<
The inspectors reviewed selected procedures in order to verify that functional responsibilities, levels of authority, and lines of internal and external communication interfaces had been incorporated in the procedures to reflect the new division of engineering responsibilities, and eng!neering organizational structure. The procedures
"
reviewed included SSP-6.5, SSP-6.3, SSP-8.50, SSP-6.3, SSP-6.4, and SSP-6.26.
Based on this review the inspectors concluded that the licensee had not completed the action items in the Engineering Reorganization Plan involving procedure revision nor had the new division of engineering responsibilities been incorporated in site level procedures. Discussions with TVA management concerning this issue and the other action items described in the Engineering Reorganization Plan revealed that the
,
licensee was unable to present objective evidence which would demonstrate completion of corrective actions associated with 14-1,14-2, and 14-3. As a result of this inspection finding TVA management on the weekend of February 8,1997, revised the site level procedures of the System Engineering group to incorporate the new division of engineering responsibilities based on the existing organization structure and functional responsibilities. TVA management was informed that the actions taken by the Systems Engineering manager in revising the procedures were necessary but they were not sufficient to address the lack of objective evidence for completion of action items 14-2 and 14-3.
Discussions with the Outage Director conceming the status of corrective action 15 revealed that this item was loaded into TROI as a completed action, i.e., there were no actions required to be taken for resolution of the deficiency associated with corrective action 15. This corrective action required the establishment of a Unit 2 cycle 7 ' Fah Impact Team, (HIT), to evaluate knowledge based arrors specific to reliabilhy improvements systems. TVA management took credit for the Balance of Plant, (BOP), HIT that was in place and functioning during the Unit 2 cycle 7 outage.
.
..-
-
-
.
,
.
The inspectors were informed that there was no need to perform any specific actions
,
when the reliability study was later issued. Based on this discussion, the inspectors
requested TVA to provide objective evidence which demonstrated that corrective action 15 had been completed. The licensee provided the following documents for the inspector's review:
Memorandum from the Outage Director to Nuclear Engineering dated
-
February 6,1996, subject; Per SQ960393-Action item #15-Closure Documentation Unit 2 Cycle 7 Refuelling Outage Critique
-
Unit 1 Cycle 7 Refuelling Outage Critique
-
Milestone Manager Handbook, Rev.1, April 1996
-
Memorandum dated August 18,1995, Subject; SONP-Kickoff meeting
-
for Unit 2 Cycle 7 High impact Team Leaders
Memorandum dated December 4,1995, Subject; Information Exchange
-
Meeting for Unit 2 Cycle 7 HIT Leaders Based on review of the above documents the inspectors were not successfulin verifying that corrective actions involving evaluation of knowledge based errors specific to reliability improvements systems had been performed. The basis for the licensee's action of taking credit for the U2C7 BOP HIT was not demonstrated by a root cause analysis of the deficiency, an extent of condition evaluation, nor developed corrective action plans for recurrence control.
Pursuant to the above inspection findings the inspectors conducted interviews with personnel from the site NA&L organization to determine the degree of involvement of this organization in the disposition of PER No. SQ960393PER. In discussions with personnel from the site NA&L organization the inspectors were informed that PER No.
SQ960393PER had been rejected twice because the PER documentation lacked sufficient details to provide a stand alone auditable record of appropriate corrective actions and/or recurrence control. On February 6,1997 a first level escalation of PER No. SO960393PER was initiated by NA&L to ensure compliance with the requirements of the corrective action program during implementation of the corrective actions for deficiencies documented in SQNP-1996 Reliability Report.
c.
Conclusion The inspectors concluded that the licensee failed to demonstrate that an adequate root cause analysis of the deficiencies identified in SONP-1996 Reliability Report had been performed and that the identified root causes had been clearly documented in the PER. Additionally, there was no evidence that an extent of condition evaluation had been performed and finally objective evidence demonstrating completion of corrective actions was not available for the inspector's review. The inspectors
-
-
.
concurred with the site NA&L organization that the PER was not being dispositioned in accordance with ths controls of the corrective action program and that NA&L was taking the appropriate actions to address this issue with TVA management.
,
E1.2 System Status Reports a.
Inspection Scope The System Status Reports (also called the System Health Reports) are issued by the Technical Support Department on a quarterly basis. The purpose of the review was to evaluate the effectiveness of communication of issues to the other departments and site management and to assist in selecting specific engineering documents for team review.
,
,
b.
Observations and Findinas The team reviewed the four reports issued during Fiscal Year 1996 (October 1,1995 through September 30,1996) and the report for the first quarter Fiscal Year 1997 (October 1,1996 through December 31, 1996). The team noted that the Executive l
Summary of each report color-coded the ratings of the systems into four groups:
green (excellent), white (acceptable), yellow (needs improvement), and red (not acceptable). The summary presented the percentage of systems in each category by
{
unit and also highlighted the systems that had improved and degraded ratings since the last report. Explanations were given for these changes in ratings. This presentation was consistent for the five reports reviewed. The team observed that, since the second quarter FY96, the number of systems in the yellow or red category had decreased or remained the same until the first quarter FY97 Report, which indicates a negative trend of yellow systems for both units and a negative trend of red systems for Unit 1. These percentages can change substantially with only a few systems actually changing color.
The team noted the following improvements in the System Status Reports during their review:
A system report card has been added to each system report. The system
-
report card indicates the trends of certain parameters for each system such as open PERs, new PERs, degraded components, total WO backlog, and the number of disabled alarms.
The content of each system report has expanded over time to supply more
-
informatien on each system.
Rating definitions were revised and minimum performance criteria for each
-
color rating was explicitly defined and standardized.
-
Maintenance rule descriptions and actions were added to each syste *
.
a
in order to gain assurance of the future closure of issues identified in the System Status Reports, the team traced a number of items from the Report to the Master issues List (MIL). From the sample selected, the items that were chosen were shown as open items on the MIL.
c.
Conclusions The review of the System Status Reports by the team indicated a steady improvement in content and ease of understanding the important issues that were affecting system performance. These reports are considered adequate.
E1.3 Quality and Seismic Reauirements For TVA Class H Pioina and Eauipment a.
Inspection Scope The inspector reviewed several work packages associated with the Condenser Cooling Water System that changed the equipment configuration in the Condenser Cooling Water Pump House Structure.
b.
Observations and Findinas The Condenser Cooling Water Pump House Structure is classified as Seismic Category 1. The Condenser Cooling Water Piping System is classified as TVA Class H Piping, Seismic Category l(L). TVA Document SON-11, Paragraph 3.2.2.6 characterizes Class E through Class V components by stating the following; "Since these components complement safety-related components during normal operation and are in close proximity to them, they are designed to code requirements that will assure the integrity of the syctems such that the minimum capability of safety components will not be compromised." TVA Document SQN-DC-V-41.0, Paragraph 3.1 states; "The purchase requisitions for Category l(L) equipment should require an attestation (perfomled by CEB Component Qualification Group / Authorized contractor and/or Procurement Engineering Group (PEG)/ Power Stores Receipt inspection) that the equipment being supplied is structurally identical to that which was qualified and to certify any changes to the commercial equipment as a result of the qualification." TVA Document SSP-3.2, Appendix 1, Paragraph 1.0.E states, "All components... and Piping... shown as WA Class H... on system flow diagrams.. and piping..
drawings are Seismic Category l(L), and are covered by the Augmented QA Program."
The inspector observed that several pieces of equipment that were replaced in the Component Cooling Water Piping System in 1996 did not adhere to these requirements. It was specifically noted that the pressure control valve which was partially replaced in accordance with WR C211641, was requisitioned (No. 992354) as
"QA Level 0", with the End Equipment Classification block checked off as "Non-Quality Related". It was further noted in the same work package that the strainer which was added to the system in accordance with WR C325693 was also requisitioned (Nc, 1039600) as "QA Level 0", with the End Equipment Classification block checked off as
"Non-Quality Related", as were the threaded couplings that were installed in lieu of the original soldered piping fittings. TVA management was asked to identify the
.
.
procedure which permitted the replacement of TVA Class H, Seismic Category l(L)
piping and equipme'nt in a seismic Category I structure with non-quality related and non-seismic qualified material. There was no such procedure identified.
c.
Conclusion The inspector concluded that requisition of materials installed under WR C211641 and WR C325693 were assigned a QA level of "QA Level 0"in lieu of Seismic category 1(L). Additionally, procedural controls for establishing requirements where materials may be procured / requisitioned at a lower QA level have not been established by the licensee. This issue is unresolved pending further NRC review (50-327,328/97-02-
01).
'
.
E2 Engineering Support of Facilities and Equipment i
System 6 - Heater Drains and Vents This system contains 21 feedwater heaters,5 heater drain pumps,2 main feed pump turbine condensers,2 heater drain tanks,6 moisture separator drain tanks,6 high pressure reheater drain tanks,6 low pressure reheater drain tanks,1 main feed pump turbine condenser drain tank, and 2 main feed pump turbine condenser drain pumps.
Each string of heaters and reheaters and drain tanks has a series of level switches, level controllers and level control valves to control heater and drain tank levels. The licensee had identihed a number of weaknesses in the original design and single point failures that can create, as a minimum, power reductions. The licensee had identified vulnerabilities mainly at the high pressure /high temperature end of the process and the need for additional heater bypass lines. The team chose a sample of documents that reflect the vulnerabilities identified by the licensee.
E2.1 PER No. SO952068PER, #2 Heater High Level Water Hammer a.
Inspection Scope The team reviewed the PER in order to verify that the root cause analysis, extent of the condition evaluation, and the developed corrective action plan provided for effective corrective action and the intent to prevent recurrence.
b.
Observations and Findinos PER SQ952068PER documented a deficiency involving placing the #2C feedwater heater into service during the Unit 1 Cycle 8 startup. The heater filled to a high-high level when the extraction low point drain and extraction bypass were opened per procedure. After the heater string isolated, significant water hammer occurred in the
- 2 extraction line. Water hammer subsided when the low point drain and bypass valves were closed. Operations performed a walkdown of the extraction line and did not uncover damage that occurred to the hangers. The unit was operated for approximately 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> without placing the #2 feedwater heaters into service. This was performed to meet the requirements to " heat soak" the turbine prior to performing
.
.
j overspeed testing. A decision was made to perform the soak at 15% power without placing the heaters'in service. During previous heat soaks, unit power was brought to 20% to 25% and the #2 feedwater heaters were placed in service as part of the normal power escalation process.
The licensee formed an investigation team of 2 operations personnel, a system engineer, and a design engineer to determine the root cause of the event and to determine corrective actions to prevent recurrence. The investigation team determined that the root cause of the event was a lack of understanding by operations of the consequences of operating the turbine for extended periods of time without the i
- 1 and #2 fee Jwater heaters in service. It was also determined that there was a lack of concem over the water hammer incident that led to untimely reporting of the event
_
to management.
Information on how to avoid water hammer was provided to Operations for the short term until Operations procedures could be revised. The other actions to prevent recurrence were as follows:
Revise the operational procedures to prevent prolonged operation without the
-
- 1 and #2 feedwater heaters in service and the procedures addressing the performance of the turbine overspeed test and temperature soak.
Revise the procedure to ensure that the line from the moisture separator
-
reheater drains to the feedwater heaters are slowly opened when placing the
- 1 and #2 heaters in service.
Develop a plan to improve feedwater heater maintenance and operational
-
reliability by reviewing maintenance and operational methods at other utilities, reviewing previously identified heater control problems and ensure corrective actions were adequate. Issue additional corrective actions as necessary.
Increase the priority matrix score to add #2 feedwater heater bypass lines to
-
the condenser.
Emphasize to the operators the need to promptly report and evaluate
-
unexpected or abnormal transients, especially water hammer and the importance of fully evaluating changes in operating conditions.
Initiate WRs to replace all feedwater heater level controllers.
-
Emphasize to the Technical Support staff the importance in properly
-
disseminating information that effects other organizations.
c.
Conclusions The team concluded that the corrective actions specified in the PER were adequate.
The DCN to install the #2 heater bypass line has not been completed and, therefore, has not been installed. Therefore, it is possible that the unit reliability could still be
l
.
challenged. The bypass line is presently scheduled to be installed in 1998 in Unit 2 (U2 Cycle 8 RFO) snd in 1999 in Unit 1 (U1 Cycle 9 RFO). The system engineer did
'
inform the team that a majority of the level controllers had been replaced. The level controllers on the low pressure / low temperature applications that have not had any reported operating problems were not replaced. During plant walkdowns, the team did observe that a large number of the level controllers had been replaced.
'
E2.2 PER No. SQ962876PER 2-LIC-6-105 Level Controller False Signal a.
Inspection Scope The team reviewed the PER in order to verify that the root cause analysis, extent of
.
the condition evaluation, and the developed corrective action plan provided for
effective corrective action and the intent to prevent recurwnce.
b.
Observations and Findinas PER SO962876PER identified a deficiency that opened a level control valve for #3 Heater Drain Tank on Unit 2 and caused a power runback from 100% power to 77%.
The licensee performed a root cause analysis by investigating the failure modes that could have caused the runback and by troubleshooting the equipment functions and conditions. The controller was removed from service and carefully examined. The controller that failed was new and had been installed approximately 2 months before
,
the incident. It was determined that debris was blocking a small hole in a nozzle. The nozzle controls the back pressure on the air relay that controls the bypass valve. The root cause analysis determined that the particles were in the controller since fabrication by the manufacturer, Masoneilan.
The licentae evaluated the extent of the condition by determining the location of all
the Masoneilan 12800 Series controllers in the plant. These controllers were extensively utilized in the Heater Drains and Vents System,6 locations in the Auxiliary Steam System, and 2 on the Gland Seal Water System. The majority of the controllers in operation were original equipment and were not considered suspect because of their prolonged operation without a problem. Level controllers had exhibited recent reliability problems, however, this had been caused by aging. These problems were being addressed by replacing the aging controllers. The licensee surveyed all the controllers used at Sequoyah and determined that 14 new controllers had been installed in Unit 2. During the next plant outage an inspection of 4 of the controllers revealed brass particles in 2 of them. An inspection of the controllers in inventory was also performed and 20% had debris. All nozzles with debris were cleaned. WRs were written to inspect and clean the nozzles of the remaining Unit 2 controllers and the Unit 1 controllers.
In order to prevent recurrence, Masoneilan was contacted and they verbally committed to correct the problem. However, in addition to this action, the licensee planned to develop a receipt inspection requirement to inspect for debri *
.
.
The licensee also indicated that a train!ng letter will be issued to all craft that maintain controllers to be aware of this problem during troubleshooting and this information will be included in their training program.
c.
Conclusions The team concluded that the licensee's root cause analysis had been adequately performed and that the corrective actions specified in the PER were adequate to address the identified problem.
E2.3 PER No. SQ963034PER #4C Heater High Level Alarm Actuation a.
Inspection Scope The team reviewed the PER in order to verify that the root cause analysis, extent of the condition evaluation, and the developed corrective action plan provided for effective corrective action and the intent to prevent recurrence.
b.
Observations and Findinas On Unit 1 the main control room received a #4C feedwater heater high level alarm without Valve 1-LCV-6-166B opening. When the LIC was opened for inspection, the LIC output increased and the valve opened. Level returned to normal and the valve closed. Shortly thereafter, the valve opened without the heater high level alarm.
Technicians at the LIC at the time observed that the flapper nozzle spring was not responding correctly. Under WR C359620 the spring tension was adjusted. Further monitoring of the LIC showed no additional anomalies.
Since the PER was dispositioned as a Level C PER, a root cause analysis is not
'
required. In accordance with Sequoyah Site Standard Practice SSP-3.4, Revision 19, dated December 3,1996, for a Level C PER, only an apparent cause is required.
This PER was initiated in November 1996 and the apparent cause and corrective action plan have not been issued at this time. The System Engineer did initiate an extension request form in December 1996 to shift the due date for these actions. The request was made by the System Engineer due to an increase in the number of PERs that impacted his time. These extension requests are in accordance with SSP-3.4.
c.
Conclusions Since the apparent cause and corrective action plan have not been completed, an evaluation of these results can not be performed. However, it was noted that the System Engineer may have too large a workload to address some problems in a timely manner. The System Engineer did indicate that a generic review was necessary at Watts Bar since they have a similar feedwater heater desig.
_.. - -
-. - - ~. -. - - - -. -.. -. _ -.
. - - -... -. -
!
'
-
,
,
.
i
.
l
'
{
E2,4 PER No. SQ963041PER Trend Analysis for Mercoid Level Switches l
j a.
hLspection Scope
'
.
f-s
!
The team reviewed the PER in order to verify that the root cause analysis, extent of T
the condition evaluation, and the developed corrective action plan provided for f
effective corrective action and the intent to prevent recurrence.
b.
Observations and Findinas
!
The Site Quality organization reviewed the plant Tracking and Reporting of Open items (TROL) printout for approximately a 3 year period and determined that there was an adverse trend identified with the failure of Mercoid level switches including micro switches which indicated ineffective corrective actions for previously closed PERs.
'
This PER demonstrated that Sequoyah's equipment history and failure trending
!
program would not identify these problems since it trends on the component and
!
requires 2 failures within a 24 month period on the same component to be classified as a repetitive failure. The cornponent identifiers were different for the 5 problems identified except for 2 items that were more than 24 months apart.
The team was advised by the System Engineer that Design Engineering is surveying the industry to address the failure of Mercoid switches, which appears to be a heat-related problem, in conjunction with a continuing problem the site has had with leaking level sight glasses on feedwater heaters, main steam reheaters, and drain tanks.
Design Engineering has written DCN R12040B to add isolation valves with ball check valves to eliminate safety concems in the event of sight glass failure but this would not address the lea' king of the sight glass. These valves have not been installed. The fix being pursued would eliminate glass from the level indicator and would have a magnetically moving indicator that would trip reed switches, as required, for alarm or control functions. This appears to be an acceptable approach to solving a number of ongoing equipment problems with a single fix.
This PER was initiated in November 1996 and the apparant cause and corrective action plan have not been issued at this time. The System Engineer has initiated 2 extension request forms in December 1996 and January 1997 to shift the due date for these actions. The latest request indicated that this PER would be used by the root cause training class to be conducted the week of February 3,1997. The licensee indicated that class comments would be incorporated and resolved as part of the final analysis. These extension requests are in accordance with SSP-3.4.-
c.
Conclusions Since the apparent cause and corrective action plan have not been completed, an evaluation of these results can not be performed. However, it was noted that the System Engineer may have too large a workload to address some problems in a timely manner as indicated by the need to extend completing these documents.
.
.
.
E2.5 DCN No. R-121978, Change to CCW Pump Motor Protection Scheme
,
System 27 - Condenser Circulatina Water The purpose of this system is to provide cooling water to the main turbine condensers. The system consists of 3 CCW pumps with synchronous motors, piping, condenser tubing, the Ameritap condenser tube cleaning system, and cooling towers for each unit. The CCW pump bearing tube oil cooling water system is also part of this system.
a.
Inspection Scope The team reviewed PER SQ952315PER, Switchyard Bus Differential Relay Operation Trips Unit 2, in order to verify that the root cause analysis, evaluation of the extent of the condition, and the developed action plan provided for effective corrective action and prevention of recurrence. The issue Identification document and plant modification DCN R-12197-8 were reviewed to observe the process and the technical adequacy of these documents.
j b.
Observations and Findinos PER SQ952315PER documented the tripping of Unit 2 in December 1995. The 161 KV switchyard breaker 974 failed, which caused the operation of the differential relays.
On Unit 1,2 of the 3 CCW pump motors tripped. Condenser pressure increased to 1.37 psia which was well below the turbine trip setpoint. The operator restarted the CCW pump motors, condenser vacuum retumed to normal, and the unit continued to operate. On Unit 2 the fault resultea in the tripping of all 3 CCW pump motors. When the turbine failed to trip on loss of condenser vacuum (5.4 psia), the operator manually tripped the reactor. After the operator stabilized the reactor in Mode 3, the CCW pump motors were restarted and condenser vacuum was reestablished.
The licensee investigated this incident with a team that included Technical Support, Operations, Customer Group (Transmission and Distribution), and Engineering and Technical Services. From the very detailed analysis performed, the licensee identified the root cause of this incident as a mill defect in the main contact porcelain section on C phase of Breaker 974. However, there were a number of equipment anomalies that were also identified. From the event in the switchyard that operated the differential relays, which was caused by a C phase to ground fault, it was determined that the differential relays operated as designed. However, a transient from the switchyard fault to the Sequoyah auxiliary power system caused 5 of the 6 CCW pump motors on Units 1 and 2 to trip. The PER identifies a number of trip mechanisms that could have caused the tripping of the CCW pump synchronous motors. The out of step relay that is upstream of the CCW PUMP motor excitz.tlor?
could have tripped the motors. This relay senses the potential between phases A and C for the 6900 volt system that supplies the pump motor and compares it to the phase B current for the same system. This relay operates on the phase angle between the monitored voltages and current which indicates a deviation from synchronous operation. Other pee
' trips could have been caused by the 50G relay that senses
_
,'
.
.
ground fault current or other various trips associated with the field contractor and the interlock circuits. There was no indication available that allowed the determination of the relay that actually caused the motor trips.
The licensee concluded that the CCW pump motor tripping was due to the inability of the design of the CCW pump motor protection to handle switchyard transients. The licensee analysis indicated that Nuclear Engineering would evaluate methods to prevent CCW pump trips associated with switchyard operations. The licensee identified a similar event in 1992 when both units tripped as the result of a C phase to ground fault on the 500 Kv system. That event caused all 6 CCW pump motors to trip due to a voltage swing in the exciter field.
The licensee initiated issue identification No. 96041 in February 1996 to address
,
some of the methods that may be used to alleviate the CCW pump trips due to switchyard transients. The objective of the circuit changes was to reduce motor
'
protection sensitivity without jeopardizing equipment or personnel safety. This document analyzed the relays in the motor protection circuit and, due to their operation during the transient, concluded that they were not involved or contributed to the tripping event. A number of options were discussed and recommendations made.
However, since the conclusions did not justify why 1 of the CCW pumps did not trip, it is stated that further testing would be performed by the Customer Group to determine the root cause of the tripping of the motors. The tests were performed but a root cause analysis of the motor tripping was not performed by the Customer Group.
DCN R-12197-B was issued in June 1996 for Unit 2 and installed in 1996. The Hitachi out-of-step relay was replaced by a GE relay and an auxiliary relay (56X) was installed since the contacts of the GE relay were not rated for 220v service. The GE relay has an adjustable time delay setting that can be set above the upstream tripping time and, therefore, prevent spurious tripping of the CCW pump motors. The j
underfrequency relay was rewired to ensure that its target would be actuated if the relay was energized. A time delay auxiliary relay was added to the excitation undervoltage tripping circuit and the control circuit relaying was diversified so that a fault on one phase would not disable all of the CCW pump motors. The i
recommendation of adding an uninterruptible power supply to the motor control circuitry as indicated in Issue Identification 96041 was not included in the design. The Design Engineer indicated that further discussions with operations and maintenance determined that this would create additional maintenance that was not justified.
The team reviewed DCN R-12197-8, WR C-210388, and the following documents for technical adequacy of the design change:
Drawing 45N763-3, Wiring Diagram 6900V Unit Auxiliary Power Schematic
-
Diagrams, Revision 8 Drawing 45N763-4, Wiring Diagram 6900V Unit Auxiliary Power Schematic
-
Diagrams, Revision 7 i
.
-
-. -.
-
. -
--..
.-
-..-
-
... - - - _ -._
.
!
.
.
.
,
2
Calculation SQN-RSS-008, 6900V Unit Boards - Condenser Circulating Water
-
j Pumps CCWP 1 A,1B,1C,2A,2B, & 2C, Revision 1 I
c.
Conclusions The team concluded that the plant modification was technically adequate even though
a root cause analysis for the motor tripping was not completed. The plant
'
modification has been effective as indicated by another lightning strike in the switchyard that tripped the Unit 1 CCW 1C pump motor but the Unit 2 motors did not
'
trip. Reference Problem Evaluation Report SQ961717PER. Plant Modification DCN
R-12197-B had been installed in Unit 2 prior to lightning strike. The Unit 1 i
modifications had not been installed at that time.
~
E2.6 DCN No. M11829A Turbine Trip Logic j
System 35 - Generator Coolina
"
~
This system is a combination of three systems that support the main generator:
Hydrogen, seal oil cooling and stator cooling water. The purpose of these systems is to maintain the operating temperature of the main generator, a.
Inspection Scope
j The purpose of the DCN was to decrease the possibility of an erroneous plant trip.
The original logic was designed as a one out of one logic. A turbine trip is generated
'
using one differential pressure switch for stator cooling water differential pressure or
'
one temperature switch that monitors stator cooling water discharge temperature.
The team reviewed the plant modification DCN-M11829A in order to verify that the design change had been prepared, reviewed, and approved in accordance with the requirements of the design engineering program. The DCN was also reviewed for technical adequacy.
b.
Observations and Findinos The DCN for Unit 2 was initiated to revise the original logic of tripping the turbine on the closure of one contact on a differential pressure switch or one contact on a temperature switch to a 2 out of 3 logic, which will decrease the possibility of spurious plant trips. In order to achieve this design change,2 additional differential pressure switches and 2 additional temperature switches were installed in the plant. DCN-M11829A included the constructibility walkdown checklist, a safety assessment and safety evaluation, procurement requests, human factors evaluation checklist, cable installation tickets, and marked up drawings that showed the location of the new switches and thermocouples and the revised wiring diagrams.
The team reviewed DCN-M11829-A and the following documents for technical adequacy of the design change:
TVA Design Standard DS-E-18.1.24. Human Factors Engineering, Revision 0
-
,'
.
Calculation SON-VD-VDC-011, 250 V DC Pwr Voltage Drop Evaluation-1,2-R-
-
71, Revisiori 1 i
c.
Conclusions The team concluded that the plant modification was technically adequate and had been prepared in accordance with the requirements of the design control program.
E2.7 PER No. SQ962543PER Arrow Hart Contractors System 201 - 480 Volt Electrical Boards
.
The 480 volt electrical boards supply power at th' 480 voit level to both safety-related and non safety-related equipment. A review of sne System Status Reports indicated that there were a number of programmatic problems identified including problems with the Motor Control Center contractors supplied by Arrow-Hart and an industry issue on the lubrication of Westinghouse DS breakers.
a.
Insoection Scope j
Due to numerous failures of the front mounted auxiliary contacts in the Arrow-Hart starters during the previous two years, the Maintenance Rule Expert Panel applied the (a) (1) status to the contactors. Some of the repeat problems identified have been sticking contacts and binding clappers and binding mechanical interlocks. The inspectors reviewed corrective actions related to these deficiencies.
b.
Observations and Findinas Even though the present status of this PER is on hold due to comments by Site Quality Assurance that have been addressed by System Engineering, the team reviewed this PER due to the long term impact of the Arrow-Hart contactors on plant reliability. The comments centered around an inadequate corrective action for a previously closed PER and the process to close the PER within the Maintenance Rule process. The purpose of this PER was to review all of the previous problems with the contactors and ensure that a corrective action plan will resolve these problems.
The licensee performed a root cause analysis of the Arrow-Hart contactor failures and determined that these contractors were responsible for 4 preventable functional failures in Systems 63 and 74 in the last two years. The root cause of the contactor failures was determined to be the failure of the front mounted auxiliary contactors due to past cleaning and lubricating practices exaggerated by a weak design. The corrective actions from the previous PERs were adequate to resolve the direct root causes listed in the PERs.
The licensee identified the following additional actions to alleviate this problem:
,
.
,
-
replace operationally important reversing and two speed starters on Units 1 and 2. These actions will be completed during the Cycle 9 refueling outages on each unit (1998 and 1999).
review the application of front mounted auxiliary contacts on all Arrow-Hart
-
contractors not currently scheduled for replacement and develop a listing of those contractors where proper operation of the front mounted auxiliary contacts is essential to the function of the component and operation of the unit for the purpose of performance monitoring. This task has been completed.
,
-
continue performance monitoring of contact resistance measurements per monthly PMs and Operations Standing Orders (refer to SQ9f005WA) until all
'
operationally important reversing and two speed contractors are replaced.
.
.
The licensee's Maintenance Rule Expert Panel has not reviewed the failure analysis, root cause analysis and corrective action plan at this time.
c.
Conclusions The team verified that the cause determinations were consistent with those previously identified by the licensee. Eased on this review, the team concluded that the licensee's root cause analy' sis and corrective action plan had been adequately performed.
E2.8 PER No. SQ962513PER nNestinghouse DS Breaker Lubrication a.
Inspection Scope A number of industry problems with the hardening or contamination of lubricants have been identified in NRC Ir4 formation Notice 95-22. Since Westinghouse recommends that the lubricants used in the operating breaker mechanisms is conservatively functional for 10 years, the licensee initiated this PER since some of the Sequoyah breakers have been in service since the early 1970's without replacing the lubricant.
Also, during regularly scheduled maintenance, the DS breakers experienced sluggish mechanisms due to hardening grease. The inspectors reviewed corrective actions related to the above deficiencies.
b.
Observations and Findings As part of the PER the licensee performed an operability evaluation that showed that the plant could continue to operate safely without replacing the DS breakers or their operating mechanisms immediately. A previous PER issued in 1995 had identified 2 failures of DS breakers to trip in the last 8 years. Since 1995 there had been no additional failures. Also, in the last 9 years, there had been no failures due to hardened grease. Even though there was no equipment declared inoperable, the system engineer recommended that the problem be corrected in a timely manne *
.
,
i
The licensee has prepared a corrective action plan that contains 43 action items to
.
address the lubrication problems with DS breakers. This plan addresses both 6900
'
volt and 480 volt DS breakers. This corrective action plan is presently under review within the licensee's approval process. The basic plan is to replace the operating
!
mechanisms of the 480 volt DS breakers important to operations within 2 years and to replace the remainder of the mechanisms for the 288 breakers within 5 years.
Westinghouse will refurbish the mechanisms that are removed.
c.
Conclusions Based on this review, the team concluded that the licensee's operability review and corrective action plan had been adequately performed.
i i
-
E.3 Equipment Walkdown Results The inspectors performed equipment walkdowns, and conducted interviews with system engineers, design engineers, and maintenance personnel. Observation and inspection findings from these walkdowns are documented below.
E3.1 Open and Unprotected Vent Port a.
Inspection Scope The inspector walked down and reviewed several work packages with the System Engineer who was responsible for the Condenser Cooling Water System.
i b.
Observations and Findinas
,
During the walkdown the inspector identified an open and unprotected vent port in the
'
Condenser Cooling Water (CCW) intake level transmitter manifold (0-LT-27-133) on Panel (1-L-1478) in the Condenser Cooling Water Pump House.
'
c.
Conclusion
.
l In response to identifying the open and unprotected vent port, the System Engineer immediately prepared a work request (WR C356027) to install a vented plug to prevent debris intrusion into the valve manifold,
-
t E3.2 Excessive Leakage at CCW Bearing Lube Water Pump B a.
Insoection Scope
.
The inspector walked down and reviewed several work packages with the System j
'
Engineer who was responsible for the Condenser Cooling WMer System.
i s
i
_. _. _.'
.
b.
Observations and Findinas During the walkdown the inspector identified excessive leakage (as confirmed by the System Engineer) coming from the Condenser Cocling Water System Bearing Lube Water Pump B (0-PMP-27-428) which was spraying water on a portion of the abandoned Caustic System No. 28 located nearby. After further review it was determined that a Problem Evaluation Report (SQ960852PER) was previously initiated on April 1,1996 describing the problem and a Work Request (WR C357870) was initiated on June 19,1996 recommending to Engineering that a T-DCN be issued to install "Rainsflo" seal packing to correct the probier.i. When the inspector contacted Engineering to determine the status of the T-DCN, he was informed that the pump seal replacement was not currently scheduled since this was considered equipment
_
enhancement and that there were other current priorities.
c.
Conclusion it appears that excessive time has elapsed since the origination of the PER without any work being done to correct the excessive leakage. The license did not provide any justification for considering the suggested seal replacement an equipment enhancement thus delaying any corrective action.
E4 Engineering Procedures and Documentation E4.1 Procedural Guidance For Maintenance Versus Desian Chance Activities a.
Inspection Scope The inspector reviewed several work packages associated with the Condenser Cooling Water System and the Generator Cooling Water System that were associated with trip sensitive BOP components and subcomponents.
b.
Observations and Findings i
The inspector determined that there was no procedural guidance provided to either the Maintenance Department nor Nuclear Engineering (NE) for determining maintenance activities versus design change activities for work packages involving two inch and smaller BOP field routed and field supported piping and equipment.
When only vender catalog information was available and multiple equipment replacement options were offered, there were no instructions for Maintenance to obtain Engineering approval before replacing one equipment configuration option with
'
another. Although it appeared to be standard practice for Maintenance to obtain Engineering approval prior to substituting a different equipment configuration option, one individual interviewed in the Maintenance Department felt it was acceptable for Maintenance to select any such option if so provided by the vender catalog. Another individualin Engineering stated that substituting catalog configuration options were not
-
.
considered design changes as long as they do not invalidate the original design basis.
However, he was uhable to identify the procedural guidance for making such a determination.
c.
Conclusion The inspector concluded that the licensee had not developed procedural guidance to clarify for Maintenance and NE what constitutes a maintenance activity versus a design change activity for two inch and smaller BOP field routed and field supported piping and equipment when control drawings do not exist. The development of procedural guidance is an Inspection Follow item (50-327,328/97-02-02).
.
4.2 Procedural Guidance For Documentina Oriainal Eauipment Confiaurations a.
Inspection Scope The inspector reviewed several work packages associated with the Condenser Cooling Water System and the Generator Cooling Water System that were associated with trip sensitive BOP components and subcomponents as noted above in 4.1.
b.
Observations and Findinas The inspector determined tha! there wa no procedural guidance provided to either the Maintenance Department or iC for documenting the original equipment configurations subject to work packages involving two inch and smaller BOP field routed and field supported piping and equipment when control drawings did not exist.
Furthermore, it was determined in one example that the original configuration apparently was not documented. Maintenance personnel were asked how a determination of the equipment original configuration could be made, assuming that incorrect parts had been installed during a previous maintenance activity. The results of this discussion revealed that there were no established program controls to ensure that inadvertent configuration changes were not made during maintenance activities for field routed and field supported piping and equipment.
c.
Conclusion The inspector concluded that the licensee had not developed procedural guidance to clarify for Maintenance and Engineering what constitutes documenting original equipment configurations subject to work packages involving two inch and smaller BOP field routed and field supported piping and equipment when control drawings do not exist. The development of procedural guidance in another example of IFl (50-327,328/97-02-02).
'
.
,
.
E5 luality Assurance in Engineering E5.1 Escalation Report Nuclear Assurance & Licensing: PER No. SQ960393PER a.
Inspection Scope The inspector reviewed the Escalation Report for PER No. SQ960393PER, dated February 6,1997, in order to verify the adequacy of the response from the Operations manager for the site NA&L organization's rejection of the corrective actions for Sequoyah Nuclear Plant -1996 Reliability Report, Finding number 1, operation support.
b.
Observations and Findinas
,
~
On December 6,1996, the site NA&L organization rejected the corrective action plan for PER No. SQ960393PER based on a review of the corrective actions for Finding
,
number 1 proposed by the Operations manager. This rejection was based on NA&L review of selected corrective actions for Finding No.1 only. NA&L recommended that
j the responsible organizations perform a review of the corrective actions for Findings
'
No. 2,3,4, and 5 to ensure that the proposed corrective actions addressed all causal factors. On January 17,1997, a response to the rejection of the correceve action plan on December 6,1996 was provided to the site NA&L organizaticn by the Operations manager. The response included a summary statement that the information provided
supports the conclusion that the rejection was without a basis. NA&L reviewed the q
response and concluded that the corrective action 1 portion of rejection 3 was missing
.
from the response. Additionally, the recommendation provided in the December 6,
"
1996 was not addressed in the response. On January 31,1997, NA&L rejected the response for a second time. NA&L advised that the closure verification of PER No.
!
SQ960393PER by the responsible organizations was required to address all corrective
!
actions developed as a result of identified causal factors to ensure adequate i
recurrence control of the deficiencies listed in the SQNP-1996 Reliability Study.
The inspectors conducted interviews with the NA&L manager and concurred with the
'
NA&L's staff observations in that similar deficiencies had been identified by the inspectors during review of the proposed corrective actions for reliability report finding d
i number 4, i.e., corrective action 14-1,14-2,14-3, and 15. Based on this discussion
,
the inspectors were informed that a first level escalation of PER No. SQ960393PER would be initiated by NA&L to ensure compliance with the requirements of the corrective action program during implementation of the recommendations contained in i
SONP 1996 Reliability Report.
The first level escalation resolution due date was February 13,1997. On
,
February 14,1997, the inspectors were informed that the Operations manager had responded and provided the information requested in the escalation. A review of this information was scheduled to be performed by NA&L.
i
.
-
.
c.
Conclusion
.
The inspectors concluded that the inspection findings related to corrective actions 14-1,14-2,14-3, and 15 corrotorated NA&L inspection findings which resulted in two rejections and one first level escalation of PER No. SQ960393PER.
C Exit Meetina Summary The inspection scope and results were summarized with those persons indicated on February 14,1997. The inspector discussed the first level escalation of PER No.
SO'60393PER and the corrective actions required by the responsible managers.
Th management was informed that the inspection findings for corrective actions 14 and 15 corroborated the results of NA&L's review of the corrective actions for finding
,
number 1. Objective evidence reviewed for implementation of corrective action 16 involving hardware reliability issues indicated that the licensee was following a sound technical approach in the disposition of degraded and nonconforming equipment. The safety assessments, root cause analyses, and apparent cause analyses reviewed were concise and reasonable; the recommended fixes and corrective actions were appropriate. No dissenting comments were received frcm the licensee. Proprietary information is not contained in the report.
Items Opened / Closed / Discussed Type Number Title i
URI 97-02-01 Installation of Nnn QA Material for QA Material. (E1.3)
IFl 97-02-02 Develop procedure guidelines.
.
(E4.1, E4.2)
Partial List of Persons Contacted
.
R. Adney, Site Vice President
- B. Atsup, QA Supervisor, Engineering Section
- J. Bajraszewski, Licensing
- L. Bryant, Outage Director
- M. Fecht, NA&L Manager
- T. Flippo, Site Support Manager
- D. Lundy, Nuclear Engineering / Program Manager
- R. Profitt, Licensing
- P. Trudell, Technical Support / Program Manager
- Attended exit interview
-
-
.....
..- - -.-
_ _..
-
-.._ -
.
-
.
...
.
Inspection Procedures Used IP 37550 Engineering IP 40500 Effectiveness of Licensee's Controls in Identifying, Resolving, and (
Preventing Problems i
Acronyms l
l BOP Balance of Plant CCW Condenser Circulating Water CFR Code of Federal Regulations DCN Design Change Notice GE General Electric i
LIC Level Indicating Controller NA&L Nuclear Assurance and Licensing
)
NPRDS Nuclear Plant Reliability Data System PER Problem Evaluation Report OA Quality Assurance
RFO Refuelling Outage TROI Tracking and Reporting of Open items TVA Tennessee Valley Authority TS Technical Support WA Work Arounds WO Work Order WR Work Request