IR 05000440/1993019

Insp Rept 50-440/93-19 on 930920-24.Violations Noted.Major Areas Inspected:Engineering & Technical Support & Related Mgt Activities

ML20059H542
Person / Time
Site:	Perry
Issue date:	10/21/1993
From:	Hausman G, Nejfelt G, Pegg W, Shafer W, Yin I NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:
Shared Package
ML20059H503	List: IR 05000440/1993019 ML20059H500 ML20059H515
References
50-440-93-19, NUDOCS 9311100118
Download: ML20059H542 (10)

Text

. _ - . _ .- . . . . - - - . . - - - - . - .- _

i i

l

-

. ,

~!

U. S. NUCLEAR REGULATORY COMMISSION  ;

i REGION III  !

!

!

Report No. 50-440/93019(DRS)  ;

e Docket No.50-440 Licens'e N NPF-58 l

Licensee: Cleveland Electric Illuminating Company Post Office Box 5000 i Cleveland, OH 44101 l Facility Name: Perry Nuclear Power Plant j l Inspection At: Perry Nuclear Power Plant, Perry, Ohio f

.!

Inspection Conducted: September 20 - 24, 1993 l

l l

l Inspectors: W?cNrfe.6. h e )o/2.1/93 l l G. HausmarO V Date  !

!

Wh T. 7h \O/2I[93 i W. Pegg v0 Date

!

G.*Nejfelt 5khh /d Date'

l '

3 N (  !

,

'

bl. be /c4 1b 10 /2i /93 I. Yin OO V Date  ;

Approved By:

'W. D. SMfer, Cptef

/ A M //S df Oitd  ;

Maintenance and Outages Section '

t

'

Inspection Summary Inspection conducted SeDtember 20 throuah 24. 1993 (Report N /93019fDRS))

Areas Inspected: Special announced team inspection of engineering and technical support and related management activities. The inspection was-  !

conducted utilizing portions of inspection procedure 37700 to ascertain whether engineering and technical support was effectively accomplished and assessed by the license i

'

!

i 9311100118 931105 PDR ADOCK 05000440

! O PDR s (

. - , - . - , _ . . _ . . _ . . _ - - . _ . _ . . . , _ . . . _ . . .__ ,. ., ..,.... _ ,.. m ,,.. . . . - .

Inspection Summary 2 Results: One violation with two examples of inadequate corrective action and four inspection followup items were identified.

System engineers were technically competent and provided support to operations and maintenance, although significant reactive responsibilities diverted attention away from long-term performance issues and contributed to a substantial work backlog. Efforts by plant management to foster teamwork, reduce workload, and support the plant are positive steps, but have occurred too recently to allow for adequate assessment. Communications between and within plant organizations appeared adequate. Overall plant housekeeping conditions were acceptable; however, contamination still exists in many area l i

i I

l l

l l

l l

l

_ _ _____

k DETAILS

,

' Principal Persons Contacted The Cleveland Electric illuminatina Company (CEI)

l * D. Igyarto, Plant Manager

• N. Bonner, PNED Director -

• K. Pech, PNAD Director

• W. Coleman, Engineering Support Manager  !

• R. Tadych, Electrical Design Section Manager

• F. Von Ahn, System Engineering Section Manager

• J. Lausborg, PNAD Acting Manager

.

• A. Migas, Plant Engineering Response Team Lead Engineer

• R. Gaston, Compliance Engineer

• L. Routzahn, Compliance Engineer U. S. Nuclear Reaulatory Commission (NRC)  !

• B. Burgess, Chief, Operational Programs

• T. Vegel, Resident Inspector

• Denotes those present at the exit meeting on Septatber 24, 1993.

Other persons were contacted as a matter of course during the inspectio .0 Licensee's Actions Reaardina Previously Identified NRC Findinas ,

(Closed) Unresolved Item (440/92014-01(DRS)): This item related to corrective ,

actions taken for three personnel errors involving the preparation of complex '

instrumentation and control testing procedures. The inspectors reviewed the licensee's response dated November 1992 and verified appropriate training records. The inspectors concluded that adequate corrective actions had been administered and that no further concerns were identified. This item is close .0 Inspection Ob.iectives The objectives of the inspection were to determine if engineering activities that supported Perry Nuclear Power Plant (PNPP) were effectively coordinated, controlled, and implemented. The inspectors focused on design changes and modifications, identification and resolution of technical issues, communications, and management support. This was accomplished by interviewing selected personnel (including engineers and engineering managers), performing system walkdowns, and reviewing records, procedures, and associated documentatio . Observations of Plant Conditions

,

During September 21-22, 1993, the inspection team members performed walkdowns l of accessible portions of safety-related systems with the responsible system '

engineers (RSE) to observe the material condition, look for indications of )

equipment problems, and discuss the system performance and operation with the )

RSE The inspectors noted that a significant number of systems were not readily i accessible due to contamination of those areas and corresponding dress out '

requirement Some of these areas included the RHR pump rooms, control rod drive areas, and the high pressure core spray (HPCS) injection valve roo ;

This illustrated the continuing contamination problems that have been  ;

experienced at PNP :

!

The inspectors noted that deficient equipment, components, and conditions were ;

not identified by deficiency tags, but were identified on a computer database :

accessible to all plant personnel. This made identification of deficient l conditions during walkdowns difficult, since it was not immediately apparent t whether or not a condition had been identified. To determine whether or not !

conditions observed during the walkdowns had been identified, the RSEs had to i log onto a computer and find out if the condition had already been documente l This labor intensive process had the potential to result in conditions not !

being identified in a timely manner if individuals assumed that something has i already been identified and did not take the time to verify i For example, i during the RCIC system walkdown, the inspectors observed oil leaking from the !

suppression pool cleanup pump. The RSE checked the computer database,  !

discovered that this condition was not documented and entered the deficiency into the databas .

i Further discussions with licensee personnel revealed that a deficiency tagging ;

program had been recently developed and was approved for implementation i beginning in October, 1993. This program will include all previously  !

'

identified, open deficiencies and should resolve the shortcomings of the current system. This is considered an inspection followup item (50-440/93019-Ol(DRS)). ,

! Enaineerina and Technical Support

,

Engineering and technical support at PNPP was provided by the Perry Nuclear !

Engineering Department (PNED). PNED was divided into four separate sections; !

the system engineering section, the electrical design section, the mechanical ,

design section, and engineering support. Most of the routine engineering '

support to plant organizations was provided t>y the system engineering sectio i The responsible system engineers (RSEs) interviewed were knowledgeable of the l operation and material condition of their systems. During interviews and  !

walkdowns, RSEs were able to answer all queries and were able to discuss I system operational history, past modifications, current problems, and planned l modifications in a depth sufficient to resolve all of the team's question RSEs were also aware of industry and regulatory concerns that impacted their systems and were taking actions to resolve the concerns where necessar [

2 ,

,

i

_ _ _ . - , , , -

-

_ _

,_ , '

Interaction and cooperation between RSEs and other industry personnel was evident in the resolution of technical issue Examples of this included interactions between an RSE and an individual at another utility to discuss issues and problems relating to the reactor vessel level indication system modification required by Bulletin 93-03 and an RSE that worked with INPO and f several contractors to improve the accuracy and sensitivity of the instrumentation used to measure containment hydrogen concentration.

The licensee was not able to produce any procedures delineating the required functions and responsibilities of the responsible system engineers (RSEs). A number of Plant Engineering Guidelines (PEG) outlined the possible tasks RSEs could be called upon to perforn, but there was no procedure that required even a minimum of essential tasks to ensure that short and long term operability of the systems was being addressed. PEG-004, System Engineering, revision 2, stated that it was the function of the system engineering leads to ensure that work priorities within the groups were established to match project goals and priorities. While this is expected to some extent, in the recent past, the project goals and priorities have been reactive to operational events and did ,

not allow RSEs the opportunity to address less urgent, but important tasks.

One task that was neglected during this time was the performance of documented system walkdowns, a task stressed by the guideline. Due to the limited scope of this inspection, the team did not review the prioritization of workload in relation to short term and long term support of the systems; however, the team will examine this area in future inspections and considers this to be an inspection followup item (50-440/93019-02(DRS)). t A review of walkdowns documented in the plant database disclosed that, of the walkdowns performed and documented by system engineers and managers, good findings resulted from the walkdowns and support to the plant was evident. On the other hand, only a small percentage of system engineers performed formal walk downs on accessible portions of their systems frequently and documented the observations. Although most of the system engineers interviewed said that they walked down their systems when possible, it was difficult to gauge the performance of walkdowns and the ability of the remaining system engineers to monitor system condition and detect abnormalities since there was little supporting documentation.

A common element among the RSEs interviewed was the large percentage of time spent on reactive work. Many were required to respond to plant events and problems rather than being afforded the time to improve long term system reliability. Management has recognized this condition and has taken steps to try to improve the situation. First, a forced outage, designated "8F", was scheduled to give plant personnel the opportunity to reduce the work backlog and improve system reliability. Second, the Perry Engineering Response Team !

(PERT) was developed to respond to plant problems and operational concerns, so l that RSEs and design engineers will not be directly responsible. The PERT is l currently chartered to remain in existence until the end of refueling outage '

four (4F), at which time the effectiveness and impact of the PERT will be assessed. At this time, no conclusions can be drawn about the effectiveness of these actions; however, these steps are encouragin I i

. Desian Chances and Modifications The team reviewed Design Change Package (DCP)92-130, which required the replacement of all four pressure gauges on the demineralizer system. These gauges were originally installed at the pump discharge, and were damaged due to cumulative vibration loadings during pump startups. The new gauges were also installed at the same location, but were liquid filled and should not be susceptible to the same loading. No deficiencies were identified during the inspectio .4 Corrective Action The team reviewed the corrective actions taken in response to conditions identified in condition reports. The findings of these reviews follow O CR 93-180 - Six platform beams supporting the main steam safety relief ,

valves were found to have an inadequate design in August 1993. The CR '

was still open at the time of the inspectio The team reviewed corrective actions and some of the re-analyses of the problems, and raised a concern about the load combination of all dynamic loadings, such as relief valve lifts and seismic accelerations. The methodology was not discussed in the USAR; however, the original load combination was based on the absolute sum method. The present re-analysis was based on Square Root of the Sum of Square (SRSS) method, which was much less conservative. The matter will be conveyed to NRC NRR for review. This will ,

be considered an inspection followup item pending further NRC review (50- '

440/93019-03(DRS)).

O CR 92-277, CR 93-099, and CR 93-100 - CR 92-277 was issued on December 5,1992, and closed on January 15, 1993. The safety related L room cooler fan for the residual heat removal (RHR) pump B was found damaged due to incorrect installation of the shaft bearing assembly locking collars. The licensee inspected seven similar units and determined that this was an isolated cas CR 93-099 was issued on May 19, 1993. Approximately six months after the identification of the failure of the RHR B cooler fan, the licensee found that the RHR A cooler fan was severely damaged due to incorrect installation of the locking collars. The fan housing was ruptured, and one of the support brackets was sheared. The bracket breakage was initiated at a sharp corner where a required stress relief hole was missin CR 93-100 was issued in conjunction with CR 93-099 on May 20, 199 During re-inspection of room cooler fans, the licensee found that stiffener plates stitch welded to the channel type fan structures were missing. The licensee determined that the stiffener plates must have broken off as a result of the excessive vibration reported in CR 92-27 I l

-

l The licensee determined that there were three potential problems requiring !

followup actions. These were incorrect installation of bearing lock collars, l damage to the stiffener stitch welds, and missing stress relief holes. At the i time of the inspection, only the latter problem was correcte The team determined that the corrective actions were inadequate in the i following areas:  !

o After the RHR room B cooler fan was found damaged, the RHR A cooler fan l was observed working properly. Six months later, the RHR A fan was ,

severely damaged due to the same cause. The corrective action plan, at I the time of the inspection, was to conduct visual surveillances. The decision to delay disassembly and inspection of the bearings and l'

performance of magnetic particle non-destructive test (MT) for the stitch welds until October 1993 and February 1994 was inappropriate i based on the previous failure !

O The corrective actions in CR 93-099, June 24, 1993, included site j generation of Work Requests (WRs) "immediately" to inspect the bearings :

installed prior to December 1992 for both M39 and M28 fan units of i similar size and design characteristics. At the time of inspection, September 22, 1993, no WRs were written for the two emergency closed cooling area cooler fans'(M28 units), and the reactor core isolation cooling pump room cooler fan (M35).

O Although MT of the stiffener stitch welds was a part of the overall inspections, no such requirement was included in written WRs for the low pressure core spray and RHR C cooler fan unit The lack of engineering justification to postpone inspections for the safety

'

related room cooler fans, inadequate implementation of planned corrective action, and inadequate corrective action scope is considered to be an example of a violation of 10 CFR 50, Appendix B, Criterion XVI requirements (50-440/93019-04a(DRS)).

All required WRs were generated prior to the completion of the inspectio O CR 89-336, CR 89-384, and CR 93-119 - CR 89-336, issued on August 21, 1989, reported that during the quarterly HPCS pump surveillance test, the two diaphragm valves on the connecting radwaste system (RW) were over pressurized, and water was observed spraying out from the seal The same problem was reported in CR 88-083. During the HPCS pump '

testing mode, the pump obtains suction from, and discharges back to, the Condensate Storage Tank (CST). The RW system is connected to the test return line without boundary separation. Water hammer noises were heard l and line movements were observed near the RW and HPCS piping during the end of the test when the HPCS isolation valves were shut and the pump !

was turned of l

!

l 5 l

. __ _ - _ _ _ _ __ _ _ . _ _ __.___ .. . _ _ _ _ _ _

_ - _ - _ - .- __ _ . .

' l

-

l l

CR 89-384, issued on October 29, 1989, reported the same problems as CR 89-j 336.

i CR 93-119, issued on June 18, 1993, reported the usual water hammer problem after the isolation valves shut and low system pressure alarm for several -

minutes. Upon review of Emergency Response Information System (ERIS) trace, !

1 the licensee found HPCS relief valve lifting prior to opening of the minimum !

,

flow (pump dead head protection system) valve. Prior to the relief valve !

lift, the HPCS pump was dead headed for approximately four seconds.

'

.

'

The team reviewed the records, studied the ERIS trace, and interviewed the l responsible system engineer, and concluded that the licensee's corrective !

actions for these problems were inadequat !

I l o The licensee's acceptance of water hammer in the RW and HPCS systems was l

unacceptable. Since 1988, the licensee adjusted pipe restraints and

!

supports and replaced an orifice in the HPCS system, but failed to '

resolved the problem. The licensee had accepted this phenomenon to the !

point that long term plastic spray catchers were installed at the two i diaphragm valves. There was no monitoring of the leakage to determine -

if the condition was getting worse, and the last time the system :

engineer observed the water hammer and valve discharge was two to three t years ag ;

i j o The licensee's actions to evaluate the effects of HPCS dead heading !

I events were inadequate. The system engineer stated that the 1992 ERIS l

,

trace also showed the same proble !

!

! (1) The licensee failed to determine how many times in the past the ;

l HPCS pump was dead headed, and for what duratio '

,

'

(2) The licensee failed to assess the root cause of the relief valve !

lifting prior to the opening of the minimum flow valv '

(3) The licensee believed that the dead heading of HPCS pump was the

GE design intent through interpretation of the GE system design i y

and performance descriptions, but failed to confirm this ,

interpretatio !

J (4) The licensee failed to investigate whether similar problems existed in other safety systems with minimum flow passage desig t i I (5) The relief valve was set to lif t when the pressure reached 1560 {

psig plus or minus 47 psig. The licensee failed to evaluate the !

3 effect of possible 2000 psig pressure spikes on the system i j components and instrumentatio !

! (6) During the HPCS pump test conducted in August 1993, the licensee

changed the pump shutdown procedure by throttling the closure of ,

j the two isolation valves and ensuring the minimum flow was '

established prior to shutting down the pump. The pump dead i

I l ,

6 l

!

I

)

$

l  !

-_.-

'

.

.. _ _

. -

i

'

.

I heading problem was resolved; however, there was no observation made on the RW valves to see if the slow closure of the HPCS i valves had also resolved the water hammer and related RW valve j

'

l spray proble The inadequate licensee action to resolve the water hammer problems on the RW and HPCS systems, and to evaluate the effects of the HPCS system deficiency and pump dead heading problem is considered to be an example of a violation of l 10 CFR 50, Appendix B, Criterion XVI requirements (50-440/93019-04b(DRS)).

l Self Assessments  ;

In responding to the violation item, the licensee presented to the team their i internal audit report, "Second Quarter 1993 Trend Report for Licensee Event I and Condition Reports, Evaluation Period April 1 - June 30,1993," distributed on September 16, 199 Under " Effectiveness of Previous Corrective Actions - CRs Caused by I Incomplete / Ineffective Corrective Actions," the two issues cited in the violation item were identified by a licensee audito For CR 93-099, the auditor concluded that the latest licensee corrective action was adequat In ;

developing a corrective action plan, the engineering staff failed to recognize

'

that visual surveillance of fans could not detect the gradual galling of the fan shafts by the loosened inner bearing rings, which are caused by improper lock collar installatio ,

The engineering staff did not conduct a detailed review of the available information or attempt to correct the problem in a timely manne O The severity of the shaft galling increased with fan operation time as l seen in the cases reported in CR 92-277 and CR 93-099. It was essential !

to correct the potential improper lock collar problem in a timely manner.

l 0 The potential crack formation at the stitch welds due to fan vibration could cause loosening of the stiffener plate. As the structure rigidity decreased, fan case body flexing occurred, and resulted in multiple fan unit internal damages as reported in CR 92-277. Again, it was essential to identify and correct the potential problem in a timely manner to prevent similar damages from occurrenc O The corrective actions committed in CR 93-099 was not implemente Not all the WRs were issued in a timely manne Although the licensee's inadequate corrective action to prevent water hammer occurrence at RW and HPCS systems and resolve the RW diaphragm valve leakage 4 during the HPCS pump tests was identified in CR 93-119, the engineering staff j failed to identify some important technical consideration !

l I

l l

, _ _

.

O The root cause of the design or installation deficiency involving relief valve lifting prior to the minimum flow valve opening when the HPCS was dead heade O Failure to investigate whether the HPCS dead heading was a part of GE design intent. Whether or not similar conditions existed in the other system O Failure to evaluate the effects of possible 2000 psig pressure spikes on the affected HPCS system components and instrumentation.

l The team concluded that these engineering evaluations lacked thoroughness and were absent of any serious technical analysis. This is significant because, for the small sample of corrective actions the team reviewed, the team found significant weaknesses in every evaluation. This weakness in engineering evaluations and corrective actions will be considered an inspection followup item and will be reviewed at a later time (50-440/93019-05(DRS)). Inspection Followup Items j An inspection followup item is a matter that requires further review and

'

evaluation by the inspector, including an item pending specific action by the licensee. Inspection followup items disclosed during this inspection are discussed in Sections 3.1, 3.2, 3.4, and 3.5 of this repor .0 Exit Meetino The inspectors met at the Perry Nuclear Power Plant with licensee representatives (denoted in Section 1) on September 24, 1993, to summarize the purpose, scope, and findings of the inspection. The inspectors discussed the likely informational content of the inspection report with regard to documents and processes reviewed by the inspectors during the inspection. The licensee did not identify any information obtained by the inspectors during the inspection as proprietary.

i

l ..