IR 05000261/1993012
| ML14182A250 | |
| Person / Time | |
|---|---|
| Site: | Robinson  |
| Issue date: | 07/27/1993 |
| From: | Peebles T, Rogers W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II) |
| To: | |
| Shared Package | |
| ML14182A248 | List: |
| References | |
| 50-261-93-12, NUDOCS 9308060075 | |
| Download: ML14182A250 (24) | |
Text
01,~
UNITED STATES
NUCLEAR REGULATORY COMMISSION REGION 11 101 MARIETTA STREET, ATLANTA, GEORGIA 30323 Report No.:
50-261/93-12 Licensee: Carolina Power and Light Company P. 0. Box 1551 Raleigh, NC 27602 Docket No.:
50-261 License No.:
DPR-23 Facility Name: H. B. Robinson Inspection Conducted:
June 14 through July 2, 1993 Inspector:
_)
. _N____________
W. G. Rogers, geam Leader te S gned Accompanying Personnel: L. King L. Mellen C. Rapp J. Shackelford NRC Consultant:
P. Kidston Approved by:
y
_
_
_
_
_/
J T. A. Peebles, Chief ate Signed Operations Branch Division of Reactor Safety SUMMARY Scope:
This special, announced inspection was conducted in the area of review of service water system operational performance, conducted June 14 through July 2, 1993, in accordance with NRC Temporary Instruction 2515/11 Results:
The team concluded that the licensee had provided minimal assurance of service water system (SWS) operability. Also, there was little design margin in the SWS at the most limiting design bases conditions with the emergency diesel generators being the critical component Engineering - Due to significant communication weaknesses between design engineering and the site's technical staff, the calculations associated with the SWS had been occasionally invalidated. Reanalysis of the corrected design inputs resulted in continued system operability. Also, in one postulated 93 8060075 930730 PDR ADC 05000261 G-PDR
scenario, design engineering had not provided adequate instruction to the site's technical staff to maintain the system within the analyzed bound Configuration control weaknesses of design documents, though minor in nature, were apparent and had programmatic implication Generic Letter Actions - The team considered H. B. Robinson's response to be sufficient inmost aspects. However, the licensee's efforts in monitoring SWS heat exchanger performance, the most critical aspect of the generic letter, were weak. A heat exchanger inspection program had been implemented instead of performance testing. The inspection program lacked acceptance criteria and contained poor documentation of the as found conditions. Also, the licensee's original initiatives at performance testing did not appear to fully review all performance testing possibilitie Testing - The Technical Specification surveillance program was properly implemented. The inservice test program was of adequate scope and, the computerized trending of the results was a strength. However, the acceptance criteria used for pump vibration was not always conservative. Also, in several instances I&C personnel had failed to perform the required evaluations for instrument calibration checks found out of toleranc Operations - The abnormal operating procedure for loss of the SWS was technically inadequate and appeared to be an indicator of procedure preparation and validation programmatic weaknesses. The operating procedure for throttling valves in the SWS was also inadequate. Operator implementation of SWS valve lineups was consistent with design documents but not with the operating procedure valve checklist. Licensed operators were properly trained on system operation and entered the limiting condition for operation when appropriate. Non-licensed operator rounds and staging of emergency equipment were goo Maintenance - The material condition of the SWS was adequate with some weaknesses noted in SWBP B's performance and in the reduced flow to the penetration coolers due to fouling. Preventive maintenance activities appeared of adequate scope and properly implemented. Corrective maintenance activities suffered from poor documentation of the work activities. Use of a laser for pump alignments was a strengt Safety Assessment - Collective internal assessments of the SWS paralleled the team's findings in most respects. The condition adverse to quality system appeared to properly identify performance, design and programmatic weaknesses within the organization. However, sometimes slow root cause analysis was noted with corrective actions addressing the consequences rather than the cause of the adverse conditio Five cited and three non-cited violations and two inspector follow-up items were identifie REPORT DETAILS 1. Licensee Employees
- S. Billings, Regulatory Compliance Technical Aide
- G. Bussell, Technical Support Project Engineer M. Clouse, Operations
- D. Crook, Regulatory Affairs Senior Specialist C. Dietz, Site Vice-President
- W. Farmer, Engineering Programs Manager
- W. Flannagan, Acting General Manager
- W. Gainey, Service Water Inspection Team Manager
- M. Hammack, Mechanical Engineer
- J. Harrison, RE&C Manager
- A. Lewis, Senior Engineer
- M. McConnel, Technical Support Engineer
- P. Musser, Engineering and Technical Support Manager
- J. Padgett, E&RC Manager
- M. Page, Civil/Mechanical Engineering Manager S. Pruitt, Inservice Test Engineer
- R. Steele, Acting Maintenance Manager
- R. Wallace, Acting Operations Manager
- D. Waters, Regulatory Affairs Manager NRC Personnel
- S. Bajwa, NRR Acting Project Directorate II Manager
- J. Jaudon, Reactor Safety Deputy Division Director
- L. King, Reactor Inspector
- P. Kinston, Atomic Energy of Canada Limited
- L. Mellen, Reactor Inspector
- C. Ogle, Resident Inspector
- W. Orders, Senior Resident Inspector C. Rapp, Reactor Inspector
- W. Rogers, Team Leader
- J. Shackelford, Reactor Inspector
- Indicates those present at the exit meeting on July 2, 1993
Acronyms and initialisms used throughout this report are listed in Appendix. Inspection Scope and Objectives Numerous problems identified at various operating plants in the country have called into question the ability of the SWSs to perform their design function. These problems have included:
inadequate heat removal capability, biofouling, silting,.single failure concerns, erosion, corrosion, insufficient original design margin, lapses in configuration control or improper 10 CFR 50.59 safety evaluations, and inadequate testing. NRC management concluded that an in-depth examination of SWSs was warranted based on the identified deficiencie Report Details
The inspection team focused on the mechanical design, operational control, maintenance, and surveillance of the SWS and evaluated aspects of the quality assurance and corrective action programs related to the SWS. The inspection's primary objectives were to:
assess SWS performance through an in-depth review of the system's mechanical functional design and thermal-hydraulic performance including the content and implementation of SWS operating, maintenance, and surveillance procedures, and operator training on the SWS,
verify that the SWS functional design and operational controls could meet the thermal and hydraulic performance requirements, and that SWS components were operated in a manner consistent with their design bases,
assess the licensee's planned and completed actions in response to Generic Letter 89-12, "Service Water System Problems Affecting Safety Related Equipment," July 1989, and
assess SWS unavailability resulting from planned maintenance, surveillance, and component failure The areas reviewed and the concerns identified are described in paragraphs 4 through 11 of this report. Personnel contacted and those who attended the exit on July 2, 1993, are identified in paragraph 1. Details pertaining to GL 89-12 action items are attached as Appendix. Licensee Action on Previous Inspection Findings (Closed) IFI 50-261/91-22-01:
Review of completed EOP and AOP upgrade program. The implementation of the EOP and AOP program will be reviewed during the follow-up to the corrective actions for violation 50-261/93-12-04. Because the open item dealt with the same subject as the violation, this IFI is close. System Description As discussed in the licensee's design bases document and the USAR, the SWS is an open loop system taking suction from Lake Robinson and eventually discharging back into Lake Robinson. Four, one-third capacity motor driven pumps located in the intake structure supply water to a common, cross connected discharge header. All four SWS pumps are automatically started on a LOCA signal and two of the four are automatically started on a LOOP signa Two main headers, the North and the South, deliver water to the plant load Each main header has an automatic, self-cleaning strainer and is sized for the maximum discharge from two pumps. Each main header can cool all the loads in the turbine, auxiliary, and containment buildings except the main control room HVAC coolers which only receive cooling from the North header. The turbine building loads can be isolated from the safety related portion of the SWS by motor operated isolation valves located in each main header and the common discharge pipe to the turbine buildin Report Details
The turbine building isolation valves have automatic and control room closure capability. Safety related cooling loads in the auxiliary building include EDGs, SI pump thrust bearings, AFW pump bearing coolers, CCW heat exchangers, control room HVAC coolers, and numerous room coolers. The SWS also serves as an emergency backup water supply to the AFW pumps. Each main header has a full capacity booster pump capable of supplying water to all four containment fan coolers and motors and containment penetration cooler Only one booster pump is normally in operation. A sampling system exists to detect radiation from any leaking containment cooler. MOVs are provided to isolate any leaking coole. Generic Letter 89-12 Implementation The NRC issued GL 89-12, "Service Water System Problems Affecting Safety Related Equipment," requesting licensees take certain actions related to their SWS. These actions included establishing biofouling surveillance and control techniques, monitoring safety related heat exchanger performance, establishing a routine inspection and maintenance program, reviewing the design to assure intended safety functions could be accomplished, and training personnel in the operation, maintenance and testing of the SW The licensee's actions to GL 89-12 were sufficient in most aspects but, the suboptimal (heat exchanger inspection) of the five possible methods for monitoring safety related heat exchanger performance was selected by the licensee. The licensee initially attempted heat exchanger testing but abandoned that method when the results were not repeatable due to limited heat loads available. The team was not convinced that testing was infeasible. The licensee's inspection/cleaning program had numerous inadequacies due to the lack of acceptance criteria and the poor documentation of as-found heat exchanger conditions. The lack of acceptable heat exchanger performance testing and weak implementation of the heat exchanger inspection program provided poor assurance of heat exchanger performance. See Appendix B for details on each GL 89-12 Action Ite. Mechanical Design Review The team reviewed the mechanical design of the SWS, including the design bases, functional requirements, design assumptions, calculations, boundary conditions, analyses and models, to determine if the design met licensing commitments and regulatory requirements. The SWS capability to meet the thermal and hydraulic performance specifications during accident and abnormal conditions was reviewed. Selected stress calculations supporting seismic qualification, single failure vulnerabilities, flooding mitigation characteristics, selected modifications, instrumentation and setpoint selection associated with the SWS were reviewed. Also, the interface between design engineering and site technical support was reviewed in par Report Details
a. Engineering Interface The design engineering organization had not established appropriate controls to assure SWS calculation model assumptions and inputs remained valid. No formal controls existed when SWS pump head characteristics changed either through refurbishment activities or service degradatio Consequently, when SWBP B was rebuilt in 1992 the new head curve was not compared to the curve used in the calculation mode The new head curve was about 200 gpm less for the differential pressures that were used in the calculation mode The failure of design engineering to transmit minimum values for acceptable SWS pump reference curves to the site technical staff was the major causal factor associated with this inadequacy. Subsequently, the calculation was performed by the licensee using the lower SWBP head curve with acceptable result The licensee had previously identified this interface weakness in their self assessment process. In response, the license was scheduled to implement a formal process for communicating non-design change calculation results, analysis results and non-design change physical plant changes affecting design calculation inputs by December 15, 199 The team concluded that the licensee's corrective actions were appropriate for the circumstances. Failure to maintain such appropriate controls was a violation. However, based upon the corrective actions, the licensee's self identification of the matter, no similar violations identified by the team in the last two years, the lack of willfulness and the nonescalated enforcement nature, this is considered a non-cited violation as authorized under 10 CFR Part 2, Appendix C, Section VII. b. Waterhammer The team noted evidence of waterhammer, while in refueling outages but not during power operations. There were many reports of pressure gauges inside containment being over-ranged. No other damage had been observed, such as damaged piping supports or leaking cooler tubes. The condition had been identified by the licensee in 1992. The licensee speculated that there was voiding in the return lines from the HVHs, presumably during changes in SWS operation. The licensee closed the root valves to all these gauges, unless taking operator readings, to prevent further damage. The licensee indicated that further investigation into this matter would be conducted during the upcoming refueling outag c. Cooling Capacity Considerations The licensee's analyses and flow tests showed that, in certain SWS accident alignments, the EDGs would not receive design SWS flo Predicted flows in the event of a LOOP with only two SW pumps operating coincident with a LOCA resulted in 574 gpm and 552 gpm for EDG A and B respectively whereas design flow was 700 gpm. The corresponding EDG
Report Details
jacket water temperatures were estimated to be above the alarm point of 195*F but below the 205OF trip setpoint. This analysis assumed the EDG heat exchangers were operating at design efficiency. A 24-hour EDG test run during the upcoming refueling outage will provide additional insights into the acceptability of this assumptio An assumption of the licensee's SWS computer model for estimating cooling flow and temperature consequences was that turbine building isolation had occurred. This assumption could be invalidated by reduced flow through the CCW heat exchangers. Reducing SWS flow would increase pressure throughout the SWS allowing SWS pressure to recover before achieving turbine building isolation. Without turbine building isolation, the EDGs would not receive sufficient cooling to acceptably perform their safety function. Design engineering personnel had not established the requisite controls to ensure these valves were not throttled below a minimum value. This is identified as violation 50 261/93-12-01a, Failure to establish appropriate design control over SWS throttle valves. Subsequently, the licensee issued additional CCW throttle valve position instructions to operators which provided the appropriate header pressure contro d. Safety/Nonsafety Related Interface Isolation of the turbine building and the nonsafety related portion of the SWS was accomplished by three valves; one in the north header, one in the south header and one in the common header. Turbine building isolation could be accomplished automatically or by operator action from the main control room. Because the automatic isolation circuitry was not completely safety qualified, the licensee considered operator action as the design basis. However, as stated previously, the SWS calculational model assumed the turbine building was isolate Therefore, the nominal ten minutes assumed for operator action was not included. The licensee provided a PRA concluding that marginal safety benefit would be provided in fully qualifying the automatic isolation as safety relate e. Design Document Accuracy Minor errors were noted in the USAR and the SWS drawings during the design review and the detailed system walkdown. Several older, noncritical drawings were of inferior quality and illegible in certain areas. The more recent and critical drawings provided acceptable information. The licensee indicated the specific deficiencies would be appropriately dispositioned. Also, before team identification, the licensee had noted a larger population of either the same or similar weaknesses in the accuracy of the design documents. ACR 93-028 initiated on February 12, 1993, documented the specific errors. The ACR characterized the problem as associated with the quality of the design change process. The team concluded the corrective actions to the ACR were comprehensive and appropriate. This will be identified as IFI
Report Details
50-261/93-12-02, NRC review and follow-up of the licensee's corrective action implementation associated with configuration contro f. Single Failure Considerations The system was always operated with the two trains interconnected at the supply header of the SWS pumps and at the suction and discharge of the SWBPs. Adequate design features were installed for mitigation of piping rupture at these locations. The ability to supply main control room HVAC cooling via only one SWS header was already recognized in NRC licensing consideration g. Instrumentation and Setpoint Methodology Instrumentation for directly reading SWS flow was not installed; therefore, SWS temperature and pressure were used to infer flow. The pressure and temperature instruments used for determining SWS performance would be accurate and available during accident condition The CCW system had adequate flow instrumentatio The radiation level setpoint for containment fan and motor cooling water and the methodology used to calculate the setpoint and cooling water flow rate was adequat h. Modifications For the three SWS design changes reviewed, no problems were note However, the licensee had used Belzona to patch leaking "U" bend thru wall cracks in the HVH-7A, HVH-7B, and HVH-8 room cooler heat exchangers. This activity was handled as a repair and not as a modification. Therefore, there was no modification review or licensee requirement for a 10 CFR 50.59 evaluation. The licensee's purchasing organization stated that using Belzona as a pressure boundary material was a misapplication of the product. Also, the purchasing organization stated the manufacturer did not recommend using Belzona as a pressure boundary materia The team considered the "U" bend portion of the room cooler heat exchanger as part of the cooler's pressure boundary. The team concluded that the licensee did not document their evaluation of the use of Belzona, did not perform a 10 CFR 50.59 evaluation for the use of Belzona as part of the pressure boundary for three safety related heat exchangers, and performed a heat exchanger modification using repair procedures instead of modification procedures. This is identified as violation 50-261/93-12-03, Failure to follow the design change process when using Belzona as a safety related pressure boundar i. Seismic Qualification Essentially all of the safety related portion of the SWS was seismically qualified. The dam that forms Lake Robinson, while not constructed to
Report Details
seismic codes, was evaluated by earlier studies and determined capable of withstanding the DBE. All selected calculations supported seismic qualification. One mathematical error in the stress calculations for the EDG cross-connect line was observed. The licensee corrected the error and the results did not invalidate the seismic qualificatio j. Flooding The seismic qualification of the auxiliary building piping minimized the risk of flooding. Sump pumps, high level alarms and procedures for operator response minimized the consequences of flooding of the discharge header pit at the intake structure. No concerns were identifie. Operations The team observed licensed operator performance during routine and abnormal situations using the plant-specific simulator. Where simulator limitations prohibited, operator performance was assessed by walkthroughs and interviews. Operator performance was also assessed in the control room and during IST of the SWBPs. Several AOs were interviewed about their knowledge of equipment and location of applicable controls. The content of SWS lesson plans, OJT checklists, the system description, and student handouts were reviewed. Normal, alarm response and emergency procedures were reviewe Inclusive in the review of normal procedures were the controls for SWS flow variations due to changing climate conditions and the methods for routine SWS temperature and flow monitoring. Also, the team conducted a detailed SWS walkdown which included all AOP and EOP action Inspection results were as follows:
a. Operator Performance (1) Some operator actions on the simulator could not be properly assessed due to modeling limitations. These actions included response to a turbine building line break and swapping SW pump The simulator model had been validated for current training needs, but did not allow for further enhancement. The current SWS model lacked proper nodilization, had no elevation changes, had incorrectly modeled heat loads, accomplished automatic turbine building isolation via reactor trip instead of turbine trip, and included inaccurate SW pump dynamics. The licensee had recognized the SWS model limitations along with limitations with other systems and a general simulator upgrade had been approve Report Details
(2) During IST of the SWBPs on June 28, 1993, A~s completed testing the A pump and commenced testing the B pump before reviewing the A pump's data. Consequently, the team identified that the A pump had not met the flow rate acceptance criteria. Once identified to the licensee, appropriate actions were taken to declare the A pump inoperable and initiate review activities. The team concluded failure to review the A pump data before proceeding with testing of the B pump was a weakness. Licensee management independently identified this weakness and took appropriate corrective action (3) During startup of the SWS, the control room operator started SWS pump A. When the second SWS pump was started the operator did not take all the requisite precautions to ensure against waterhamme During discussions with personnel following this event, the team concluded the operators had sufficient knowledge of the SWS, but failed to correctly apply that knowledge during this event. No problems were observed during SWS shutdown on the simulato (4) Operator responses to SWS related alarms for loss of SWS, CCW heat exchanger high outlet temperature, turbine building line break and SWBP suction leak were adequate. The operators were knowledgeable about the necessary local actions to be performed and where those actions would be performe (5) AOs who were interviewed were knowledgeable of procedural steps assigned to them and understood the function of the SW b. Operations Training Material (1) Some significant informantion present in the lesson plans was not presented in the training material provided to the students. For example, there was no discussion of the common turbine building isolation valve's automatic transfer to MCC-9 on loss of MCC-10, the 12 psig low suction pressure trip for the SWBPs, and why a SWS pump discharge isolation valve was closed before starting the first SWS pum (2) The turbine building isolation circuit was incompletely described in training materia The system description and the training lesson plan did not describe the individual turbine trip signals to the isolation valve c. Procedures (1) System operating procedure, OP-903, revision 45, contained incorrect and incomplete guidance as follows:
Attachment 9.1, Service Water System Valve Checklist, listed valve SW-424 as throttled open. However, the design documents, including the P&IDs, identified the valve as full ope Report Details
Also, containment penetration cooler valves SW-442, SW-422, SW-446, SW-450, SW-454, SW-458, and SW-462 were identified as throttled on the P&Is, normally were throttled, and currently were throttled but were not designated as throttled in the procedure. This is identified as violation 50-261/93-12-01b, Failure to establish appropriate design control over SWS throttle valve As follow-up, the team reviewed the two most recent completed valve lineups and noted these valves were signed off without exceptio Failure of the operators to request and obtain a change to the applicable procedure section before continuing the valve lineup is a violation of 10 CFR 50, Appendix B, Criterion V, "Instructions, Procedures and Drawings."
However, there were no safety consequences associated with the operator actions and the licensee was taking adequate corrective action. Therefore, in accordance with 10 CFR 2, Appendix C, Section VII.B.1., this is considered a non-cited violatio *
There were inconsistencies between the precautions and limitations section and the body of the procedure about which parameters, flow or pressure and temperature, were to be used to set the maximum opening of the CCW return valve *
There was no guidance for initial positioning of the CCW return valves in attachment 9.1. This could result in operation of the SW pumps in an unstable region on the pump head curv *
The precautions and limitations referenced a reactor trip versus a turbine trip for automatic closure of the turbine building isolation valve (2) The operator actions, referenced indications, and probable causes associated with the SWS ARP were adequate. Alarm setpoints were consistent with the system description documen (3) The only AOP applicable to the SWS was AOP-022, "Loss of Service Water," Revision 9. A walkdown of AOP-022 and its attachments identified the following:
Attachment 2 step 6, which established cooling flow from the PWST to the SI pump thrust bearings via gravity feed could not accomplish its intended safety function. The step directed installation of a hose between the PWST and the SI pump thrust bearing cooling water inlet. The SI pump thrust bearing crossover piping was about one foot five inches above the PWST TS minimum leve Therefore, PWST water would not gravity drain. This is identified as violation 50-261/93-12-04a,
Report Details
Failure to provide adequate instructions in AOP-02 *
Section C and Attachment 3 aligned the fire protection system as backup cooling for the AFW pump and the control room HVA The licensee was not aware of the maximum potential pressure of the fire protection system as it related to this AOP. The pressure effects on the AFW pump and control room HVAC SWS heat exchanger had not been analyzed with the maximum potential pressure from the fire protection system pump Subsequent licensee review determined the effects on the AFW system were within the design of the system. However, the effects on the control room HVAC SWS heat exchanger were no The AOP did not include instructions to reduce the fire protection system pressure to within the design rating of the control room HVAC cooler. This is identified as violation 50 261/93-12-04b, Failure to provide adequate instructions in AOP-02 *
The required equipment to accomplish the AOP was staged and readily availabl (4) Because of the inadequacies noted above in AOP-022 the team reviewed the V&V process used for the backup water sources for AOP-022. The V&V was performed in accordance with administrative procedure OMM-043, "Verification and Validation,"
which provided adequate guidance. The team concluded the AOP-022 inadequacies occurred in the implementation of the V&V procedure and was not the result of specific procedural weaknesse (5) The reactor trip response and loss of all electrical alternating current were the only EOPs that required SWS manipulations. The procedural steps were clear, and the equipment was available to the operator d. System Walkdowns Valves and controls were labeled according to drawings and procedure However, there were minor weaknesses in the way equipment was designated including the abbreviation on some local equipment labels requiring the operator to trace the instrument lines to find the component's us. Maintenance The team reviewed maintenance history on selected equipment, maintenance procedures, completed work packages, the preventive maintenance schedule, preventive maintenance procedures, LERs, and a listing of all outage electrical and mechanical WRs from 1990 to present. Selected completed WRs were reviewed in detai The availability of spare parts was evaluated by reviewing the licensee's design basis data system. The inspection results were as follows:
Report Details
a. SWS Preventive Maintenance The preventive maintenance program was comprehensive and included repacking of all the SWS valves on a regular frequency, biological assessments for Asiatic clams, and internal piping inspections. The Shearon Harris biological assessment unit conducted routine inspections to determine if Asiatic clams were present in Lake Robinson and none were found. The team reviewed documentation and photographs of the inspection of the 30-inch and 18-inch pump discharge piping that were taken during the 1991 refueling outage showing cement patching of the concrete lined 30-inch piping. Photographs of the repairs were taken of the B header piping during the 1992 refueling outage which indicated the repairs experienced no deterioration. Further visual inspections were scheduled for the upcoming refuel outage along with some ultrasonic examinations (which will be performed on the piping downstream of the CCW exchanger return piping).
b. Spare Parts Availability There were few spare parts available for the butterfly valves, the predominate valve type in the SWS, due to the manufacturer having gone out of business. A purchase order authorized the procurement of a 30-inch, a 20-inch, and two 24-inch butterfly valves as replacement Complete spare assemblies were available for the SWS and SWBP c. Heat Exchanger Inspection and Cleaning The work requests and procedures that implemented the inspection and cleaning of safety related heat exchangers did not define the as found conditions in sufficient detail to enable the team to determine the condition of the HVHs. The instructions for the inspection of the heat exchangers and requirement for documentation of the results were inadequat The lack of acceptance criteria for the inspections had been identified by licensee self-assessments and corrective actions to revise the procedures were in progress. Failure to establish appropriate acceptance criteria is a violation of 10 CFR 50, Appendix B, Criterion XI, "Tests and Inspections."
However, based upon the corrective actions, the licensee's self-identification of the matter, no similar violations identified by the team in the last two years, the lack of willfulness, and the nonescalated enforcement nature, this is considered a non-cited violation as authorized under 10 CFR Part 2, Appendix C, Section VII. Additional SWS heat exchanger inspections were scheduled to be performed during the upcoming refueling outage using the revised procedures. This is identified as IFI 50-261/93-12-05, NRC review and follow-up of any SWS heat exchanger inspections and tests during the 1993 refueling outag Report Details
d. Corrective Maintenance (1) The corrective maintenance WRs reviewed commonly had poor documentation of the failure mechanisms. Several WRs were reviewed which performed overhaul of the SWBPs and investigated tripping of the breakers. There were no conclusions to indicate the cause of the problem had been identifie (2) The team's observation of the rebuilding of the spare SWBP in the shop was stopped when an "o" ring listed on the vendor drawing was not available. The failure to acquire and stage the "0" ring reflected a weakness in the licensee planning for this work activit (3) The licensee's procedure for laser alignment of pumps and the accuracy of this method was superior to the previous alignment techniques. This method was considered a strengt (4) Suboptimal performance was exhibited by the B SWBP. The pump had been rebuilt on three different occasions as identified in WRs 90-ABMM1, 91-ALBAl and 92-ANBU1. Always, the pump was disassembled, overhauled, and returned to service. Unexplained head losses occurred following the last two pump overhaul Throughout 1992 to the present, the run time for the A SWBP had been substantially greater than that of the B SWBP. The B SWBP had an oil seal leak which was identified by WR/JO 93-ABPN1 and the work was on hold waiting on part. Surveillance and Testing The team reviewed preoperational test procedures, surveillance procedures, and the licensee's IST program and implementing procedures to determine if sufficient testing had been conducted to confirm system design requirements and system operability. Also reviewed were the licensee's procedures, controls, and other activities associated with the calibration of instrumentation in the SWS. The results of those reviews were as follow a. Inservice Testing of Pumps and Valves (1) The licensee had been granted relief from the pump vibration testing requirements as specified by paragraph IWP-4500 of ASME Section X Instead of the Section XI requirements, the licensee had been approved to use the acceptance criteria of ASME/ANSI OMa 1988, Part 6 for pump vibration testing. No other relief requests were applicable to SWS pump testing. The acceptance criteria for vibration testing specified that for the alert range, the most limiting of 2.5 times the reference vibration velocity or 0.325 in/sec be used. The required action range had been specified to be the most limiting of 6 times the reference vibration velocity or 0.7 in/se Report Details
The most limiting values for each range had not always been use For example, the reference vibration velocity (in the vertical direction) for SWBP A was 0.075 in/sec. The licensee had used a value of 0.325 in/sec as the alert value when 2.5 times the reference value would be 0.1875 in/sec. Similarly, the licensee had used a value of 0.7 in/sec for the required action value when 6 times the reference value would yield a required action level of 0.48 in/sec which would also have been more limiting. Similar non-conservatisms were noted for the vertical vibration velocity measurements for SWS pump A and SWBP B. Additionally, a non conservatism was noted in the alert range for the horizontal measurements for SWBP A. The licensee had improperly interpreted the requirement to read that if the 0.325 in/sec alert value or the 0.7 in/sec required action value requirement were most limiting, then that range would be applied to all vibration velocity measurements for the entire pump. Otherwise, the times value for alert range and 6 times the reference value for required action would be applicabl The licensee interpreted that if the maximum vibration velocity in one direction was limited to a certain value for a particular bearing, then the maximum value should be the same for the orthogonal directio This is identified as violation 50-261/93-12-06, Failure to establish appropriate vibration acceptance criteria for pumps as required by the IST program. The licensee stated the acceptance criteria would be changed and no safety related pumps were found inoperable when using the more conservative acceptance criteri Based on these factors, no response to this violation is require (2) The SWS valves were being appropriately tested and on schedule with no repetitive problems note b. Equipment Performance Trending The licensee had implemented a computerized system for trending inservice testing results (use of the computerized system had allowed the licensee, including onshift operations personnel, to identify adverse trends and implement enhanced monitoring for the affected components). Overall, the system provided an effective method for maintaining a current perspective on the status of the affected equipmen However, several screen display limitations were noted. Specifically, check valve test results were originally recorded on a pass/fail basis and later with a quantitative value; therefore, comparison of the results were inappropriate. Also, the pump vibration units were in mils instead of in/sec and the scaling factors used were poor. The licensee stated that these problems had been previously identified and
Report Details
that the appropriate software corrections were in developmen c. Instrument Calibrations (1) Occasionally, instruments were found out-of-tolerance when performing instrument calibration checks. However, the required documentation associated with the evaluation and disposition of the unsatisfactory as-found conditions had not been complete Section 6.4.3 of MMM-006, Calibration Program, required such action Examples where the required evaluations had not been performed were:
FI-1698C, HVH-3 Return Water Flow on December 23, 1991,
DPS-1608B, North Service Water Strainer on January, 7 1993
DPS-1698A, HVH-1 Return Water Flow on December 8, 1991
DPS-1698D, HVH-4 Return Water Flow on October 8, 1991
DPS-1608A, South Service Water Strainer on January 7, 1993
FI-1698A, HVH-1 Return Water Flow on October 26, 1992
DPS-1698B, HVH-2 Return Water Flow on October 26, 1992 This is identified as violation 50-261/93-12-07, Failure to follow procedures associated with instrument calibration check (2) Several temperature indicating instruments associated with monitoring of the SWS had not been included in the calibration program. In particular, TI-163A and B, TI-1662A,B,C, and D, TI-1685, TI-6619A and B, TI-6620A and B, and TI-6617A and B had not been included. The licensee stated that the instrument manufacturer had recommended these particular instruments be replaced rather than undergo calibration. However, the licensee did not have a mechanism to detect when the instrumentation should be replaced. The licensee indicated that these instruments would be considered for incorporation into the calibration progra d. Technical Specification Surveillances TS surveillances were performed within the required periodicity. The implementing procedures accomplished the surveillance requirement with suitable acceptance criteri e. Other Testing (1) The licensee's SWS flow balancing test was adequate with no problems noted in the way the test was performed. However, the test was not regularly scheduled. The licensee indicated that an established frequency would be considere Report Details
(2) Section 7.3.2 and 7.3.10 of the last CCW heat exchanger SWS flow test, SP-895, provided results that the CCW system was cooling the SWS by 4 or 5 degrees. The licensee indicated awareness of the situation and stated it was due to the use of uncalibrated temperature instruments. Also, the temperature data was not essential since it was not used as part of the acceptance criteria. The team considered signing off the procedure without annotating and dispositioning the uncalibrated instruments as a weaknes (3) The technical content of the procedures used and results of the last calibration of the containment air cooler radiation monitor, R-16, were acceptabl.
Equipment Availability The team reviewed the availability records of the SWS for the past two years. This data was compared to that used as input to the IPE repor The data used in the availability calculations for the IPE report was derived from SWS availability from of January 1, 1985 through December 31, 1989. The team determined that the recent availability data compared favorably with the availability assumed in the IPE report. However, the team noted that SWS pumps were occasionally in alert. The SWS pump was stopped and redundant equipment was used. Corrective maintenance was then deferred until the next refueling outag.
Quality Verification and Corrective Actions The team reviewed the licensee's self-assessments and select ACRs associated with the SWS. The team attended on-site and offsite safety committee meetings conducted during the inspection period. Results of these reviews were as follows: The licensee's collective self-assessments, including a nuclear assessment department report and the licensee's SWS team, paralleled the team's findings. Weaknesses in communications between engineering and technical support, configuration control, and inspection acceptance criteria were examples of problems identified by both the team and the licensee. Due to the recent completion of some self-assessments, an evaluation of the licensee's corrective action implementation was not possibl The licensee had yet to ascertain the root cause of two ACRs of significance from 1992. The two ACRs dealt with a waterhammer in the SWS piping while in refueling outages and the tripping of SWBPs during refueling outage station blackout testing. In both instances, corrective actions only dealt with the symptoms of the problem rather than the cause. The licensee indicated a more aggressive investigation of the two events would occur during the upcoming refueling outag Report Details 16 On-site review committee membership and discussions were consistent with Technical Specification requirement The offsite review committee was a new initiative of the license The committee appeared to be making good progress in focusing on safety significant issues. The committee should enhance management's safety oversight of H. B. Robinson's performanc Due to recent inspection activities associated with the licensee's commitment tracking system (IR 50-261/93-11), the commitment tracking system was not evaluate.
Exit Interview The team conducted an exit meeting on July 2, 1993, at the H. B. Robinson Nuclear Power Station to discuss the major areas reviewed during the inspection, the strengths and weaknesses observed, and the inspection results. Licensee representatives and NRC personnel attending at this exit meeting are documented in paragraph 1 of this report. The team also discussed the likely informational content of the inspection report with regard to documents reviewed by the team during the inspection. The licensee did not identify any documents or processes as proprietar There were no dissenting comments at the exit meeting. However, before issuance of the report in telephone conversations, the licensee dissented violation 50-261/93-12-06 concerning IST vibration acceptance criteri Also, the licensee indicated additional information may be provided on violation 50-261/93-12-03 concerning use of Belzona. However, the information was not provided before report issuanc ITEM NUMBER STATUS PARAGRAPH DESCRIPTION 91-22-01 Closed
IFI - Review of completed EOP and AOP upgrade program 93-12-01 Open 6 & 7.c VIO - Failure to establish appropriate design control over SWS throttle valves 93-12-02 Open IFI - NRC review and follow-up of the licensee's corrective action implementation associated with configuration control 93-12-03 Open VIO - Failure to follow the design change process when using Belzona as a safety related pressure boundary
Report Details
93-12-04 Open VIO - Failure to establish adequate instructions in the SWS operating procedure and AOP-022 93-12-05 Open IFI - NRC review and follow-up of any SWS heat exchanger inspections and tests during the 1993 refueling outage 93-12-06 Open VIO - Failure to establish appropriate vibration acceptance criteria for pumps as required by the IST program 93-12-07 Open VIO - Failure to follow procedures associated with instrument calibration checks
APPENDIX A Acronyms and Abbreviations ACR Adverse Condition Report AFW Auxiliary Feedwater AO Auxiliary Operator AOP Abnormal Operating Procedure ARP Alarm Response Procedure ASME American Society of Mechanical Engineers CCW Component Cooling Water CFR Code of Federal Regulations Cl Chlorine DBE Design Basis Earthquake EDG Emergency Diesel Generator EOP Emergency Operating Procedure gpm Gallons per Minute HVAC Heating Ventilation and Air Conditioning HVH Containment Air Cooler IFI Inspector Follow-up Item IPE Individual Plant Examination IR
Inspection Report
Inservice Test
LER
Licensee Event Report
Loss of Coolant Accident
Motor Control Center
Motor Operated Valve
mpy
Mils per Year
NRC
Nuclear Regulatory Commission
Nuclear Reactor Regulation
On-the-Job Training
Part per Million
Probabilistic Risk Analysis
psig
Pounds per Square Inch Gauge
PWST
Primary Water Storage Tank
Reliability Centered Maintenance
Safety Injection
SWBP
Service Water Booster Pump
Service Water System
TI
Temperature Indicator
TS
Technical Specification
Work Request
Updated Safety Analysis Report
Verification and Validation
APPENDIX B
Generic Letter 89-12 Action Items
I. Biofouling Control and Surveillance Techniques
Action I of GL 89-12 requested licensees implement and maintain an ongoing
program of surveillance and control techniques to significantly reduce the
incidence of flow blockage problems as a result of biofouling. The actions
requested included intake structure inspections, periodic SWS flushing/flow
testing and chemical treatment of the SW Intake Structure Biofouling Inspections - Inspections of the SWS intake
structure were performed in 1989, 1991, and 1992. The initial inspection in
1989 was used as a baseline for subsequent inspections. These inspections
were recorded on videotape as documentation. The team reviewed the
videotapes of inspections conducted in 1989 and 1992. The 1989 inspection
covered the general SW intake area while the 1992 inspection consisted of
detailed inspection of pump bays, impellers, and the intake structure. No
evidence of biofouling or silting was present in either of these
inspections. Based on these inspection findings, the licensee did not plan
an inspection for the upcoming September refueling outage and will conduct
inspections every other refueling outag Containment Air and Motor Coolers Biofouling Monitoring - A monitor was
installed inline to the SWS to monitor the containment air and motor coolers
(HVHs 1-4) for biofouling. The monitor simulated conditions inside the HVHs
by measuring the heat transfer and pressure drop through a sample of the HVH
cooler tube materia Any decrease in heat transfer or increase in pressure
drop would indicate biofouling. However, the monitor had a high
unavailability and no historical database had been established between the
monitor and actual HVH conditions. Two coolers were inspected during the
previous refueling outage with no biofouling found. Therefore, the
inspection cycle was extended to 15 months. The other two HVHs have not
been inspected for 30 months and will be inspected during the upcoming
refueling outage. If no biofouling is found, then a 30-month inspection
interval will be used. Once a historical database has been established the
inline monitor's data will be used to determine the HVH inspection cycl Infrequently Used Heat Exchangers and Piping Dead Legs - The licensee
considered the AFW oil coolers, EDG heat exchangers, and EDG SW cross
connect piping as infrequently used. The EDG heat exchanger and the AFW oil
cooler were flushed once per week by failing the temperature control valves
open. Also, SWS chlorination was coordinated with biweekly EDG
surveillance. The EDG SW cross-connect was tested for biofouling by
supplying SW to the EDG from the opposite SW header. The lack of dual
isolation valves in the cross-connect did not allow for visual inspectio SWS Chemical Treatment - The licensee routinely shock treated the SWS with
sodium hypochlorite for about one hour to obtain 0.5 ppm free Cl during
treatment. The treatment schedule depends on water temperature as
determined by historical seasonal variations. Procedure CP-009 provided the
treatment frequencies. The maximum treatment frequency was once per day
during the summer month Appendix B
Initially, the free Cl requirement was established at 1.0 ppm and was
effective in preventing biofouling. However, this resulted in higher
corrosion rates for both the 90/10 and Admiralty copper coupons. The free
Cl requirement was decreased to 0.5 ppm to reduce the copper corrosion rat A biobox was installed inline to the SWS to provide visual indication of
biological growth. The copper corrosion rate and biobox will be used to
determine the free Cl and treatment frequency requirement Due to the low pH of Lake Robinson, a corrosion rate for copper coupons of
greater than 2 mpy had been found resulting in a serviceable life for copper
alloy heat exchanger tubes of about 10 years. Because the SWS was an open
system, a suitable corrosion inhibitor had not been found. The corrosion
affected the CCW heat exchangers, the EDG heat exchangers, AFW pump coolers
and numerous room coolers. The containment air and motor coolers were not
affected because the piping to the coolers and the cooler tubes were
stainless stee To date, no correlation between the coupons and actual
corrosion rates had been established. Inspections during the upcoming
refueling outage are expected to provide the correlatio II. Monitoring Safety Related Heat Exchanger Performance
Action II of GL 89-12 requested licensees implement a test program to
periodically verify the heat transfer capability of all safety related heat
exchangers cooled by the SWS. The test program was to consist of an initial
test program and a periodic retest progra In response to this action item the licensee attempted to measure the
performance of the CCW heat exchangers during the 1991 and 1992 refueling
outages. The results of the 1991 test were analyzed as inconclusive and the
test results from 1992 were not analyzed. Testing of the HVHs in 1991 had
also been inconclusive. In lieu of performance testing the licensee had
decided to inspect and clean the heat exchangers. The team considered the
licensee's decision to implement visual inspections of the heat exchangers,
rather than conduct performance testing, to be prematur The data obtained from the 1992 CCW heat exchanger performance test appeared
to be valid, however, the licensee had yet to conduct an analysis of the
test results. The conditions which had invalidated the 1991 test results
were not observed in the recent test. Additionally, enhanced performance
monitoring of the EDG heat exchangers could be achieved by trending the SW
delta T rather than the current practice of monitoring the SW outlet
temperature. Also, the licensee had yet to explore new industry initiatives
which appear to improve testing accuracy under conditions of low temperature
gradient The team considered the failure to analyze the 1992 CCW test as another
example of communication weaknesses between the site technical staff and
design engineering as discussed in section 5.a. The licensee agreed to
analyze the 1992 test for validit Appendix B
II Routine Inspection and Maintenance
Action III of GL 89-12 requested licensees implement a routine inspection
and maintenance program for open-cycle SWS piping and components. This
program was to ensure that corrosion, erosion, protective coating failure,
silting, and biofouling would not degrade the performance of the safety
related systems supplied by the SWS. In response to the action item the
license had performed extensive repair, replacement and inspection Piping - In response to microbiological attack a large part of the piping
within the containment had been replaced with more microbiologically
resistant piping (the last section of piping was scheduled for replacement
at the upcoming refueling outage).
Piping at the discharge of the SW pumps
had been replaced due to external erosion. Select sections of large bore
piping had been inspected in the last two refueling outage The piping inspection program to date was adequate with the piping
inspections conditional upon technical judgements from refueling outage to
refueling outage. Some fouling of the containment penetration coolers lines
had been experienced with chemical cleaning of penetration cooler lines
scheduled for the upcoming refueling outage. Based upon an existing
licensee engineering analysis the coolers were not needed to perform a
safety functio Pumps and Valves - Pumps were routinely refurbished or rebuilt ones
installed at refueling outages. Valves were routinely repacked during
refueling outages. The maintenance schedules for valves and pumps were
satisfactor Heat Exchangers - Due to corrosion, the EDG heat exchangers were retubed
with 90/10 copper in 1987 and the CCW heat exchangers were retubed with
90/10 copper in 1990. Eddy current testing of one CCW heat exchanger will
be performed during the upcoming refueling outage. However, the number of
tubes examined will be conditional on when the licensee feels consistent
results have been achieved. Follow-up of this inspection is considered part
of IFI 50-261/93-12-05. The containment air coolers were replaced with
stainless steel to minimize the effects of microbiological attac Hydrostatic testing of the EDG heat exchangers occurred every refueling
outage. Visual inspections of the heat exchangers were being performe However, as discussed in section 7, these inspections were inadequat IV. Design Function Verification and Single Failure Analysis
Action IV of GL 89-12 requested licensees confirm that the SWS would perform
its intended function in accordance with the licensing basis for the plan This confirmation was to include a review ensuring requisite safety
functions were accomplished even with the failure of a single active
componen Appendix B
In response to this action item, the licensee conducted a review of the SWS
for single failure, performed numerous walkdowns, and reviewed design
document. The licensee did not identify any single failure problem However, the team identified a single failure scenario as discussed in
section 5.c associated with inadequate CCW heat exchanger throttle valve
limitations as it relates to SW pump discharge header pressure contro The
licensee identified a number of minor configuration control discrepancies
during system walkdowns. The team confirmed that the configuration control
discrepancies were being dispositioned under the licensee's condition
adverse to quality syste V. Training
Action V of GL 89-12 requested licensees confirm that maintenance practices,
operating and emergency procedures, and training involving the SWS were
adequate to ensure safety related equipment cooled by the SWS would function
as intende In response to this action item the licensee conducted an in-depth review of
the SWS and documented their findings in report NED-G-6219 dated January 18,
199 Five specific recommendations were identified; (1)
develop five
additional procedures for valve maintenance, (2) add components to I&C
calibration program, (3) revise AOP-14 to identify the specific heat loads
to be isolated and valves to be closed, (4)
make maintenance procedures
consistent with the operations procedure, PLP-030, for independent
verification, and (5) include abnormal events in training module The team reviewed the licensees actions to these recommendations and found
that appropriate corrective actions had been or were being taken. An
independent review of selected training records of maintenance personnel
indicated adequate training was provided. Also, based upon the information
in section 6, operator training was adequate.