IR 05000361/2004009

IR 05000361-04-009; 05000362-04-009, on 09/13/2004 - 10/26/2004; San Onofre Nuclear Generating Station, Units 2 and 3; Biennial Baseline Inspection of the Identification and Resolution of Problems

ML043450229
Person / Time
Site:	San Onofre
Issue date:	12/10/2004
From:	Laura Smith Division of Reactor Safety IV
To:	Ray H Southern California Edison Co
References
IR-04-009
Download: ML043450229 (44)

Text

ber 10, 2004

SUBJECT:

SAN ONOFRE NUCLEAR GENERATING STATION - NRC PROBLEM IDENTIFICATION AND RESOLUTION INSPECTION REPORT INSPECTION REPORT 50-361/2004009; 50-362/2004009

Dear Mr. Ray:

On October 26, 2004, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at your San Onofre Nuclear Generating Station, Units 2 and 3. The enclosed report documents the inspection findings which were discussed on September 23, 2004, with Mr. D. Nunn and other members of your staff. Following additional in-office inspection, a discussion was held by telephone with Dr. R. Waldo and other members of your staff on October 26, 2004, to notify you that two issues that were unresolved when the onsite inspection was completed had been resolved, as discussed below.

This inspection was an examination of activities conducted under your license as they relate to the identification and resolution of problems, and the compliance with the Commissions rules and regulations and the conditions of your operating license. Within these areas, the inspection involved examination of selected procedures and representative records, observations of activities, and interviews with personnel. The team also reviewed your progress on improvement plans for human performance and equipment reliability which were corrective actions for a White performance indicator in Unplanned Scrams.

On the basis of the samples selected for review, the team concluded that in general, problems were properly identified, evaluated, and corrected. Your staff was proactive in performing self-assessments which were probing and aggressive in addressing negative behavior trends at a low level. However, the quality of root and apparent cause evaluations continued to be inconsistent, although appropriate actions were being taken to address this issue. The team identified that the operability assessment process was not being implemented consistent with procedures, in part because control room operator responsibilities were not proceduralized.

Also, your program was not always rigorous in evaluating and addressing industry operating experience.

There were three findings identified, one of which was determined to be a violation of NRC requirements. These findings were evaluated under the risk significance determination process as having very low safety significance (Green), and the violation is being treated as a noncited

Southern California Edison Company -2-violation, consistent with Section VI.A of the Enforcement Policy. The noncited is described in the subject inspection report. If you contest the violation or significance of this noncited, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 611 Ryan Plaza Drive, Suite 400, Arlington, Texas 76011; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at the San Onofre Nuclear Generating Station, Units 2 and 3 facility.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

//RA//

Linda Joy Smith, Chief Plant Engineering Branch Division of Reactor Safety Dockets: 50-361; 50-362 Licenses: NPF-10; NPF-15 Enclosure:

Inspection Report 05000361/2004009; 05000362/2004009 w/Attachments:

A --Supplemental Information B --Southern California Edison Position Paper: Electrolytic Capacitor Preventative Maintenance Regulatory Basis C --Southern California Edison Position Paper: SCE Assessment of Engine Systems, Inc.

Part 21 Notification D -- Southern California Edison Position Paper: Regulatory Assessment Licensee Event Report 3-2004-001, Unit 3 Trip Due to Sea-grass Intrusion cc w/enclosure:

Chairman, Board of Supervisors County of San Diego 1600 Pacific Highway, Room 335 San Diego, CA 92101

Southern California Edison Company -3-Gary L. Nolff Power Projects/Contracts Manager Riverside Public Utilities 2911 Adams Street Riverside, CA 92504 Eileen M. Teichert, Esq.

Supervising Deputy City Attorney City of Riverside 3900 Main Street Riverside, CA 92522 Joseph J. Wambold, Vice President Southern California Edison Company San Onofre Nuclear Generating Station P.O. Box 128 San Clemente, CA 92674-0128 David Spath, Chief Division of Drinking Water and Environmental Management California Department of Health Services P.O. Box 942732 Sacramento, CA 94234-7320 Michael R. Olson San Onofre Liaison San Diego Gas & Electric Company P.O. Box 1831 San Diego, CA 92112-4150 Ed Bailey, Chief Radiologic Health Branch State Department of Health Services P.O. Box 997414 (MS 7610)

Sacramento, CA 95899-7414 Mayor City of San Clemente 100 Avenida Presidio San Clemente, CA 92672 James D. Boyd, Commissioner California Energy Commission 1516 Ninth Street (MS 34)

Sacramento, CA 95814

Southern California Edison Company -4-Douglas K. Porter, Esq.

Southern California Edison Company 2244 Walnut Grove Avenue Rosemead, CA 91770 Dwight E. Nunn, Vice President Southern California Edison Company San Onofre Nuclear Generating Station P.O. Box 128 San Clemente, CA 92674-0128 D

SUMMARY OF FINDINGS

IR 05000361/2004-09; 05000-362/2004-09; September 13 to October 26, 2004; San Onofre

Nuclear Generating Station, Units 2 and 3: biennial baseline inspection of the identification and resolution of problems. A violation was identified in the area of control of non-conforming spare parts.

The inspection was conducted by two resident inspectors and two regional inspectors. Three Green findings of very low safety significance were identified during this inspection, one of which was classified as a noncited violation. The findings were evaluated using the significance determination process. The significance of most findings is indicated by their color (Green,

White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG1649, Reactor Oversight Process, Revision 3, dated July 2000.

Identification and Resolution of Problems The team concluded that the licensee was effective in identifying, evaluating, and correcting problems. The licensee was proactive in performing self-assessments which were probing and aggressive in addressing negative behavior trends at a low level. However, the quality of root and apparent cause evaluations continued to be inconsistent, although appropriate actions were being taken to address this issue. The team identified that the operability assessment process was not being implemented consistent with procedures, in part because the procedures did not cover control room operator responsibilities. The team also concluded that the licensee's program was not always rigorous in evaluating and addressing reports of industry operating experience.

NRC-Identified

and Self-revealing Findings

Cornerstone: Mitigating Systems

Green.

A noncited violation was identified for failure to establish measures to ensure non-conforming aluminum electrolytic capacitors that were part of an assembly and that were beyond their recommended shelf life would not be installed in the plant. The licensee's program controlled only electrolytic capacitors stocked individually. The licensee's apparent cause evaluation for a premature failure of an electrolytic capacitor in the Channel C power supply of the plant protection system noted that the failed component should not have been installed in the plant because it was beyond its recommended shelf life at the time of installation. This may have contributed to the power supply failure 18 months into a 10-year service life. The team noted that the licensee failed to subsequently implement measures to ensure that replacement parts with electrolytic capacitors were within their shelf-life prior to installation in safety-related systems, which was a violation of 10 CFR Part 50, Appendix B, Criterion XV.

This finding affected the mitigating systems cornerstone, and was more than minor because, if left uncorrected, it could become a more significant safety problem, since it could increase the failure rate of the plant protection system. This issue screened as Green in a Phase 1 assessment, since the finding did not result in a loss of function, since only one train was affected by the only known failure. (Section 4OA2.e(1))

Cornerstone: Initiating Events

Green.

A finding was identified concerning non-rigorous troubleshooting and post-maintenance testing of an intermittent failure, which led to declaring equipment operable without reasonable assurance that the system would reliably perform as intended. Between September 10 and 16, 2004, an intermittent failure spuriously tripped in the Steam Generator 2 low flow or pre-trip bistables in Channel A of the plant protection system in Unit 3 five times. On separate occasions, maintenance technicians replaced different circuit cards that were thought to possibly cause the failures observed. Each time, operators declared the system operable based on passing its surveillance test without any additional monitoring for further intermittent failures and without confirming that any of the removed parts were bad. The team determined that no formal troubleshooting plan was used until the team asked to review the plan. Upon creating a formal plan, the licensee promptly identified debris in a circuit card connector thought to have been introduced during a recent calibration activity.

The performance deficiency associated with this event was a failure to follow the guidance of Generic Letter 91-18 to ensure there was a reasonable assurance that the system would perform reliably before declaring it operable. This resulted in receiving a repeat spurious bistable trip signal during unrelated testing on a redundant channel, necessitating unplanned limiting condition for operation actions, which placed the plant closer to a trip condition. This finding was greater than minor because it affected the equipment performance attribute of the Initiating Events cornerstone and directly affected the cornerstone objective of limiting events that challenge plant stability. The finding was of very low safety significance because it did not contribute to the likelihood of a loss-of-coolant accident initiator, did not increase the likelihood of both a reactor trip and simultaneously affect the unavailability of mitigation equipment, and did not increase the likelihood of a fire or flooding event as described in the Significance Determination Process Phase 1 screening worksheet. (Section 4OA2.e(2))

Green.

A self-revealing finding was identified because plant procedures for responding to debris at the plant intake did not address responding to a large influx of sea grass.

The licensee identified this issue during their root cause assessment of the July 24, 2004, manual trip of Unit 3. Sea grass blocked the sea water intake for the circulating water system, and operators unsuccessfully attempted to keep the traveling screens clear until they eventually failed and stopped rotating. Neither Procedure SO23-2-5,

"Operating Instruction for Circulating Water System," Revision 21, nor SO23-2-7,

Operation of Traveling Rakes and Screens," Revision 10, nor Alarm Response Instruction SO23-15-99.C, "Annunciator Panel 99.C," Revision 6, provided adequate guidance. Specifically, these procedures contained no guidance to take plant-level actions to reduce circulating water flow if actions to clear the screens were inadequate to restore normal flow. The licensee implemented appropriate corrective actions to improve the applicable response procedures, and improved policies for proactive monitoring of tide and weather conditions which could produce excessive sea grass intrusion. Reference Licensee Event Report 05000362/2004001-00 and Action Request 040600324. The team determined that the finding did not represent a noncompliance because it related to nonsafety-related secondary plant equipment.

The finding is a performance deficiency because the alarm response and system operating procedures did not provide adequate instructions to operators to allow them to appropriately respond to more significant episodes of sea grass intrusion at the plant intake. The licensee's root cause assessment stated that more debris was expected during periods of strong surf and large tidal scope, such as was experienced in this event. This finding was more than minor because it was similar to Example 4.b in Manual Chapter 0612, Appendix E. This finding affected the Initiating Events cornerstone objective that procedures have sufficient quality to limit the likelihood of those events that upset plant stability during power operations. This issue screened as having very low safety significance in a Phase 1 assessment, since the finding affected only the initiating events cornerstone, and it affected a transient initiator without simultaneously affecting transient mitigation equipment. (Section 4OA3.1)

REPORT DETAILS

4OA2 Identification and Resolution of Problems

a.

Effectiveness of Problem Identification

(1) Inspection Scope The team reviewed approximately 128 Action Requests (ARs), along with supporting documentation, including root-cause analyses to determine whether problems were properly identified, characterized, and entered into the corrective action program.

Specifically, the team's review included a selection of action requests that had been opened or closed or that related to issues of regulatory non-compliance since December 1, 2002. These issues were selected across the seven cornerstones of safety. Additionally, the team reviewed a number of other databases, which tracked problems separately from the corrective action database.

The team evaluated the action requests to determine the licensee's threshold for identifying problems and entering them into the corrective action program. Also, the team evaluated the licensee's efforts in establishing the scope of problems by reviewing pertinent work orders, engineering modification packages, self-assessment results, and action plans.

Team members attended meetings intended to screen, assign significance and cause assessments, as well as review cause assessment reports and effectiveness of corrective actions. These meetings included several Action Request Committee meetings (September 14-17), a Corrective Action Review Board meeting (September 23), a Human Performance Improvement Team meeting (September 14),and an Equipment Reliability Improvement Team meeting (September 15). The team also reviewed documentation from numerous licensee audits, self-assessments, and directed assessments related to the problem identification and resolution. The effectiveness of the audits and assessments was evaluated by comparing the audit and assessment results against self-revealing and NRC-identified findings.

Inspection Procedure 71152 requires performing a detailed review of equipment performance in one system for a 5-year period. The team performed a review of problem identification associated with the emergency chilled water system for a 5-year period.

Specific documents reviewed by the team are listed in Attachment A.

(2) Assessment

Introduction.

The team determined that the licensee was effective at identifying problems and entering them into the corrective action program. Licensee audits and self assessments were of sufficient breadth and depth and identified issues similar to those that were raised during NRC inspections.

Quality of Self-Assessments The team concluded that the licensee was proactive in performing self-assessments.

The team reviewed numerous self-assessments. San Onofre Nuclear Generating Station performed a large number of self-assessments. These included quarterly departmental self-assessments, station-level assessments, quality assurance audits, and focused assessments. The self-assessment reports documented thorough reviews, and the reports identified issues that were consistent with those identified in NRC inspection reports. Directed self-assessments were performed in response to performance trends identified at a low level during such activities as root cause evaluations and departmental self-assessments. Many of the more significant self-assessments were conducted with the assistance of industry peers or outside experts to ensure a balanced perspective.

As part of the departmental self-assessments, the licensee performed numerous (sometimes hundreds in a single department in a calendar quarter) observations of plant activities under the Leadership Observation Program. This process was used effectively to identify behaviors and low-level human performance trends and correct them on the spot. This problem identification tool was documenting positive as well as negative observations.

Quarterly departmental self-assessments provide insightful analysis of useful data, and identify areas to focus Leadership Observation Program observations and other actions.

However, the team noted that in some cases, particularly areas of human performance, low-level behavior trends were repeatedly identified during the 2-year assessment period covered by this inspection and not corrected. Some examples are mispositioning events, equipment control issues, rework, and procedure adherence. While the practice of handling low-level issues in a low-level manner, such as directed Leadership Observation Program observations and coaching was acceptable; however, there was no clear guidance on when to place such trends firmly in the corrective action program and address them at a more formal level.

b.

Prioritization and Evaluation of Issues

(1) Inspection Scope The team reviewed approximately 128 corrective action documents and supporting documentation, including root cause analyses. The intent was to ascertain whether the licensee staff properly evaluated the problems identified and considered the full extent of conditions, generic implications, common causes, and previous occurrences. In addition, the team reviewed Action Requests to ascertain if the provisions of NRC Generic Letter 91-18, Resolution of Degraded and Non-Conforming Conditions, and 10 CFR Part 50, Appendix B, were satisfied regarding timeliness of corrective action.

The team also performed a review of problem prioritization and evaluation for the emergency chilled water system for a 5-year period. Specific items reviewed are listed in Attachment A.

(2) Assessment

Introduction.

The team found that, in general, the licensee was effective in prioritizing and evaluating issues within the corrective action program. Corrective actions were generally implemented in a timely manner. However, the team noted that documentation was often unclear, and some actions did not get properly completed or were cancelled without clearly stating why. For Apparent Cause Evaluations and Root Cause Evaluations, the team noted that there was clear alignment of corrective actions to the cause(s) identified. However, more routine issues did not always have corrective actions which directly addressed the associated problem statement or cause. The team identified that portions of the operability assessment process were not proceduralized, or that actual practices did not match existing procedures and responsibility assignments; as a result, the quality of assessment and documentation for many assessments of operability were inconsistent and often did not provide an adequate basis to show that the equipment could perform all its intended functions. Licensed control room operators were not assigned responsibility under the licensee's operability assessment processes.

Quality of Root Cause Evaluations and Apparent Cause Evaluations The team reviewed 6 completed apparent cause evaluations (ACE) and 7 completed root cause evaluations (RCE) to assess the thoroughness of the effort. Most of the RCEs reviewed by the team clearly documented a root cause that was fully supported by the facts presented in the report. Root cause evaluations were performed by root cause assessment teams, each of which was assigned a qualified root cause evaluator.

However, some of the evidence gathered in assessing the root cause was not included in the root cause reports, which made it difficult to determine the accuracy of some cause determinations. The team noted that because of software limitations, the licensee's Action Request system did not have the capability to attach data, such as logs, plant printouts, written personnel statements, or similar documents, which are commonly included in root cause assessments. Also, the station's documentation policy limited what was included in root cause reports. Procedure SO123-XV-50.39, "Cause Evaluation Standards, Methods, and Instructions," Revision 4, required the writer to remove discussion that did not support the conclusion. While this was intended for brevity, this could potentially cause information to be removed which conflicted with the stated root cause and should be included to allow readers to assess the relative strength or weakness of the conclusion reached. The team interviewed root cause team members from two root cause assessment teams, and determined that some useful information had been considered and not included in reports, but none that would have significantly weakened the conclusion stated in the reports.

The station's favored cause evaluation method was the Events and Causal Factors Charting method, which developed a flow chart which could not be attached to the RCE.

By not being able to attach the final chart, much of the presentation value of this methodology is lost. In some cases, the remaining report did not adequately describe the information available on the chart.

The team noted that in many cases, the cause assessment method used a supplemental method, without using any of the primary methods as intended by the station procedure. This condition was more prevalent in apparent cause assessments than in the more formal root cause assessments. The team also noted that the cause evaluation method used did not always include the method listed in the procedure as the most appropriate for the circumstances. For example:

• RCE 040600324 was performed to assess the cause of a Unit 3 trip associated with sea grass clogging the circulating water intake screens. The RCE stated that operator action had always been effective in avoiding a plant trip in the past, but did not perform a change analysis to identify what changes or differences from past events contributed to this trip. Further, the RCE identified that the local operator was initially effective in addressing the sea grass buildup, but after a time returned the screens to automatic and left the area; ten minutes later the screens became so clogged that their shear pins began to break. This apparent human performance issue was not reviewed using the human performance assessment tool, as intended in the procedure. This lapse potentially affected the completeness of the corrective actions from this event. This issue is also discussed in Section 4OA3.1.

• RCE 030200027 assessed a Unit 2 trip on loss of generator excitation. This was assessed using events and causal factors method alone, when barrier analysis and change analysis were appropriate since the task involved was a regularly scheduled test and numerous barriers were not adequate to have prevented the event.

• RCE 040400600 was performed to assess the cause of a Unit 2 trip on loss of feedwater flow. This root cause evaluation documented the results of troubleshooting performed over the weekend of the trip. The root cause assessment team was not assigned until the following Monday, potentially limiting the effectiveness of their review. The RCE used only a supplemental analysis method, a technical evaluation (which is permitted but not actually defined as a methodology in the program). No primary assessment method was used. The RCE concluded that two separate grounds in the feedwater control system caused a close signal to the main and bypass feedwater regulating valves. However, the troubleshooting team located only one ground. The cause was attributed to a manufacturing defect that pinched the one damaged wire found, even though the equipment was over 20 years old. No evidence was included to support manufacturing as the cause or rule out other causes, even though evidence was used to rule out maintenance work.

The team reviewed a sample of apparent cause evaluations performed during the review period. Many of the samples reviewed did not use a primary cause assessment method, and the majority listed Technical Evaluation as the only method. The level of documentation in most ACEs was significantly less than in root cause evaluations. The team reviewed training that was provided to approximately 120 apparent cause evaluators during early 2004 in order to address the perceived inconsistent quality of ACEs. While it was still too soon to determine whether the training was effective, the training was appropriate corrective action.

Quality of Operability Assessments The team reviewed a selection of Action Requests with operability aspects (see list in Documents Reviewed) for completeness and quality in assessment of the operability issue, as well as the appropriateness and formality of any compensatory actions specified to ensure operability. For issues with less-formal documentation of the basis for operabilty, the team reviewed the adequacy of the basis for operability provided.

This included an assessment of the licensee's threshold in assessing issues for operability. The team reviewed the procedures, which govern problem identification and operability assessment, then compared this to interview results intended to determine how the process was actually implemented.

The team concluded that control room operators were assigned very little responsibility or specific actions for assessing operability issues in procedures. However, operators did participate in the assessment of operability issues, because operations and station management expected that these licensed operators should make the key decisions.

Therefore, the actual operability assessment process was not reflected in the governing procedures. In particular, these procedures did not include actions or minimum documentation requirements for routine operability determinations made by control room personnel without significant assistance. As a result, the majority of Action Requests reviewed by the team did not contain sufficient documentation to be able to determine the quality of the determination. For example, formal operability determinations were generally performed by engineering personnel, and were governed by Procedure SO123-XV-52, "Operability Assessments and Reportability Evaluations," Revision 5.

This procedure appropriately provided instructions to identify the affected safety functions and document the ability to perform these functions. The team concluded that in formal operability assessments, the scope of the review was generally thorough and the basis for the conclusion was generally clearly documented. These formal assessments were typically performed and approved by engineering personnel. In contrast, the quality of the assessment of more routine operability issues, performed by control room operators, generally could not be determined from the available documentation. The potentially degraded or non-conforming condition was not clearly stated in a majority of the routine operability assessments reviewed, the required functions that were potentially impacted were not identified, and the basis for the conclusion was frequently not clearly stated. Typically, a simple conclusion was all that was stated. The team also noted examples where the stated basis for operability was that the equipment was passing its surveillance tests; the team pointed out that Generic Letter 91-18 provided specific guidance that passing surveillance tests may not be a sufficient basis for operability when the tests do not address the specific issue. The team noted that training on operability assessment did not include Generic Letter 91-18 in sufficient detail to include this point.

In general, the licensee had a conservative attitude toward operability determinations. In particular, the Action Request Committee was observed to treat unclear operability issues as inoperable. However, the team noted that there was no standardized way for the AR Committee to ensure that when they concluded equipment was inoperable that control room operators entered any applicable Technical Specification actions for operability. This was at least in part due to the lack of procedural guidance. However, the team identified no specific examples where the licensee failed to enter an action statement due to inadequate communications.

The station did not have a procedure to cover the entire operability determination process. The process starts in SO123-XX-1 ISS2, "Action Request/Maintenance Order Initiation and Processing," Revision 15-2 as part of the problem reporting process. A complex issue that requires research would receives a formal operability assessment, covered in Procedure SO123-XV-52, "Operability Assessments and Reportability Evaluations," Revision 5. However, the bulk of operability issues would typically fall somewhere in between, and were not covered by a procedure or policy.

Procedurally, Operations did not own the operability determination/assessment process.

This is inconsistent with their license responsibility in this area. Operability assessment responsibility was procedurally assigned to the AR Committee (which has a non-shift senior licensed operator as a permanent member) and Engineering. The procedures allowed either the AR Committee or two qualified engineers to determine whether equipment is operable.

c.

Effectiveness of Corrective Actions

(1) Inspection Scope The team reviewed problem evaluation requests, trending reports, effectiveness reviews, and self assessments to verify that corrective actions were identified and implemented in a timely manner commensurate with safety. Corrective actions were evaluated for effectiveness in correcting the problem, including corrective actions to address common cause or generic concerns. The team reviewed corrective actions for selected violations to verify compliance was restored. The team also performed a review of corrective actions for emergency chilled water system for a 5-year period. A listing of specific documents reviewed during the inspection is included in Attachment A.

(2) Assessment

Introduction.

The team concluded that, in general, the licensee implemented effective corrective actions to address problems in a timely manner.

The licensee's corrective action program had a low threshold for treating problems with greater than routine significance. The team observed that the program effectively screened newly reported problems in the Action Request Committee, and issues with greater than routine significance were assigned apparent cause evaluations or root cause evaluations. The actions and level of review assigned to these categories were appropriate for conditions adverse to quality and significant conditions adverse to quality, respectively.

Issues addressed under root cause evaluations received a formal root cause evaluation by a multi-disciplined team. This team was generally led by a subject matter expert, and a qualified root cause evaluator was assigned to the team. The corrective actions appropriately addressed the actual problem and potential generic implications, with the goal of preventing recurrence. During the review period, the licensee performed 8-12 root cause evaluations per year. Each of the root cause evaluations reviewed by the team clearly developed corrective actions that were directly tied to the cause(s)identified.

Issues addressed under apparent cause evaluations received a less formal cause evaluation. During 2004, 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> of cause evaluation training was provided to individuals who perform apparent cause evaluations. This action was undertaken to address concerns with the inconsistent quality of apparent cause evaluations. Also, the newly-formed Corrective Action Review Board was formed to review the quality of selected apparent cause evaluations and the associated corrective actions. This group was composed of senior plant managers and was intended to raise the standards for apparent cause evaluations. The team observed a Corrective Action Review Board meeting on September 23, 2004, and concluded that the members were vigorous in challenging the stated conclusions. Some apparent cause evaluations were rejected for additional work, demonstrating a high current standard for quality. However, the team also noted that the additional information discussed at the review was not always added into the documented apparent cause evaluation, so the completed product still did not necessarily reflect the total body of information on the issue.

Effectiveness of the Human Performance Improvement Program Partially in response to a White NRC performance indicator for unplanned scrams, the licensee implemented a Human Performance Improvement Program. The team reviewed the progress on this plan and observed the September 14, 2004, meeting of the Human Performance Improvement Team, as well as other routine meetings. Human performance was observed to receive a significant level of management attention. The scope of the actions was appropriate to address the site-wide problems observed, though some specific initiatives were still being developed. The team concluded that progress was being made in this area.

Effectiveness of the Equipment Reliability Improvement Program Also partially in response to the White performance indicator in unplanned scrams, the licensee implemented an Equipment Reliability Improvement Program. The team reviewed the progress on this plan and observed the September 15, 2004, meeting of the Equipment Reliability Improvement Team. The program was appropriately addressing the scope of the problems identified with equipment reliability, though this effort was still too new to assess the effectiveness of the actions.

Review of Troubleshooting Effectiveness During the course of the inspection, the team followed the progress of an intermittent failure in the Unit 3 plant protection system. Control room operators identified failures in the Steam Generator 3B loop flow instrument on September 10, 14 (three times), 15, and 16. After each failure, maintenance personnel decided what the most likely problem was and replaced a different component. The instrument was returned to service and declared operable after each repair. Each incidence was treated as a separate issue, rather than a continued pattern. The team concluded that no formal troubleshooting was performed until after the failure on September 16, when the team asked to see the troubleshooting plan. The team identified that maintenance personnel concluded that the repair was effective in each case without checking that the removed part was actually bad, accepting that the instrument was performing acceptably even though it was known to be an problem that appeared only intermittently. When a formal troubleshooting plan was developed, the problem was promptly identified as being caused by some foreign material partially blocking a circuit board electrical connection, and was readily identified during a visual inspection. Through discussions, the team identified that the licensee had a proper procedure for performing formal troubleshooting, but did not always implement it until the more likely potential causes were already exhausted. In this case, this resulted in repeatedly restoring equipment required to be operable by Technical Specification to service without locating or correcting the problem. This issue is discussed further in Section 4OA2(e)(2).

Review of Response to Industry Operating Experience The team attempted to assess the licensee's response to industry operating experience related to electrolytic capacitors. The nuclear power industry has reported premature failures of aluminum electrolytic capacitors in safety-related applications due to the electrolyte leaking or drying out over time (collectively known as "dryout"). This failure mechanism was separate from failures due to normal aging. Dryout failures could result in the useful life being much shorter than those of other capacitors. Electrolytic capacitors were commonly used in power supplies of electronic components, as well as other applications. These issues have been discussed in NRC Information Notices, reports pursuant to 10 CFR Part 21, and other sources of industry operating experience.

Between 1995 and 1997, the licensee formed the Electrolytic Capacitor Task Force to address the issue of age-related failure of electrolytic capacitors and develop a program of replacement for the capacitors to ensure equipment reliability. In October 1997, the licensee initiated Action Request 971001889 to track the actions of the Task Force. The Task Force was to address a number of industry operating experience reports, including Information Notice (IN) 94-24, Inadequate Maintenance of Uninterruptible Power Supplies and Inverters, IN 94-33, Capacitor Failures in Westinghouse Eagle 21 Plant Protection Systems, and IN 95-10, Potential for Loss of Automatic Engineered Safety Feature Actuation, Supplements 1 and 2. A Unit 2 trip at San Onofre in August 1986 was attributed to age-related failure of an electrolytic capacitor, and resulted in the issuance of an Engineering Memorandum dated May 26, 1987, which recommended a complete review of all critical systems to ensure that maintenance orders existed for the periodic replacement of electrolytic capacitors in those systems. Action Request 971001889 noted that this recommendation had not been implemented, and initiated corrective action to develop replacement maintenance orders for all safety critical or trip initiator components on a generic ten-year frequency based on vendor general service life estimates. The team noted that this service life did not account for applications in a high temperature environment, which could shorten the actual service life.

The team discussed the licensee's actions with members of the Electrolytic Capacitor Task Force. The team identified that the Task Force had not defined the categories of safety critical and trip initiator components, and had based decisions on engineering judgement that was not documented. The licensee was not able to provide documentation of what systems or components had been considered or excluded from the program, or the reasons why. Based on the statements of the Task Force members interviewed, the team was concerned that the equipment in these categories did not include all safety-related components in the plant subject to 10 CFR Part 50, Appendix B quality requirements.

The team attempted to review the history of specific failures of electrolytic capacitors in safety-related applications at San Onofre. However, no additional examples beyond those mentioned above could be located in the licensee's records. The team concluded that this was more indicative of not having a process to identify such failures, rather than a lack of failures for the following reasons:

• The licensee did not have a component engineer assigned to track electrolytic capacitor issues across system boundaries;

• The Maintenance Rule Program did not contain reliability or performance monitoring criteria for electrolytic capacitors across system boundaries; and

• The licensee's corrective maintenance practices did not seek to identify the cause of failures down to the level of a capacitor.

The team concluded through discussions that the performance goals of the licensee's operating experience program did not address all of the insights for this program that were described in Information Notice 2003-15, "Importance of Followup Activities in Resolving Maintenance issues." The following excerpts from Information Notice 2003-15 relate to this issue:

Nuclear power plant licensees establish and implement programs to maintain the capability of a wide range of components to perform their safety functions and provide assurance of safe plant operation. In addition to prescribing planned work activities based on industry standards or vendor recommendations, these maintenance programs address component performance issues that are identified through plant-specific and industry-wide operating experience, and through NRC and industry research activities.

Appendix B, Quality Assurance Criteria for Nuclear Power Plants and Fuel Processing Plants, to Part 50 in Title 10 of the Code of Federal Regulations specifies requirements applicable to the assessment of operating experience that can reflect the inability of safety-related [equipment] to perform their design-basis functions. For example, Criterion V, Instructions, Procedures, and Drawings, in 10 CFR 50, Appendix B, states, in part, that activities affecting quality shall be prescribed by documented procedures of a type appropriate to the circumstances and shall be accomplished with these procedures. . . . During Inspections at nuclear power plants, the NRC identified instances where licensees failed to satisfy the criteria in Appendix B to 10 CFR 50 with respect to their responses to . . . operating experience and other performance information.

When responding to operating experience and other component performance information, it is important for licensees to have a clear plan of action to identify specific potentially affected components and to address and track to completion those components in a reasonable amount of time based on their safety significance.

Addressees are reminded of the quality assurance requirements in 10 CFR Part 50, Appendix B, to address operating experience issues and component performance information and the maintenance requirements in 10 CFR 50.65 to consider industry-wide operating experience and component performance information (including issues described in NRC information notices), where practical, when setting goals and performing periodic evaluations.

The team reviewed the failure of the safety-related power supply for Channel C of the plant protection system in November 2001 in which an electrolytic capacitor had failed prematurely due to age-related loss of electrolyte (dryout). This issue is discussed in detail in Section 4OA2.e(1). The licensees evaluation correctly noted that electrolyte loss in storage was not reversible by reforming the capacitor (a process of energizing under controlled conditions to electrically restore the capacitor). However, the team noted that the licensee's program reformed safety-related capacitors stored individually as safety-related spare parts every 2 years with no limit on shelf life, despite this evaluation. Further, the licensee was testing electrolytic capacitors with test equipment, which had been reported to provide falsely acceptable results for electrolytic capacitors which were known to have experienced dryout.

The team concluded that, in the case of electrolytic capacitors, the licensee's operating experience program was not rigorously assessing lessons learned in industry and applying them at San Onfre. The licensee was assessing the specific points documented above under AR 041101130. A noncited violation was identified related to this issue, which is discussed in Section 4OA2.e(1).

d. Assessment of Safety-Conscious Work Environment

(1) Inspection Scope Each team member interviewed several members of the licensee's staff, which represented a cross-section of functional organizations and supervisory and non-supervisory personnel, regarding their willingness to identify safety issues. These interviews assessed whether conditions existed that would challenge the establishment of a safety-conscious work environment.

(2) Assessment During interviews, station personnel communicated a willingness to enter issues into the corrective action program. The team evaluated whether the Event Trend Record process or other department-specific tracking processes were being used to address problems outside the corrective action program. No significant findings were identified.

e.

Specific Issues Identified During This Inspection

(1) Inadequate Measures to Control Spare Assemblies (a)

Introduction.

The inspectors identified a Green noncited violation for failure to establish measures to ensure non-conforming aluminum electrolytic capacitors that were part of an assembly and that were beyond their recommended shelf life would not be installed in safety-related equipment in the plant.

(b)

Description.

The nuclear power industry has reported failures of aluminum electrolytic capacitors due to the electrolyte leaking or drying out over time. These failures could cause the useful life to be much shorter than those of other capacitors. Electrolytic capacitors were commonly used in power supplies of electronic components, as well as other applications. These issues have been discussed in NRC Information Notices, reports pursuant to 10 CFR Part 21, and other sources of industry operating experience.

The team reviewed a failure of the safety-related power supply for Channel C of the plant protection system in November 2001. The licensee identified that an electrolytic capacitor had failed in the power supply. Since this same capacitor had been replaced following a failure less than 18 months earlier, the licensee initiated an apparent cause evaluation as part of AR 011100769. The cause of the failure was identified as age-related loss of electrolyte (dryout). This AR stated that the failed capacitor had been installed when it was twelve years old, significantly beyond the manufacturers recommended shelf life. The licensees evaluation specifically noted that electrolyte loss in storage was not reversible by reforming the capacitor at regular intervals.

The team attempted to assess the rigor of the licensee's response to the insights gained from this failure. The team was specifically concerned about how the licensee was addressing the dryout failure mechanism to ensure that safety-related systems would maintain the high degree of quality and reliability collectively required by 10 CFR Part 50, Appendix B. The team reviewed the licensee's programs along a number of potential success paths. Each is discussed below.

As corrective action, the licensee replaced similar power supplies throughout the plant protection system in order to remove older parts. Repetitive maintenance orders were created to replace these power supplies every 10 years to limit the service life. The licensee also implemented actions to control the shelf life of spare electrolytic capacitors stored individually. However, the team noted that this corrective action did not completely address the stated cause, since it did not prevent installing old electrolytic capacitors stocked as part of spare assemblies (such as part of a circuit card) into plant equipment. The apparent cause evaluation included a recommended corrective action to conduct training for appropriate maintenance personnel to inform them not to install capacitors greater than two years old. This corrective action might have been effective as a barrier to installing non-conforming electrolytic capacitors independent of shelf-life tracking, but this recommendation was never implemented.

As discussed in Section 4OA2.c, the corrective maintenance and Maintenance Rule programs were not tuned to identify an increased failure rate of electrolytic capacitors, and the pre-installation testing was not implementing operating experience recommendations that improved detection of electrolytic capacitors degraded due to dryout.

The team considered that failure to adequately address industry operating experience for electrolytic capacitors and control the shelf-life of spare parts with electrolytic capacitors installed in plant protection system power supplies was a performance deficiency, since it had the potential to reduce the reliability of this safety-related system.

In the specific case in AR 011100769, it led to an actual failure. Further, absent a specific evaluation, a safety-related spare part beyond its shelf-life represents a non-conforming condition.

Further, the team was concerned because other industry operating experience indicated that this issue was generically applicable to aluminum electrolytic capacitors in other applications. When SCE assessed this issue in 1995 - 1997, the Electrolytic Capacitor Task Force did not include all safety-related equipment in the periodic replacement program and did not fully address the shelf life issues from their own and industry operating experience.

The licensee provided two position papers related to this inspection topic. These are attached to this report as Attachments B and C.

(c)

Analysis.

This finding was more than minor since the failure to have a program to ensure the quality of spare parts, if left uncorrected, could lead to a more significant safety concern. Specifically, the failure to control the shelf life for spare parts which contained aluminum electrolytic capacitors or otherwise ensure that the replacement parts were not non-conforming reduced the reliability of safety-related systems due to an increased probability of failure from those electrolytic capacitors due to dryout prior to scheduled replacement. This finding affected the mitigating systems cornerstone, and was more than minor because, if left uncorrected, it could become a more significant safety problem, since it could increase the failure rate of the plant protection system.

This issue screened as Green in a Phase 1 assessment, since the finding did not result in a loss of a safety function.

(d)

Enforcement.

Appendix B to 10 CFR Part 50, Criterion XV, Nonconforming Materials, Parts, or Components, requires that measures shall be established to control materials, parts, or components which do not conform to requirements in order to prevent their inadvertent use or installation. These measures shall include, as appropriate, procedures for identification, documentation, segregation, disposition, and notification to affected organizations.

Contrary to the above, the licensee had a program to control the shelf-life of individual spare capacitors, but had no effective measures to prevent installation of a non-conforming electrolytic capacitors that were part of an assembly and that were beyond their designated shelf-life. Specifically, on November 16, 2001, an electrolytic capacitor in the power supply for Channel C of the plant protection system failed. The failed capacitor was non-conforming to requirements because it was beyond its recommended shelf-life when it had been installed 18 months earlier. The licensee's failure analysis concluded that the cause was age-related dryout of the electrolyte. This is being treated as a noncited violation consistent with Section VI.A of the NRC Enforcement Policy; NCV 05000361/2003003-01, "Failure to Control Shelf-life of Spare Assemblies with Electrolytic Capacitors." The licensee staff entered this issue into their corrective action program as Action Request 041101130.

(2) Inadequate Assessment of Operability (a)

Introduction.

The inspectors identified a Green finding where inadequate troubleshooting led to the declaration of equipment operable without reasonable assurance that the system would reliably perform as intended.

(b)

Description.

On September 10, 2004, the Steam Generator 2 low flow bistable spuriously tripped in Channel 'A' of the Plant Protection System (PPS) in Unit 3. Since the alarm had cleared shortly after being received, maintenance technicians reviewed the circuit drawings and component history for the affected PPS cabinet to determine potential causes for the spurious alarm. The technicians identified three circuit cards that could possibly cause the alarm, one of which - the rate variable setpoint card - had caused this sort of problem in the past. The maintenance technicians replaced the rate variable setpoint card and performed a surveillance test on the cabinet to verify operability. Operators declared the system operable since it had passed its surveillance test and the alarm had cleared.

On September 13 and again on September 15, the spurious low flow alarm and a spurious low flow pre-trip indication were received, respectively, and the operators again declared the instrument inoperable. In each case, maintenance technicians replaced a different circuit card, performed the surveillance test, and restored the PPS to operable status.

The team noted that although the replacement circuit cards were thoroughly calibrated and tested prior to installation in the PPS cabinet, no such calibration test was performed for the cards that were removed. Such testing should have indicated whether the cards that were removed were malfunctioning as thought. The team also noted that the system would have passed the surveillance test had the test been performed immediately prior to each of the maintenance activities, since the spurious alarms had already cleared prior to replacing the circuit cards. This demonstrates that the surveillance test by itself should not have been sufficient to establish whether the problem that had caused the spurious alarm had been identified and resolved. The inspectors determined that the guidance of Generic Letter 91-18, which states that performance of a surveillance requirement alone may not be sufficient to prove operability, was applicable in this case, and that further investigation and/or monitoring of the system was required in order to provide the necessary reasonable assurance that the system would perform as intended.

On September 16, operators again declared the low flow bistable inoperable following a spurious trip received while planned maintenance testing was being performed on Channel C of the plant protection system. With one channel already in bypass for testing, Technical Specifications required that the inoperable bistable be placed in trip within one hour. Whereas on previous occurrences, a technician would manually trip only the affected bistable, the short time frame allowed by Technical Specifications in this case necessitated that operators use an approved alternate method of removing a connector cable in the cabinet. This method inserted trips for both the affected bistable, as well as, unnecessarily inserting trips in 11 other bistables in Channel 'A,' such that a single signal from any corresponding bistable in either Channel 'B' or 'D' would result in an automatic reactor trip, somewhat increasing the chance of a spurious plant trip.

Following the spurious actuation on September 16 and questioning from the team, plant maintenance personnel developed a formal troubleshooting plan in accordance with site Procedure SO23-XV-2, Troubleshooting Plant Equipment and Systems, Revision 0.

On September 21, technicians implementing the troubleshooting plan discovered a piece of foreign material lodged in the edge connector for one of the circuit cards.

Technicians suspected that the debris (a plastic key insert for the edge connector) had become lodged in the connector when the card was removed to be calibrated approximately 2 weeks earlier, shortly before the first spurious trip occurred. The inspectors noted that had the troubleshooting plan been implemented following the first spurious actuation, and the guidance of Generic Letter 91-18 had been followed prior to declaring the system operable, the short-term LCO and additional unnecessary PPS trips on September 16 would not have occurred.

(c)

Analysis.

The performance deficiency associated with this event was a failure to follow the guidance of Generic Letter 91-18, which resulted in receiving a repeat spurious trip signal coincident with scheduled testing on a redundant channel, necessitating unplanned LCO actions which placed the plant closer to a trip condition. This finding was greater than minor because it affected the equipment performance attribute of the Initiating Events cornerstone and directly affected the cornerstone objective of limiting events that challenge plant stability. The finding was of very low safety significance because it did not contribute to the likelihood of a LOCA initiator, did not increase the likelihood of both a reactor trip and simultaneously affect the unavailability of mitigation equipment, and did not increase the likelihood of a fire or flooding event as described in the Significance Determination Process Phase 1 screening worksheet.

(d)

Enforcement.

No violation of regulatory requirements occurred since Technical Specification action statements were followed while the failures existed. This finding is identified as FIN 0500362/2004006-02, "Inadequate Troubleshooting and Postmaintenance Testing for an Intermittent Failure in Plant Protection System."

4OA3 Event Followup

.1 (Closed) Licensee Event Report 05000362/2004001-00, Manual Reactor Trip Due to

Sea Grass at Intake Structure a.

Introduction.

A Green self-revealing finding was identified because plant procedures for responding to debris at the plant intake were inadequate for responding to a relatively large influx of sea grass. This performance deficiency contributed to a manual trip of Unit 3 on July 24, 2004.

b.

Description.

On July 24, 2004, plant operators manually tripped Unit 3 due to sea grass degrading the circulating water intake. Unusual tides and rough surf conditions contributed to a heavy influx of sea grass at the sea water intake. As the tide approached its low point, the grass influx increased to the point where the traveling screens were running continuously in automatic. At 3:30 a.m., operator action was necessary per the alarm response procedure to manually operate the screens to improve flow because grass clogging reduced flow through the screens sufficiently to cause a 15-21 inch level difference across the screens. After about 20 minutes of manual operation, level differential appeared steady. The local operator placed the screens in automatic and left the area. At 4:00 a.m., grass buildup on Screen 3 was sufficient to overload the screen and break the shear pin (weak link by design) on the drive assembly. This caused the traveling screen to stop rotating and prevented further grass removal. When the local operator returned a few minutes later, the level differential across the screens was 32 inches. The operator attempted manual operation of the remaining screens, but the rest of the shear pins broke over the next 15-30 minutes. The maximum level differential observed was 72 inches. The lowest tide level was -1.8 feet at 5:05 a.m., which was the lowest observed since January 2004.

At 4:34 a.m., control room operators began reducing power. One circulating water pump was secured in order to reduce the rate of grass intake at the screens. In order to secure a second pump to further reduce circulating water intake flow, procedures required reducing power below 60 percent. However, when power was approximately 80 percent, indications of circulating water pump flow oscillations were observed. At 4:45 a.m., operators manually tripped the reactor.

The licensee performed a formal root cause evaluation of this event under Action Request 040600324, and concluded that the root cause was inadequate operating instructions. The licensee concluded that the only way to have successfully responded to these conditions was to recognize the need to reduce circulating water intake flow early enough to avoid overwhelming the traveling screens. Neither SO23-2-5, "Operating Instruction for Circulating Water System," Revision 21, nor SO23-2-7, "Operation of Traveling Rakes and Screens," Revision 10, nor Alarm Response Instruction SO23-15-99.C, "Annunciator Panel 99.C," Revision 6 provided guidance to reduce flow rates if the screens were being overwhelmed.

The team noted that the procedure changes made in response to this event would improve operator's ability to recognize similar conditions and respond better, but the changes stopped short of being proactive.

The team noted that the low tide level was within the design low water level (-2.6 feet)listed in the Updated Final Safety Analysis Report (UFSAR). Also, the plant was designed to withstand the effects of a tsunami, which bounded the surf conditions experienced. However, the impact of sea grass on plant operation was not discussed in the UFSAR. The team concluded that having inadequate guidance in procedures for responding to abnormal, off-normal, or alarm conditions associated with debris buildup at the plant intake was a performance deficiency that was within the licensee's ability to foresee. Specifically, these procedures did not alert operators that debris buildup at the intake screens which exceeded the self-cleaning capability of the screens may require reduction of power level and circulating water flow.

The team questioned the thoroughness of the root cause evaluation, since operator performance during the event was not formally evaluated. It appeared likely that allowing the operator to place the traveling screens back in automatic and leave the area was inappropriate because it allowed excessive buildup of sea grass which manual action had been adequately coping. However, the licensee's contention was that the quantity of grass could only be successfully dealt with by reducing circulating water flow rate, and manual operation merely prolonged the time until a trip would have been necessary. The licensee provided the team with a position paper on this issue, which is included as Attachment D to this report.

Corrective actions improved procedures to better alert operators to the potential need to reduce intake flow and reduce power, as well as providing more operational flexibility to secure circulating water pumps and manually operate multiple traveling screens.

Guidance was also provided to operators to remain aware of ocean conditions which might create a large influx of sea grass and to perform proactive briefings on operator response. These actions adequately addressed the issues identified in the root cause evaluation.

c.

Analysis.

The team determined that the finding is a performance deficiency because the SCE alarm response and system operating procedures did not provide adequate instructions to operators to allow them to appropriately respond to more significant episodes of sea grass intrusion at the plant intake. Such episodes were expected during periods of rough surf and large tides. This finding was more than minor because it was similar to Example 4.b (procedure error that resulted in plant trip) in Manual Chapter 0612, Appendix E. This finding affected the Initiating Events cornerstone objective for procedure quality to limit the likelihood of those events that upset plant stability during power operations, because guidance was not provided for a controlled plant-level response to debris buildup that exceeded the removal capability of the screens. This issue screened as having very low safety significance in a Phase 1 assessment, since the finding affected only the initiating events cornerstone, and it affected a transient initiator without affecting transient mitigation equipment.

It should be noted that this performance deficiency was self-revealing as a result of the event described in this Licensee Event Report. However, this performance deficiency was a contributor to the event. The corrective actions were appropriate to address the lack of guidance. However, the guidance might not prevent all possible future trips due to debris influx.

d.

Enforcement.

No violation of regulatory requirements occurred. The team determined that the finding did not represent a noncompliance because it related to non-safety-related secondary plant equipment. This finding is identified as FIN 0500362/2004006-03, "Inadequate Procedure to Respond to Debris on Circulating Water Traveling Screens."

.2 (Closed) LER 05000361/2004002-00, Manual Reactor Trip in Response to Loss of Main

Feedwater On April 10, 2004, Unit 2 was operating at 97 percent power when both feedwater pumps tripped on high discharge pressure. Control room operators manually tripped the reactor in response. The auxiliary feedwater system actuated as expected.

Troubleshooting determined that the main feedwater regulating valve and its associated bypass valve closed due to a control system problem. This condition caused the high discharge pressure condition at the feedwater pumps. The licensee was able to locate one ground in the control system, where a wire was pinched and its insulation cracked.

When the wire was replaced, troubleshooters verified that the ground was cleared, but also realized this was an ungrounded system, so a second ground would be needed to cause the indications actually observed during this event. The licensee was unable to locate a second ground, and presumed it was either intermittent or was cleared during troubleshooting and repair activities. No subsequent monitoring was performed to check whether the second ground came back, even though the licensee had a formal process to do so ("Spurious Actuation" actions within the corrective actions program). The licensee concluded that the pinched wire was a result of initial fabrication, since no normal maintenance work would have disturbed the terminal strip that had pinched the wire. Therefore, corrective actions were implemented to upgrade the control system to a digital system which would be less susceptible to this failure mode, and to implement ground checks within the feedwater control system during refueling or forced outages.

The team noted that the ground checks would not prevent grounds, but were intended to identify and correct an initial ground before a second one could appear and have an actual impact on system operation. Also, Unit 2 (the unit that tripped) would receive a new control system before the new ground checks would be performed. The licensee did not believe that such checks could be performed on line, since they could cause a trip. Therefore, the frequency of the checks could not be increased. Based on this, corrective actions were reasonable to address the observed issues.

The team also reviewed the circumstances surrounding a Unit 2 loss of feedwater event due to a failure in the feedwater control system on November 2, 2002, and concluded that the earlier event did not involve a ground. No issues were identified during the review of this LER.

4OA6 Meetings, Including Exit

On September 23, 2004, the inspectors presented the inspection results to Mr. D. Nunn, Vice President, Engineering and Technical Services, and members of his staff who acknowledged the findings. The inspectors confirmed that proprietary information was not provided or examined during this inspection.

Following in-office inspection during September 27 to October 13, a supplemental exit was conducted by telephone with Dr. R. Waldo on October 26, 2004, to present the results of the issues regarding electrolytic capacitors and the plant trip due to sea grass.

ATTACHMENTS:

A --Supplemental Information B --Southern California Edison Position Paper: Electrolytic Capacitor Preventative Maintenance Regulatory Basis C --Southern California Edison Position Paper: SCE Assessment of Engine Systems, Inc. Part

N o

t i

f i

c a

t i

o n

D -- Southern California Edison Position Paper: Regulatory Assessment LER 3-2004-001, Unit 3 Trip Due to Sea-grass Intrusion ATTACHMENT A

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee personnel

C. Anderson, Manager, Emergency Preparedness

B. Ashbrook, Project Manager, Chemistry

D. Axline, Technical Specialist, Nuclear Regulatory Affairs

R. Bockhorst, Manager, Maintenance Engineering

D. Breig, Manager, Maintenance Engineering

C. Cates, Project Manager, Maintenance

W. Frick, Manager, Nuclear Safety Concerns

D. Hall, Manager, Operations

M. Love, Manager, Maintenance

J. Madigan, Manager, Health Physics

C. McAndrews, Manager, Nuclear Oversight and Assessment

M. McBrearty, Engineer, Nuclear Regulatory Affairs

D. Niebruegge, Manager, Maintenance Engineering

D. Nunn, Vice President, Engineering and Technical Services

J. Osborne, Project Manager, Nuclear Oversight and Assessment

R. Sandstrom, Manager, Nuclear Training

A. Scherer, Manager, Nuclear Regulatory Affairs

M. Short, Manager, Systems Engineering

O. Thomsen, Manager, Nuclear Fuels Management

J. Todd, Manager, Security

C. Townsend, Manager, Work Control

R. Waldo, Station Manager

J. Wambold, Vice President, Nuclear Generation

T. Yackle, Manager, Design Engineering

NRC personnel

C. Osterholtz, Senior Resident Inspector

A

-2-

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

05000362/2004009-01 NCV Failure to Control Shelf-life of Spare Assemblies with Electrolytic Capacitors (Section 4OA3.(e)(1))

05000362/2004009-02 FIN Inadequate Troubleshooting and Postmaintenance Testing for an Intermittent Failure in Plant Protection System (Section 4OA2(e)(2))

05000362/2004009-03 FIN Inadequate Procedure to Respond to Debris on Circulating Water Traveling Screens (Section 4OA3.1)

Closed

05000362/2004001-00 LER Manual Trip Due to Sea Grass at Intake Structure (Section 4OA3.1)

05000361/2004002-00 LER Manual Reactor Trip in Response to Loss of Main Feedwater (Section 4OA3.2)