IR 05000323/2009009
| ML100700281 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 03/09/2010 |
| From: | Caniano R Division of Reactor Safety IV |
| To: | Conway J Pacific Gas & Electric Co |
| References | |
| IR-09-009 | |
| Download: ML100700281 (41) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION
REGION IV
612 EAST LAMAR BLVD, SUITE 400 ARLINGTON, TEXAS 76011-4125 March 9,2010 John Senior Vice President-Energy Supply and Chief Nuclear Officer Pacific Gas and Electric Company P.O. Box 3 Mail Code 104/6/601 Avila Beach, California 93424 Subject: DIABLO CANYON POWER PLANT - NRC SPECIAL INSPECTION REPORT 05000323/2009009
Dear Mr. Conway:
On January 25,2010, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Diablo Canyon Power Plant reviewing the circumstances and extent of condition related to the failure of emergency core cooling system interlocks discovered by your staff on October 23, 2009. The enclosed special inspection report documents the inspection findings, which were discussed on January 26, 2010, with Mr. James Becker, Site Vice President, and other members of your staff and members of the public at the Embassy Suites Hotel, 333 Madonna Road, San Luis Obispo CA (ADAMS ML100390010).
The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's ruies and reguiations and with the conditions of your iicense.
The inspection team reviewed selected procedures and records, observed activities, and interviewed personnel.
This report documents two NRC identified violations of very low safety significance (Green) and one Severity Level IV violation. All of these findings were determined to involve violations of NRC requirements. However, because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as noncited violations, consistent with Section VI. A. 1 of the NRC Enforcement Policy. If you contest the violations or the significance of the noncited violations, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 612 E.
Lamar Blvd, Suite 400, Arlington, Texas, 76011-4125; the Director, Office of Enforcement, U.S.
Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the NRC Resident Inspector at the Diablo Canyon Power Plant. In addition, if you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Resident Inspector at the Diablo Canyon Power Plant.
The information you provide will be considered in accordance with inspection Manual Chapter 0305.
Pacific Gas and Electric Company -2-In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, and its enclosure, will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of NRC's document system (ADAMS).
ADAMS is accessible from the NRC Web site at (the Public Electronic Reading Room).
Sincerely,
~~
Rote8niano, Director Division of Reactor Safety Region IV Docket: 50-323 License: DPR-82
Enclosure:
NRC Inspection Report 0500323/2009009 wiAttachments:
Attachment 1: Documents Reviewed Attachment 2: Special Inspection Charter Attachment 3: Timeline of Events Attachment 4: Phase 3 Significance Determination Evaluation Attachment 5: System Drawing Attachment 6: Picture of Limit Switches
REGION IV==
Docket: 05000323 License: DPR-82 Report: 05000323/2009009 Licensee: Pacific Gas and Electric Company Facility: Diablo Canyon Power Plant, Unit 2 Location: 7 Y2 miles NW of Avila Beach Avila Beach, California Dates: November 30, 2009 through January 25, 2010 Inspection Michael Runyan, Senior Reactor Analyst Team: Michael Peck, Senior Resident Inspector, Diablo Canyon Paul Elkmann, Senior Emergency Preparedness Inspector Megan Williams, Reactor Inspector Approved By: Roy Caniano, Director, Division of Reactor Safety Region IV
~
- I - Enclosure
SUMMARY OF FINDINGS
IR 05000323/2009009; 11/30/2009 -1/25/2010: Diablo Canyon Power Plant, Unit 2, Special
Inspection Report.
The report covers a special inspection by a region-based inspection team. Two Green non cited violations of very low significance and one Severity Level IV noncited violation were identified.
The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, "Significance Determination Process." Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG 1649, "Reactor Oversight Process," Revision 4, dated December 2006.
NRC-Identified and Self-Revealing Findings
Comerstone: Mitigating Systems
- Green.
The inspection team identified a noncited violation of 10 CFR 50, Appendix B,
Criterion III, Design Control, which requires licensees to implement measures to assure that applicable regulatory requirements and the design basis are correctly translated into specifications, drawings, procedures, and instructions. These design control measures include verifying or checking the adequacy of design, such as by the performance of design reviews, by the use of alternate or simplified calculation methods, or by the performance of a suitable testing program. Specifically, on February 16, 2008, plant engineering personnel failed to implement the design control process for a modification to the Unit 2 residual heat removal containment sump valves when they inappiOpriately used maintenance piOcedures to reduce the valve stroke lengths from 15.5 to 13.8 inches. The invalid design change resulted in the inoperability of both emergency core cooling trains between April 8, 2008, (when the plant entered Mode 4) and October 22,2009. The reduced sump valve stroke length also caused a portion of the sump valve disc to remain in the residual heat removal suction flow path, reducing the available residual heat removal pump net positive suction head. The licensee entered this condition into their corrective action program as Notification 50277252.
The inspection team concluded that the failure of plant engineering to use the design control process was a performance deficiency within the licensee's ability to foresee and correct.
The finding is more than minor because it affected the Mitigating Systems Cornerstone initial design control attribute and objective to ensure the availability, reliability, and capability of systems that respond to initiating events. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding required a Phase 2 analysis because the finding represented the loss of a safety system function. The Phase 2 analysis determined that this finding was potentially greater than Green; therefore, a Phase 3 analysis was completed by a regional senior reactor analyst. The Phase 3 analysis determined that this issue was of very low safety significance (Green), owing principally to the fact that operators could have opened the affected valves locally with a very high probability of success. This finding had a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensee did not thoroughly evaluate the failure of the valves to meet the specified stroke time to ensure that the resolution fully addressed the causes and extent of condition, as necessary P.1(c).
- Green.
The inspection team identified a noncited violation of 10 CFR 50, Appendix B,
Criterion XI, Test Control, which requires that a test program be established to assure that all testing required to demonstrate that structures, systems, and components will perform satisfactorily in service. Specifically, the licensee failed to perform testing to assure that the interlock circuitry associated with the residual heat removal containment sump valves (SI-2-8982A and B) would perform satisfactorily in service following a modification on February 16, 2008, that changed the stroke lengths. As a consequence, remote operation of the valves needed to initiate high pressure recirculation was lost for an entire operating cycle. The licensee entered this issue into their corrective action program as Notification 50277252.
The failure to establish adequate post-modification testing requirements was a performance deficiency within the licensee's ability to foresee and correct. The finding is more than minor because the Mitigating Systems Cornerstone initial design control attribute and objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences was affected. Using Manual Chapter 0609.04,
"Phase 1 - Initial Screening and Characterization of Findings," the finding required a Phase 2 analysis because the finding represented the loss of a safety system function. The Phase 2 analysis determined that this finding was potentially greater than Green; therefore, a Phase 3 analysis was completed by a regional senior reactor analyst. The Phase 3 analysis determined that this issue was of very low safety significance (Green), owing principally to the fact that operators could have opened the affected valves locally with a very high probability of success. This finding had a crosscutting aspect in the area of problem identification and resolution associated with the operating experience component because the licensee failed to implement a corrective action program with a threshold sufficient to identify issues associated with the failure to meet sump valve post-modification test acceptance criteria [P.1 (a)].
- Severity Level IV. The inspection team identified a noncited violation of 10 CFR 50.59, which states that a licensee may make changes to the facility as described in the final safety analysis report without obtaining a license amendment if the change does not result in a departure from a method of evaluation described in the final safety analysis report used in establishing the design bases or in the safety analyses. This regulation further requires the licensee to include a written evaluation providing the basis for concluding that a license amendment is not required. On November 21,2005, the licensee failed to provide a written evaluation concluding that a license amendment was not required for a change to the facility as described in the final safety analysis report. Specifically, the licensee identified a condition where large differential pressure across the residual heat removal suction containment sump valves could cause them to fail to open during certain small break loss of coolant accidents. On October 5, 2005, the licensee revised Emergency Operating Procedure E-1, "Loss of Reactor or Secondary Coolant," to add an operator action to align component cooling water to the residual heat removal heat exchanger. On June 16, 2009, the licensee again revised Emergency Operating Procedure E-1 to specify that operator action to align component cooling water within 30 minutes was a time critical operator action. The licensee did not evaluate either change to determine if prior NRC approval was required for the new manual actions. The licensee entered this issue into their corrective action program as Notification 50276288.
The failure of the licensee to perform a 10 CFR 50.59 evaluation of a new manual action supporting the plant's design basis was a performance deficiency within the licensee's ability to foresee and correct. The inspectors evaluated this issue using the traditional enforcement process because the performance deficiency had the potential for impacting the NRC's ability to perform its regulatory function. The inspectors concluded that the issue was more than minor because of a reasonable likelihood that the change to the facility would require Commission review and approval prior to implementation. The inspectors also evaluated the significance of this issue under the Significance Determination Process using Inspection Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings." The inspectors concluded that the issue affected the Mitigating Systems Cornerstone and screened Green because the finding was a design or qualification deficiency confirmed not to result in loss of operability. The issue was classified as Severity Level IV because the violation of 10 CFR 50.59 involved conditions resulting in very low safety significance by the significance determination process. This finding had a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensee did not thoroughly evaluate the change to the facility as described in the Final Safety Analysis Report Update to determine if prior NRC approval was required P.1(c).
B. Licensee Identified Findings None
REPORT DETAILS
REACTOR SAFETY
Cornerstone: Mitigating Systems
1R1 Special Inspection Scope Event Description On October 22, 2009, during refueling operations, licensee technicians identified that the Unit 2 residual heat removal (RHR) heat exchanger discharge valves to the high and intermediate head pumps (SI-2-8804A1B) and to the containment spray pumps (CS-2-9003A/B) would not open from the control room. This condition resulted from an incorrect manipulation of limit switches associated with the RHR containment sump suction valves (SI-2-8982A1B). The SI-2-8804A1B valves open to provide RHR pump discharge to the suction of the intermediate and high head emergency core cooling pumps, in order to align the emergency core cooling system to the cold leg recirculation mode following an accident. The CS-2-9003A1B valves open to provide recirculation suction to the containment spray pumps.
Inspection Charter The inspection charter required the team to
- (1) develop a chronology (time-line) related to the failure of emergency core cooling system interlocks and a description of the motor-operated valve interlock operation,
- (2) evaluate the licensee's root cause analysis and corrective actions for the known failures, and assess the adequacy of the licensee's corrective actions,
- (3) evaluate the extent of the condition and extent of cause reviews performed by the licensee in response to this event,
- (4) evaluate the adequacy of the licensee's post-rnaintenance testing program for activities involving motor-operated vaive interlocks,
- (5) evaluate the licensee's response to industry operating experience to determine whether the licensee took timely and effective measures to address the issues, including review of the licensee's commitments to Generic Letter 96-01, "Testing of Safety Related Circuits,"
- (6) collect data as necessary to complete a determination of risk significance for the failed interlocks, including an assessment of the circumstances of each failure (i.e., plant conditions, testing mode), what equipment would not function, time since previous successful performance, and the potential for operator recovery actions, and
- (7) determine if the licensee met NRC reporting requirements of 10 CFR 50.72 and 10 CFR 50.73.
The team conducted their reviews in accordance with NRC Inspection Procedure 93812, "Special Inspection Procedure." The special inspection team reviewed procedures, corrective action documents, and design and maintenance records for the equipment of concern. The inspection team interviewed key station personnel regarding the events, reviewed the root cause analysis, and assessed the adequacy of corrective actions. The team walked down and inspected the equipment in the field. A list of specific documents reviewed is provided as Attachment 1. The charter for the special inspection is provided
as Attachment 2. The timeline of events is provided as Attachment 3. The significance determination process (SOP) Phase 3 analysis is provided as Attachment 4.
1R2 Review of the Maintenance Activity on Valves SI-2-8982A and SI-2-8982B during Refueling Outage 2R 14 (including post-maintenance testing)
a. Inspection Scope
The inspection team conducted a thorough and systematic inspection of the actions taken by licensee personnel to modify and test the RHR containment sump valves SI-2-8982A1B. The inspection team conducted this review using the guidance provided in Inspection Procedure 93812, "Special Inspection." The inspection team reviewed design, maintenance, and testing documentation and interviewed licensee personnel.
b. Findings and Observations
On October 22, 2009, the licensee identified that the RHR containment sump suction valve interlock switches (Valves SI-2-8982A1B) were misaligned on both Unit 2 trains, that this misalignment occurred on February' 16, 2008, and that the equipment had been returned to service on February 21,2008. The misaligned interlock switches would have prevented remote manual operation of the motor operated valves supplying RHR discharge to the suction of the safety injection and centrifugal changing pumps (Valves SI-2-8804A1B) and containment spray pumps (Valves CS-2-9003A1B). The loss of capability to operate Valves SI-2-8804A1B remotely rendered both trains of the emergency core cooling system inoperable. The loss of capability to operate Valves CS-2-9003A1B rendered both trains of containment spray inoperable (see drawing in attachment 5), though the loss of this capability had minimal impact on risk because the containment spray system would not be needed for the size of loss of coolant accidents that VJould render Va!ves CS=2=9003A1B inoperable from the control room.
The motor operators for the RHR containment sump valves (SI-2-8982A1B) include a number of limit and interlock switches (see picture in Attachment 6). Four of these switches are operated from mechanical cams or rotors. Each limit switch is actuated following a preset number of cam rotations corresponding to valve stem travel. Rotor #1 controlled the total stroke distance of the valve stem. The valve motor operator stops after the Rotor #1 limit switch engages on the open stroke. The Rotor #3 limit switch
\A/8S included in a permissive circuit that allo\AJed remote manual operation of Valves S!-
2-8804A1B and CS-2-9003A1B from the control room. This interlock is designed to prevent plant operators from opening SI-2-8804A1B or CS-2-9003A1B before the containment sump valves (SI-2-8982A1B) are fully open. The Rotor #3 limit switch must be set to engage before the Rotor #1 limit switch engages for the interlock to operate successfully.
The plant emergency core cooling system incorporates three modes of post-accident operation:
1. Cold leg injection - The emergency core cooling system maintains core cooling by transferring water from the reactor water storage tank (RWST) to the reactor coolant
system. The emergency core cooling system will stay in this alignment until approximately 300,000 gallons of water have been transferred from the RWST to the reactor coolant system. This corresponds to an RWST level of 33 percent.
2. Cold leg recirculation - The RHR pump suction is realigned to the containment sump.
Water from the break is cooled by the RHR heat exchanger and pumped back to the reactor coolant system. The RHR discharge is also aligned to the suction of the high and intermediate head emergency core cooling system pumps to provide adequate net positive suction head. Plant operators are instructed to open remotely (using control room switches) the containment sump valves (SI-2-8982A1B) and RHR heat exchanger discharge flow valves (SI-2-8804A1B) to complete the transition to cold leg recirculation. This high pressure recirculation mode (through Valves SI-2-8804A1B) is primarily needed for small and medium-sized loss of coolant accidents.
For larger-sized loss of coolant accidents, the RHR system is independently capable of providing adequate recirculation flow.
3. Hot leg recirculation - The emergency core cooling system pump discharge is
aligned to the reactor coolant system hot legs.
In July 2005, the licensee received operating experience indicating that the motor operators of Valves SI-2-8982A1B may not have sufficient capacity during some types of loss of coolant accidents due to high differential pressure (Action Request A06431 07).
The motor operators for these valves were originally designed to open with a maximum of 64 psi differential pressure between the RHR suction piping and the containment sump. As described in the operating experience, this differential pressure may increase up to 450 psi (the suction relief valve setpoint) after long periods of RHR operation on minimum flow recirculation. This situation would likely occur following small break loss of coolant accidents where the reactor coolant system pressure remains higher than the shutoff head of the RHR pumps. To ensure reiiabie operation of the containrnent surnp valves, the licensee implemented the following changes:
1. Installed a higher gear ratio in the containment sump valves (SI-2-8982A1B) to
provide a higher opening thrust capability, and
2. Revised plant emergency operating procedures to direct plant operators to initiate
cooling to the RHR heat exchangers within 30 minutes of the accident.
The action to provide cooling to the RHR heat exchangers within 30 minutes lowered the maximum expected differential pressure from 450 psi to 368 psi. Both of these changes were needed to open the RHR sump suction valves from the control room with an acceptable margin.
Modification to the Containment Sump Valves In August 2007, the licensee completed the design work for Minor Modification A0702739 to change out the helical gear set on the containment sump valve (SI-2-8982A1B) motor operators. Plant engineers determined that the valve torque could be increased to open against a 368 psi differential pressure by increasing the pinion and
worm shaft gear ratios. Plant technicians completed the modification on February 14, 2008, per Work Orders C0214725 (for Valve SI-2-8982A) and C0214751 (for Valve SI-2-89828). Post-modification testing included Procedure STP V-3L 17, "Exercising Containment Recirc Suction Valves SI-2-8982A and SI-2-8982B." Procedure STP V-3L 17 was performed to verify that the open stroke time of the modified sump valves was within 25 seconds as specified in final safety analysis report update (FSARU)
Table 6.3-5, "Safety Injection to Recirculation Mode; Sequence and Timing of Manual Changeover." During the inspection, the licensee was not able to retrieve the completed post-modification test; however, the plant motor-operated valve engineer stated that the modified valve stroke time was greater than 25 seconds. The inspection team identified that the failure to meet the post-modification testing criteria was not entered into the corrective action system as required by Procedure AD13.ID2, "Post Modification Testing." Section 5.6.6 of this procedure stated that problems encountered during testing must be entered into the corrective action program to be resolved by the implementing organization. Section 5.6.7 also required a review by the test performer and the design change sponsor to verify that test acceptance criteria were met prior to releasing the equipment back to operations.
The valve engineer stated that the calculation supporting the helical gear set change concluded (incorrectly) that the modified sump valve stroke time would be less than 25 seconds. Consequently, the failure to meet the stroke time specification was unexpected. After discovery of the non-conforming condition during testing, the valve engineer determined that the sump valve stroke time could be brought back into compliance by shortening the containment valve (SI-2-8982A1B) stroke length from 15.5 inches to 13.8 inches. The valve engineer also concluded, based on an informal analysis, that the valve disc would be fully withdrawn from the RHR flow path despite the reduced valve stroke length. The valve engineer calculated the new Rotor #1 limit switch values corresponding to the shortened stroke and entered these values into the controlled design component database. Plant valve technicians subsequently reset the Rotor #1 limit switches to the new design values and returned the sump valves to service on February 21, 2008. No test was performed to ensure that the interlocks controlled on Rotor #3 were operating satisfactorily.
The inspection team determined that the valve engineer bypassed station design control processes when specifying the new sump valve stroke length. The licensee controlled changes to sump valve limit switch settings by the use of Drawing M000073, "Torque and Limit Switch, Spring Pack, Control Transformer Changes for MOVs," Revision 4.
Changes to the information contained in Drawing M000073 required design engineering personnel to review all affected calculations and the plant test group to scope applicable testing to demonstrate that the component will perform as required. The inspection team concluded that the failure of the valve engineer to follow the design control process resulted in both the failure of plant personnel to coordinate the adjustment of Rotor #3 to ensure proper function of the interlock switches and the failure to verify that the new stroke length would not occlude the RHR flow area.
On December 1, 2009, the inspection team requested that the licensee provide documentation to verify that the RHR pump sump suction valve stroke length modification did not adversely affect the RHR flow path. The licensee stated that the
requested documentation was not available. The iicensee subsequently determined that the valve disc would not be fully withdrawn with the modified full-open position, that the disc would intrude 0.54 inches below the top of the pipe cross-sectional area, and the disc would block approximately 4 percent of the RHR suction flow path. The licensee completed a prompt operability assessment (Notification 50286743) on December 3, 2009. The licensee concluded that 0.8 ft. of RHR pump net positive suction head margin was maintained in this condition. Before the valves were modified, 1.0 ft. of net positive suction head was available. The inspection team determined that, although the margin had decreased, the remaining available net positive suction head was acceptable.
Failure to Follow Design and Configuration Control Requirements
Introduction:
The inspection team identified a Green noncited violation of 10 CFR 50, Appendix B, Criterion III, Design Control, after a plant engineer improperly modified the Unit 2 RHR containment sump valves. This modification rendered both emergency core cooling system trains inoperable between April 8, 2008, and October 22, 2009.
Description:
On February 16, 2008, a plant engineer specified that the stroke length of RHR containment sump Valves SI-2-8982A/B be reduced from 15.5 to 13.8 inches. The manner in which this change was implemented disabled the emergency core cooling system permissive that allowed remote manual operation of Valves CS-2-9003A1B and Valves SI-2-8804A1B. The reduced sump valve stroke length also resulted in a portion of the valve disc to protrude into the system flow path, degrading the available net positive suction head to the RHR pumps. The inspection team identified that the plant engineer failed to meet plant design configuration control requirements as specified in Drawing M000073. This drawing required plant Design Engineering personnel to review the affected calculations and the Plant Test Group to scope applicable testing to demonstrate that the component would perform as required prior to retuning the sump valves to service. Because the design control process was circumvented, the stroke length was not verified by independent personnel nor was the modification evaluated against other design requirements. The loss of remote manual operation of Valves SI-2-8804A1B rendered the emergency core cooling system inoperable. Further, the shortened stroke length resulted in the valve disc occluding the valve flow area, reducing the available net positive suction head.
Analysis:
The failure of plant personnel to use the design control process for the RHR containment sump valve (SI-2-8982A1B) stroke length modification was a performance deficiency within the licensee's ability to foresee and correct. The finding is more than minor because the performance deficiency affected the Mitigating Systems Cornerstone design control attribute, related to plant modifications, and objective to ensure the availability, reliability, and capability of systems that respond to initiating events. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings,"
the finding required a Phase 2 analysis because the finding represented a loss of a safety system function. Because the Phase 2 analysis concluded that the finding was potentially of low to moderate significance, a Phase 3 analysis was completed by a regional senior reactor analyst. The Phase 3 analysis determined that this issue was of very low safety significance (Green), owing principally to the fact that operators could
have opened the affected valves locally with a very high probability of success. This analysis is included as Attachment 4 to this report.
This finding had a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensee did not thoroughly evaluate the failure of the valves to meet the specified FSARU stroke time to ensure that resolutions addressed causes and extent of conditions, as necessary
Enforcement:
Title 10 CFR 50, Appendix S, Criterion III, Design Control, requires licensees to implement measures to assure that applicable regulatory requirements and the design basis are correctly translated into specifications, drawings, procedures, and instructions. These design control measures include verifying or checking the adequacy of design, such as by the performance of design reviews, by the use of alternate or simplified calculation methods, or by the performance of a suitable testing program.
Contrary to the above, on February 16, 2008, the licensee did not check the adequacy of the design through the performance of design reviews, by the use of alternate or simplified calculation methods, or by the performance of a suitable testing program.
Specifically, the licensee changed the design specification for the stroke length of Valves SI-2-8982A1S without verifying or checking the adequacy of the design and did not utilize design control procedures that would have engaged the licensee engineering organization. Because this finding is of very low safety significance and was entered into the corrective action program as Notification 50277252, this violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000323/2009009-01, "Failure to Follow Design and Configuration Control Requirements."
Failure to Conduct an Adequate Post-Modification Test
Introduction:
The inspection team identified a Green noncited violation of 10 CFR 50, Appendix S, Criterion XI, Test Control. Specifically, the licensee failed to conduct an adequate post-modification test of the Unit 2 RHR containment sump valves prior to placing the components back in service on February 21, 2008.
Description:
The licensee implemented Modification A0702739 to change out the helical gear set on the containment sump valve (SI-2-8982A and S) motor operators. The post-modification test specified the use of PiOcedure STP V-3L 17 to verify that the sump valves would stroke fully open in less than 25 seconds. Plant technicians identified that the modified sump valves did not meet the post-modification test acceptance criteria on February 16, 2008. However, the technicians did not document the failure to meet the post-modification test acceptance criteria. Procedure AD13.ID2, "Post Modification Testing," Section 5.6.6, required that problems encountered during testing be entered into the corrective action program to be resolved by the implementing organization.
Procedure AD13.ID2, Section 5.6.7, also required the test performer and the design change sponsor conduct a review to verify that test acceptance criteria were met prior to placing the equipment back in service. A plant engineer shortened the stroke length to address the test discrepancy but failed to ensure that additional post-modification testing
'vA./as seoped to
- (1) verify that the neVJ Rotor #1 setting vi/ould not adversely affect the
Rotor #3 interlock circuitry for RHR heat exchanger discharge Valves SI-2-8804NB and CS-2-9003NB, and
- (2) ensure that the valve disc in the fully open position would clear the RHR flow path. The inspection team concluded that the failure to enter the stroke time problem into the corrective action system was the underlying cause of the performance deficiency.
Analysis:
The failure of plant personnel to use appropriate procedures to address a noncompliance with test acceptance criteria and the resultant failure to perform a necessary post-modification test was a performance deficiency. The finding is more than minor because the performance deficiency affected the Mitigating Systems Cornerstone design control attribute, related to plant modifications, and objective to ensure the availability, reliability, and capability of systems that respond to initiating events. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding required a Phase 2 analysis because the finding represented a loss of a safety system function. Because the Phase 2 analysis concluded that the finding was potentially of low to moderate significance, a Phase 3 analysis was completed by a regional senior reactor analyst. The Phase 3 analysis determined that this issue was of very low safety significance (Green), owing principally to the fact that operators could have opened the affected valves locally with a very high probability of success. This analysis is included as Attachment 4 to this report.
This finding had a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensee did not enter the failure to meet the post-modification test acceptance criteria into the corrective action system [P.1 (a)].
Enforcement:
Title 10 CFR 50, Appendix B, Criteria XI, Test Control, requires that a test program be established to assure that all testing required to demonstrate that structures, systems, and components will perform satisfactorily in service is identified and performed in accordance with written test procedures which incorporate the requirements and acceptance limits contained in applicable design documents.
Contrary to the above, on February 16, 2008, the licensee failed to assure that testing demonstrated components would perform satisfactorily in service. Specifically, the licensee failed to perform a test of the interlock circuitry to assure that Valves SI-2-8804A1B and CS-2-9003A1B could be opened remotely from the control room. Because this finding is of very low safety significance and was entered into the corrective action program as Notification 50277252, this violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy:
NCV 0500032312009009-02, "Failure to Conduct an Adequate Post-Modification Test."
Failure to Evaluate a Change to the Facility as Described in the Final Safety Analysis Report Update Associated with the Addition of Manual Actions in the Safety Analysis
Introduction:
The inspection team identified a Severity Level IV noncited violation of 10 CFR 50.59 after the licensee failed to perform an evaluation of new manual actions added to an emergency procedure to ensure that the emergency core cooling system met the design basis.
Description:
The inspection team determined that the licensee failed to perform an evaluation to determine if prior NRC approval was required for the introduction of new manual actions into the plant's accident analysis design basis. The licensee evaluated operational experience in Action Request 0643107 that revealed conditions under which the RHR suction containment sump valves could fail to open during certain small break loss of coolant accidents. This evaluation included a review of RHR system pressurization following a loss of coolant event. The licensee concluded that the differential pressure across the sump valves would be within the capability of the sump valve motor operators if plant operators aligned cooling to the RHR heat exchangers within 30 minutes of a loss of coolant accident. On November 21,2005, the licensee concluded without an evaluation that prior NRC approval was not required for the introduction of a new method for demonstrating that core cooling could be achieved. On October 5,2005, the licensee revised Emergency Operating Procedure E-1, "Loss of Reactor or Secondary Coolant," to add the requirement to align component cooling water to the RHR heat exchanger. On June 16, 2009, the licensee again revised Emergency Operating Procedure E-1 to specify that operator action to align component cooling water within 30 minutes was a time critical operator action. The licensee did not perform an evaluation of either procedure change to determine if prior NRC approval was required for introducing the new manual actions into the design basis.
Regulatory Guide 1.187, "Guidance for Implementation of 10 CFR 50.59, Changes, Tests, and Experiments," stated that NEI 96-07, "Guidelines for 10 CFR 50.59 Evaluations," Revision 1, provides methods that are acceptable to the NRC staff for complying with the provisions of 10 CFR 50.59. NEI 96-07, Section 4.2.1, required an evaluation for the changes to Emergency Operating Procedure E-1 because the activity was a change to procedures as described in the FSARU demonstrating that emergency core cooling design functions could be accomplished.
Analysis.
The failure of the licensee to perform an adequate evaluation in accordance 10 CFR 50.59 and the endorsed guidelines of NEI 96-07 prior to changing the facility as described in the FSARU is a performance deficiency. The inspection team evaluated this issue using the traditional enforcement process, including NRC Enforcement Policy, Supplement I, Reactor Operations, because the performance deficiency had the potential for impacting the NRC's ability to perform its regulatory function. The inspection team concluded that the issue was more than minor because of a reasonable likelihood that the change to the facility would require Commission review and approval prior to implementation. The inspection team also evaluated the significance of this issue under the Significance Determination Process using Inspection Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings." The inspection team concluded that the issue affected the Mitigating Systems Cornerstone because the change affected the quality of plant emergency procedures. The finding was screened as having very low significance (Green) because the inspection team confirmed that the finding did not result in the loss of operability of a safety function or screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. Because the issue screened Green under the Significance Determination Process, the inspection team concluded that the finding was a Severity
Level IV violation. The finding had a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensee did not thoroughly evaluate the importance of the design basis change, such that the resolutions addressed relevant causes and the full extent of conditions, as necessary P.1(c).
Enforcement:
Title 10 CFR 50.59, "Changes, Tests and Experiments," states that a licensee may make changes to the facility as described in the final safety analysis report without obtaining a license amendment if the change does not result in a departure from a method of evaluation described in the final safety analysis report used in establishing the design bases or in the safety analyses. This regulation further requires the licensee to include a written evaluation providing the basis for concluding that a license amendment is not required. Contrary to the above, on November 21, 2005, the licensee failed to provide a written evaluation concluding that a license amendment was not required. Specifically, the licensee changed the facility as described in the final safety analysis report to incorporate new manual actions into the method of evaluation described in the FSARU used in establishing the design bases without performing a written evaluation concluding that a license amendment was not required. Because this finding is of very low safety significance and was entered into the corrective action program as Notification 50276288, this violation is being treated as a noncited violation in accordance with Section VI. A. 1 of the Enforcement Policy: NCV 05000323/2009009-03, "Failure to Evaluate a Change to the Facility as Described in the Final Safety Analysis Report Update Associated with the Addition of Manual Actions in the Safety
Analysis.
"
1 R3 Review of Root Cause Analysis and Licensee Corrective Actions
a. Inspection Scope
The inspection team evaluated the licensee's root cause analysis (RCA), corrective actions, Event and Causal Factors Chart, Kepner-Tregoe analysis, and extent of condition review associated with the failure of the emergency core cooling valve interlocks. The inspection team also interviewed licensee personnel involved with the modification and testing of the Unit 2 RHR containment sump valves and personnel assigned to the licensee's root cause investigation team.
b. Findings and Observations
The inspection team concluded that the licensee's root cause and extent of condition review was incomplete. The licensee presented the inspection team with the results of the root cause evaluation as described in "Root Cause Analysis (RCA) Report, SI-2-8982A1B Interlocks Out of Adjustment, SAPN 50277252," November 19,2009. The RCA stated that that the licensee's investigation identified two root causes of the condition:
.. Legacy issue from 1991 resulted in Procedure MP E-53.1 OV1, "MOV Diagnostic testing with the Viper System," not including guidance for rotor coordination if a limit switch is reset. and
.. Calculation V-OJ (Sizing Calculation for Valves SI-2-8982NB) does not provide adequate guidance regarding the limitations for its use in predicting actual stroke time. That resulted in the Design Change Development Group relying on the results of the calculation and not considering alternatives to meet the 25-second stroke time specified in the licensing basis.
The license also concluded the following contributory causes:
.. Conversion of STP V-7B, "Test of Engineered Safeguards, Valve Interlocks and RHR Pump Trip from RWST Level Channels," to PEP V-7B, "Test of ECCS Valve Interlocks," facilitated an organizational decision to not perform V-7B in its entirety every outage in order to reduce critical path time and/or outage duration.
.. There was no rigorous method of evaluating, documenting, and communicating information regarding work performed that would be needed to make decisions about conditional post-maintenance testing. Informal and incomplete communication between the motor-operated valve engineer and the post-maintenance testing coordinator led to the rejection of PEP V-7B as a post maintenance test (this seemed to have resulted from a misunderstanding that the interlocks were not affected because Rotor #3 was not adjusted), and
.. A legacy issue from 1998, in Calculation V-07, Appendix K, resulted in narrative information not being formatted for ready retrieval or use. That led to a human error by the engineer having to remember the need to coordinate Rotor 3 with Rotor 1.
The licensee's RCA considered motor-operated valve rotor coordination as a maintenance activity. However, plant procedures only provided for maintenance activities to restore rotor settings to the design values specified in the component database. The inspection team concluded that the licensee's focus on plant surveillance testing and corrective maintenance procedures prevented the licensee from recognizing that the change of the containment sump valve stroke length constituted an unauthorized modification. As a result, the RCA did not identify the significance of the failure of plant technicians to initiate a corrective action program record after initial post-modification testing identified that the 25-second stroke time acceptance was not met.
Had the issue been entered into the corrective action program, station procedures would have required the test performer and the design change sponsor to review the condition.
The inspection team concluded that plant design controls would have required action for rotor coordination and additional post-modification testing following a change in the specified stroke length before the sump valves were placed back in service.
Because the RCA did not recognize the existence of the unauthorized modification to the containment sump valves, the licensee's extent of condition failed to identify that the new valve stroke was too short to fully withdraw the disc from the RHR flow path. As a result, the licensee failed to identify a loss of margin in the available net positive suction head.
The inspection team also identified the following contributing conditions to the event:
" Some plant personnel interviewed by the inspection team were unaware of the differences between the testing requirements in Procedure AD13.ID4, "Post Maintenance Testing, and Procedure AD13.ID2, "Post Modification Testing."
" Some plant engineers were generally confused of the difference in requirements between maintenance and modification activities.
" Controls for the plant Component Database were not well understood by some engineering staff that had access to change safety-related design input values.
" Some plant engineers authorized to perform 10 CFR 50.59 screenings and evaluations were unaware of the NEI 96-07 screening criteria for safety evaluations.
The inspection team conciuded that the immediate actions taken by the licensee to correct the interlock problems in Unit 2 and to assure that the same problem did not exist in Unit 1 were piOmpt and adequate. Because the root cause analysis did not identify the failure to follow design control procedures, the licensee's extent of condition review was incomplete.
1R4 Review Recovery Actions (manual operation of valves)
a. Inspection Scope
The team reviewed procedures, conducted interviews, and performed an in-plant walkdown to gather information needed to estimate the probability that operators would be able to open Valves SI-2-8804A1B locally in time to prevent an interruption of core cooling following a loss of coolant accident.
b. Findings and Observations
Emergency Operating Procedure EOP E-1, "Loss of Reactor or Secondary Cooling,"
instructs an operator to open Valves SI-2-8804A1B locally if they fail to open from the control room. Based on interviews with plant personnel and a walkdown of the access pathways and physical configuration of the valves, the inspection team determined that it would be likely take approximately 20 minutes to open the valves locally.
Both of these valves are in unobstructed locations in the auxiliary building with manual handwheel operators at approximately shoulder height above the floor. There were no ergonomic factors that would make local operation difficult. The valves require 360 turns to change the position from fully closed to fully open and, based on a test conducted by the licensee (though not witnessed by the NRC), it took 4 minutes 30 seconds to complete the operation. During this test, the valve was opened against a differential pressure approximating the maximum worst-case design conditions (368 psid).
This information indicated that local operation of Valves SI-2-8804A1B is uncomplicated and could be accomplished within 20 minutes. A possible exception to this timing assumption is the case where emergency onsite facilities are fully activated prior to reaching the procedural step where the valves fail to operate remotely. This condition would likely occur in a small-break but not a medium-break loss of coolant accident. In this case, the administrative complexities of executing recovery actions might extend the time to 40 minutes, according to the licensee. The inspection team considered this to be a conservative estimate, and concluded that the 20-minute estimate was appropriate for all scenarios.
1R5 Review Radiological Conditions associated with Recovery Actions
a. Inspection Scope
The inspection team performed an independent assessment of the potential radiation exposure to plant operators while manually operating Valves SI-2-8804A IB with a loss of coolant accident in progress. The inspection team used data from the FSARU to evaluate the maximum radiation exposure expected from RHR system piping and from the containment structure for loss of coolant accident leak rates from 50 to 1000 gpm.
The analysis considered both normal and design basis concentrations of radioactive material in reactor coolant water and assumed operator exposure times between 15 and 45 minutes. The inspection team also reviewed licensee calculations of the maximum radiation exposure to plant operators while manually operating Valves SI-2-8804A1B.
This was compared to the independent (NRC) assessment of radiation exposure to determine whether plant operators could remain within the licensee's emergency radiation exposure limits while manually operating Valves SI-2-8804A1B. The specific documents reviewed during this inspection are listed in the attachment.
b. Findings and Observations
The inspection team estimated the maximum plant operator's radiation exposure as 950 mrem to open Valve SI-2-8804A valve and 82 mrem to open Valve SI-2-8804B, for a total of 1032 mrem for a single operator opening both valves. The difference in dose estimates for these two valves was primarily because Valve SI-2-8804A is much closer to the containment wall. This assessment was based on radiation shine from the containment building, radiation shine from the residual heat removal piping, the design basis reactor coolant activity as described in the licensee's FSARU, a loss of coolant accident leak rate of 1000 gpm accumulating for two hours, and a 15-minute exposure time for the operator at each valve. The analysis was a bounding calculation that included the effects of radioactive decay and radiation shielding provided by the containment structure wall, but did not include radiation shielding internal to the containment structure or shielding provided by the auxiliary building walls, floors, ceilings, or intermediate equipment.
The licensee evaluated the plant operator's maximum potential radiation exposure as 167 mrem for each valve, for a combined radiation exposure of 334 mrem for a single operator to operate both valves. The licensee's radiation exposure calculation was
based on a 20-minute exposure to radioactive material in the residual heat removal system piping, using the highest concentration of radioactive material observed in the previous five years, as calculated using the MicroShield code. The licensee determined that the radiation shine from containment was negligible under these conditions.
The licensee established an administrative exposure limit of 10 rem for a recovery action of the type evaluated for this finding. Given the conservatisms in the calculations, it was anticipated that the actual exposure would be well less than 1 rem.
While the NRC and licensee's assessments differed in method and assumptions, both assessments indicated that necessary operator actions could be taken within the normal and emergency radiation exposure limits. The inspection team concluded that plant operators could manually operate Valves SI-2-8804NB during a loss of coolant accident without exceeding the licensee's emergency radiation exposure limits.
1 R6 Review Licensee Response to Industry Operating Experience
a. Inspection Scope
The inspection team reviewed the licensee's response to industry operating experience having relevance to the subject of this inspection.
b. Findings and Observations
OE 20893 (Catawba Small Break Loss of Coolant Accident (SBLOCA)) was issued in July 2005, identifying a potential problem with the sizing of motor-operators used to open the RHR sump suction valves. The OE stated that in certain sized loss of coolant accidents, where the RHR pumps are running in recirculation without injecting (RCS pressure greater than pump shutoff head), the resulting differential pressure can reach the limit of the suction relief valve (450 psid). This could result in a failure of the motor operator to open the valve because the initial sizing was not designed for this scenario.
Although the licensee determined that Unit 2 SI-2-8982 NB valves fell into this category and initiated their organizational corrective action (AR A06431 07), substitution of new gearing was not completed until February, 2008. The time frame for response was slowed by an initial confusion that the Catawba OE was not applicable to Diablo Canyon because of a difference in design between the two plants.
Generic Letter 96-01, issued January 10, 1996, required licensees to perform two actions in response to recurring incidents of safety-related logic throughout the industry:
- (1) Compare electrical schematic drawings and logic diagrams for the reactor protection system, emergency diesel generator (EDG) load shedding and sequencing, and actuation logic for the engineered safety features systems against plant surveillance test procedures to ensure that all portions of the logic circuitry, including the parallel logic, interlocks, bypasses and inhibit circuits, are adequately covered in the surveillance procedures to fulfill the technical specification (TS) requirements. This review should also include relay contacts, control switches, and other relevant electrical components within these systems, utilized in the logic circuits performing a safety function.
- (2) Modify the surveillance procedures as necessary for complete testing to comply with the technical specifications. Additionally, the licensee may request an amendment to the technical specifications if relief from certain testing requirements can be justified.
These actions were to be completed prior to startup from the first refueling outage commencing one year after the issuance of this generic letter.
The following note was also included:
Some licensees may have already performed the requested reviews and taken appropriate corrective actions. These licensees do not need to perform any additional review unless modifications have been made to the logic circuits for these systems. In these cases the modifications should be reviewed. Licensees are reminded that following modifications to safety-related logic circuits, full functional testing of the modification should be conducted. Licensees should not rely on routine surveillance testing to confirm proper performance of logic circuits following modifications.
The licensee responded before the final implementation due date with letters dated October 10 and December 15, 1997, identifying successful surveillance of solid state protection systems in response to the generic letter. On January 22, 1998, the licensee forwarded a letter to the NRC stating that all actions required of Generic letter 96-01 were completed.
The licensee's initial response letter (DCl 96-090) included a commitment to complete all actions by March 20, 1998. However, it did not include any detail of how the licensee intended to implement the required actions, i.e. specific staff assigned to particular actions, specific drawing reviews or other measures, or milestone dates for completion.
There was also no mention of any attention to the note quoted above regarding full functional testing of logic circuits following modifications.
The team identified as an observation that the response to Gl 96-01 was a missed opportunity for the licensee to implement actions that could have prevented the failed interlock condition; however, the team recognized that the generic letter was not specific to motor-operated valve interlock circuits.
1R7 Review Reporting Requirements The inspection team verified that the licensee reported the condition appropriately in accordance with the requirements of 10 CFR 50.72 and 10 CFR 50.73. The reports pursuant to these requirements were determined to be complete and accurate. The licensee event report was issued one day after the 60-day limit imposed by 10 CFR 50.73 had expired. The team concluded that this was a minor violation in accordance with Example 3.d of IMC 0612, Appendix E, because it was a minor discrepancy in time that had no actual significance.
40A3 Event Foilow-up (71153)
(Closed) LER 05000323/2009003-00, "Containment Sump Recirculation Valve Position interlock Failure Due to Inadeguate Testing."
On October 23, 2009, the licensee identified that valves necessary to initiate cold leg recirculation could not have been opened from the control room for a period of approximately 20 months. The details are described in this inspection report. The NRC identified two Green noncited violations of very low significance that were directly related to this condition, as discussed in this report. This licensee event report (LER) is closed.
40A6 Meetings
Exit Meeting Summary
The inspection team briefed Mr. Peters, Station Director, and other members of the licensee's management staff on December 3, 2009, following completion of the onsite portion of the inspection. An exit meeting \,vas performed on January 26, 2010, with Mr. Becker, Site Vice President, present along with other members of the licensee staff and the public. The licensee acknowledged the findings presented. The inspector asked the licensee whether any materials examined during the inspection should be considered proprietary. One piece of proprietary information was identified and was stated by the team to be handled appropriately.
On March 2, 2010, the team obtained permission from Mr. Tom Baldwin, Licensing, to include the drawing and photographs presented in Attachments 5 and 6 to this report.
ATTACHMENT:
SUPPLEMENTAL INFORMATION
- 19 - Enclosure
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- J. Becker, Site Vice President
- M. Ginn, Emergency Planning Manager
- K. Peters, Station Director
- T. Baldwin, Manager Regulatory Services
- M. Williamson, Motor Operated Valve Engineer
- P. Nugent, Engineering Manager
- M. McCoy, Regulatory Services, NRC Interface
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened and Closed
05000323/2009009-01 NCV Failure to Follow Design and Configuration Control
Requirements05000323/2009009-02 NCV Failure to Conduct an Adequate Post-Modification Test
05000323/2009009-03 NCV Failure to Evaluate a Change to the Facility as Described
in the Final Safety Analysis Report Update Associated
with the Addition of Manual Actions in the Safety Analysis
Closed
05000323/2009003-00 LER Containment Sump recirculation Valve Position Interlock
Failure Due to Inadequate Testing
Attachment
1: Documents Reviewed
DIABLO CANYON ACTION REQUESTS
A07072739, Design Change to Improve Design Margin of SI-2-8982A1B, July 9, 2007
A0659468, Evaluate V-7B Test Time, February 8,2006
A0659468, Evaluation V-7B Test Optimization to Reduce Drain Down Window, February 8,
2006
A0674066, SI-2-892A1B MOV Gear to Improve Design Margin, July 26,2006
A0551483, Evaluate Deferral of STP V-7B Scope, March 20, 2002
DIABLO CANYON NOTIFICATIONS
277252, 8982A1B Interlocks Out of Adjustment,
261612, NSAL 09-6, ECCS Flow Interruption/REDU, August 13, 2009
500336425, Design Change Motor Pinion Gear, Unit 1
201640, Evaluate PEP V-7B Frequency, February 14, 2009
27752, 8982A1B Interlocks Out of Adjustment, October 23,2009
CALCULATIONS
NUMBER TITLE REVISION
STA-220 RHR System Pressurization Due to INPO OE 20893 o
SBLOCA Scenario, November 21,2005.
N-100 Maximum Flow from ECCS Pumps and Minimum 4
Flow to Containment Spray header
Motor Operated Valve Sizing and Switch Setting Calculations,
June 25, 2009
Rising Stem MOV Limit Switch Setpoints
Attachment
DRAWINGS
NUMBER TITLE REVISION
M000073 Maintenance Modification Drawing Torque and Limit Switch, 4
Spring Pack, Changes for MOV's
57725, Sheet 1 Plan and Elevation, 91 '0" and 100'0" Auxiliary, Containment, 29
and Fuel Handling Buildings
57723, Sheet 1 Plan at Elevation 73'0", Auxiliary and Containment Buildings 16
MODIFICATION PACKAGES
NUMBER REVISION
MP 05-1003 Trip Throttle Valve Gear Ratio Change 100: 1 o
PROCEDURES
NUMBER TiTLE REVISION
EOP E-1 Loss of Reactor or Secondary Coolant 28
CF3.ID9 Design Change Development 30
STP V-3L 17 Exercising Containment Recirc Suction valves, SI-8982A and 10
SI-8982B, Unit 1, completed on February 19, 2009
EOP E-1.3 Transfer to Cold Leg Recirculation, Unit 2 17
EOP ECA 1.1 Loss of Emergency Coolant Recirculation 16
AD8.DC58 Outage Scope Control 4
AD8.DC60 Outage Schedule Preparation o
MP E-53.1 OV1 MOV Diagnostic Testing with the Viper System 8
AD13.ID4 Post Maintenance Testing 17
AD13.ID2 Post Modification Testing 4
PEP V-7B Test of ECCS Valve Interlock
Attachment
OPERATING EXPERIENCE INFORMATION
NRC IN 2006-29, Potential Common Cause Failure of Motor-Operated Valves as a Result of
Stem Nut Wear.
Work Oder C0214725, 8982A Change Gears to Improve Design Margin, February 17, 2008
Work Oder C0214751, 8982B Change Gears to Improve Design Margin, February 17, 2008
MISCELLANEOUS
Root Cause Analysis Report, SI-2-8982NB Interlocks Out of Adjustment, November 19, 2009
E-mail, Michael Davida, Update on MOVs PEP-V-7B, February 19, 2008
Prompt Operability Assessment, POA AR A06431 07 AE03, Compensatory Measure Requiring
EOP E-1 and MA 1.101, December 2, 2009
Licensing Basis Impact Evaluation Screen AT-MM A0702739, Improved Design Margin of SI-2-
8982NB, Rev. 0
FSAR Table 6.3-5, Safety Injection to Recirculation Mode Sequence and Timing of Manual
Changeover, Rev. 18
Response to NRC Generic Letter 96-01, Testing of Safety Related Logic, April 18, 1996, PG&E
Letter DCL-96-090
Westinghouse, Small Break LOCA Assessment to Support Past Operation with Failed ECCS
Interlock, November 25,2009
2: Special Inspection Charter
November 19, 2009
MEMORANDUM TO: Michael
- F. Runyan, Senior Reactor Analyst
Division of Reactor Safety
Michael Peck, Senior Resident Inspector
Project Branch B
Division of Reactor Projects
Paul Elkmann, Senior Emergency Preparedness Inspector
Plant Support Branch 1
Division of Reactor Safety
Megan
- J. Williams, Reactor Inspector
Plant Support Branch 2
Division of Reactor Safety
FROM: Roy
- J. Caniano, Director IRAI
Division of Reactor Safety
SUBJECT: SPECIAL INSPECTION CHARTER TO EVALUATE THE FAILURE
OF EMERGENCY CORE COOLING SYSTEM INTERLOCKS AT
DIABLO CANYON NUCLEAR POWER PLANT UNIT 2
A Special Inspection Team is being chartered in response to the Diablo Canyon Unit 2 failure of
emergency core cooling system interlocks on October 23, 2009. Michael F. Runyan is
designated as the Special Inspection Team Leader. Michael Peck, Senior Resident Inspector,
Mr. Paul Elkmann, Senior Emergency Preparedness Inspector, and Megan Williams, Reactor
Inspector, are designated as team members.
A. Basis
On October 23, 2009, during the performance of Plant Engineering
Procedure PEP V-7B, "Test of ECCS Valve Interlocks," containment recirculation
suction to residual heat removal (RHR) Pump 2-1 Valve SI-2-8982A, and containment
recirculation suction to RHR Pump 2-2 Valve SI-2-8982B, interlock position switches
failed to work correctly when attempting to open RHR pump isolation to Spray Header 1
and 3 valve, CS-2-9003A, and RHR pump isolation to Spray Header 2 and 4 valve,
CS-2-9003B. The interlock also affects the RHR pump discharge to charging pump
suction valve, SI-2-8804A, and RHR pump discharge to safety injection pump suction
valve, SI-2-8804B.
The licensee's initial investigation revealed that Pacific Gas and Electric Company
performed maintenance on valves SI-2-8982A/B during the Spring 2008 Unit 2 refueling
outage. The maintenance activity adjusted the travel stop and indication limit switch.
This adjustment resulted in a shorter stroke that stopped the valve before the interlock
limit switch was made up. Pacific Gas and Electric Company did not perform any post
maintenance test to verify the interlock functionality.
Pacific Gas and Electric Company's extent of condition review determined that the same
condition did not exist on Unit 1 through a review of maintenance records.
A regional Senior Reactor Analyst (SRA) preliminarily estimated the Incremental
Conditional Core Damage Probability for this issue to be 7.4E-6, which falls in the region
which recommends a special inspection. A special inspection will be performed since
there are questions about the ability of the facility to be operated within its design basis
under these conditions.
B. Scope
The team is expected to address the following:
1. Develop a chronology (time-line) related to the failure of emergency core cooling
system interlocks and a description of the motor-operated valve interlock
operation.
2. Evaluate the licensee's root cause analysis and corrective actions for the known
failures, and assess the adequacy of the licensee's corrective actions.
3. Evaluate the extent of condition and extent of cause reviews performed by the
licensee in response to this event.
4. Evaluate the adequacy of the licensee's post maintenance and testing program
for activities involving motor-operated valve interlocks.
5. Evaluate the licensee's response to industry operating experience to determine
whether the iicensee took timely and effective measures to address the issues.
Review the licensee's commitments to Generic Letter 96-01, "Testing of Safety
Related Logic Circuits."
6. Collect data as necessary to complete a determination of risk significance for the
failed interlocks. This should include an assessment of the circumstances of
each failure (I.e., plant conditions, testing mode), what equipment would not
function, time since previous successful performance, and the potential for
operator recovery actions.
7. Determine if the licensee met NRC reporting requirements of 10 CFR 50.72 and
eFR 50.73.
Attachment
C. Guidance
Inspection Procedure 93812, "Special Inspection," provides additional guidance to be
used by the Special Inspection Team. Your duties will be as described in Inspection
Procedure 93812. The inspection should emphasize fact-finding in its review of the
circumstances surrounding the events. Safety concerns identified that are not directly
related to the event should be reported to the Region IV office for appropriate action.
The team will report to the site, conduct an entrance, and begin inspection no later than
November 30, 2009. While onsite, you will provide daily status briefings to Region IV
management, who will coordinate with the Office of Nuclear Reactor Regulation to
ensure that all other parties are kept informed. Depending on the outcome of the
inspection, inspection results will be documented in NRC Special Inspection
Report 05000323/2009009. This report will be issued within 45 days of the completion
of the inspection.
This Charter may be modified should the team develop significant neVJ information that
warrants review. Should you have any questions concerning this Charter, please
contact Ray Kellar at (817) 860-8164.
Attachment
3: Timeline of Events
Timeline Associated with Diablo Canyon, Unit 2, Disabled ECCS Interlocks
Date Activity
January 10, 1996 NRC issues Generic letter 96-01 (Licensees responses due March 10,
1996)
February 14, 1996 NRC revises due date for responses to Gl 96-01 to April 18, 1996
April 18, 1996 PG&E forwards response to Gl 96-01 (DCl 96-090) to NRC; commits
to completing evaluation and final response by March 20, 1998
October 10, 1997 PG&E forwards DCl 97-172 for lER 1-97-016-00 regarding
surveillance testing on solid state protection system
December 15, PG&E forwards DCl 97-206 for lER 1-97-016-01 regarding additional
1997 surveillance testing on solid state protection system
January 22, 1998 PG&E forwards DCl 98-009 stating actions required by Gl 96-01 have
been completed.
July 2005 PG&E receives OE20893 (Catavvba SBLOCA) and initiates AR
A0643107
November 22, PG&E completes preliminary calculation identifying stroke time =24.3
2005 <<25 seconds)
July 26, 2006 PG&E Safety Injection System Engineer requests design change for
neVJ gear ratio in SI-2-8982,L\J8 (/\R /!l*,0674066) to compensate for
pressure differential
August 23, 2007 Design change for new gear ratio is approved and funded for 2R14; AR
A0702739 is initiated
November 15, Work Orders C0214725 and C0214751 issued to replace gears
2007
Attachment
February 14, 2008 During execution of work orders C0214725 and C0214751, valve stroke
time> 25 seconds on SI-2-8982A1B
February 16, 2008 AR A0674066 documents geometry assessment to change open limit
setpoint range for MOVs to 13.8" from 13.95" from full closed.
February 21, 2008 AR A0702739 closes all 10 items necessary to return Valves SI-2-8982
AlB to service
April 8, 2008 Unit 2 enters Mode 4 with disabled MOV interlocks
February 12-16, PG&E completes gear switch on Valves SI 8982s in Unit 1, resetting
2009 rotors and interlock actuations
March 1, 2009 PG&E completes testing on SI 8982s in Unit 1
October 22, 2009 PG&E conducts refueling outage test on ECCS in Unit 2; Valves SI-2-
8804A/B and CS-9003A1B failed to open on demand; rotor #3 and
interlocks were reset and tested.
October 24, 2009 PG&E forms IPRT to address immediate issue and evaluate safety
significance.
November 2, 2009 Integrated Problem Resolution Team (IPRT) transitioned to Root Cause
Team (RCT), to determine what happened, recommend corrective
action, and prevent recurrence.
November 30, NRC Special Inspection Team convened on site
2009
December 2, 2009 PG&E issues prompt operability assessment (POA) to investigate
resulting position of modified valve stroke on net positive suction head
available (NPSHA) for recirculation.
11l. .-9 Attachment
4: SDP Phase 3 Analysis
Phase 3 Analysis
Diablo Canyon
Disabled ECCS Interlocks
Performance Deficiency
Improper coordination of limit switches on Unit 2 residual heat removal (RHR) sump suction
Valves SI-2-8982A/B resulted in the disablement of interlocks which permit operators to open
the valves that provide RHR flow to the suction of the safety injection and charging pumps (SI-2-
8804NB). This resulted in a loss of the capability for operators to initiate cold leg recirculation
from the control room for loss of coolant accidents that do not depressurize the reactor coolant
system below the shutoff head of the RHR pumps. The valves were accessible and could have
been re-positioned locally.
Assumptions
1. The condition existed for approximately 20 months during the entire operating cycle
between Refueling Outages 2R14 and 2R15. Therefore, in accordance with the
guidelines in IMC 0609, an exposure period of 1 year was assumed.
2. All emergency core cooling system (ECCS) pumps, including the RHR pumps, safety
injection pumps, and high-head centrifugal charging pumps start automatically in
response to a loss of coolant accident. When the refueling water storage tank reaches
percent level, the RHR pumps stop automatically and operators use procedures to
align for cold leg recirculation. This process must be completed before the refueling
water storage tank (RWST) level reaches 4 percent, at which time the other ECCS
pumps are stopped. Failure to complete the actions by this time would result in a loss of
forced flow in the core and a high probability of uncovering the fuel.
The operators would not discover the disabled interlocks until some time after the 33
percent level is reached, within the framework of completing the procedural steps to
initiate cold leg recirculation. The time available between the 33 and 4 percent level
depends primarily on the break size and whether containment spray is running.
Based on thermal-hydraulic analyses performed by Westinghouse and Diablo Canyon,
the containment spray system is not expected to actuate for small and medium break
<<5.0-inch) loss of coolant accidents. For breaks larger than approximately 5.5 inches,
containment spray would likely actuate but the reactor coolant system would
depressurize and remain depressurized below the shutoff head of the RHR pumps. For
breaks in the size range 5.0 - 5.5 inches, it is expected that forced flow could be
interrupted to the core, but in this scenario, based on a bounding thermal-hydraulic
analysis reviewed by the NRC, the peak cladding temperature would be limited to less
than 1500 degrees F even if Valves SI-2-8804NB were not opened in time, and
l\ttachment
therefore fuel damage would not occur. Because the RHR pumps would be capable of
supplying sufficient recirculation flow, breaks larger than 5.5 inches (principally
incorporating the range of large-break loss of coolant accidents) would not result in a
change to the core damage frequency. In other words, for breaks of this size, the failure
of the 8804 valves to open would not represent a change in the probability of core
damage.
For the largest small-break LOCA, the estimated time available for operators to manually
open the 8804 valves before the RWST reaches 4 percent is 77 minutes. Although for
smaller breaks, the time available could be significantly longer, it was assumed that this
time applied to all small-break LOCAs. For medium break LOCAs, this time would be 72
minutes. These times assume that containment spray is not running. As stated above,
it is assumed that any break of sufficient size to initiate containment spray will enable
RHR by itself to provide sufficient core cooling to avoid fuel damage.
3. The nominal time that would be needed for operators to open the 8804 valves is 20
minutes. It has been estimated by the licensee that for breaks of smaller size
(essentially the small break category) the onsite emergency facilities would be activated
by the time the valve failures would be discovered (because of increased time from 100
to 33 percent RWST level) and that this would have the effect of adding administrative
elements to the recovery and thereby potentially extend the total time to 40 minutes.
The analyst determined that this estimate was overly conservative and considered that
the 20-minute estimate was appropriate for all scenarios.
4. The radiological conditions in the area of the 8804 valves or in any of the access
pathways to these valves would not affect the recovery actions. The maximum dose for
a 30-min stay time was conservatively calculated to be 1 rem. The licensee is
authorized an exposure up to 10 rem for this type of recovery action. The time needed
to open the valve manually is approximately 5 minutes, such that the actual exposure
would likely be less than 1 rem.
5. Both of the 8804 valves are located in unobstructed areas with the manual handwheel
easily accessible at approximately shoulder height. The ergonomics associated with
local operation are favorable.
6. The remote positioning capability of valves supplying RHR flow to the containment spray
valves (9904A1B) vvas also disabled by the improper coordination of rotor limit svvitches
on Valves SI-2-8982A1B. This had only a negligible impact on the risk significance. The
containment pressure would not under any conditions be greater than the threshold of
psig at the time that containment spray would hypothetically be lost when the RWST
reaches 4 percent. Also, the Diablo Canyon SPAR model does not credit containment
spray for accident mitigation. Additionally, the loss of containment spray recirculation
would not affect the large early release probability.
7. Using information in the assumptions above, the following estimates were made of the
probability that operators will fail to open the 8804 valves before the RWST level falls to
percent and all injection is lost. In the SPAR model, the major sequences contributing
to the change in CDF are small-break LOCAs, medium-break LOCAs, and loss of offsite
,11,-11 ,L\.ttachment
power. The loss of offsite power events involve small-break LOCAs- the failure of RCP
seals. Although slight differences in time available exist and the predicted time to
perform the actions is different, the non-recovery values for small and medium break
LOCAs were estimated to be the same values using the performance shaping factors in
the SPAR-H methodology.
The nominal time to open the 8804 valves is 20 minutes. The time available is 77
minutes for a small break and 72 minutes for a medium break. Very little time is applied
to diagnosis because the failure of the valves to open would be immediately observed.
DIAGNOSIS (0.01 NOMINAL)
Performance Level Factor
Shaping Factor
Available Time Nominal 1
Stress High 2
Complexity Obvious 0.1
Experience/Training Nominal 1
Procedures Nominal 1
Ergonomics Nominal 1
Fitness for Duty Nominal 1
Work Processes Nominal 1
Diagnostic Result =(0.01) (0.1) (2) =2.0E-3
Available Time: It is estimated that the nominal time to diagnose the condition would be
several minutes- arriving at the procedural step to open the 8804 valves.
Stress: The situation would be high stress for the operators, but not extreme, because
immediate threats to health and life would be absent.
Complexity: This would represent an obvious diagnosis because of the immediate on-panel
indication of a failed valve stroke.
Experience/Training: Operators are we!! versed to notice anomalies in their indications.
Attachment
Procedures: Procedures were avaiiabie and complete.
Ergonomics: There are no ergonomic impediments
Fitness for Duty and Work Processes: These factors were considered nominal.
ACTION (0.001 NOMINAL)
Performance Level Factor
Shaping Factor
Available Time Nominal 1
Stress High 2
Complexity Nominal 1
I _ ...
Experience/Training LUW 3
Procedures Nominal 1
Ergonomics Nominal 1
Fitness for Duty Nominal 1
Work Processes Nominal 1
Action result =(0.001) (2) (3) =6.0E-3
Available Time: It is estimated that the nominal time to perform the actions would be 20
minutes, with approximately 70 minutes available.
Stress: The scenario would be high stress for the operators, but not extreme, because
immediate threats to health and life would be absent.
Complexity: The steps needed to perform the recovery are not complex.
ExperiencelTraining: Operators do not have experience in performing this recovery. It has
never been performed.
Procedures: Procedures directing manual opening valves SI-2-8804A!B are available and
complete.
Ergonomics: There are no ergonomic impediments.
Fitness for Duty and Work Processes: These factors were considered nominal.
Attachment
Total HRA result =2.0E-3 + 6.0E-3 =8.0E-3
Internal Events Analysis
The Diablo Canyon SPAR model, Revision 3.50, dated October 4, 2009, was used to estimate
the risk significance of the finding. Because the exposure period was a full year, average test
and maintenance values were used. The truncation level was set at 1.0E-13.
As discussed in Assumption 6, the Diablo Canyon SPAR does not model the containment spray
system. For the reasons discussed above, this exclusion is appropriate for the scenarios being
evaluated. Therefore, only the 8804 valves contributed to the risk increase.
The basic event HPR-MOV-CF-8804AB, CCF of RHR Supply to CVC/HPI Valves CV8804A and
S18804B, was set to TRUE, which models the inability of these valves to open, thereby
preventing the initiation of high pressure recirculation. The SPAR does not model locally
opening these valves. Therefore, the SPAR result of 2.443E-5 is the no-recovery delta-CD
- F.
The dominant sequences (collectively contributing 99 percent of the total risk), are presented
below. The SPAR was then re-run by assigning the non-recovery value of 8.0E-3 to the HPR-
MOV-CF-8804AB basic event (the base case value is much smaller, 2.28E-5, therefore
assigning the non-recovery value is mathematically appropriate), with results in the final column:
FINAL
DELTA- PERCENTAGE RECOVERY
SEQUENCE DESCRIPTION DELTA-
CDF OF TOTAL FACTOR
Loss of offsite power, loss
of AFW, feed/bleed
LOOP 12 initiated, loss of high 1.657E-5 67.8 8.0E-3 1.33E-7
pressure recirculation
(HPR)
Medium break LOCA,
failure to cool down/
MLOCA4 4.000E-6 16.4 8.0E-3 3.20E-8
depressurize, loss of
HPR
Loss of offsite power, loss
of AFW, feed/bleed
LOOP 14 initiated, failure to recover 1.767E-6 7.2 8.0E-3 1.41E-8
offsite power in 6 hrs.,
loss of HPR
Small break LOCA,
failure to cool down/
SLOCA 7 1.200E-6 4.9 8.0E-3 9.60E-9
depressurize, failure of
HPR, failure to perform
ECA 1.1, loss of
recirculation procedure
Loss of offsite power,
PORV sticks open, offsite
LOOP 9 2.507E-7 1.0 8.0E-3 2.00E-9
power not recovered in 2
hrs., loss of HPR
Partial loss of main
PLMFW 21 feedwater, loss of aux 2.233E-7 0.9 8.0E-3 1.79E-9
and main feedwater,
Reactor trip, loss of AFW,
loss of main feedwater,
feed/ bleed initiated, 5.05E-
TRANS 21 6.313E-8 0.3 8.0E-3
failure to re-establish 10
secondary cooling, loss of
HPR
Loss of main feedwater,
loss of AFW, feed/ bleed
4.96E-
LOMFW 21 initiated, failure to re- 6.199E-8 0.3 8.0E-3
establish secondary
cooling, loss of HPR
Loss of condenser heat
sink, loss of AFW, feed/
3.97E-
LOCHS 12 bleed initiated, failure to 4.959E-8 0.2 8.0E-3
re-establish secondary
cooling, loss of HPR
DELTA-CDF INCLUDING RECOVERY 1.94E-7
The estimated significance of the finding for internal initiators is therefore 1.94E-7/yr.
External Events:
Seismic
The analyst used seismic data obtained from the licensee's seismic PRA analysis. This
included spectral acceleration bins and related equipment fragilities.
The predominant risk contribution to this issue from seismic events is a loss of offsite power.
SPAR LOOP sequences 12 and 14 provide most of the risk increase, and involve loss of AFW,
an initiation of feed and bleed and subsequent failure of high pressure recirculation. For this
analysis, it is assumed that risk-significant mitigating equipment, all of which has a much higher
fragility than offsite power, is not lost in the seismic event. The CCDP for a seismic LOOP
A-i5 Attachment
assuming no recovery of offsite power and no loss of mitigating equipment, is 2.724E-4. With
both 8804 valves failed closed at the above-calculated non-recovery probability of 8.0E-3, the
seismic LOOP CCDP rises to 2.766E-4, for a difference of 4.2E-6. The difference is small
because most LOOPs that don't include offsite power recovery go to core damage due to
station blackout conditions, which makes the interlock problem inconsequential, and also
because the non-recovery probability of 8.0E-3 is close to the total base case value for a failure
of high pressure recirculation. This delta-CCDP result of 4.2E-6 is applied in the table below.
The following table illustrates the results:
SEISMIC FREQUENCY PROBABILITY FREQUENCY DELTA-CDF OF
RANGE (G) OF OF LOOP OF LOOP RANGE
SPECTRAL RANGE(PER
YEAR)
0.2-1.25 1.72E-2 12% 2.07E-3 8.68E-9
1.25-1.75 8.69E-4 69% 6.01E-4 2.53E-9
1.75-2.00 1.56E-4 86% 1.35E-4 5.65E-10
2.00-2.50 1.24E-4 96% 1.19E-4 5.01 E-1 0
2.50-3.00 2.94E-5 99% 2.92E-5 1.23E-10
3.00-4.00 7.64E-6 100% 7.64E-6 3.21E-11
4.00-5.00 2.37E-7 100% 2.37E-7 9.95E-13
Total Seismic Delta-CDF 1. 24E-B/yr
Fire
Based on a review of the important sequences from the internal events assessment, the analyst
concluded that the only potentially significant contribution of fires to the significance of the
finding is fires that cause a loss of offsite power. The only other significant sequences involve
loss of coolant events, which are very unlikely to be coupled with fires. The data used to
establish plant-centered and switchyard-centered loss of offsite power events are included in
the baseline LOOP frequency used in the internal analysis, and therefore, the increase in risk
would be associated with difficulties imposed by the fires in mitigating the event. Based on the
low percentage of fires that could result in offsite power events coupled with the expectation of
recovery possibilities similar to the internal analysis, the analyst determined qualitatively that fire
events would not appreciable add to the risk associated with internal initiators.
Attachment
These events are not expected to involve loss of coolant situations. The internal LOOP
frequency already accounts for wind-induced LOOPs.
Flooding
Neither internal nor external flooding is expected to contribute more than negligibly to the
significance of the finding.
Large Early Release
Based on information provided in IMe 0609, Appendix H, for a PWR large, dry containment,
core damage sequences resulting from loss of coolant accidents and loss of offsite power
events do not contribute more than negligibly to the probability of a large early release of
radiation following a core damage event. Therefore, the significance of this finding is
determined by the core damage frequency.
/aJtachment
5: System
- -' -',- - '- 'l'-
fI.eli..Elng .
Il'.IalEr
81ca!il1l:
Tarf.
C-+::
- m~'1
'-
t
1,),
Attachment
6: Limit Switches
,Ll,ttachment