IR 05000247/2013011
| ML14118A124 | |
| Person / Time | |
|---|---|
| Site: | Indian Point  |
| Issue date: | 04/29/2014 |
| From: | Bill Dean Region 1 Administrator |
| To: | Ventosa J Entergy Nuclear Operations |
| Marjorie McLaughlin | |
| References | |
| 1-2012-036, EA-13-076 IR-13-011 | |
| Download: ML14118A124 (48) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION
REGION I
2100 RENAISSANCE BLVD., SUITE 100 KING OF PRUSSIA, PA 19406-2713 April 29, 2014 EA-13-076 John Ventosa Vice President, Operations Entergy Nuclear Operations, Inc.
Indian Point Energy Center 450 Broadway, GSB P.O. Box 429 Buchanan, NY 10511-0249 SUBJECT: INDIAN POINT ENERGY CENTER UNITS 2 & 3, NOTICE OF VIOLATION, NRC INSPECTION REPORT NOS. 05000247/2013011 & 05000286/2013011, AND NRC OFFICE OF INVESTIGATIONS REPORT NO. 1-2012-036
Dear Mr. Ventosa:
This letter provides you the NRC enforcement decision for apparent violations (AVs) identified during an investigation by the NRC Office of Investigations (01) conducted between March 30, 2012, and March 26, 2013, at the Entergy Nuclear Operations (ENO) Indian Point Energy Center (IP). The investigation was conducted to determine if the IP Chemistry Manager deliberately entered false data into a Chemistry database pertaining to an Emergency Diesel Generator (EDG) fuel oil storage tank (FOST) and the reserve fuel oil storage tank (RFOST).
Per the IP Technical Specifications (TS), the fuel oil is sampled every 92 days and analyzed to determine if it is within limits for specified parameters, including total particulate concentration.
If the particulate concentration is above the stated limit, it must be restored to below the limit within 7 days for an FOST, or 30 days for the RFOST; otherwise, ENO must immediately declare the associated EDGs inoperable. For the RFOST, all the EDGs would be declared inoperable, which would require ENO to shutdown both operating units.
Based on the 01 investigation, the NRC identified two AVs involving: (1) operating IP Units 2 and 3 in violation of site TS requirements when the 22 FOST and the RFOST exceeded the TS limit for particulate concentration; and, (2) failing to initiate a condition report (CR) or otherwise report the condition such that a Licensee Event Report (LER) could be timely submitted to the NRC. The AVs were described in an enclosure to the NRC letter sent to you on December 18, 2013 (ML13354B806 1). On January 22, 2013, a pre-decisional enforcement conference (PEC)
Designation in parentheses refers to an Agency-wide Documents Access and Management System (ADAMS) accession number. Unless otherwise noted, documents referenced in this letter are
. publicly-available using the accession number in ADAMS. was conducted in the NRC Region I office with you and members of your staff to discuss the AVs. During the PEC, ENO staff: (1) acknowledged that the AVs occurred; (2) described the circumstances surrounding the AVs and ENO's identification of the issue; (3) reviewed the corrective actions taken and planned for both IP and the ENO fleet; and (4) provided their view of the significance of the AVs. A summary of the PEC and a copy of the ENO presentation are provided as Enclosures 2 and 3 to this letter.
Based on the information developed during the investigation, and that ENO presented during the PEC, the NRC has determined that three violations of NRC requirements occurred, as described below. Two of the violations are cited in the Notice of Violation (Notice) which is provided as Enclosure 1 to this letter. The two cited violations involve: (1) ENO's operation of IP Units 2 and 3 in violation of TS 3.0.3 after the Chemistry Manager identified that the fuel oil in the 22 FOST and the RFOST exceeded the TS limit for particulate concentration; and (2) the failure by the Chemistry Manager to initiate a CR or otherwise report the issue such that an LER could be submitted to inform the NRC of the condition prohibited by TS, as required by 10 CFR 50.73(a)(2)(B). The third violation involves additional examples of operation of IP Units 2 and 3 in violation of the TS due to the RFOST and/or 22 FOST particulate limits being exceeded. As discussed below, this third violation is not cited because it is not willful and is of very low safety significance.
Specifically, during a self-assessment conducted in January/February 2012 to prepare for an upcoming NRC Component Design Bases Inspection, ENO staff at IP reviewed the EDG fuel oil delivery systems and storage tanks. The IP self-assessment team identified that: ( 1) results of RFOST samples taken on June 17, 2011, and December 1, 2011, were not entered into the Chemistry Department database until July 14, 2011, and January 23, 2012, respectively; and (2)
although both samples exceeded the TS particulate limits, no CRs had been written to document the issues and notify site operations and, evidently, no re-sampling performed to confirm that the oil had been restored to below the limit within the 30-day allowed outage time.
On February 2, 2012, the IP self-assessment team inquired of Chemistry department staff, including the Chemistry Manager, about this issue. Subsequently, on February 5, 2012, the Chemistry Manager entered information in the Chemistry database indicating that re-samples for the June 17, 2011, and December 1, 2011, RFOST samples had, in fact, been performed on June 29, 2011, and December 9, 2011, respectively (i.e., within the 30 day period allowed by TS), and that the re-samples were below the TS particulate limit. However, during the 01 investigation, the Chemistry Manager admitted to 01 that the re-samples had actually not been obtained. The manager informed 01 that he had entered false values in the database instead of documenting the issue in a CR or otherwise informing the IP Operations Department that the site was operating in violation of its TS.
The manager testified to 01 that he entered the false values because he believed the original sample results were incorrect as a result of poor IP Chemistry Department sampling practices.
Namely, the samples had been obtained from the bottom of the RFOST and shipped in a tin-coated can; both practices that were specifically not recommended by newer industry guidance because sediment could collect at the bottom of the tank and the tin coating could contaminate the samples. The manager said that he did not report the out-of-specification results because he wanted more time to prove his theory and incorporate new test methods, and he did not want the plant to shut down when he did not believe it really needed to. The manager also admitted that he similarly entered false re-sample data for the IP 22 EDG FOST after identifying that the TS particulate limit had been exceeded for a November 18, 2011, sample taken from that tank. Namely, on February 6, 2012, the manager entered data to indicate that a resample had been performed on December 7, 2011, and that the resample was below the TS particulate limit. Based on the evidence gathered during the 01 investigation, the NRC concluded that the Chemistry Manager deliberately: (1) caused ENO to operate IP Units 2 and 3 in violation of their TS requirements for a longer period than if he had written a CR (or otherwise notified the Operations Department of the issue); and (2) prevented ENO from informing the NRC of this TS-prohibited condition.
Because licensees are responsible for the actions of their employees, and because the manager's actions were willful and impeded the regulatory process, these violations were evaluated under the NRC's traditional enforcement process, as set forth in Section 2.2.4(b) and (c) of the NRC Enforcement Policy. After careful consideration, the NRC concluded that these violations are best categorized at Severity Level Ill (SL Ill). In reaching this determination, the NRC considered that the underlying technical findings would have been evaluated as having very low safety significance (i.e. green) under the Reactor Oversight Process (ROP) because the higher fuel oil particulate concentration would not have impacted the ability of the EDGs to fulfill their safety function. Specifically, the TS limit is conservative and the IP EDGs utilize two sets of filters on the fuel oil booster pump, which alarm on high differential pressure, and IP staff could have changed the filters as needed without interrupting EDG operation. The violations associated with these findings are evaluated using traditional enforcement, and would have been best categorized at SL IV, absent willfulness, in accordance with the Enforcement Policy.
However, because they involved willfulness, the NRC increased the significance of these violations, in accordance with Section 2.2.1.d of the Enforcement Policy. Willful violations are of particular concern because the NRC's regulatory program is based, in part, on licensees and their employees acting truthfully and with integrity. In consideration that the violations were directly related to each other and were both caused by the Chemistry Manager's deliberate actions, the NRC has categorized the violations collectively as a single SL Ill problem.
In accordance with Section 8 of the Enforcement Policy, a base civil penalty in the amount of
$70,000 is considered for a SL Ill problem involving power reactors. Because the violations were willful, the NRC considered whether credit was warranted for Identification and Corrective Action in accordance with the civil penalty assessment process in Section 2.3.4 of the Enforcement Policy. The NRC has concluded that credit is warranted for ENO's identification of the violations. Specifically, after receiving an employee concern questioning the validity of one of the supposed RFOST re-samples, ENO hired a consultant to conduct an internal investigation, and independently concluded that the re-samples had not been obtained and that the Chemistry Manager had likely provided false information indicating that they had. The NRC notes that the manager resigned without cooperating with ENO's investigation and did not respond to ENO's requests to be interviewed about the issue.
The NRC has also concluded that credit is warranted for ENO's corrective actions in response to these violations. Namely, ENO: (1) performed an extent of condition review and verified that the fuel oil in all of the FOSTs (including the 22 FOST) was below the TS particulate limit; (2)
after filtration of the RFOST was unsuccessful in reducing the particulate level to below the TS limit, cleaned the tank and replaced the fuel oil with new oil; (3) submitted an LER to the NRC regarding the issue, after it became known to site management; (4) modified the fuel oil sampling process to meet the industry standards; (5) trained the Chemistry staff on expectations and TS requirements; (6) added a second check of all Chemistry data entry and implemented a data trending analysis requirement by Chemistry supervisors; (7) added the chemistry sample schedule to the IP work management schedule to increase visibility of such TS-related samples; (8) conducted an independent Safety Conscious Work Environment assessment; (9) amended the Chemistry Manager's Personnel Access Data System entry with information pertaining to his actions; (1 0) briefed IP staff on the issue and reinforced expectations for communicating nonconforming conditions; and ( 11) implemented a more intrusive management oversight program that includes monthly reinforcement of standards and expectations. Therefore, to encourage prompt identification and comprehensive correction of violations, I have been authorized, after consultation with the Director, Office of Enforcement, not to propose a civil penalty in this case. However, significant violations in the future could result in a civil penalty.
The NRC has concluded that information regarding: (1) the reasons for the violations; (2) the actions planned or already taken to correct the violations and prevent recurrence; and, (3) the date when full compliance was achieved, is already adequately addressed on the docket in the NRC letter dated December 18, 2013, and in this letter and its enclosures. Therefore, you are not required to respond to this letter unless the description therein does not accurately reflect your corrective actions or your position. In that case, or if you choose to provide additional information, you should follow the instructions specified in the enclosed Notice.
The NRC notes that, through its investigation, ENO identified additional examples of operation of IP Units 2 and 3 in violation of the TS due to the RFOST and/or 22 FOST particulate limits being exceeded. Specifically, the IP Chemistry staff that entered the initial sample results in the Chemistry Database for the June 17, 2011, and December 1, 2011, RFOST samples did not identify that the samples exceeded the TS particulate limit. The results of the June 17, 2011, RFOST sample were provided to IP by the offsite laboratory on July 13, 2011, and loaded into the database on July 14, 2013. The results of the December 1, 2011, RFOST sample were provided to IP on December 30, 2011, and were loaded into the database on January 23, Z012.
Even though entry of the high particulate values created a visual warning on the computer database screen, the responsible staff did not recognize that the TS limit had been exceeded and/or did not believe that the result was a cause for concern because they erroneously believed the related equipment was no longer in service. Additionally, the results of a November 18, 2011, sample from the 22 FOST were provided to IP on December 7, 2011, yet were apparently not reviewed by Chemistry staff and were not entered into the Chemistry Database until February 6, 2012, when the Chemistry Manager entered the data along with false information pertaining to a supposed re-sample. For each of these occurrences, members of the Chemistry Department did not write condition reports or otherwise notify site management or the Operations Department that TS limits were exceeded. Consequently, actions were not taken within the required timeframes to restore the parameters to within limits, requiring the EDGs to have been declared inoperable.
The circumstances surrounding the violations are of concern to the NRC because they indicate a lack of consideration for (and/or knowledge of) TS requirements by ENO Chemistry staff. The NRC also notes that the Chemistry Manager would not have had the opportunity to commit the violations had ENO staff exhibited the proper regard for the site TS. However , because these violation examples were not caused by the Chemistry Manager's willful actions, they were evaluated under the ROP and determined to be of very low safety significance (i.e., green), for the reasons previously documented. Because ENO generated CRs to address these issues, they are being treated as NCVs, consistent with Section 2.3.2 of the Enforcement Policy. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosures will be made available electronically for public inspection in the NRC Public Document Room or from the NRC's document system (ADAMS), accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html. To the extent possible, your response, if you choose to provide one, should not include any personal privacy, proprietary, or safeguards information so that it can be made available to the Public without redaction. If personal privacy or proprietary information is necessary to provide an acceptable response, please provide a bracketed copy of your response that identifies the information that should be protected and a redacted copy of your response that deletes such information. If you request withholding of such information, you must specifically identify the portions of your response that you seek to have withheld, and provide in detail the bases for your claim of withholding (e.g., explain why the disclosure of information will create an unwarranted invasion of personal privacy or provide the information required by 10 CFR 2.390(b) to support a request for withholding confidential commercial or financial information).
The NRC also includes significant enforcement actions on its Web site at (http://www.nrc.gov/reading-rm/doc-collections/enforcement/actions/).
Sincerely, William M. Dean Regional Administrator Docket Nos. 50-247; 50-286 License Nos. DPR-26; DPR-64 Enclosures:
1. Notice of Violation 2. Summary of January 22, 2014, Pre-decisional Enforcement Conference 3. ENO Presentation at the January 22, 2014, Pre-decisional Enforcement Conference cc w/enclosures:
Distribution via ListServ In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosures will be made available electronically for public inspection in the NRC Public Document Room or from the NRC's document system (ADAMS), accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html. To the extent possible, your response, if you choose to provide one, should not include any personal privacy, proprietary, or safeguards information so that it can be made available to the Public without redaction. If personal privacy or proprietary information is necessary to provide an acceptable response, please provide a bracketed copy of your response that identifies the information that should be protected and a redacted copy of your response that deletes such information. If you request withholding of such information, you must specifically identify the portions of your response that you seek to have withheld, and provide in detail the bases for your claim of withholding (e.g., explain why the disclosure of information will create an unwarranted invasion of personal privacy or provide the information required by 10 CFR 2.390(b) to support a request for withholding confidential commercial or financial information).
The NRC also includes significant enforcement actions on its Web site at (http://www.nrc.gov/reading-rm/doc-coll~ctions/enforcementlactions/).
Sincerely,
/RAJ William M. Dean Regional Administrator Docket Nos. 50-247; 50-286 License Nos. DPR-26; DPR-64 Enclosures:
1. Notice of Violation 2. Summary of January 22, 2014, Pre-decisional Enforcement Conference 3. ENO Presentation at the January 22, 2014, Pre-decisional Enforcement Conference cc w/enclosures:
Distribution via ListServ Distribution: see next page ML14118A124 DOCUMENT NAME S :\ En f-allq'\ En forcement Propose dA . FOST Samples LICENSEE NOV-Ill EA-13-076.docx
- ctions Reqion1 \ In d ian Po1nt X Non-Sensitive X Publicly Available X SUNSI Review/ MMM D D Sensitive Non-Publicly Available OFFICE RI/ORA RI/DRP RI/DRP RI/ORA 01 J Teator/ JAT via NAME M Mclaughlin/ MMM* A Burritt! ALB* M Scott! E Benner for* B Klukan/ BMK NLO*
email*
DATE 2/19/14 2/25/14 3/10/14 3/12/14 3/13/14 OFFICE RI/ORA OE NRR OGC RI/RA M Lemoncelli NLO via NAME B Bickett! BAB* R Carpenter via email D Willis via email W Dean email DATE 3/14/14 4/24/14 4/23/14 4/17/14 4/26/14
' See prev1ous concurrence page OFFICIAL RECORD COPY
ENCLOSURE 1 NOTICE OF VIOLATION Entergy Nuclear Operations, Inc. Docket Nos. 50-247 & 50-286 Indian Point Nuclear Generating Units 2 and 3 License Nos. DPR-26 & DPR-64 EA-13-076 During an NRC investigation conducted between March 30, 2012, and March 26, 2013, violations of NRC requirements were identified. In accordance with the NRC Enforcement Policy, the violations are listed below:
A. Indian Point Nuclear Generating Unit 2 (IP2) Technical Specifications (TS)5.5.11 and Indian Point Nuclear Generating Unit 3 (IP3) TS 5.5.12, "Diesel Fuel Oil Testing Program," in part, require verification every 92 days that total particulate concentration of the fuel oil in the on site and reserve storage tanks is less than or equal to 10 mg/1.
IP2/3 TS 3.8, "Electrical Power Systems," Section 3.8.3, "Diesel Fuel Oil and Starting Air," requires that whenever the total particulate concentration of fuel oil in the fuel oil storage tanks (FOSTs) exceeds the limit, it must be restored within limits within 7 days (30 days for the reserve fuel oil storage tank (RFOST)), otherwise, the associated diesel generators must be immediately declared inoperable.
IP 2/3 TS 3.0.3 states that when a TS Limiting Condition of Operation is not met and the associated Actions are not met, action shall be initiated within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> to place the unit, as applicable, in: MODE 3 within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />, MODE 4 within 13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br />, and MODE 5 within 37 hours4.282407e-4 days <br />0.0103 hours <br />6.117725e-5 weeks <br />1.40785e-5 months <br />.
Contrary to the above, on or about February 2, 2012, Entergy Nuclear Operations (ENO)
identified that test results for a November 18, 2011, fuel oil sample from the IP 22 FOST and for a December 1, 2011, fuel oil sample from the IP RFOST indicated total particulate concentration for both tanks was in excess of the Technical Specification limits of 10 mg/1. Although the total particulate concentration for these systems had not been demonstrated to be within limits within 7 days and 30 days, respectively, ENO did not declare the associated diesel generators inoperable and did not place the units in MODE 3 within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />, MODE 4 within 13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br />, and MODE 5 within 37 hours4.282407e-4 days <br />0.0103 hours <br />6.117725e-5 weeks <br />1.40785e-5 months <br />.
B. 10 CFR 50.73(a)(2)(B) requires the holder of an operating license to, within 60 days after discovery, submit a Licensee Event Report to the NRC for any operation or condition which was prohibited by the plant's Technical Specifications.
IP2 TS 5.5.11/IP3 TS 5.5.12, "Diesel Fuel Oil Testing Program," in part, require verification every 92 days that total particulate concentration of the fuel oil in the onsite and reserve storage tanks is less than or equal to 10 mg/1.
IP2/3 TS 3.8, "Electrical Power Systems," Section 3.8.3, "Diesel Fuel Oil and Starting Air," requires that whenever the total particulate concentration of fuel oil in the reserve fuel oil storage tank (RFOST) exceeds the limit, it must be restored within limits within 30 days, otherwise, the associated diesel generators must be immediately declared
Notice of Violation 2 inoperable.
IP 2/3 TS 3.0.3 states that when a TS Limiting Condition of Operation is not met and the associated Actions are not met, action shall be initiated within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> to place the unit, as applicable, in: MODE 3 within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />, MODE 4 within 13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br />, and MODE 5 within 37 hours4.282407e-4 days <br />0.0103 hours <br />6.117725e-5 weeks <br />1.40785e-5 months <br />.
TS 5.4, "Procedures," Section 5.4.1, states, in part, that written procedures shall be established, implemented, and maintained covering the applicable requirements and recommendations of Appendix A of Regulatory Guide 1.33, Revision 2 (except as provided in the quality assurance program described or referenced in the Updated FSAR for Unit 2).
RG 1.33, Rev.2, App A recommends chemical and radiochemical control procedures to prescribe the nature and frequency of sampling and analyses. Implementing Procedure EN-CY-1 01, "Chemistry Activities," includes guidance related to chemistry sampling and analysis. Section 5.5 states that out of specification conditions should be identified and corrective actions initiated as quickly as possible. Implementing Procedure 0-CY-121 0,
"Organization and Responsibilities of the Chemistry Department," includes guidance related to chemistry sampling and analysis. Step 4.1.4 requires a condition report to be initiated to report any condition exceeding any procedural limits.
Contrary to the above, on or about February 2, 2012, ENO staff identified that, on two occasions: 1) fuel oil sample test results had been received indicating total particulate concentrations that exceeded TS limits of 10 mg/1; 2) the total particulate concentration for these systems had not been returned to within limits within the TS-required timeframe; and 3) the associated diesel generators had not been declared inoperable or the units placed in the appropriate operating modes. However, the ENO staff did not initiate condition reports or otherwise report the condition such that a Licensee Event Report could be written. Specifically the ENO staff identified that: 1) on July 13, 2011, Entergy received an RFOST sample result indicating total particulate concentration of 13.4 mg/1, and the parameter was not restored to within limits until September 2, 2011; and, 2) on December 30, 2011, Entergy received an RFOST sample result indicating total particulate concentration of 13.2 mg/1, and, as of February 5, the parameter had not been restored to within limits. The NRC was not informed via an LER that the plant was operating in a condition prohibited by its TS until August 20, 2012, more than 60 days after discovery by the ENO staff.
These violations are categorized collectively as a SL Ill problem (Enforcement Policy Example Section 6.1 ).
The NRC has concluded that information regarding the reason for the violations, the corrective actions taken and planned to correct the violations and prevent recurrence and the date when full compliance was achieved is already adequately addressed on the docket in in the NRC letter dated December 18, 2013, and in the letter forwarding this Notice of Violation (Notice).
However, if the description therein does not accurately reflect your position or your corrective actions, you are required to submit a written statement or explanation pursuant to 10 CFR 2.201 within 30 days of the date of the letter transmitting this Notice of Violation. In that case, or if you
Notice of Violation 3 choose to respond, clearly mark your response as a "Reply to a Notice of Violation," and send it to the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, One White Flint North, 11555 Rockville, MD 20852-2738, with a copy to the Regional Administrator, U.S.,
Nuclear Regulatory Commission, Region I, 2100 Renaissance Boulevard, King of Prussia, PA 19406, and a copy to the Resident Inspector at Indian Point Nuclear Generating Units 2 and 3.
If you choose to respond, your response will be made available electronically for public inspection in the NRC Public Document Room or from the NRC's document system (ADAMS),
accessible from the NRC Web site at http://www ..!2.!:£:.9Qv/reading-rm/adams.html. Therefore, to the extent possible, the response should not include any personal privacy, proprietary, or safeguards information so that it can be made available to the Public without redaction.
In accordance with 10 CFR 19.11, you may be required to post this Notice within two working days of receipt.
Dated this 29th day of April 2014
ENCLOSURE 2 SUMMARY OF JANUARY 22, 2014 PRE-DECISIONAL ENFORCEMENT CONFERENCE Licensee: Entergy Nuclear Operations Facility: Indian Point Nuclear Generating Units 2 and 3 License Nos.: DPR-26 & DPR-64 Docket Nos.: 50-247 & 50-286 EA-13-076 On January 22, 2014, representatives of Entergy Nuclear Operations (ENO) met with NRC personnel in the Region I office located in King of Prussia, Pennsylvania, to discuss the apparent violations identified in NRC Inspection Report Number 05000247/2013011 &
05000286/2013011. The apparent violations involved the deliberate actions of the Indian Point Chemistry Manager to enter false data pertaining to the total particulate concentration of oil in an Emergency Diesel Generator (EDG) fuel oil storage tank (FOST) and the reserve fuel oil storage tank (RFOST). The conference was held at the request of Region I so that ENO could provide information to better help the Region understand how the violations occurred and corrective actions that ENO has taken.
The ENO staff acknowledged that the apparent violations occurred as characterized in the NRC inspection report. The licensee provided its view that the actual impact of the violation on reactor safety was not significant. Specifically, ENO acknowledged that, although high particulate levels may plug or clog EDF fuel oil filters, the IP EDGs have two sets of Duplex filters on the inlet and outlet side of the fuel booster pump. The filters alarm on high differential pressure, and plant operators would be able to change them without interrupting EDG operation. Additionally, ENO stated that the Technical Specification (TS) particulate limit of 10 mg/1 is conservative, and that industry guidelines consider 20-25 mg/1 to be the maximum acceptable usable limit. The highest fuel oil sample at IP was 22 mg/1 on the 22 FOST.
However, Entergy staff considered that value to be inaccurate and likely the result of a typographical error because samples taken before and after that value was recorded were below the TS limit. Aside from that anomalous result, the other high samples (all of which were from the RFOST) averaged around 13 mg/1.
The licensee also provided a summary of the corrective actions taken in response to these violations. Namely, ENO: (1) performed an extent of condition review and verified that the fuel oil in the FOSTs (including the 22 FOST) was below the TS particulate limit; (2) after identifying that the RFOST fuel oil remained above the limit, filtered the RFOST fuel oil and then, after the particulate level remained above the limit, cleaned the tank and replaced the fuel oil with new oil; (3) submitted an LER to the NRC regarding the issue, ( 4) modified the fuel oil sampling process to meet the industry standards; (5) trained the Chemistry staff on expectations and TS requirements; (6) added a second check of all Chemistry data entry and implemented a data trending analysis requirement by Chemistry supervisors; (7) added the chemistry sample schedule to the IP work management schedule to increase visibility of such TS-related samples; (8) conducted an independent Safety Conscious Work Environment assessment; (9) amended the Chemistry Manager's Personnel Access Data System entry with information pertaining to his actions; (1 0) briefed IP staff on the issue and reinforced expectations for communicating
Enclosure 2 2 nonconforming conditions; and ( 11) implemented a more intrusive management oversight program that includes monthly reinforcement of standards and expectations.
List of Attendees:
NRC William Dean, Regional Administrator, Region I Michael Scott, Acting Director, Division of Reactor Projects (DRP), Region I Eric Benner, Acting Deputy Director, DRP, Region I Brice Bickett, Team Leader, Allegations/Enforcement Staff, Region I Arthur Burritt, Chief, Projects Branch 2, DRP, Region I Brett Klukan, Regional Counsel, Region I Diane Gallagher, Senior Special Agent, Office of Investigations Marjorie Mclaughlin, Senior Enforcement Specialist, Region I Thomas Setzer, Senior Project Engineer, DRP, Region I Nick Hilton, Chief, Enforcement Branch, Office of Enforcement (OE)
Robert Carpenter, Senior Enforcement Specialist, OE Carleen Sanders, Senior Enforcement Specialist, Office of Nuclear Reactor Regulation Jeromy Petch, Project Engineer, DRP, Region I Brandon Pinson, Project Engineer, DRP, Region I Louis McKown, Project Engineer, DRP, Region I Entergy John Ventosa, Site Vice President, Indian Point Energy Center (IPEC)
John McCann, Vice President, Regulatory Affairs, Entergy Pat Conroy, Site Vice President Coordinator, IPEC John Kirkpatrick, Director Regulatory and Performance Improvement, IPEC David Mannai, Senior Manager, Regulatory Assurance, Entergy Robert Walpole, Manager, Regulatory Affairs, IPEC Jeanne Cho, Senior Counsel, Entergy Chuck Thebaud, Independent Investigator, Partner, Morgan Lewis
-
- -
N
-
TI "[O]n or about February 2, 2012, Entergy Nuclear Operations ( NO) identified that test results for a ovember 18, 2011, fuel oil sa pie from the IP2 FOST and for a December 1, 2 11, fuel oil sample from the IP RFOST, indicated total particulate concentration for both tanks was in excess of Technical Specification limits of 10 mg/L. Althouah the total oarticulate concentration for these systems had not been demonstrated to be within limits within 7 days a d 3 days, respectively, ENO did not declare the associated diesel generators inoperable and did not place the units in MODE 3 within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />, MODE 4 within 13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br />, and MOD 5 within 37 hours4.282407e-4 days <br />0.0103 hours <br />6.117725e-5 weeks <br />1.40785e-5 months <br />.
This Apparent Violation is accurate.
ough the former Chemistry anager was aware of the non-comoliance with Technical Soecifications on February 2, 2012, others were not aware because on February 5 and 6, 2012, the former Chemistry Manager created data purporting to show timely re-samoles from RFOST and IP2 FOST were within Technical Soecification limits.
, "[On] or about Feb ary 2, 2012, ENO staff identified that, on two occasions: 1) fuel oil sample test results had been received indicating total particulate concentrations that exceeded TS limits of 10 mg/L; 2)
the total particulate concentration of these systems had not been returned to within limits "thin the TS required timeframe; and 3) the associated diesel generators had not been declared inoperable or the nits placed in the appropriate operating mod s. However, ENO staff did not initiate condition reports or otherwise report tne condition such that an LER could be written.
(continued)
Specifically, the NO staff identified that: 1) on July 13, 2011, Entergy received an R OST sample result indicating total particulate concentration of 13.4 mg/L, and the parameter was not restored to within limits until September 2, 2011; and 2) on December 30, 2011, Entergy received an RFOST sample result indicating total particulate concentration of 13.2 mg/L, and, as of February 5, the para eter ha not been restored to in limits. The NRC was not informed via an LER that the plant was operating in a condition prohibited by its TS until August 20, 2012, more an 60 days after discovery by the ENO staff.
This Apoarent Violation is also accurate.
The falsification of data bv th former Chemistry Manager on Februarv 5, 2012, of two timely "re-samples" purported to show results within Technical Specifications.
Entergy submitted an ER within 60 days of receiving the June 21, 2012 results of a out-of-specification laboratory analysis for a sample taken from the RFOST.
To understand the eve t a d to assess the significance of the event and our corrective actions, a review of the facts is in order.
This factual discussio consists of the following sections:
2011 Fuel Oil Sampling and Testing Discovery of the Anomalies Greater Management Focus and Initial Corrective Actions
2011 Fuel Oil Sampling and Testing 6/17/11 Quarterly GT 2/3 RFOST fuel oil drawn.
7/13/11 High out-of-specification (13.4 mg/L) laboratory test results of 6/17/11 GT 2/3 RFOST sample from Herguth Laboratories received bv Chemistry Department.
7/14/11 Chemistry Specialist enters laboratory results into Chemistry database for 6/17/11 GT 2/3 RFOST sample. No CR 9/02/11 Quarterly GT 2/3 RFOST fuel oil drawn.
9/29/11 in specification ( 8.0 mg/L) Herguth laboratory test results of 9/2/11 RFOST sample received by Chemistry Department. (Results entered in Chemistry database on 2/5/12, by former Chemistry Manager.)
11/18/11 22 EDG FOST fuel oil arawn.
12/01/11 Quarterly GT 2/3 RFOST fuel oil drawn.
12/07/11 High out-of-specification (22 mg/L) Herguth laboratory test results received by Chemistry Department for 11/18/11 22 EDG FOST sample. (Results entered in Chemistry database on 2/6/12, by former Chemistry anager. No CR written.)
12/30/11 igh out-of-specification (13.2 mg/L) Herguth laboratory test results of 12/1/11 RFROST sample received by Chemistry Department.
1/23/12 Chemistry Specialist enters laboratory results into Chemistry database for 12/1/11 GT 2/3 RFOST sample, but does not write CR.
I Discovery of Anomalies 2/02/12 A self assessment team questions Chemistry personnel about out-of-specification GT 2/3 RFOST test results from June and December 2011.
2/05/12 Former Chemistry Manager enters two false GT 2/3 RFOST test results in Chemistry database, using the log-in identity of another person. False data purports to show a GT 2/3 RFOST sample on 6/29/11, with a particulate value of 6.90 mg/L, and another RFOST sample on 12/9/11, with a particulate value of 8.20 mg/L.
2/06/12 Former Chemistry Manager enters false 22 EDG FOST test results in Chemistry database, using the log-in identity of another person. False data purports to show a 22 EDG FOST sample on 12/7/11, with a particulate value of 2.30 mg/L.
2/13/12 e self assessment team writes CR-IP2-2012-00901 to document the late entries and failure to write Condition Reports.
2/17/12 Cover-up continues by the former Chemistry IVIanager, en he writes CR-IP2-2012-01 039, falsely indicating at RFOST fuel oil re-samples were tested in-house, but not documented "in a laboratory format."
2/29/12 The self assessment team writes CR-IP2-2012-0 1253: 1) No CR written nor notifications made for entry into TS actions following receipt of out-of- _
specification GT 2/3 RFOST test results; 2) Quality records backlogged and not entered for approximately1 0 months; 3) No analysis/record (other than database) supporting exit from a TS action statement.
Knowledge/Understanding as of Early March 2012 We had performance issues within the Chemistry Department (late-posting of lab results, no CRs or operational response for out-of-specification results, and poor documentation of purported in-house sample analyses).
Our understanding, as expressed in the last three bullets, turned out not to be accurate.
Employees Raise Questions 3/12/12 Employee initiates concern with Entergy Ethics Line, questioning validitv of 12/9/11 GT 2/3 RFOST "re-sample."
3/17/12 Anonymous concern to IPEC CP questions validity of 6/29/11 and 12/9/11 GT 2/3 RFOST "re-samples."
Concern alleges, "There may be violations of 10 CFR 50.5 and other federal laws."
3/19/12 Entergy retai s Morgan Lewis to conduct non-privileged investigation.
(NOTE: the Senior Resident was otified of the concerns raised in mid arch 2012)
Greater Management Focus and Initial Corrective Actions 2/28/12 Quarterly GT 2/3 RFOST sample drawn.
3/13/12 High out-of-specification (12.0 mg/L) Herguth laboratory test result received by Chemistry Department for 2/28/12 GT 2/3 RFOST sample. The self assessment leads to improved response by Chemistry Department personnel.
3/14/12 Chemistry Supervisor writes CR-IP2-2012-01831 and CR-IP3-2012-00805 noting: that the high particulate value is above TS, that the testing method was not in accordance with TS methodology, and that are-sample and re-test were required.
Operations declares RFOST inoperable and enters 30 day action statement for Unit 3 (Unit 2 creates a mode-hold).
3/15/12 IPEC Chemistry Department procedure 0-CY-3180 revised to allow on-site testing of fuel oil for particulates.
3/15/12 Sample drawn from GT-2/3 RFOST. On-site analysis using new revision of procedure yields a result 1.2 mg/L for particulates.
Result is within TS. Operations declares tank operable.
nbeknownst to all but some administrative personnel, the former Chemistry Manager also sends a split sample to Martel Laboratories for analysis, but "for information on._ -"
3/29/12 Martel Laboratories provides results from Mar. 15, 2012 RFOST sample to former Chemistry Manager and Chemistry admin only. Results are high out-of-specification (18 mg/L). No CR written. Former Chemistry Manager does not disclose results to management or subordinates.
4/1 - 4/17/12 Multiple attempts to interview the former Chemistry anager were unsuccessful due to his post-outage vacation and unspecified family issues.
4/17/12 Former Chemistry Manager resigns from Entergy after having refused to cooperate in Entergy investigation.
4/18/12 Because of the former Chemistry Manager's resignation, Senior Management instructs the acting Chemistry Department Manager to have technicians take another fuel oil sample from GT 2/3 RFOST. Split samples to be analyzed in-house and by Herguth.
4/19/12 In-house analysis of 4/18/12 RFOST sample yields particulate level of 2.59 mg/L; within TS.
4/30/12 Chemistry Department receives Herguth laboratory report for 4/18/12 RFOST sample. Results are high out-of-specification (12.8 mg/L) and inconsistent with in-house analysis.
5/02/12 Acting Chemistry Manager writes CR-IP2-2012-03135 and CR-IP3-2012-01325,which reports the conflict between in-house analysis and Herguth analysis for 18/12 RFOST sample. Evaluation concluded in-house testing results were valid. Fuel oil in RFOST remained operable.
5/25/12 Sample drawn from GT 2/3 RFOST.
6/21/12 Herguth reports particulate level of 10.8 mg/L for 5/25/12 RFOST sample. CR-IP2-2012-04132 and CR IP3-2012-01914 written to document high out-of-specification result. Both units enter 30-day action statements. 60-day LER clock initiated.
6/22/12 NSA Director writes CR-IP2-2012-04164, noting lack of credible evidence to support alleged "re-samples" on 6/29/11 and 12/9/11.
6/21 - 6/28/12 Multiple fuel oi1 samples taken for the RFOST.
Although some test results were within Technical Specification limits, I EC continued to consider the RFOST out of specificatio ...
7/05/12 IPEC provides total required reserve fuel oil wnn new fuel oil stored in temporarv on-site trailers to allow for ltering of fuel oil.
7/16/12 IPEC completes an extent of cond on review, which found no credible evidence for the 2.3 mg/L particulate level of an alleged 12/7/12 "re-sample" drawn from 22 FOST. Former Chemistry Manager made this false entry into Chemistry database on 2/6/12. CR-1 P2-20 12-04617 written to memorialize the discovery. (Herguth testing of 22 FOST samples drawn on 2/1 0/12 and 5/2/12 confirmed fuel oil was within TS after the anomalous 22 mg/L test results of 12/7/11.)
7/18/12 Samole d GT 2/3 RFOST.
7/19/12 ergutn reports oarticulate level of 7/18/12 RFOST sample of 7.44 mg/ -*
7/19/12 Sample drawn from GT 2/3 RFOST.
7/20/12 Herguth reports pa culate level of 7/19/12 RFOST samole of 7.68 mg/L 7/20/12 Filtering of R OST fuel oil complete.
7/24/12 GT 2/3 OST declared operable.
7/25/12 organ Lewis issues Report of Investigation.
8/20/12 Entergy submits LER to NRC reporting operations in a condi on prohibited by Technical Specifications.
12/6/12 FOST fuel oil replaced with new oil.
7/23/13 .S. Atto ey issues criminal comolaint against former Chemistry 1v1anager.
Failure to:
enter important laboratory results in the chemistry database and analyze in a timely ma ner; enter out-of-specification results in CAP, and to take appropriate actions to address discrepancies.
Knowledge weakness of a chemistrv soecialist.
Former Chemistry Manager's creation of false data and other actions to conceal *olations of Technical Specifications; These issues resulted in op rating outside Technical Specifications and failing to provide a timely LER.
Our corrective acnons nave een comprenensive.
We have addressed:
The specific technical/operational issues qualitv of the fuel oil in RFOST; The process and performance weaknesses in the Chemistry Department; Safety culture and SCWE implications; and Deliberate misconduct of former Chemistry Manager.
uality of Fuel in RFOST The earlier discussion the facts identified the immediate actions taken to understand and address this issue.
On July 20, 2012, we completed ltering the fuel oil and returned it to R OST. Off-site testing confirmed fuel oil met Technical Soecifications oarticulate requirements.
Subsequent tests showed accepta le, but higher than anticipated results. As a result, ntergy decided to replace all of the fuel oil in the RFOST.
By December 6, 2012, we cleaned e tank and had replaced all of the fuel oil in RFO new fuel.
Testing since replacement nas ed that the fuel oil eets TS.
Process and Performance Assigned new Chemistry 1v1anager.
Reinforced trending requirements for chemistry parameters.
Reinforced CR initiation protocols and requirements for chemistry in continuing training.
Reinforced expectation for tracking of vendor analyses and entry of data into chemistry database.
Reinforced requirements for timely evaluation/disposition of test results.
Reinforced required actions for out-of-specification conditions.
Chemistry Department Process and Performance (cont'd)
Reinforced requirements to validate data in chemistry database (second check) and also includes trend analysis and identification of out of specification conditions.
Discontinued onsite fuel oil particulate analysis.
$ Provided case study covering el oil events as part of continuing chemistry training.
TS/TRM fuel oil surveillances are tracked in INDUS work management and the IPEG site sched Track drawing the sample Track shipping the sample Track receipt of analysis results and enter and trend the results in the database.
Safety Culture/SCWE Independent assessments of safety conscious work environment (SCWE) in Chemistry Department confirm existence of healthy work environment.
Independent investigation- Anonymous concern of March 17, 2012, asserted that "Several members of the department are aware of this issue [i.e., the absence of there-samples] and fear reprisal if they come forward."
Investigation included interviews with Chemistry supervisors, Specialists, and Chemistry Technicians involved in fuel oil sampling.
Investigation found that the employees in the Chemistry Department are not reluctant to raise issues or concerns. No one interviewed expressed reluctance to raise an issue.
Chemistry personnel consistently reported department management and supervision encouraged the raising of concerns and respected the priority of nuclear safety.
Synergy Survey resu m February-April2012 reflect healtnv worK e ent 1n Chemistry Departme .....
Safety culture and safety conscious work environment were also assessed by external parties.
RC PIR October/November 2013 INPO Evaluation and Assessment December 2013
Deliberate Misco Site VP/GMPO conducted "All Hands" meeting on 6/28/12 to address deliberate misconduct. (JAF "mask fit" event used as case study).
Points of emphasis included:
Procedure compliance Seriousness and consequences of deliberate misconduct Reporting misconduct Completeness and accuracy of information/communications New Chemistry Manager briefed Department on results of Entergy investigation (Aug/Sept 2012).
Site Nuclear Safety Culture Monitori g Panel (NSCMP) reviewed the falsification event and recommended briefing of management team.
Safety Culture Leadership Team and NSCMP concluded the falsification was a discrete issue, without broader implications. Site management team received briefing at Site VP meeting in December 2012.
Deliberate Misconduct (Cont'd Site briefed on IN 13-015 re: Willful Misconduct at Leadership and Alignment meeting. Entergy added discussion of Entergy events (Pilgrim operator inattentiveness, ANO E-plan falsification of records, JAF mask fit event, and misco duct of former IPEC Chemistry anager) (Sept 2013).
GMPO briefed Department anagers:
Reinforced compliance with 10CFR50.9 and 50.5; Reinforced expectation for generation of CRs; Reinforced importance of timely, accurate communications; Reinforced obligations to safety and public, "If we have to shutdown the plant we will do so" (Dec. 2013).
Department Managers briefed their departments.
II I
Entergy Fleet Actions Prepare summary of event for distribution to Entergy fleet.
Reviewed current CBT for need to amolify 50.5/50.9 training.
Performed an assessment of corrective actions for adequacy and thoroughness.
Will request post-sentencing interview of former Chemistry Manager for insight and potential inc1usion in fleet summary of event discussed above.
Instituted monthly Department Performance Review Meetings to drive intrusive reviews of department performance.
Implemented the monthly "one-on-one" process to improve communication and individual work performance and to reinforce standards and expectations.
Fleet Nuclear Oversight initiative driving more intrusive oversight.
ithout diminishing the se *ousness of the event, it is portant to recognize some positive aspects:
The self-assessment team discovered the anomalies; Earlier 50.5/50.9 fleet training was effective in helping employees to recognize, report, and document the issue; An employee raised a concern with the Entergy Ethics Line; Other(s) raised a concern through the IPEC Employee Concerns Program; We conducted an investigation, which identified the wrongdoing and performance weaknesses; Our extent of condition investigation uncovered the issue with the FOST for 22 EDG; and We have taken comprehensive corrective action.
Actual safety signn1cance:
Evaluation found the actual impact on reactor safety was not s1gn1ncant.
High particulate level may cause plugging or clogging in EDG fuel oil system filters.
Tech Spec limit of 10 mg/L is conservative.
EPRI's Diesel Fuel Oil Guide considers 20-25 mg/L to be the aximum accepted usable limit for particulates in fuel oil.
Additionally, each EDG has two sets of Duplex filters on the inlet and outlet side of the fuel booster pump. These filters alarm on high DP and actions are provided for the operator to change filters without interruption of EDG.
36