ML103020432

Lr - Doc List Docs Needed & Civil Question

ML103020432
Person / Time
Site:	Diablo Canyon
Issue date:	10/28/2010
From:	- No Known Affiliation
To:	Division of License Renewal
References
Download: ML103020432 (42)

Text

DiabloCanyonNPEm Resource From: Tan, Miranda [M1TF@pge.com]

Sent: Thursday, October 28, 2010 5:50 PM To: Pick, Greg Cc: Grebel, Terence

Subject:

RE: Doc List Docs needed & Civil Question Attachments: 2005Chemistry.doc; 2009 Chemistry and Radiochemistry Audit.doc

Greg, I'd like to double check with you on Firewater SAPN 50445449. Our system is saying that number doesn't exit.

Fuel Oil Chemistry 2005 & 2009 chemistry audits are attached.

Best, Miranda Tan 805 781 9415 From: Pick, Greg [1]

Sent: Thursday, October 28, 2010 1:00 PM To: Tan, Miranda Cc: Grebel, Terence

Subject:

Doc List Docs needed & Civil Question Good afternoon, As I stated in my voice mail, I need to have the following documents uploaded electronically. I have 7-8 sections still to compare the doc list to the electronic files - this means I may have a similar small request.

We would also like a discussion related to the plans by civil engineering in regards to the strain gauge cover plates. Specifically, we wanted to know what your likely choice of CAs would be going forward. We have no immediate concerns but would like to know the answer prior to signing out the report.

A time next week will work since both J. Melfi and I are in the PDT zone.

Have a great day!!

1

Hearing Identifier: DiabloCanyon_LicenseRenewal_NonPublic Email Number: 2049 Mail Envelope Properties (D065043718A59C4B99DDA1862BB067DE01BA241E)

Subject:

RE: Doc List Docs needed & Civil Question Sent Date: 10/28/2010 5:49:58 PM Received Date: 10/28/2010 5:50:10 PM From: Tan, Miranda Created By: M1TF@pge.com Recipients:

"Grebel, Terence" <TLG1@pge.com>

Tracking Status: None "Pick, Greg" <Greg.Pick@nrc.gov>

Tracking Status: None Post Office: exchange12.Utility.pge.com Files Size Date & Time MESSAGE 1107 10/28/2010 5:50:10 PM 2005Chemistry.doc 325184 2009 Chemistry and Radiochemistry Audit.doc 239680 Options Priority: Standard Return Notification: No Reply Requested: No Sensitivity: Normal Expiration Date:

Recipients Received:

Date: March 11, 2005 File #: EDMS #043360019 To: MANAGER, CHEMISTRY AND ENVIROMENTAL OPERATIONS From: SUPERVISOR, QUALITY VERIFICATION

Subject:

2005 Chemistry and Radiochemistry Program Audit LANCE HOPSON Quality Verification (QV) performed the 2005 Chemistry and Radiochemistry Program Audit from February 1, 2005 through February 18, 2005. The audit included four site personnel and a technical specialist from Fermi 2 Nuclear Power Plant. The purpose of this biennial audit was to verify the graded quality assurance requirements were properly implemented for the overall implementation and control of various chemical and radiochemical limits and processes.

The enclosed report provides the results of the audit. A summary of the results was presented at the audit exit conference held on February 18, 2005.

This audit verified that the DCPP Chemistry and Radiochemistry Programs are effectively implemented. The major areas for improvement are documented timely review of samples and analyses, documented evaluation and review of chemistry process changes, and increased leadership in resolving safety issues. Other areas for improvement include timely review of Operating Experiences and inclusion in engineering reviews, more effective use of the event trend reporting program, increased support of the self-assessment program, consistent application of the three error reduction techniques to reduce human errors, and clearer procedural guidance on the transferring of quality documents from the originating departments to the records management system.

The audit team documented several positive observations during the performance of this audit.

There were several observations of good tailboards and effective use of the error reduction techniques. Laboratory and sampling area cleanliness has improved since last audit.

If you have questions regarding the audit or the report, please contact Robert Clark at 3670.

Signature on Original Robert R. Prigmore cc: See audit distribution list

2005 CHEMICAL AND RADIOCHEMICAL PROGRAM AUDIT EDMS # 043360019 DISTRIBUTION GRueger 77Beale/B32 RLocke 77Beale/B30A DOatley 104/6/600 JBecker 104/5/500 PRoller 104/5/504 LHopson 104/5/517 JGardner 104/5/11B SPigeon 104/5/9A SDavid 104/5/519 RSnyder 109/2/211 AVersaw 104/3 CBelmont 104/5/501 DTaggart 104/3/328 NRC Resident 104/5/543 JMartin NSOC LPruett for NSOC 104/5 Audit Team

2005 CHEMISTRY AND RADIOCHEMISTRY PROGRAM AUDIT EDMS # 043360019 February 1 through February 18, 2005 Chemistry Section Report Prepared by: Signature on Original 3/11/2005 Audit Team Leader Robert N. Clark Date Report Approved by: Signature on Original 3/11/2005 QV Plant Support Robert Prigmore Date Supervising Engineer

2005 Chemistry and Radiochemistry Program Audit EDMS # 043360019 TABLE OF CONTENTS 1.0 SCOPE 2.0 EXECUTIVE

SUMMARY

3.0 AUDIT RESULTS & CONCLUSIONS 3.1 Audit Topics, Auditor, and Results 3.2 Positive Observations 3.3 Self-Assessment Program 3.4 Training and Qualifications of Personnel 3.5 Identification and Resolution of Equipment and Personnel Safety Issues 3.6 Optimum Protection of Plant Systems 3.7 Program responses to Out of specification Chemistry Conditions 3.8 Implementation of Outage Lessons Learned 3.9 Radionuclide Analysis and Quality Control 3.10 Chemical Analysis and Quality Control 3.11 Compliance with Technical Specifications and Equipment Control Guidelines 3.12 Communication and Coordination with Operations 3.13 Human Performance 4.0 CORRECTIVE ACTION RESPONSES 5.0 EFFECTIVENESS EVALUATION 6.0 APPENDICES 6.1 Quality Problem Reports - A Type ARs 6.2 Non-Quality Problem Reports - N Type ARs 6.3 Event Trend Reports 6.4 Comments and Recommendations 6.5 Audit Team Composition 6.6 Personnel Contacted and Meeting Attendance 6.7 Source Documents and References 6.8 Departments and App. B Binning

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 1.0 SCOPE The Chemistry and Radiochemistry Program Audit is required by Chapter 17 of the UFSAR. The scope of this audit for 2005 included verification that the graded quality assurance requirements are properly implemented for the overall implementation and control of various chemical and radiochemical limits and processes.

Requirements for these programs include the Updated Final Safety Analysis Report (UFSAR), Technical Specifications (Tech. Specs.), Equipment Control Guidelines (ECG),

and plant procedures. The audited organization was the Chemistry Section with field observations of interactions with the Operations Section.

Quality Verification (QV) reviewed the following areas during the performance of this audit:

• Corrective Action Response and Program

• Self-assessment Program

• Training and Qualification of Personnel

• Identification and Resolution of Equipment and Personnel Safety Issues

• Optimum Protection of Plant Systems

• Program Response to Out of Specification Chemistry Conditions

• Implementation of Outage Lessons Learned

• Radionuclide Analysis and Quality Control

• Chemical Analysis and Quality Control

• Compliance with the Technical Specifications and Equipment Control Guidelines

• Communication and Coordination with Operations

• Human Performance The audit team obtained objective evidence of the program implementation through the review of program procedures, implementation documentation, interviews of management personnel and technicians, and observations of sampling and analyses.

This audit was performed within the required entrance due date of May 16, 2005 for the audit on the Quality Verification - Plant QA Group Master Internal & External Audit &

Review Schedule of December 7, 2004.

Page 3 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 2.0 EXECUTIVE

SUMMARY

Quality Verification performed the audit of DCPPs Chemistry and Radiochemistry Program as required by Chapter 17 of the FSAR. The audit team included four site auditors and a technical specialist in the area of Chemistry from Fermi 2 Nuclear Power Plant.

This audit verified that the DCPP Chemistry and Radiochemistry Programs are effectively implemented. Although there are areas for improvement, the program, as currently implemented, is meeting the requirements of Program Directive CY1 Chemistry /

Radiochemistry. The following three areas do not currently compromise the program, but require improvement to ensure safe and effective program implementation.

• Documented timely review of samples taken and analyses performed to support Tech. Spec., ECG, and procedural requirements.

Three areas warranted improvement in the documentation of the review of samples obtained, analysis results, and identification of out of specification conditions.

1. Review of Sample Schedule Various methods are used, by each chemistry foremen, to ensure that the assigned sample and analysis tasks are performed. Informality of tracking and review has led to missed samples.

2. Review of Sample Analyses The audit team identified several data sheets containing Tech. Spec. and ECG sample analyses data that did not have timely documented reviews.

3. Trending Data Base Flags WINCDMS is a chemistry database used to collect sample data from the chemistry technician, store the data for trending, and transmit the data to Operations and other plant personnel. Chemistry foremen imbed flags into WINCDMS that identify the out of specification (OOS) parameters. These imbedded flags are not verified and validated in any systematic and documented manner. This has led to OOS conditions being missed by Chemistry management.

• Documented evaluation, review, and approval process for chemistry process changes that have a potential affect on plant systems or personnel safety.

There is currently no documented process for evaluating, obtaining cross discipline reviews, or approvals for new or unusual processes, chemistry regimes, or plant system alignments sponsored by the Chemistry section. The lack of formality in the process has led to three plant events that might have been prevented.

• Increased leadership in resolving safety issues.

The audit team determined that there are several examples of personnel safety issues that need additional leadership by the Chemistry section. Some of the issues constitute major projects like fixing the buttress building roof or adding a closed cooling water injection skid. Some of the issues are process changes that require personal contacts or scheduling changes like maintaining the outfall or coordinating the filling of the TSC liquid nitrogen dewer. One of the issues was not responding expeditiously to minor fire and chemical cabinet deficiencies. QV is concerned that by not resolving safety issues in a Page 4 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 timely manner, Chemistry personnel will perceive that the conditions are acceptable and will decrease their advocacy for resolution.

Additional Areas for Improvement Self-Assessment Program:

The use of Operating Experience (OE) by the Chemistry section has some strengths including the use of OEs in structured tailboard notes and a web page access to relevant OEs.

Weaknesses included the lack of systematic searching for, and incorporating of the OEs into chemistry engineering evaluations and the excessive time taken to formally review assigned OEs. Three significant plant events involving the Chemistry section were partially the result of a weak operating experience search and application of the information. The current average of 2003 & 2004 OE evaluations by Chemistry is 116 days. This is in excess of the OM4.ID3 requirement of 60 days.

Chemistry performed a reasonable number of subject appropriate, good quality Self-Assessments (SA) with good recommendations. However, the lack of resources to implement the recommendations and the limited management support for the program has constrained the effectiveness and commitment to the SA program.

Training and Qualifications of Personnel:

Some Chemistry foremen did not have a clear understanding of the expectations for procedure use, technician qualifications, or tailboard guidance. With the turnover of the Chemistry foremen occurring and the problem of ensuring that adequate knowledge, understanding, and expectations for the position is maintained, QV believes a formal training program for Chemistry foreman is warranted.

In the area of ANSI qualifications, two quality problems were identified. Four of five ANSI qualification worksheets and the attached resumes did not have sufficient detailed information to validate the months assigned to overhaul experience and job coverage experience. Another ANSI qualified technicians ANSI worksheet documentation could not be retrieved and supplied to the auditor for review.

Quality Records Management:

AD10.DC3 lacks the necessary detail required to transmit quality records in accordance with UFSAR 17.17-1 and ANSI N45.2.9 - 1974 "Requirements for Collection, Storage, and Maintenance of Quality Assurance Records for Nuclear Power Plants. This is based on the fact that a desk guide contains specific information on maintaining chain of custody for the records, rather than a procedure. One of the desk guides approved options is to leave the quality records unattended in the mailroom for periods of time, typically hours, and in rare cases overnight.

Human Performance:

The audit team observed excellent application of the three error reduction techniques; tailboards, self-verification, and three-way communication. Examples included a detailed morning tailboard that incorporated safety and a discussion of sampling and sink radiological practices, a stator core cooling water sample that used the procedure in hand, and three-way communication with the operations crew.

There was also an observation of poor error reduction techniques used during the zinc acetate addition where the procedure was in the field but not followed, Operations Page 5 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 personnel accepted a degraded piece of equipment as normal, and did not write a problem report on the equipment.

The auditors identified an increased number of human errors in the Chemistry section during 2004. Examples were in various areas. The errors spanned both the chemistry laboratory and count room, and were attributed to technicians, foremen, and engineers.

The Chemistry section needs to be more aggressive at understanding the root causes of this trend, raise the human performance expectations, and monitor the application of the expectations to ensure consistent application of error reduction techniques.

Corrective Action Program:

Chemistrys use of the Event Trend Reporting (ETR) program to identify low level issues before they become problems is not effectively implemented. The augmented trending codes are only focused on human performance issues and do not include equipment or program issues. The Chemistry augmented trend section is only trending 12% of the total Chemistry ETRs.

The auditors review of various corrective actions identified a tendency to close ARs to promised future actions or no action. These ARs included non-quality evaluations closed to the procedure change request database, a prudent action to a QE that was closed stating that a trending policy exists, when in fact there is no written policy, and a recent AR concerning a gamma detector calibration error that was closed with statements on evaluation and correction of past data, but no correction of the cause of the original calibration error.

Positive Observations The audit team noted a number of areas where the program performance has improved since last audit or met or exceeded established standards, including the following:

• Morning tailboards were comprehensive and covered human performance areas such as industrial safety, radiological practices, operating experiences, and equipment condition.

• Laboratory and sample area cleanliness has improved since last audit.

• Some field observations of plant sampling demonstrated good procedure in hand use, self-verification, and three-way communication.

Page 6 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 Quality Problem Reports - A Type Action Requests (ARs)

AR # Org Quality Problem Report Title A0633285 PGPT Incomplete review of new chemistry process or plant alignments A0633290 PACR Insufficient information to support ANSI qualification A0633291 PACR ANSI Qualification worksheet for a C&RP technician is not retrievable A0633298 PAGS AD10.DC3 lacks necessary detail for transmitting quality records A0633305 PGPT Documented chemistry data review is not timely A0633308 PGPT Lack of a completed and reviewed sample schedule Non-Quality Problem Reports - N Type Action Requests (ARs)

AR# Org Non-Quality Problem Report Title A0633286 PGPT Ineffective use of Event Trend Report (ETR) program A0633287 PGPT Closing ARs to a promise or an action that is not completed A0663292 PGPT No formal chemistry foreman training A0633293 PGPT Chemistry personnel safety issues A0633294 PGPT Excessive maintenance trend on Cel-900 (RCS DO2/H2)

A0633296 PGPT Safety discrepancies in secondary laboratory A0663297 PGPT Inconsistent breaker identification A0633302 PGPT WINCDMS OOS flags are not verified and validated A0633310 PGOE OP B-1A: XV procedural non-compliance A0633314 PGPT CAP O-15 procedural non-compliance A0633316 PGOF Failure to document problem with AR A0633317 PGOM Failure to meet management expectations A0632519 PTEE ZAIP 2-1 Pulsation damper does not hold nitrogen pressure Page 7 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 3.0 AUDIT RESULTS & CONCLUSIONS 3.1 Audit Topics, Auditors, and Results The following table lists the topics that were audited, the key auditor, and the overall results:

1. Topic Auditor Results 1 Corrective Actions R Clark 1 Quality Problem 2 Non-Quality Problem 2 Self-Assessment Program R Clark 3 Training and Qualifications of Personnel R Clark 2 Quality Problem 1 Non-Quality Problem 4 Identification and Resolution of Equipment R Clark 1 Non-Quality Problems and Personnel Safety Issues 5 Optimum Protection of Plant Systems M Mosher 6 Program Response to Out of Specification M Mosher 1 Non-Quality Chemistry Conditions 7 Implementation of Outage Lessons Learned M Mosher 8 Radionuclide Analysis and Quality Control Dominic Dale & 1 Quality Problem Jeff Harker 9 Chemical Analysis and Quality Control Dominic Dale & 1 Quality Problems Jeff Harker 3 Non-Quality Problem 10 Compliance with the Tech. Specs. And J Hutcherson 1 Quality Problem ECGs 11 Communication and Coordination with J Hutcherson 5 Non-Quality Problems Operations 12 Human Performance Audit Team 3.2 Positive Observations During this Audit, QV made the following positive observations:

• Improved housekeeping over observations during last outage. This was specifically noted at the NWT feedwater filter sampling station and primary and secondary laboratory.

• Good review of radiochemistry QC charts. This is a noted improvement from last audit.

• The chemistry clerk and the chemistry foremen do a good job of working together to back each other up, ultimately ensuring that all required records are completed and turned into the clerk for processing in RMS.

Page 8 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019

• Morning tailboard discussed and reinforced the use of good radiological practices, operating experiences, and equipment condition.

• Good self-verification of sample point and sample bottle labels during the RCS sampling.

• Good communication with operation personnel, in hand procedure use, three-way communication, and self-verification during stator core cooling water sampling.

• Chemistry quarterly training covered 1R12 lessons learned, including C&RP Tech responsibilities pertaining to the Steam Generator Chemical Cleaning (SGCC) during 2R12, the injection of titanium dioxide into the SG after the SGCC, and the outcome of the 1R12 SGCC.

• Operations delayed the start of forced oxygenation during 2R12 in order to come back on track with the RCS Letdown Filter Plan.

3.3 Self-Assessment Program The auditor reviewed the Chemistry SA for 2001 through 2004. The following observations were made:

• The number of medium to high impact SAs was four in 2001, one in 2002, nine in 2003, and five in 2004.

• The 2003 peak performance of nine SAs in 2003 corresponded to the emphasis placed on SAs by the plant.

• 2003 has three open SA ARs.

• In 2003 one SA had 9 of 29 AEs rejected due to excessive workload and another SA that accepted only 6 of 25 AEs.

• In 2004 one SA report has not been issued and two others do not have reports in EDMS.

• One SA had all five AEs closed with no action, three were closed due to lack of resources, one was rejected, and one was closed to a promise for future action.

Another SA had two of four rejected, one due to lack of funding.

In summary, during 2003 and 2004 Chemistry performed a reasonable number of subject appropriate, good quality self-assessments with good recommendations. However, the lack of financial and personnel resources to implement the recommendations and the limited management support for the program has limited the effectiveness and commitment to the SA program.

The auditor reviewed the availability and use of Operating Experiences (OE) by the Chemistry section. The review included the INPO web page, the Chemistry web page, ARs relating to OE evaluations, and significant chemistry events at DCPP.

• The Chemistry Web Page has a section dedicated to Operating Experiences. This is a good listing of relevant OEs. The lists were last updated on June 29, 2004.

• Chemistry does a good job of incorporating plant and industry operating experiences into structured tailboard notes.

Page 9 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019

• Three significant plant events involving the Chemistry section were partially the result of a weak operating experience search and application of the information. If the operating experience lessons had been applied to these events, the problems might not have occurred.

• Chemistry is not timely in evaluating operating experiences. The current average of five 2003 & 2004 OE evaluations is 116 days. This is in excess of the OM4.ID3 Assessment of Industry Operating Experience requirement of 60 days.

In summary, the use of operating experience by the Chemistry section has both strengths and weaknesses. The strengths include the use of OEs in structured tailboard notes and a web page access to relevant OEs. The area of weakness is in the systematic searching for and incorporating the OE experiences into chemistry engineering evaluations and the excessive time taken to formally review assigned OEs.

This area is satisfactory, though improvements are warranted in the self-assessment and operating experience programs.

3.4 Training and Qualifications of Personnel The auditor reviewed the chemistry technician training program in the areas of ANSI qualifications of new hires, personnel chemical safety training, and qualified technicians performing assigned tasks.

In the area of ANSI qualifications, two quality problems were identified. Four of the five ANSI qualification worksheets and the attached resumes reviewed did not have sufficient detailed information to validate the months assigned to overhaul experience and job coverage experience. This is a quality problem AR A0633290. Another ANSI qualified technicians ANSI worksheet documentation could not be retrieved for the auditors review.

These are quality records that need to be retrievable. This is a quality problem AR A0633291.

A review of personnel chemical safety training demonstrated an effective initial and continuing safety training program. The review of technician tasks performed against verified qualifications did not result in the identification of any problems.

During performance observations and interviews with foremen, the audit team observed weaknesses in administrative program knowledge. These weaknesses included:

• A foreman demonstrated little ownership or knowledge of the procedure revision control process or his responsibility to ensure correct procedures are used.

• A foreman stated that it was the training departments responsibility to verify and ensure that the technicians were qualified for their assigned tasks.

• A foreman was unaware of the Chemistry section policy A-03 Use of Key Behaviors in Chemistry.

With the continual turnover of the foremen staff and the problem of ensuring clear and complete knowledge and understanding of plant and section procedures, policies, and expectations, QV believes a formal documented process for training chemistry foremen is warrented. This is a non-quality problem AR A0633292.

This area is satisfactory with two quality problems and one non-quality problem. The ANSI qualification problems are of low significance because they are isolated records problems that should be easily corrected and prevented in the future.

Page 10 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 3.5 Identification and Resolution of Equipment and Personnel Safety Issues The auditor reviewed the current status of personnel safety in the Chemistry section by interviewing Chemistry foremen and technicians and by reviewing the Chemistry web site Chemistry & Environmental Operations Safety and Work Around Issues. During this review several issues were noted. They included:

• The Unit # 1 & 2 turbine building buttress roof leaks causing water to leak into light fixtures, onto equipment, and to cause wet slippery floors.

• Isothiazolin and glutaraldehyde are routinely handled in quantities for closed cooling water chemical additions, which require the use of respiratory protection instead of engineering controls. The glutaraldehyde issue was identified on 3/4/03 in AR A0577176 and includes an evaluation for the installation of a permanent chemical addition skid. QV did not find any AR to evaluate the hazard of isothiazolin or the routine use of respirators and the impacts on the priority of the permanent chemical addition skid.

• The portable closed cooling water addition skid requires heavy lifting to move it around pipes and berms from one location to another.

• Filling the TSC nitrogen dewer entails either delays for security to unlock the back door to the TSC or heavy lifting of the dewer up the stairs.

• The primary and secondary laboratory fume hoods are required by the Chemical Hygiene Plan to be inspected quarterly. The laboratory hoods were last inspected on 10/29/04. Some of the chemistry foremen were not aware of the required frequency of inspection.

• The outfall stairs are muddy and slippery, and there were electrical cords laying in the mud and water.

Some of these issues are facility or equipment repair or modification issues and some are process change issues. In all cases, though, increased awareness and leadership by Chemistry management is needed to resolve them. This is a non-quality problem AR A0633293.

This area is satisfactory with one non-quality problem. This problem is of medium significance because although some problems have been resolved. Continued attention and advocacy by the Chemistry organization for resolving personnel safety issues is important to prevent unsafe plant conditions. By not resolving personnel safety issues in a timely manner, personnel perceive that the conditions are acceptable to work around and over time will stop advocating for resolution.

3.6 Optimum Protection of Plant Systems This area was reviewed in more detail in the section on Out of Specification Chemistry Conditions.

3.7 Program Response to Out of Specification Chemistry Conditions The audit team reviewed the history and future plans for a variety of plant systems that have a track record of out of specification chemistry or problems associated with them.

These included the bio-fouling of closed cooling water heat exchangers, and iron transport into the Steam Generators. In general, DCPP has been slow to recognize and react to the adverse conditions, both of which have been pointed out by Chemistry, QA, and others for Page 11 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 years. In the case of the bio-fouling, Chemistry did utilize consultants as far back as 2002; however, the consultants concluded that our process for treating the closed cooling systems was adequate. We may have relied too heavily on their conclusion, since bio-fouling was already evident. Once INPO recognized both problems via Areas for Improvement in 2003, DCPP finally took action to resolve the situations. Currently Chemistry is actively participating in or leading teams to address the issues of closed cooling water bio-fouling, feedwater iron transport, condenser leakage, and RCS radioactivity clean up.

When looking at past ARs, the audit team identified 13 ARs in 2004 specifically associated with RCS hydrogen monitor, Cel-900. This piece of equipment or its components has created a great amount of work and should be evaluated to determine if it is cost efficient to keep it in service or if the underlying maintenance problems are being effectively resolved. This trend is a non-quality problem AR A0633294.

This area is satisfactory with one non-quality problem to evaluate an excessive maintenance trend.

3.8 Implementation of Outage Lessons Learned Lessons Learned (LL) from both 1R12 and 2R12 were reviewed. An overriding theme between the two outages pertained to problems with RCS Letdown filter change out.

Chemistry, Operations, and Radiation Protection developed a Letdown Filter Plan prior to 2R12. However, the plan was not followed at one point, causing Operations to delay Forced Oxygenation by almost 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> in order to come back into alignment with the plan.

Quarterly training includes information learned from 1R12 with respect to the Steam Generator Chemical Cleaning. Technician responsibilities relating to the cleaning during 2R12 were discussed, as were the results of the 1R12 cleaning and the use of titanium dioxide in the SGs after the cleaning. This is a positive observation for Learning Services.

Overall, the Lessons Learned module appears to be appropriately utilized by those in Chemistry, and generally the issues appear to be resolved in a timely manner. The auditors verified that appropriate steps were added to the Chemistry Pre-Outage Task Schedule for 1R13. The auditors recommend that AR numbers be added to the LL forms when they are filled out to track lessons learned to the corrective action program.

This area is satisfactory with one recommendation.

3.9 Radionuclide Analysis and Quality Control Approved procedures for the count rooms are available, followed, and adequately describe the guidance necessary for successful completion for the task. The radiochemical quality control charts are appropriately maintained, reviewed, and trended.

This was noted as an improvement in this area over previous audit findings. Additionally, the radiochemistry counting instruments are calibrated at the prescribed frequency and the sources used for that calibration are traceable to National Institute of Standards and Technology (NIST) standards.

A review of a sample of data sheets showed that when the data sheet is designed to be completed in one day, one days worth of data, the foreman review is performed in a timely manner. Other auditors observed that this was not true for data sheets that contained information for longer periods of time, days to months on a single data sheet.

A positive observation was made regarding the method the radiochemistry foreman and the chemistry clerk have to ensure that correct quality records are completed, reviewed, Page 12 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 and sent to RMS. However, one deficiency was identified in the area of the control of quality records. QV believes AD10.DC3 lacks the necessary detail required to transmit quality records in accordance with UFSAR 17.17-1 and ANSI N45.2.9 - 1974 "Requirements for Collection, Storage, and Maintenance of Quality Assurance Records for Nuclear Power Plants. This is based on the fact that a desk guide contains specific information on maintaining chain of custody for the records, rather than a procedure. In addition, the fact that one of the desk guides approved options is to leave the quality records unattended in the mail room for periods of time, typically hours, and in rare cases overnight. This is a quality problem AR A0633298.

This area is satisfactory with one quality problem. This issue of the transfer of quality records from the originating departments to RMS is a site wide issue that is of medium significance but is not directly related to radiochemical analysis program.

3.10 Chemical Analysis and Quality Control The audit team observed that chemical analyses are performed in accordance with sampling and analysis procedures and CAP Q-1. This determination is based on two observations of sampling and analyses, document reviews, and interviews with section personnel.

Procedures were noted to be readily available in the chemistry foremans office, followed during the sampling and analysis processes, and adequate to provide sufficient guidance to complete the tasks successfully. The Chemistry section does a good job of notifying operations personnel when its appropriate and the use of three-way communications was observed each time.

Instruments are calibrated at the required frequency, documentation is appropriate, and quality control charts are reviewed adequately to identify system trends.

Data sheets with one sample per sheet are completed and reviewed in a timely manner.

However, data sheets with multiple samples are not reviewed in a timely manner. The following chemistry analysis data sheets use a form that allows multiple samples on one data sheet. The data sheets have a documented foreman review when the data sheet is full. This process results in sample data that does not have an official documented Chemistry foreman review for many days after the initial sample. The following is a list of recently reviewed data sheets and the number of days from the initial sample to the date of the documented foreman review.

Component Cooling Water Data Sheet - Unit 1 - greater than 71 days Service Cooling Water Data Sheet - Unit 1- 68 days Refueling Water Storage Tank 1-1/2-1 Data Sheet - 53 days CVCS Mixed Bed Demin Data Sheet - 33 days Spent Fuel Pool 1-1 / 2-1 Data Sheet - 31 days PZR Liquid - 27 days Boric Acid Reserve Tanks 0-1 / 0-2 Data Sheet - 25 days Primary Water Storage Tank 1-1 / 2 23 days Boric Acid Storage Tanks and RWSTS data Sheet - 18 days This lack of timely data review has resulted in the untimely identification of an out of specification condition as identified in Chemistry AR A06311228. In this case, Unit #1 PWST was sampled on 1-26-05 and TOC was identified as out of specification.

Chemistry management did not recognize this out of specification condition until 2/2/05 when reviewed by a chemistry technician during an observation by the auditor.

Page 13 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 This is a quality problem AR A0633305, because some of the samples that are receiving untimely reviews are Tech. Spec. and ECG required samples. The sample reviews exceed the required surveillance frequency and the required action completion time.

WINCDMS is a chemistry database used to collect sample data from the chemistry technician, store the data for trending purposes, and transmit the data to the Operations section and other interested plant employees. Chemistry foremen and other authorized personnel imbed flags into WINCDMS that identify out of specification conditions on each sample parameter. There is no documented verification or validation of the out of specification flags in WINCDMS. This chemistry data collection, review, and transmittal process led to a problem observed by QV during this audit. On 2/2/2005 QV observed that total organic carbon (TOC) for the 9/15/2004 primary water storage tank (PWST) 1-1 was listed in WINCDMS as 106 ppm and was not flagged as out of specification.

Chemistry wrote AR A0631228 to identify this and another database data error. In discussions with chemistry personnel it was determined that there is no formal documented process for verification and validation of the out of specification flags in WINCDMS. This is a non-quality problem AR A0633302.

Only two instances of expired chemicals were noted in the lab that had passed their expiration dates by a week or more. This is a noted improvement over previous audits.

One instance of a discrepancy between the description of a circuit breaker in a procedure and the description on a lamicoid in the field was noted. This is a non-quality problem AR A0633297.

The audit team observed housekeeping and safety issues and noted an improvement in the area of lab cleanliness. Secondary Laboratory glassware was cleaned, dried and placed back on racks and shelves. The lab floor was in very good condition. However there were several low significance safety issues involving fire and chemical lockers that were identified by the audit team and were not resolved in a timely manner. This is a non-quality problem AR A0633296.

The problems related to timely document review of data are a significant issue because the data supports Tech. Specs. and ECGs. This area is satisfactory because the impact of these problems were isolated to a few examples.

3.11 Compliance with Tech. Specs. and Equipment Control Guidelines Based on a review of the data, discussion with Chemistry section personnel, and procedure review, the audit team concluded that the Chemistry group has procedures and policies in place, to assure that sample results required by Tech. Specs. and ECG are obtained, analyzed and trended. The system relies on an experienced staff to ensure this happens, rather than a consistent sample schedule review and a formal process to ensure that all samples are taken as required.

One problem, noted during the audit, concerned the untimely sampling of the ASW chlorine as required by CAP A-9 Auxiliary Sampling Schedule. While performing STP I-1C on Feb 6, 2005, operations personnel discovered a potential missed surveillance for ASW chlorine samples on both Units. ECG SR 17.2.1 requires chlorine samples to be taken every 7 days. The last known sample of ASW 1-2 and ASW 2-2 trains was on Jan.

21, 2005. This exceeds the surveillance interval and the 1.25 allowable extension. This also exceeds the 14 day allowed action period of ECG 17.2-A. ASW/CCW heat exchanger 1-2 train and ASW/CCW heat exchanger 2-2 train were declared inoperable.

Page 14 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 Chemistry does not have an integrated schedule that incorporates a timely foreman review of completed work. This is a quality problem AR A0633308.

During the investigation of this missed sampling event, the auditor noted an inconsistency between CAP A-3 and CAP A-9. CAP A-3 Technical Specification Sampling Schedule states: This procedure covers the chemical and radiochemical routinely scheduled surveillance requirements described in the Technical Specifications (Tech Specs) and Equipment Control Guidelines (ECG) created from relocated Tech Specs. Sampling of the ASW chlorine is in CAP A-9 Auxiliary Sampling Schedule which says in the discussion section On occasion, it is permissible to omit some sampling with approval of the senior chemistry engineer to attend to more urgent matters. It is mandatory, however, that all tests required by the Tech. Specs. be performed in accordance with their designated frequencies in CAP A-3. The placing of ASW sampling in CAP A-9 makes it appear less important than CAP A-3 sampling, yet failure to perform sampling on the ASW at the required frequency can render an ASW train and CCW train inoperable.

This area is satisfactory with the exception of one quality problem. The review of completed samples is a significant issue because the sample collection frequency supports Tech. Specs. and ECGs.

3.12 Communication and Coordination with Operations Observations and interviews, verified that Operations is notified in a timely and effective manner of plant sampling evolutions. Operations and Chemistry procedures were followed and coordinated the joint activities except as noted for the Zinc Injection Tank.

Three-way communication was consistently used by both Operations and Chemistry.

The auditor observed a zinc acetate addition to the Unit 1 / 2 zinc skid. The crew consisted of one Chemistry technician and three Operators, one qualified and two under instruction (UI). One UI Operator would be performing the Unit 1 OP B-1A:XV Zinc Injection Skid Operation procedure and the other would perform the Unit 2 evolution. When the Operator got to step 6.2.5 of the procedure, the operator was informed that the chemistry technician would actually be performing steps 6.2.5 thru 6.2.7 of the Operations procedure, which unlocks the pipe fill cap and adds the chemicals to the tank. OP B-1A:XV has a note that states per CAP O-15, chemistry should accompany the operator for concurrent verification of the addition. CAP O-15 Preparation of Zinc Acetate Dihydrate for Injection Into the Reactor Coolant System states in the responsibilities section that Operations is responsible for operating the zinc injection skid, including chemical additions to the injection tank. As performed, the verification was weak because the chemistry technician had already placed the containers on top of the tank and one of the operators only went part way up the portable stairs to look at a container. This is two non-quality problems AR A0633310 & AR A0633314. One problem was routed to Chemistry and one was routed to Operations Sections for procedure non-compliance and possible procedure revisions.

The auditor continued to observe the chemical addition and filling of the tanks with primary water. Both UI operators used three-way communications and self-verification.

During the restoration of the system and restarting of the zinc injection pumps, step 6.2.12 of the procedure has a note that this step only needs to be performed if the dampener is not working properly or if the system has been idle for weeks. The Unit 2 pump did not have the required 60-75 psig, so the Operator contacted the Operations Support Team (OST) to add nitrogen to the dampener. No AR was written by Operations for the failure of U-2 Zinc Pump pulsation dampener to hold pressure. A non-quality problem AR Page 15 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 A0633316 was initiated for not writing the equipment AR. QV wrote AR A0632519 to document the pulsation dampener problem.

The Unit 2 Operator could not restart the pump and complete the procedure until nitrogen was added by the OST later in the day. Because Operations called the OST directly to have nitrogen added after a zinc tank fill, and because there was no AR documentation of the equipment problem, QV could not determine the frequency of the problem. This is a type of operator burden that should not be accepted and taught to new operators. A non-quality problem AR A0633317 was written for Operations accepting a problem and not pursuing a resolution.

The communication and coordination between Chemistry and Operations is satisfactory, but the implementation of human performance and corrective action programs was unsatisfactory and resulted in the initiation of five non-quality problems.

3.13 Human Performance During this audit, the audit team performed numerous performance based observations of tailboards, sampling evolution, sample analyses, and chemical additions. The audit team observed very good application of the three error reduction techniques, tailboards, self-verification, and three-way communication. Examples included a detailed morning tailboard that included safety discussions of sampling and sink radiological practices, a stator core cooling water sample that used the procedure in hand, and three-way communication with the operations crew. There was also an observation of poor error reduction techniques used during the zinc acetate addition where the procedure was in the field but not followed, as discussed above.

The auditors identified 27 ARs initiated in 2004 that were a result of Chemistry personnel human error. Examples of these are required samples not collected, documentation errors, communication errors, errors with calculations in procedures, incorrectly performed calculations, not following procedures, missing data, operating wrong valves during sample evolutions, improper signature collection for procedure revisions, and others. The errors spanned both the chemistry lab and count room, and were attributed to technicians, foremen, and engineers. In February 2005, the Chemistry section created a human performance trend AR A0631334. The AR only addresses lax radiological practices and self-verification and documentation/data entry errors. It also appears to address errors only made by technicians. At the auditors request, the scope of this AR was expanded to include the other types of human errors listed above.

This area demonstrated inconsistent performance with notable examples as identified in several areas of this report. The Chemistry section needs to be more aggressive at understanding the root cause of thee examples and this trend, raise the human performance expectations, and monitor the application of the expectations to ensure consistent application of error reduction techniques.

4.0 CORRECTIVE ACTION RESPONSES The auditors review of the corrective actions from the 2003 Chemistry and Radiochemistry Program Audit determined that all of the quality issues were satisfactorily resolved. Some of the non quality and recommendation corrective actions were either not completed or were declined to implement.

Chemistrys use of the ETR program to identify low level issues before they become problems is not effectively implemented. The augmented trending codes are only focused Page 16 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 on human performance issues and do not include equipment or program issues. The Chemistry augmented trend section is only trending 12% of the total of Chemistrys ETRs.

This is a non-quality problem AR A0633286.

The auditors review of various corrective actions identified a tendency to close ARs to promised future actions or no action. Three non-quality evaluations were closed with the stated intent to change the procedure and to track the changes in the procedure change request database. None of these procedure changes were made. Although these procedure changes were not required, and therefore permissible to be tracked in the database, the actions stated in the evaluations were not completed. The prudent action to a Quality Evaluation (QE) was to establish a policy for Chemistry to trend specific parameters on a periodic basis. The tracking AR was closed stating that a trending policy exists, when in fact there is no written policy. A recent AR concerning a gamma detector calibration error was closed with statements on evaluation and correction of past data, but no correction of the cause of the original calibration error. This is a non-quality problem AR A0633287.

The auditor reviewed four QEs and six Apparent Cause Evaluations (ACE) performed by or involving the Chemistry section in 2003 and 2004. The auditor observed that there were three significant plant events that were the result of new processes implemented by the chemistry section. The three events were:

1) QE Q0012323 Cation bed produces unexpected reactivity results

2) QE Q0012380 Unit 1 RCS pressure transient during solid plant operations

3) AR A0609958 Startup venting released hydrazine above RQ and NCR N0002184 Hydrazine odors in various plant areas In each of these events the corrective actions included revising existing procedures to enhance the caution statements or review operating experiences. However, none of the cause analyses of these events identified that the tasks associated with these events were new processes, or changes from established procedures and process, and that they lacked a documented formal review and approval process prior to implementation. This is a quality problem AR A0633285.

This area is satisfactory with one quality problem and two non-quality problems. The quality problem is significant because the insufficiently reviewed Chemistry actions can affect plant operations.

5.0 EFFECTIVENESS EVALUATION Based on interviews with responsible personnel, performance based observations in the field, and objective evidence reviewed during the audit, the Audit Team has concluded that the DCPP Chemistry and Radiochemistry Program is effectively implemented. There are three major areas for improvement:

• Documented timely review of samples taken and analyses performed to support Tech Spec., ECG, and procedural requirements.

• Documented evaluation, review, and approval process for chemistry process changes that have a potential effect on plant systems or personnel safety.

• Increased leadership in resolving safety issues.

Page 17 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 6.0 APPENDICES 6.1 Quality Problem Reports - A Type Action Requests (ARs)

AR # Quality Problem Report Title A0633285 Incomplete review of new chemistry process or plant alignments A0633290 Insufficient information to support ANSI qualification A0633291 ANSI Qualification worksheet for a C&RP technician is not retrievable A0633298 AD10.DC3 lacks necessary detail for transmitting quality records A0633305 Documented chemistry data review is not timely A0633308 Lack of a completed and reviewed sample schedule 6.2 Non-Quality Problem Reports - N Type Action Requests (ARs)

AR# Non-Quality Problem Report Title A0633286 Ineffective use of Event Trend Report (ETR) program A0633287 Closing ARs to a promise or an action that is not completed A0663292 No formal chemistry foreman training A0633293 Chemistry personnel safety issues A0633294 Excessive maintenance trend on Cel-900 (RCS DO2/H2)

A0633296 Safety discrepancies in secondary laboratory A0663297 Inconsistent breaker identification A0633302 WINCDMS OOS flags are not verified and validated A0633310 OP B-1A: XV procedural non-compliance A0633314 CAP O-15 procedural non-compliance A0633316 Failure to document problem with AR A0633317 Failure to meet management expectations A0632519 ZAIP 2-1 Pulsation damper does not hold nitrogen pressure 6.3 Event Trend Reports AR# Event Trend Report Description None Page 18 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 6.4 Comments and Recommendations QV Comments and Recommendations The trending policy referenced in QE Q0012339 and AR A0591435 is still in draft form and has not been issued.

The meaning of the rad tape around sinks to identify them as surface contamination areas is not clearly communicated to new rad workers to the site.

There were two instances were the PWST TOC sample results were out of specification and were not correctly identified, 1/26/05 and 9/15/04.

Observed the surface contamination area sample sink radiological practices were awkward and unpracticed. The technician needed to be prompted to dose rate the sample bottle.

The placing of ASW sampling in CAP A-9 makes it appear less important than CAP A-3 sampling, yet failure to perform sampling on the ASW at the required frequency can render an ASW train and CCW train inoperable.

OP F-5 is out of date. This procedure was last revised in 1992. The procedure revision is tracked in the Chemistry Procedure Change database and is overdue.

There were two chemicals in the secondary laboratory that were expired for greater than five days and two others for less than five days. There was also a container of expired vacuum pump oil.

No formal guidance could be found as requested in AE 2 of A0563958 for the addition of nitrogen pressure to the zinc addition tank pulsation dampener.

OP F-5:III limit for SCW dissolved oxygen (DO) is greater than 2000 ppb when removing nitrates. Currently Chemistry is adding nitrates. The procedure is not clear as to the DO limits when adding nitrates.

There is no glove policy when handling the DX500 or ICP to prevent cross contamination.

The implementation of the reagent and standards database program appears to be spotty due to unfamiliarity and because the training was premature to the implementation of the database program.

6.5 Audit Team Composition Team Member Title Audit Role 1, 2 Robert Clark QV Auditor Audit Team Leader 1,

• Dominic Dale Sr. Nuclear Chemistry Instructor Technical Specialist 1, 2 Jeff Harker RP Supervisor and Qualified Auditor Auditor 1, 2 John Hutcherson QV Auditor Auditor 2 Margo Mosher QV Auditor Auditor

• Conducted a debrief with the Chemistry Section Manager Page 19 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 6.6 Personnel Contacted and Meeting Attendance:

Name Title/Department 1 Dave Oatley Vice President and General Manager of DCPP 4 Paul Roller Director, Operations Services 1, 2 Lance Hopson Manager, Chemistry and Environmental Operations 1, 3 Jeff Gardner Chemistry Senior Engineer 3 Clint Gans Chemistry Engineer 3 Keith Bieze Chemistry Engineer 3 David Chen Chemistry Engineer 3 John Knemeyer Chemistry Engineer 1, 3 Scott Pigeon Chemistry Quality Supervisor 3 Donnie Shippey Chemistry Foreman 3 David Cortina Chemistry Foreman 3 Ken Cortese Chemistry Foreman 3 Rick Wallwork Chemistry Foreman 3 Trevor Rebel Chemistry Foreman 3 Jim Climer C&RP Technician 3 Bruce Ryan C&RP Technician 3 Doug Coleman C&RP Technician 2 Susan Westcott Acting Operations Manager 3 John MacIntyre Operations Foreman 3 Jim Welsch Operations Foreman 3 Frank Lowe Nuclear Operator 3 Dan Berens Nuclear Operator 3 Alex Brown Nuclear Operator 3 Bob Murach Nuclear Operator 3 Mark May Senior Control Operator 3 Henry Clardy Senior Control Operator 3 Jack Trigg Senior Control Operator 3 John McDonald Operations Support Team Foreman 3 Bruce Tripp System Engineer 1, 2, 3 Robert Snyder C&RP Learning Supervisor 3 Ed Davidson C&RP Training Instructor 1, 2 Chuck Belmont Director of Nuclear Quality & Licensing 1, 2 Dave Taggart Manager, Nuclear Quality Verification 1, 2 Bob Prigmore Supervisor, Plant Quality Assurance 3 Chris Joyce Operating Experience Assessment Engineer 3 Rick Schirmer INPO Chemistry Liaison for DCPP 1 Attended the audit entrance February 1, 3 Interviewed or observed during the audit.

2005 2 Attended the audit exit February 18, 2005 4 Attended an individual audit exit Page 20 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 6.7 Source Documents and

References:

DCPP Procedures referenced for this audit:

Type Procedure Rev Procedure Title No. #

UFSAR Chapter 17 15 Quality Assurance Tech. 3.4 3 Reactor Coolant System (RCS)

Spec.

Tech. 3.5 6 Emergency Core Cooling System (ECCS)

Spec.

Tech. 3.6 3 Containment Systems Spec.

Tech. 3.7 6 Plant Systems Spec.

Tech. 3.8 4 Electrical Power Systems Spec.

Tech. 3.9 3 Refueling Operations Spec.

ECG 7.0 1 Reactor Coolant System - Chemistry ECG 8.8 0 Chemical and Volume Control System - Reactivity Control Systems - Borated Water Source - Shutdown ECG 8.9 1 Chemical and Volume Control System - Reactivity Control Systems - Borated Water Source - Operating ECG 11.1 9A Nuclear Sampling System - Post Accident Sampling System PD CY1 1A Chemistry / Radiochemistry IDAP AD10.ID1 4A Storage and Control of Quality Assurance Records IDAP OM4.ID3 7 Assessment of Industry Operating Experience IDAP TQ2.DC5 6 Chemistry and Radiation Protection Technician Training DLAP AD10.DC3 2A Records management system coordinator (RMSC)

DLAP CY1.DC1 5 Analytical Data Processing Responsibilities DLAP CY1.DC3 5B Chemistry Data Reporting to Other Sections DLAP OM12.DC1 29 Relieving the Watch CAP A-1 16 Primary Cycle Sampling Schedule CAP A-2 18 Secondary Cycle Sampling Schedule CAP A-3 15 Technical Specifications Sampling Schedule CAP A-9 26 Auxiliary Systems Schedule CAP A-15 2 Count Room Sample and Analysis Control CAP A-16 4 Control of Chemical Analysis Data Sheets CAP A-21 2 Chemical Laboratory Safety Rules CAP C-70 4 Flash Point by Pensky-Martens Closed Tester for Fuel Oil CAP E-19 13 Plant Vent Radioactive Effluent Sampling CAP O-1 9 Chemical Degas of the RCS CAP O-1A 3 Forced Oxygenation and Shutdown Chemistry of the RCS CAP O-15 9 Preparation of Zinc Acetate Dihydrate for Injection Into the Reactor Coolant System CAP Q-1 15 Chemical Laboratory Quality Control OP B-1A:XV 16/10 Zinc Injection Skid Operation OP F-5 5 Chemical Control Limits OP F-5:I 30 Chemical Control Limits and Action Guidelines for the Primary Systems Page 21 of 22

2005 Chemistry and Radiochemistry Program Audit, EDMS # 043360019 OP F-5:II 29 Chemical Control Limits and Action Guidelines for the Secondary Systems OP F-5:III 17 Chemical Control Limits and Action Guidelines for the Plant Support Systems 2 DCPP Chemical Hygiene Plan CE&O A-01 2 Procedure Use and Adherence Policy CE&O A-02 1 Chemistry Management Observation Policy CE&O A-03 1 Use of Key Behaviors in Chemistry Policy CE&O A-04 0 Chemical Additions Policy CE&O B-02 2 Chemistry Lab Sink Drain Guidance Policy CE&O Writing Chemistry Event Trend Reports (ETRs)

Guide CE&O Chemistry - General Expectations Guide CE&O Chemistry - WINCDMS Expectations Guide Regulatory Documents:

Document Section Title ANSI N45.2.9 Requirements for Collection, Storage, and Maintenance of Quality

- 1974 Assurance Records for Nuclear Power Plants 6.8 Departments and App B Binning Audited Organizations PGPT Chemistry 10CFR50 Appendix B I Organization II Quality Assurance Program V Instructions, Procedures, and Drawings VI Document Control XII Control of Measuring and Test Equipment XVI Corrective Actions XVII Quality Assurance Records XVIII Audits Page 22 of 22

Date: May 22, 2009 File #: EDMS #083500025 To: OPERATIONS SERVICES DIRECTOR From: GENERAL SUPERVISOR, QV - PLANT QA

Subject:

2009 Chemistry and Radiochemistry Programs Audit JIM WELSCH:

Quality Verification conducted an audit of the DCPP Chemistry and Radiochemistry Programs from January 21 through April 23, 2009. This audit fulfilled the requirement for a biennial audit of these programs as specified in FSAR chapter 17.18. The enclosed report identifies the scope and the details of this audit. A summary of the results was also provided to the Chemistry staff at an exit meeting held on April 23, 2009.

QV concluded that the DCPP Chemistry and Radiochemistry Programs were being effectively implemented and that overall performance was satisfactory.

Weaknesses were identified in the control and storage of expired and waste chemicals awaiting transfer to the Hazardous Waste Group. This area needs improvement.

The control of closed cooling water systems chemistry remains a challenge due to the availability of Operations personnel to support chemistry additions in a timely manner.

Significant progress has been made in procedure updating to support Human Performance standards and Error Prevention Tool usage.

QV reviewed 1R15 start-up chemistry control as requested by the Site Vice President. The planning and execution of the Chemistry evolutions involved in returning the plant to service was satisfactory.

If there are additional questions, please contact Gloria Lautt (x4150) or Scott Pigeon (x3471).

Signature on Original GLORIA LAUTT cc: See audit distribution list

AUDIT REPORT QUALITY VERIFICATION DEPARTMENT 2009 Chemistry & Radiochemistry Audit EDMS# 083500025 DISTRIBUTION JRBecker 104/6/601 SJCairns 245 Mkt/488 JTConway 77 Beale/3221 LAHopson 104/3/315 JMWelsch 104/5/504 GALautt 104/3/23C CDTerry NSOC DBPetersen 104/3/328 JLPost (Law) 77 Beale/B30A BCHinds 104/5/523 LEWalter 119/2/246 NRC Resident 104/5/544 Reg Svcs 105/5 Audit Team

January 21 through April 23, 2009 Report Prepared by: Signature on Original 5/22/2009 Scott A. Pigeon Date Audit Team Leader - In - Training Report Reviewed by: Signature on Original 5/22/2009 Gloria Lautt Date Audit Team Leader Report Approved by: Signature on Original 5/22/2009 Gloria Lautt Date General Supervisor, QV - Plant Quality Assurance EDMS #083500025 Page 1 of 14

TABLE OF CONTENTS SCOPE .........................................................................................................................................3 EXECUTIVE

SUMMARY

..............................................................................................................4 AUDIT RESULTS AND CONCLUSIONS ......................................................................................5 Organization, Qualifications, and Documentation .....................................................................5 Conduct of Chemistry Operations .............................................................................................5 Chemistry Lab Practices ...........................................................................................................6 Chemistry Instrumentation ........................................................................................................6 Data Evaluation and Monitoring ................................................................................................6 Chemistry Sampling and Controls .............................................................................................6 Chemical Controls .....................................................................................................................7 Corrective Actions .....................................................................................................................7 Software and Data Control ........................................................................................................7 Self Assessments, Benchmarking, and Operating Experience .................................................7 Performance Indicators .............................................................................................................8 Previous Audit Review ..............................................................................................................8 Observed Evolutions .................................................................................................................9 CORRECTIVE ACTION FOLLOW-UP .......................................................................................10 Non-Quality Problems.................................................................................................................10 EFFECTIVENESS EVALUATION...............................................................................................11 APPENDICES ............................................................................................................................12 SAP Notifications ....................................................................................................................12 Audits Reviewed .....................................................................................................................12 Audit Team Composition .........................................................................................................13 Personnel Contacted and Meeting Attendance .......................................................................13 Procedures and Documents Reviewed ...................................................................................14 Binning ....................................................................................................................................14 EDMS #083500025 Page 2 of 14

SCOPE This biennial audit was scheduled on the Quality Verification Group Master Internal & External Audit & Review Schedule dated March 5, 2008.

This audit was performed to verify compliance with the requirements for implementing the Chemistry and Radiochemistry Programs.

The audited areas include:

• Organization, Qualifications and Documentation

• Conduct of Chemistry Operations

• Chemistry Lab Practices

• Chemistry Instrumentation

• Data Evaluation and Monitoring

• Chemistry Sampling and Controls

• Chemical Control

• Corrective Actions

• Software and Data Control

• Self Assessments, Benchmarking and Operating Experience

• Performance Indicators

• Previous Audit Review

• Observed Evolutions A checklist was prepared for each area based on regulatory and procedural requirements. The audit team utilized these checklists as guidance for performance-based observations of activities, document reviews, and interviews with plant personnel.

Detailed checklists of the audited areas are available upon request.

EDMS #083500025 Page 3 of 14

EXECUTIVE

SUMMARY

The audit team concluded that the Chemistry and Radiochemistry Programs were effectively implemented during the audited period. Chemical Control was determined to need improvement.

There were a total of 14 SAP Notifications (SAPN) created during this audit which consisted of two Level 2, seven Level 3, and five Level 5 severity SAPNs. There were no audit findings.

In the area of Chemical Control, five notifications were created to document deficiencies in expired chemical and waste chemical handling and storage.

Recent actions taken to address biofouling of the closed cooling water systems have resulted in noticeable improvements to the Intake Cooling Water and Service Cooling Water systems.

However, continued monitoring is necessary to determine the effectiveness of these actions.

Adjustments to closed cooling water system chemistry, chemical additions and feed-and-bleeds, continues to be a burden for Operations personnel. This burden impacts the effectiveness of the chemical additions.

Significant progress has been achieved incorporating human performance standards and error prevention tools into Chemistry procedures. However, the required annotations for Procedure Commitment Database commitments are not included in many of these updated procedures.

The significance of these issues did not detract from the overall effectiveness of the Chemistry and Radiochemistry programs.

QV reviewed 1R15 start-up chemistry control as requested by the Site Vice President. The planning and execution of the Chemistry evolutions involved in returning the plant to service was satisfactory.

EDMS #083500025 Page 4 of 14

AUDIT RESULTS AND CONCLUSIONS Organization, Qualifications, and Documentation Chemistry section personnel have been in their positions for many years and are very familiar with their responsibilities. Additionally, QV identified that the entire Chemistry engineering staff is eligible, or will be eligible, for retirement within the next 4 years. While there is some sharing of position responsibilities during vacations and business trips there is no established plan to ensure duties can be fulfilled long term. QV recommends Chemistry designate alternate contacts for each management area of responsibility and provide cross training to ensure continuity of established programs. A candidate for an engineering position that has been vacant for 2 years is currently being sought.

Chemistry has made excellent progress on updating procedures that contain instruction for manipulating plant components. All of the sampling procedures have been updated to the format that allows formal placekeeping. The manager's expectation that placekeeping be used for all plant component manipulations is evident during field observations. Necessary changes are made in a timely manner and the current procedures are available for technician use.

Eight of ten Chemistry procedures that had major revisions performed since July of 2005 did not identify the commitments of the Procedure Commitment Database (PCD) as described in AD1.ID1, "Nuclear Generation Procedure Writer's Manual" (SAPN 50211506). A review of five of the eight procedures revealed that the commitment requirements have been incorporated into the procedures, but are not annotated as required.

QV identified 2 procedures that had On-The-Spot-Changes (OTSC) that were not incorporated in a timely manner. A Chemistry engineer had requested that Procedure Services delay the incorporation of the OTSC until an in-progress revision was completed. This resulted in the OTSC being in the field for over 6 months before being identified by QV (SAPN 50210821).

Administrative qualifications (Knowledgeable Electrical Worker and Foreign Material Exclusion Worker) that do not affect access to the Protected Area but are related to the work being performed by Chemistry personnel have not always been treated with the same rigor as worker qualifications. Four Chemistry foremen and two Chemistry technicians had expired administrative qualifications during this audit period.

This area is satisfactory.

Conduct of Chemistry Operations QV observed shift turnovers, tailboards and in plant evolutions. All management expectations were performed by all observed parties.

QV found three instances of outdated revision datasheets in the chemistry labs in the ready for use slots (SAPN 50213652). The datasheets were disposed of and replaced with current revisions. There is no indication that the outdated datasheets were used for plant monitoring.

Proper use of the human performance tools was evident during all of the observed evolutions.

This area is satisfactory.

EDMS #083500025 Page 5 of 14

Chemistry Lab Practices The chemistry labs are adequate and generally well maintained; however, there is a need to update the lab infrastructure to address process functions and ergonomics. All required safety equipment is readily available and inspected regularly.

Exhaust hoods have chemicals and equipment stored inside contrary to guidance in the Chemical Hygiene Plan (CHP) (SAPN 50227437 and 50227436). The position of equipment that must remain stored inside the exhaust vent hoods for safety reasons has been adjusted to minimize the adverse effect identified in the CHP. Chemical storage inside the exhaust hoods has been minimized and focused on the overall safety to Chemistry personnel.

This area is satisfactory.

Chemistry Instrumentation Review of instruments and calibration documents indicate that the proper care and maintenance of the instruments is being implemented.

This area is satisfactory.

Data Evaluation and Monitoring Review of instruments and documentation indicate that data is being properly evaluated and all required parameters are being monitored as required by implementing procedures.

The peer auditor found it difficult to verify compliance with chemistry control guidance since the sample schedules and the required parameters are located in different sets of procedures. The auditor recommended combining the information into one location to facilitate verification.

This area is satisfactory.

Chemistry Sampling and Controls QV reviewed the chemistry controls and sampling requirements of plant primary, secondary, and auxiliary systems. All requirements are met with the current chemistry strategies, and sampling schedules effectively monitor chemistry parameters.

Changes to chemistry control to support recent plant changes such as Steam Generator Replacement were effectively implemented in a timely manner.

Recent actions taken to address biofouling of the closed cooling water systems have resulted in noticeable improvements to the Intake Cooling Water (ICW) and Service Cooling Water (SCW) systems. ICW head tank cleanliness and SCW corrosion monitoring coupons performance indicate these actions have had the desired effect. However, continued system operation and monitoring is necessary to verify the effectiveness of these actions.

Closed cooling water chemistry control continues to require an increased number of chemical adjustments to maintain chemistry parameters within specifications. This increased attention to these systems involves Operations, as well as Chemistry personnel. Operations is not always able to support chemical additions in the time frame requested by the Chemistry engineer. This results in chemicals being added in rapid succession which can significantly limit the desired effects on the system and reduces the time before the next add will be needed.

This area is satisfactory.

EDMS #083500025 Page 6 of 14

Chemical Controls QV identified six expired chemicals that were still in the labs and stored with the in-use chemicals (SAPN 50213394). Chemistry management expectation is to remove expired chemicals daily to prevent inadvertent use.

In addition, some containers of liquid chemicals are not routinely stored inside secondary containments as required by the CHP (SAPN 50227438).

Waste chemicals were not segregated and stored in accordance with chemical hazard storage requirements in the labs while awaiting delivery to the Hazardous Waste group for disposal (SAPN 50206220 and 50216621).

This area needs improvement.

Corrective Actions The threshold at which Chemistry personnel enter problems into the Corrective Action Program (CAP) has been lowered with encouragement from the Chemistry Manager. As a result, the development and implementation of corrective actions to address identified issues is timelier.

This area is satisfactory.

Software and Data Control The computer applications being used by Chemistry were reviewed and verified to be listed in the Application index. They have been appropriately screened for Software Quality Assurance (SQA) Plans and proper review has been completed. The SQA plans were verified for those applications requiring them.

QV identified vulnerabilities with the ChemCalc Excel spreadsheet (SAPN 50211144 and 50211520) and actions were taken quickly to address these problems.

This area is satisfactory.

Self Assessments, Benchmarking, and Operating Experience Chemistry performed two Quick Hit Self Assessments (QHSA) and two Benchmarking activities during this audit period. Two other QHSAs were scheduled for the end of the period but not completed and an informal Benchmarking to assess the applicability of the "Smart Chemworks" program to DCPP is planned for later in 2009.

Three of the four performed activities were reactionary in nature, included investigative actions for resolving long term process problems. The fourth was a proactive Benchmarking activity to investigate the advantages of Reduced Inventory Forced Oxidation of the RCS during outages.

A Chemistry Formal Benchmarking activity, Condensate Polisher Benchmarking, scheduled for 2009 has been dropped from the schedule by the Self Assessment Review Board to allow closer alignment with plant priorities.

EDMS #083500025 Page 7 of 14

Chemistry has not performed a formal Self Assessment in 5 years but has performed 11 QHSAs during that period. Therefore Chemistry has not had the benefit of having outside organizations provide valuable insight as they would have with a formal Self Assessment.

Chemistry has incorporated plant and industry Operating Experience (OE) into formal tailboards for complex and infrequently performed evolutions, and is included during routine tailboards. In addition, Chemistry personnel utilize OE in accordance with the site standards.

This area is satisfactory.

Performance Indicators Chemistry uses internal Performance Indicators (PI) to drive toward excellence as well as participating in Institute of Nuclear Operations (INPO) and Nuclear Regulatory Commission (NRC) mandated PIs.

Chemistry meets regulatory requirements by submitting information regarding Reactor Coolant System Specific Activity to the NRC in accordance with AWP O-001, "NRC Performance Indicators: Reactor Coolant System (RCS) Specific Activity."

Participation in the INPO PIs allows DCPP to monitor and compare chemistry performance to other United States nuclear power plants. Combining the INPO PIs with the more extensive internal PIs, some of which contain more restrictive limits, enables Chemistry to more closely monitor performance and drive to excellence.

The INPO Chemistry Effectiveness Index (CEI) is a PI based on an 18-month rolling composite of several plant chemistry parameters and allows comparison to the nuclear industry.

The Unit 1 INPO CEI is in the top quartile and represents excellent performance.

The Unit 2 INPO CEI was in the top quartile prior to 2R14 and entered the second quartile due to startup feedwater iron concentrations following 2R14. The 18 month CEI was returning to the first quartile when 2T15, the C-Phase Main Bank Transformer Outage, occurred. Upon startup from 2T15 feedwater iron concentration drove the CEI into the fourth quartile (SAPN 50076930).

An Apparent Cause Evaluation was performed and identified the lack of a well defined decision making process for unplanned outages regarding control of Feedwater iron transport.

Procedure changes have been completed and should prevent a similar problem when the next unplanned outage occurs. Although the current value for feedwater iron is in the first quartile, the recovery of the 18-month CEI from the effects of 2T15 high iron transport is not expected until April 2010.

This area is satisfactory.

Previous Audit Review A review of the previous Chemistry audit (EDMS # 063480001) did not identify any repeat findings during the conduct of this audit. Three continuing minor issues were identified in both audits but actions are being taken to address these problems and have not yet been completed.

Progress on these issues is satisfactory.

This area is satisfactory.

EDMS #083500025 Page 8 of 14

Observed Evolutions QV observed system sampling and chemical addition evolutions during this audit period.

Observations were entered into the Management Observation Program.

Chemistry personnel followed the expected behaviors in each of these evolutions.

Four SAP Notifications were written during these observed evolutions that were assigned to other organizations and had no impact on Chemistry. These SAPNs are:

50206538 - Package Boiler 0-1 Signage Evaluation 50208791 - U1 Plant Vent SISIP Concern 50208792 - U2 Plant Vent SISIP Concern 50213118 - FME during Breaches for Chem Adds This area is satisfactory.

EDMS #083500025 Page 9 of 14

CORRECTIVE ACTION FOLLOW-UP All of the corrective actions for the three quality problem and ten non-quality problem Action Requests that were initiated in the previous audit (EDMS # 063480001) have been completed and closed. The actions taken were both timely and effective.

The Action Requests reviewed include:

Quality Problems AR # Title A0689281 Ineffective Implementation of Review of New Chemistry Processes A0689284 C&RP Technician Hired Without Meeting ANSI Qualifications A0689288 ANSI Quals for Two C&RP Techs Not Correctly Evaluated Non-Quality Problems AR# Title A0687660 Poor Safety Practice Observed During Unit 1 SCW Chemical Addition A0689263 QC Checks Not Performed for Tennelec A0689265 Chemistry Sampling and Data Review Process is Inefficient A0689270 Change Management Was Not Instituted for Procedure Place-Keeping A0689273 System Engineering Guidance to Utilize System Chemistry Parameters A0689276 Chemical Control of Closed Cooling Systems is a Burden on the Plant A0689279 Chemistry SA and BM Do Not Focus on Program Improvements A0689291 Closing ARs to a Promise or an Action that is Not Completed A0689292 Failure to Correct Safety Issues in a Timely Manner A0689754 Consumable Materials List is Not Up To Date EDMS #083500025 Page 10 of 14

EFFECTIVENESS EVALUATION The audit team concluded that the Chemistry and Radiochemistry Programs were effectively implemented during the audit period. There were no audit findings. However, the area of Chemical Control needs improvement.

EDMS #083500025 Page 11 of 14

APPENDICES SAP Notifications SAPN# Issue 50205908 Chemistry Lab DFO sample waste storage 50205909 Hazardous Waste storage in Chem Lab 50208006 U1 Plant Vent SISIP concern 50208009 U2 Plant Vent SISIP concern 50210821 OTSCs untimely incorporation 50210938 PCD commitments not referenced in Chem procedures 50211141 ChemCalc worksheets out of revision 50211144 ChemCalc needs instructions for changes 50212939 Outdated Chem datasheets in use in lab 50213118 FME during breaches for chem adds 50213394 Audit Recommendation - Inspect lab for expired chem 50221975 Hexane extractors position in hood 50221976 Chemicals stored in vent hoods 50221977 Secondary containment for chems in glass Audits Reviewed EDMS Number Audit Title 063480001 2007 Chemistry and Radiochemistry Audit EDMS #083500025 Page 12 of 14

Audit Team Composition Team Member Audit Role Gloria Lautt Audit Team Leader Scott A. Pigeon Audit Team Leader - in Training Ben Neufeld Technical Specialist Personnel Contacted and Meeting Attendance Note Name Title/Department 1, 2 Brad Hinds Chemistry and Environmental Operations Manager 1, 2 Jeffrey Gardner Senior Chemistry Engineer 1, 2 Clint Gans Chemistry Engineer 1, 2 David Cortina Chemistry Foreman 1 John Knemeyer Chemistry Engineer 1 Keith Bieze Chemistry Engineer 2 Ken Cortese Chemistry Foreman 2 Eric Wessel Chemistry Engineer John Cianci Chemistry Foreman Rick Wallwork Chemistry Foreman Rich Dong Chemistry Foreman Dina Brazil Chemistry Clerk Jane Sutton Chemistry Procedure Writer Chance Siri Digital Systems Engineer Cory Sloan C&RP Technician Joe Iliff C&RP Technician Marty Sanders C&RP Technician Kent Grasmick C&RP Technician Patrick Likes Nuclear Operator Josh Meade Nuclear Operator Notes:

1 Attended audit entrance on January 21, 2009 2 Attended audit exit on April 23, 2009 EDMS #083500025 Page 13 of 14

Procedures and Documents Reviewed Procedures Rev. Title CY1 2 Chemistry/Radiochemistry CY1.ID2 1 Closed Cooling Water Chemistry Program CY1.DC1 5 Analytical Data Processing Responsibilities CY1.DC3 6B Chemistry Data Reporting to Other Sections Control of Material and Equipment Used for Analysis CY1.DC4 4 for Chemistry and Radiochemistry Program CAP A-1 20 Primary Sampling and Analysis Schedule CAP A-2 21 Secondary Cycle Sampling Schedule CAP A-3 16 Technical Specifications Sampling Schedule Laboratory Instrumentation Calibration and routine CAP A-4 16 Maintenance Schedule CAP A-9 30 Auxiliary Systems Sampling Schedule In-Line Instrument Calibration, Correlation, and CAP A-12 13 Routine Maintenance Schedule CAP A-15 2 Count Room Sample and Analysis Control CAP A-16 6 Control of Chemical Analysis Data Sheets Monitoring of Radionuclide Indicators During Routine CAP A-20 2 Operation CAP A-21 2 Chemistry Laboratory Safety Rules Guidelines for New and/or Unusual Chemistry CAP A-22 0 Processes Waste Gas Analyzer Sampling - System Operable by CAP E-2:IIIB 1 Not Running CAP O-1 13 Chemical Degas and Forced Oxidation of the RCS CAP O-16 3 Reactor Coolant System Chemical Additions CAP Q-1 21 Chemical Laboratory Quality Control CAP Q-2 2 Guidelines for Analytical Data Review 3 DCPP Chemical Hygiene Plan Binning 10CFR50 Appendix B, Criteria I, II, IV, V, VI, VII, VII, XII, XV, Topics XVI, XVII, XVIII Audited DCPP Chemistry Organization EDMS #083500025 Page 14 of 14