ML20141B672

Final Rept Configuration Control Self Assessment

ML20141B672
Person / Time
Site:	Fort Calhoun
Issue date:	04/11/1997
From:	Foley J, Geschwender J, Hyde K OMAHA PUBLIC POWER DISTRICT
To:
Shared Package
ML20141B670	List: ML20141B663 ML20141B672
References
LIC-97-085, LIC-97-85, NUDOCS 9705150376
Download: ML20141B672 (50)

Text

{{#Wiki_filter:., i a i l i LIC-97-085 Attachment Y Configuration Control Self Assessment 4 Final Report W April 11,1997 t 1 8 1 4 f f f .o 9705150376 970508 E PDR ADOCK 05000205I G PDR l,,

t Configuration Control Self Assessment Final Report, April 11,1997 Prepared by the CCSA Team: /L - _t c /g/ r J. R. Geschwender' . th J. J. Foley ' ',.;M 7 / i K. C. Hyde f/' M4 h /l.6. /I73 & A-S. K. Kaira D. R. Taylor Cree f II .A 0 L. P. Hopkins tir-E. J. Bernal J. K. Gasper hk[deM ii / H. J. Faulhaber y M. R. Core M /"A R. L. Phelps Accepted by: < l,hM S. K. Gambhir .l I

- ~ .. - ~. - .) l i INDEX j b 'Section ' Page.No. . 1. Executive S ummary.................................................... 2 o i 2. ' Purpose and Scope of the CCS A......................................... 3 i , 3. Review of Current Procedures (Task I of the Plan)........................... 4 ~4. CCSA Condition Report Review (Tasks 2 and 3 of the Plan).................... 8 - 1 ' 4.1 Methodology for Analysis.......................................... 8 '{ '.2 ' ' Analysis Resul ts..................................................~ 12 4 4.3 US AR Implications............................................. 13 4.4 Labeling Errors................................................. 14 Walkdowns for Configuration Issues (Task 4 of the P!.m)...................... 16 5. 5.1 S pecific Walkdo wns............................................. 16 5.2 - Operator Rounds................................................ 17 1 ' 6. Industry Experience Review (Task 5 of the Plan)............................. 18 o 7. Conc l usio ns.......................................................... 21 1 8. Recommendations..................................................... 25 ' Appendix A. Configuration Control Self Assessment (CCSA) Plan 13. . Description of the CR Data 'I i 1-April 11,1997 j . O.,' 4,.

SECTION 1 EXECUTIVE

SUMMARY

This Configuration Control Self Assessment (CCSA) was performed to investigate whether there is a current problem with unauthorized configuration changes at Fort Calhoun Station (FCS). A Team of managers and highly experienced individuals from the functional areas that could potentially affect plant configuration was assembled and an assessment plan prepared. The experience level of the Team was essential for the task of reviewing Condition Reports (CRs), screening for configuration issues, and assigning selected CRs to categories appropriate for analyses. These assessment categories were then malyzed to provide insights into the extent and causes for the configuration issues. De methodology used to analyze the CR data involved separation into types of configuration change (authorized or unauthorized), process involved (modification or Maintenance Work Document), and document type involved, and as well as consideration of when the event occurred. Further breakdowns using specifically designed assessment categories provided insights on human . performance factors. The results demonstrated that events occurring prior to September 1995 dominated the data. The number of current, unauthorized configuration changes was a small part of the CRs. Most of these items involved a part being replaced with a similar but not identical part, without required documentation. It is important to note that the results confirmed very few safety significant configuration control issues being experienced at Fmt Calhoun Station. A further review of the category of Unauthorized Configuration Change CRs indicated that some of the identified issues affected the configuration as described in the Updated Safety Analysis Report (USAR). Analysis of walkdown information and observations from operator rounds supported the overall conclusions of the CCSA. A review of procedures controlling configuration changes and the procedure goveming Maintenance Work Documents (MWDs) demonstrated that the currently used processes appear to be adequate fbr properly controlling plant changes. Nevertheless, several areas for further enhancements to processes and procedures were recommended. The team reviewed oflessons from current industry experience during analysis of the data. In summary, although heightened awareness for the existence of potential configuration issues may result in continued identification of CRs, the overall causes for the CRs are expected to remain consistent with the results of this self assessment. Improvements in document processing for modifications and maintenance will help reduce configuration discrepancies. The current plant emphasis on overall high quality human performance is also important. I i 2 April 11,1997

SECTION 2 PURPOSE The purpose of the Configuration Control Self Assessment (CCSA) was to evaluate' the adequacy of current procedures and processes for maintaining control of the Fort Calhoun Station configuration, and to evaluate recently identified configuration contro! discrepancies to determine whether they indicate a need for programmatic changes. i SCOPE The scope of the CCSA included: Review of current procedures associated with configuration control. Review of Condition Reports (CRs) to identify configuration control discrepancies. Analysis of selected CRs involving configuration control discrepancies. Field observations. . Review of applicable industry experience. The plan used to conduct this assessment is provided in Appendix A, including a list identifying the members of the multi-discipline assessment Tearr.. Additional details regarding the methodology used during the assessment are provided in subsequent sections of this report. i 3 April 11,1997

P 4 SECTION 3 REVIEW OF CUPhNT PROCEDURES (Task I of the Plan) This task consisted of reviewing procedures with potential to alter plant configuration. These included procedures mntrolling modifications, maintenance, and processes for identification, assigmnent, and resolution of configuration control issues. In conjunction with this effort, configuration-related Ccndition Reports (CRs) were reviewed using the methodology in Section 4.1 for indications of which processes, or stages thereof, were most likely to allow configuration control deficiencies. Procedures were selected based on their purpose of controlling configuration change processes, or for providing mechanisms for identifying and correcting configuration discrepancies. Proce-dure not directly affecting configuration control processes were considered to be outside the scope of this assessment. The following procedures were reviewed: PED-QP-2 Configuration Change Control, Rev. 24 PED-gel-3 Preparation of Design Change Packages, Rev.18 PED-gel-29 Facility Change Evaluation, Rev. 4 PED-gel-35 Preparation of EARS for Minor Configuration Changes / Replacements, Rev. 0 ) Pl!D-gel-51 Document Change Evaluations; Rev. O PED-gel-56 Configuration Change Closecut, Rev. 4 S0-G-21 Modification Control, Rev. 62 S0-M-101 Maintenance Wei'c Control, Rev. 42 S 0-0-25 Teniporary Modification Control, Rev.49 S0-R-2 Condition Reporting and Corrective Action, Rev. 3 i The purpose of the review was to determine if current procedures contain weaknesses which might allow the plant configuration to be changed without appropriate design reviews or without procedures, drawings, and other documents being updated to reflect the change, and to assure that the plant remained in compliance with its design and licensing bases. The following questions were considered during this review: Do current procedures address all processes by which plant configuration may be affected? Do current processes contain sufficient barriers to prevent configuration discrepancies? Arc activities with potential to alter plant configuration reviewed by appropriate person-nel to assure that configuration discrepancies do not occur? Do appropriate personnel review configuration discrepancies to assure proper technical resolution? is there a correlation between the procedural weaknesses and the Condition Reports anal-ysis results described in Section 4.2? 4 April 11,1997

Procedure Review Commerg S0-G-21 comments: This procedure allows a modification to be declared operational prior to all affected documents being updated. Documents necessary to operate the modified system safely are required to be updated (e.g. Control Room drawings, Operating Instructions) prior to declaring a modification 1 operable. However, other documents refice ag plant configuration are not required to be updated until modification completion, whi a is often several months after the modification be-came operable. (Note: Temporary Modification Control procedure S0-0-25 does not allow a TM to be put in effect until all affected drawings and procedures are updated.) Conversely, no clear restrictions exist to prevent a procedure, drawing, or other document or da-tabase revision from being submitted, processed, and updated prior to the configuration change being made operable. This is even more likely to occur for modifications which are not completed in a one-time consolidated effort. PED-gel-56 comments: Currently, drawings are reviewed by Design Engineering prior to being submitted to the drafting department for update. The drafting department has an intemal peer review process for drawing revisions. Ilowever, the drawing receives no Engineering review prior to being issued. (Note: This post-drafling Engineering review had formerly been eliminated to reduce what was consid-cred to be an administrative burden on the engineers.) Revisions to other configuration-related documents (e.g., FACTS, ELDL, Breaker Program, Fuse Control Program) are submitted prior to modification closecut. Though other procedural pro-cesses exist to control these documents once the updates are submitted, there is no " closure" in the modification process whereby Engineering verifies the changes have been completed as sub-mitted. gel-56 does not provide any requirements for updating CIIAMPS. Modifications and FC ECNs do require the CliAMPS Coordinator to sign the FC-1137 form. The SRI ECN gel-60 form requires CHAMPS updates to be submitted, but provides no verification that the CHAMPS data-base is accurately updated. S0-M-101 comments: Step 5.1.1b item 29 allows replacec.:nt or fabrication of non-CQE mounting hardware under tool pouch maintenance. This appears to create the potential for compromising the affected compo-nent's or structure's seismic classification. PED-QP-2 comments: Configuration change processes have recently undergone some refinements. These were directed towards assuring compliance with design and licensing bases while eliminating non-value added engineering work when personnel and nuclear safety and plant reliability were clearly unaffected. This resulted in procedural definitions for " nonsignificant configuration changes" (PED-QP-2, S0-M-101) and " minor configuration change / replacement"(PED-QP-2, PED-gel-35). As these 5 April 11,1997.

l processes are still relatively new, it is not practical to draw conclusions regarding their effective-ness in preventing configuration discrepancies. Ilowever, these procedures appear to implement suflicient controls to assure that configuration discrepancies are not created, and that preexisting discrepancies are promptly and thoroughly documented and resolved. S0-R-2 comments: The Condition Reporting procedure, SO-R-2, does not clearly require any configuration discrep-ancy to be identified on a CR. SO-R-2 Attachment 1," Guidelines for Identification of Condi-tions to be Reported" describes configuration related issues under Operational Non-conformances. However, the examples are limited to CQE, LCQE, and Fire Protection equipment. Other types of configuration discrepancies (e.g., radioactive waste disposal) may be interpreted as outside the boundary of a condition requiring a CR. This may, in turn, result in a known configuration discrepancy bypassing the PED-QP-2 configuration control process, and either remain uncorrected or be resolved via a process which bypasses Engineering review. It is important that configuration control issues be assigned to Engineering to assure that appropriate reviews and documents are generated to resolve the issue. Engineering management t is included in the Condition Review Group (CRG); however, CRG quorum requirements may i allow a valid CRG quorum (6 members including alternates) without Engineering management representation. (This may arguably have no effect on whether a configuration control issue is assigned to Engineering.) Similarly, if a configuration control issue is designated a Level 5 or Level 6 CR, then an Owner may not necessarily be assigned, in which case the issue would bypass PED-QP-2 screening. Condition Renort Cause Comoarison The weaknesses described above were compared with the Condition Report analysis results described in Section 4.2. One-hundred and ninety-seven (197) CRs were evaluated by process (reference Charts 111-1 and 111-2). It was not practical to quantitatively break down each CR by the process involved (i.e., modification process, maintenance process) and tic each CR to a specific procedural weakness. Ilowever, the analysis results (see Section 4.2) do indicate qualitative correlations between the procedural weaknesses and the CR analysis results. For example, several CRs involve coordination / timing issues. This corresponds to the S0-G-21 and PED-gel-56 weaknesses, which involve modifications being made operable prior to all affected documents being updated, or conversely, to documents being submitted and issued prior to a modification being made operable. This issue may require additional management attention and/or procedural controls. Ilowever, the relatively small number and low severity of these occurrences may indicate that excessive procedural controls would impose high costs with diminishing returns in temis ofimproved nuclear safety, personnel safety, or plant reliability. 6 April 11,1997

Another correlation involved document updating errors. This is not indicative of procedural weaknesses, but is characterized as a human performance issue. However, newer, computerized document search tools such as the EDITS database and the ISYS document search engine significantly reduce the probability of documents being overlooked. The number and nature of CRs reviewed during this assessment indicate that Fort Calhoun Station personnel are not hesitant to identify any kind of configuration discrepancy in the CR system. Though some procedural clarifications may be appropriate, the perceived SO-R-2 weakness described above is not validated by the data. I i 1 I i 7 April 11,1997 i - i

SECTION 4 CCSA CONDITION REPORT REVIEW (Tasks 2 and 3 of the Plan) 4.1 Methodology for Analysis The Configuration Control Self Assessment (CCSA) was structured to provide not only a review of the scope and type of configuration issues recently being experienced at Fort Calhoun Station (FCS) but to provide insight to the extent, causes and contributing factors for the issues, which in turn would generate recommendations for improvement. In order to ensure a reasonably comprehensive base ofinformation to use in the assessment, the Condition Report (CR) System was used. Condition Report System data was screened to select CRs for further analysis. In accordance with the CCSA Plan in Appendix A, the selected CRs were further analyzed to establish categories that are descriptive of both why the configuration issue occurred and to what causal element it could be attributed. In this phase of the self assessment, the Team make-up was essential to proper categorization. The Team consisted of members experienced in areas associated with plant configuration control. The categories into which the data was separated were designed specifically for this assessment. The assignment of CRs to these categories involved thejudgement and experience of the Team members. Selection of Condition Reports As stated above, the CR system was used to assess the nature of recent configuration control discrepancies identified at Fort Calhoun Station. Potential configuration control issues were identified from CRs initiated between September 1995 and March 31,1997 (approximately 2,400 CRs). The potential configuration control CRs included CRs that had been assigned an " event code" of" CON,"(indicating an event involving plant configuration control as determined by the Corrective Action Group) as well as additional CRs that had previously been identified by a Design Engineering Nuclear (DEN) effort to track and trend configuration control issues. The CRs in this preliminary set were then individually checked against the following scoping criterion. The Condition Reports (CRs) included in the scope of the assessment are those that involve a discrepancy between: - the as-built configuration, and - the design / configuration documentation or equipment labels. i A set of 222 " Configuration Control" CRs were selected for further review based on this scoping criterion. Further analysis of these CRs by the Team established appropriate categories to assess the nature of the identified configuration discrepancies. 8 April 11,1997

l Classification by Type of Configuration Issue The purpose of categorizing the " Type" of CRs was to distinguish CRs involving configuration changes that were not properly authorized, from CRs involving authorized changes that were not properly documented. CRs involving only a labeling discrepancy were assigned to a separate category. The definitions used for the " Type" categories were: UNAUTHORIZED - CRs involving a discrepancy between the approved and/or analyzed configuration (at the time ofinstallation) and the as-built configuration. Examples ofissues that were assigned to this category include: improper installation; configuration change implemented without formal controls or prior to fonnal approval; original installation issues requiring a configuration change for resolution. AUTHORIZED - CRs involving a discrepancy between design / configuration documentation and the as-built configuration. Examples ofissues that were assigned to this category include: failure to update a drawing, pme:dre or other design / configuration document; original installation issues not requiring a configuration change for resolution. LABELING - CRs involving only a labeling discrepancy. Examples ofissues that were assigned to this category include: wrong label, missing label. Classification by Process Involved The purpose of categorizing CRs in this way was to determine what process was being used at the time of the error. This provides insight into where improvements are needed. The CRs were divided into the following categories: FACILITY CilANGE ENGINEERING CilANGE NOTICE (FC-ECN)- Error occurr:d at any stage during facility change design, review, installation, document updating or closing process. MODIFICATION REQUEST (MR) - Error occurred at any stage during modificatio a design, review, installation, document updating or closing process. SUBSTITUTE REPLACEMENT ITEM ENGINEERING CHANGE NOTICE (SRI-ECN)- Error occurred at any stage during SRI ECN design, review, installation, document updating or closing process. MAINTENANCE WORK DOCUMENT (MWD)- Error occurred during MWD process. ORIGINAL INSTALLATION - Error occurred during the original design or installation. 9 April 11,1997

OTHER - Error occurr:d during Project 1991, Labeling Project. or any other design project. e NA - Not applicable to any of the previously established categories. Classification by Docurnent Tyne Involved The purpose of categorizing CRs in this way was to determine the types of documents involved in the discrepancy. The documents are categorized as follows: DESIGN DOCUMENT - Error occurred in updates of Design Basis Documents (DBDs), Vendor Manuals, or the Critical Quality Equipment (CQE) List. DRAWING - Error occurred in the drawing during the preparing, installing or updating phase. LABEL - Error occurred during labeling phase. PROCEDURE - Error occurred in any Operating or Installation / Test procedure during the preparing, installing or updating phase. NA - Neoc of the categories above apply. Cassificatica?y Date of Occurrence The purpose of this classification was to distinguish discrepancies resulting from " current" activities from discrepancies that occurred in the past but have just recently been discovered. September 1995 was arbitrarily selected as the threshold between " current" and "past" discrepancies. This date places discrepancies occurring within about the last 18 months (the approximate duration of an operating cycle) in the " current" category, and also separates events that occurred before implementation of the CR system from those that have occurred since. This classification is therefore defined as: BEFORE 9/95 - Available information indicates that the discrepancy occurred before September 1995. AFTER 9/95 - Available information indicates that the discrepancy occurred after Septem'x, 1995. " Assessment" Categories in order to better understand the causes of the CR issues, the Team developed the concept of assigning " Assessment" categories to the CRs. These categories were developed after initial review of the CRs in order to allow insights from the initial data review to be used to ensure meaningful classifications. This categorization was used to determine where in the process the loss of configuration control occurred. If multiple parts of the process failed, the event was coded to the point in the process that most likely caused the problem. 10 April 11,1997

O Eight main categories were defined. The main categories were subdivided, as appropriate, to better indicate where in the process the problem occurred. The following are the main categories with a brief description: DOCUMENT UPDATE - This category was used when the field condition was evaluated to be correct and a document such as a procedure, drrwing, etc. was not updated properly. Included in this category were Design Change Package (DCP) preparation o closecut deficiencies, DCP As Built deficiency, or a drafting error. filSTORICAL - This category was used when no documentation (work document or DCP) of when the event took place was available. Therefore, it was either an original construction event or an undocumented configuration change which had taken place in the history of the plant. IMPROPER WORK PRACTICE - This category was used when a maintenance or modification installation error took place. This category also included improper use of parts by maintenance, poor maintenance work instructions, etc. SRI ECN - This category was used when an SRI ECN was involved. Included in this category are design deficiencies resulting in incorrect work or parts, procedures not followed, etc. PROGRAM - This category was used to document an event associated with ongoing FCS programs such as the CQE list, Probabilistic Risk Assessment (PRA), Design Basis, etc. COORDINATIONfflMING - This category was used when the DCP closcout process was too long or concurrent DCPs associated with the same drawing resulted in documentation being incorrect when needed by the plant. It was also used when a procedure was approved prior to the DCP being installed. NOT AWARE OF CONFIGURATION CIIANGE - This was used when the personnel j involved in the loss of configuration where not aware that they had created a problem. OTilER - This category was used when none of the other categories fit. Additional Innuts to the Analysis In addition to the classifications. assigned by the Team, categories assigned to CRs as a part of the corrective action process were also considered Specifically, the CR Significance Level and CR cause code information were used. Condition Reports are assigned a Significance Level of 1 through 6 by the Condition Review Group (CRG). Of these, CRs classified as Level 1 or Level 2 are considered "Significant." One or more cause codes are assigned to each CR by the Corrective Action Group (CAG) following closure of a CR. As a result, CRs that have not been closed (including many of the " Configuration Control" CRs being reviewed by the team) do not have assigned cause codes. I1 April 11,1997

l I 4.2 Analysis Results in order to interpret the data, various graphical and tabular presentations of the data were - prepared and reviewed (See Appendix B). The "Qafiguration Control" CRs were first reviewed .to assign each to a Type category. P'the 222 " Configuration Control" CRs,38 percent (85 CRs) were categorized as "Unauthr,rized, 51 percent (112 CRs) were categorized as " Authorized," and 1 I percent (25 CRs) were categorized as " Labeling." Subsequent review efTorts emphrimd the " Unauthorized" and the " Authorized" Configuation Chu ges (i.e.,197 CRs). Labeling CRs l were also considered, but they were analyzed separately (See lection 4.3). i In further analysis of the 197 Authorized /Unauiluiud CF ihms, the following results were j notable: A. Condition Report Level / Significance Very few of the CRs were classified as'"Significant"(i.e., Level 1 or 2) by the Condition Review Group. Only 6 of the 197 CRs were determined to be Level 1 or 2 items (Charts ill-3 and 111-4). B. Before/After September 1995 Most of the CRs involve conditions that occurred before September 1995. Approximately 86% (170 CRs) were classified as "Before 9/95," leaving only about 14% (27 CRs) in the "After 9/95" category.

The number of CRs generated in each of these categories was plotted versus time to assess the rate ofidentification of configuration issues (Chart 11-1). This plot shows that more configuration control CRs were generated in March 1997 then in any of the previous 18 months (28 CRs versus an overall average ofless than 10 per month for the previous 18 months). About 86 percent (24 CRs) of the March 1997 CRs were classified as "Before 9/95." The fact that the increased number of March 1997 CRs maintains the high percentage of"Before 9/95" items, indicates that the increased rate of CR initiation represents greater sensitivity to identifying configuration control issues, rather than reduced effectiveness in current configuration control programs.

C. Authorized Versus Unauthorized The analysis showed more configuration control discrepancies associated with authorized changes that were not properly documented, than with unauthorized changes (i.e.,112 " Authorized Change" CRs versus 85 " Unauthorized Change" CRs, Chart 1-1). D. Processinvolved The " Process involved"information showed differences between the activitiec associated - with " Unauthorized" and " Authorized" configuration control discrepancies. In pvticular, 4 12 April 11,1997 w - v.- - ~, - -

O most of the CRs involving an " Unauthorized" configuration change were associated with the MWD process. 'I he three largest process categories for "Unwhorized" change CRs were MWDs (53%), Original Installation (23%) and Modification Requests (12%) (Chart III-2). In contrast, the three largest process categories for " Authorized" change CRs were Modification Requests (34%), Other (22%) and Original Installation (! 7%) (Chart 111-1). E. Assessment Categories The analysis of" Assessment" category Sformation showed that overihree quarters of the CRs fell in three categories: llistorical Cmstruction (37%), Document Update (22%) and Improper Work Practices (19%)(Chart 11-3). The IIistorical Construction CRs involved mostly original installation errors, midocumented changes and the past history of not maintaining appropriate control d drawings. The Document Update CRs mostly involved procedures or drawings not beb g identified or being incorrect in the Design Change Package and/or Field Design Change Request. The Improper Work Practice CRs involved mostly the wrong part being issued /procused/used, Maintenance planning or work instruction deficiencies, and errors during installation (Tables 1 and 2). Four other " Assessment" Categories each accounted for 4 to 6 percent of the CRs: SRI-ECN, Coordination / Timing, Program Related, and Not Aware of Configuration Change. F. Assessment of Recent Unauthorized Changes Of the relatively small number (9 in 19 months) of CRs involving recent (after September 1995) " Unauthorized" changes, most involved a part being replaced with a similar, but not identical part, without required documentation. G. Assessment of Recent Authorized Changes Of the CRs (18 in 19 months) involving recent (after September 1995) " Authorized" changes, most involved document updating or coordinating / timing issues. A flowchart of the analysis methodology and overview of results is shown in Figure 4-1. 4.3 USAR Imnlications Condition Reports that were classified as " Unauthorized Configuration Changes" were reviewed to determine if they were in conflict with the USAR. A total of 85 condition reports were reviewed by the self assessment team. Of the 85 unauthorized configuration change condition reports reviewed by the team,23 required further review by System Engineering. At the completion of the system engineering review,11 of the 23 were determined to be, in some manner, in conflict with the intended configuration or the intent of the USAR. 13 April 11,1997

Of the 11, eight were the result of human error during maintenance or modification activities and three were the result of original installrin of the equipment or original plant design. One of the unauthorized changes (regarding tornado venting) was significant, reported to plant management via the condition report system, assigned a leve! "1", reported to the NRC, and is currently under design review. Others were less significant, such as temporary removal of a fresh air intake pre-filter due to icing without recognizing that removal constituted a configuration change cz a configuration different than that stated in the USAR. All but two of the issues have been resolved by placing the equipment back to the correct configuration or changing the configuration through an authorized configuration change document. The two remaining issues are currently scheduled for resolution. 4.4 Labeling Errors A separate review of the 34 Condition Reports (CRs) that were identified by the Configuration Control Self Assessment (CCSA) Team as labeling issues was conducted. These 34 CRs were divided into the following causes: Document change improperly implemented: this includes issues where a drawing was changed, but plant procedures or labeling were not updated following completion of the document change ECN Error / scope during Label Upgrade Project: this includes issues where the direction provided to the " Project 1991" labeling team was insufficient to ensure all plant equipment was labeled or errors were apparently made during the Project Incorrect drawing: self explanatory Installation error: this includes label installation errors, that is, the labels were incorrectly installed in the field, even though the appropriate drawing / design document was correct Lack of walkdown when preparing fuse list: this identifies a singular instance where a design engineer failed to walk down a system during preparation of a fuse list for that equipment Skid mounted equipment: this category identifies equipment that, during criginal installation, did not have separate components identified because the equipment was purchased as a skid. llistorically, when skid mounted equipment was installed, none of the " support" components for the skid were labeled. This was consistent with the original design philosophy. Unknown: this category includes issues where it is not possible to determine the cause or investigations have not been completed sufficiently to determine the cause. (Preliminary investigations indicate that a high level of confidence exists that the errors were introduced during original installation). 14 April 11,1997

1 i The CRs were separated into categories by reading the complete condition report, assigning a preliminary cause code to each, then refining the cause codes into those mentioned above. The review was conducted by a Team member who is an experienced engineer who has been assigned to Fort Calhoun for a period of time sufficient to have witnessed the evolution of the i configuration change process from early operation to present. A summary of the errors noted is as follows: 13 of 34 (38%) were caused by lack of, or inadequate, procedural guidance (" Error /scopa during label upgrade project" and " Skid Mounted Equipment") 13 of 34 (38%) were caused by field error (." Installation Error") 4 of 34 (12%) were caused by Engineering personnel error (" Document Change ECN improperly implemented"," Incorrect drawing", and " Lack of walkdown when preparing fuse list") 4 of 34 (12%) had unknown causes. In most cases, the error was likely introduced in the field during original installation or historic maintenance. 1 Although the labeling CRs were maintained separately in the data base for analysis, the overall conclusions point to human performance issues (50%) such as installation error, incorrect drawing, and document change ECN improperly implemented. The second largest category was procedural guidance issues (38%) such as scope during labeling upgrade project and skid mounted equipment. Most of the labeling CRs were the consequence of"Before 9/95" events. i i i 15 April 11,1997

47% 56 % MR-ECN Document (FC or SRI) '{ . Update ~ 51 % } Analyze by wg See Note 2 . i 7% _, j construeen i 53 % h mj .O -oI. asiisel q I 7* 1 co n" Auca. by o QQf j- / 7 [ Note 1 i \\ t_._ ,/ 23 % originai Cons >@n i 16 % Note 1: Six of 222 significance level one or two 7_ _11%_ [ Note 2: 84% occurred before 9/95 ] Procedural I Guidance or Note 3: 89% occurred before 9/95 cowsores k._ _ _. _ _ ins agah Em M% t Not e Contgurabon 7--- lasue Rounds issues ~- Analyze Operator Category M% j Con 6guration ca wntten [ Figure 41 q .J Analysis Methodology l v T ,v. .w.*--.---.w-- ---.-w,.,-,r.--e,vrwv -a -- ---,r,-vm-

mvee,

~.---rm4( e-.,..-. utor--w ,v .w-+r, c., .y-, ~~s 1.w-wr v-re.~.. =~ w -,~ww.. wyr.w..A-o---

e i SECTION 5 WALKDOWNS FOR CONFIGURATION ISSUES (Task 4 of the Plan) 5.1 Specifie Walkdowns The Configuration Control Self Assessment walkdown was completed on the Auxiliary Feedwater (AFW) system. The walkdown was a simple comparison of drawings to installed equipment to determine if all installed equipment was reflected on the drawings and all drawing requirements were met by the installed equipment. The selection of drawings was made at random. Various component tag numbers were chosen at random and several drawings for each component were chosen at random for verification. The drawings were taken to the field and a verification ofinstalled equipment was completed. The installed equipment was also verified to be properly represented on the drawings. All discrepancies were noted. The discrepancies were reviewed to determine if unauthorized modifications had been performed. Two condition reports were generated as a result of this review. In each instance the configuration changes appeared to have taken place 10 to 20 years ago. One appeared to be 1 inadequate drawing update after original installation. The other appears to be a change generated j as a result of a maintenance activity. In both cases the current configuration control procedures would have reduced the possibility of the unauthorized change. The results of the conFguration Control Self Assessment walkdown ofisometric drawings were similar to the results of the Licensing Basis Assessment walkdown. These walkdowns were i performed independently and both identified specific issues associated with system ) configuration. These issues were similar between the two walkdowns. The similarities of the two waikdowns supports the conclusion that the Licensing Basis Assessment walkdowns provide a good check of configuration control at the Fort Calhoun Station. Licensing Basis Assessment walkdowns have been performed on the Charging and Safety Injection systems. These walkdowns identified issues that were considered configuration control issues. The configuration changes appeared to have taken place long ago. These configuration control issues would have been less likely to occur today with the current improved configuration control procedures. The ongoing process of walkdowns should provide additional assurance that configuration control is being maintained. i 16 April 11,1997

5.2 Operator Rounds in addition to the specific walkdowns above, the operating crews were asked to prepare a list of observations from their nonnal rounds which might raise configuration control questions. The list was evaluated through March 31,1997 by the CCSA Team to the extent practical and the results documented on CRs when required. The results are consistent with the walkdowns and with the CCSA database analysis. The Configuration Control Self Assessment Team reviewed items that were identified during the month of March 1997. Items identified were investigated using the following methodology as applicable:

1. Review of maintenance history

2. Review of modifications where applicable

3. Review of historical incident Reports

4. Confirmatory Field walkdowns

5. Review of CR data

6. Review of ECNs

7. Review of the EDIT system

8. Part verifications in warehouse

9. Interviews with applicable personnel Thirty six (36) items from the list of observations had been walked down during development of this report.

Four CRs had been previously written on items contained in the list. CRs 199700243, 199700250 and 199700257 have been reviewed and closed as not configuration control issues. CR 199700255 is in the population that was included in previous evaluations. Fifteen (15) additional CRs were written as a result of walkdown of the listed items. Of the population of fifteen (15), two (2) were determined to be not configuration issues by the CAG. Of the remaining thirteen (13) CRs, preliminar'y screening was performed resulting in the following insights: Of the remaining thirteen (13) CRs,(9) nine CRs or sixty nine percent (69%) are before 9/95 (historical) issues The remaining four (4) are after 9/95(current) and are configuration control issues related to lack of attention to detail by plant personnel. i Of the nine (9) before 9/95 CRs, six (6) or 66% are unauthorized configuration changes. Of the thirteen (13) CRs, six (6) or forty six percent (46%) are unauthorized configuration changes related to the MWR process. This small sampling should not be extrapolated to indicate programmatic trends or issues. The insights alone are generally consistent with the overall findings of the CCSA. 17 April 11,1997

4 SECTION 6 INDUSTRY EXPERIENCE REVIEW (Task 5 of the Plan) A review ofindustry experience was conducted. This review included the following: e INPO AP-905," Configuration Change Process Description" e INPO AP-906 (Preliminary)," Design Change Process Description" INPO AP-922 (Draft)," Design Basis Change Process Description" e NRC Inspection Report 50-336/96-201; 50-423/96-201, "Special Inspection of Engineering and Licensing Activities at Millstone Units 2 and 3" NRC Inspection Report 50-237/96-201; 50-249/96-201, " Independent Safety Inspection of Dresden Nuclear Power Station" e NRC Integrated Inspection Report 50-213/96-08 (Connecticut Yankee Atomic Power Company) e NRC Independent Safety Assessment of Maine Yankee Atomic Power Company i e Visit to Millstone by Fort Calhoun Staff Manager and member of the Self Assessment Team o Combustion Engineering Owners Group (CEOG) Resultai INPO Documents Review: A review of the INPO documents was conducted against the configuration control philosophy at Fort Calhoun Station. The INPO documents are very similar in nature to the FCS methods of configuration change, design change processes, and design basis changes. The INPO documents provide valuable insight into the processes with the use of graphics that ce not included in the FCS procedures or in the FCS training for design / system / field engineers. Additionally, the INPO documents frequently refer to " electronic" updating of configuration control records. While many of the configuration control records at FCS are updated electronically, many records / drawings are updated manually. There is an opportunity to improve in this area as computerized expertise continues to improve. l8 April 11,1997

NRC Reports Review: A review of the NRC Inspection Reports for Millstone Units 2 and 3, Dresden Nuclear Power Station, Maine Yankee Atomic Power Station, and Vermont Yankee Atomic Power Station was conducted by four of the self assessment team members with an emphasis on configuration control issues. The results of this review indicate that many of the examples of poor or non-existent configuration control issues cited in the reports are similar in nature to issues raised at Fort Calhoun Station in the mid-1980's. Since the mid-1980's, FCS has undergone many program enhancements in all areas of plant operation, maintenance, and configuration control. In the area of configuration control, these enhancements include the following: e USAR Upgrade Project e Procedures Upgrade Project e Design Basis Reconstitution Project e Improvements ta the Configuration Change Proceses e Continual selfimprovement processes Millstone Visit Review: Northeast Utilities provided a briefing on the Millstone Unit 2 restart issue to representatives from CE plants on April 8,1997. Conclusions and actions relative to the Fort Calhoun configuration management program and actions are listed: e The design basis restoration and reconstitution being conducted at Millstone Unit 2 are similar in scope to the design basis reconstitution previously conducted at Fort Calhoun. The methods used at Millstone to reconstruct the licensing and design bases are similar to those previously used at Fort Calhoun. Based on the work being conducted at Millstone, the Fort Calhoun USAR verification should include the Fort Calhoun risk significant and safety related systems. The USAR Chapter 14 Safety Analyses and associated engineering analyses should e be reviewed to identify and document system requirements before conducting system verifications. Combustion Engineering Owners Grouo (CEOG) Review: One of the team membeu is the OPPD representative to the CEOG. In October 1996 and in March 1997, the CEOG (representing nine utilities) met, and as part of the agenda discussed current configuration control issues. Among the topics discussed were: The confirmatory action letters for Millstone and the extent of their recovery action e plans to restart e The cabling configuration control issues at Maine Yankee 19 April 11,1997

n e Procedures for making simplified (or minor) configuration changes and utilities' experiences ; e .importance of complete identification of all documentation and drawings affected by configuration changes e NRC's proposed recommendations for clarifying the 10CFR50.59 review process - e Recent efforts to verify USAR design bases and the potential advantage ofidentifying commonalities of design among CE NSSS plants o. In addition, copies of presentations and other material Millstone 2 used to describe their configuration control problems v ere distributed. The industry reviews indicate that, although FCS does not have problems to the extent noted in the NRC reports, that there is room for improvement. Areas that should be targeted include: e-Continued self-improvement. Additional procedural enhancements after the INPO draft procedures are issued e -Improved awareness of what is and is not a configuration change. 'e - Continued education on the expectations of management and the NRC on USAR issues e Continue improvements in the computerization of configuration issues e Continued improvement in the time from installation of a configuration change to close-out of all documents 20 April 11,1997 -m. e, ++re w +- m s

SECTION 7 G)Nt'LUSIONS The results from the categorization of CRs are provided in Appendix B. Review by the Team of these graphs and tables provided the following insights and conclusions:

1. Ilistorical Events Tables 1 and 2 clearly show that events occurring "Before" September 1995 dominate the data with 170 out of 197 CR so categorized (86 percent). In particular, CR categorized as "Ilistorical" (older events, including original construction issues) comprise 74 out of the 170 (44 percent). The additional walkdown data and the CR from the operator rounds (described in Sections 5-1 and 5.2) support this conclusion. Based on Chart Il-1 it is reasonable to expect that the trend to find and identify historic configuration issues will continue.

Considering Charts 111-3 and 4, a very small number (6 total) of all configuration control CRs are significant (Condition Levels 1 and 2).

2. Current Events The smaller number of events occurring "After" September 1995 (27 out of 197, or 14 percent), best reflect the current configuration control process and work practices. These CRs show Document Updating as a major cause category with failure to identify affected procedures and drawings required to be updated as part of the configuration change package as the areas of major concern. They also show Improper Work Practices and Coordination and Timing as contributing causal categories with failure to follow procedures and SRI-ECN parts issues as the major concern.

It should be noted that the large proportion of the CRs attributed to historic issues in no way implies that there was any inadequacy in the design basis reconstitution activities performed for FCS following the 1987 SSOMI. Those reconstitution tasks, including the DBD project and the procedures upgrade project were designed to identify and verify functional performance and used walk-down mformation to verify plant conformance to P&lDs and to validate procedural objectives were being met. The historic issues identified in the CCSA are primarily component level issues, most of them of very low significance, and are arising from both heightened awareness towards plant configuration and a threshold level for discrepancies far lower than that which existed at the time of the design basis reconstitution. In this respect, the results from the CCSA support the prior reconstitution work, since such a small number of the total CRs were considered significant, and since the majority of the procedure related CRs indicate human performance issues rather than validation errors. Recall from Section 4.2 that " Authorized" configuration changes involve a discrepancy between design / configuration documentation and the as-built configuration, whereas " Unauthorized" configuration changes involved a discrepancy between the approved and/or analyzed (documented) configuration (at the time ofinstallation) and the as-built configuration. These categories were selected because they lead to different sets of causes and provide insights to recommendations for reducing future occurrences. 21 April 11,1997

1. Events categorized as " Authorized" Configuration Changes comprise 51 percent (112) of the total CR (Chart I-1). Of this group, only 3 are Condition Level 1 or 2 (Chart III-3). The j

processes causing the configuration issue were primarily design change documents (MR-ECN) at 47 percent and historic or original installation at 17 percent (Chart III-1). Review of the assessmer.t categories associated with the Authorized MR-ECN category shows 56 percent were due to document update problems and 15 percent resulted from historical construction issues (Chart 111-9). Further analysis of the assessment categories of Table 1 shows the predominant problems in document updating are due to procedures and/or drawings not being identified in the design change document or being incorrectly identified.

2. Events categorized as " Unauthorized" Configuration Changes comprise 38 percent (85) of the CR (Chart I-1). Of this group, only 3 are condition Level 1 or 2 (Clu 1III-4). The processes causing the configuration issue were primarily MWDs (53 percent) t j historic or original installation (23 percent)(Chart III-2).

Review of the assessment categories associated with the Unauthorized changes due to MWDs shows 23 percent due to historic construction and 48 percent due to improper work practice (Chart 111-11). Further analysis of the assessment categories of Table 2 shows the predominant problems in historic construction are undocumented configuration changes or original installation error. Problems in work practices are due to parts issues. The data from Tables 1 and 2 are important, because of the dramatic comparison between historic and current problems: 1. " Unauthorized," "Aner 9/95" issues, that is, the category most representative of whether FCS has a current problem with unauthorized configuration changes, shows only 9 CR, whereas the "Before 9/95" portion of this group contains 74. This is a very strong indication that the current procedures for configuration changes reviewed in Section 3 nave been effective in controlling plant configuration. 2. The " Authorized" category tinm Table 1 exhibits the same conclusion. Only 18 CR are current issues versus 04 that are "Before 9/95." Further review of detailed analysis categories on Table I shows that failure to identify affected drawings or procedures in the design change documents is identified 23 times for the "Before" CR versus only 3 times for the current CR, which demonstrates further that the current configuration control procedures are effective. Regarding the procedural review in Section 3, modification and maintenance procedu es currently in us: were concluded to have sufficient controls to assure that plant configuration changes are appropriately documented and reviewed for effects on nuclear safety and compliance with design and licensing bases, including industry design codes and standards to which FCS is committed in its USAR. 1 22 April 11,1997

Some weaknesses were encountered in various engineering procedures. These are characterized as minor weaknesses in that barriers do exist to limit the probability of creating a configuration discrepancy. Those instances in which barriers are not clearly evident involve processes in which the sequence of events allows a configuration discrepancy to exist for a limited time until the modification is completed or the affected document is issued. Another weakness involved the SO-R-2 process for identifying configuration discrepancies and assigning them appropriately. Any configuration control issue, whether it is a document discrepancy discovered via routine personnel observation or a formal request for engineering assistance via the EAR process, should be filtered through the governing configuration control ] procedure, PED-QP-2 (depicted in PED-QP-2, flowchart 1). The current condition reporting procedure, S0-R-2, may not assure that any discrepancy in the configuration control boundary, defined in PED-QP-2, is necessarily identified on a CR or assigned to Engineering for resolution via the PED-QP-2 configuration control procedure. The following genetic conclusions have been reached based on the information collected and analyses performed by the CCSA Team:

1. Configuration control discrepancies are being identified at a rate that is substantial (over 10 per month) and increasing. Only a small minority of the discrepancies is significant, however, and most of the items identified are pre-existing problems that are being identified at an increasing rate due to increased sensitivity to configuration control issues.

2. Recent configuration control discrepancies have included unauthorized use of non-identical parts, inadequate document updates and inadequate coordination between document updating and installation of changes.

3. A review of procedures identified the following areas for possible enhancement 1.

Timeliness and sequencing of engineering document and drawing updates. 2. Engineer final review of completed drawings. 3. Screening for effects on the USAR during CR evaluation and resolution.

4. The industry experience review identified several areas that could be targeted for improvement.

5. USAR reviews may not be completed, where appropriate, for Condition Reports. The current CR system does not prompt the owner to review the USAR during corrective action implementation.

6. Efforts to prevent future configuration control discrepancies should focus on the following areas:

i Ensuring that non-identical replacement parts are properly evaluated before installation; Ensuring appropriate identification and updating of procedures and drawings; j Ensuring that documentation is completed at the proper time; 23 April 11,1997 i

. _. = -.. -. .-........ -. ~ -. - =.. - -.... -.. - h Ensuring uniform understanding and implementation of configuration control . programs /requirem:nts. in summary, the data reduction selected for use in this assessment and shown in Appendix B is only one way to view the many possible correlations. The Team did consider the CR data in many different ways, including the Analysis Matrix in Section 4.' The insights described in this section accurately represent the generic conclusions for the CCSA. 1 4 5 4 ( i i t 4 4 ? 24 - April 11,1997 e e. .y r y

1 l s b SECTION 8 ) 4 RECOMMENDATIONS Based on the analyses and conclusions from the Configuration Control Self Assessment, the r following actions are recommended. 1. - Since historical and current CRs related to configuration control are expected to - continue into the near future (Chart 11 1), the Corrective Action Group should develop q a standardized trending capability based upon the assessment categories in this report. In this way, the causes for the deficiencies can be monitored and periodically i exammed to see if the conclusions of this CCSA remain valid. i i l' 2. Many of the major causes from the assessment categories analyzed in l'and 2 indicate human performance related issues. FCS should continue to emphasize the current

Coaching and Counseling management style for supervisors and managers in order to improve human performancec This will help to clarify expectations for workers and raise awareness for the need to enhance overall performance. The institution of

" Human Performance Days"is a positive initiative. l 3. All FCS personnel should be encouraged to use the CR system to identify - configuration control issues. Consistent with the data in this report, results should be carefully reviewed at the Corrective Action Group (CAG) daily meeting to ensure i i significance is appropriately prioritized. i 4. The procedure enhancements related to the conclusions from Section 7 should be i considered for incorporation. j i i 5. Coaching and continuing training on the us'e of the recent non-significant configuration change procedures must be emphasized to ensure the potential for i future unauthorized configuration changes is reduced as much as possib!r This effort should also address the areas identified in Conclusion 6 of Section 7. 6. . The configuration change process should be reviewed when INPO issues the new configuration change documents. In addition, other recommendations resulting from i the Section 6 Industry Experience Review should be reviewed and implemented as appropriate, i 7. The Condition Report program should be revised to ensure that the USAR is i. reviewed, when appropriate, for impact on plant configuration. 9 i i 25-April 11,1997 1 ,y>. -p o 4-p a --m,,

APPENDIX A CONFIQURATION CONTROL SELF ASSESSMENT PLAN

Background

The physical condition of Fort Calhoun Station (FCS) is dynamic over time due to maintenance, modifications and aging efTects. The documentation that assures FCS is configured at any point in time to operate consistent with its licensing and design basis is also dynamic and must be continually updated as the plant configuration changes. The regulations and implementing procedures for performing and documenting plant configuration changes are also dynamic, and j have continually been upgraded over time. The objectives for this upgrading are to ensure that the following attributes are incorporated into the safe operation of FCS: 4 (1) increasing levels of awareness of design basis requirements (2) analytical sophistication to ensure all interactions important to safety are addressed 3 (3) awareness that maintenance and operation activity can affect configuration (4) increasing standards of excellence throughout the nuclear industry . Because of this dynamic environment, there are many potential sources of configuration discrepancies, especially in an older plant like FCS: 1 ' (1) Original equipment installed differently from the plant drawings (2) Older modifications which may not have adequately updated drawings and documentation (3) Replacement part evaluations that were not well documented, done prior to the current Substitute Replacement item (SRI) process (4) Past maintenance work practices that did not recognize a configuration change had occurred i (5) Unauthorized configuration changes (6) Lack of procedural guidance to plant staff on what constitutes a configuration change i 4 (7) Contractor support groups not well informed on configuration change issues J Following a milestone SSOMI in 1987, FCS undertook a series of major activities to ensure our design basis was reconstituted and mainta%ed. In particular: (1) FCS completed the Design Basis Reconstitution project in 1990 (2) Several walkdowns were conducted to verify that the physical plant matched plant drawings and other documents (3) Modifications implemented prior to the DBD project were reviewed for adequacy of j 10 CFR 50.59 evaluations j .(4) 1770 open items were identified. Most (about 93%) of them have been closed out (5) Many new calculations have been generated as a result of the DBD project (6) New modification control and 10 CFR 50.59 procedures were developed 26 April 11,1997 1

l 1 (7) A USAR upgrade project was completed in 1995 (8) A Procedure upgrade project was completed in 1993 (9) A Vendor Manual upgrade project was completed in 1991 Concurrently, in 1988-1989, a proceduralized methodology for controlling configuration changes was put into place which, with improvements and upgrades, is in place today. This methodology addresses the spectrum of work that can result in changes to the physical plant, from major modifications to less significant configuration changes. j lt is important to realiw that due to: (a) the many years of operation prior to implementation of the post-SSOMI controls and reconstitution; and (b) the inherent limitations of the walkdowns to verify plant configuration, it is unreasonable to expect that one hundred percent of the potential configuration discrepancies have been identified. However, as a result of the reconstitution verification activities described in the nine items above, and continuing review of deficiencies documented in Cono: tion Reports, it is reasonable to expect that there are no significant historical configuration control deficiencies that impact plant safety. As the operating staff continues to monitor the plant, additional historic configuration issues may be found. These are documented with the CR process and resolved. They are not indicative of current unauthorized configuration changes. The configuration control procedures and the conduct of maintenance procedures in use since 1989 have also evolved, and potential issues associated with configuration changes made using these procedures have also been experienced. These issues were generally directed at how configuration changes were implemented (process) rather than whether or not they were authorized. Recent issues, however, have indicated that a Self-Assessment of how configuration change is controlled and authorized is appropriate at this time. Examples of these issues are: (1) Improper authorization for removal of Post Accident Sampling System (PASS) equipment ] (2) Changing of Like-for-Like components prior to authorization by engineering (3) Lessons leamed from Industry (4) Insights from 50.54(f) project Objectiye 1 This Self Assessment will focus on determining whether or not there is a current issue regarding l unauthorized configuration changes at FCS. 27 April 11,1997

1 l N l The following plan will be implemented: Task 1 Perform a review of the current procedures for configuration changes. The objective is to provide the confidence that any proposed change to plant configuration, large or small, is being properly evaluated, documented, approved installed, retumed to service and all supporting documentation updated. In particular, this review will determine whether existing change control tools are sufficient to address all levels and sources of potential configuration changes. Task 2 Perform a review of recent configuration control issues using the Condition Reporting (CR) system. The CRs identified will be assigned causes and categorized so that the specific type of configuration change problem is identified and the events causing it are understood. Task 3 Using the categorized information from the CR review, each configuration control issue will be assessed to determine whether it is a historical issue as discussed in the back-ground section and considered not part of this assessment, or a configuration change under the current configuration control procedures. The results of Task I will be used as the baseline. Appropriate categories for dispositioning the CRs reviewed will be developed to allow assessment of the issdes. Possible categories include: CR issue was addressed by procedures but procedures were not followed. CR issue was not addressed by procedures at the time it occurred, but is now. CR issue is addressed by the currently evolving procedures for controlling non-significant configuration changes. CR issue was an unauthorized configuration change. Any event that falls into one of these categories must be further evaluated for generic implications and/or definable trends. Categories may be modified as the assessment progresses. Task 4 Walkdowns of selected systems or equipment will be conducted to attempt to identify configuration issues which represent the very detailed scrutiny currently expected of maintenance and modification work. Credit may be taken for walkdown information currently being provided by Operations personnel from their rounds and from other current system walkdowns. Issues identified from this walkdown data will be evaluated using the method and criteria from Task 3. Task 5 Perform a review of appropriate industry experience with comparable procedures and potential unauthorized changes to ensure the results of this self assessment incorporate lessons learned. 28 April 11,1997

l j Result 4 The conclusions from this self assessment will be summarized in a report to EOS Division Management and will also be submitted to NRC Region IV as the basis for the recommended resolution of the issue of unauthorized configuration changes at FCS. Any recommendations resulting from the report will be assigned for resolution. Schedule This assessment will be completed by April 30,1997. SPONSOR: Sudesh Gambhir(Division Manager) LEAD: Ralph Phelps/Merl Core (Engineering Management) CO LEAD: liarry Faulhaber (Maintenance Management) TEAM: Joe Gasper (Design Engineering, Nuclear Projects Management) Jim Geschwender (Nuclear Safety Review Group) l Jim Foley (System Engineering) Kevin flyde (Design Engineering, Mechanical) Sudhir Kalra (Design Engineering, Electrical) Dave Taylor (Quality Assurance) l Larry Hopkins (Planning and Scheduling) Ed Bernal (Nuclear Constmetion Management) i l 4 r 29 April 11,1997 1

7 1 APPENDIX B DESCRIPTION OF THE CR DATA i bata Analyses Matrix Grouping I - Use all data in CCSA database 'l. Plot Authorized, Unauthorized and Labeling categories as a percent of the total data. Grouning II - Eliminate " Labeling" data from CCSA database 1. Plot After September 1995 and Before September 1995 categories as a ftmetion of time I 2. Plot A'uthorized and Unauthorized data by Cause Code from CR System t ~ 3. Plot Assessment Categories as a percent of the database ' Grouping 111 - Separate CCSA database (less " Labeling") i,nto Authorized and Unauthorized data l. Plot Authorized data by Process Involved 1 2. Plot Unauthorized data by Process Involved 3. Plot Authorized data by CR Significance Code L 4. Plot Unauthorized data by CR Significance Code 5. Tabulate Authorized data by Before September 1995 and by Assessment Category (Table j 1) 6. Tabulate Unauthorized data by Before September 1995 and by Assessment Category (Table 2) c .7. Tabulate Authorized data by After September 1995 and by Assessment Category ( Table 1) 8. -Tabulate Unauthorized data by After September 1995 and by Assessment Category (Table ". 2) ~ 9. Plot Authorized data by Process Involved - MR and ECN and by Assessment Category

10. : Plot Unauthorized data by Process Involved - MR and ECN and by Assessment Category

? 11,' ! Plot Unauthorized data by Process Involved - MWR and by Assessment Category o. 30 April 11,1997 ~ y y, y a e . w. r v., m E-,

~ \\ i n i i Configuration Change CRs 1 1 Labebng 11 % s i "Buborized Changes i 38 % i i r i I Aumorized Changes 51 % l 222 CRs CRs Through March 1997 i i i i Chart-l.1 n t

]- j

i!

4 ,jt j; ' 3 1 f i!{!:; i I !l lil !il + 5 9S E R O FE B E 5s2 s s 5eu E M 1<' ~ 5t, I T ^ F O ~ N 8so O IT C E 8sz N U . P s F O ?. W 86o T jj T T C - b x ^ g 8$m E S' [ i h [ P S ^ E r R s 8$< v H s T I 1 W S 84 R T A E o p E R I h R {s. ,y. 5 l A Y O Q. 8s, T H H C G ON E T M A 8ks C ,Q ( O-k[ 5 l s 9 / 8i< 9 c

+S R

S t I E y' T 8ss F A ~ . U 4 s N

.d-

_[ D

~5~

g 3 ^ A b 84w + 7Nd 5 9 ~ / n 9 ^ $i, E ,ma. ,[j -) w y 2 R l l O .s - (l ,F 1 F i $e6 B g .w ' E l 3$- s ,1 f~ ,c'[!'y* f ^ mf$ h-c 'v. v- $&z ,* wd c" Q~2 i ^, Xz ' ~"/ i ^ p

u~

s $so J y _e ag <g?! uA ~

• au s

4 x M @d r,w;. pi i a P J $$e Sr \\ k 8 3 5 0 o s 0 1 2 2 s 79 N=U rlE / 1 1 / 4 4

[, i l

l~ > j ill,;ll, {ll!l ll l

Unauthorized and Authorized by CRS Cause Codes i i Other 1 i 10% Design Unknown 22% 6% r; n , q. ':. J: ^ ' - } + t's.. i l i l Training dy: ST[.

1. • $(($[k j

l 4% .u?> =- ^- }- l s jg. n.a 4 i i t Reliability rh ( 6% immediate Supervision 4% 4 7 t i l t t k

f-..,.

Procedures ,ei ' ~ j 13 % + \\- F Management Systems l I F#j 's - . [ 17% a m i t Personnel i 18% CRs Through March 1997 i 1 l 1 Chart-il.2 i 1 l f I l

I 4 t l Authorized and Unauthorized l Assessment Categories l l l i' j NOT AWARE OF CONFtGURATION i OTHER COORDtNATNIMIN 4% i i DOCUMENT UPDATE PRO M O 22 % 4% 1 SRt ECM CONCERN 6% }' ij, _ cf N 4 . "',y ' L ~ ( ( \\ .-c m t i \\ {' tMPROPER WORK PRAC I t \\ 1 HtSTORtCAL CONSTRUCTION \\ i 37 % e l 197 CRs CRs Through March 1997 i t d t Chart-il.3 + i m

4 1 ] 4 t l Authorized Changes l i Percent by Process i SRI FC i 7% 6% l 1 1 l t i i 4 i I 7 4 OTHER 22 % l I i MR l l 34 % l i 1 i I i I i l h l ~ l l ^ l r j 7:gr { +s l .2 , ~ i 5 i~..$.) k; ' ~, l 4 g L ~^ ? f -> j ORIGINAL l .31 - ~ m. 17% _;{d ' -. c. -. ~ MWD l 7% NA I 112 CRs 7% CRs Through March 1997 i t Chart-ill.1 r

Unauthorized Changes Percent by Process SRI j OTHER 5% MR i 1 2% 12 % i i e l \\ 1 4 o ORIGINAL Pjp ~ %, { m. (., 23 % ,h.@!?frliJ VV!.' _ W i l gr a:c;.:. :m i c%, t. $ .,4

m..

.c'..99,. ~ l a., i . y 7; 3 s x

4 7.7[-

i m. . -u,. c '? ,.~1'+), i (. O 4 f <:p+ i l \\ l ...>,","<.,,/

te e -.

I i I I i NA ) i 5% \\ I l MWD i i 53 % \\ I 85 CRs CRs Through March 1997 i Chad-lli.2 4

~. -. I b Authorized LR Level 3 -Condition Lesel6 3 -Condition Level 1 g \\ f ? 3 -Condition Level 5 ' x-s 3 -Condition Level 3 \\ i g t -...e,- _ y. ' v: a s. . s~ 4 _ w Wi.~ht, ^ /YA t $= ? ^, _.$?fi. _s. '4:n. O ^ 7 . ~- - p, .'z,. t - :.u m v_ >:w : . ^~ n? n a n, .m. Q' 'hP"

; y

'_e' 4. -y. ey% .~ ..m y

o. e..._,.. ;r -

v_- w ,g s. ..ec < g u% ; w-; i z a M. N y,. 'n g ' ~ ^ ^I'~ m.. 7- ".$.,^. .E +- Y: s 4.4 3 .. g., : Q n g t - _:: yq ..& c ', ,n

++

a^, s .x - 4 g ^U'. ,[ N D , t / .gv ,y +:__ -tr -:.c.. ,. -. - w...;.. r. c: u - . m, e %, .c t 17 l

4..

'^ - / l 4 ~ ] 2 t 1 100-Condition Level 4 i .e . s,. P., 4 g,, 112 CRs CRs Through March 1997 y Chart-lll.3

1 k I = I l i Unauthorized CR Level a y 1 -Condition Level 6 2 -Condition Level 1 .i 6 -Condition Level 5.~ \\l -Condition Level 2 ~ ~ ' 1 -Condition Level 3 f I I u ' ,J' l fiU l 19:j '. s x;g 3, -., - ,~ t an:, :W: m v,

- g l

mi - egN;

<,...._ : : W'L, /,. _ ' L' w 'w;. m,,. ', k f -'of 9

s.

gw s_ ; 1 e y -h.'- -3J-b T 7'<,.. x. h : p g: : -._.,? ,c i-- &H - ; e ._m.g... ,;p, t 9 a s WQg - ' Q L2 y ' %m'.. +r ..m - v. sm._h, ; c 3G.;A v w:, > o %,- 'v:%m 3 -' t. ;:; ; ', j _ 0 ~,. .% e s ri;. e + L 9 q4, p: ~- 1, i 1.; v r 'l _q._ ~~y-;y ~ .,L ' W, _ l pp g7% T ,.. :) -. i n;, _-. w ~- g: Jg., =,!f. 4 y - y.... * .+ A .,s f% 2 se l m g...w?,,: ; ']l} ;' ',_ s 4-n f l ' - %...Q..L..,2 v 4 y l' , g y~ v l <'n' - p-: s f ,w-(g i .-,g l t a ou t.:a s a 3 v , w -n . j# ) .. = y ^z . ~.... y ..,9t .( _ s. y.,G;' + s. ~g y:'.- <c s' m;= >,.1 q m %.; w e-g / L, .y .+ OO741 6/6/96!DURIN CALORI 4'2 OTHER PROCEDURE O 10I 13 i 199600755 6/7/96THE E THIS C 4 1 ORIGtNAL DRAWING O 20 25 199600765 6/10/96DURIN THE CO 4 1 ORIGINAL DRAWING O 20 21 199600767 6/11/96,HEATE OlFFERE 4 1 MWD DRAWING O 40 41 199600768 6/11/96!DURIN MWDS 4 1 MWD PROCEDURE -1 10 11 199600780 6/14/96[PER G THE ORI 4 1 ORIGINAL DRAWING O ~ 4 20 199600787 6/15/96lDURIN THE LA 1 1 MR DRAWING O 20~ 23 24 199600793 6/17/96tDURIN 53 lMR LABEL O 199600798 6/17/96THE IN PAST P 43 ! ORIGINAL LABEL 0 199600827 THE C 4 1 [ MWD Other O 30 33 199600829 6/25/96lWHILE AFTER 4 1 MWD LABEL C 20 21 199600855 7/1/96%HILE THE LIM 4 1 ORIGINAL DRAWING O 20 21 199600880 7/9/964HE W THE MO 42 ORIGINAL DRAWING O 50 5 i 199600920 7/19/96OPERATHE AP 42 MR DRAWING O 10 11 199600926 7/22/96WHfLE THE AF 42 MR DRAWING O 10 12 199600939 7/24/96WHILE THE WR 4 2 MR PROCEDURE -1 30 32 199600943 7/25/96PMO THE CA 42 OTHER DRAWING O SO 33 199600954 7/29/96WHILE SYSTE 4i1 ORIGINAL DRAWING O_ 20 21 199600969 8/1/96,DURIN AN EVA 4'2 OR!GINAL LABEL 0 10 12 A 4 __ 199600970 8/1/96NOT C THE CA 42 MR PROCEDURE 10 10;4 11 i 199600997 8/11/96THE O THE OP 42 MR gPROCEDURE -1 10 11 199601003 8/13/96 USAR USAR A 42 MR PROCEDURE O 104 21 { 199601005 8/13/96THE C CHARGI 4.2 MR DRAWING O 20 25 199601013 8/15/96DURIN CAUSEI 42 MR PROCEDURE -1 11 i 199601021 8!16/96 PI-511 CAUSE 4 1 MWD DRAWING O 2OI 21 199601039 8/22/96SL-36 HE CON 4 1 MWD DRAWING -1 3OI ~ 36 i 199601066 8/29/96,DURIN MWOHI 4 1 ORIGINAL DRAWING O 20 21 199601112 9/12/96lCH-19 CAUSEI 4 1 ORIGINAL DRAWING O 20 21 199601126 9/16/96{MWO9 THE DR 42 SRl_ (DRAWING f-1 60 61 199601145 9/20/96?tP-5 C THIS C 42 OTHER ! DRAWING O 20 25 199601154 THERE 2 1 MR fProcedure O 20 21 199601172 9/29/96THE IN CAUSEI 43 OTHER ! LABEL O i i 133601187 10/2/96 DURIN THE WI 511 IORIGINAL IDRAWING f0 ~ 30 35 199601190 10/3/96THE E FAILUR 4Y fC_ PROCEDURE [1_ 10 16 i 199601205 10/6/96,WHILE IT IS NO 3 3 OTHER ' LABEL 199601212 10/8/96!DURIN 52 ORIGINAL DRAWING 10 12 10/7/96fNHILE 199601219 THE CA 4 1 ORIGINAL DRAWING 199601245 10/13/96M/HILE IT WAS 42 OTHER DRAWING 20L 21 20! 25 1 Page 3 L

CCSA DATABASE 4/13197 ' ate CR kvent f)Cause ftl CR Level l Type of Change Process involvbocument Type involv l Event Occured after 9/95 l Assessas.t1 l Assessment 2 Cordtion Report D 199601265 10/16/96lECN 9 CAUSE 4jl iMWD DRAWING 10 40{ 42 199601271 10/16/96TWHILE 4;2 ORIGINAL DRAWING O 2O[ 23 199001281 10/19/96]PA913 THE DR 4'2 jMR LABEL 0 20' 25 19 601284 10/19/9600RIN BREAKE 4 1

MWD DRAWING jo 30 34 199601295 10/22/56 WHIL THE CA 4

3

}OTHER jLABEL !O 199601298 10/22/96,CIAS THE CA 4!1 (MR jDRAWING 'O 3 32 199601329 10/26/96ICRAFT 4!1 lMR iPROCEDURE O 3 32 199601331 10/2'1/96 ALKD PAST P 42 I ' RAWING ORIGINAL D THE CA 4 1 MR DRAWING 'O 2 25 10/31/9kCIAS 199601355 O 30i 72 199601357 10/31/96; PED-G THE SU 4 1 IMWD DRAWING -1 40 43 199601365 11/1/96lTHE B THIS C 4 1 ' MWD DESIGN DOC 0 20, 24 199601366 11/1/96!FW-16 5 1 ORIGINAL DRAWING O 3OL _34 199601367 11/1/96pP-11, THE CO 42 ORIGINAL PROCEDURE O 80 80 199601368 11/1/96lTHE F CAUSE 4 1 MWD DRAWING O 30 31 199601443 11/13/96!EA-FC-INADEO 4 2 OTHER DESIGN DOC O 50 52 199601451 11/14/96fTHE V APPARE 4 2 ORIGINAL DESIGN DOC 0 20 2b 199601472 11/16/96pVHEN THE CA 11 ORIGINAL ! DRAWING O 20 21 199601486 11/19/96! FUSE A SEAR 1 MWD %RAWING O 30 34 199601505 11/19/96 DURIN THE CA 3 OTHER (OTHER LABEL 0 199601519 11/25/96;DURIN 1 2 OTHER -1 50 52 199601551 12/4/96'DURIN IT IS UN 42 MR DESIGN DOC O 10 11 199601556 12/5/96IDURIN WHILE I 4I MWD DRAWING -1 40 ~ '3 4 199601563 12/6/96THERE UNAUT 4 1 MWD DRAWING O 20 21 4 199601595 12/12/96 CRAFT THE DR 42 iMR DRAWING O 60 61 199601610 12/18/96 SO-M-THE PRI 4 1 {M_WD pTHER O 30 34 4 199601630 12/23/96'MWR THIS C 4 1 iMH ORAWING O 20 21 199601643 12/30/96480 V THE SU 4 1 ' MWD DRAWING O 70 70 199700008 THE JOI 4 1 SRI Procedure O 30 33 4 199700019 1/7/97 PARTS 3 1 MWD DRAWING -1 40' 41 199700035 1/11/97 RAD PMO W 4 1 MWD DRAWING -1 70 70 199700061 1/18/97 PC 47 THERE 52 MR PROCEDURE -1 60 62 199700073 1/21/97 DISCO 42 ORIGINAL DRAWING O 20. 4 24 199700078 1/22/97 THE R THE CR 4 1 l MWD DRAWING O 70 70 199700083 1/23/97 T WA 42 fMR ORAWING O 20 25 199700120 1/30/97 LECT 4 1 !NOT DETERM ORAWING O 20 21 199700125 1/31/9 URIN 42 ' MWD DESIGN DOC O 10 16 199700127 1/31/97THE N AS STA 41_ MWD DRAWING -1 30 34 ~ 199700135 2/5/97CIRCU 41 199700136 2/5/97CIRCU 5 1 MR DRAWING O 20. 21 MR 193700148 2/10/97DURIN LEVEL 3 5 1 SRI _ DRAWING O 20 21 PROCEDURE O 40 43 199700151 2/11/97 THIS EPLACE 4 1 NA -NA O 40 43 199700157 2/12/97WHILE PMO 96 4 1 MWD RAWING 30 35 4 4 199700159 2/12/97 THE S ECN 93-42 F_C { DRAWING O 60 63 199700161 2/12/97 DURIN ISSUE D 542 >MR jDRAWING O 10 17 t 4f3 199700164 2/14/97jDURIN IT WOU MR 4 LABEL 0 199700170 l 4l2 MR ! Procedure O 10 16 Page 4

_m..- g I CCSA DATABASE 4/13/97 Condelen Report hate CR kvent ik:ause fr l CR Level l Type of Change Process h - @ =-- _ _ _; Type inwohr l Event Occured after 9/95 l A--- _ _ _ _;1 l A- .;2 199700176 2/19/97]THE IN THE PH 4l3 iOTHER

LABEL O

199700180 2/21/97pHE IS THERE !42 ! ORIGINAL ) DRAWING O 20i 25 199700184 2/21/97 WHILE 42 iMWD - IDRAWING O 4 41 199700189 2/24/97WHILE 4 1 ! ORIGINAL DRAWING O 2 21 199700191 SEVERA 32 Other Procedure -1 80{ 80 199700192 2/25/97 DISCO 4 1 FC DRAWING O 2OI 21 199700196 2/27!96 WHILE 4 1 MWD DRAWING O 40 43 199700204 4 1 Not Determin NA O 20 21 199700206 THE CA 4 1 MWD Procedure O 33 199700213 42 MR Drawmg O 20 25 199700214 4 1 ORIGINAL DESIGN DOC 0 30 34 199700220 42 Other Label 0 10 12 199700221 42 Other Labeling O 20 25 ( 199700225 4 1 MWD DESIGN DOC O 30 34 L 199700226 4 1 MWD Procedure O 20 21 I 199700227 4 2 MWD Drawmg O 70 70 m 199700229 6 1 ORIGINAL Drawings O 20 24 g 199700232 42 other procedure -1 10 -13 199700233 62 FC Drawmg -1 10 15 199700241 62 Other Label 0 10 16 l 199700251 4 1 MWD Procedure -1 40 43 199700255 4 1 MWD Procedure O 30 33 199700258 4 1 Not Determe NA O 70 70 199700261 4 1 MWD NA O 20 21 l 199700262 43 Other Label O 1 199700264 4 1 M_WD Other O 199700268 42 FC Other O 30 34 10 13 5 199700271 42 MR Drawing O 10 16 199700275 4 1 MWD Drawing O 30 33 199700277 42 FC Drawmg -1 10 ~ 34' 15 199700293 4 1 MWD NA O 30 199700294 42 ORIGINAL Drawmg O 20 25 ( 199700297 42 Not Determm Orawing O 20 21 l-199700298 42 Origmal Drawmg O 20i 25 199700310 62 Not Determm Drawing O 20 25 199700312 42 Not Deteimin Drawny ,0 20. 25 f 199700314 42 SRI DESIGN DOC O 40 y 199700318 42 .Other Label O 20 25 I i 4 i i k Page5 l 1 l l

Configuraton Self Assessment Labeling Errors 'f a C!R Number; Description- ~ ~ Document Change ECN irnproperly implemented 1 Cause t 199500357 {10 to 15 labepfo'r EHCf3A/B notiupdated when changed fro' rn l O-13A/B[ 199700241 jlG-1664 historically assigned incorrect loop number Document Change ECN improperty implemented 2 199600077 : Labels associated with stator cooling skid cont.v!vs. invalid ~ ~ ~ ~ ~ - j Error / scope during label upgrade project 3 I abels fori^l64A'5ie nit correct' ~~ ~ 199600121 [!A viv to FCV-959 has no labelino tag but is shown on P&lD ~' ~ [Edor/ scope during latsi upgrade pioject~ 4 ~ ~ - ~ 199600237! ~ ~ ~ ~ l Error / scope during label upgrade project 5 199600658 iia piping C-ID38 has 2 drain vivsTnot labe!ed/not'oiP&iD~ ~ ~ ~ ~ ~ IError/ scope during label upgrade piojec~ t~~ 6 ~ 199600678] Drain vivibetweedih-I-ICW2899D-B & lA-HCV-2899C-B 6as no labellErrir/ scope during latSI upgrade pioject_. [7 ~ ~ ~ ' ~ 1996_00798 {lAjlv. to _Al-179 & condenser !vt. gauges nct on P&lD [ Error / scope during label upgrade project _ _ _ 9 8 199601172._IA viv. to PCV-1057/TCV-1058A/TCV-10588 not shown on P&lD l Error / scope during label upgrade project 199700164[incirrect[disignationilocation foriabels__ _.. ~ ~ ~ ~~ 199700221,CF-1,2,3 agitators and CG-3 tank heaters not labeled __ { Error / scope during laisi~ upgrade' project))]j0 [ Error / scope during label upgrade project 199601281 :QSPDS labelis in PSIG when process measured is PSIA

lncorrect drawing

_ g 11 12 199600061 jYl-6767A & Yl-67678 not labeled ' Installation Error 13 ~ 199600305 AB-235 mis-labeled as AB-325 Installation Error ^ ~ ~ ~ ~ ' I14 199600352[ Cables mistabele[ ~ [ Insta!!ation Error '15 ~ ~ ~ 199600364 Approx.16 labels associated with Hydrogen coolers attached to wrong equipment _instattation Error 17 ERF computer wire labels do not match print (eq. operable) Installation Error 16 199600373,CH-320 & CH-322 labels swapped _ _ _ _ _ _. _ stallation Error ln 199600640 18 199600684_ Fire barrier penetration improperly labeled installation Error 199600969 Wire labels did not match drawing 19 Installation Error 20 199601205 T1 A-2 sec pot fuses _ drawer label reversed with gmd fault fuses drawer _ __, Installation Error 199601505]RC-114 & RC-115 labels switched 21 instaitation Error 22 199700176 Battery room indicators reversed _ installation Error 23 .lnstallation Error 199700262,.Two valves improperty labeled Installation Error (preliminary) 25 199700220 IA line to PCV-6680B-1 components missing labels /does not agree with pnnt 24 199601295 Al-107 fuses not labeled with normal convention 199600471_ Equipment for SI-1 A/1B/2A/2B/2C/3A/3B/3C has no labels _ Lack of walkdown when preparing fuse list 26 199600666 Fire pump FP-1B components not labeled . Skid Mounted Equipment 27 Skid Mounted Equipment 28 199600670_. Casing vent v!vs. on CW-_3A 8, 3B not labeled or on P&lD . Skid Mounted Equipment 30 199600793 RCP local instruments do not have labels. Skid Mounted Equipment. 29 199600359 Label for air dryer viv. found temporarily attached to a non-air system drain viv. Unknown - probably maintenance 31 199600437 :G1 Watt Meter wire labels do not match pnnt (meter is operational) Unknown - probably original installation _ 32 199600829 ; Wire labels do not match print (does not affect equip. operability) junknown - probably onginal installation 33 199700318 : Wrong ckt. bkr listed in procedure iUnknown - probably original installation 34 Page 1 -}}