ML20137L003
| ML20137L003 | |
| Person / Time | |
|---|---|
| Site: | Saint Lucie  |
| Issue date: | 09/09/1996 |
| From: | Lowens D, Walls J FLORIDA POWER & LIGHT CO. |
| To: | |
| Shared Package | |
| ML17354B293 | List:
|
| References | |
| FOIA-96-485 QSL-PM-96-17, NUDOCS 9704070136 | |
| Download: ML20137L003 (42) | |
Text
._ __ _. _ _ _ _ _ . _ _ _ _ _ _ _ _ . . . _ _ _ _ _ _ _ . _ _ . _ . . . _ . _ . _ _ _ . _ _ . . _ . _ . . _
s s
- 4
!- FPL Nuclear. Division Quality Assurance AvdirReport-
_ 1 3
~- a
- l 1 l i
t i
MONTHLY PERFORMANCE MONITORING AUDIT I
l QSL-FM-96-17 July / August,1996 l
i
}
- i i
i 1
i !
Audit Team:
I L.J. Bearror l D. C. Lowens I B.J. Lowery !
C.E. Norris l L.L. Pannessa J.J. Walls l
L. W. Bladow QA PSL l
l I
9704070136 970325 l PDR FOIA ,
BINDER 96-485 PDR j
1
@p:53L
~
AUDIT REPORT QSL-FM-96 47-Page 2 of 40 Table of Contents i
a Executive Summary . . . . . . . . . . . . . . . . . .... .. . .... .. , .... 3 1
Operations .. . . ...... .... ... . . . ... . .. . . . . .. 5 PMON 96-028 - Health Physics Practices - Unit 1 Outage . . . . ... . .. .. 5 PMON 96-032 - Steam Generator Tube Plugging Activities ABB/Framatome . . . .. 6 PMON 96-043 - Conduct of Operations - July . . . . . . .... . .. .8 l PMON 96-047 - Zach Pate INPO CEO Letter Verification .. .. .. 10 Maintenance . . . . . ... . .... . ..... .. ... 11 PMON 96-039 - Insp. and Repair of K-Line AC Circuit Breakers - 10 CFR 21 ...I1 S ervic es/E n gin ee rin g . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .... 11 PMON 96-046 - Compliance With Overtime Limit Requirements - July 96 . . . . . . . . I1 j PMON 96-052 - PC/M 009-195 EAcore NI Drawer Replacement . . . . . . ..... ....I1 i PMON 96-048 - Corrective Action - July 1996 . . . . . . . . . . . . . . . . . . . . . .... 13 j PMON 96-049 - Examination of Maintenance Rule SSC Historical Review . .. 16 I Independent Technical Review (ITR) Activities . . . .. ... .. .. .. 17 Findings . . .. .. ... .. . .. .. .. .. ... . . . ... ... .. .. 25 Finding 1: Several instances were observed in which Radiation Workers did not follow approved plant health physics procedures. ... . 25 Finding 2: Nine instances ofimproper configuration control of plant valves were identified in connection with PCM 153-194 . . . . . . ..... 27 Finding 3: Modification of the Unit 1 Excore Nuclear Instrumentation was allowed to /
proceed to completion despite multiple indications of design process breakdown . . . . ... .. ... .. . .... 29 Finding 4: In two cases. Condition Report dispositions provided corrective action of unsatisfactory depth . .. ... .. .. 31 Finding 5: In three cases inadequate inter-departmental coordination has caused Quality Assurance audit finding responses to have unsatisfactory content 36 Audit
Participants:
. ... . . .. . . . . .38 ' ' "lI-5 {k 1
AUDIT REPORT
. _ . _ . . _ _. c .. _ . _ _ _ _ _ _ . - - -
QSL-PM-96-17 Page 3 of 40 Audit Location: St. Lucie Plant Date of Audit: July / August 1996 Audit Scone: St. Lucie Plant Performance Monitoring Audit i
Executive Summary:
Performance Monitoring summaries for QA evaluations completed during the July / August time frame are included in this report. This period included the fmal portion of the Unit 1 Cycle 14 refueling outage and encompassed follow-up on activities monitored during the outage:
e The results of these evaluations indicate that in the areas of health physics, plant modification and corrective action, administrative controls are not entirely successful in controlling plant activities. Observed problems are attributed to a lack of training and administrative corarols not being treated with the necessary importance. ;
l e Operations Results
)
1 Periodic reviews of Operations activities were performed during the audit includmg:
monitoring of routine technical specification surveillances. random checks of valve required position, and all non-routine reactivity manipulations . Satisfactory performance j
was noted during these reviews. Conservative decision making and improved '
communications were also noted during these evolutions. i e Condition Reports - The following Condition rtcports issued by QA during this period indicate continuing weakness in maintenance work control and a weakness in procedure /
document control.
, Maintenance Work Control CR 96-1770 Loose debris control during entries into the Unit 1 Containment while in Mode 3 does not appear to meet site management expectations.
1 AUDIT REPORT
___p
___ _ __ _ _.-- - QSL-PM-96-17 l Page 4 of 40 1
1 l
l CR 96-1938 A new stem nut was fabricated by a machinist using measurements of an '
i.
old nut. No material traceability provided with the completed PWO.
'~
CR 96-1937 Post review noted typass of QC hold point. Procedure GMP-01 Step 2.5 l bolting inspectio s requireB.
l CR 96-2005 QI-l1-2 Appendix B sheet altered after ISLT inspection completed. l Procedure / Document Control CR 96-997 Requirements contained in Safety Evaluation JPN-PSL-SEMS-96-026 were not completely transferred to 1-LOI-MM-46 for underwater repair of l the reactor vessel flange. i CR 96-1941 U-2 Switchgear has out of revision power distribution breaker lists posted.
CR 96-1821 OP-1-0110056 Rev. 22 - Procedure instructions do not match Data sheet 1, lines 2B and 3E for the same figure.
4 CR 96-1892 Nine open " Discrepant Field Condition" documents were found. These documents were written in 1993 and 1994.
. i l
e Findings l l
4 1. Several instances were observed in which Radiation Workers did not follow
- approved plant hesith physics procedures. (See Page 24)
- 2. Nine instances ofimproper configuration control of plant valves were )
identified in connection with PCM 153-194. (See Page 26)
- 3. Modification of the Unit 1 excore nuclear instrumentation was allowed to proceed to completion despite multiple indications of design process breakdown. (See Page 28)
- 4. In two cases, Condition Report dispositions provided corrective action of unsatisfactory depth. (See Page 30)
1 i l AUDIT REPORT
-QSL-PM-96 --
Page 5 of 40 l summmmmmmmmmmmme 1
- 5. In three cases inadequate inter-departmental coordination has caused i Quality Assurance audit Finding responses to have unsatisfactory content
{
(See Page 34) l
.c Operations:
PMON 96-028 - Health Physics Practices - Unit 1 Outage This PMON was conducted to evaluate radiological work practices during the Unit I refueling ,
outage. Tours were made of the Radiation Control Area to observe compliance with clothing l requirements, handling of material, Radiation Work Permit (RWP) limitations, contamination I control, survey instrument and dosimetry use, postings, pre-job briefings, hot particle areas, diving operations, resin transfer, Special Nuclear Material control, and Health Physics l Technician and Radiation Worker field activities. I The results of these observations were generally satisfactory. Observation of discrepant conditions were documented in the following condition reports:
- 1. CR 96-607 Radiation worker practices are not in accordance with requirements. l Observations included workers moving potentially contaminated material l without using protective clothing, a radioactive material drum stored next to I the Unit I containment ramp with rust holes in the side and material within l it not bagged. I CR 96-607 was closed on 5/30/96 but continued observations of poor worker practices prompted the issuance of a Quality Assurance Finding. See Finding
- 1 below.
- 2. CR 96-790 Inconsistent direction by Health Physics in the disposal of cotton liners.
Conflicting guidance was being provided by the HP generic containment pre-job briefing and HP personnel concerning whether or not removal and disposal of cotton liners was required prior to leaving the posted
, contaminated area outside the containment air lock. Further investigation showed lack of consistency in the understanding of HP personnel about management expectation in this area. This condition report was closed on 5/30/96. Repetition of this problem was not observed.
AUDIT REPORT QsL.PM-96-M_- -
Page 6 of 40 Other poor worker prer:tices noted during this PMON included: movement of potendally contaminated and/or adioactive material to non <iesignated areas (not rosted), opening ofdesignated storage areas without Health Physics assistance (no prior contact of Health Physics ), a worker
" handling a hot particle bag (bag open) in the drumming room in an area that was not designated as a Hot Particle Area, and handling of contaminated items without adequate protective clothing.
Discrepant areas from previous findings in the Health Physics area were re-reviewed during this PMON. These included: logging out of instruments for use. pcumg of radiation areas arid movement of Special Nuclear Material. These areas were found to be satisfactory, but will be monitored on an ongoing basis.
Selected portions of the Radiation Controlled Area were walked down using copies of the latest surveys to verify posting requirements based on the survey results. The survey verification was conducted at various times during the outage. During this activity doors locked for radiological purposes were verified to be secure. It was noted that several locking gates on Unit I are showing signs of wear. In time these gates may not be able to successfully perform their function of 1 excluding personnel from necessary areas. Conditions noted were loose hinges, missing or loose j bolts and doors bent up to about 15 degrees at the lock area. This was discussed with the Health Physics Supervisor and he indicated that this condition had been noted but that correction was postponed for budgetary reasons. CR 96-2083 was issued to document this problem.
An event related to Special Nuclear Material control event at Salem (excore fission chambers lost and may have been shipped to Barnwell for burial without documentation ) was reviewed for applicability to St. Lucie in Perfe:mance Monitoring Report 96-031 with no discrepancies noted.
Performance Monitor: Joe Walle with contributions from other QA personnel PMON 96-032 - Steam ';enerator Tube Plugging Activities ABB/Framatome This PMON was conducted to observe Unit 1 Steam Generator tube plugging operations and related activities. The following work activities were reviewed during the Unit One Outage:
- 1. Eddy Current Testing (ECT).
- 3. In-Situ Hydro Testing.
- 4. Tube Plug Installation.
- 5. Eddy Current Data vs Plugging List Verification.
- 6. Post Completion independent Review of Tube Plugging Activities.
s AUDIT REPORT Qs),PM-96-17 Page 7 of 40 A Nuclear Assurance team of QA and QC personnel was assembled to monitor steam generator tube plugging activities. The plan for this surveillance was to conduct 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> observations during S/G p!ugging for a sufficient period of time to establish a confidence level in vendor procedure compliance and performance. The 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> coverage was established with two people per shift (3 shifts) and continued for the first ten days of plugging activities. With approval from the Facility Review Group, coverage was then cut back to one person per shift, and subsequently to one person overall.
The plugging process included five specific steps, three of which required ABB/CE QC verification and sign-off. ABB/CE Traveler PSL-001, used to control the activity, also required thm a position verification (PV) be performed at least every 10 tubes during the marking activity. Additional PVs )
were performed when requested by FPL. FPL QA/QC documented observations on the process steps witnessed. Adherence to procedures and attention to detail by ABB/CE operators and QC were 1 considered good. Any questions or concerns raised by FPL QA were addressed promptly and professionally. 1 In-Situ Hydro Testing was conducted on a small number of S/G tubes in accordance with ABB/CE I Traveler PSL-007 and FPL Safety Evaluation JPN-PSL-SEMP-96-052,"In-Situ Hydrostatic Testing of St. Lucie Unit 1 Steam Generator Tube Flaws." This testing was observed by ABB/CE. FPL/CSI i and FPL/QA Personnel. The testing was also observed by an NRC Inspector from Region II. The l
above procedure, as well as safety precautions, were strictly adhered to during this testing. As required, Control Room personnel were kept informed throughout the period during which testing was being conducted.
During plugging operations, three Condition Reports were written by FPL QA/QC. Two were the result of three mis-marked tubes detected during subsequent process steps. Marking operations were suspended until root cause was determined and corrective actions taken. Corrective actions included correcting a zone boundary (row 50) and requiring that Genesis fixture calibration checks be performed when an " arm break" was required due to movement from low rows to high rows in the '
S/G. Training was held to ensure all operators understood these requirements. The third CR written dealt with a procedure inadequacy in which the operator was not directed to verify that a vacuum attachment was tumed on when using a cutting tool. A procedure change was generated to add this step.
At the request of CSI. PSL/QA conducted an independent verification of the accumulated ECT data prior to generation of the final plug list. The intent of this verification was to ensure that available documentation was that from the final analysis of the subject S/G tubes. and that codes warranting tube plugging had resulted in placement of the identified tubes on the final S/G tube plugging list.
As a result of this independent verification. it was determined that the final tube plugging list issued
AUDIT REPORT
_ _ ._ _. _ _ _ . . _ _ _ _ _ _ __ _ _ ._ _ _ _ _ _ __ . _ Q s t, P M-96-G Page 8 of 40 i
)
l 6/27/96 accurately denoted the S/G tubes to be plugged based upon documented test results and !
analysis.
As a result of errors during data transposition subsequent to the generation of the final plugging list.
and a deficient final QC review by ABB/CE, it das later found that one S/G tube plug and two stabilizers had not been installed as required. The plug was identified during the final FPL/CSI document review. Prior to the completion of CSl's review, ABB/CE QC again reviewed their documentation, and identified two missing stabilizers. Due to these three errors found, FPL/QA again independently verified the plugging documentation. No other plugging operation errors were identified. Independent Technical Review ITR 96-017 was issued to provide details on the QA l review and root cause investigation associated with the detection of these errors.
To facilitate the installation of the plug and stabilizers associated with the discrepant conditions, a video monitor and other related plugging equipment was set up on the 18' elevation of Contamment.
The installation was performed by ABB/CE personnel with verification by ABB/CE. FPL/QA also ;
verified that correct tube locations were being addressed. These installations and final reviews concluded the Steam Generator activities for the Unit 1 Cycle 14 outage.
l l
Performance Monitor: Lee Bearror PMON 96-043 - Conduct of Operations - July This PMON was conducted to observe various areas of performance by the Operations Depanment and ensure compliance with St. Lucie Plant Policies and procedures during the July / August timeframe. Observations included both outage and non-outage activities. Day, Peak, and Mid-Shift i activities in the Control Rooms and in the power block were observed during this period. Several l deficiencies were identified:
- 1. CR 96-1821 was written by QA to address a procedure discrepancy identified during Unit 1 MTC testing. A procedure inconsistency in Op 1-0110056 R22 existed which referenced two line items contained in Data Sheet 1 of that procedure incorrectly. The data sheet content was correct, but the line numbers mentioned in the procedure were not accurate.
Also addressed was an inconsistency found in the Plant Physics Curve Book. Four Physics
, Curves did not show a revision number consistent with the index. When the Curves were prepared. the revision number had been omitted. These two concerns are documented in the CR are are being addressed by the Reactor Engineering Department. <
- 2. CR 96-194I was written by QA to address out-of-revision Power Distribution Breaker Lists posted on larce (i.e. 480V. 4.16kV, and 6.9 kV) electrical equipment in the Unit 2 Turbine
1
% \
9
_ __9 AUDIT REPORT
_QSIPh%17- -
l Page 9 of 40 l Switchgear Room. Administrative procedure AP l-0010720 " Unit 1 Power Distribution i Breaker List" no longer requires these lists to be posted on specified equipment. The postmgs found had never been removed. The Unit 2 ANPS was informed of this condition.
The CR recommended that a check of other electrical equipment be conducted during l normal operator rounds for any other out-6f-revision postings that may still exist.
l
- 3. Finding #2 (see below) was written to address discrepancies noted in the area of locked valves. It was noted during a plant tour that six locked valves related to the Unit 1 Waste
, Monitor Tanks did not have " Locked Valve" tags attached. I Paragraph 8.1.11 of Operations Guideline No. 0G-012 states the following: "All plant
- valves listed as LOCKED on Flow Diagrams (P&lDs), Valve Lineups, and Procedures shall l have the appropriate locked valve tag attached with the identifying tag." After reviewing ;
the PC/M package that had changed the SRD status of the noted valves to " Locked Status," )
a walkdown of random locked valves listed was conducted. One other valve was found !
without a " Locked" tag attached. Also found during this walkdown were two valves I associated with the Unit One Main Steam Isolation Valves in a locked position status that was inconsistent with that shown on the applicable drawing. The Drawing shows them as i locked open and plant procedures list the valves as locked closed. The Valves are tagged l
and maintained consistent with plant procedures, in a locked closed position.
1 Special instructions associated with the CRN disposition that had changed these valves to l locked status, stated the following: " Plant should review applicable plant operating i procedures to assure consistency with this CRN and plant drawings." A review of this nature would have addressed problem described above. A CC Mail containing the details contained in Firding 5 was prepared and immediately forwarded to the Operations Supervisor and the Operations Support person responsible for tagging issues. Finding 2 (see below) provides a further description of this problem.
Throughout the period covered by this report, close attention was paid to the performance of the Control Room staff. Areas such as tumovers. communications. reactivity manipulations. key control, procedure use and compliance, and other Control Room functions were observed and found to be satisfactory. Use of the phonetic alphabet was noted to be greatly improving since its ,
establishment as a required communications method. Also accomplished by QA during this period was an independent verification of the identification of fuel assemblies stored in the Spent Fuel Pool that wer; to be retumed to the reactor and independent verificication of the final mapping associated with the Unit One core load. These activities verified fuel and CEA serial numbers, locations and orientations consistent with the FRG approved reload core map. All activities reviewed with the exception of those noted in the paragraphs above were found to be satisfactory.
e l.
i ___ _ _ . _ _ _ __
AUDIT REPORT QSL,PM-%1"L _
FPL. Page 10 of 40 Performance Monitor: Lee Bearror PMON 96-047 - Zach Pate INPO CEO Letter Verification This PMON was conducted to determine PSL conformance with the J. Broadhead response letter to Z. Pate, INPO, dated 5/15/95. This letter regarded Mr. Pate's speech entitled "The Control Room. " A checklist of forty-three selected response letter statements was researched to verify PSL conformance with these statements. Of these forty-three statements researched, three items requiring comment were identified. The identified items are as follows:
- 1. Reference to procedure AP 0010120. " Conduct of Operations" is found in several response statements. Not all of the stated information is currently contained in AP 0010120.
Although accurate at the time of the response, subsequent revisions to the procedure have relocated some information to various " Operations Policies" or to Q1 1-PR/PSL-2,
" Operations Organization." All topics were verified to exist in these alternate documents.
- 2. Statement 14 - Although Turkey Point uses the STAR method of self-verification, St. Lucie uses the STOP me' hod, which is very similar. The PSL Operations Manager plans to convert to the STAR method in the future.
- 3. Statement 18 - The commitment statement refers to main Control Room board access authorization by the Control Room SRO. PSL policy does not distinguish between Control Room access and control board access. Additionally, access to the PSL Control Room may be granted by a Control Room crew member who may be either an RO or SRO. CR96-1902 l was generated by QA to address the lack of specific SRO authorization to approach the I control board area. !
As part of the verification process, interviews with Operations and Training Department personnel were held. Observations of simulator training sessions, and Control Room activities were conducted.
Applicable document reviews were also completed. Operations and Training department personnel were found to be very helpful during this research. With the exception of the three differences listed.
all statements were found to be accurate. I Performance Monitor: Lee Bearror l
l
s AUDIT REPORT QSL-PM-96-17 Page 11 of 40 Maintenance:
PMON 96-039 -Inspection and Repairs of K-Line AC Circuit Breakers - 10 CFR 21 This PMON was initiated to evaluate actions associated with a 10CFR Part 21 report regarding !
480VAC Circuit Breaker Solid State (SS) Trip Devices manufactured by ABB. This work activity I was conducted during the Unit 1 Cycle 14 Outage. 1 i
In accordance with PSL Administrative Procedures, CR 96-685 was initiated by the PSL Corrective Action Group to address the Part 21 Notification provided by JQA-96-518. Engineering Evaluation JPN-PSL-SEES-96-029 identified plant components that were affected by the Part 21 Notification i and established a plan to inspect and evaluate affected trip devices for defective conditions similar l to those found by the manufacturer. i QA monitored initial inspections of several 480V circuit breakers to ensure that manufacturer specific instructions for inspection of the solid state trip device were followed. Based upon initial evaluation of the conditions discovered, which were arclike cracks developing around soldered pins (considered to be precursors to faihue), a conservative approach was selected and the decision was ,
made to commence a replacement program for each breaker unit whose noted Serial Number l sequence was specified by the manufacturer Part 21 Report. l During the Unit 1 Outage, fourteen 480V circuit breakers were worked, either to replace the suspect l SS Trip Device, or replace the er.m e breaker with an available spare breaker that had a satisfactory SS Trip Device installed. He work activity was well documented and confonned to the program requirements of the maintenance work process. It was confirmed by QA that the replacement SS Trip Devices conformed to the material requirements of PC-1 applications.
Further activities associated with this Manufacturer Part 21 Report are being tracked through the PMAI process at PSL. PMAls # PM-06-068,069,070 and 071 have been initiated by JPN to accomplish this task.
1 Performance Monitor: Lou Panessa Sen ices /Encineerine:
PMON 96-046 - Compliance With Overtime Limit Requirements - July 96 This PMON was performed at the request of the St. Lucie Site Vice President as a spot check of site j compliance with plant overtime guidelines. Gate logs and payroll time sheets were obtained and I reviewed for fifteen specific 1&C personnel for the period of 6/24/96 through 7/6/96. !
l
AUDIT REPORT
_ _ _. - QS L-P M-96-4 7 Page 12 of 40 l
i No violations of the established overtime limits were identified. Most of the personnel within the I target population were working 12 to 13 hour1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br /> shifts but did not violate the overtime limits when ,
consideration was applied for shift tumover and lunch breaks. Tumover time as noted on the time !
sheets was consistent at oi,e hour per day, i.e.,30 minutes before and after shift.
Overtime checks will continue in the future.
l Performance Monitor: Joey Lowery l l
r PMON 96-052 - PC/M 009-195 Excore NI Drawer Replacement This PMON was performed to review the overall performance of the FPL program for design control with respect to PC/M 009-195 "RPS NI Drawer Replacement" PC/M 009-195 was performed to replace the Unit I excore nuclear instrumentation system with an updated system designed by Gammametrics Inc. Included in the scope of the replacement were source range / wide range detectors, containment pre-amplifier assemblies, and the signal processing drawers in the nuclear instrumentation panels. A replacement of the same type had been successfully completed on Unit 2 during the last refueling outage. I 1
The organization responsible for generation of the new design was the FPL Nuclear Engineering Department (JPN). Manufacture of the new drawers was contracted to Gammametrics Inc. and I installation of the system was performed by the PSL Instrument & Control Maintenance (ICM)
Group.
Implementation of the PC/M was undertaken during the Unit 1 Cycle 14 refueling outage. From the outset of the implementation activity, difficulties were encountered. Gammametrics had difficulty meeting the fabrication schedule deadline. The FPL engineer with primary responsibility for the project left the company in December 1995, four months prior to the start ofinstallation of the new system.
During the implementation period, installation and system performance problems were encountered, largely traceable to the fact that FPL had provided improper design information to Gammametrics in the ppecification for the project. Problems included at least one case of rolled wires in Gammametrics technical drawings. Approximately half way through the implementation period.
the FPL Instrument & Control Supersisor with primary responsibility for the project left the company.
AUDIT REPORT
_ ggi,-FM-96-17 -- -- --
Page 13 of 40 As the project progressed towards conclusion, noise problems in the source range / wide range instmments presented a significant obstacle. This were due to several factors, including cable termmation problems deriving from the incorrect specification of previously installed cables. At least two cases were encountered in which the new design did not provide the proper output signals for the reactor protection system functions. Following reactor start-up, and power ascension, flux shape testing by reactor engineering disclosed that the detectors for all four power range safety channels had been improperly wired to the new instrumentatior One of the four detectors had an I offsetting error in the detector wiring, and as a result provided me correct indication.
1 A great deal of change paper was produced during implementation of the modification. Thirty two Change Request Notices (CRN) were written against the PC/M package. The installation work order used twenty scope changes. and four separate work orders written for troubleshooting purposes during the installation. The pre-operational test recorded forty six deviations. including some which were used to change acceptance criteria within the test. Workers on the job frequently complained l about the unmanageability of the implementation documentation. '
1 Concem was evidenced by several parties due to the nature of the problems encountered, their
~
source, and the marginal level of compliance with the quality assurance program requirements during implementation activities. Fourteen Condition Reports were written during implementation, four of these by Quality Assurance (QAK The NRC performed a reactive inspection in response to one of the CRs written by QA.
On the basis of overall review of this project performed both during installation and during this PMON, it is concluded that implementation of the quality assurance program was not effective in activities associated with PC/M 95-009, and that there.were significant warning signs of quality program breakdown throughout the implementation period. These warning signs were not properly heeded.,either by the design organization, or by the implementing organization, or by the Quality Assurance Group. Significantly absent, were the presence of a questioning attitude and a concern for generic implications in cases where discrepancy was located. Finding 3 details specifics of non-compliance with quality program requirements.
Performance Monitor: Dave Lowens PMON 96-ed - Correcti iction - July 1996 This PMON is the second in a series of reviews designated to examine the St. Lucie Nuclear Plant Corrective Action Program. This PMON examined the timeliness and effectiveness of corrective action associated with Condition Reports. Seven Condition reports were reviewed, in a sample that
AUDIT REPORT QSLPJ4-96 4 __ -- --- -
Page 14 of 40 represented all severity levels. Several problems were identified, both with the Condition Report process and with two individual Condition Reports:
Timeliness of and Accuracy of Processine Two discrepant conditions were identified in this area:
- 1. As designated by AP 0006130," Condition Reports", the time period for CR processing I begins with the Plant General Manager (PGM) signature in Block 5 of the CR form and ends with PGM approval of the disposition in Block 13. Of the seven Condition Reports reviewed. six had not been processed within the time period required by their severity level designation. This problem was recently identified in Finding #1. of audit QSL-OPS-96-13, ,
" Corrective Action." Corrective action for this Finding is currently in progress. The !
Finding will not be ased until the corrective action is proven to be effective and the absence of future problems is verified.
- 2. Condition Report 96-1516 was written to document the alarm points associated with a security door having failed in the alarmed position. In the CR, the Nuclear Plant Supervisor indicated in Block 4 " Operability /Reportability Determination", that the problem was potentially reportable. During the PMON it was learned from the security information report, that an officer arrived at the problem area three minutes after the alarm began and i found the c'oor to be locked and secure. For this reason. the event was found to be neither I logable nor reportable. The disposition of the Condition Report was reviewed, and found to be very thorough concerning description of the failed alarm. but neglected to l
- satisfactorily address the indication of potentially reportability as indicated in Block 4.
This is considered to be a minor lapse in attention-to-detail, but undesirable in a document that becomes part of the plant quality record system.
Condition Reoort Discosition l In two cases reviewed. Condition Report disposition was unsatisfactory in that it lacked required depth ofinvestigation.
l
- 1. Condition Report 96-1543 was wrinen to document a discrepant result associated with the Unit 1 Polar Crane Load Test. The CR was written because the final weight shown on the
, load cell used as the primary measuring instrument, was 5.6% lower than that required by the acceptance criteria for the test. An engineering evaluation associated with the CR disposition performed a calculation of test weight using data from a back-up, non-M&TE calibrated flow totalizer. This totalizer was intended to measure the volume of water added to Kevlar bags to monitor filling of the bags. The engineering evaluation states that the total test weight, as measured by the totalizer. was well within the acceptance criteria and
\
l AUDIT REPORT QSL-PM-96-17 Page 15 of 40 accuracy required by the test procedure. On the basis of this evaluation, the Unit 1 Polar Crane was concluded to have been successfully tested at approximately 125% ofits rated load.
The conclusion above is based on an assumption that the non-M&TE totalizer was calibrated within .5% ofits full scale rating and was operated within the maximum and minimum flow rates for which the device has known accuracy. Information concerning the flow rate during the water bag filling process was not included in the engineering evaluation.
Acceptance of the use of non M&TE equi pmer.t for this important test, along with a lack of attention to detail in ensuring that the equipment was operated within process botmdaries within it has known accuracy, calls into question the validity of the totalizer output reading.
The depth of the corrective action provided ir, the CR is evaluated as unacceptable due to a lack of substantive information pertinent to the actual flow rate during the filling process.
This problem is discussed further in Finding #4 below.
- 2. Condition Report 96-1434 documents a failure to provide adequate / correct design input for PC/M 009-195, the Unit 1 Excore Nuclear Instrument (NI) drawer replacement. This CR was written because of misidentification of actual field conditions concerning the presence of 16 gauge power wiring for the N1 amplifiers (18 gauge was actually installed),
misidentification of coaxial cable for installed signal cabling (portions of triaxial cable were actually installed), and an incomplete and incorrect design in the mounting et the NI replacement drawers.
The engineering evaluation in the CR disposition states that the misidentification of cable sizes and types was due to the use of data from a previously prepared Unit 2 specification.
The disposition states that root cause and generic implications of this problem would be programmatically addressed as part of a different Condition . Report. CR 96-1766. On this basis, without the generic implications or root cause having been analy:ed, the original Condition Report, CR 96-1434, was closed. In addition to the fact that closure of one CR based upon a promise of future action in another CR is not permitted by AP 0006130, closure under this circumstance introduced other substantial problems into the situation .
, This problem is discus:.; $ further in Finding d4 below.
Interdenartmental Coordination in Disnosition of Condition Renorts Associated with OA Findines.
AP 0006130 tasks the assignee of a Condition Report with responsibility for coordinating all actions associated with the disposition. Although not explicitly stated. the implication is clear that where
AUDIT REPORT
- QSL-PM-96-1-7 Page 16 0f 40 interdepartmental coordination and cooperation are necessary, the assignee has responsibility for accomplishing this function.
" This process has not worked well in the disposition of CRs associated with several recent Findings issued by the QA Department. In several instances, it has been clearly demonstrated that Departments not assigned direct responsibility for resolution of a Finding feel little incentive to participate in the process. even though they may have had a role in the causation of the problem. On the other side, Departments tasked with a Finding response do not feel that they are responsible for providing a response that includes actions' that are outside of their area of departmental responsibility. These problems were clearly demonstrated in the actions surrounding three recent responses to QA Findings. This problem is further discussed in Finding #5 below.
Overall Summary Corrective Action at St. Lucie is still a troubled area. The Condition Report process, while processing a high volume of documentation, lacks the elements of compliance with administrative requirements, and a sufficient level of quality in disposition activities. It is noted that efforts are in progress to address these areas. However the results of these efforts were not yet evident in the cases examined by this PMON.
Performance Monitors: Joey Lowery / Dave Lowens a
PMON 96-049 - Examination of Maintenance Rule SSC Historical Review This PMON was performed to evaluate the process used for historical review of plant structures systems and components (SSC) for assignment to their current status under the Maintenance Rule.
This review was performed as a follow-up for indeterminate status of this area identified during audit QSL-MR-014 " Maintenance Rule Compliance."
Two systems were chosen for review: Unit 1 Auxiliary Feedwater. and Unit 2 Low Pressure Safety Injection. In each case, documentation generated by the completed historical review was evaluated t
and compared to the present status of the system as reported in the 1996 Maintenance Rule Second Quarter Report.
Unavailability for both systems is monitored for purposes ofINPO reporting. The process for collection of this data has been in existence for a significant period of time and has now been adapted to the collection of unavailability hours for Maintenance Rule purposes. No deficiencies were noted in this area.
i AUDIT REPORT
-QSIrPM 96 --
Page 17 of 40 l
Determination of reliability was accomplished by the analysis of maintenance preventible functional failures (MPFF) over a 36 month period. This was performed through collection of available In-House Event Reports and NPRDS failure reports for 53Cs that are part of each system. These
" documents were then distributed to the applicable system engineers for the analysis of the failure history.
This process suffered from several weaknesses. Lapses in NPRDS reporting had the potential, to introduce error into the input of the analysis. In many cases, the system engineers who performed the reviews had not been responsible for the systems during the time period being examined and were limited to the information described in the NPRDS reports for detail on the nature of the failures. In addition, this review was performed by system engineers during a period when many other priority uses existed for their time. ,
The preceding deficiencies were counterbalanced to some extent by oversight from the Maintenance Rule Administrator and his staff, in some cases where information was questionable, conservative assumptions were made. In this regard it is noted that the 1C Auxiliary Feedwater Train is currently in a (1) status due to three maintenance preventible functional failures during the period exarr.ined.
Documentation of the completed historical review is poor. Copies of the PMAls issued to the system engineers are maintained in several binders, together with marked up copies of the applicable l NPRDS reports. Some of the markings on these reports are questionable in nature and require verbal l explanation from the Maintenance Rule Administrator to resolve questions that they raise. l No errors were detected in the status of the two systems reviewed by this PMON. However the overall posture of this area is consistent with the evaluation of marginal regulatory compliance deported in the report for audit QSL-MR-96-14. Personnel contacted were aware of the deficiencies i l
in this area and stated that they were the result of the speed with which the program was required to be assembled, combined with the lack of comprehensive system engineer coverage during the necessary period. Actions are currently in progress to refine the documentation associated with the Maintenance Rule Program and strengthen the PSL system engineer program.
Performance Monitor: Dave Lowens I Independent Technical Review (ITR) Activities
- 1. Independent Technical Review Activities in Progress
AUDIT REPORT QSL-PM % 17 -
Page 18 of 40 The following ITRs are currently in progress.
4 e 196-016 " Review of Cranes and Hoists" e 196-020 Review of LER # 335/96-007, " Inadvertent Start of the 1B Emergency Diesel Generator During "B" Channel Containment Isolation Actuation Signal Testing Due to ProceduralInadequacy" e 196-022 Review of LER # 389/96-002. " Manual Reactor Trip Due to High Main Generator Cold Gas Temperature Caused By Valve Failure" II. Independent Technical Review Reports ITR 96-018 was conducted to perform an independent technical review of the causes and corrective actions listed in licensee event report (LER) 335/96-005, " Wide Range Nuclear Instrumentation Channel Inoperable When Required to be in service for Fuel Movement," dated June 13,1996, 1
On May 14.1996, with St. Lucie Unit 1 in Mode 6 and the reactor being defueled. Operations personnel performed a functional surveillance test of the nuclear instrumentation used to monitor count rate during fuel movement. This test. conducted in parallel with fuel movement, rendered one of the two nuclear instrument channels required by Technical Specification 3.9.2, inoperable. The '
Reactor Engineering Supervisor suspended the fuel assembly removal when the recorder used to monitor neutron flux showed an increase in counts. Further core alterations were suspended until it was determined that the increase in counts had been caused by the surveillance in progress.
4 The ITR engineer determined that the LER contains the information required by 10CFR50.73, including: operating conditions before the event, a description of major occurrences. component failures corrective action taken and planned, and a narrative description of what occurred and its j known causes. Also, the LER was submitted within the 30 day time limit required by the CFR.
A review of the event causal factors and the recommended corrective actions documented in LER 335/96-005 determined that the corrective actiom taken to prevent recurrence effectively addressed
)
the specific event initiating causes. includinc
. nnel performance and generic issues conceming
, placing equipment out of service during surveillance testing. However. generic implications relating to inadequate procedures were not identified. Although Operating Procedure 1-120051 and 2-120051, " Wide Range Nuclear Instrumentation Channel Functio-al Test." and Pre-operational Procedure 3200090. " Refueling Operation." were revised to add pre.autionary steps and count rate 2
l l
I
l AUDIT REPORT QSL-PM-96-17 Par,e 19 of 40 recorder monitoring requirements, the LER did not note that procedural inadequacy contributed to the event.
" Procedural inadequacy as a contributing factor was previously discussed in ITR 96-012. " Trending of PSL 1 & 2 Licensee Event Reports From January 1994 through March 1996," dated May 16, 1996. ITR 96-012 documented that 19% of the causal factors documented in the LERs were related to procedures. A breakdown of these problems determined that 82% were caused by inadequate procedural guidance or a lack of detail. Procedure problems continue to be a factor effecting plant performance, and continued management attention to the ongoing effort to upgrade site procedures ,
is warranted. I
)
ITR Engineer: C.E. Norris i
ITR 96-019 was conducted to evaluate the causes and corrective actions for the loss of contamment audible neutron count rate indication. Full details of this event are included in licensee event report !
(LER) 335/96-006, " Inadvertent Loss of Containment Audible Count Rate Indication Due to Procedural Deficiency," dated June 26,1996.
On June 1,1996, with St. Lucie Unit 1 in Mode 6 (Refueling), wide range audible neutron count rate indication in containment was temporarily rendered inoperable. Technical Specification 3.9.2,
" Refueling Operations, Instrumentation " requires that audible neutron count rate indication be in operation in Mode 6 whenever core alterations or positive reactivity changes are being made. The loss of the audible neutron count rate occurred when the electrical bus (1B2 MCC) supplying power to the containment audio amplifier was removed from service during switching operations, in support of a maintenance activity to change out the bus supply breaker. The power supply and audible neutron count rate indication were restored after approximately five minutes.
An independent technical review of the LER determined thst the report contains the information required by 10CFR50.73, including: operating conditions before the event, a description of major occurrences. personnel errors, corrective action taken and planned, and a narrative description of what occurred and its known causes. This review also verified that the LER was submitted within the time limit (30 days) required by the CFR.
An independent review of the Control Roo.n and refueling station chronological logs determined that the LER accurately reported the details of the event. Additionally, a review of the procedure goveming removal and restoration of buses. Operating Procedure 1-0910024. " Cross tying / Removal / Restoration of 480V Buses." Revision 7. and the unit 1 breaker list. Administrative Procedure 1-0010720. "llnit 1 Power Distribution Breaker List." Revision 10. determined that
AUDIT REPORT
-QSL-FM-96-17 Page 20 of 40 critical components, that would be lost when the 1B2 MCC was removed from service, were not adequately identified. The corrective actions included in the LER adequately address applicable event causal factors and generic implications. When fully implemented, the corrective actions will also establish secondary barriers to prevent recurrence.
a ITR Engineer: C. E. Norris ITR 96-021 was conducted due to indications of an unanticipated deficiency in the design.
installation, or operation of a safety related system. An independent technical review was conducted !
to evaluate the causes and corrective actions for licensee event report (LER) 335/96-008
" Inadvertent Actuation of the SIAS and the CIAS Due to Loss of the 15 VDC Regulated Power Supply During Maintenance," dated July 8,1996.
On June 8,1996, with the unit in operational mode 6 and the "B" side electrical train out of service for maintenance and testing, an inadvertent actuation of Channel "B" Safety Injection Actuation Signal (SIAS) and the Containment Isolation Actuation Signal (CIAS) occurred. He actuation was caused by loss of the 15 VDC regulated power supply during maintenance. During trouble shooting activities to determine the cause. Instrumentation & Control (I&C) personnel identified that the drawings in the vendor manual did not match the as-found wiring of the ESFAS cabinet.
Specifically, the vendor manual depicted the affected 15 VDC regulated power supply monitor card to be associated with the Containment Spray Actuation Signal (CSAS) and Recirculation Actuation Signal (RAS) functions. and not the SIAS function.
An Event Response Team (ERT) was formed to determine why the actuations occurred during the maintenance activity and to restore the system to normal operational status without further inadvertent actuations. Examination of the affected power supplies (S3 and M3) discovered a blown fuse in the M3 power supply and the S3 power supply output near 0 volts due to crowbarring (an inte: 'al protection feature that shunts the output on high voltage). Based on the sequence of events and the as found condition of ESFAS (i.e., there were no indications ofinitiating signals / alarms present and the actuation occuned during the replacement of power supply monitor cards A10 and Al1) the ERT concluded that the spurious Channel "B" SlAS and CIAS were the result of the maintenance activity and not a coincidental external event.
Personnel from I&C. Site Engineering, and the ESFAS cabinet vendor inspected the "B" ESFAS cabinet to determine wiring differences. To fully determine wiring differences the scope of the inspection included the "A" ESFAS cabinet. An evaluation of the cabinet wiring against the vcador's wire lists demonstrated that the as found field condition of the ESFAS cabinets was correct, and that the cause of the unanticipated result of the loss of two auctioneered power supplies was that
I O AUDIT REPORT
_ gg[,pg.96 -
Page 21 of 40 the drawings supplied by the vendor, (i.e., the design drawings of record), did not represent the actual power supply wiring in the ESFAS cabinets. Condition Report (CR) 96-1323 was written to document the wiring problem. Engineering learned, through discussions with the vendor, that the
" wrong revision of the drawings may have been sent with the cabinets. However, the Engineering root cause evaluation, included in the evaluation for'CR 96-1323, did not confirm that this was the root cause. The evaluation stated: "The root cause of the unanticipated result of the loss of the two auctioneered power supplies was thet the drawings supplied by the vendor did not represent the actual power supply wiring in the ESFAS cabinets. The vendor believes that the wrong revision of the cabinet drawings may have been sent, however, this potential reason for delivery of the wrong drawings has not been confirmed."
Corrective action resulting from CR 96-1323 included re-wiring PSL Unit 1 ESFAS cabinets to the configuration specified in current design drawings of record. However, an independent review of the inadvenent actuation by QA found that, if the drawing was determined to be incorrect, then re-wiring ESFAS to conform with the drawing constituted a design change. The ESFAS cabinet re-wiring was completed without the use of an approved Engineering design package or a 10CFR50.59 review to determine that the changes did not constitute an unreviewed safety question. This modification ~was performed and documented using the PWO process. However, this work should have been performed under the controls of an Engineering Package to ensure that design verification, documentation, and configuration control requirements applicable to safety related equipment were met. Also additional discrepancies with respect to installed configuration and design drawings may exist. The generic implications identified in the corrective action developed for CR 96-1323 were limited to channel"A" Unit 1 ESFAS cabinet and the Unit 2 ESFAS cabinet. Generic implications should be expanded to include checks of all Eaton/ Consolidated Controls equipment installed in a safety related application that were manufactured / installed during the same time frame as the Unit 1 ESFAS cabinets. These concems were discussed with Site Engineering personnel and condition report CR 96-1920 has been issued to address them.
ITR Engineer: C. Norris ITR 96-023 was conducted due to indications of a tampering event, discovered on August 14,1996, involving systems or components that could affect nuclear safety and a similar event, discovered on July 26,1996, that involved 9 padlocks / cores. and 2 door locks / cores, containing foreign material, that effected both units. This review was conducted to evaluate the effectiveness of the plant's preparedness to evaluate, respond and implement compensatory actions for these types of events.
On August 14.1996. with Unit I and 2 in Mode 1 at 100% power. the I&C maintenance department was conducting the ESFAS monthly functional surveillance on Unit #2. Key switches for the Safety
1 AUDIT REPORT
. QSL-P-M-9647-Page 22 of 40 Injection Actuadon System (SIAS) block circuits, located on the Hot Shutdown Control Panel 1 (HSCP), were found to be inoperable due to the presence of foreign material in the keyhole. Based on a suspicion that the key switches had been tampered with, a decision was made to inspect the tait l
- 1 HSCP for evidence of similar tampering. This inspection revealed key switch 1-CSI17 which operates the Power Operated Relief Valve (PORV) (V1404) on the unit #1 HSCP also contained foreign material.
Independent review determined that the actions taken by the plant for the August 14 th tampering event complied with the requirements of 10CFR73.55(h), " Response requirement" and the St. Lucie Plant Safeguards contingency plan. These actions included: declaration of a security alert and an unusual event. NRC notification, informing Local Law Enforcement Agencies (LLEA), increasing the security of the site by restricting visitor and vehicle access, and verifying that perimeter gates are locked or guarded. Additionally, compensatory actions taken to heighten security awareness and post security officers in normally unoccupied areas, such as the' electrical equipment rooms, effectively isolated the tampered components and provided an effective deterrent to additional tampering during the investigation. After the initial plant inspections were completed, the continuous coverage provided by the posted officers was transferred to random coverage by two
{
roving patrols in each unit. Nuclear Assurance personnel performed random spot checks of the I security patrols and found coverage of unoccupied areas to be satisfactory.
Plant personnel determined through examination and test of plant structures, systems, components and review of past maintenance history, through May 22,1996, that the tampering was limited to the locks and cores identified on July 26.1996 and the HSCP PORV control and SIAS block switches identified on August 14, 1996. Engineering evaluation, JPN-PSL-SENS-96-0383,
" Evaluation of Equipment Tampering On Safe Plant Operation." dated August 19.1996 determined that, for the identified tamper conditions, safe operation and safe shutdown of the plant were not affected. The evaluation also concluded that cooldown from the HSCP would be delayed while repairs are made; however, delays are permitted for repairs of safe shutdown equipment as outlined in the Final Safety Analysis Report. Section 7.4. This engineering evaluation provided assurance that the tampering had negligible impact on plant safety and justified continued plant operation. The ITR Engineer determined through review ofinspection and test documentation and JPN-PSL-SENS-96-0383 that the process had identified the scope of the tampering and had verified that the plants could be operated safely during the event and subsequent investigation.
This independent review determined that current plant and security procedures provide only minimal guidance for response to acts of tampering or vandalism. Deliberate acts of tampering at other plants in the industry and those discovered in July and August 1996 at St. Lucie illustrate that the potential for this type of activity at the St. Lucie Plant must be recognized and prepared for. Independent review determined that the response guidance provided in the " Safeguards Contingency Plan."
1 1
- AUDIT REPORT QSL-PM-%-17 Page 23 of 40 l
1 Revision 11 and " Security Procedure, SP 0006027, " Safeguards Contingency Plan Implementing !
Procedures," Revision 27, should be enhanced to provide more detailed instruction and decision processes. Specific guidance should be provided to commit resources and define responsibilities.
Procedure enhancements addressing response activities to determine the scope of the event and its l effects on safe operation shoula include: '
e Verification of system or component alignment / condition e Affected system control logic checkout
- Instructions to determine interrelated systems and direct inspections of them l Inspection of in plant and Control Room panels, safety-related electrical panels and switchgear e A historical review of condition reports and plant work orders to determine if other tampering may have occurred that should be included in the inspection, examination, and evaluation scope e
Performance of selected surveillance activities to verify. system or component operability e A decision matdx to aid in determining the classification of security events and to increase i the sensitivity to the potential of tampering or sabotage events occurring at PSL Independent review of the actions taken in response to the July tampering event determined that actions taken were not sufficient to bound the event and evaluate the affect on plant operation. The I
response to this event included only inspections oflocks and cores. Other plant equipment located near the tamper areas (i.e.. HSCPs) was not included in the examination scope. The ITR Engineer determined, through review of the July tampering issue and suspect precursor events related to missing relief valve tamper seals and failed generator gas temperature control valves, that implementation of the Safeguards Contingency Plan in response to the July event would have increased security posture and directed additional response activities to determine the full scope of the event and its effect on safe operation of the units. There has been a relatively low frequency of these types ofincidents in the industry which may have resulted in reduced sensitivity to tampering events. Reluctance to initiate and follow the contingency plan, for events considered as mischief, futher demonstrates the need for procedure enhancements to prompt additional safeguards activities including: additional inspections. tests and evaluations of other related plant systems. components, and structures to improve the response to tampering and vandalism issues. The ITR Engineer determined during review of the actions taken in response to the August event that the actions taken included and effectively bounded the issues resulting from the July event.
4 I
Recommendation 1 Review of PSL Administrative Procedure AP 0010509. " Personnel and Material Control." Revision
- 17. indicated that a potential weakness may exist which authorizes individuals continuous access to
AUDIT REPORT
_ _ - ._. _ _ _ - _ _ . qst.py.94,g 7 Page 24 cf 40 muse vital areas without a justifiable reason for such authorization. AP 0010509 directs Department Heads to review their Department's Protected Area and Vital Area access requirements and to ensure that such access is requested only as needed for individuals requiring the access to accomplish assigned duties. Guidance addressing specific criteria for ascertaining the need for continuous unescorted access is not provided. During an interview the plant security Access Control Supervisor disclosed that approximately 800 out of 1400 personnel badged at PSL have access authorized for at least one vital area. The supervisor also stated that this is an estimate and that the actual number is probably higher. It should be recognized that granting access to personnel for expediency or convenience increases the risk of sabotage and vandalism by inside personnel and reduces the probability that the perpetrator /s will ever be identified in follow up investigations. Actions should be taken to minimize the number of personnel granted unescorted access to vital areas of the plant. Specitic criteria should be established, such as restricting unescorted access authorization to personnel required for emergency response or performing a specific job on equipment located in the area. Consideration should be given to removing unescorted access authorization for those personnel with infrequent or administrative needs. Escorted access should be sufficient when infrequent access is needed. This is indicative of a potential problem in authorizing unescorted access that will be evaluated during future Quality Assurance PMON activities.
This review determined that the plant's response to the August 14,1996 tampering event complied with the requirements of the governing procedures and 10CFR73.55(h). These actions effectively bounded the equipment affected and evaluated the effects on safe operation of the plant for both the July and August events. Effects on plant operation from these events have been evaluated by engineering evaluation. JPN PSL-SENS-96-0383 and have been determined to have no adverse effect on the safe operation of the plant. One recommendation has been identified relating to improvements in security event response procedures.
Recommendation 2 196-023.001. Revise existing response procedures or develop new ones to provide for a comprehensive response and evaluation of tampering and vandalism issues.
Revisions should also include a security response and event classification decisions matrix to assist in accurate and timely classification of events.
. i Expected Responsibility: Security Department)
ITR Engineer: C. E. Norris
l 9
AUDIT REPORT
_g _ __ _ ___ -
---QSL-EM 96 17__ _ .
Page 25 0f 40
- nummmmm m e Findinge Finding 1:
Several instances were observed in which Radiation Workers did not follow approved plant health physics procedures.
m- 6 Criteria: HP-2 FPL Health Physics Manual. Rev.10 Paragraoh 3.1.1 Resoonsibilities. Individuals.
" Individuals requiring unescorted access to the RCA are responsible for:
- 1. Cooperating fully with the Radiation Protection Program for his own protection and the protection of others within the facility:
- 2. Maintaining their doses ALARA and minimizing the generation of radioactive waste;
- 3. Using protective and pbrsonnel monitoring equipment properly;
- 4. Following the HP instructions and procedures (including Radiation Work Permits);
- 5. Notifying management of any potential or existing radiological hazards or improper practices; . ."
Discussion: Observation of health physics practices during the Unit 1 Cycle 14 refueling outage i detected worker practices that were not in compliance with established plant procedures. These included the following examples:
l 1
- 1. Transferring a bag of material over a contamination boundary and receiving '
it with no protective clothing.
- 2. Moving material in the RCA to an area that was not designated as a Radioactive Material Storage Area.
- 3. Reporting to a work area without ennugh protective clothing or tools requiring personnel to again leave rad return to the work area. increasing ;
exposure. This is in violation of ALARA. '
- 4. Observation of the condition of some Locked High Radiation Area doors which indicates that they may have been climbed upon to gain access. This is based on the fact that they are bent about 15 degrees at the lock area. It must be noted that this condition may not be a recent occurrence.
- 5. Workers observed placing potentially contaminated scaffolding parts, from Connex storage boxes. on the ground outside of the designated Radioactive Material Storage Area.
The area Health Physics Technician or the Health Physics Supenisor was notified of each of these problems and corrective action was obtained. This Finding'is
1 AUDIT REPORT QSL-PM46-1-7
_ P P t._._ _ _ _
Page 26 of 40 written to obtain and document correction of the underlying causal factor that led to the occurrence of these events.
i l
These basic radiological practices appear to have been over looked. Supervision in the field is not reinforcing the requfrements and is sometimes not providing a good example to the other personnel.
Recommendations:
ne following recommendations are offered to aid you in responding to the finding.
However, additional or alternate actions may be necessary based on your investigation of the finding and the causal factors and generic implications identified.
Your response must address each of the five elements in the audit cover letter.
1.
Training and retraining must emphasize the fundamental practices with radiation workers and their supervisors. An opportunity for improvement would be to revise the training to provide fundamental practices to ali requalification classes.
- 2. Management must provide a clear statement of expectations and for proper l
l work practices in the Radiation Control Area.
- 3. Health Physics personnel should be diligent in their assigned areas and exercise their authority to stop work when procedures and RWPs are not being followed. '
i
I O
1 AUDIT REPORT
_ ___ _. p g _ _. ____. . _ _ .
QSL-PM-96-17 Page 27 of 40 Finding 2: Nine instances ofimproper configuration control of plant valves were identified in connection with PCM 153-194 Criteria: Onerations Guideline No. OG-012. Rev. 0-Parneranh 8.1.11 a
"All plant valves listed as LOCKED on Flow Diagrams (P&lDs), Valve Lineups, and Procedures shall have the appropriate locked valve tag attached with the identifying valve tag."
CRN #153-194-5001. Rev. 0-Soccial Instructions No. 2 '
l
" Plant should review applicable plant operating procedures to assure consistency with this CRN and plant drawings."
1 Discussion:
PC/M 153-194 was generated in part to effect changes to plant drawings regarding the " locked" status of several plant valves shown on Selected Record Drawings (SRD). As a result of this PC/M several administrative changes were incorporated in SRDs by various Change Request Notices (CRN) generated by the Operations Department .
CRN Numbers 153-194-4998 through 5002 requested that " Locked status"be designated on SRD's, for a number of valves based upon plant criteria for having valves locked. As a result of these CRNs approxiinately 166 valves had " locked status" added to the appropriate SRD's. Included in the CRN Dispositions was a "Special Instruction" from Engineering to the Plant (Special Instruction #2 above) on how to review the final PC/M product.
During a plant tour, QA identified the following six plant locked valves as not having
" Locked" tags attached along with the identifying valve number tags:
V06209 V06217 V06229 V06213 V06225 V06232 After identifying the first six valves without Locked tags attached. a random walkdown of other valves referenced in the PC/M package was conducted. During this walkdown an additional valve in the same status was identified.
FCV-25-6 During the same research effort. QA identified valves V08569 and V08579. which are actually Locked Closed valves. but shown on P&lD 8770-G-079 as Locked
}
AUDIT REPORT
___p g __ _ . _ . . _ _ . _ . _ . . _ _ _ _ . _ _ _ _ _ _ . - - - --._ _---- - -
--QSL.P.W41-7 Page 28 of 40 l
l l
Open. The Locked Open status shown on the drawing is as listed in the PCM {
package. i Recommendations:
. l
\
The following recommendations are offered to aid you in responding to the finding.
However additional or altemate actions may be necessary based upon your investigation of the finding and the causal factors and generic implications identified.
Your response must address each of the five elements in the audit cover letter.
1.
Conduct a review of all locked valves to insure that a " Locked Valve" tag is attached to the valve identifying tag and that it reflects the " Correct" required valve position. This could possibly be accomplished in conjunction with the performance of the periodic locked valve list verification.
- 2. Ensure CRN - PC/M reviews are conducted against applicable plant i procedures when requested or required by CRN dispositions.
u_____
AUDIT REPORT
_ pg __ _ _ . _ _ _ . . _ _ _ . - -- -
--- . - QSL-PM-96-1 - -- -
Page 29 of 40 Finding 3:
Modification of the Unit 1 Excore Nuclear Instrumentation was allowed to proceed to completion despite multiple indications of design process breakdown Criteria: TOAR 3.0 "Desien Control" Rev.12 "A Quality Assurance Program shall be established for design-related activities. The design control program shall ensure that the design is defined, controlled and j
verified; that applicable design inputs are specified and correctly translated into design output documents; that design interfaces are identified and controlled; that design adequacy is verified by persons other than those who designed the item; and that the design changes, including field changes, are governed by control measures commensurate with those applied to the original design."
Discussion: PC/M 009-195."RPS NI Drawer Replacement" was written to replace the Unit 1 Excore Nuclear Instrumentation with an upgraded design. At several points during the implementation of this PC/M, indications of failure of the design process were received. These occurred both in the form of Condition Reports (CR) issued to document problems encountered during the work, and Change Request Notices (CRN) issued to document field changes necessary in order for the work to proceed.
Significant examples are listed below:
Number Title Date Issued CR 96-0656 Cable 10062B actually triax -connectors Do 5/6/96 Not Fit CRN 6155 Rolled wires in connector J3 5/17/96 CR 96-0928 Joumeymen noted that Recorder JR-011 5/20/96 (Reactor Power) read =60% when it should have been reading 100% '
CRN 6170 Second power wire run in paralleled to boost 5/21/96 voltage supplied to containment amplifiers and clear locked in power trouble alarm l
CRN 6193 Input to RPS high stan-up rate bistable 5'23/96 designed incorrectly CR 96-1358 Possible loss of design control on Nls 6/12/96 i
CR 96-1434 Failure to provide adequate / correct design 6/17/96 input for PC31009195
AUDIT REPORT
_g________.. . . _ . _ _ _ - -
QSL-PM %17 --
Page 30 of 40 CR 96-1400 Steps necessary to apply PC/M closecut 6/17/96 process to partial tumover of a PC/M are not well defined -great difficulty in turning over Ul wide range ell annel"A" CR 96-1787 NI Output relay that provided RCS low flow 7/20/96 set point to RPS was not provided in NI drawer design, resulting in a constant low-flow trip Several critical personnel on the project were lost with little or no turnover.' As the project progressed, due the difficulties encountered, the implementation procedure and pre-operational test procedure became voluminous and difficult to manage properly.
Despite indicadons of serious design input and output problems, and extreme difficulty in implementation the safety consequences of a possible loss of design control on the project were not properly considered. Work was not stopped and a re-verification of the quality attributes of the modification was not performed.
Following the return of Unit I to power it was discovered that the signal cables for all four power range safety channels had been connected backwards.
Recommendations:
'Ihe following recommendations are offered to aid you in responding to the finding.
However additional or alternate actions may be necessary based upon your investigation of the finding and the causal factors and generic implications identified.
Your response must address each of the five elements in the audit cover letter.
Re-evaluate the adequacy of the approach taken towards change management in the case of this PC/M. This evaluation should not center on procedural adequacy but rather on the following attributes:
- 1. Management control and accountability
- 2. Continuity of personnel assigned /tumover practices
- 3. Time allowed for work to be accomplished
- 4. Verification of quality attributes during design )
- 5. Proper monitoring and overall evaluation of difficulties encountered during l implementation
- 6. Criteria for work stoppage.
AUDIT REPORT QSL-PM-96-17 Page 31 of 40 i
Finding 4: In two cases, Condition Report dispositions provided corrective action of unsatisfactory depth Criteria: TOR 16.0. Corrective Action Resision Paragraoh 2.2 1 "It is the responsibility of the organization which identifies the significant condition I adverse to quality to verify that corrective action description not only cornets the immediate condition, but also precludes the condition from recurring."
I AP 0006130 Condition Renorts Rev.1 Paraprach 8.7. l .b (in eart)
The cause of the condition / problem. For CRs requiring NP-700 reports (assigned in Block 5), and/or Root Cause Analysis, a formalized Root Cause Analysis shall be l performed in accordance with plant procedures and/or Departmental Instructions.
1 AP0006130 Condition Renorts Rev. 2 Paraprach 8.71. (in cart)
The assignee (from Block 5) shall evaluate the condition described in block 2 and report the following:
A. Results of the investigation and analysis of the condition.
B. The cause of the condition / problem
- 2. For CR's requiring a root cause to be performed, it shall be performed in accordance with ADM-08.04," Root Cause Analysis." The desired format for level 1 and 2 root cause analyses are found in the ADM.
D. Corrective actions needed to correct the condition and prevent recurrence.
Discussion: Polar Crane Load Test The Unit 1 Polar Crane test was conducted to demonstrate the operability of the main hoist by lifting a load approximately 125% of the hoist's full rated capacity.
CR 96-1543 was wTitten to document the discrepancy between the tinal load cell reading of 412,730 lbs and the criteria required by the test instructions of 437,500 lbs F.) 2000 lbs. An Engineering Evaluation associated with the CR disposition attempts tojustify the 125% load lift by using the How totalizer readings and a vendor supplied weight value for the tilled water bags.
O
\
AUDIT F O'dRT
_._ - -qst.py g i7 Page 32 of 40 The Engineering evaluation states the following:
i "The observed load cell discrepancy of 24,770 pounds was approximately 5.6%
~ lower than the weight calculated from the Flow Totalizer. The Load Cell vendor indicated that an approximate error of 5% was normal. The Flow Totalizer values determined that the test weight met the LOI criteria. Since the Polar crane is not safety-related the totalizer is not required to be part of the M&TE program.
Conclusion:
Test results are acceptable."
The above engineering evaluation uses a combination of flow totalizer data and the known weight of the water bags when filled. (supplied by manufacturer), to support its satisfactory conclusion. Data contained within the evaluation states that, "the expected capacity of each of the 35 metric ton water bags in the configuration used (2 bags per shackle) is 70,900 lbs and the expected capacity of the 10 metric ton water bag is 20,000 lbs." He evaluation does not account for filling differences associated with the 7 bags used in the test resting against each other during the filling process.
Emerson Electric Company (totalizer manufacturer) drawing SC-4316 and name plate data on the totalizer. indicate a maximum flow rate of 1295 g.p.m. and a minimum flow rate of 259 g.p.m. for calibrated operation. The evaluation does not demonstrate that the totalizer (which was a non-M&TE calibrated, back-up instrument) was used within its calibrated range. Subsequent conversations with Engineering revealedthat the flow rate was not verified during thefilling process. Conversations with QA personnel observing the test indicated that theflow rate was less than 150 g.p.m..
Lack of verification of the proper Dow rate represents a substantial oversight in the disposition of the CR and weakens the basis for the conclusion that this important test had a satisfactory result.
Engineering work of this caliber in the resolution of a test discrepancy on a crane that routinely lifts heavy loads over irradiated nuclear fuel is judged to be unsatisfattory with respect to quality assurance program corrective action requirements.
RPS Nuclear instrument Drawer Reclacernent CR 96-1434 was written to document the failure to provide adequate / correct design input for PCM 009-195. "RPS N1 Drawer Replacement. Specification SPEC-IC-004 "RPS Nuclear Instrumentation System Replacement", identified the preamp power cabling size as =16 gauge wire. The as installed cable was = 18 gauge wire. The signal cabling was identified as coaxial cable however it was discovered during installation that portions of triaxial cable were actually installed. The specification did not contain detailed mounting plans for the Nuclear instrumentation drawers. He disposition of this CR addresses these items as nonconformance issues and identifies the Change Request Notices to incorporate these discrepancies into the design as
o
\
AUDIT REPORT
_ _ _QSL-P31-96-17 . _ - _ _
Page 33 of 40 a
correcting the problem. The disposition states that root cause and generic implications of this problem would be programmatically addressed as part of a different Condition Report, CR 96-1766. On this basis, without the generic implications or root cause having bmn analyzed, the original Condition Report, CR 96-1434, was closed.
Several problems were identified with the disposition of this CR and are listed below:
- 1. :1P0006130 eives no enddance in transferrine corrective actions or evaluations to another CR.
In approving CR 96-1434 the PGM had checked the Investigate And Correct block along with the Root Cause Analysis block. The CR documentation package does not contain a root cause evaluation as delineated in ADM-08.04 Revisio,n 1. A review of the CR disposition revealed several items, i.e.,
assumptions made, lack of detail in dnwings and an apparent lack of detailed planning preceding the actual drawer replacement, that would have benefited from an in-depth root cause analysis. As an alternative, the disposition states that root cause and generic implications of this problem would be programmatically addressed as part of a different Condition Report, CR 96-1766.
- 2. The disoosition ofCR 96-1434 incorrectiv references CR 96-1766.
The referenced CR number was incorrect. The intended CR reference should have been CR 96-1787, written to evaluate a missing NI output relay on the same project. This change was made by the engineer via a supplement during this PMON at the prompting of QA.
- 3. CR 96-1787 (the intended reference) does not cross reference CR 96-1434 or describe the conditions associated with it.
A review of CR 96-1787 on 8/12/96. revealed that Block 2 did not direct or describe any action associated with CR 96-1434. An additional review on 8/20/96 revealed that the CR had been signed off by engineering with no mention of CR 96-1434 in the root cause evaluation. A statement was included in the CR disposition which read " Generic considerations and corrective actions regarding the adequacy of the PC/M design process and verification for PC/M 009-195 are addressed in CR 96-1878."
4 Generic considerations and corrective actions reeardine the adeauacy of the PC/M desien process and verincarion for PC/M 009-195 were transferred to another Condition Report. CR 96-18~R
Y AUDIT REPORT QSL-P31-96,17.. _
1 This CR documents concerns that were inconsistent with expectation conceming data taken during the shape annealing factor test for excore nuclear l instrument detectors. A review of CR 96-1878 prior to close out revealed ;
there was no reference to the two previous condition reports. l A meeting was held with engineering and the problems concerning the continuing transference process was discussed. Engineering stated the issues would be addressed and closed within CR-1878. They also agreed to include references in the CR to tie the corrective actions back to the original two CR's. l l
l \
- 5. The time frame between initiation o(CR 96-143J and the disvosition ofCR 96-1787 was excessive.
i CR 96-1434 was approved by the PGM on 6/16/96. CR 96-1787 was approved by the PGM on 7/20/96. The 34 day time frame between the two CR's mdicates that inappropriate / inadequate attention was displayed relating to the i senousness of the conditions described. Had the original CR been l dispositioned properly, problems identified later in the modification might have been been avoided. In addition. the act of successively deferring the root cause analysis had the effect of artificially extending the due date for the original CR in excess of the required 30 day period.
Overall Conclusion The Condition Report process is intended to resolve previously identified discrepancies. The measures that it prescribes must be taken seriously, since they prescribe a safe path out of a situation that has already deviated from the desired amcunt of defense in depth. The cases above demonstrate the liability that exists when the measures of the Condition Report process are not properly implemented.
Recommendation:
l l Re following recommendations are offered to aid you in responding to the finding.
l However. additional or alternate actions may be necessary based upon your investigation of the finding and the causal factors and generic implications identified.
! Your response must address each of the tive elements in the audit cover letter.
1
- 1. Determine the input flow rate of the totalizer during the tilling process and insure it fell within the vendor specified range.
1 l
, AUDIT REPORT i _ . _ _ _ _ . p g ._. .__ _ . ._
. _ _ _ . _ _ _ . _ . . _ . ._ QSL-PM-96-17 Page 35 of 40 ;
4 2. Provide treining on the instances above to personnel responsible for the activity of dispositioning Condition Reports. Reemphasize the critical importance of l
a conservative approach to analysis and compliance with the established
{
administrative procedures. i
- 3. Ensure that the work load of personnel responsible for dispositioning Condition !
Reports is maintained at a '.evel where effective functioning of the process is possible.
J
)
t I
i
.'(l l AUDIT REPORT e
QS L-PM-96-17
______. 'Pa (36oT40-l Finding 5: In three cases inadequate inter-departmental coordination has caused Quality Assurance audit finding responses to have unsatisfactory content.
)
l Criteria: AP 0006130 " Condition Reports", Revision 2 Paragraoh 8.7.1 "The assignee (from B!ock 5) shall evaluate the condition described in Block 2 and report the following:
A. Results of the investigation and analysis of the condition B. The cause of the condition / problem.
C. For CRs issued as the result of Quality Assurance Audit findings, the cause of i the condition shall be determined and the results of an examination for l potential weakness in departmental self-assessment programs widch may have !
impeded self-identification of the problem shall be included in the evaluation.
In addition, a determmation of the generic impact of the problem shall be made.
D. Corrective actions needed to correct the condition and prevent recurrence.
Paragraoh 8.7.2 "The assignee retains overall responsibility for completion of the CR from Block 7 through Block 13, within the time frame assigned, unless the Plant General Manager approves a due date extension, severity level downgrade or reassignment of j responsibility." I Discussion: Several recent audits by Quality Assurance have identified problems that involve multiple plant departments. In accordance with the requirement of AP 0006130, '
" Condition Reports" the need for response to the Findings that documented these problems were tracked via the initiation of condition reports (CR). The CRs were assigned to one or more of the departments having responsibility for the problems by the Plant General Manager.
In two cases the Finding responses generated in resp &se to these CRs and returned to QA were incomplete. In another case a Finding response was returned to the Vice President - PSL for tinal approval without necessary components. Two types of situations existed:
- 1. The CR for the Finding response was assigned to two departments, but routed to only one department and was completed addressing only that department's involvement. (CR 96-1292)
.n (
ik FPL.
') AUDIT REPORT QSL-PM-96-17
~
-- -- -- - - Page 37-of 40 - - - -
- 2. The CR for the Finding response was routed to one department which then identified the need for response from other department. A response from the second department was not obtained prior to the finding response bemg completed and submitted. (CR 96-1290, CR 96-1681)
Two of the CRs associated with these Finding responses were closed by the Corrective Action Group and the Finding responses forwarded to the QA Department.
These two Finding responses were rejected by QA. In a third case the Finding response was rejected by the Vice President - PSL during final review prior to being submitted to QA.
In connection with the information above, it is noted that a generic problem exists with obtaining inter-departmental coordination with respect to Finding responses. In one case the second department was not notified of the need to respond to the Finding.
In another case the lead department did not feel that it was their responsibility to obtain response from other departments that it had identified as contributors to the problem. In a third case, an effort was made to obtain response from the second department but cooperation was not provided.
Recommendations The following recommendations are offered to aid you in responding to the finding.
{
However additional or attemate actions may be necessary based upon your '
investigation of the finding and the causal factors and generic implications identified.
Your response must address each of the five elements in the audit cover letter.
I
- 1. Establish a clear method for assignment of the responsibility for generation of l finding responses.
- 2. Establish clear accountabilities for the lead department and supporting l departments in the Finding response process.
.t . Establish a plant mechanism for generation of multiple CRs in response to a single finding when warranted.
- 4. Establish clear expectations for Finding response completeness prior to acceptance by the Corrective Action Group and CR closure.
p
) AUDIT REPORT QSL-PM-96 -
Page 38 of 40 Audit Particinants:
Name Department / Group PMON No.
H. Buchanan Health Physics ,96-028 H. Mercer Health Physics96-028 R. Mc Cullers Health Physics96-028 A. Wier Health Physics96-028 V. Munne Health Physics96-028 D. Haithcox Health Physics96-028 l J. Mann Information Services96-043 I C. Marple Operations96-043 W. Mead Reactor Engineering 96-043 ;
C. O'Farrill Reactor Engineering 96-043 D. Pottorff Operations Suppon 96-043 B. Sommers Health Physics96-028 j H. Johnson Operations Manager 96-047 l M. Allen Tr;ining Manager 96-047 I C. Marple Operations Supervisor 96-047 P. Fulford Operations Support / Testing 96-047 M. Dryden Licensing 96-047 L. Rich Training 96-047 R. Weller Operations96-047 P.11oneysett Operations96-047 D. Carpenter Training 96-047 K. Carpenter Training 96-047 D. Brown Training 96-047 P. Kendrick Nuclear Engineering 96-039 J. Campbell System & Component Eng.96-039 D. Melody Quality Control 96-039 B. Cole, man Electrical Maintenance 06-039 B. White Security 06-046 J. Lahera Budget 96-046 M. Snyder System & Component Eng.96-049 B. Walcheski System & Component Eng.06-049 L. liiegel Instrument & Control Maint.96-051 D. Wolf Nuclear Engineering 96-051
O t'
o AUDIT REPORT QSL-PM-96-17 Page 39 of 40 Audit Participants (Continued)
Name Department / Group PMON No.
R. Gil Nuclear Engineering 96-051 O. Schmidt ICM '96-051 J. Barbieri Nuclear Engineering 96-051 P Dutt Nuclear Engineering 96-051 W. Bush Nucelar Engineering 96-051 C. Butler ICM 96-051 A. Pawley 1CM 96-051 Robert Hawley Quality Control 96-048 Bruce Parks Quality Assurance 96-048 Steve Sanders Quality Assurance 96-048 JeffBarbieri Nuclear Engineering 96-048 Michelle Tarascio Nuclear Engineering 96-048 Warren Bush Nuclear Engineering 96-048 Rudy Gil Nuclear Engineering 96-048 Pre-Audit Notific2 tion:
. Location: St. Lucie Plant Date: July 1.1996 Post-Audit Conference:
Location: St. Lucie Plant Date: -
Sept. 5.1996
\
AUDIT REPORT
-- p
__ _ __. . -QSirPM-96-17 Page 40 of 40
- Summary of Post-Audit Conference:
The results of the Performance Monitoring activities < inclusive of the findings and recommendations were discussed with the attendees: J. Voorhees, D. Fadden, M. Allen, J. Walls, W. Bladow, J.
Scarola, B. Acosta, R. McCullers. D. Denver, A.' Stall, E. Weinken, H. Johnson, B. Dawson.
i Location of Audit:
St. Lucie Plant i
Accompanying Auditors: L.J. Bearror D. C. Lowens
- B.J. Lowery i C.E. Norris L.L. Pannessa i
, Princinal e Auditor: -> MM k#[
d J.J.Shlis Date d
Quality Assurance - PSL Review ed by:
Ol^f(0 Qlb bi(
D. C. Lowens Date QA Supervisor - PSL
,