ML20059E536
| ML20059E536 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 08/19/1993 |
| From: | AFFILIATION NOT ASSIGNED |
| To: | |
| References | |
| OLA-2-I-MFP-078, OLA-2-I-MFP-78, NUDOCS 9401110269 | |
| Download: ML20059E536 (27) | |
Text
l 3
j O - 275lw=2.3 -OU - L MFP %M lt
~E-P-77 glMuws m-i 88 NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993
'93 MT 28 P6 :10 MANAGEMENT
SUMMARY
. fy This NCR covers two events as follows:
i On August 21 1991, ASW pump 1-2 was declared operable contrary to the requirements of TS 3.7.4.1 and ASME Section XI, Article IWP 3230, since the pump was above the action high level for Differential Pressure (DP). It was determined that ASW pump 2-2 volume 9 curve was ured to define the action high levels for ASW pump 1-2.
When the proper data was obtained from ASW pump 1-2 volume 9 curve, it showed that the pump was above the high level for DP.
On November 14, 1991, a PG&E surveillance test reviewer failed to j
recognize that ASW pump 1-2 should have been placed on alert in j
cccordance with TS 4.0.5 and ASME section XI, 1977 Edition (Summer 1978 Addenda), Article IWP 3230.
The STP P-7B surveillance test reviewer failed to recognize that ASW pump 1-2 was below the alert low j
1cvel for DP.
On October 10, 1992 while reviewing an STP P-7B on ASW pump 1-2, these ovents were discovered.
The root causes for both events were attributed to personnel errors cognitive: (1) In the first event the STA did not ensure that the j
correct curve from " volume 9" was used; and (2) In the second event, the second surveillance test reviewer assumed that the test results w:re acceptable, since the data was similar to previous test results.
The corrective actions to prevent recurrence include : First Event:
g1) Nuclear Engineering to revise the IST Pump procedures to add an independent verification of volume 9 data entry; (2) Operations to write an incident summary to reemphasize the importance of the Surveillance Test process and train the Operations group in the importance of this problem; (3) Due to its significance, the Training group to add this event to the operators requalification training; S:cond event: (1) An AGME Section XI computer trending. program has been implemented and will be discussed in the next revision of Administrative Procedure (AP) C-3S1; (2) A training session covering ASME Section XI requirements for Alert frequency tracking was held with the appropriace procedure reviewers.
On February 4, 1993, the TRG reconvened and reviewed the write-up and root cause analysis for final comments.
NCR CLOSURE:
MAY 1, 1993.
92NCRWP/92TPN052.PGD Page 1 of 23 9401110269 930819 PDR ADOCK 05000275 g
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 NCR DC1-92-TP-N052 i
Missed Alert Frequency STP I.
Plant Conditions On August 21, 1991 and on January 25, 1991 Unit 1 was in Mode 1 (Power Operation) at 100% power.
II. Description of Eve.nt A.
Summary:
First Event:
On August 21, 1991, Auxiliary Salt Water (ASW) pump 1-2 was declared operable contrary to the requirements of TS 3.7.4.1 and ASME Section XI, 1977 Edition (Summer 1978 Addenda),
l Article IWP 3230, since the pump was above the action high level for Differential Pressure (DP).
Second Event:
In the second event, on November 14, 1991, a PG&E surveillance test reviewer failed to recognize that ASW pump 1-2 should have been placed on alert in accordance with TS l
4.0.5 and ASME section XI, 1977 Edition (Summer 1978 Addenda), Article IWP 3230.
The STP P-7B surveillance reviewer failed to recognize that ASW pump 1-2 was below the alert low level for DP.
B.
Background:
ASW pumps are required to be functionally tested in accordance with STP P-7B on a 92 day frequency to meet the requirements of TS 4.0.5.
ASW pump operability requirements are defined in T.S.
3.7.4.1.
ASME section XI, 1977 Edition (Summer 1978 Addenda), Article IWP 3230, " Corrective Action" states that, "If deviations ~
l fall within the Alert Range of Table IWP-3100-2, the frequency of testing shall be doubled until the cause of the i
deviation has been determined and the condition corrected."
l The requirement to increase test frequency ensures that attention is given by the second reviewer to.the parameter that is indicated within the acceptable range.
92NCRWP/92TPN052.PGD Page 2 of 23 t
a NL'rtEAR REGULATCf:Y COMMISSION f Of3-OQ Ofr:3l gg g3,@((
Docket Ne v'
- - :: PadIFl(_615 %d C LCc TB.JC_ 00 10Ell11' ICD v C10:i't r> /
W -~
/-... _
M,2 7 L.,
~*
C C"~
? '.i[
Y-/77 3 w"
v:a,u, C s'a R2; tter o t_
I l
t
i 1
i NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 C.
Events
Description:
1 First'avents On August 21, 1991, ASW pump-1-2 was removed from service'to-j perform STP P-7B " Routine Surveillance Test on-Auxiliary ~ Salt-Water Pumps".
This placed Unit'1 in a 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />' action statement per TS 3.7.4.1. On. August 21, 1991;ASWapump.1-2 was.
declared. operable contrary _to-the requirements of TS 3.7.4.1' and ASME section XI, 1977 Edition (Summer:1978_ Addenda),
Article.IWP 3230, since the-pump-was above the1actionLhigh) level for DP.
ASW pump-1-2 should have remained inoperable.
second events l
In the second. event, on November =14,11991, ca PGEE--
surveillance test eviewer' failed to recognize (that ASWLpump-3 1-2 was below the' alert low level for Differential ~ Pressure l
(DP) and should be placed on alert.
l On November 29, 1991, a surveillance test of ASW pump'l-2_was, l
subsequently performed.
The results of this test wereL l
satisfactory, with ASW Pump 1-2 differential pressure:within 2
l the acceptance range.
However, ASW pump'1-2 technically remained-in an alert status and should have been maintained l
l in that status.
on January 25, 1992, the surveillance'testffor-performing 1In.
' I Service Inspection and Testing (IST) activities for.ASW pump 1-2 was not performed on an acceleratedLfrequency as: required by the applicable addenda.of the ASME' Boiler'and Pressure Vessel Code section XI, and as itfapplies toLTechnical' Specification 4.0.5. 'ASW pump 1-2 should_have been declared inoperable and the 72-hour action statement of(T.S.
3.7.4.1' entered.
On January 29, 1992, at 0000 PST, the 72-hour _
~
action statement for T.S. 3.7.4.1 was exceeded..This date i
includes the maximum allowable extension ofi25% Land the'72 i
hour action-statement..
On October 10,~1992, while. reviewing.a surveillanceLtest-l' procedure (STP)-P-7B on ASW pump 1-2,:the pump _ history;was questioned since the results of the current test indicated the need for this pumpfto be placed-on alert, and on:
accelerated testing as required _byLTechnical; Specification
~;
4.0.5.(see Reference 4).
. l J
92NCRWP/92TPN052.PGD Page '3 of 23_
- f
,ea
,r
,,...~.,.
~.
. -.~..
.-,c
l l
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 On October 15, 1992, a Technical Review Group (TRG) determined that in these two events the requirements of TS 3.7.4.1 and 4.0.5 have been exceeded and that these events were reportable in accordance with 10 CFR 50.73 (a) (1) (B).
l D.
Inoperable Structures, Components, or Systems that l
contributed to the Event:
None.
E.
Dates and Approximate Times for Major Occurrences:
1.
August 21, 1991:
Event date: TS 3.7.4.1 and 4.0.5 were not met when ASW pump 1-2 was declared operable.
l 2.
November 14, 1991:
STP P-7B was performed and the l
reviewer failed to recognize that ASW pump 1-2 differential l
pressure was in the alert range for accelerated surveillance.
3.
January 25, 1992:
The alert status. surveillance due date was exceeded.. Entered i
72-hour action statement per tr 3.7.4.1.
4.
January 29, 1992:
Event date. The 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> (plus 25%) statement of TS 3.7.4.1. TS j
3.7.4.1 was exceeded.
l 5.
October 10, 1992:
Discovery date..In reviewing a current completion of STP P-7B on ASW pump 1-2, the pump history was questioned.
F.
Other Systems or Secondary Functions Affected:
None.
G.
Method of Discovery:
On October 10, 199'2 during a review a of the completed STP P-7B a system engineer questioned the pump test data history since it revealed that ASW pump 1-2, was required to be placed on alert (accelerated testing) frequency per ASME section XI.
92NCRWP/92TPN052.PGD Page 4 of 23
4 t
NCR DCl-92-TP-N052 Rev.00 Draft: F & uary 4, 1993 H.
Operator Actions:
None required.
I.
Safety System Responses:
None required.
III.
Cause of the Event A.
Immediate Cause:
First Event:
ASW pump 1-2 was declared operable contrary to the requirements of T.S.
3.7.4.1.
and ASME section XI, 1977 Edition (Summer 1978 Addenda), article IWP 3230, since 'he pump was above the action high level for DP.
second Event:
A PG&E surveillance test reviewer failed to recognize that ASW pump 1-2 should have been placed on an Alert frequency.
B.
Determination of Cause:
See root cause analysis (reference No. 11).
C.
Root Cause:
Personnel errors.(cognitive) due to inattention to details:
In the first event:
The PGEE STA failed to recognize that the data was~ incorrect.
The incorrect volume 9 data was incorporated in STP P-7 Band.
The STA read ASW pump 2-2 curve instead of 1-2 curve which should have placed ASW pump 1-2 above_the action high level l
l DP.
On August 21, 1991, ASW pump 1-2 was declared operable by the Shift Foreman contrary to the requirements of~T.S. 3.7.4.1 and ASME Section XI, 1977 Edition (Summer 1978 Addenda)~,
Article IWP 3230, since the pump was above the action'high~
level for DP.
92NCRWP/92TPN052.PGD
.Page 5 of 23 i
l h
I
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 In the second event:
A PG&E surveillance test reviewer failed to recognize that ASW pump 1-2 should have been placed on an Alert frequency.
D.
Contributory Cause:
First Event:
1.
The shift foreman failed to recognize that the wrong pump curve was attached to the data package (the unit designation was clearly marked on the pump curves).
2.
The method of determining the base line for the ASW pump is cumbersome.
Operators are required to.go to another document to pull the pump base line data rather than have an appendix in the procedure for each-pump.
3.
The test performer failed to identify that the wrong pump curve was used.
4.
The Annubar readings are crucial to the proper data collection.
Based on other valid tests, the data obtained in this specific test is questionable, second Event:
Proper training / instructions were not provided'to the surveillance test reviewer.
IV. Analysis of the Event A.
Safety Analysis:
First event:
On August 21 1991, ASW pump 1-2 was declared operable contrary to the requirements of TS 3.7.4.1 and_ASME Section XI, 1977 Edition (Summer 1978' addenda), Article IWP 3230, since the pump was above the' action high level for DP.
It was determined that ASW pump 2-2 volume 9 curve was used to define the action levels for ASW pump 1-2.;
When'the proper l
data was obtain from the ASW pump 1-2 volume 9 curve, it i
showed that the pump was above the action high level for DP.
Volume 9 curve action high limit for ASW pump 2-2 is.54.1 psid and for ASW pump 1-2, 52.2 psid. The recorded DP was 53.26 psid.
92NCRWP/92TPN052.PGD Page 6 of 23 i
i i
4 i
4 4
NCR'DCl-92-TP-N052 Rev'.00 i
Draft: February:4, 1993 i
l Subsequent to the " Action High" test recorded;an August'21, 5
1991,.there.have been tests performed with.no other high i
4 i
values recorded.-From the data trended, this particular-
)
finding was one standard deviation from.the meancwhich statistically'is insignificant'(mean being,99.01% and.its i
standard deviation being 4.46).c Furthermore,nannubar
.)
i mispositioning problems have been' documented and j:
dispositioned on several~ Action Requests.- Proper annubar_and; resultant flow calculations (from the' proper flow constant).
i t
j are crucial to the successful performance of.STP:P-7A and b a i
I P-7B.
PGEE believes;that the positioning ofLtheLannubar may-have caused the', abnormal readings of August;21, 1991.and November - 14 ~, 1991.
second event:
']
The requirement to increase test' frequency.' ensures 5that
-l
~
attention is given to the parameter that'.is andicated within:
4 i
the alert range.;
j
(
']
The pump differential ~ pressure alertilow, warns.plantL engineers that the pump is'only'slightly within the' i
acceptable range and should be. tested-and: evaluated on a more 4
frequent basis. ~ Although'ASW pump-1-2 reached-itszalert-limits, it_was considered to be operable.
~
conclusion:
4 Accordingly, PG&E believes that ASW pump.1-2'was' fully).
functional and capable ofLperforming;itsvintended1safetya j
function during the time period in questio'ni(August 21,s1991 to October.15, 1992).
In bothiinstances,.these. errors did not ciente an unreviewed safety. question and did not1 adversely affect the health and. safety _of.the.public.-
B.
Reportability:
1.
Reviewed under QAP-15.B and determined [to be7 y
non-conforming in-accordanceJwith SectionL2.1.8.
]
1 2.
Reviewed under 10 CFR 50.72iand410 CFRL50.73 per NUREG' 1022 and determined to be reportable in'accordance with 10 CFR 50.73 (a) (2) (1) (B)..The report associated with this NCR is LER 1-92-024-00.
j 92NCRWP/92TPN052.PGD Page 7 of. 23 I
v,
9 g
4
- = -
ev w
,,,m
---9,e-+...-
m-mr-r
.-.-1 s--+
%k+. [w
..,v ee e
9
i
~
NCR DCl-92-TP-N052 Rev 00' l
Draft: February 4, 1993 Prepare and submit a License Event Report-(LER)-per 50'.73' (a)-(2) (v) (D).
RESPONSIBILITY:
P. Dahan.
Tracking AR:
A0280549,.AE #.03 i
DEPARTMENT:'
Regulatory Compliance.
Outage Related? No OE Related?
' No -
NRC ' Commitment? No
}
CMD Commitment? No STATUS:-
COMPLETE.
l 3.
Reviewed under 10 CFR.Part 21 and. determined that-this problem,will not require a 10 CFR 21 report, since'it does not involve defects in vendor-supplied.-
L,
services / spare parts in stock.
4.
This problem will not be reported via an INPO Nuclear' Network entry.
.i J
5.
Reviewed under 10 CFR 50.9.and determined the event to be:
not reportable since this event does not have a=
significant implication for public health and. safety or common defense and security.
6.
Reviewed under the criteria of AP C-29' requiring;the issue and approval of an OE and determined that'an-OE is j
not required.
V.
Corrective Actions l
l A.
Immediate Corrective. Actions:
ASW pump 1-2 was placed on acceleratec alert testing frequency.
B.
Investigative Actions:
i 1.
Evaluate'other pump test reviews-performed by the same_
test reviewer to determine if..theys are correct.
RESPONSIBILITY: L. Cossette.
DEPARTMENT:
Nuclear' Engineering Tracking AR:-
A0280549, AE # Ol' STATUS:
COMPLETE.
92NCRWP/92TPN052.PGD Page 8 of.23 t
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 2.
Clarify if missed surveillances are reportable.
RESPONSIBILITY: D. Sisk and D, Benkhe.
DEPARTMENT:
Regulatory Compliance Tracking AR:
A0280549, AE # 02 and 10.
STATUS:
RETURN.
3.
Develop an action plan for forming a surveillance engineering task group to review an enhance the (STP P-7B) surveillance test process.
RESPONSIBILITY:
L.
Cossette.
DEPARTMENT:
Plant Engineering.
Tracking AR:
A0280549, AE / 04 STATUS:
RETURN.
Replaced by a prudent action. See AR A0285203.
4.
Contact the Operations Department Shift supervisor and initiate a discussion with the surveillance Engineer, STA and other operations personnel involved in the conduct of STP P-7B surveillances.
RESPONSIBILITY: Paul Dahan.
DEPARTMENT:
Regulatory Compliance.
Tracking AR:
A0280549, AE / 05 l
STATUS:
COMPLETE.
5.
The TRG chairman and the senior nuclear engineer to discuss with engineering management.(Steve Banton and McCoy Burgess) the feasibility to split the surveillance test procedures (STPs) to be unit and equipment specific i
to reduce the possibility of errors.
RESPONSIBILITY:
C.
Groff DEPARTMENT:
Asst. to Tech. Manager.
Tracking AR:
A0280549, AE # 6 STATUS:
RETURN.
i l
92NCRWP/92TPN052.PGD Page 9 of 23
t NCR DC1-92-TP-N052 Rev.00 Draft: February 4, 1993 C.
Corrective Actions to Prevent Recurrence:
1.
Write an operations incident summary to re-emphasize the importance of the surveillance test process and train the Operations group in the importance of this problem.
RESPONSIBILITY:
J. Dye ECD:
1/31/93 DEPARTMENT:
Operations.
Tracking AR:
A0280549, AE # 07 Outage Related? No OE Related?
No NRC Commitment? No CMD Commitment? No STATUS:
RETURN.
2.
Revise the IST pump procedures defined on AE'# 08 of AR A0280549) to add an independent verification of volume 9 data entry.
RESPONSIBILITY:
L. Cossette.
DEPARTMENT:
Plant Engineering.
Tracking AR:
A0280549, AE # 08 Outage Related? No OE Related?
No NRC Commitment? No CMD Commitment? No STATUS:
RETURN.
3.
Due to its significance, add this event to the operators requalification training.
Training will empnasize proper positioning of the annubar.
RESPONSIBILITY:
J.
Becerra.
ECD:
2/15/93 DEPARTMENT:
Training.
Tracking AR:
A0280549, AE # 09
)
Outage Related? No OE Related?
No NRC Commitment? No CMD Commitment? No STATUS:
ASSIGNED.
l l
1 l
92NCRWP/92TPN052.PGD Page 10 of 23 l
l
i
\\
~
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993
)
i 4.
An ASME Section XI computer trendihg program has been implemented.
This program provides the trending of all i
section XI IST component data and compares the current need for an alert frequency.
This information is provided to the test reviewer at the time of data input into the section XI trending program.
This alert evaluation feature program will be discussed in the next revision of Administrative Procedure (AP) C-3S1.
(See QE Q0009506 and AR A0258602).
5.
A training session covering ASME Section XI requirements for alert test frequency tracking was held with the surveillance test reviewers.(See QE Q0009506 and AR A0258602).
D.
Prudent Actions (not required for NCR closure)
Develop an Action Plan for forming a surveilaance engineering task group to review and enhance the IST pump surveillance test process.
The group shall include HPES representative, Operations and Surveillance Group participation. Consider volume 9 pump curves and pump data.
RESPONSIBILITY:
L. Cossette DEPARTMENT:
Nuclear Engineering Tracking AR:
A0285203.
STATUS:
RETURN.
VI. Additional Information A.
Failed Components:
None.
B.
Previous Similar Events:
1.
NCR DC1-92-TN-N003 -
MISSED ALERT FREQUENCY STP.
l On December 16, 1991 the time allowed for the performance of a conditional surveillance, STP P-14B, including the 25 percent extension allowed by TS was exceeded.
The cause of this event was personnel error, cognitive, in.
that the Boric Acid transfer Pump.1-2 differential i
pressure was in the alert range on October 18, 1991.
i 92NCRWP/92TPN052.PGD Page 11 of 23
i NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 Since the TRG determined that this event did not meet the definition of an non-conformance, the NCR was closed. The corrective actions tracked by QE Q0009506 were as follows:
1.
A training session covering the section XI requirement for " alert" was held with the. appropriate procedure reviewers.
2.
A section XI trendm.g program which will enhance the reviewers awareness to an " alert" situation has been implemented. This program provides the trending of all ASME Section XI IST component data and compares the current test data'to the previous test data to evaluate the need for an " alert" frequency.
These corrective actions should not have prevented this event since these actions were taPen after this event occurred.
2.
NCR DC1-86-TN-N086 - MISSED SURVEILLANCE.
The test frequency for FCV-366 was not increased to 31 days when the valve stroke time exceeded 150% of previous valve stroke test.
On 02/07/86, AR 0016622.was generated to increase the stroke time frequency of. Unit 2 FCV-366 valve from 92 days to 31 days. On 03/07/86 Unit'l FCV-366 stroke time was also found to exceed the stroke time by more than 150%.
The root cause was determined as inadequate engineering controls on the alert status items.
A training session was conducted within the engineering department. Surveillance test department implemented routine program of reviewing all alert status STPs being conducted to determine if alert condition is still valid
{
and increased emphasis in updating the Alert Status Board. All PPE's (Surveillance Test Department) were retrained to verify all Alert status components when' determining if item is already in Alert.
The corrective actions taken on this event did not i
preclude the present event since the operations department personnel were.not included in the training.
92NCRWP/92TPN052.PGD Page 12 of 23 i
i 1
l '~
l l
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 3.
NRC INSPECTION REPORT NO. 50-275/86-14 AND 50-323/86-15.
On May 12, 1986, the NRC issued a severity level IV Notice of Violation for DCPP Units 1 and 2. This notice cited a concern regarding the effectiveness of PG&E's Technical Specification surveillance test program.
l l
On December 9, 1985, with both Units in Mode 1, plant l
engineers identified a missed surveillance on Containment Spray Pumps (CS) 1-2 and 2-2. The missed surveillance occurred on October 30, 1985, for CS Pump 1-2 and on November 1, 1985 for CS Pump 2-2.
The missed surveillance was a result of failure to increase the surveillance test frequency (as required by Technical Specification 4.05) after performance of tests on l
September 2, 1985 for CS Pump 1-2 and September.4, 1985, for CS Pump 2-2.
The cause of the missed surveillance was personnel error in that the plant engineer evaluating the test results failed to initiate an increase in the surveillance test frequency upon identifying the CS pumps had reached an alert high range on differential pressure.
The surveillance tests had subsequently been performed on December 4, 1985 for pump 1-2 and December.5, 1985 for CS Pump 2-2.
The results of these tests were satisfactory, the CS Pump 1-2 differential pressure within the acceptance range and CS Pump 2-2 differential pressure, again, in the alert high range. CS Pump 2-2 surveillance test frequency was increased from at least once per_92 days to at least once per 46 days. The plant engineers reviewed all other 1985 IST pump tant resulto and found no additional discrepancies.
The corrective actions taken on this event did not preclude the present event, since the test was rerun satisfactorily and no other action was taken.
C.
Operating Experience Review:
1.
NPRDS:
Not applicable.
2.
NRC Information Notices, Bulletins, Generic Letters:
No similar event found.
92NCRWP/92TPN052.PGD Page 13 of 23
NCR DCl-92-TP-N052 Rev 00 Draft: February 4, 1993 3.
No similar event found.
D.
Trend code:
First event:
Responsible department OP,'and cause code A3.
Second Event:
Responsible department TP,'and cause code A2.
E.
Corrective Action Tracking:
1.
The tracking action request is A0280549.
2.
Are the corrective actions outage related?
No.
F.
Footnotes and Special Comments:
Based on the change in position.by the NRC (See AE # 2), PSRC interpretation 85-02 will be rescinded.
The new PSRC i
position-is that missed surveillance by themselves are no i
longer reportable.
As long as the missed surveillance is performed prior to the action statement being exceed, the event is not considered reportable.
On 11/23/92 the PSRC as approved rev. 1 of 85-02 which rescinded the PSRC interpretation, based on the change of.NRC position.
RESPONSIBILITY: ' D. Behnke.
DEPARTMENT:
Regulatory Compliance.
Tracking AR:
A0280549, AE # 10.
STATUS:
RETURN.
l G.
References:
1.
Initiating AR A0280428 - Evaluate Missed Placement of ASW Pump 1-2 on Alert 2.
Tracking AR A0280549.
3.
QE Q0009506 - Personnel error on review of STP P-14B.
92NCRWP/92TPN052.PGD Page 14 of 23 v s
-r-r--
r w
e t-
-n r
n*
-w s
w y
s-r v---
~e-e~
v w,
w wr rt'
i I'
I I
I:
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 l
l 4.
AR # A0280427 - Require ASW 1-2 to be placed on alert i
frequency.
1 5.
AR # A0268730 - Inadequate review of STP'P-14B.
f 6.
Technical. Specification 3.7.4.1
]
7.
Technical Specification 4.0.5 i
8.
STP V-7B ran on 08/20/91.
i STP V-7Biran on 08/21/91-
)
STP V-7B ran-on 11/14/91.
i STP V-7B ran on 11/21/91..
1 9.
VOLUME 9 pump curve for ASW 1-2.
.1 f
- 10. ASME SECTION XI, IWP 3000 - Inservice Test Procedures.
A i
- 11. Root cause analysis.
f H.
TRG Meeting Minutes.'
i 1.
On October 15, 1992, the TRG convened and considered the following:
1 i
5 a.
Review of the events of OS/21/1991; 11/14/1991,.and j
10/09/1992.
i b.
Determination of reportability of these events.
i
'l
]
c.
Immediate actions.
I j
Looked at the human factor side of the curve. : Operators' j
are required to go to another. document-to pull'the base i
j line of the.ASW pump rather to have an appendix for each j
pump attached to the STP.-
1
]
A typical process for completion of1STP P-7B was reviewed-which consists of:
}
The STA prepares the procedure following.the' schedule i
a.
that says run the test.-
1 1
i b.
The STA coordinates'with I&C to have the annubars j
installed.
i 92NCRWP/92TPN052.PGD Page 15 of 23 1
1 4-
_,... ~
i
(
ITR DCl-92-TP-N052 Rev.00 Leaft: February 4, 1993 The STA gives the procedure to,the senior operator c.
who coordinates with the operator.
2 d.
The intake AO records the data and the turbine watch AO records the data at the annubar.
e.
After the procedure is performed, it is returned to the STA.
4 f.
The STA performs
...e datt reduction, gathers the data from volume 9 and enters the action limits on the data sheet.
g.
The shift foreman reviews the test results and determines operability.
h.
The completed procedure is send to the surveillance group who reviews for ALERT LIMITS from volume 9 and l
update the In Service test data records.
i.
The section XI coordinator review the test procedure and results.
The August 21, 1991'and November 14, 1991 surveillance reviews were completed by a procedure writer.
The October 9, 1992 review was completed by a system engineer.
The change to have system Engineers review completed STP's took place in early 1992.
It is agreed that the form where the tests dtn.. are logged does not have any place for recording ne alert
- ~
hi/lo range. There is not enough information on the data sheet to make an accurate determination.- The -acond check is for the IST program engineer to look once a month for alert frequency.
There is a need to look at the shift foreman log of August 21, 1991 to find out when the TS action statement was exited.
An investigative action was established that required 4
Nuclear Engineering to evaluate other pump test reviews performed by the same test reviewer in order to determine if they are correct.
i 92NCRWP/92TPN052.PGD Page 16 of 23 W
NCR DC1-92-TP-N052 Rev.00 Draft: February 4, 1993 The reportability of the missed alert condition was reviewed.
It was determined that the TS was not' complied with on the two events.
An action was given to Regalatory Compliance to submit-an LER to the NRC, to report this event.
Nuclear Engineering will look back at the surveillance records to evaluate the data and determine if there were additional instances of alert.
The frequency between the past recorded tests and the required accelerated frequency for equipment on alert were discussed.
Proceed with the LER, describing the two events and associated corrective actions.
The TRG was informed of specific test measurement issues involving the ASW Pump STP's.
In particular, the proper operation of the annubar and adequate venting of pump discharge pressure indication gauges.
TRG TO RECONVENE ON FRIDAY OCTOBER 23. 1992 TO REVIEW THE INVESTIGATIVE ACTIONS RESULTS AND ESTABLISH THE ROOT CAUSE AND CORRECTIVE ACTIONS.
2.
On October 23, 1992, the TRG reconvened and considered the following:
a.
Agenda for the TRG:
1.
Discuss Root Cause.
2.
Review of past events.
3.
Determine any additional CAPR's.
b.
Review of the NCR write-up.
c.
Immediate actions:
Per procedure the STA's actions are to identify the prerequisites and to look up the coefficients for the annubar which is to be used.
Therefore, there was no failure to review the pump curve, but the wrong pump curve was attached to the work package.
92NCRWP/92TPN052.PGD Page 17 of 23
l l
l NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 1
It may be helpful to split these tests into Unit 1 and Unit 2 specific.
This will permit to attach unit specific pump curvcs to their respective Unit Test Procedure.
This method, will reduce the chances for an STA to make a personnel error by attaching the wrong pump curve to the test data sheets.
l Discussion on replacing the two STP's.(P-7B for the ASW and P8B for the CCW) with either four procedures l
(unit specific) or ten procedures-(for each Unit and pump specific). It was agreed that this should be considered.
l Since it has been established that the control room personnel is appropriately trained.
It appears in this instance that the test reviewer was not appropriately trained, Potential contributor / causal factor is that Volume 9 is not well organized.
For the purpose of this TRG, a surveillance engineering group will be formed to: (1) review and enhance the surveillance process; and (2) develop a l
reviewer check list.
Larry Cossette will form and lead the group.
Operations management should reemphasize the importance of the surveillance's review.
Discuss the importance of writing guides for STP's, conduct and performance of STP's.
It was agreed to include the corrective actions stated in QE 9506 for the second event.
One investigative action was assigned:
1.
Regulatory Compliance (P. Dahan) to contact the Operations Department Shift supervisor and initiate a discussion with the surveillance l
Engineer, STA and other operations personnel l
invrlved in the conduct of STP P-7B surveillances. Event 1, this is an operations issue.
Event 2 this is an Engineering issue.
TRG TO RECONVENE ON 10/30/92 TO REVIEW THE ROOT CAUSE, INVESTIGATIVE ACTIONS AND CORRECTIVE ACTIONS TO PREVENT RECURRENCE.
92NCRWP/92TPN052.PGD Page 18 of 23 i
l
NCR DC1-92-TP-N052 Rev.00 Draft: February 4, 1993 3.
On November 2, 1992, the TRG reconvened and considered the following:
Discussion on the reason for the high reading.
The reading was erroneous due to the' suspected flow readings caused by the positioning of the annubar.
When the high reading of the test were noted, the test: shall have been reruned.
Nuclear Engineering (L. Cossette) will explain in the initiating AR A0280428,- the reason for the high readings.
Regulatory Compliance (David Sisk) will clarify his answer in the AE / 2 of the tracking AR.
The corrective actions will be addressed in the QE.
Recommendation # 2 for STA participation will not be accepted as a corrective action.
The current administrative burden for alert ranges should be included in the STP.
It should address that subject, but should not be made a procedure requirement.
It should be considered that anytime _the volume data is split and the need arises to consult more than one document, the chances for human errors increase.
There is a need to verify that the pump tests cre in accordance with volume 9.
It was recommended to split the procedures for pumps and attach the pump curves to the unit specific procedure.
The pump curves are not required to become an attachment to the current procedures.
In this event, the STA attached his own curve to the test procedure and the shift foreman did not review the package to ensure that the correct pump was attached.
Nuclear Engineering (Larry Cossette) to look at the
)
process and come back with some recommendations.
The 1
problem with the ASW pumps is the measurement.
This shows a weakness in the program.
The STA run the pump tests together for both units.
i i
l
-l i
l 92NCRWP/92TPN052.PGD Page 19 of 23 l
J i
s NCR DCl-92-TP-N052 Rev.00 f
l Draft: February 4, 1993 If we stick to volume 9, we need to get some recommendations and take a hard look if we should split l
the pump tests.
l The need was stressed to look at the volume 9 process. It I
was recommended to split the pump tests and make them l
Unit and pump specific.
A corrective action to prevent recurrence was assigned to the TRG chairman and
...a senior nuclear engineer to discuss with engineering management (Steve Banton and MCoy Burgess) the feasibility to split the Surveillance
(
Test Procedures (STPs) to be Unit and Equipment specific l
to reduce the possibility of error.
TRG TO RECONVENE IN EARLY DECEMBER TO GET AN UPDATE ON THE ASSIGNED ACTION AND TO FINALIZE THIS NCR.
4.
On November 6, 1992, the TRG reconvened and considered l
the following:
The TRG reconvened to review the write-up of LER 1,
024-00.
A proposed root cause submitted by Regulatory Compliance (David Sisk) was read and tabled.
Actions taken:
The LER was reviewed and the root causes were changed to read as follows:
Both events were due to personnel errors (cogi._ cive) caused by inattention to details.
Event 1.
On August 21, 1991, data was recorded from the wrong pump curve.
Subsequent reviews by a PG&E Shift Foreman and a surveillance test performer failed to recognize that the wong ASW pump curve and pump data had been used in performing STP P-&B.
The shift foreman then incorrectly determined the pump to be operable.
Event 2.
A PG&E Engineering test reviewer in reviewing l
STP P-7B performed on November 14, 1991, l
failed to recognize that the as-found ASW 1-2 DP was in the Alert Range.
92NCRWP/92TPN052.PGD Page 20 of 23 i
l
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 The Safety Analysis was changed to read as follows:
Subsequent to the " Action High" test recorded on August 21, 1991, there have been seven tests performed with no other high action value recorded.
From the data trended, 4
this particular finding was one standard deviation from the mean which statistically is insignificant (mean being 99.01% and its standard deviation being 4.46).
Furthermore, the annubar mispositioning problems have been documented and dispositioned on several Action Requests.
Proper annubar use and resultant flow calculations (from the proper flow constant) are crucial to the successful performance of STP P-7A and STP P-7B.
PG&E believes that the positioning of the annubar may have caused the abnormal readings of August 21, 1991 and November 14, 1991.
The corrective actions to prevent recurrence wera changed to read as follows:
Event 1:
1.
Independent verification steps will be added to the IST pump test procedures to check that Volume 9 data has been properly _ selected and entered into the STPs.
2.
This event is being included in the-operations update requalification training.
3.
An operations incident summary will be written and discusses with Operations personnel.
Event 2:
I 1.
An ASME Section XI computer trending program is being l
implemented.
This program provides the trending of all section XI IST component data and compares the current need for an alert frequency.
This information is provided to the test reviewer at the time of data input into the section XI trending _
program.
This alert evaluation feature program will be discussed in the next revision of Administrative Procedure (AP) C-3S1.
2.
A training session covering ASME Section XI l
requirements for alert test frequency tracking was held with the appropriate procedure reviewers.
92NCRWP/92TPN052.PGD Page 21 of 23 l
)
,i4 NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 Three additional corrective. actions ~to prevent recurrence were initiated as follows:-
Operations'to write an' incident summary-to a.
reemphasize the importance.of tho' surveillance Test process and train the" operations group;in the importance of this problem.-
b.
Nuclear Engineering to:rsvis~e.the'IST Pump procedures to add an independent verification of.
volume 9 data; entry.
1 Due'to its significance, the Training' group to c.
add this event to the operations requalification training.
The following investigative action.has been' changed to a prudent action:
Develop an-Action Plan for formingia surveillance task group to' review and enhance.the.IST pump.
surveillance test process. 'The group;.shall include.
HPES representative, Operations ~-and SurveillanceL groups participation.
THE TRG TO RECONVENE ON DEC N ER 15. 1992'TO REVIEW Ymm Ce a wdiva ACTIOMs RMn Fi==LIIE'THE MCR WRITE-UP BEFORE PSRC.
The TRG did not' reconvene on December 15, 1992 nore on January 13, 1993 as scheduled.
5.
The TRG reconvened'on January 15, 1993 and' considered the following:
Review of the write-up and the root.-cause analysis.
Discussions on rewording the root cause analysis'.
A new draft of the NCR write-up.to be distributed to'all TRG members by. 01/22/93.,
Comments by the TRG members of the NCR Draft' write-up-to?
Regulatory Compliance-(Paul Dahan) and the-TRG' chairman (Chris Groff)' by 01/28/93.
NCR CLOSURE ECD:
MAY-1, 1993.~
92NCRWP/92TPN052.PGD Page 22 of 23-
.. e NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 TRG TO RECONVENE ON FEBRUARY 5, 1993 FOR FINAL' REVIEW AND 81GN-0TT.
6.
On February 4, 1993, the TRG reconvened and reviewed the write-up and root cause analysis for final comments.
The TRG will not reconvene, the minutes will bedistruted to the members and the NCR will'go to PSRC and QA for i
~
closure.
I.
Remarks:
None.
J.
Attachment.
l Root cause analyris.
92NCRWP/92TPN052.PdD Page 23 of 23
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 ROOT CAUSE ANALYSIS
- Page 1 of 3 CAUSE EFFECT EVIDENCE BARRIER FIRST EVENT: ASW Pump 1-2 It was not recognized Test The method of declared operable contrary to that ASW pump 1-2 was procedure determining base the requirements of TS 3.7.4.1
( ] $e 7 1 n high history sheet line for the ASW and ASME Section XI Article for STP P-7B.
pump is cumbersome.
IWP 3230.
Operators are required to go to another document to obtain base line data rather than having the data available as an attachment. CC.
WHY?
It was not recognized Acceptance criteria
" volume 9" The shift foreman that ASW pump 1-2 was above was recorded from Asw pump curves.
failed to recognize the action high level for DP.
[n that the wrong pump u P curv r
ASW pump curve 1-2.
curve was attached to the package.
Annubar readings are crucial to proper data collection.
Based on other valid tests on this pump, data obtained in this specific test is questionable.
CC.
WHY?
Acceptance criteria wts The wrong ASW pump ASW pump 1-2 Independent recorded from ASW pump curve curve was used from history data verification of
1""* 9"*
2-2 instead from ASW pump sheet and STP data obtained from curve 1-2.
P-7B test
" volume-9's."
data.
Use separate
" volume 9" for each unit.
CC.
=
CONTRIBUTORY CAUSE.
92NCRWP/92TPN052.PGD Pace -24 of 23
?-
j NCR DCl-92-TP-N052 Rev.00
(
Draft: February 4, 1993
~'
i:
i ROOT CAUSE ANALYSIS - Page 2 of 3 CAUSE EFFECT EVIDENCE BARRIER WHY? The wrong ASW The tott performer T.S.
4.0.5 and ASME Color coding of Unit i
pump curve was used failed to identify that Section XI.
l'and 2 pump curves
- {"Q"#'*"**
in " volume 9" from
" volume 9".
WHY?. The test Personnel error.
Investigations.
Write an operation performer failed to Inattention to.
summary to identify that the-detail, in that the reemphasize the wrong 1 curve-was' STA did not ensure importance of the-utilized. CC.
that~the proper STP' process.
l curve from " volume 9" was used.
Root Cause.
4
.i CC.. =
CONTRIBUTORY CAUSE.
l i
3 l
i i
f e
-4 i
r
[
.1
.i -
4 s
)
1 92NCRWP/92TPN052.PGD
'Page 25 of 23
NCR DCl-92-TP-N052 Rev.00 Draft: February 4, 1993 ROOT CAUSE ANALYSIS
- Page 3 of 3 CAUSE EFFECT EVIDENCE BARRIER SECOND EVENT: ASW Pump 1-2 Asw pump 1-2 Test declared inoperable in increased testing procedure accordance with TS 3.7.4.1 due f"*"*"CY "** *1***d' history sheet to missed ASME section XI IWP for STP P-7B.
3230, increased frequency testing.
WHY?
ASW pump 1-2 increased STP P-7B surveillance
" Volume 9" System Engineer to testing frequency was missed.
test reviewer failed pump curves.
review the test to recognize that ASW T.S.
4.0.5 results.
pump 1-2 was above the alert level for and ASME DP.
Section XI.
ASME Section XI trending program..
WHY?
STP P-7B surveillance Personnel Error. The ASW pump 1-2 Adequate training test reviewer' failed to surveillance test history-data on-ASME section XI recognize that ASW pump 1-2
- 'i [ **'"
sheet and STP test reviews was j
was'above the alert level for acceptable since the P-7B test not provided. CC.
DP.
was similar to data.
3 Previous test Provide adequate results.
. training on ASME Root Cause.
Section XI test reviews.
j CC.
=- CONTRIBUTORY CAUSE.
1 4
i 9?NCRWP/9?TPN057.PGD Pace 26 - ' of r
) -