ML20202G712
Text
.- . .. _
_.-___._..m _ _ _ _ _ _ . . _. -_ _ . _ . _ _ . _ . .
4 ,
d, .
- c. 4-00f '
4W November 20,1997 -
c rom: K. Jenison, Senior Resident inspector, Susquehanna (SSES), Division of Reactor Projects (DRP), NRC Region I (RI) l To: C. Anderson, Chief,- Reactor Projects Branch 4, DRP, RI I l
Through: - P. Swetland, Project Engineer, RBP4, DRP, RI:
Attached
References:
- 1. Fewell issue (amended 11/23/97)-responses provided to Region I in a previous email
Subject:
Review of Allegation Related issues On November 5,1997, the NRC attended a meeting with PP&L management and
- representative from PP&L Corporate Auditing (CA) to discuss a set of 4 related issues _
involving operator performance and equipment panel testing. These issues were reviewed by an allegation panel on November 19,1997 during which the decision was made to interview the PP&L corporate auditors again to determine what their progress was since
. the November 5,1997 meeting and determine the status of the licensee actions. The interviews were performed on November 20,1997. -
The four issues will be referred to out of numerical sequence in order to increase the clarity I of discussion. i 1
y issue #2 - ESS Transformer Testina.
l The inspectors determined that licensee management (Markowski, Adelasi, Chamblis) was )
aware of the ESS transformer surveillance problems and computer problems on April 9, and I March 21,1997. At the time that PP&L's April 9,1997, letter was written the ESS I transformer surveillance problems known to SSES management were not assumed to be related to the NPO performance errors, and were thought to be related to computer hardware and software problems. No new information was determined by the inspectors
- during the November 19,1997 interviews with the PP&L Corporate Auditor which disputes the correctness of the April 9,1997 letter with respect to the ESS transformer issue. A PP&L CA report that addressed issue #2 was produced by the licensee and reviewed by the inspectors issue #2 was addressed in the attached file which was forwarded by email (Jenissn/Swetland).
Issue #3 - Radwaste Control Panel Testina There is no data to indicate that a Radwaste control room personnel performance issue or a test data issue was known to licensee management prior to April 9,1997. Performance issues were identified in a review that was commenced by PP&L CA after April 9,1997.
1This review ia in its final stages of completion and a draft report is expected to be placed in a final form prior to November 30,1997.
4 9902050260 990129 F x
PDR FOIA t SORENSEN99- 36 PDR >m
^
^
i t jj { (
h o r- _ . . .. . . . _.
t ?. : ;
1 2-l The inspector reviewed a sample of the raw CA data, a draft CA report and a PP&L memo
, describing the issues (Urban / Jones dated November 19,1997). Based on this review of preliminary data, the inspector determined that there are performance related issues with ,
approximately half of the ASOs assigned to the Radwaste control room. The following i performance related issues were identified:
During an interview with PP&L CA, one ASO admitted signing for an unperformed panel test on one occasion.
One ASO admitted that there may have been an occasion that their signing was inaccurate Three ASO stated that when they became aware of the requirement that they performed and documented the panel testing (meaning that prior to some date they had not performed testing as documented)
Testing / computer data discrepancies exist that suggest + 3 the statements made by other ASOs with respect to their January to June 19Y adwaste alarm panel perforrrance may not be validated by the plant computer. -
Computer accuracy with respect to the Radwaste panel testing is very high. Three separate known conditions exist which could prevent the unit 1 and 2 computers from not recognizing a valid radwaste panel alarm test. The three possible conditions are of an extremely low probability and as far as the available data would indicate were limited to two instances over a period of July 1997 to December 1997. This preliminary data implies that the test data for January 1996 to July 1996 should be relatively unaffected by the three identified conditions. A fourth testing configuration was identified by the inspectors. This configuration was communicated to the CA on November 20,1997. The CA agreed to review this fourth possibility and to determine if based on the completion of their preliminary review there is a need to expand their radwaste control panel sample period.
The inspectors concluded:
There are personnel performance and integrity issues associated with
- radwaste alarm panel testing.
The PP&L and NRC preliminary radwaste alarm panel testing data reviews indicate that there is no computer software or hardware related root cause for the computer to not record the performance of the alarm test. This is despite the possibility of some very low l probability computer / unit configuration related degradations.
L:
j . There is no evidence that SSES or PP&L corporate management was J awaie of.the personnel performance when the April 9,1997 letter to L ~ the NRC was written. The radwaste control panel alarm test issues
!- were identified by CA after April 9,1997, (and therefore also the l
. 3/21/97 PEC). l l
Issue #4 - Main Control Room Panet Testino i l
f l
l
+ .,- - - - , - - - , ,, , - - - - - , - - - - ,- - , . . , , ,
The inspectors reviewed a sample of the raw CA data and a PP&L memo describing several testing issues (Urban / Jones dated November 19,1997). The inspectors reviewed the raw data to determine if there were personnel performance, computer data, and/or computer / plant configuration related issues associated with main control room panel testing.
The inspector also conducted interviews with Mr. Chamblis, Mr. Markowski, Mr. Peal, Mr.
Kuzynski, approximately six Plant Control Operators, three Unit Supervisors (US), and two Shift Supervisor (SS). None of the interviews identified that SSES management was aware of PCO performance related issues prior to April 9,1997. Each of the individuals interviewed by the inspectors maintained that the panels were tested as directed by SSES administrative requirements and shift supervision (ie SS direction). One discrepancy was identified by the inspectors and CA independently. Different SS had different expectations of what the administrative requirements meant and required. The management expectation issues is being addressed by both SSES management and the CA. A CA report is in its final stages of completion and is expected to be placed in a final form prior to November 30,1997.
The inspectors came to the following conclusions:
There is no data to indicate that a control room personnel performance issue was known to licensee management prior to April 9,1997 (and therefore the 3/21/97 PEC). A performance issue was identified in a review that was commenced by PP&L CA(after April 9,1997). In addition, a management expectations issue (see below) was identified by CA after April 9,1997.
There were significant differences between the testing schemes used to perform control room nanel testing depending on which shift management was requiring the testing.
With one exception there is no data to indicate that a control room (PCO) personnel performance issue exists with respect to the alarm testing of the main control room. During an interview with PP&L , one PCO admitted signing for an unperformed panel test on one occasion. This admission seems to be a serious situation for a licensed operator. It is the opinion of the inspector that the licensee has not been quick to resolve this singular instance.
Testing / computer data does not exist to validate the performance of some control room test activities. For those activities that had computer validation capability no computer data entry or personnel performance issues were identified to PCO performance.
Issue #1 - PP&L Manaoement Statements An April 9,1997 PP&L letter states:
i .. - .
The NPO's pattern of behavior demonstrated by the performance of surveillances and rounds was found only in a particular activity (routine, repetitive NPO rounds) in a specific location (E Diesel Building).
During a March 21,1997, enforcement conference PP&L management made several statements that support the above written statement.
A November 5,1997, meeting between the NRC and PP&L CA was held to discuss the activities of the CA that were intended to determine whether the written statements in the April 9,1997, PP&L letter and the statements made on March 21,1997, by PP&L management were factual.
A number of interviews were conducted by CA in pursuit of the April 9,1997 letter, prior to November 5,1997. However, the NRC requested, at the completion of the November 5,1997 meeting, that Corporate Auditing conduct additional interviews with Mr. Peal, Mr.
Markowski, Mr. Kuzynski and Mr. Chamblis; and review the statements made during the taped March 21,1997, meeting. On November 19,1997, the inspectors confirmed that a communications error had occurred. The PP&L Corporate Auditors had not conducted the additional interviews and had not reviewed the statements made during the March 21, _
1997 meeting. PP&L licensing (Mr. Kenney) informed the Lead CA(Mr. Urban) that there were no statements of interest in the taped conversations and as a result the Corporate Auditors did not do an independent review of the taped meeting. Following discussions at the November 20,1997 meeting between the resident inspectors and the Corporate Auditors, the Corporate Auditors agreed to conduct these additional interviews.
The inspectors discussed the PP&L March 21,1997 meeting statements with Mr.
Swetland, Mr. McDermott, Mr. Kenney, Mr Kuzynski, Mr. Chamblis, and Mr. Urban. The inspectors could find no evidence to indicate that the licensee made incorrect or inaccurate statements during the March 21,1997 meeting or in the April 9,1997 letter. However, subsequent to the April 9,1997 letter, the licensee (CA and Mr. Jones) became aware of performance problems in the main control room and the radwaste control room which may indicate that statements made by PP&L management were inaccurate at the time that they were made. However, these potential inaccuracies were not known to PP&L management at the time that they were made and are still preliminary at this date.
The inspectors came to the following conclusions:
There is no data to indicate that personnel performance issues concerning panel testing other than at the E diesel alarm panel were known to licensee, prior to the April 9,1997 PP&L letter or the March 21,1997 enforcement conference.
Subsequent to the April 9,1997 letter, the licensee became aware of performance problems in the main control room and the radwaste control room which may indicate that statements made by PP&L management were inaccurate at the time of the March 21,1997 meeting and the April 9,1997.
The licensee has not taken quick action to address two individuals that have admitted signing SSES documentation for activities that were not perforneed.
CA has been aware of this information for several weeks, Mr. Jones has
- - - - --- - - - - = ' -- '
I. .
l o. - .
1 i
. been aware of the data for several days. The involved individuals have not '
i either been retrained, placed in the SSES bahavior modification program, been found capable of performing safety related activities or removed from !
performing safety relate activities, in the inspectors' opinion this is a very '
serious situation.
i i
l l
l l
4 9 6 4
l l i
I 4
i l
l
Attachment 1 This attachment was provided by a previous emali to Region 1. However, because of update modem problems at the site it looks like Region I did not receive a copy. The concerns are as understood by NRC on receipt of the issues and relate in a general way to concerns received by PP&L under separate circumstances.
The following sections address concerns raised to J. Fewell, Regional Counsel, on June 20,1997, by an attorney for the allegers. As directed by Regional management, subsequent conversations were held with the allegers to obtain details of their concerns.
These conversations were documented in memorandums to Allegation File RI-97 A-145.
Concern 1: "ESS transformer alarm tests were not routinely performed prior to the June 1996 time frame based on computer records."
The resident inspectors reviewed a PP&L Corporate Auditing (CA) report dated October 15,1997, and Operations Department reviews dated March 26,1997, (Revision 2) and April 29,1997,(Revision 3). The inspectors reached the following _
conclusions:
Unit 1 computer records (revision 2) supplied to Keith Davis, Union Steward, by PP&L show sporadic records of alarm tests at the ESS transformer local alarm panels. Following a detailed review of the alarm circuits and field tests of the -
alarms, several equipment problems were identified and the capability of the Unit 1 and Unit 2 computers to keep records for the ESS transformer alarms was better understood by the licensee.
- a. Failures exist in the alarm circuits from ESS transformers 101 and 111 to the Unit 1 computer.
- b. The scan rate for the Unit 1 computer is 2 seconds and the functional ESS transformer alarms can be tested and reset without creating a Unit 1 computer record.
- c. Failures existed in the alarm circuits from ESS transformers 111 and 201 to the Unit 2 computer.
- d. The scan rate for the Unit 2 computer is 0.10 seconds and the functional ESS transformer alarms can NOT be tested and reset without creating a Unit 2 computer record.
- e. A comparison of the Unit 1 and Unit 2 alarm records for ESS transformer 211 was made and provides additional evidence that the conclusions reached above are supportable.
Both the Operations Department and CA reviews of Unit 2 computer records for ESS transformers 101 and 211, for the period January 1996 to June 1996, found no instances were a required alarm test was not performed. This pcrtion of the l allegation can not be substantiatud.
Recommended NRC Inspection Report input:
O2.1 ESS Transformer Local Alarm Panel Tests i
- a. Scoce (71707) i i
The licensee identified an apparent problem with the alarm circuits for the !
ESS transformer alarm panel. The inspectors reviewed two revisions of an i Operations Department review of ESS transformer alarm test performance dated March 26,1997 and April 29,1997. In addition, a PP&L corporate review of these issues dated October 15,1997, was also reviewed. Each ,
PP&L review examined the reliability of ESS transformer alarm circuits and l the accuracy of related Unit 1 and Unit 2 computer records. I
- b. Findinas and Observations i i
The inspectors made the following observations regarding PP&L's findings:
Failures exist in the alarm circuits from ESS transformers 101 and _ j 111 to the Unit 1 computer.
i
- The scan rate for the Unit 1 computer is 2 seconds and the functional ESS transformer alarms can be tested and reset without creating a i Unit 1 computer record. l Failures existed in the alarm circuits from ESS transformers 111 and 201 to the Unit 2 computer.
- The scan rate for the Unit 2 computer is 0.10 seconds and the functional ESS transfc.<mer alarms can NOT be tested and reset without creating a Unit 2 computer record. .
- A comparison of the Unit 1 and Unit 2 alarm records for ESS transformer 211 was made and provides additional evidence that the conclusions reached above are supportable.
The inspectors' evaluation of documentation from the Operations Department reviews and the PP&L corporate review of ESS transformers alarm test records found no examples of a required alarm test not being performed as a result of a human performance error. Based on the' reliable computer records for specific ESS "ansformers (captured by the Unit 2 computer), the inspector concluded that the pattern of activity showed that NPOs were performing the required tests during routine rounds between January 1996 and June 1996.
The inspector reviewed a comparison of Unit 1 and Unit 2 computer records for tests of ESS transformer 211 and found that the records support PP&L's conclusion that ESS transformer alarm tests of shorter than 2 seconds will not appear on Unit 1 computer records.
- 1 l
i i
Concern 2: "Other alarm tests were not performed and PP&L management knew this l when the April 9,1997, letter to the NRC was written. The April 9,1997 letter states " The NPO's pattern of behavior demonstrated by the performance of surveillances and rounds was found only in a particular activity (routine, repetitive NPO rounds) in a specific location ("E" Diesel Building). Moreover, we found no evidence that these issues were widespread." l This issue was reviewed by the inspectors. It was noted that Kevin Chamblis, SSES j Operations Manager and John Edwards, SSES Human Resources, were specifically named !
by both allegers as individuals who knew the tests were not being performed. One alleger stated that this information was provided to Chamblis and Edwards by Keith Davis, Union l Steward, during a February 1997 meeting. One alleger also named Tom Markowski, Operations Supervisor, Mike Peal, Shift Supervisor, and George Kuczynski, General Manager. Three examples were provided by both allegers: the ESS transformer alarms, the radwaste OC322 alarms, and the control room alarms.
The PP&L CA investigation questioned a number of managers involved with writing the April 9,1997 letter regarding what they knew at that time. They intervie,wed the Sr. VP _
Generation / Chief Nuclear Officer, VP- Nuclear Operations, Manager - Nuclear Assessment Services (former), and the Supervisor - Nuclear Licensing. Each person told CA that they had no knowledge as of April 9,1997, regarding other alarm tests that were not l performed.
I interviews between the resident inspectors and the CA personnel indicated that all of the personnel indicated above were not interviewed and the CA agreed to perform additional interviews.The CA interviews that were conducted used a standard set of questions that would lend themselves to the broad questions regarding who knew what and when.
The allegation details received by PP&L were apparently different from those received by the NRC and the NRC did not refer officially its set of allegation iccucs to PP&L. This situation made resolution and comparison of all the issues difficult. The PP&L investigation sought to establish whether a specific Shift Supervisor was told by Keith Davis that plant i control operators (PCOs) were not testing the control room annunciators. Based on this, l CA interviewed both Davis and the SS. CA determined that both of them denied that a conversation of this sort took place.
During the enforcement conference, PP&L representatives made several statements about a number of questions regarding alarm tests. In order to establish the basis for PP&L's statements, the inspectors interviewed Mike Peal, Tom Markowski, Kevin Chamblis, and George Kuczynski. Each was asked to explain the comments made at the enforcement
, conference. Each person was also asked whether they personally had any information regarding other alarms panels that were not being tested as a result of human error, prior
! to April 9,1997. In each case they stated no.
Based on the information available as of November 12,1997, and NRC conclusions regarding the ESS transformer alarm test records, the inspectors found no information to suggest that PP&L management was aware of alarm panel test performance errors, other than on OC577E, prior to April 9,1997, Note: this is a performance related statement and not a statement that addresses hardware problems or computer soft ware problems. SSES
._ __ _ - . . _ _ _ . . _ ~ _ _ _
6 management were each aware of ESS transformer computer and hardware problems as early as the end of February 1997, but did not relate these issues to human performance problems and were taking (in the opinion of the inspector) adequate and timely corrective action.
It is the intention of the inspectors to review the technical issue related to ESS transformer '
tests, radwaste control room panel .est, and main control room panel tests when the data is made available by the licensee.
e 1
e 4 s
n r
I I
I 1
I l
1 l
j
4 s
Attachment 2 The following writeup was provided by email to RI and is attached to increase the clarity of the memo to which this is attached.
1 "E" Emeraency Diesel Generator (EDG) Alarm Panel OC577E Ooeration l On February 13,1997, the licensee performed routine alarm testing of the OC577E Alarm l Panelin the "E" EDG building. During the performance of this routine test a plant control operator (PCO) noted that a specific control room alarm panel (AR-016-F02)did not reflash as designed. The inspector reviewed this panel test failure and the licensee's corrective I actions.
Observations and Findinos Following an alarm panel test failure, the licensee initiated condition report (CR) 97-0289 and work authorizations (WAs) S70808, S70529, S70675 and W66261. The inspector reviewed the associated WAs and drawing FF65111. WA S70529 was initiated on February 15,1997 to determine why the control room did not get a reflash during the _
February 13,1997, panel test. The investigation WA revealed that a failed panel reflash relay caused the test failure. I Each unit has two computer points (EGZ12 and EGZ14) which respond to the OC577E panel alarm test but have no further noun name designation. I l
At the time that the February 13,1997, failed alarm test was performed: l Unit 2 computer point EGZ12 was not functional- WA W66261 was written on March 10,1997, to repair Unit 2 computer point EGZ12, which had been degraded since August 13,1996.
Unit 2 computer point, EGZ14, and Unit 1 computer points EGZ12 and EGZ14 were determined to be active and functional during the period of time that included the panel test in question.
A separate plant testing activity was being performed under test procedure (TP) TP-024-149. This second test procedure controlled the positions of breakers OB56502A and OB56503A under permits 1-96-1808 and 1 0325. The TP and the two permits caused several alarms to be activated in the control room and locally. The presence of these alarms associated with TP-024-149 prevented an initializing alarm from the February 13,1997 test, because the associated control room alarm was already alarmed. Because the control room alarm that was associated with the February 13, panel test was already alarmed it should have reflashed. However, it did not reflash because the reflash function had failed.
The February 13,1997, f ailure of the reflash unit affected the performance of the EGZ14 computer points on Unit 1 and Unit 2 by not allowing the repeat of an appropriate signal to
e s
be recorded at the EGZ14 points. The reflash unit failure did not affect the EGZ12 points.
The Unit 1 EGZ12 computer point provided an accurate indication that the February 13, 1997 OC577E test was correctly performed despite the other computer point degradations.
After completing a review of the above data, the inspector determined that the alarming /reflashing of specific control room annunciators related to OC577E panel tests were affected by activities conducted under TP-024-149in combination with the failure of l a reflash unit. The Unit 1 and 2 computer points EGZ14 were affected by the failure of the l reflash function as these points did not a receive a repeat input signal. Computer point )
EGZ12 on Unit 2 was degraded during the February 13,1997 panel testing and did not l always accurately register the panel test. Unit 1 computer point EGZ12 was functional 1 throughout the February 13,1997 OC577E test and was unaffected by the other testing or j the reflash relay failure. There was no evidence to indicate that the interaction of the two tests combined with the failure of the reflasn component was other than a random j occurrence. Further, there is no evidence of previous repeated failures of the reflash units.
The licensee's corrective actions and root cause determination associated with CR 0289 were determined to be adequate and no violations of NRC requirements were identified. Despite the equipment failures, there was reliable Unit 1 (EGZ12 point) computer data available throughout the period in question, the alarm panel testing reviewed was _
performed from only one panel (OC577E).
The inspector reviewed the possibility that computer scan rates could have affected the recording of data on the Unit 1 EGZ12 point. A set of licensee test data was reviewed and it was determined by the licensee and verified by the inspector that it was not possible to ;
test and reset the OC577E alarm panel test function (using the methods normally employed during the NPO rounds) in a manner that was affected by the Unit 1 scan rates. Therefore the inspector determined that the speeds of the computer scan rates for units 1 and 2 were not a factor in the failure to record data on February 13,1997.
The inspector reviewed a sample of test and maintenance data to determine if there were similar or repeated test failures of control room reflash relays. No evidence of repeated similar failures could be identified.
The inspector compared the findings from this inspection to two other NRC inspection activities that are still in progress. The two activities are a review of radiological waste control room alarms and a review of ESS transformer alarms. The inspector was not able to identify any common initiating or contributing conditions in either of the other reviews that corresponded to the root cause of the failed February 13,1997 test.
Conclusions Following a February 13,1997, alarm panel test failure, the licensee initiated a condition report and work authorizations to determine why the control room did not get an alarm reflash. At the time that the alarm panel test was performed a separate plant testing activity was being performed. The presence of the alarms associated with the second test caused the control room alarms that would have indicated the February 13,1997, OC577E alarm panel test to be in the alarmed condition prior to the performance of the alarm panel test. As a result, the alarm panel test should have caused a reflash in the control room.
The reflash condition did not occur because of a f ailed reflash component. Computer points EGZ12 & 14 are normally actuated by the OC577E alarm test. Although the
.. 4*l l I
$ l
)
functioning of the Unit 1 and 2 EGZ14 computer points was affected (did not record the February 13,~ 1997 test correctly) by the interaction between the two tests and the reflash component failure, there was no evidence to indicate that this condition was other than an isolated low probability instance. The Unit 2 computer point EGZ12 was inoperable on February 13, but the Unit 1 computer point EGZ12 was operable throughout the February 13,1997 test and accurately recorded the performance of the test. The licensee's corrective actions and root cause determination associated with CR-97-0289 were
' determined to be adequate. No violations of NRC requirements were identified.
j e
4 l
i l
.p n . -
1 s ,, --
ALLEGATION DISPOSITION RECORD Rev. 10/01/96 j
'All:g tion No.: RI-97 A-0145 Branch Chisf (AOC): Anderson l
M At M7-4-00d Sita: Susquehanna Acknowledged: Yes X No __ N/A ___ l Pen:1 Date: 11/19/97 Confidentiality Granted: Yes ._._ No X l
Issus discussed (if other than original allegation): ,
- Allig::r contacted prior to referral to licensee (if applicable)? Yes No ALLEGATION PANEL DECISIONS (Previous Allegation Panels on issue: Yes __ No _ ,_)
Att:ndees: Chair - Nicholson Branch Chief (AOC) - Swetland ;
l
' SAC - Vito J
- l. 01 rip. - Logan RI Counsel - Fewell Others - Walker, Linville DISPOSITION ACTIONS: (State actions required for closure (including special concurrences), responsible person, l ECD cnd expected closure documentation) l I
l 1) Ris establish what computer system questions were referenced at the 3/21/97 PEC. Verify not ESS, CR I and radwaste alarm panels. Document in memo to file for the allegation file and provide to SAC. j
- I Responsible Person: Anderson ECD: 11/30/97 i Closure Documentation:' Completed: l 2)- Ol interview PCO's including one who PP&L says admitted falsifying CR logs (within 30 days of PP&L report). Ol to open an Assist to inspect to obtain and review PP&L report. Of will contact PP&L Corporate Audit staff to secure info regarding PCO falsifying CR logs.
Responsible Person: Letts/ Logan ECD: 11/30/97 i Closure Documentation: Completed: {
I
- 3) Repanel after receipt of PP&L report of ASO falsification of radwaste alarm panel logs (PP&L report due ~
1/98).
Responsible Person: SAC ECD: 12/97 Closure Documentation: Completed:
l . ,
- 4) Send update letter with results, excepting PCO and ASO problems. !
Responsible Person: SAC ECD: 12/15/97 !
Closure Documentation: Completed:
I I Scfsty Significance Assessment: ;
i l Priority of Of Investigation: l M .
ARB MINUTES ARE REVIEWED AND APPROVED AT ARB i 0$2 44 C-GJ- A -004% $128 m&cck/#d wuAD $ VME
'A # \
a n ic y + b Js m w w & rs %~' W p ww(Bfo;c
~&W aswa #(7 p.~v, op p;-
}6M#h ') W,