IR 05000528/1983047
| ML17298B470 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 10/16/1984 |
| From: | Gage P, Johnson P, Miller L NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML17298B469 | List: |
| References | |
| 50-528-83-47, NUDOCS 8411060514 | |
| Download: ML17298B470 (24) | |
Text
U. S.
NUCLEAR REGULATORY COMMISSION
REGION V
Report No.
Docket No.
50-528/83-47 50-528 License No.
CPPR-141 Licensee:
Arizona Public Service Company P.
O. Box 21666 Phoenix, Arizona 85036 Facility Name:
Palo Verde Nuclear Generating Station - Unit 1 Inspection at:
Palo Verde Site (Mintersburg, Arizona)
Inspection conducted:
September 19-23, October 4-7, October 18-21, and November 1-4, 1983 Inspectors:
P.
H. J nson, Reactor Inspector P.
C.
Ga e, Reactor Inspector Date Signed lo/Ir/ay Date Signed Approved By:
eactor ProJects Section No.
L. ilier, Chi f R
o ti Date Signe Summary:
Ins ection on Se tember 19-'23 October 4-7 October 18-21 and November 1-4 1983 Re ort No. 50-528/83-47 Areas Ins ected:
Special inspection to evaluate allegations received regarding conduct of the prerequisite test program.
The inspection consisted of 234 inspector hours onsite and offsite by two NRC inspectors.
Results:
No violations were identified.
I 8411060514 801016 PDR ADOCK 05000528
DETAILS Persons Contacted
'AJ
~
IJ G.
>'cP W.
P.
-K.
'G J.
'C Kirby, Startup Manager Hayes, Unit 1 Startup Manager Roedel, Corporate (}A Manager Arasz, Startup QA/gC Document Review Group Supervisor Brandjes, Electrical Test Group Supervisor, Unit 1 Craig, Project Shift Director (formerly Startup (}C Supervisor)
DeVine, Test Review Group (TRG) Supervisor Gross, Compliance Supervisor Pankonin, Startup (}A/gC Manager Reardance, (}C Inspector (Level III)
Russo, Operations QA/gC Manager The inspectors also talked with and interviewed other licensee personnel, including test engineers, (}C document reviewers, document control personnel, and TRG members.
-'Denotes those present at the exit interview.
Pur ose of Ins ection This inspection was conducted to examine licensee activities related to the Unit 1 prerequisite test program, in response to allegations of improper or ineffective licensee performance associated with that program.
These allegations were set forth in an August 31, 1983, sworn statement made by the alleger to an investigator from the NRC Office of Investigations, San Francisco Field Office.
The alleger was further interviewed by the inspectors on September 19, 1983.
The alleger requested anonymity, although he acknowledged during the September
interview that inspection followup of his specific allegations might enable some persons at the Palo Verde site to ascertain his identity.
However, his name or identity was not disclosed or confirmed by the inspectors throughout the course of the inspection.
General Ins ection Findin s The inspectors began the inspection by interviewing the alleger.
In addition to the alleger a total of 16 other interviews were conducted with persons involved in the direction, conduct, or review of the test program.
These 17 interviews involved a total of 21 hours2.430556e-4 days <br />0.00583 hours <br />3.472222e-5 weeks <br />7.9905e-6 months <br /> of interview time.
The inspectors also examined the licensee's administrative controls and test procedures pertaining to the areas of concern; examined completed test documents and licensee reviews thereof; and discussed various aspects of the test program with cognizant licensee personnel.
Inspection findings related to the specific allegations are described in paragraph 4, below.
General inspection findings which relate to the overall scope or context of the allegations, or provide general background information, were as follows:
a.
The initial tests to be conducted as part of the Palo Verde startup test program were called prerequisite tests.
These generally
t ~
I consisted of wiring checks, instrument calibrations, circuit continuity checks, and other verifications of proper individual component operation or installation.
The licensee identified approximately 32,700 individual prerequisite tests for Vnit 1, as set forth in a Detailed Test Schedule (DTS).
Approximately one-third of these prerequisite tests were quality-related (i.e., in systems falling within the scope of the licensee's approved equality Assurance Program).
The startup test organization at Palo Verde was initially composed of APS and Bechtel personnel integrated into a combined Startup Department.
One licensee representative estimated that 1000 to 2000 electrical prerequisite tests were completed under this organization before October 1981.
At that time a distinct division of, responsibility was implemented, with Bechtel assigned principal responsibility for the conduct of mechanical and electrical prerequisite testing.
The licensee stated that this change was made to facilitate the accomplishment of prerequisite tests by Bechtel before systems were turned over to APS for preoperational testing.
Measuring and Test Equipment and Nuclear Steam Supply System Instrumentation and Controls (ISC) prerequisite testing were kept under APS control until January 1982, when all IGC was brought under APS Control.
During this period of separate testing responsibilities, the APS and Bechtel organizations functioned under separate administrative control procedures.
Mechanical and electrical prerequisite tests were conducted under Bechtel procedure AD-108, "Prerequisite Testing," while IRC tests were conducted pursuant to APS procedure 90GA-OZZ28, "Prerequisite Test Conduct."
The responsibilities for prerequisite testing continued as above until July 1983, when APS, Bechtel, and contract personnel were recombined into a single APS Startup Department.
The licensee stated that this restructuring was done to obtain more control of test completion on a system-by-system basis.
The administrative control remained in effect as before (i.e., Bechtel procedures governed mechanical and electrical tests, and APS procedures governed ISC tests) until September 27, 1983, when revised APS procedures governing all prerequisite testing were placed into effect, The licensee halted prerequisite testing in June 1982 due to questions regarding whether all test data demonstrated proper equipment performance.
A group within APS known as the Barrows Committee reviewed all prerequisite tests completed up to that time (approximately 4000, primarily electrical).
This committee made judgments on the acceptability of test data and identified certain tests to be repeated.
Testing resumed after an interruption of approximately two months.
The Test Review Group (TRG) was formed about this time to provide an additional level of test results revie Another examination of prerequisite test. results was conducted by the licensee in July-September 1983.
This included all electrical tests, all quality-related mechanical and ISC tests, and 50% of those pertaining to fire protection, except that tests accepted by the Harrows Committee were not reviewed further.
Licensee representatives stated that retests were required for approximately seven percent of the electrical tests.
Reviews to date called for retest of about two percent of ISC tests, one mechanical test, and no fire protection tests.
Most of the retests were required not because of unacceptable test results, but because of non-interpretable data or to provide better trackability (e.g.,
component serial numbers or nameplate data).
The licensee stated that integration of the test organizations, implementation of new administrative controls on September 27, 1983, and training provided to test personnel on the new administrative procedures were intended to address the weaknesses observed in the control and conduct of prerequisite testing.
Each plant system was also assigned to a principal startup engineer (PSE),
who was responsible for coordinating all prerequisite and preoperational testing for that system.
4.
S ecific Alle ations Allegations presented by the alleger and inspection findings related to them were as stated below.
The allegations are stated as characterized from the alleger's sworn statement.
a.
provided regarding the documentation program, and there were no established written procedures for the start-up documentation review program.
A roach to Resolution:
The inspectors interviewed cognizant members of the licensee's staff and examined pertinent records and procedures to determine the level of direction given for QC review of prerequisite.test documents.
Ins ection Findin s:
In his sworn statement the alleger indicated that he and an associate were assigned to perform a
QC review of completed test documents.
The statement noted that the two reviewers were told something to the effect, "Here's the document vault.
Review all the Q related tests."
The statement further indicates that when he asked about procedures covering the program, he was told there were none.
The QC reviewers'upervisor, a Level III QC Inspector, was interviewed regarding this statement.
He stated that the two were provided a copy of a September 1982 memorandum to QC personnel.
This memorandum, dated September 14, 1982, enclosed a copy of Startup QA/QC Department Instruction No. G-OOl, which concerned the filing, handling, and transmittal of prerequisite test data.
It did not address the QC review of the data.
The Startup QC Supervisor stated during an interview, however, that the reviewers were given verbal instructions to ensure the documentation was filled out
properly and the data were in accordance with the acceptance criteria, using administrative procedures AD-108 and 90GA-OZZ28 as applicable.
The alleger.'s statement indicates that he suggested to the Startup gC Supervisor that a written procedure (for the conduct of the (}C reviews)
was a necessity.
The alleger's statement indicated that the (}C Supervisor agreed, and that the alleger then wrote a
procedure which was approved and issued as Start-up Procedure (}CI-001.
This procedure was used by the two QC reviewers to conduct the QC reviews of completed prerequisite test documents.
The inspectors concluded that the alleger's statement was corroborated i.e., that no procedure for the QC review was initially provided, but that management was responsive in providing one following the gC reviewers'xpression of concern.
inadequate.
A roach to Resolution:
The inspectors examined the comments prepared by the alleger and an associate regarding test documents reviewed by the gC reviewers.
Interviews were also conducted and pertinent licensee procedures were examined to determine the licensee's established methods for conducting, reviewing, approving, and filing prerequisite test documents.
Ins ection Pindin s:
During a period of approximately six weeks in ofay and June 1983, the two QC reviewers reviewed 117 completed prerequisite test documents.
The reviews were conducted using a
five-page checklist developed by the gC reviewers.
This checklist was supposedly based upon procedures AD-108, 90GA-OZZ28, and gCI-001, discussed earlier.
Using this checklist, all of the 117 prerequisite test documents reviewed were rejected.
The inspectors examined seven of the 117 checklists in detail.
Each comment was evaluated against administrative procedures in effect at the time the test was conducted, to determine the validity of the comment.
The gC reviewers had made 92 comments on the seven checklists examined.
The inspectors'onclusions were as follows:
None of the 92 comments involved valid technical issues (i.e.,
which would place the test result in question).
Seventeen of the 92 were valid comments of an administrative nature (e.g.,
nameplate data not recorded;
"N/A" not entered in blank; "N/A" not initialed and dated; name 'signed but not printed; pages or attachments not numbered)
which did not affect the validity of the test results.
The remaining 75 comments were determined by the inspectors to be not valid.
Generally, these resulted from the reviewers either not understanding the applicable administrative
Il IE
procedure or using criteria in the review checklist which did not apply at the time the test was conducted.
In view of the documents examined, interviews conducted, and the re-reviews conducted by the licensee (as discussed in paragraph 3.c above),
the inspectors saw no basis for concluding that the documentation of the prerequisite test program was inadequate.
- 1 <<
an entry for the start and for the stop of the test, but there was no written explanation for the actual test.
A roach to Resolution:
The inspectors examined,,selected test log sheets against the requirements of governing administrative procedures.
Ins ection Zindin s:
Most prerequisite tests were conducted using generic test procedures; i.e.,
a procedure specifying test methods and data recording for a number of similar components.
Bechtel procedure AD-108 did not require a test log, but did provide a
comments section with each data sheet.
The governing APS procedure, 90GA-OZZ28, provided for use of a test log, but generally required log entries only for abnormal or unacceptable conditions.
Commencement and completion of the test were prescribed log entries, but descriptive entries were not required so long as the test was done acceptably in accordance with the generic procedure.
One IGC test engineer interviewed stated that if no problems were encountered, the test log would normally include only entries for starting and completing the test.
The test logs examined by the inspectors appeared to be consistent with procedural requirements.
should have been used and it also, in some situations, would not have provided the test results which were recorded on the documentation.
A roach to Resolution:
The inspectors examined selected prerequisite test documents, including a sampling of those reviewed by the (}C reviewers, to determine whether improper test equipment was used.
Ins ection Pindin s:
Examination of the seven document packages discussed in paragraph 4.b and several other prerequisite test packages did not identify any instances wherein improper test equipment appeared to have been used.
Test equipment calibration due dates were recorded on the test documents as required.
It should also be noted that administrative procedures allowed equivalent test equipment to be substituted, with the approval of a Level II test engineer, for the specific equipment called for in the procedure.
found" condition and in the "as-left" conditio II,
N
A roach to Resolution:
The inspectors examined a sampling of completed prerequisite test documents and held discussions with licensee representatives.
Ins ection Findin s:
This allegation related to an issue referred to by the alleger as
"dead nuts."
This term was used to mean that an instrument calibration agreed exactly with the standard.
The alleger's inference was that this was very unlikely and indicated that the individual performing the testing may have filled in the desired data without actually performing the calibration.
The "dead nuts" issue was raised in several of the interviews conducted (discussed in paragraph 3) and was not corroborated.
One IGC test engineer interviewed was not familiar with the term; however, he stated that some instruments could be calibrated to.read exactly as specified, and that directions from his supervisor were to set the instruments as close to the specified standard as possible.
Examination of 16 test documents reviewed by the two QC reviewers identified one case (1JAFZZF041B/K018) in which the data sheet indicated the instrument to be calibrated in exact agreement with the standard.
In three other cases (1JAFZZF040A/K015, 1JAFZZF040B/K016, and 1JAFZZF041A/K017), the QC reviewer comment sheet indicated that the calibration was "dead nuts" when in fact it was not (one, two, and four data points, respectively, were not in exact agreement with the desired setting).
The records examined and interviews conducted by the inspectors gave
.no'indication that prerequisite tests involving instrument calibrations were not being handled responsibly.
different individuals, all of whom apparently were not aware that
,. the subsystem had already been tested.
-A roach to Resolution:
The inspectors questioned these interviewed about this concern and examined a sampling of test documents to determine the significance of this issue.
Ins ection Findin s:
Persons interviewed by the inspectors acknowledged that, repeat testing had been a problem, although related more,to efficiency or productivity than any other concern.
They stated that some components were tested more than once when a
generic test procedure was used.
If a specific test procedure (a
~ procedure written to accomplish only one specific test)
was used, only one Official Test Copy (OTC) of the procedure would be prepared, and it would be logged out by document control.
Several OTCs of each generic procedure were prepared, however, since each generic procedure was used to test a number of similar components, and more than one copy could be in use concurrently.
Although the OTC was signed out and in by a test engineer, the test data sheet (which became the official test document)
was not signed out by DTS number, and the component would still be shown on the DTS listing as
I'
I'P C
t
,i I )
not having been tested until the test document was completed and approved by the test group supervisor.
The licensee stated that this resulted in some tests being performed more than once due to poor communications among test personnel, a possibility which was increased by,multi-shift testing 'activities.
The number of repeat tests conducted was considered to be a small percentage of the total, however.
Licensee management stated that when a component was tested more than once, the last test performed was considered to be the official record of the test, and was reviewed as such.
The licensee also stated that methods had been changed to minimize duplicate testing on Unit 2: (1) test data sheets were being issued by DTS number, and (2) each plant system will have a principal startup engineer assigned to control testing on the system (a method not used for Unit 1 after prerequisite test responsibility was given to Bechtel).
Examination of test record files by the inspectors did not indicate duplicate testing to have been a frequent occurrence.
The inspectors could see no basis for a safety concern related to this issue, although as noted earlier, it may have affected the licensee's productivity.
booking" was being done on the test documentation; i.e., the data was not being obtained from actual tests.
A roach to Resolution:
This matter was explored with several members of the licensee's organization in interviews conducted during the inspection (discussed in paragraph 3).
Ins ection Findin s:
Persons interviewed did not corroborate the alleger's concern.
They consistently felt that test personnel were doing a conscientious job of testing.
The inspectors could identify no basis from the interviews conducted or records reviewed for substantiating the alleger's concern.
~1 wherein each individual within the Test Review Group is required to pass approximately 500 documents a week, is unacceptable.
A roach to Resolution:
This issue was addressed in interviews with those involved in performing and reviewing prerequisite tests.
Ins ection Findin s:
The Supervisor of the TRG stated that no guidance or goals were given to him on how many test documents were to be reviewed by the TRG in a given period, although he gave his supervisor a weekly status on test documents received, reviewed, and pending.
One TRG member interviewed stated that no pressure was exerted to review any specified number of tests, that quality was stressed over quantity.
He also stated that no goals or quotas had been established.
Another TRG member agreed, stating that a test document had to be done to his satisfaction or he would not sign it off.
When interviewed, the alleger's associate (QC reviewer) stated
g I J
that he was unaware of a "bean count" applied to either QC reviews or TRG reviews.
The Startup QA/QC Document Review Group Supervisor and two QC Document Reviewers, when interviewed separately, stated that no pressure had ever been felt to maintain any certain flow of documents through the review process.
Three test engineers also stated in separate interviews that pressure for production was not present in the conduct of prerequisite testing.
The inspectors concluded from the interviews that no "bean count",
review quotas, or goals existed.
No indication was received that.
pressure was ever applied to sacrifice quality of reviews for quantity.
returned to the Test Review Group were in the alleger's office at the time of his termination.
He does not believe that they were ever sent to the TRG.
A roach to Resolution:
The inspectors obtained and examined the alleger's comment sheets for the tests and discussed their disposition with cognizant licensee personnel.
Ins ection Findin s:
The alleger and his associate reviewed 117 prerequisite test documents during their tenure as QC reviewers.
As discussed earlier in paragraph 4.b, the reviewers found none of the 117 tests to be acceptable.
The licensee had retained the 117 test review checklists for disposition.
The Startup QA/QC Document Review Group Supervisor had been assigned the responsibility of reviewing and resolving the comments, and this was in progress during the inspection.
The pertinent test documents had been retained in the prerequisite test document files.
valid test program, the QA Manager did not want him at the site.
A roach to Resolution:
The inspectors interviewed the Startup QA Supervisor and examined relevant test documents and review checklists.
Issues of a personal or personality nature between the alleger, his supervisor, and others were avoided.
Ins ection Findin s:
In an interview, the Startup QC Supervisor stated that from the beginning he was responsive to the alleger's concerns and went to lengths to satisfy him.
The supervisor stated further that he had given the alleger a lot of help, but that he was unable to complete any QC reviews.
The alleger also expanded the scope of his assignment beyond the document review function management intended.
Examination. of test documents indicated that the alleger sought to expand his task beyond the QC document review function to which he was assigned.
The. alleger confirmed this during his interview with the inspectors, indicating that he had extended his review to include verification of test activities in the plant.
As discussed in paragraph 4.b, a substantial portion of the reviewers'omments
P
on completed test documents were also incorrect.
For example, the test document for electrical scheme lEGA01NC1/E064 (a control power cable continuity verification) was rejected with 13 comments, none of which was valid.
Scheme 1EGA01AC2/E054 was similarly rejected with 12 comments, all incorrect.
It was the inspector's conclusion that the alleger was treated fairly'y licensee management, but that he was released because he was not accomplishing the task to which he was assigned; i.e.,
gC review of completed prerequisite tests.
Interviews indicated that factors beyond the scope of NRC concern may also have been involved.
ineffective audit.
More specifically, the alleger's sworn statement included the following on page 7:
"I was aware that NRC inspectors conducted an audit of some elements of the Start-Up Program in the latter part of May 1983 or early June 1983.
I believe that the audit involved employee performance or training.
I was not directly involved with the audit, however, I heard comments from other individuals concerning the audit.
The comments reflected that the NRC inspectors were too "casual" in their relationship with their APS counterpart, and that the NRC inspector appeared to be led around by the APS repr'esentative.
There were no identified deficiencies in that audit, whereas most employees believe there are deficiencies if the inspector looked more closely.
I have no reasons to believe that there was collusion or wrongdoing on the part of the NRC inspector, but I simply consider such an audit to be ineffective."
A roach to Resolution:
The inspectors examined Region V records of inspections conducted at Palo Verde and discussed this issue during interviews, Ins ection Findin s:
Examination of the record of inspections conducted at Palo,Verde, as reflected in Region V Inspection Report docket files, indicated that the only inspections of startup test program activities conducted during the period in question were by the resident inspectors.
This was discussed with the alleger during a September 19, 1983 interview, in which he described the inspector as "small built, young,'iddle twenties."
This description did not match any of*the resident inspectors assigned to the Palo Verde site at that time.
The alleger also stated that he had only heard this, but had neither seen nor talked to the inspector.
In addition, the allegersaid that the deficiencies identified by the audit were so obvious the inspector wouldn't write them up.
J
In a separate interview, the alleger's associate (gC reviewer)
stated that he had no feelings 'at Palo Verde that NRC dealings with the licensee were too casual.
A test engineer stated during an interview that the Senior Resident Inspector observed flushing of the fuel pool cooling system and testing of the containment spray rings, and that the inspector had been businesslik )
I'f
The inspector concluded that this allegation was not substantiated, particularly because the only inspections of startup test activities during the time period of concern were done by the resident inspectors, who did not match the description provided by the alleger,.
5.
Other Concerns During the September 19, 1983 interview, the alleger stated that he had recently heard about other actions which had taken place since his release.
Concerns were expressed as follows:
a.
Concern:
A QC reviewer reviewed three prerequisite tests within five minutes.
Ins ection Findin s:
Many of the prerequisite test documents are very brief; i.e.,
one page.
Some are simple verifications such as proper orientation of the flow arrow on a check valve body, calibration of one gauge, or continuity check of one conductor.
Instructions provided for the QC reviews indicated that they were intended to verify completeness, legibility, and proper administrative form of the test documents.
Technical adequacy is not a QC function, this being accomplished by the Test Group Supervisor and the TRG. It appeared to the inspector that an experienced QC reviewer could properly review three prerequisite test documents in five minutes if they were simple tests and involved no discrepancies.
b.
Concern:
TRG now makes any changes or signatures needed for administrative correction of the test documents.
Ins ection Findin
In separate interviews, the TRG supervisor and two TRG members stated that test personnel were asked to correct minor administrative discrepancies (initials, dates, explanation for
"NAs", etc),
and that cooperation with test groups in this regard had been good.
They stated that technical discrepancies were few, but that when these were detected the test was returned to the test group with a memo requesting a retest.
Examination of TRG records and the review process corroborated the statements of the TRG members.
c ~
Concern:
Subsequent QC personnel reviewed and signed off all prerequisite test documents in a period of approximately two weeks.
Ins ection Findin
Examination of QC review records and interviews with QC personnel did not substantiate this concern.
The QC review effort was resumed on July 14, 1983 (following the alleger's termination)
and was still in progress during the various site visits which comprised this inspection.
Review records and discussions with the Startup QC Document Review Group Supervisor indicated that four QC reviewers reviewed an average of approximately 300 test documents per'ay during the month of August.
This averages to 75 documents per day per reviewer, or less than ten per hour.
Considering the brevity of many of the documents,
~ 3 i
this did not appear to the inspectors to question the quality of the reviews.
d.
Concern:
The Startup QC Supervisor took all the QC stamps and identification records and threw them away.
Ins ection Findin
In an interview with the inspectors, the Startup QC Supervisor stated that APS QC inspectors had never used QC stamps.
Bechtel had used these for their QC inspectors in the past when Bechtel had its own QC group (November 1981 to February 1982).
However, these were not used after February 1982, when Startup QA/QC was returned to the APS organization.
6.
Conclusions Followup on the allegations discussed herein did not identify any technical concerns.
The inspectors also concluded that the prerequisite test program was being managed effectively by the licensee.
7.
Exit Interview The inspectors met with licensee representatives (denoted in paragraph 1)
on November 4, 1983 at the Palo Verde site.
The inspection findings identified in paragraphs 2 through 6 were discusse p,'5