ML20107G817
| ML20107G817 | |
| Person / Time | |
|---|---|
| Site: | Crane  |
| Issue date: | 02/25/1985 |
| From: | Jordan W, Weis E, Weiss E HARMON & WEISS, UNION OF CONCERNED SCIENTISTS |
| To: | |
| Shared Package | |
| ML20107G787 | List: |
| References | |
| SP, NUDOCS 8502260558 | |
| Download: ML20107G817 (163) | |
Text
,
e'
- e O_
DOCKETER UShAC February 25, 1985 W FEB 25 PS:38 UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION GEC 0F SE~
TAk
M
. K:E BEFORE THE ATOMIC SAFETY AND LICENSING BOARD
~~~.4,
- s:: % *, :g.
- I.. ?
In the Ma tter of
)
,jfs, 5 '
METROPOLITAN EDISON COMPANY
)
Do c k e t No. N0'-$8 9 S $ ~ ' # '
)
(Restart Remand on
)
Ma nagement - Training)
(Three Mile Island Nuclear
)
Station, Unit No. 1)
)
)
UNION OF CONCERNED SCIENTISTS PROPOSED FINDINGS OF FACT AND CCNCLUSIONS OF LAW ON THE ISSUE OF LICENSED OPERATOR TRAINING AT TMI-l
()
Ellyn R. Weiss William S. Jordan, III General Counsel Union of Concerned Scientists 2001 S Street N.W.
Suite 430 Wa shington, D.C.
20009 (202) 328-3500 lll22ID$$! $ ob g9 0
Wb
. _ ~. -
i TABLE OF CONTENTS 4
I.
Introduction and Background 1
l~
A.
The Remand on Training 1
t l
B.
Participation in the Proceeding 6
C.
The Scope of the Proceeding 9
II. Proposed Findings of Fact
. 13 A.
The Reconstituted OARP Review Committee's 4
Assessment of.the TMI-l Licensed Operator Training Program.
. 13 J
l 1.
The Ma keup, Purpose, and Responsibil-j ities of the Reconstituted OARP Committee. 13 2
2.
The Committee's Me thodology.
18-S.
Summary
. 18 b.
The Me thodology - that Must Be Used to I-Review-a Training Program and Reach L-Supportable Conclusions 20 4
1.
The Standard for Determining an Appropriate Methodology.
20 i
11.
The Methodology Necessary to 1
Evaluate the TMI-l Licensed
+
- Opera tor-Training ; Program.
24:
4.
-(a) - Dr.' Regan! s Proposed
~
j Methodology........
...25 (b)~ ~The Staff's Proposed t
.Methodologyf.......'.~.. 32 111. Conclusions :as ito an' Approp-zriate Me thodology.
.E.
43
.4 a
c.
The Methodology Employed by the' -
Committee in-Preparing,the Special' c...
1.
.-'45 Report.
.c.......
~i.
-The Committee Could Not Have-.
1 Implemented.an adequate-Method-ology in the' Time Available~... J4 5'
()
~
t hi Il
ii.
The Committee's Me thodology in Preparing the Special Report was Inadequate 54 d.
The Me thodology Used by the Committee After the Special Report 66 i.
Impact of Cheating 67 11.
Effectiveness of Training 70 lii. Propriety of Personnel Decisions. 84 iv.
Adequacy of Communications 85 3.
Conclusions With Respect to the OARP Committee 86 B.
The Licensed Operator Training Program at TMI-l 89 1..
Organization 89 2.
Impact of Cheating 98 3.-
The Adequacy of the Licensed Operator Training Program 100 O
1.
The Standard 'for Decision...;.
100 ii.
Licensee Examinations 106
-lii. L Program Development and Implemen-tation 123 iv.
Trainee:and Operator Selection and
~ Evaluation 153 1473 v.
Employee At titudes...
III.
Conclusions of Law
. ~..
153
-L t
~
. l'
~),
.,m,,
'Y
February 25, 1985 UNITED STATES OF AMERICA NUCLEAR REGULATORY CCMMISSION BEFORE THE ATOMIC SAFETY AND LICENSING BOARD In the ba tter of
)
)
METROPOLITAN EDISON COMPANY
)
Docket No. 50-289 SP
)
(Restart Remand on
)
Management - Training)
(Three Mile Island. Nuclear
)
Station, Unit No. 1)
)
)
UNION OF CONCERNED SCIENTISTS PROPOSED.
FINDINGS OF FACT AND CONCLUSIONS OF LAW ON-THE ISSUE OF LICENSED OPERATOR TRAINING AT TMI-l I.
Introduction and Background A.
The Remand on Training 1.
In Ma y 1984, the Appeal Board remanded this proceeding
()
to'the Licensing Board for further hearings on three discrete issues related to the competence and~ integrity or character of licensee.
Me tropolitan Ed ison Co. (Three Mile Island Nuclear Station, Unit 1), ALAB-772, 19 N. R.C. 1193 (1984).
.This partial initial decision addresses the remanded issue of whether the operator training program at Three Mile Island, Unit 11("TMI-1")'
is. adequate to ensure that the plant will be safely operated' 2.
In -1979, the Commission ordered TMI-l to remain! shut.
down pending a hearing on a~ number lof issues,. including the
~
'v.
. i management capability and technical resources of the Licensee.1 The Commission also directed that two relevant "short term" actions be taken as necessary but not sufficient conditions of restart of TMI-1.
First, all TMI-l operators were to be retrained and re-examined.
Second, the licensee was ordered to demonstrate the adequacy of its managerial capability and resources, including "the technical capability and training of operations staff.
CLI-79-8, 10 N.R.C. 141, 143-146, 149 (1979); see also CLI-80-5, 11 N.R.C. 408 (1980).
After an extensive hearing on management issues, including the substantive adequacy of the TMI-l licensed operator training program, the Licensing Board issued a decision in August 1981 generally favorable to Licensee.
LB P-81-3 2, 14 N.R.C. 381 (1981).
Be ca use of the contemporaneous discovery of cheating on NRC licensed l ()
operator examinations, however, the Board retained jurisdiction of the case to consider the impact of this new information on its findings and conclusions on Licensee's' management competence.
Id. a t 403 (1 45).
The Board subsequently reopened-the' management proceeding and -appointed a Special Master to hear-evidence on the impact of the cheating incidents at TMI-1..
t During the course of the further hearings, much evidence was I
taken-demonstrating that the p'roblems with GPU's training program l
In ?l981,..the operating. license for TMI-1 was transferred-s, J'
. f rom Metropolitan Edison Company-to.the. newly' formed ~ General.
Public Utilities subsidiary, GPU. Nuclear. Corporation. ICLI-81-17, 14 N. R.C. - 2 99_ - (19 81).
At the same time, the Commission instructed the Licensing Board to consider the management-competence of GPU : Nuclear, rather than : that of Metropolitan i
(]'
Edison; the' original'TMI-l? licensee.
I_d.
- V
's 6
4 C
=
_ _ _ _ _ _ _ _ _ _ went well beyond instances of cheating.
The Special raster concluded on the basis of the evidence that the training program was poorly administered, weak in content and ineffective in its method of instruction.
LB P-9 2-3 48, 15 N.R.C.
918, 1020 (1 251).
He concluded that the GPU training program did not meet the requirements of the Commission's 1979 Order.
Id.
The Licensing Board decision found that there had been a breakdown in the integrity of Licensee's training and testing program at TMI-1.
LBP-82-56, 16 N.R.C. 281, 300 (1 2082).
The Board imposed several requirements, primarily future audits, directed at obtaining future assurance of the adequacy of the training program.
Id. a t 365, 384 (11 2347, 2420).
The Board also concluded, however, that the identified weaknesses in the program did not undermine the Board's carlier decision favoring restart.
O d..
t 3 01 < < 2 0 8 9).
3.
In ALAB-772, the Appeal Board reviewed the entire record in the TMI-l restart proceeding on the ability of GPU Nuclear Corporation's management to operate TMI-l safely.
ALAB-772, supra, 19 N.R.C. at 1201.- In response to issues argued on appeal primarily by UCS and TMIA, the Appeal Board endorsed the Licensing Board's characterization of the question that had to be answered - following the cheating incidents at TMI
-1, viz., "is-the instruction adequate to prepare the -operators to operate the plant-safely?"
It disagreed, however with the Board, "on its affirmative answer to that question."
ALAB-772,c supra, 19 N. R.C.
at 1232-33.
The Appeal Board believed that the record in em
_ _ the reopened proceeding had perhaps raised more questions than it
()
satisfactorily had answered.
"For example, does the training program actually enhance the operators' knowledge or simply encourage memoriza tion for test-taking purposes?
Are the licensee and N.R.C. examinations an ef fective way to measure an operator's ability to run the plant?
Do the format and content of the examinations encourage cheating?"
Id. at 1233.
l 4.
The Appeal Board ruled that the " principal difficulty" of the Licensing Board's decision was its " failure to reconsider, as promised and in a meaningful way, its earlier finding that licensee's training program was ' comprehensive and acceptable,'"
Id,.
Instead, the Board " relied on the post-cheating testimony of only licensee and the Staff."
Id.
Of particular additional significance was the large degree of reliance by the Licensing
()
Board on the testimony of a panel of GPU consultants known as the l
OARP Review Committee who reviewed licensee's accelerated operator retraining program in 1980-81 before the cheating became known.
As the Appeal Board stated:
"[T]he Board essentially presumed that the earlier, favorable expert testimony by the outside consultants would not have been altered by the cheating
~
revelations."
Id.-
5.
In addition to the questions already discussed,1the Appeal Board raised many other specific issues and concerns throughout its decision.
Among these are:~-
the presence in supervisory positions 1of several' a.
individuals implicated in the cheating episodes.
ALAB-772, supra at-1233.
p
,V
~.
b.
the failure of employees, including training instructors, to take the courses or examination process seriously.
Id,.
c.
whether the positive assessments of Mr. Kelly and Dr.
Christensen regarding the " pride and enthusiasm" of employees in their training program and the professionalism of the instructors would have been altered by post-cheating testimony reflecting "a lack of these qualities."
Id,. a t
~
1234.
d.
whether the OARP Committee's brief but favorable comments on the written exams would be revised by the new evidence.
Id. a t 123 5.
e.
whether the Committee's favorable parceptions of the instructors would be changed by knowing that one or more of those they evaluated were involved in the cheating episodes.
Ld -
f.
whether the criteria for training instructors are adequate.
Id.
g.
whether top management is awa re of the - real-and perceived problems of its employees.
Id. a t 123 6.
h.
the appropriateness of the promotion and current assignments of several persons within the GPU organization, specifically including Dr. Robert Long, director. of Training
()
and Education during the cheating incidents, promoted to GPUN Vice President. for Nuclear Assurance; Dr. Richard P.
Coe, who has replaced Dr. Long; Sanuel. Newton, former Operator Training Manager, promoted to Manager of Plant Training; Edwa rd J..Fre6erick, a TMI-2 control room operator'at the time of the accident, promoted to supervisor of Licensed operator Training.
Id. a t n.
56, p. 1236 6.
The Appeal Board summarized its holding as follows:
The most.significant issue requiring further hearing.is l
. training.
Because the safe operation of 'the plant 1,s so I
-heavily dependent upon'the operators' skill, the importance of training cannot be overstated. sThe cheating and related l.
incidents called into. question.the adequacy and integrity of licensee's entire training and testing program.. Although we have-found'that the reopened record on the cheating itself was as fullyLdeveloped as possible,.the; impact of those ifindings on the Licensing Board's' earlier conclusionsfon' licensee's: training program was.not-given the full consideration it warrants..
In particular,-the Board should have sought further testimony, in light of the cheating incidents,. from the OARP Review Committee, whose, views the:
L Board previously'found so persuasive.
f.
ALAB-772, 19 N. R.C. a t 1279.
m-o.
l
' i 1
B.
Participation in the Proceeding 7.
On June 28, 1984, the Licensing Board presided over a prehearing conference among the parties for the purpose of defining the issues and providing for prehearing procedures in the proceeding remanded by the Appeal Board in ALAB-772.
At that conference, in addition to Licensea and the NRC Staff, the i
Commonwealth of Pennsylvania ("the Commonwealth"), Three Mile Island Alert ("TMIA") and the Union of Concerned Scier.tists
("UCS") were parties to the restart proceeding that expressed an interest in participating in the remand on training.
Tr. 27,281 (Commonwe alth) ; 27,280 (TMIA); 27,280-81 (UCS).
A schedule was set for discovery and the commencement of the evidentiary hearings.
See Memorandum and Order Following Prehearing Conference, July 9, 1984.
The evidentiary hearing began on O
De cembe r 19, 1984 and continued intermittently for 11 days, closing on January 18,.1985.
8.
The Board encouraged'and the intervenors agreed ~to l
utilize lead intervenors in the remanded proceeding.
Tr. 27,294
. (Chairman Smith, Jordan) ; Memorandum 'and Order on Lead-Intervenors,' July-13, 1984.
This arrangement was made with the
' understanding that no -intervenor : waived its right. to pursue its
~
l separate interests where the lead intervenorldid.not-fully
. represent the others.
However,'intervenors were required to make-good faith effo'rts to consolidate their case with the.
. presentation of.the lead intervenor before L proceeding
. independently.- If these. efforts failed, the intervenor could:
(~/ -
'\\
"sm
=
i..
then seek leave of the Board to proceed separately.
See Board Memorandum & Order on Lead Intervenors, July 13, 1984; see generally 10 C.F.R.
S 2.714(e) and Part 2, Ap p. A, S III(a) (4).
9.
The intervenors specified the issues for which they proposed to take primary responsibility.
UCS identified two issues, which were modified and approved by the Board.
These issues related to the substantive adequacy of training to prepare operators to operate TMI-l sa fely.
Memorandum and Order on Lead Intervenors, July 13, 1984 at 3; Memorandum and Order on Licensee's July 31, 1984 Comments on Lead Intervenors and Motion to Partially Exclude UCS From Management Phase, August 30, 1984 at 3-4; see also Tr. 315736, 31,757 (Chairman Smith).
UCS' proposed subissues were:
()=
(1). Are.the operators equipped to safely operate the.
plant particularly in emergency situations?-
.(2)
Do the-NRC and company examinations reliably measure the operators' ability to safely. operate the plant?
Memorandum and Order on Lead Intervenors, July 13, 1984'at'2.
L After further comments were' submitted by the' parties.on the-scope of the. proceeding,.UCS;subissue (1) wa s modifie'd una
~
follows:
(1)
Are the operators trained to safely operatef the-
. plant in accordance with approved procedures,-
particularly in emergencies?
Memorandum and. Order on Licensee's-JulyJ31,1198'4 Comments on Lead
~
-Intervenors-and Motion to ' Partially Exclude' UCS f rom q
~
Management Phase, August 30, 1984, at 3.
10.
The Board ruled that the adequacy of the NRC examinations could not be litigated by the Intervenors in this reopened proceeding unless the OARP Review Committee relied upon these exams as a measure of operator competence.
On the other hand, neither may GPU or the Staf f claim any credit for the NRC exam process; the issue of operator competence must stand or f all i
solely on the basis of GPU's own training and testing process.
The OARP Committee's Special Report did indicate a degree of reliance by them on the NRC exams as providing assurance of I
operator competence.
S.R. a t 46.
However, after being informed of the Board's ruling the Committee communicated to the Board through GPU counsel asserting that it has not relied on the NRC exams.
Letter from GPU counsel to the Board, September 27, _
.( ) '
1984.
During cross-examination, this question again became clouded.
Kelly,-31,966, 32,085.
Although the Board has not -
permitted any inquiry into the adequacy of the NRC-exams, the Committee's equivocation on this question raises doubts about its credibility.
11; The subissues identified by :TMIA were,the following:'
(1)
Has GPU properly responded to' the problems in its training program identified / internally.and/or~by the-Special Master,. the Licensing Board and the ~ Appeal Board?
(2)- Are the people responsible [for the management -and implementation of the training program properly-equipped-
~ by their own experience and ' attitude to impart 'the information and values necessary for safe operation of L TMI-l?
-(3)
Do 'the operators have. the appropriate-attitude towa rd the - training ' program; do.they believe it is;ef fective?-
)
s
'wl u
()
(4)
How does the history of GPU's problems with training and its current training program reflect on the competence and integrity of GPU management?
Because the wording of TMIA's proposal suggested to the Board that TMIA'might pursue matters that wete res judicata, the Board-simply approved TMIA's lead on the training issues to the extent that ALAB-772 authorized an inquiry into cheating and integrity as they relate to training.
Memorandum Order on Lead Intervenors, supra, at 3; see also Tr. 31,757-58, 31,784-85 (Chairman Smith).
C.
The Scope of the Proceeding-12.
In ALAB-772, the Appeal Board raises numerous questions about Licensee's training program, most of which are delineated above.
Because of the importance of the issue of training, see (k
ALAB-772, supra, 19 N. R.C. a t 1279,-and our independent responsibility to ensure that the record in this proceeding is complete,2 : the Board was reluctant to interpret narrowly the' Appeal Board's directive remanding _the issue of training.
Thel right of other parties'to confront'the views.of.the OARP Committee to some degree dictated the scope of the hearing.
SeeI
- In ' ALAB-772, : the - Appeal Board made clear that the f remand was precipitated by "the Licensing Board's failure ~to reconsider, as
~<
promised,and in a meaningful way,,its. earlier finding: that licensee's training program was ' comprehensive and acceptable. '"'
~ supra,.19 N. R.C. a t 1233.
In this _ remanded proceeding,-
ALAB-772, the Board therefore is~ charged with a particular responsibility conferred on it by.the Commission Ein gits ' original ~ institution of '
this proceeding.
See CLI-7 9-8, 10 : N. R.C. ~141', f147-49_ (1979).
p Ju) '
- l^
~
~
Memorandum & Order Following Prehearing Conference, July 9, 1984 at 3.
However, ALAB-772 specified several limitations on the scope of this proceeding and, by applying those limitation, the Board essentially provided a framework within which the evidentiary proceeding ensued.
13.
The broad issue on remand was the adequacy of the training program to prepare the TMI-l licensed operators to operate the plant safely.
See Board Memorandum & Order Following Prehearing Conference, July 9, 1984 at 2-3; see, e.g.,
Tr.
32,270-74 (Chairman Smith).
However, this broad issue was confined by the Appeal Board in Section III.
C. o f ALAB-772 to the implications of cheating and other deficiencies that came to light in the reopened proceeding on cheating.
In addition, management findings (including findings on the TMI-l training
( )8 program) that were not placed -in issue by the Appeal Board were ruled to be res judicata in the remanded proceeding.
Id,. at 3.
For example, the remand did not permit the relitigation of the cheating incidents themselves.
Id. a t 3, 6.
In addition, the Board ruled that only licensed operator training would be considered.
14.
ALAB-772 remanded the issue of training in order.for the Board to. assess the implications of, the cheating incidents on the adequacy of the operator' training program currently in existence a t TMI-1.
ALAB-772, supra, at 1232-37.
However, parties had the
~
'right to pursue a particular past problem insofar as that problem could shed some light on the adequacy of the current program.
Se e ALAB-774, 19 N.R.C.
1350, 1356 (1984).
fm L.)
8
~
f
. 15.
Finally, as discussed above, the Board permitted litigation of the NRC license examination only insofar as this process formed a basis for the OARP Review Committee's opinion of Licensee's program.
16.
The NRC Staff considered the remand to be limited strictly to the views of the OARP Review committee about licensed operator training at TMI-1.
The Staf f testimony therefore I
addressed the issue of whether the "me thodology" used by the OARP Committee was sufficient to support its conclusions.
See Testimony of Julius J.
Persensky, Joseph J. Buzy and Dolores E.
4
(
Morisseau on the Remanded Training ' Issue f rom ALAB-772
(" Staff"),
f f. Tr. 3 3,14 8, at 2.
UCS presented an expert witness, Dr. James J.
Regan, who offered his recommended methodology for ensuring the adequacy of training at TMI-1.
Testimony of Dr. James J.
.()
Re gan ( " Reg an" ), ff. Tr. 33,532; see also Surrebuttal Testimony of Dr. James J.
Re g a n, ff. Tr. 32,693.
The Licensee presented the panel of five experts who made up the Reconstituted.OARP Review Committee.
See Testimony of the Reconstituted CARP committee (Dr. Julien Christensen, Dr. Er ic Ga rdne r, bk. Frank i
Kelly, Dr.' William Kimel and Dr. Robert Uhrig) on.the TMI-l Licensed Operator Training Program
(" Committee"), ff.
3
. In Ma y 198 4, the OARP' Review committee was reconstituted.
The membership remained the same with the exception of Mr.
Ma rzec, who was unavailable.
He wa s replaced with Mr. Frank Kelly, who had previously testified in this proceeding.
See 1 224, infra; see LBP-81-32, supra, 14 N.R.C. at 460-61 (11
^~
226-29)..
b
~-
t Tr. 31,749; Rebuttal testimony of the Reconstituted OARP Committee, f f. Tr. 3 3,3 2 0.
Licensee also presented three panels of company witnesses who described the TMI-l licensed operator training program.
This testimony addressed questions contained i
in Section III.C of ALAB-772 about post-cheating management actions related to training.
See Licensee's Testimony of Dr.
Robert L.
Long and Dr. Richard P.
Coe on the Issue of Licensed Operator Training at TMI-1 ("Long & Coe"), f f. Tr. 3 2,202; Licensee's Testimony of Mr. Samuel L.
Newton, Mr. Bruce P.
Leonard and Mr. Michael J.
Ross on the Issue of Licensed Operator Tr a ining a t TMI-1, f f. Tr. 3 2, 4 0 9 ; Rebuttal Testimony of Dr.
Ronald A.
Knief and Mr. Bruce P.
Leona rd ("Knief & Leonard"), ff.
Tr. 33,364.
UCS and TMIA further challenged the substantive adequacy of the licensed operator training program, throug h
()
cross-examination of Licensee's witnesses and through the introduction of exhibits of fered for -the purpose of establishing; l
inadequacies in the-program.
See UCS Training Exhs.1-34;1TMIA l
Training Exhs. 1-11.
I
~
l l
j')h-u
l l,
II.
Proposed Findings of Fact
()
A.
The Reconstituted OARP Review Committee's Assessment of the TMI-l Licensed Operator Training Program.
1.
The om keup, Purpose, and Responsibilities of the Reconstituted OARP Committee.
17.
The Reconstituted OARP committee testified on behalf of licensee about its review of the current TMI-1 licensed operator training program.
Dr. Robert E.
Uhrig is Chairman of the Committee and Vice President, Advanced Systems and Technology for Florida Power and Light Company, Miami, Florida, with 28 years of utility and engineering education experience.
Dr. Julien M.
Christensen is Chief Scientists Human Factors and Logistics, for Universal Energy Systems, Dayton, Ohio.
Dr. Eric F. Gardner is Professor Emeritus of Psychology and Education, Syracuse University, Syracuse, Ne w Yo r k.
Dr. William R. Kimel is Dean of
(}
the college of Engineering at the University of Missouri, Columbia, Missouri, and inter alia, has served as President (1978-9) of the American Nuclear Socie ty.
See_ Committee, ff. Tr.
31,749, resumes.
Dr. Kimel is af filiated with INPO and is currently serving as an alternate member of the INPO Accrediting Board.
Kimel, Tr. _ 3 2,0 4 6.
Mr. Frank Kelly is _ President of the PQS Corporation and former Chief of the operator Licensing Branch of the AEC.
18.
The members of-the OARP Committee were under contract ~
- with ' licensee for this testimony.. With the exception of Mr.
Kelly, the members of the OARP Committee had been under contract y
.L)
i with GPU for the original OARP report.
Committee, Tr. 31,964-6.
Several of the Committee members had other contracts with GPU.
4 Mr. Kelly, who receives 100% of his income from the nuclear industry, performed a review of the training and requalification l
program for the TMI-2 operators in April and May of 1979.
Tr.
31,964.
He also, in December 1979, reviewed the OARP program records, and his company had a contract with GPU to administer the so-called mock N.R.C. examinations for licensed RO's and SRO's at TMI-l in 1980.
He appeared as a witness for licensee in the earlier phase of this restart proceeding.
In April 1984, he gave a two day seminar on oral examination administration at Oyster Creek.
Id.
Dr. Gardner was employed by GPU to give a seminar on test construction about two years ago.
Tr. 31,965.
Dr. Christensen served on a contract to redesign the control room.
Tr. 31,966.
Dr. Uhrig, while he did not have any other
- s contracts with the Licnesee, has been a member of the Atomic Industrial Forum Committee on Three Mile Island Two Recovery
("the AIF Committee") since its formation in 1980.
Before and during the time that Dr. Uhrig was a member of the Reconstituted OARP Committee, the AIF committee sent three letters to' Chairman Palladino urging that the Commission allow restart of'TMI-1.
Stipulation concerning Dr. Uhrig 1985.
The Board recognizes that, as individuals, the members of the Reconstituted OARP Committee may have certain professional.and contractual obligations to licensee and the nuclear industry as a whole and
~
/~'N U
strong opinions about TMI-1.
In the Board's opinion, this underscores the need to seek objective information in the Committee's testimony and to be skeptical of subjective statements.
Certainly Dr. Uhrig's participation on the AIF Committee and his support for the letters to Chairman Palladino are typical of this type of personal activity on an issue such as this.
We must for example, take into account the fact that Dr.
Uhrig has joined in three public calls for the Commission to restart TMI-1.
19.
The Committee was called together in order to respond to the Appeal Board's remand in ALAB-772.
As discussed below in greater detail, the Committee performed two basic functions for Licensee.
First, the Committee developed a Special Report, dated June 12, 1984, but actually issued on June 28, 1984, which
()
Licensee provided directly to the Commission.4 Second, the Committee prepared testimony, including the Special Report, which the Committee presented in this hearing on behalf of'the Licensee.
20.
According to the Committee, it prepared the Special Report "specifically for the- [then] impending Nuclear. Regulatory Commission meeting" at which the Commission had indicated that it would " formally consider the issue of restarting TMI-1."
S.R.
a t 3.
When the Committee. was - first called together, the 4
The Special Report was received into evidence as Attachment 1 to the Committee's direct testimony, which appears _ following Tr. 31,749.
We will cite the Special Report as "S.R."
&(m
)
Commission meeting in question was scheduled for June 27, 1984.
Dr. Uhrig, who chaired the Committee, testified that the Licensee wanted the Committee to prepare "a report that would be available for that meeting."
Tr. 31,790.
In preparing the Special Report, Dr. Uhrig and the other members of the Committee were aware that the Commission might well rely upon the Special Report in the June 27 meeting in deciding whether TMI-l should be allowed to restart.
Tr. 31,791.
Indeed, the very purpose of the report was to persuade the Commission to vote to restart TMI-1.
21.
The Board finds, therefore, that the Conmittee prepared the Special Report 'for the purpose of and with the knowledge that the Commission might well rely upon the report in voting to restart TMI-1.
Thus, the Committee knew that its findings on the
()
adequacy of the training program at TMI-1 could have a direct-effect upon the public health and safety.
It necessarily follows that the Committee was aware of the need to undertake the investigation necessary to assure that the conclusions in the Special Report are accurate and well supported.
2 2. - For these reasons, the Board considers'the adequacy-of the Committee's work in preparing the Special Report to be.an extremely important indicator of the weight to be given the.
Committee's testimony.
Given the time available before the
~
commission meeting, the Committee had two choices.
The' members could.have decided that the time-was not sufficient to permit them tx) make informed ijudgements and declined ' to prepare a report.
Having decided, on the other hand, to prepare a report
.. ON within the available time, the Committee had an obligation to do
()
the work necessary to support the conclusions it reached and urged upon the Commission.
Had it done so, its findings would be particularly persuasive.
If, on the other hand, the Committee failed to insist upon an adequate investigation and was willing, nonetheless, to reach an unqualified conclusion that the TMI-l licensed operator training program adequately trained operators to run the plant safely, this Board could not give significant weight either to the Special Report or to any later testimony.
23.
It is an improper methodology and an unacceptable practice to reach conclusions based upon inadequate evidence and then to attempt to support those conclusions with evidence gathered at a later time.
That is particularly true where the conclusions were not simply initial or tentative opinions, but
()
were sent to the Commission or any other regulatory body for tha
(
purpose of influencing a decision that could have a major 'ef fect on the public health and safety.
It is common sense-that having committed themselves publicly to. specific conclusions the authors of the report are likely, consciously or unconsciously, to seek out the facts that support their predetermined conclusions and to disregard those that are contrary.
Such'a practice severely undermines any claim of objectivity and adversely-af fects the weight that can be given to the conclusions.
It also requires this Board to insist upon the presentation of objective facts, rather than subjective judgements, as the basis -for any conclusions.
l.
L
2.
The Committee's Me thodology v
a.
Summary 24.
The Committee presented factual conclusions and recommendations at three distinct times.
First, the Committee f
issued the Special Report on June 28, 1984.
Second, the Committee filed written direct testimony on November 1, 1984.
i f
Third, the Committee presented live testimony in the hearing.
l beginning on December 19, 1985.
The methodology employed by the Committee in each case is crucial to the Board's evaluation of the Committee's conclusione and recommendations.
25.
As detailed more fully below, the Committee's overall methodology was as follows.
First, the Committee met for i
.approximately six daye, during which time it received and reviewed various documents provided by Licensee management, and
()
it received briefings from various representatives of Licensee's-l management.
Members of the Committee also reviewed documents and-held discussions at other times when~the Committee was not
'actually meeting.. This initial work did not-include,. for.
[
example, interviews with licensed operators or with participants l
l in the licensed operator training ~ program, observation.of any.-
L j.
classes, review' of many fundamental program -documents, nor did u it L
' include many.other actions essential"to a' thorough review.. Ba sed upon.this information, the Committee-reached'the conclusions set-out inithe Special Report, including particularly _the;. conclusion that:"the GPU Nuclear.tra'ining program produces qualified.
f N l.
l, w
operators and is adequate to support the restart of TMI-1."
S.R.
T at 83.
26.
When the Committee issued the Special Report for consideration by the Commission, Licensee had not engaged Dr.
Uhrig or the Committee for any further work on the question of the adequacy of training at TMI-1.
As far as the Committee members were then aware, the Special Report was the final product of their work on the Reconstituted Committee.
27.
In mid-August, Licensee once again called the Reconstituted Committee together, this time to develop testimony for this remanded proceeding.
This was the first time that the Committee members became aware that they would be asked-to provide additional information to the Commission, or that they would have the opportunity to examine the Licensee's training
()
program in greater depth than they had done for the Special Report.
28.
From August until late October, Committee members received further briefings on various issues.
They also interviewed various operators and others involved in training, sat'in on training classes, and pursued other activities.
This work culminated in the. Committee's direct testimony,' which was filed on November 1, 1984. -The direct testimony reiterated and expanded upon the conclusions reached in_'the-Special Report.
29.
Af ter the Committee had filed its-direct ~ testimony, and particularly after-depositions by UCS' revealed weaknesses'in the Committee's work, Committee-members continued to gather.-
r
. Y.))
_.u information and conduct interviews.
This additional work was reflected in part in the Committee's rebuttal testimony, which was filed on November 28, 1984, and in its live testimony.
b.
The Methodology That Must Be Used To Review a Training Program And Reach Supportable Conclusions 1.
The standard for determining an appropriate methodology.
30.
Our reaction to the conclusions in the Special Report, and particularly to the conclusion that "the GPU Nuclear training program produces qualified operators and is adequate to support the restart of TMI-1,"
S.R.
a t 8 3, is the same as Dr. Re g a n ' s.
"This is quite a strong conclusion, particularly because it is not qualified or limited in any way."
Regan, f f. Tr. 3 3, 5 3 2 a t 17.
Although the Committee mentions early in the Special Report that time limitations precluded the opportunity to undertake an i
L ()
in-depth study, S.R. at 3, the Committee in no way emphasized this limitation, nor did the Committee repeat the limitation when f
,it. stated its conclusions.
Thus, the' Committee knew,.or should-
[
have known, that the Commission would read the conclusions in the Special Report as embodying the Committee's considered judgment,-
F based upon an. investigation that was a'dequate to support the.
conclusions as stated in the.Special Report..
I
-31.
With this-in mind, we must first resolve the ' issue of.
what constitutes an adequate investigation' on the basis ;of'which this. Committee could reasonably develop and present unqualified conclusions as to the adequacy 'of the ' licensed operator training program at Three Mile Island: Unit 1.
Our approach.in this regard
~
()
I l
is consistent with that of the NRC Staff and Dr. Re g a n.
The
. ()
Staf f's testimony as a whole suggests, and Dr. Regan explicitly I
testified that the first step in an investigation of the type undertaken by the Committee is to develop a methodology for the investigation.
Staff, ff. Tr. 33,148, Reg a n, ff. Tr. 3 3,532 at 18; f f. Tr. 3 2, 6 9 3 a t 7.
32.
We approach this task f rom two dif ferent, but closely related, points of view.
First, we must judge the credibility of I
the. Committee's work on the basis of the work done by the Committee in order to support the conclusions that it stated in the Special Report and in its testimony.
Second, we must determine whether the investigation undertaken by the Committee was adequate to comply with the Appeal Board's mandate in
~
ALAB-772.
As discussed below, the Committee's investigation was
()
inadequate on both counts.
33.
There is no evidence that the Committee itself attempted to develop an appropriate methodology before it began its investigation and produced the Special Report.$
Ra the r, four of the five members of the Committee met with licensee management 5
The Committee appears to have begun its work by reading and discussing ALAB-772, Uhrig, Tr. 31,793, and determining the.
approach the Committee would use to respond to it.
Gardner, Tr.
31,797.
Beyond the fact that the Committee assigned roles to its members based largely upon their participation in the original OARP Committee, Gardner, Tr. 31,797, the Board is unable to identify.any methodology that the Committee developed as a guide to its work, either on the Special Report or on ' its later testimony.
We'are concerned, therefore,.that the Committee's actions were dictated by the information provided'by Licensee,-
rather than by an independent judgement.as to the appropriate methodology.
~
r
, on short notice at Three Mile Island on May 30 - June 2, 1984.
O
(_/
Their purpose was to develop a report to be presented to the Commission for a meeting then scheduled for June 27, 1984.
Uhrig. Tr. 31,790-791.
The best evidence of what the Committee considered to be its task at that point is the Special Report itself.
The adequacy of the Committee's methodology in I
developing the Special Report must be judged on the basis of the conclusions presented in the report.
34.
In response to the Staf f's testimony critical of its methodology, the Committee argued that the Appeal Board in ALAB-772 did not intend the Committee to " validate" the licensed operator training program, but simply to provide the collective judgment of a group of individuals with expertise in the relevant areas.
Tr. ff. 33,320 at 9-10.
Similarly, Licensee now urges us 7s()
to find that the Appeal Board did not intend the Committee to undertake an " accreditation-type review" of the training program.
Licensee Proposed Finding No. 226.
35.
We consider these assertions to be of'little help to this discussion.
First, whatever was required by ALAB-772, the Special Report, and thus the remainder of the Committee's work, must be judged on the basis of the conclusions reached by the Committee, which unquestionably do purport to validate the TMI-l training program and urge restart on the basis of its adequacy.
S.R. at 82-83.
As the Commission and its adjudicatory boards have emphasized innumerable times, the Commission's regulatory scheme depends upon licensees providing accurate and complete r")
(_/
- information.
Thus, licensee assertions must be based upon
)
investigations that are adequate to support them.
Similarly, conclusions by licensee consultants such as this Committee must be based upon investigations that are adequate to support the conclusions as stated by the consultants.
If the Commission cannot be confident in those facts, it cannot be confident that its regulatory scheme is functioning effectively.
As the Appeal Board stated in this proceeding, "A licensee's responsibilities are increased by the Commission's dependence on the licensee for accurate and timely information about the facility and its operation."
ALAB-772 a t 1208, Citing Petition for Remedial and Emergency Action, CLI-78-6, 7 N. R.C.
400, 418-19 (1978).
36.
Second, the Licensee's reference to an " accreditation-type review" is simply a semantic argument.
It is certainly true
()
that the Appeal Board did not specifically require a review similar to what might be necessary to obtain accreditation from INPO or from any other accrediting body.
On the other hand, neither did the Appeal Board say that such a review would not be.
necessary.
Ra the r, the Appeal Board raised various issues.
The question now is whether the Committee undertook ~the investigation necessary to address those issues.
Thus, setting aside Licensee's semantic approach, we accept Licensee's proposed finding that the Appeal Board intended the Committee's review to have been f ashioned such that the Committee could provide us with knowledgeable and competent -testimony.regarding its opinion of the adequacy of the training program in light of the remanded a
s.
issues.
Hence, the threshold question is whether the Committee's method of review was suf ficient to allow it to knowledgeably address the remanded issues.
37.
We appreciate Licensee's emphasis on the credentials and expertise of these individuals.
Licensee Proposed Finding 224.
We, too, were impressed with the credentials and expertise of the Committee.
With adequate information and time, the Committee could likely provide a reliable evaluation of a licensed operator training program.
At the same time, we have no doubt that neither this Committee, nor any other group, no matter how expert, can develop a reliable evaluation without the factual information necessary tc reach reliable conclusions.
It is always tempting, even comforting, to rely upon expert opinion without questioning the adequacy of the information on which the
()
opinion was based.
We will resist that temptation.
Since the Committee's opinions are only as good as the facts on which they are based, we first consider to the nature of the methodology that is necessary in order to develop those facts.
11.
The methodology necessary to evaluate the TMI-l l
licensed operator training program.
38.
We received testimony from the NRC Staff and from UCS' witness Dr. James J.
Regan specifically on.the question of the methodology that is necessary to evaluate the TMI-1 training program.
Licensee did not present testimony directly on this point, although the Committee explained and defended what it had done.
39 We emphasize at the outset that no witness sought to identify a single methodology as the only one that would be appropriate to the situation.
Regan, ff. Tr. 32,693 a t 7,
- Staff, ff. 33,148 at 36.
Ra ther, both the NRC Sta f f and Dr. Regan identified appropriate mechanisms for developing the facts on which an evaluation could be based.
In our view, therefore, the question is not whether the Committee did what the Staff or Dr.
Regan describe as appropriate, but whether the Committee gathered the information necessary to undertake the evaluation.
The significance of the methodologies presented by Dr. Regan and the NRC Staff, therefore, is that they establish both the type of information necessary to support an evaluation and various means of obtaining that information.
(a)
Dr. Regan's Proposed Me thodology
()
40.
Dr. James J.
Regan testified on behalf of UCS.
Dr.
Regan is trained as an industrial psychologist with approximately 31 years of experience as a researcher in the area of training and education.
He spent most of that time as a civilian employee of the United States Navy, which has both extensive and complex training requirements.
'In particular, he served from 1973.to.
1982 as the founding Technical Director of the Navy-Personnel Research and Development Center.
His experience includes-extensive research in the area of technical training, as' well as o
e v
m.
the. initial implementation of Navy training programs on an experimental basis.
Although he is not extensively familiar with the design of nuclear reactors or with the training of nuclear reactor operators, we can discern no reason that his expertise is not directly applicable to the training of nuclear reactor operators.6 41.
We note that Dr. Regan has twice been called upon to advise the Commission in areas related to the training of nuclear reactor operators.
First, in 1981-82, Dr. Regan was the chairman of a national advisory committee to the NRC that was asked to advise the Commission concerning whether reactor operators and senior reactor operators should be required to have college degrees.
Second, in 1983, Dr. Regan was a member of a similar committee that advised the Commission on the question of whether n
(_)
each nuclear power plant should have a site-specific simulator.
6 We have been given no reason to believe the licensed operator training program at TMI-1, or at any nuclear f acility, dif fers from Navy training or f rom other industrial training in any significant way such that Dr. Regan's expertise is not directly applicable here.
Licensee itself does not suggest as much, although it seeks to denigrate Dr. - Regan's expertise by sug g es ting, for example, that he does not have much personal experience with the conduct of training.
Licensee Proposed Findings at 216, n. 104.
We reject Licensee's. invitation to minimize the significance of Dr. Regan's testimony on that basis.
This is a case in which the experts, the OARP Committee' and Dr. Regan, unquestionably-have the experience and training that is necessary to address the issues.
The question is not whether they are adequately qualified, but whether they gathered the information necessary to support.their conclusions.
O V
2 t
Regan, ff. Tr. 33,532, at 4.7 The fact that Dr. Reg a n ' s
(
expertise was recognized and sought for those committees supports our conclusion that his expertise in training applies to the training of nuclear reactor operators as well as to other types of technical training.
42.
In his direct testimony, Dr. Regan described the ef fort that would have to be undertaken to do a reliable evaluation of the TMI-1 training program in order to answer the questions posed by the Appeal Board in ALAB-772 and to reach conclusions of the sort reached by the Committee in its Special Report.
We agree with Dr. Regan's view that ALAB-772 poses "the fundamental question of whether the training program adequately prepares 7
Licensee notes that the process by which these advisory committees gathered information and developed their conclusions
()
is " strikingly similar to the approach initially used by the OARP Review Committee here."
Licensee Proposed Findings at 217, n.
109.
We agree.
We note, however, crucial distinctions.
The advisory committees were specifically intended to provide advice from experts who were not necessarily knowledgeable of the specifics of nuclear power or the training of reactor operators.
Tr. 32,727-728, 730, 32,820-821.
Moreover, as Dr. Regan explained, questions specific to nuclear training were not significant to their deliberations.
Tr. 32,731.
Finally, the advisory committees were not asked to pronounce on the quality of training at any particular plant.
By contrast, the role of the OARP Committee was to undertake an evaluation of a particular training program at a specific reactor and to advise the Commission and this Board on the fundamental factual question of whether that program is adequate to assure that operators are trained to run the plant safely.
That purpose is specifically reflected in the Special Report, in which the Committee reached precisely that conclusion.
- Thus, there is absolutely no basis for judging the OARP Committee's work favorably by comparison to the work done by Dr. Re g a n ' s advisory committees.
Ra the r, to the extent that the OARP Committee's work was similar to that done by the advisory committees, the comparison indicates that it was-inadequate to the task.
6
1 I reactor operators to operate the plant safely."
Regan, ff. Tr.
33,532 at 17.
In essence, the Licensee and the Committee sought e'
to respond to ALAB-772 by demonstrating the overall effectiveness of the training program, rather than by narrowly addressing each specific issue.
Thus, we must approach the issue as did Dr.
Regan, by determining what sort of investigation is necessary to i
form conclusions, such as those presented by the Committee, on the fundamental question of whether the training program adequately prepares reactor operators to run the plant safely.
43.
Dr. Regan wo uld use a team of personnel with both training and nuclear expertise.
Their first task would be to gather basic information and develop a plan of action.
Th is i
initial.information would come from the company's managerial
-i J
personn'el through briefings and. documents.
In addition, Dr..
()
.Regahi s team would review ' primary-materials such as examinations t and' progran documents, and ;it. would conduct -interviews with -
appropriate personnel.. Re g a n, f f. Tr. a t - 33,53 2 a t 18.
44.
Dr. Regan's ' fundamental premise is that "the training'
~
- +
program must be assessed - again'st. operational: pe'rformance of.
He considersL this measurement. to be 2
. individuals and-s'ystems."
the only reliable means-:cf-measuring'the: effectiveness-of ys training. 'Regan, f f TrII$ 3,532 a t' 9.. ThusI his_ methodology'is.
~
designed to'developithe best_possible information concerningJthe-
~
. relationship, betweenJ training and; job performance.
Qi
~
{YL
,(
3
- 7..
f
,7 a
'j,"
t
,g W
+
s-1 p
_f 1
W
.,s_
'11 45.
According to Dr. Re g a n, there are essentially three
()
questions that must be addressed by such a methodology.
- First, what measures of job performance apply to the job in question.
Second, what are the standards by which an operator's performance is judged.
Third, what are the actual performance levels of those who receive the training?
Regan, f f. Tr. 33,532 a t 11.
46.
In some cases, it may be simple to measure job performance, as in the job of producing golf clubs.
In others, objective performance standards may not be so clear.
For the latter situations, which we understand to include the job of reactor operator, Dr. Regan identified various other measures that may be used as indicators, includ ing examinations, fine tuning training through a process of constant development and evaluation known as formative evaluation, comparing instruction
()
methods to the state-of-the-art, and measurement of trainee attitudes to determine trends over time.
Regan, ff. Tr. 33,532 at 13-14.
}
47.
According to Dr. 'Regan,- the essential elements of a sound training program include diagnosis of the skills and knowledge of incoming students in order to tailor training to i
their needs, particularly for a limited' number of students as at TMI-1.
They also include thorough - job-task analyses that go beyond. mere descriptions of the actual operations done by the incumbent to analysis of various behaviors required of the.
p/ -
u.
L.
incumbent, including the type and frequency of the behavior and (1
the conditions under which the incumbent must act.
These elements then form the basis for determining standards of job performance and measurement of job performance.
Regan, ff. Tr.
33,532 at 5-9.
48.
Dr. Regan emphasized that a sound training program must include objective measures of performance at each point at which performance is to be assessed, and that there must also be reasonably objective measures of job performance.
These are necessary in order to allow an objective correlation between performance in training and performance on the job.
Regan, ff.
Tr. 33,532 at 10.
It follows, as Dr. Regan also noted, that performance measures must also be properly constructed, administered, and validated in order to assure that they are
()
measuring what they purport to measure.
Regan, f f. Tr. 3 3,532 a t 11.
14 9.
Findings 43-48 are based upon Dr.. Regan's discussion of essential elements of a sound training program.
It is clear from his testimony that the goal; of any. evaluation of a training.
. program must be.to assure that these elements are in place.' Dr.
Regan also identified specific-actions that he believes should be taken in any such evaluation.
50.
First, Dr. Regan's team would sample-the relevant ~
job-task analyses and learning objectives to determineEwhether
'they are technicallyLaccurate and contain'all ofLthe required
'o-d
=
information.
Where appropriate, this would include an examination of whether the material was consistent with the current design of the facility.
Regan, ff. Tr. 33,532 a t 18-19, 51.
Second, the team would review the training materials and instruction to determine both whether materials were technically correct and whether they were correctly administered.
Th is wo uld include an examination of the training and evaluation of instructors.
Id,.
52.
Third, the team would both observe simulator instruction and conduct examinations appropriate to determining whether the instruction resulted in any interference with the operators's ability to run the plant.
Id,.
53.
Where, as at TMI, job incumbents and trainees are subject to frequent changes in procedures and requirements, the
()
team would review the question of whether prior learning and experience may significantly inhibit both initial learning and retention of new material, an issue that can'be particularly important in emergency situations where an incumbent may revert to previous procedures.
Regan, f f. Tr. 3 3,532 a t 20 54.
Sixth, the team would review examinations and other devices used to assess training or job performance.
The issues are whether these devices rely upon objective and standar' ized-
~
d measures-of performance and whether the questions and. assessments are ' properly constructed such -that they test the ~ right behavflors and mental processes.- This review includes a determination of 1 [.
g-L.:
how assessment devices used in training correlate with measures of job performance, a step that must be taken in order to have any sound basis for an opinion as to whether the training program is producing operators who can operate the plant safely.
- Regan, f f. Tr. 33,532 at 21.
55.
The team would then review how the company uses the results of its training and job performance evaluations to feed back into the training program.
Ici.
56.
Finally, the team would systematically investigate the attitudes of trainees toward the training and the jobs.
Th is would be done through an anonymous random sampling technique developed by survey research experts.
Id.
57.
Based upon his admittedly limited knowledge of the TMI-l licensed operator training program, Dr. Regan estimated that it
()
would take a team of five qualified people three months to complete the effort that he believes necessary to respond to ALAB-772 and to reach conclusions such as those presented by the Reconstituted OARP Committee.
This estimate does not include the time necessary to draft any report that the team might produce.
Assuming a twenty-day work month,. Dr.. Regan's estimate involves three hundred person-days of work, not including the ~ time to l
write the report.
Regan, ff. Tt. 3J,532 at 22.
(b) The Sta f f's Proposed Me thodology 58.
We received testimony from the NRC 'Staf f which wa s limited to its assessment of the adequacy of the methodology employed by. the Reconstituted OARP. Review Committee ' during l its p)
~
l evaluation of the TMI-l licensed operator training program with
)
respect to the remanded training issues.
Staff, ff. Tr. 33,148, at 2.8 The Staff presented what it considers to be an appropriate methodological approach to assessing the licensed operator training program in light of the remanded training issues.
Tht: Staff presented its evaluation of the Committee's methodology by comparing its recommended methodology to that l
utilized by the Committee.
Staff, f f. Tr. 3 3, 14 8, at 3 1
59.
_The Staf f outlined approximately 110 steps that it felt should be taken or items that should be reviewed in an evaluation i
j of the TMI-1 licensed operator training program in accordance with the Appeal Board's remand.9 As did Dr. Regan, the Staff I
acknowledged that other reviewers might develop their own approachs. Tr. ff. 33,148 a t 3 6.
However, the. Staff asserted
/'(_-)
_ The three Staff witnesses were Dr. Julius J.
Persensky, Ps.
' Delores S. Morisseau and Mr. Joceph J. Buzy.
Dr. Persensky is Section Leader of The Personnel' Qualifications'Section, Licensee Qualifications Branch, in the NRC's Division.of Human. Factors Sa f e ty.
He holds a B.A.
in Psychology, an M. A.
in Experimental Psychology.
Ms. Morrisseau-is a Training and Assessment
. Specialist, Licensee Qualifications Branch, Division of' Human Factors Sa fety.
She holds a B.A.
in Psychology and an M.A.~
in Industrial' Psychology. 1k. Buzy, the Staf f's subject Lma tter-expert in this case, is a Systems Engineer (Training &
As sessment), Personnel Qualifications Branch, LDivision of Human Factors Sa fety.
Mr. Buzy holds a B.S.
in Marine Engineering in addition to his experience in the nuclear power field over the past 20 years. lSta f f, f f. Tr'. ~ 3 3,148,. attached qualifica tion.
l statements.
9 Licensee's Proposed Finding No. 259 character' zesl the i
Staff's-testimony as identifying " steps that'it' felt could be..
taken or actions that could be reviewed..."
(Emphasis added).
This is inaccurate.
While the_ Staff acknowledged that another group of professionals might use a;different approach, it testified that an _ appropriate methodology _ should include -'the' steps that it identified.
Se e e.g., ' f f. Tr. 33,148 at"8, 9,111, 36.
O
- that it had identified the essential elements of an appropriate
()
review, subject only to changes in specific details.10 60.
In summary, the Staff recommended that an appropriate review of the training program should include the review of the following documents:
the 1980 OARP Committee Review Report, LBP-81-32, LBP-82-34B, LB P-8 2-5 6, ALAB-772, the DDL Report (September 10, 1982), NUREG-0680 (June 1980) including Supplements 1 through 5, the RHR Report ( Ma rch 15, 1983), BETA Repor t (February 28, 1983),11 and the INPO Annual Report (1983), in addition to the training procedures and training materials (e.g., lesson plans, learning objectives, 10 In its testimony, the Staff acknowledges that it does not conduct its normal reviews of licensee programs using the methodology it outlined here..The Staf f explains this as reasonable because the Staff is constrained by law and N.R.C.
i rules in what it reviews, and its inspection program provides the Staff with regular input on the status of training at licensee facilities.
Staff, ff. Tr. 3 3,14 8, at 38.
We agree with the Staff.
This is not a normal review of a utility training program in a licensing case.
It is a proceeding in response to specific questions raised by the Appeal Board about a training program.
that was unique in.that it was plagued by cheating that occured in the context of a special restart proceeding in which operator retraining and competence were particularly emphasized as a i
result of the TMI-2 accident.
Moreover, licensee and the Committee chose in ' this instance _ to respond to ALAB-772 by a full review and defense of'the training program.
Given that approach, comparison of the Committee methodology to normal-Staf f reviews.
undertaken for an entirely dif ferent purpose : in 'an entirely -
different context would'be inappropriate and misleading.
- 11. - GPU Nuclear conmissioned Ba sic ' Energy Technology -
Associates, Inc. (BETA)- to review GPU Nuclear,1 including Lthe TMI-l ~ Training Departmenti from a cost efficiency standpoint in-December, 1981.
BETA's-findings.are documented'in:'its report-entitled:
" A Review of Current and Projected Expenditures. and Manpowerf Utilization for GPU Nuclear ' Corporation."' (Februaryf 28, -
1983).. See ALAB-738, supra,: 18 L N. R.C. a t ' 198-9 9.'
+=t-
'nW&
d
~
examinations) relevant to the remanded issues.
Staff, ff. Tr.
, ()
33,148 at 8-9.
The Staff's methodology would also include interviews with training managers, licensed operator ins tr uc to r s,
shift supervisors and operators, and observations of classrooms, instructors, and examination administration (written, oral and
, simulator).
Staff, ff. Tr. 33,148 at 9.
The Staff would approach its assessment of the training program in accordance 1
1 with the three topical categories that it derived f rom the remanded training issues.
These categories are: 1) ma nag emen t/commun ica t ions /a t t i tud e s ; 2) training systems / programs ; and 3) GPU Nuclear examinations.
Id. a t 10.
61.
In the Staff's opinion, an appropriate assessment of its first category (management / communications / attitudes) wo uld essentially consist of a series of interviews with training
()
management, instructors and operators, as well as an extensive review of relevant documents (e.g., documents concerning communications mechanisLs, management resumes and job descriptions, instructor resumes and performance evaluations, documents concerning the instructor development program, the RHR Re por t, NUREG-0680, Supp. 4, the DDL Report, and documents concerning attrition rates and absenteeism).
Id,. ~ a t 11-19.
Specifically, an evaluation utilizing the Staf f's methodology would include interviews with
- 1) all levels of training management'regarding communications mechanisms, implementation of-changes in training and operating procedures, and the. effectiveness of the instructor development program;
[v
- 2) operators with respect to implementation of changes in
)
training and operating procedures, their opinion of the quality of the instructors, their perception of communications, their attitudes toward the training program, their level of " pride and enthusiasm," and their perceptions of differences since implementation of new programs and criteria; and
- 3) the training staff to assess implementation of changes in training and operating procedures, and to ascertain their assessment of operator attitude and communications mechanisms.
Staff, f f. Tr. 3 3,148 a t 11-19.
62.
The Staff emphasized the need to structure the various interviews in order to assure that all points are covered and that interview results can be compared so that'it is possible to
()
draw consistent conclusions.
In particular, the Staff testified that questions related to attitudes should parallel questions used for the RHR Report so that the Committee could determine trends over time. -Staff-Tr. ff. 33,148 at 14, 16, 33, Tr.
33,140, 33,163, 33,189.
The Staff also testified that an evaluator would usually keep some kind of written. record of interviews.
On cross-examination, Ms. Morisseau agreed that if written records are not kept, the interviewer's instincts and other matters might interfere with accurate L recall -and judgment -
about the information obtained in the interview.
Morisseau, Tr.
38,163-64.
m_.
__ 63.
The Staf f's methodology would also include classroom observations by a subject-ma tter expert and a trained specialist to review the quality of instruction, instructor attitude, operator attitude and course content.
The Staff's observation of instructor performance would include evaluating the instructors against the Licensee's evaluation criteria.
Staff, ff. Tr.
33,148 at 16-17.
Finally, the Staf f recommends that an evaluating party review:
organizational documents to determine the structure of the training program and its relationship to corporate and plant management structure; documents concerning training department staf f qualifications and job specifications, especially those of Messrs. Long, Coe, Newton and Frederick, to ensure that they are qualified to serve in their positions; documents related to the cheating incidents to investigate the
()
involvement (if any) of these individuals; and documents describing the instructor development program, includ ing -
instructor criteria and instructor evaluation procedures and records of instructor attendance in addition to a review of the a forementioned documents.
Staff, ff. Tr. 33,148 at 16-18, 64.
A major theme of the Staff's testimony on evaluation of the management / communications / attitudes category of issues is the need for a quality assurance check.. In this category, the, QA check must cover, inter :alia, documentation of communications
. review of activities and evaluations' of instructors.
Staff, ff.
Tr. 33,148 at~12, 13, 18.
As discussed below, the. nee'd for a QA check is the most important point in the Staff's testimony.
' lm o
Simply put, unless the evaluator has checked information first hand, there is little basis for confidence in the evaluator's conclusions.
65.
The Staff's recommended approach to the evaluation of its second category (training system / programs) is dedicated in large part to ensuring that the licensed operator training program is performance-based.
The Staff's methodology involves the review of the job / task analyses, including correctness of the tasks, and the procedures for linking job / task analysis data to learning objectives.
The Staf f's methodology further recommends the review of learning objectives to ensure that they are clearly stated and that they properly reflect the task analyses for each job.
The evaluating party should observe on-the-job training to ensure that it is consistent with job / task analyses and actual
()
plant operations.
It should review or observe classes, lesson plans, handout material and simulator training to ensure that these items are consistent with program descriptions.
With respect to classes, Ms. Morisseau testified that as an evaluator she would sit in on more than one class given by a instructor in any_particular area in order to assure consistency over ' time.
Tr. 33,156-57.
Ms. Morisseau similarly testified that she would also sit in on classes given by other instructors in the same subjects.
Tr. 33,157.
66.
Also, the reviewing group should review simulator training to determine whether it is centered on problem-solving and symptom-based analyses.
The Staf f further believes that an appropriate review would include the evaluation of the.
.p v
performance evaluations of the operators who have gone through
()
the training program in order to identify deficiencies that could be traced back to training.
Staff, ff. Tr. 33,148, at 19-20.
67.
As a part of the inquiry into the program's adequacy, the Staf f would consider the Appeal Board's question whether deficiencies in testing were symptomatic of more extensive failures.
Here, the Staf f would review the cheating decisions to l:
ascertain what deficiencies existed before, and review the relevant areas of the training program, including lesson plans, to determine whether more extensive failures now exist in the training program.
Id. at 21.
As to whether-the training program enhances knowledge rather than encouraging memorization, the i
Staf f recommends first determining the extent of memorization f
-required to perform as an operator.
The reviewing group should
()
then inspect lesson-plans and class instructional plans for
~
inappropriate repetition and integration of training concepts i
with plant operation requirements.
Classes should be observed to determine if instructors encourage memorization through repetition-and to' determine 11f there'are opportunities for discussion and team work.
If memorization is required the reasons'for it should be explained to the reviewing group.
Quizzes and examinations. should be inspected' to determine the types of. questions asked, and the balance between the mental-processes required to' respond to these questions.
They.also should be reviewed to ensure that.the questionsLencourage discussion of the relationship!between concepts Land' operational; 3
e I
~{N
_ requirements.
At the simulator, lesson plans should be reviewed and exercises observed to ascertain whether a variety of situations are presented.
Simulator examinations should present situations novel to the trainees, and oral examinations should include discussion of concepts as related to plant operations.
Id. at 22-23.
68.
The Staff's recommended evaluation also would require i
I the examining party to:
visit the TMI training center to assess its adequacy and to observe the instructors' use of training aids; evaluate training expenditures to determine the adequacy and approprietness of the training programs; observe and evaluate the training instructors'; qualifications against the documented instructor criteria, in addition to reviewing the new instructor evaluation forms; interview instructors concerning
()
their training; observe classes and evaluate instructors against the company's criteria; review the simulator tra'ining lesson plans and learning objections for consistency.with task analyses; have a subject-matter expert observe simulator training at PSI (B&W) and on the BPTS; and review the GPU Nuclear performance evaluations of simulator instructors.
Staff, ff. Tr.
33,148 at 23-26.
69.
Once again, a fundamental point of the Staf f's testimony is the need for quality assurance checks.
In this area, this is perhaps most significant with respect to the job-task analyses and learning objectives on which licensee relies to develop _its training program.-
/
_Y 70.
Finally, with respect to the Staff's recommended
()
methodology for assessing its third category (GPU Nuclear examinations), the Staff witnesses testified that exam development procedures, security procedures, content, format and administration should be reviewed by direct inspection of the exams and by observation of the administration of exams.
To this end, the Staff testified that a party conducting an evaluation of this issue should review:
documentation describing exam security; construction and administration procedures for written; oral and simulator exams; exam content to er-*Me that it is consistent with job / task analyses,. behavioral:lt: sing objectives and current plant design; exam questions to determine balance between responses testing memorization and those testing an operator's ability to' solve problems and address plant
()
systems.
The evaluating group should:also observe and review the content and methodology of simulator. and oral. exams, as well as the standard for oral' exams.
These observations should cover examinations by various examiners.in'different subject areas.in; order-to obtain a reasonable sample:of the. oral exams.-
Morisseau,.Tr. 33,164.
Significantly, the evaluating group should check examination resultsfagainst personnel evaluations ~to determine program validityf and_ to ensure L that Licensee's written exams offer an effective means of measuring an operator',s; ability
- to run the plan'.,.
The; reviewing party should, finally,: observe the administrataon of-examinations,, review exam answer keys (for technical.accurtcy, and interview trainees ~to.ascertainLtheir opinion of the 7portance of exam: integrity.
St a f f,. f f '. Tr.'
'33,Y48,at'27-3J.-
1 s~ =
IL y
s
)
l.
h'
. 71.
In addition to discussing the appropriate methodologies j
for resolving the three categories of issues, the Staff testified to two methodological concerns that the Board believes are of crucial importance to this proceeding.
First, in a colloquy with Mr. Jordan, the Staf f agreed that, depending upon the use to which conclusions may be put, it would be inappropriate to reach conclusions if constraints of resources and time prevent an evaluator from adequately implementing the methodology necessary to address the questions at issue.
Morisseau and Persensky, Tr.
33,174-77.
Second, it follows from that discussion that it would be an inappropriate methodology to develop conclusions on the basis of inadequate information and then to attempt to justify
~
those conclusions through later investigation.
72.
The Board would not need Staf f testimony to establish
()
these propositions.
They are simple common sense.
Any time someone has previously committed hinself to a particular conclusion, his later investigation of the question will be a f fected by that fact.
This is particularly true where the conclusion has been presented in a formal context for the. purpose of influencing a government decisionmaker, as was true of the conclusions in the OARP Committee's Special-Report.
73.
This leads us once'again to note the importance of the Staf f's emphasis on the need for quality assurance checks in order to assure that the~ underlying. facts support appearances and that programs work as program documents indicate they should.
It-
\\-)
is clear to us that the evaluation of a training program is to a
. ()
large extent a subjective exercise.
Thus, it is essential to introduce as much objectivity as possible into the evaluation.
Where, as here, an evaluator reached conclusions that it sought to justify by later investigation, the problem of subjectivity is even greater, as is the need for objectivity.
iii.
Conclusions as to an appropriate methodology 74.
We agree with all parties that there is no one detailed metbodology that must be followed in order to address the issued remanded in ALAB-772.
Dr. Regan and the NRC Staf f have both presented acceptable methodologies.
Neither the OARP Committee nor the Licensee sought to do so during the course of the hearing.
For the most part they do not dispute that the types of methodologies proposed by Dr. Regan and the NRC Staf f would be
()
appropriate to address the questions, although they disagree with respect to some details.
- See, e.g., Committee, ff. Tr. 33,320 at 14-15, 33,356-357.
75.
Accordingly, we do not hold that an appropriate i
methodology required consideration of all of the detailed points suggested by Dr. Regan and the Sta f f.
We do find, however, that a methodology is inadequate unless it achieves the fundamental goals of those discussed by Dr. Regan and the Staf f.
O)
(_
76.
In our view, an adequate methodology for addressing the
()
remanded issues and reaching the conclusions stated in the Special Report and testimony of the OARP Committee must includ e the following:
1.
Investigations to assure that various aspects of the training program work as intended.
Thus, for example, if the question is whether communications i
i are adequate, the investigation must exanine the documentation of communications mechanisms and interview those to whom information was intended to be communicated to determine whether, in fact, the communication was effective. 'This is the sort of i
quality assurance check recommended by the Staff.
It need not, however, comply with formal quality
(
assurance requirements as contained in Appendix B to 10 C.F.R. Part 50.
2.
An attempt to maximize the -objectivity and reliability of information obtained in the investigation.
With respect to attitudes, for example, this' would require some degree of standardization in interviews, as well as a written 4
record of all interviews.
3.
Investigations to assure that all facts are as claimed by the company.
This includes, for example, reviewing; job-task analyses to assure that they'are adequate on their face, and comparing the (K
L_)
. analyses to actual plant conditions to assure that
., ~.
(
the analyses are are both technically accurate and up to date.
4.
Assurance that the evaluator has considered all information relevant to the evaluation and has been able to make an independent judgement as to the significance of the information.
5.
Coverage of all significant issues.
i-77.
These are the principles that govern our review of the l
work done by the Reconstituted OARP Committee.
To avoid 1
- redundancy, we have not described an appropriate or ideal j
methodology in any greater detail.
In the' discussion that f
follows, we will address the' Committee's methodology in light of these ' principles and of the more detailed testimony of Dr. Regan
()
and the NRC Staff.
I c.
The methodology employed by the Committee ~in j
preparing the Special Report, i
1.
The Committee could not have implemented 'an adequate methodology in the time available.
78.
The Committee's methodology -in preparing the special i-Repor t is most clearly : established by a review of the. Committee's actions f rom May. 30, 1984,. to - June : 28,1984, the day.the Special Report. was completed.
Committee members were contacted either.by..
the company or by' Dr..Uhrig within approximately.tte weeks - before j
Ma y - 3 0, 1984.
Tr. 31,788-790.. When Dr. L Long contacted Dr.
- Uhrig, he explained that-the purpose of,the meeting was to answer some of the issues raised in ALAB-772 and 'that ~ this. needed to-be -
t done very quickly so that the report would be available for a meeting of the Commission that was then scheduled for June 27, 1984.
As a result of that discussion, Dr. Uhrig understood that the Commission might vote on whether to restart TMI-l in the same meeting in which the Commission considered the Committee's report.
Uhrig, Tr. 31,790-91.
Although it is not clear how other Committee members were informed of the purpose of the Special Report and its involvement in the Commission's meeting, they testified that their understanding of the matter was the same as Dr. Uhrig's.
Id,.
79.
Four of the Committee members first met as a Reconstituted Committee on May 30, 1984, at Three Mile Island.
The mee ting lasted f rom May 30 to June 1,1984.
Dr. Kimel had a conflict and was not present.
In addition, Dr. Uhrig was not
()
present on June 1, 1984.
Tr. 31,789, 31.791-92.
Th us, Dr. Uh r ig spent two days at the meeting, while the other three participants spent three days each, for a total of 11 days spent by Committee members during the first meeting.
- 80..The first time that the Committee received any information about the current training program at TMI-1 was at the meeting on May 30.12 Tr. 31,792-793.
12 This does not, of course, include information obtained by the Committee in the course of its original work several years earlier.
It also does not include information that Dr. Gardner
~
may have obtained in the course of giving a seminar on test construction some two years earlier. : Ga rdner, Tr. 31,792.
Dr.
Gardner did not testify that he received:any significant information during that seminar, and the Board concludes that the first-time that any member of the Committee received and began to review information significant to this proceeding was May 30, 1984.
p,.
qp L'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 81.
During the meeting of May 30 - June 1, 1984, at Three A
' (_)
Mile Island, the Committee received various briefings f rom 1
Licensee management personnel, toured the facilities at Three Mile Island, and had individual discussions with various management personnel whom the Committee identified in the Special Report.
Uh r ig, Tr. 31,793.
t 82.
As far as we can tell, the Committee spent approximately half of its time from May 30 to June 1 in meetings with GPU management.
In addition, the Committee appears to have spent between one-third and one-half of the remaining time during that period in discussions with management.
Tr. 31,794-796.13 Thus, the Committee spent between two-thirds and three-quarters of its time from Pay 30 to June 1 with Licensee management personnel.
The Committee appears to have spent the some
! ()
additional time reviewing documents, particularly ALAB-772, which Committee members saw for the first time on Ma y 30, 1984.
According to Dr. Gardner, however, the purpose of this meeting was to determine what the Committee would focus on and what approach it would use in attempting to respond to ALAB-772.
Ga rdner, Tr. 31,797.
Finally, the Committee assigned tasks for
. preparation of the Special Report, including which members of the Committee were to draft which sections of the report.
Tr.
31,793-798..
13 Dr. Christensen appears 'to have estimated that the Committee spent slightly less time in briefings with Licensee management personnel.
Tr. 31,796.
Although the Board has no doubt of Dr. Christm..?n's integrity, he consistently revealed a poor memory for specific facts and dates.
Se e, e.g., Tr.
31,822-824.
Thus, the Board finds that. an estimate consistent
(~~}
with figures given by Dr. Uhrig and Mr. Kelly is more likely ~ to V
be correct.
l
. 3 83.
When the Committee members left Three Mile Island, they
,N.
appear to have taken a number of documents with them for review.
Dr. Uhrig, in particular, took home approximately five inches of material.
Uhr ig. Tr. 31,796.
Af ter he lef t Three Mile Island on Pa y 31, Dr. Uhrig spent some 12 to 15 hours1.736111e-4 days <br />0.00417 hours <br />2.480159e-5 weeks <br />5.7075e-6 months <br /> reading these materials before the Committee reconvened on June 6.
Dr.
't Christensen could not recall how much time he spent on this activity, or whether he read all of the material that he took with him.
Mr. Kelly estimated that he spent about 10 hours1.157407e-4 days <br />0.00278 hours <br />1.653439e-5 weeks <br />3.805e-6 months <br /> reading TMI-1 materials during this period, and Dr. Gardner estimated that he spent two or three days in that activity.
Tr.
31,800-801.
4 84.
Converting these estimates to a reference point of days, Committee members spent some 5 to 6 days on their TMI-l work
, ()
during this period, plus an unknown amount of time put in by Dr.
Christensen.
The Committee must have spent all of this-time reviewing information that it had received during its first meeting because no member of the Committee received any further information before the Committee's second meeting began on June 6,
1984.
Tr. 31,799-800.
85.
The Committee reconvened on June 6, 1984, in Parsippany, New Jersey.
This time, the entire Committee was present, includ ing Dr. Kimel.
Uh r ig, Tr. 31, 8 01.
Dr. Kimel first received information concerning his -work with the Committee when he arrived on June 6.
Kimel, Tr. 31,800.
All members of the Committee were present for the entire time, except that Dr. Kimel O
i '
visited Three Mile Island on June 7, 1984, the second day of the meeting.
Tr. 31,802.
Thus, the Committee spent a total of 15 days on its TMI-1 work during the meeting of June 6-8, 1984.
86.
On June 6, 1984, the Committee spent most of its time throughout the day discussing ALAB-772, including who was going to prepare material dealing with particular issues, what the general approach should be, and what the approach should be for responding to particular issues.
Uh r ig, Tr. 31, 8 0 3.
87.
On June 7, 1984, while Dr. Kimel visited Three Mile Island, the remaining members of the Committee held a series of meeting at which they discussed how they were progressing.
They appear to have spent time individually preparing the report, f
including writing their portions of it, and meeting periodically, l
which they did three or four times during the day.
According to O)
(,
Dr. Uh r ig, the day was basically spent "in our own wr iting --
reading and discussions," the " iterative process" by which the Committee developed and wrote the report.
Uhr ig, Tr. 31,804.
88.
While at Three Mile Island, Dr. Kimel talked with Mr..
Newton and Mr. Gif ford, a GPUN Vice-President who is responsible for communications at TMI.
He also talked with Mr. Irizarry and Mr. Boltz, who demonstrated the Basic Principles Training Simulator.
Kimel, Tr. 31,803-804.
He must have been there approximately half a day since he lef t Newark Airport for TMI a t 8:00 a.m. and started back f rom TMI about 1:00 p.m.
He rode back f rom Three Mile Island with Mr. Newton, with whom he discussed the remanded issues.
Upon his arrival in Parsippany, (O
_/
i he rejoined the Committee for another meeting and further individual work, which involved writing and discussions with other Committee members.
Kimel Tr. 31,802-803.
89.
On the evening of June 7, 1984, Dr. Uhrig was able to i
pull together all of the material that the Committee members had prepared so that by the morning of June 8, 1984, the Committee had the first semblance of an outline of the report.
Uh rig, Tr.
31,805.
90.
The Committee met early on June 8, 1984, to discuss the materials that it had developed by that point and to decide what additional material was needed.
It then spent the remainder of the day in the same iterative process of individual writing and t
meetings, including some three or four meetings of the Committee as a whole, each lasting half an hour to an hour.
By very late
()
that night, the Committee had what Dr. Uhrig considered to be the first draft of the report.
Uh r ig, Tr. 31,805-806.
91.
Sometime near the end of the meeting of June 6-8, the Committee learned that the Commission meeting for which it was preparing the Special Report had been delayed, and that the Committee now had approximately two more weeks before the report was due.
Uh r ig, Tr. 31,807-808.
92.
Dr. Uhrig stayed at Pa rsippany into June 9,1984, to get-the first draft put together and turn it over to Dr. Coe.
- Uhrig, Tr. 31,805-806.
Af ter leaving Parsippany on June 9,1984, Dr.
Uhrig spent "a good deal" of his weekend on the project, including substantial time on the telephone with other members of
- ()
l f the Committee.
The purpose of those conversations and of the remainder of his work during this period was to discuss and edit the draft, which Dr. Coe had sent to the other Committee I
members.
As far as we can tell, Dr. Uhrig spent approximately
)
two days on Committee work between the time he lef t Parsippany on June 9 and when he returned on Wednesday, June 13, 1984.
- Uhrig, Tr. 31,810-811, 31,807.
93.
While he was in Parsippany on June 13 and 14, 1984, Dr.
Uhrig was involved in continuous editing and draf ting of the Special Report based upon his telephone conversations and some material that Committee members had mailed in.
Uhrig, Tr.
31,807.
By the end of June 14, 1984, the Committee had produced a semi-final document, after which final editing was done over the telephone with other Committee members.
Id,.
()
94.
Af ter Dr. Uhrig lef t Parsippany on June 14, 1984, he spent some 20-25 hours on the Special Report before the document left his office on June 28, 1984.
This translates into 2 1/2 to 3 days.
Uh r ig, Tr. 31,811.
95.
Dr. Kimel estimated that he spent approximately three days working on the Special Report between the time he lef t Parsippany on June 8,1984, and June 28.
Kimel, Tr. 31,812.
Dr.
Christensen estimated that he spent 75-80 percent of his working hours on the Special Report between June 8 and June 28, plus additional hours on weekends, and probably work hours that he did not record. 'Christensen, Tr. 31,812-813.
Re ferring to a calendar for the period, this translates into somewhere between
10 and perhaps 15 days.
Mr. Kelly spent about two days on the project during this period, Tr. 31,813-814, and Dr. Gardner spent 10 to 15 days on the Special Report between June 8 and June 28.
Gardner, Tr. 31,814.
96.
Based upon the Committee's testimony, we are able to determine that the Committee spent between 63.5 and 75 days on the Special Report.
We cannot determine precisely how many of f
those days the Committee spent before June 12, the day the Committee originally expected to complete the report, Uhrig, Tr.
31,807-808, and June 28, the day the report was finally completed.
At best, we can estimate that Dr. Gardner must have spent a minimum of 9 to 11 days on Committee work af ter June 12.
i (He apent 10 to 15 days af ter June 8, and there were only four days between June 8 and June 12.)
Dr. Christensen must also have
()
spent a minimum of 9 to 11 days af ter June 12.
Mr. Kelly could have spent all of his time before June 12.
Dr. Uhrig spent 2 1/2 to 3 days after June 14.
Dr. Kimel could have spent all of his time before June 12.
Thus, subtracting the time that was or must have been spent after June 12, we conclude that the Committee could have spent approximately 45 to 50 days on the Special Report before June 12, the date the report was originally due.
97.
The time spent on this report is significant because, as i
we have previously emphasized, the Committee knew that the Commission might rely upon this report in deciding wnether to allow the restart of TMI-1.
The - figures in the preceeding paragraph demonstrate one disturbing thing.
The Committee was (v~)
'3 3 -
presumably prepared as of June 8, 1984, to file the report on the
()
original deadline of June 12.
Only on June 8 did the Committee know that more time was available for its work.
Committee members then found it appropriate, and presumably necessary, to spend some 18.5 to 25 additional days on the report, which they could not have spent if the deadline had not been relaxed.
That is between 29% and 33% of the total time spent by the Committee.
Although hardly conclusive, it is disturbing that the Committee was prepared to issue the Special Report, with its unqualified conclusions and potentially significant influence on the Commission's decision, without doing a third of the work that it eventually felt necessary to do on the report.
98.
Our concern is only increased when we consider what the committee actually did in preparing the Special Report.
As the
()
Committee's testimony reveals, and as the Staff has testified, S.R. a t 3, and Sta f f, f f. Tr. 3 3,14 8 a t 7, 15, 16, the Committee obtained information only from Licensee management personnel, and it spent its time examining documents rather than undertaking the investigation necessary to verify anything that it learned.
In fact, a review of the time spent by the Committee reveals that it began drafting the report on June 7, 1984, after the Committee members had spent only some 27 days on the project.
At that point, Dr. Kimel had received relevant information only the i
previous morning, yet he participated in the draf ting.
The Committee spent a large part of its remaining time drafting and editing the report, so that the time that the Committee actually-spent in any sort of investigation must have been minimal.
We A'
believe it would be charitable to conclude that the Committee
(
half of its time obtaining and analyzing information, as spent opposed to discussing the issues, deciding how to proceed, and i
massaging the drafts.
Although we do not consider ourselves to we find it both be bound by Dr. Regan's estimate on this point, instructive and disturbing that if we are correct in our estimate, the Committee spent between 10% and 13% of the time that Dr. Regan estimated would be necessary to do an adequate study.
Re gan, ff. Tr. 33,532 a t 22.
We do not believe that Dr.
Regan's estimate could be in error by a factor of 10.
ii.
The Committee's methodology in preparing the i
Special Report was inadequate.
99.
Our review of appropriate actions that the Committee failed to take in preparing the Special Report further heightens
! ()
our concerns:
1.
The Committee did not review any actual training in the licensed operator training program, whether in the classroom or at the simulator.
No member of the Committee even visited the B&W simulator.
I Uhrig, Tr. 31,816, 31,823, Kelly, Tr. 31,818,-919 Christensen, Tr. 31,822-24, 31,905-906.14 l
l
.14 ' Dr. Christensen exhibited ' considerable ' confusion about when
~
he first monitored a training class.
Tr.'31,822-24.
He
. ultimately confirmed,'however,~that he did not' attend any. classes
- until August 1984.
Tr.-31,905-906.
Dr. Gardner testified that he did sit on one class, but it was not 'part of the : licensed operator training program.
Tr. 31,820-821.
f
. l 2.
The Committee did not interview any trainees or i
)
operators who were not part of Licensee management.
Uhrig, Tr. 31,816, Ke lly, Tr. 31,818, l
31,836-838.15
{-
-3.
The Committee did not undertake any evaluations of 1
l licensed operator training instructors.
Uh r ig, Tr.
31,825.
4.
The Committee did not review ^ any particular job-task analyses.
Uh r ig, Tr. - 31,825.16 5.
1&ie Committee did not review operating procedures, emergency procedures, or ATOG procedures to determine --if the training. program wa s consistent with those procedures.
Uh r ig,.Tt. 31,823-824.
In f act, neither Dr. Uhrig, Mr. Kelly, nor Dr.
()
Christensen reviewed these procedures at-all.
Uhrig, Tr. 31,816, : Kelly, : Tt. 31,818, Ch ristensen, Tr. 31,822.
We have no evidence in the: record as 15-Mr. Kelly did discuss operator attitudes withL Mr. Boltz, 'a simulator instructor, but.the conversation -was apparently limited a
to-the proposition that a. lack of1 turnover: among operators indicated that morale was high.
We 'have no evidence that Mr.
Kelly' addressed any. other issues with Mr. Boltz,~ ' including attitudes -towa rd the training ~ program, reactions to ' the Lcheating::
incidents,; or. anything of that sort. ' Tr. - 31,838. -
. 1 61 Dr. Kimel' asserted that he did peruse.the Plant Operations.
Ma nual, which '" looked -like task analysis" ' to. him. - He - spent about half an. hour in.this perusal.
,Tr.431,825. 'The1 Plant Operations:
Manual, which 'is based on learning objectivesiand Lthus would' at.
E least be relevant 'to job-task analyses,f is '9 ' volumes and 'several' ifeet' thick.
Kimel, Tr. 31,826.
While we s were ; impressed with:Dr.
Kimel's backg round and qualifications,: we ' cannot imagine' that: he; undertook a. serious? review of'either. job-task analyses or that 1PlantcOperations Manualitself in that half hour.
g.
~
' \\;
f, i
1 ?'
- )r1-
-. to whether Dr. Kimel or Dr. Gardner reviewed these procedures at all before issuance of the Special Report, but it is clear that without nuclear expertise, Dr. Cardner did not have the competence to undertake such a review, Gardner, Tr. 31,822, and Dr. Kimel did not have time to do an extensive one if he did any.
Since Licensee must carry the burden of proof and has not presented any proof on the point, we conclude that no member of the Committee reviewed the procedures in question before issuance of the Special Report.
6.
The Committee did not observe the administration of any examinations of any sort.
Uh r ig, Tr.
31,824.17
(
17 The Special Report asserts that-Dr. Ga rdner observed the administration of two oral examinations during the OARP Committee's original work.
Sr. a t 5 0.
We took this to mean that Dr. Gardner had seen oral examinations administered by the
. Licensee's personnel, which would -make the testimony relevant to this proceeding and could support the Committee's favorable conclusions.
Indeed, that is the only reasonable reading of the Special Report.. We are quite -disturbed to learn that the ' oral' examinations discussed in the Special Report were not even administered by Licensee personnel,' but by Mr. Kelly and.one of
~
his employees.
Gardner, Kelly,rTr. 31,820.
If the Committee had read its Special Report with care, it would have known - that we and the Commission would interpret this aspect of the Special Report as supporting the Licensee's. training program, when it did no such-thing.
We doubt that the. Committee intended to be.
deceptive, but the report is deceptive on this point..This sort tj of carelessness seriously undermines the Committee's presentation' and-testimony, particularly because the ' Committee -relied so heavily on its subjective judgment and its expertise, as opposed'
,to data obtained through the careful and standardized procedures..
' proposed by Dr. Regan and ' the NRC Sta f f.-
i-T U[r P
. 7.
Only Mr. Kelly appears to have reviewed any written llh examinations, two reactor operator and two senior r
reactor operator examinations in the 1982/1983
[
requalification cycle.
Tr. 31,814-815.10 Mr.
Kelly first testified that he took the examinations r
with him for extensive review, but not for input into the Special Report.
Id,.
When asked to expand on that answer, Mr. Kelly simply stated that he did
=
review those examinations prior to the Special Re por t.
Kelly, Tr. 33,275-276.
We cannot determine on the basis of this testimony whether 1
Mr. Kelly undertook any review of examination h
construction, balance of questions, potentially excessive memoriza tion, technical accuracy, or any I;
of the other matters at issue in this proceeding.
{
ggg 8.
Dr. Uh r ig, the Chairman of the Committee, did not Y
spend any significant time reviewing training
}
materials, and he spent none reviewing curricular s
materials.
Uhrig, Tr. 31,815.
He also did nothing to check the accuracy and consistency of the curricula materials or examinations against the 18 The record is unclear as to whether Dr. Gardner reviewed E
any written examinations before issuance of the Special Report.
{
At one point he testified that he did not, Tr. 31,815, but a t-another point he testified that he did.
Tr. 31,819.
It makes no L
difference, however, because he also testified that at the time h
the Special Report was issued he had reached no judgment as to whether the examinations adequately covered the material, or whether they required the proper balance of mental activities.
s Id.
5 L$
r i
.. actual design of the plant.
Uh r ig, Tr. 31,817.
()
There is also nothing in the record to suggest that any of the other members of the Committee undertook such an effort prior to issuance of the Special Re po r t.
We are constrained to conclude that the Committee undertook no such review if it presented no testimony to that effect af ter the question was raised.
Mr. Kelly's testimony that he did not review any training program descriptions prior to issuance of the Special Report, Kelly, Tr.
31,818,19 is consistent with our conclusion on this point.
9.
Prior to issuance of the Special Report, no member of the Committee did anything to evaluate the-()
consistency of the examinations and answer keys with the actual cur rent TMI-l design.
Uhrig, Tr.
l 31,826.
10.
Prior to issuance of the Special Report, the Committee did not review the responses by TMI-l I-operators to the -RHR Report, which had assessed employee attitudes with apparent negative results in 1982.
Uhrig Tr. 31,826.
Since the Committee.
obtained all of its information with ~ respect to 19 Mr. Kelly later testified that he ~ reviewed ;the scope and -
content of the requalification program in' connection.with his review of examinations.' Tr. 33,283.- Since he reviewed examinations both before and after issuance of_the special-
. report, this is consistent with his -statement - that he did not review program descriptions-priorLto issuance of the1special
- (^3 -
report.
Thus, we_ conclude t'aat he must have _exami'ned the program-..
(_/
descriptions af ter ; issuance' rif the special report.
1 m.
~. operator attitudes from management personnel and
()
did not even attempt to interview the operators themselves, the Committee should have insisted upon reviewing any documentary evidence related to operator attitudes.
Since they did not even learn of the existence of the RHR report in any detail until October, Tr. 31,851, 32,038, they clearly did not assure that they had seen all of the relevant evidence.20 100.
In light of these deficiencies, the Committee's work in preparing the Special Report was grossly inadequate to reach an independent judgment on the issues remanded in ALAB-772 or to support the unqualified conclusions presented in the Special Report.
A comparison to the methodologies discussed by Dr. Regan
()
and the NRC Staff r? veals that the Committee did virtually none of what it needed to do.
101.
A summary of Dr. Regan's specific recommendations appears in Findings 50-56.
These actions, or some substitute for them, are necessary elements of an evaluation of a training 20 Licensee sought on cross-examination to minimize 'the significance of the Committee's failure to review the RHR report.
In essence, Licensee's point is that-later information-about attitudes in NUREG-0680, Supplement 4, may establish that the RHR report findings were incorrect, or :at least that the-situation was not as bad as it might have seemed from reading the '
RHR report itself.
Tr. 33,235-245.
This ef fort is. irrelevant to.
the issue-here.
Our concern is whether the Committee undertook-an independent and sufficient effort to evaluate'the.TMI-1 training program.
If the Committee f ailed to review information such as the RHR report and make its own independent ~ judgment of
' the significance. of the ;information, Tm question whether the' Committee's actions were suf ficiently. f ree of Licensee
~
' influence.
The argument that the Committee would not have~ been concerned about the RHR report does not alleviate this concern.
/~N
(-)
This.is another example of the needffor care and objectivity;in a review of this sort.
Se
I In the context of TMI-1, review of job-task analyses program.
()
and learning objectives is probably most important because those the primary means by which the Licensee seeks to assure that are its operators are adequately trained to operate the plant safely.
The Committee did not review these matters for the Special Report.
Nor did it attempt to evaluate the quality of the instructors, observe the use of the simulator, review examinations to any significant extent, or interview trainees to determine their attitudes toward the training and the jobs.
Mo st important, the Committee did nothing that we can identify either to investigate the correlation between performance in training and performance on the job or to determine-whether Licensee undertakes such a correlation.
That is the single most important question, and the Commi ttee f ailed to do what was necessary to
()
answer it.
102.
A review of the Staf f's proposed methodology as set out in Findings 61-70 leads to similar conclusions.
The Committee did not do most of the interviews discussed by the Staff, but relied in the Special Report entirely on information f rom Licensee management, which can hardly be characterized as an The objective source of information in this particular. context.
Committee did no quality assurance checks, either formallor informal, to assure that the information and assurances provided by the Licensee were correct.
The Committee did not review job-task analyser and related documents, nor -did it even cbserve any training.. The-Committee did nothing beyond its general document review to address the-issues of excessive memorization-f))
Q,
'q
~
or impacts of and potential for cheating.
The Committee did not
()
observe the administration
' any examinations or any other evaluations on which the Licensee relies to decide whether to permit trainees to become qualified reactor operators.
103.
Finally, we refer to what we identified in Finding 76 as the essential elements of an adequate methodology.
The Committee met none of these relatively general standards.
It did nothing to assure that various aspects of the training program work as intended.
Not only did it f ail to seek objective information, it did not seek any independent information, however subjective.
It relied upon documents and assurances provided by Licensee.
It failed to obtain and make an independent judgment as to the significance of all relevant information.
Since it did review the job-task analyses and learning objectives on which not
()
the TMI-l training program is based, it failed to cover even the most important issue before it.
104.
One major f ailing of the Special Report is representative of the Committee's overall. failure to undertake a thorough and independent review of the TMI-l training program.
At page 17 of the Special Report, the Committee discussed the
_ qualifications of Edward J. Frederick, :then Supervisor of Licensed Operator Training at TMI-1:
Edward J. Frederick _(Supervisor, TMI Licensed Operator Training) provides the TMI training program with extensive education and experience in nuclear power He is a graduate of :the Navy Nuclear plant operation.
Power Program and has completed courses to qualify him as a certified TMI-l senior' reactor operator instructor.
and - is s licensed TMI-2 senior reactor. operator.
Le A
um S.R. at 17.
The Committee further emphasized Mr. Frederick's
()
significance to its review by noting that "as an experienced licensed TMI s tation operator, [he] directs and monitors the TMI-licensed operator program so as to ensure orderly transition f rom the Navy program to the TMI program.
Additionally, his experience during the TM1-2 accident and his subsequent education and training on small break LOCA phenomena qualifies his credentials in this area."
Id. at 18.
105.
This language indicates that Mr. Frederick plays an extremely important role in the training program.
The Committee identifies attributes and experience on his part that apply to no one else.
As it turned out, however, this language was misleading, at best.
Although FJ. Frederick had taken the courses necessary to qualify as a certified TMI-1 senior reactor
()
operator instructor, he had also, in Ma rch of 1984, failed the examination for which those courses were intended to prepare him.
Uhrig, Tr. 31,750.. Dr. Uhrig later testified not only that this failure was relevant to the committee's evaluation of Mr.
Frederick, but that-it was a very important consideration.
Uhrig, Tr. 31,960-9 61.
Thus, the Special Report created a serious misimpression as to the status of this important individual.
106.
The Staff testified that the Committee's failure to obtain accurate information about Mr. Frederick was a 1 imitation..
~
of its methodology.
Persensky, Tr. 33,177.
Thisiis s'-
~
considerable understatement.
This matter concerns.us>for several reasons.
First, the Committee failed to assure the
,\\
j accuracy of the Special Report on this point.
The Committee did
()
not speak to Mr. Frederick about his status.
Tr. 32,087.
It also must not have made any other independent effort to assure the accuracy of its statement.
107.
Second, the Committee included this misleading information in the Special Report although it was apparently told by Dr. Coe at the beginning of its work about the status of Mr.
Frederick.
Uh r ig, Tr. 31,750, Coe, Tr. 32,354.
Thus, through oversight or otherwise, the Committee failed to assure that the report was accurate and failed to reflect in the report what the chairman himself considered to be significant negative information concerning the TMI-l training program.
108.
Third, although many Licensee personnel reviewed draf ts of the Special Report in which this language would have appeared,
()
they f ailed to correct the misimpression that the language creates.
Uhrig, Tr. 31,829-834, Long, Tr. 32,216-218, 253, Coe, Tr. 32,366-370,21 Leonard, Tr. 32,469, Newton, Tr. 3 2,4 62.
This 'is significant with respect to the Committee's work as a whole, but particularly with respect to the Committee's work on the Special Report.
In reaching the conclusions stated'in the Special Report, the Committee relied almost entirely upon i
21 We are quite disturbed by Dr. Coe's testimony on this point.- He asserts that the statement in the Special Report is not inaccurate because Mr. Frederick did take the courses mentioned in the statement.
Indeed, Dr. Coe considered the statement'_to be candid.
Tr. 32,366-371.
This astonishing testimony leads us _ to question Dr. Coe's judgment quite.
seriously.
If-he cannot understand why_the-statement is deceptive, we find that1we cannot give his -testimony -any '
.significant weight because it may be. based on such poor judgment.
We -had thought that Dr. Coe might ' serve an important
~
part as a new and' objective participant-in:the training program.
gS()-
We we re. wrong on that point.
C unverified information provided by Licensee.
Since Licensee,
()
given several opportunities, failed to correct these statements, the integrity of the entire Special Report is in question.
109.
Fourth, although the Committee reconvened in August, Uhrig, Tr. 31,963, it did not learn of Mr. Frederick's status until UCS raised the matter in Dr. Uhrig's deposition.
Committee, Tr. 31,958.
That deposition took place on October 23, 1984.
Uh r ig, Tr. 31,794.
The Committee filed its direct testimony, including the Special Report, on November 1, 1984, yet it still did not correct its statement.
We consider this omission to be peculiar at best.
Indeed, we think it inexcusable.
The fact that the Committee finally corrected the record at the beginning of its live testimony does not change our conclusion.
By then, the Licensee and the Committee knew that
()
UCS intended to pursue the matter an that they were better off revealing it themselves.
Moreover, although the Committee finally acknowledged that it should have noted that Mr. Frederick had failed his exam, it never acknowledged a need to re-assess its views in light of the special reliance it had explicitly placed in Mr. Frederick's place in the training program and his qualifications therefor.
This belies the Committee's claim that it was keenly aware of its obligation to correct any l
misunderstandings or errors in the Special Report after completing its first-hand review.
Committee, f f. ln. 31,749 at 25 and Uh r ig, Tr. 31,750.
110.
We emphasize, however, that we believe Licensee had an a f firmative responsibility to correct this language and to i
th
. assess the ef fect on its conclusions, both in the Special Report
(
that Licensee submitted to the Commission, and at the first opportunity in the testimony that accompanied the Special Re po r t.
We consider Licensee's failure to make this correction to be extremely serious and to reflect very badly on Licensee's judgment and its integrity.
We believe the Committee was negligent in this matter, as it was in reaching the conclusions l
l l
of the Special Report as a whole, but we do not believe that this matter raises any serious questions about the integrity of the members of the Committee.
111.
As this discussion suggests, this matter heightens the need to assure that the evaluation of the TMI-l training program was thorough, objective, and independent.
It also emphasizes the need to assure that the training program itself employs objective
()
criteria and does not depend too much upon subjective judgments.
j This is particularly true where Dr. Uhrig, the Chairman of the Committee, has publicly committed himself, through the AIF-Committees, to promote restart of TMI-1.
Stipulation concerning Dr. Uhrig.
Since his mind has.already made up on the ~ ultimate
~ issue, we have difficulty basing favorable conclusions on his or the Committee's subjective judgements.
112.. The members of the Reconstituted OARP Committee certainly have the backg round and qualifications to evaluate a training program and respond to ALAB-772.
They know as well as the intricacies, complexities, and inherent ' subjectivity
- anyone that characterize a training program such as this one. ~We also have no reason to doubt their personal-integrity.
(m
's_
113.
That is why we are so disappointed in the quality of
()
their work on the Special Report.
There is simply no serious case to be made for the proposition that the work that led to the Special Report was adequate to support the conclusions stated by the Committee.
Yet the Committee members allowed themselves to be pushed into a producing unqualified conclusions on the basis of which the Commission might well decide to authorize the restart of TMI-1.
They knew better.
In fact, they knew that if someone was going to rely on their report, there was going to be a need for verification of what they had said.
Uh r ig, Tr.
31,973, 32,022-023, 32,104.
They should simply have insisted that they could not complete a reasonable evaluation in time for the Commission's meeting.
The fact that they did not taints not only the Special Report, but their entire testimony.
In this, of all cases, it must be clear that the Commission's and this
(}
Board's decisions are based upon objective, factual information and upon evaluations that meet at least minimal standards.
The Special Report did not meet these standards.
d.
The methodology used by the Committee af ter the Special Re po r t.
f 114.
When the Committee issued the Special Report, it had no knowledge that it might be called upon to do any further work.
Uhrig, Tr. 31, 9 7 2, 32,102.
Although the Committee recognized that further work was necessary and appropriate, it did not recommend that it do anything further.
It lef t. that decision-entirely in'the hands of the Licensee.
Tr. 31,973, 32,103-104.
~
p This, again, is troubling because Licensee had already pushed the
()
Committee to issue the Special Report prematurely.
115.
In any event, the Committee was contacted again in mid-August and met at Three Mile Island beginning in August l
1984.
Uhrig, Tr. 31,963.
The Committee's purpose was now to develop testimony to present to this Board.
116.
There is no question that the Committee now began to do many of the things that it should have done before it issued the Special Report.
In the paragraphs that follow, we address the Committee's methodology with respect to the various substantive issues in this proceeding.
i.
Impact of Cheating 117.
The primary reason for this hearing is the f act that cheating in the TMI-l licensed operator training program
[ ()
following the Committee's original work may have altered the Committee's previous favorable opinions.
We expected, therefore, that the Committee would at least attempt some analysis of the cheating incidents in order to determine how they might af fect the Committee's views.
That is not what the Committee did.
Instead, it decided, in essence, that the best way that it could-address the issue today was to evaluate the training program as a whole, S.R. at 65-8, which is the reason that this hearing is necessarily broader than the Board might otherwise have ruled.
118.
To the extent that the Committee sought to address: this issue by evaluating the program as a whole, we discuss its methodology in'later sections of this decision.
At this' point, we. address the cheating issue more narrowly.
~
'T (3
x 119.
The Committee testified quite emphatically that it is
(
very difficult to predict whether people will cheat or even to understand why they have cheated.
Each case depends upon its own facts and is highly situational.
Christensen and Gardner, Tr.
32,033-034, 32,149.
Even students with very good records may cheat in some situations.
See e.g.,
Ch r istia nsen, Tr. 32,033.
This testimony was supported by Mr. Leona rd, whose experience at the Naval Academy was consistent with that conclusion.
Leona rd,
Tr. 3 3,4 0 6-4 07.
120.
The fact that even good students may cheat and that cheating is difficult to predict does not, however, resolve the question of why people cheat in general or why they cheated at TMI-1.
More important, it does not address the question of whether there was anything peculiar to the TMI-1 training program
()
at the time that might have contributed to cheating and that has not been changed.
~
121.
Dr. Regan suggests that people may cheat if they are concerned that they do not know the answers, or if they do not consider the examination to be relevant to their training or their job. Re g an, Tr. 32,771-772.
The OARP Committee agreed with the latter possibility and with-the possibility that cheating could relate to the trainee's perception of the fairness of the-examination.
Tr. 3 2,14 9.
The -OARP Committee could not preclude the. possibility that people might cheat.as a result of.
deficiencies in the training program itself.
Christensen, Tr.
32,140, Ga rdne r, Tr. '3 2,14 6.
Mr. Leonard similarly agreed that.
people may cheat out of"a-perception that they do not adequately know~ the material.
Tr. 33,413-414.
./~]
v W
v 122.
We conclude, with all of the witnesses, that cheating
(
is, indeed, highly complex and situational.
It may arise, as it did at the Naval Academy, out of mere convenience.
Leonard, Tr.
33,412-413.
And it may arise for the reasons discussed above.
We can see no way to determine what causes it without discussing the matter with those who actually cheated.
Only they are information capable, directly or indirectly, of providing the necessary to determine whether the cheating arose from deficiencies in the training program.
We agree with Dr.
Christensen that, "There probably should be an investigation of each individual case."
Tr, 32,165.
123.
The Committee, however, did not talk to any of those identified as having cheated.
Tr. 31,915.
In fact, the Committee did not attempt to determine. the cause of any specific (f
cheating' incident.
Tr. 31,916, 32,035-036.
Moreover, in the course of its original work, the Committee did not attempt to
. make any determination that the Operator Accelerated Retraining Program was adequate such that cheating should not occur, nor did it investigate any aspect of cheating in 1980.
Uhrig, Tr..
And the Committee did not-review.the exam'inations on l~
32,148.
I.
.which the cheating. occurred.
Uhrig, TTr. 3 2,150. -
124.
Despite the fact that-the Committee did not undertake such-an1 investigation, Dr. Uhrig was willing to state "the I
categorically what he, simply could not know, 'namely that, a
cheating had'nothing to do with the content of the training D
f program."
Tr.
31,918.
He'did so not only despite the11ack of an investigation of the' issue, but.also despite; the if act that l.
-pi
)
u_
the Committee undertook no efforts to determine how any aspects of the Licensee's training program at the time might have contributed to the cheating event.
Tr. 31,924.
Once again, the Board is disturbed by this willingness to reach conclusions f avorable to the Licensee without an adequate investigation or adequate information.
We consider this to reflect poor judgment l
on the part of Dr. Uhrig in this instance and on the part of the Committee in general.
125.
We conclude that the Committee's methodology for addressing one of the fundamental issues of this case is inadequate because the Committee made no effort to determine the causes of the cheating incidents and to relate those to the training program in existence at the time, or to the program that exists today.
()
11.
Effectiveness of Training 126.
The Committee's approach to the impact of cheating and to the remand as a whole was to address the effectiveness of the training program.
This is reflected not only in specific Committee assertions to this effect, Committee, ff. Tr. 31,749 at 6, but.also in the Committee's fundamental conclusion that the training program adequately prepares operators to run the' plant safely.
S.R. a t 8 3.
Thus, we must review the Committee's methodology for reaching that conclusion.. Mo re accura tely, since the Committee had already reached the _ conclusion, he must review the Committee's methodology _ for attempting to justify its prior
- conclusion.
p
c
! 127.
The Committee noted with approval the fact that the Licensee's examination process for reactor operators and senior reactor operators " includes written examinations, oral examinations, on-the-job evaluations, and simulator l
examinations.."
Committee, ff. Tr. 31,749 at 28-29.
As Mr.
Kelly explained, the combinations of different types of examinations is important because, "The dif ferent portions of the examinations as a complete package judge the ef fectiveness of the j.
training program and judge the effectiveness of the operators to il indicate what they have learned in the program."
Kelly, Tr. a t l
31,863.
In addition, the combination is important because each examination serves as a check on the other three.
Id.
1 128.
In light of the importance of the examination process, we conclude' that an adequate methodology would have to include a i
()
review of the. adequacy of the various examinations.
In fact,-
however, the Committee did not. evaluate or observe any oral examinatio'ns or any simulator examinations.- Ke lly, Tr.
31,864.22 Nor did any. member of the Committee observe any.
- Mr. Kelly testified that he evaluated the "results".of. oral 22-examinations.
.Tr.-31,864.
We areL hard pressed, however, to seel how this.could have been of any use. -The :only results that can.
be evaluated are checklists such as UCS Training Exhibits: 29 and-
-30, which may list subjects addressed, but.which do notfreveal-the exam questions. or answers.-
In. addition, 'Dr.1 Christensen recalls _ observing operators at the B&W simulator, but.he :does 'not. know whether he observed 'a training session orLan examination session.
Ch ristensen,, Tr.
niat'being the-case, he has'not provided us with any-31,865.-
usef ul information ' concerning ' the Committee's review of - simulator.
examinations.
In f act,- if he' cannot ' remember what he wa s is doubtful: that his. observation was so structured-
~
wa tching, -it as to.. result in a careful. evaluation.of. a : simulator examination..
LO y
_ _. l on-the-job training evaluations.
Kelly, Tr. 31,865, Committee,
()
Tr. 33,340.
Thus, the Committee failed to evaluate three of the 4
four types of examinations on which the validity of the TMI-l training program depends.
This is a serious inadequacy in the Committee's methodology.
129.
Oral examinations, in particular, are problematic.
As f
Dr. Regan explained, oral examinations are dif ficult to administer and peculiarly subject to extraneous influences such as the examiner's personal reaction to the examinee.
Thus, it is particularly important that those who administer oral examinations are adequately trained to do so, and they must be more than simply subject matter experts.
Regan, ff. Tr. 33,532 at 15-16.
No ne thele ss, the Committee relied solely upon their understanding that many of the personnel who administer oral
()
examinations are supervicory personnel with presumed subject matter expertise.
The Committee did not otherwise review the qualifications of-these individuals.
Tr. 31,869-871.
130.
It turns out' that Mr. Kelly' and Dr. Ga rdner did eventually undertake more review of the Licensee's written I
examinations.
Mr. Kelly, for example, evaluated all of the 1982 and :1983 requalification written examinations and results.
Kelly, Tr. 31,864-865.- The purpose of this review ' initially was '
-to address the scope and content of the examinations.- Kelly, Tr.-
31,882.
It was not until' November, af ter the testimony was filed, however, that Mr. Kelly' and Dr. Gardner togetherireviewed 1984 examinations :for such' purposes as, determining whether the examinations called for too much -memoriza tion.
Kelly,. Tr.
G )-
31,882.
As o f Oc tobe r 25, the date of his deposition, (and only
()
three working days before his testimony was filed) Dr. Ga rdner had given no more than a glance at the 1984 requalification examinations, and he could not at that point even state an opinion as to whether the Licensee's written examinations contained a proper balance between questions that call for rote memorization and those that require a demonstration of other types of concepts.
Tr. 31,891, 31,951..
We have no details on how the November review was conducted, but we do find, once again, that review of a major issue, one specifically identified by the Appeal Board, ALAB-772, 19 N. R.C. a t 1233, was not conducted until after the Committee had reached its conclusions.
In this case, the Committee reached its conclusions once in the Special Report and then again in its direct testimony, which was
()
filed on November 1.
In this circumstance, we find it hard to I
imagine that anyone would have made findings that contradicted l
the conclusions they had already published twice.
- Moreover, virutally all of the "first-hand" reviews done by the Committee were only done af ter UCS's interrogatories, depositions and comments to the Commission had emphasized the Committee's failure to gather _this specific information.
131.
Even this eventual post-testimony review did not include the type of quality assurance check necessary to determine whethe--the examinations were properly used.
The Committee did not check to see. if the grading 'was correct, and;it did not' check to see if the answers themselves were-correct.
Kelly, Tr. 31,884-885.
The written examination process could be seriously flawed, but the Committee would not know it.
E 132.
We conclude thattheCommittee'l,evaluationofthe
()
Licensee's examination process was inadec. ate.
The Committee ignored three of the four types of examiMstions altogether, and k
itsreviewofwrittenexaminationswasbflated, at best.
Since the Committee did review some written exkminations, we can give some weight to the Committee's conclusions based upon that review.
However, since the other three types of examinations serve as check on the written version, we are unable to give much weight to the Committee's view of the written examinations as supporting their ultimate conclusions.
133.
To the extent that the Committee reviewed the construction and preparation of written examinations, we may be able to give greater weight to its testimony on the subject.
In fact, Dr. Gardner gave a seminar on test construction at TMI
()
approximately two years ago, which he understands has been incorporated into the current instructor development program.
Tr. 31,879-880..
At that time, he observed.Dr. Knief present similar training on the construction of_ examination questions.
f Gardner, Tr. 31,881.
Since that seminar, however, neither he nor-l-
any other member of the Committee has observed Licensee's training in the construction of written examinations.- Ga rdne r,
Tr. 31,881.23 23 In addition, Dr. Gardner believes that training in test
-construction is a requirement that must be met before anyone may.
develop questions for an examination.
Tr. 31, 8 8 2.
In fact, that is not the case.
Under the procedures, any licensed operator may be -assigned to prepare examination questions, although Mr.
Leonard has chosen not to assign anyone wholis not an instructor.
Leona rd, Tr.- 32,495-497.
Even then, however, an exam writer may not have been. trained ~ in examination wr iting.
3rg Mr. Ma a g, for example, who is Supervisor-of Licensed Operator-(/
Training, has not completed the Instructor Development Program of f which this training is a part.
Leona rd, Tr. 32,4 83.
'. C; sz v,.
()
134.
In addition, neither FJ. Kelly nor other members of the Committee appear to have reviewed the examination bank from which questions are often drawn, nor do they know whether it is upda ted in any way other than through the process of constructing an actual examination.
Ke lly, Tr. 31,888-90.
135.
In light of these facts, the Committee's views on written examinations at TMI-l remain of little worth to us.
136.
The effectiveness of the TMI-l training program depends upon the quality of the job-task analyses and learning objectives on which the program is based.
There is no dispute on this point.
The Committee, however, failed to review the job-task analyses to assure their accuracy or adequacy, nor has the Committee compared the analyses to the training curriculum.
Kelly and Ga rdner, Tr. 31,94 8-9 50.
Thus, this Board can-hardly l
rely upon what the Committee might have to say about these
[
fundamental aspects of the training program.
137.
Whatever the quality of.the job-task analyses and learning objectives, they are of little worth'if current i
information is not incorporated into the lesson. plans that govern the actual. training.
Dr. Gardner purported to address this issue, but he admitted that he does not have the competence to do t
r so.
Tr. 31,944-945.
Mr. Kelly claims 'to have reviewed the lesson plans for this purpose.
Tr. 31,946.
It turns out, however, that. he actually reviewed the lesson plans - for the classes he attended against the ryerations Plant Ma' ual, not.
n against the plant itself.
'Tr.
31,948.
He appears to have viewed all the emergency procedures and ' ATOG procedures at some m
point, but as of November 5, 1984, the date of his deposition, he was unable to state on the basis of his personal knowledge that those procedures were up-to-date with the current plant design.
Tr. 31,948.
Other members of the Committee stated that they were given various assurances that mechanisms are in place to assure that training is kept current with the design, but none of them undertook an independent effort to verify that the mechanisms actually work.
Tr. 33,276-277, 33,287A-289.
And Mr. Kelly's further testimony indicates that the source of his opinion is not a specific review to update information, but the fact that as he did his reviews he recognized various information as updates,. and thus he concluded.that the material had been updated.
Tr.
33,277-278, 33,290-291.
138.
In addition to the job-task analyses and lesson plans-on which the training is based and-the examinations that seek to f
test the trainee's knowledge, the actual presentation.of the training and the quality of the instructors.are crucial matters.
that the Staf f and.Dr.
Regan have testified must be reviewed.
Licensee has not disputed this point.
139.
Af ter the Special-Report, Committee members did eventually observe a number of ' classes, although they overlapped in many classes, and some of the classes that they ob' served were not part of. the licensed operator training program.
See,-e.g.,
Tr. 31,896-898.
Most'of the classes attended by the Committee
- were given during -the August meeting at-Three Mile Island.
Others were given'in November,t after theiCommittee:had: filed its.
direct testimony.. Tr. 31,893-910.-
O
. In observing these classes,.
only Mr. Kelly appears to have had the TMI evaluation sheet available, but he did not fill it out, and he never reviewed the Licensee's use of its instructor evaluations.
Ke lly, Tr.
31,913-914.
Dr. Gardner did review some instructor evaleations, but there is no evidence that he compared them to performance that he observed or otherwise attempted to determine their accuracy, and he did not review these documents until November, once again after the direct testimony and after the Committee had already twice reached favorable conclusions as to the instructors.
Ga rdner, Tr. 31,913-914, 32,156-7.
140.
This record demonstrates that the Committee did see one or more instructors more than once, but the reviews appear barely adequate, if at all, in light of the need, demonstrated by the Staff, to monitor several instructors in several subject areas.
()
Finding No. 48.
Moreover, Dr. Regan testified to the inadequacies of evaluating instructors by simply observing their classroom performance.
Re g a n, Tr, 32,825.
According to Dr.
Regan, an evaluator should use some other indicator, such as the performance of those trained by a particular instructor, which will give a more objective indication of the ins truc tor's performance.
Id.
The Committee did nothing of the sort.
Moreover, the Committee did not even know whether the licensed operator training program is currently fully staffed.
Uhrig, Tr.
32,057.
141.
After it reconvened in August, the Committee interviewed a number of reactor operators and trainees, apparently in an attempr to determine attitudes toward (a'h
- = _ -. -
i ;
i training.
- See, e.g.,
Tr. 31,839.
As Dr. Regan testified, such
()
attitudes may serve as an intermediate measure of the effectiveness of a training program.
Regan, ff. Tr. 33,532 a t 14.
At the same time, however, it is clear that trainee opinions are of relatively little worth as direct indicators of the quality of training.
They are more likely to reflect the entertainment value of training rather than its substantive worth.
Re g a n, Tr. 32,773-778.
i' 142.
Dr. Gardner emphasized that attitudes are quite transient, Tr. 3 2,03 9, and Dr. Regan explained that the question i
is not so much what the attitudes are at any given point, but what the trends are over time.
Regan, f f. Tr. 3 3, 53 2 a t 14.
For these reasons in particular, we agree with the Staf f's testimony 4
that attitude interviews must be structured in a way that will
. ()
allow comparisons against other information that may be available on the subject.
Staf., ff.Tr.
33,148 at 15, 33,140.
143.
The Committee members did eventually conduct some interviews, although. as with several other actions taken by the J
Committee, a number of these occurred af ter the Committee filed-its direct. testimony.
Kelly, Tr. 31,844-845 (interviews after deposition, which was taken: November 5, 1984,1four days after direct testimony was filed).
Moreover,-noLmember of the 3
' Committee used'a written checklist to assure consistency, nor did the Committee' pattern its-interviews'after those done for the RHR Report or' for NUREG-0680,- Supp.
- 4.
Tr.-31,839-846.
5hus, we. a re a
unable to determine trends over time,.or even to know with e certainty what attitudes are today.'
O s
~
\\'j A -
, 144.
Undoubtedly, one of the major questions about operator
()
attitudes at Three Mile Island is operator reaction to the cheating incidents and to the actions taken by the Licensee in response to the cheating.
The Committee recognized the seriousness this matter both through its condemnation of the cheating itself, Tr. 31,929, and through its agreement that disciplinary actions may constitute an important signal to the troops.
Tr. 31,938.
Despite this recognition, the Committee did not discuss with any of the operators one of the significant developments with respect to disciplinary actions.
The Committee wa s awa re tha t Mr. H, one of the cheaters, had first been given a two-week suspension without pay, and had later received his pay when the Licensee decided to remove his operator's license and l
prevent him from becoming a reactor operator for TMI-1 if it ever l
i ()
reopened.
Tr. 31,931-937.
Since many other licensed operators who had not cheated, those from TMI-2, were ef fectively given the r
same punishment by being denied an opportunity to operate TMI-1, one might expect some negative reaction to the Licensee's trea tment of Mr.
H.
We have no evidence that the Committee made any attempt to determine the reaction.
145.
To the contrary, the Committee's reaction to the treatment of Mr. H was to decide that he was an example of someone who had been too harshly treated.
Mr..H's situation was the only evidence that the_ Committee could cite for its statement in the Special Report that the Licensee's disciplinary actions may have denied the company the services of qualified
(%,
V i
people.
Tr. 31,938-939.
We consider this to be slim evidence,
)
at best.
In fact, we find that the Committee's statement to this effect was absolutely unsupported and was sheer speculation.
146.
This matter of attitude interviews troubles us because it is central to ALAB-772, yet all of the information presented to us has been very general hearsay.
We believe that the Staff and Dr. Regan are correct in their assertions that attitudes should be determined by structured surveys with anonymous t
answers.
They are certainly correct that some sort of written record is necessary to have confidence in the results of attitude interviews.
The question of attitudes is simply too subtle and complex to be addressed as informally and subjectively as was the case.here.
For example, Mr. Kelly told those he interviewed that he was a member of the OARP Committee, which was preparing
()
testimony for these hearings.
Tr. 31,847.
We have no doubt that this sort of information can af fect either an operators actual attitude or the attitude that he expresses to an interviewer.
We are generally aware that the operators want very much to bring TMI-l back on line.
Thus, we would be astonished if any operator, with this introduction, would exhibit any but the most positive attitudes.
Dr. Ga rdner himself. testified that this information might wel'1 af fect operator attitudes, and that even -
his precautions may not have been adequate to detect the change.
Tr. 33,290.
147.
In the case of Mr. Kelly, there were some questions in specific areas, but the preponderance of the interviews were involved in general questions-to elicit general' feelings.
Tr.
r~s
31,848.
Essentially the same characteriza tion appears to apply to the interviews done by other members of the Committee, with some small effort to assure that the interviews covered the same general areas.
Tr. 31,839 (Kelly re: interviews in which he was joined by Dr. Ga rdner or Dr. Christensen) ;
Uhrig, Tr.
31,855-857; Ch ristensen, Tr. 31,860-861; Commi ttee, Tr.
32,062-069, 32,150-32,156.
We are constrained to agree with the Staff that the lack of structure to these interviews and the inability either to determine the results with any specificity or to compare the results to attitudes at other points in time is a serious inadequacy in the Committee's methodology.
148.
Th e RHR Re po r t, UCS Training Exhibit 6, was the subject of much discussion.
On its face, this is a report of a survey done by psychologists with expertise in the evaluation of
)
human attitudes.
Although the language of the report may at times be unclear, it must certainly be i'nterpreted overall as reflecting significant negative attitudes as of the time the survey was conducted in 1982.
There is also no doubt that the RHR Report is one of only two sets of data on operator attitudes at Three Mile Island.
The other is NUREG-0680, Supplement 4.
Staff, f f. Tr. a t 15.
Thus, whatever its validity at the time or today, the Committee should have reviewed it, particularly because it would have contributed to the determination of attitude trends, which all agree is more important -than the determination of attitudes at a particular point in time.
f)
\\.)
i 1
149.
According to Dr. Uhrig, the Committee first learned of
()
the existence of the RHR Report during the meeting at Three Mile Island on Ma y 30-June 1, 1984.
However, Licensee management did not brief the Committee on the report in any significant detail.
At that time the Committee did not believe the report was relevant to their work because it was concerned only with operator attitudes.
Although the Committee ultima tely recognized that operator attitude, at least toward the training program, is relevant to this proceeding, the Committee itself never met to discuss the contents of the RHR Report.
Tr. 32,038.
150.
While the Committee as a whole did not review the RHR Repor t, Mr. Kelly and Dr. Gardner eventually read it.
Id.
Mr.
Kelly first saw the RHR report in October 1984, after he had done most of the interviews in which he discussed operator attitudes.
()
Tr. 31,850-851.
At-that time, he did not ask to see the raw data on which the report was based, and he did not actually see the raw data until af ter UCS addressed the subject in his deposition.- Tr. 31,850-55.. HDr. Ga rdner also did not see the RHR Re po r t un t il Oc tobe r,. Tr. 33,296, spent very little ' time on it before his deposition, and did not look at the raw data until.
after his deposition.
Tr.
31,851-852.
151.
We recognize that for various reasons Dr. Gardner does not consider either the RHR Report or NUREG-0680, Supp. 4,; to be of great significance, nor does he think that the Committee's failure to review the:RHR Report reduced the value of the:Special Repo r t.
Tr. - 3 3,293-298.' He does accept both ' documents. as -
providing useful data' points.
Id.
(~v)
- 152.
Our question here is the adequacy of the Committee's
()
methodology.
Dr. Ga rdner's ultimate conclusions favorable to the Special Report and the training program provide us with little comfort.
Both RHR a nd NUREG-0680, Supp. 4, we re potentially important data points to be considered in the Committee's review of operator attitudes.
But the Committee did not even learn about them in any detail until October, and the Committee was not in a position to decide whether or not to consider them for the purpose of the Special Report.
Tr. 33,299.
By October, the Committee had done most of its interviews-for the purpose of addressing operator attitudes, so that it could not, prior to the preparation of its direct testimony, conduct interviews that addressed the issues raised by the RHR Report.
De spite - tha t inadequacy, the Committee issued its direct testimony, in which
()
it reiterated without qualification its-favorable opinions.about attitudes.
153.
We have here another situation in which the Committee reached its conclusions before it did the work necessary to support-_them.
The question 'of _ operator attitudes. is highly subjective even when careful.psychologicalisurveys'are_used.to' address it.. ~ He re, the: Committee'usedfonlyl general conversation, with no detailed structure and 'no significant written record. : We
.~do.not.believe'that.the;Committeelis-dishonest'in' expressing its:
- opinions _ on this subject, but we 'believe
- that -the inherent problem of ' subjectivity, coupled-with :the subconscious pressure of having.already published -favorable conclusions,: rendersithe
~
Committee's views of operator _ attitudes and.of the RHR Report; Luseless to our deliberations.
~
c.
4 _:
-e, g-
._ 154.
Finally, in connection with employee attitudes, one of
()
the specific issues raised by the Appeal Board was the attitudes of the instructors.
Although Committee members observed classes given by TMI-l instructors, they did not tap one of the most important sources of information on this subject.
They did not inquire of operators as to their perceptions of the attitudes of the instructors.
Tr. 32,065.
155.
Since the fundamental issue addressed by the Committee was the adeqr
't of training to prepare operators to run the plant safely, one of the major resources for their investigation
~
should have been reviews of operator performance.
We have no evidence, however, that the Committee undertook any such reviews.
- See, e.g., Kelly, Tr. 31,955-956, 33,340.
We do know, however, that the Committee was ignorant as to the evaluations 9[( )
done by the Company.
Although Mr. Kelly seems to be aware who evaluates operators, he believed that the Licensee did a written evaluation of the operators at some set interval.
Tr. 31,956.
Tha t is not true.
The union contract ef fectively prohibits 'such evaluations, according to Mr. Ross.
Tr. 33,419-422.
iii.
Propriety of Personnel Decisions 156. - One of the specific issues raised by the Appeal Board was the propriety of various personnel ~ decisions 'concerning, for example, Dr. Long and Mr. Frederick. _By its own admission,-the Committee'did not attempt to second-guess Licensee management's-decisions in this area.
S.R. at 16, Tr..
31,928.
Dr.[Uhrig and Mr. ~ Kimel.did eventually speak: to Mr. ' Clark, President of Licensee, on this subject, ~ but notLuntil. August 14 and November jy
.N
.. e
!) A
- /C
'\\\\
_ _ 8, respectively.
Id.,
and Tr. 33,282.
De spi te Mr. Frederick's perceived importance to the training program, as reflected in the Special Repor t, the Committee did not review hk. Fr ede r ick 's performance as Supervisor of Licensed Operator Training, and they did not interview him at length as part of their investigation.
Kelly, Tr. 31,96;, Uh r ig, Tr. 31,996-997, 32,096-098, Commi t tee,
Tr. 32,087.
The Committee did eventually speak to Mr. b2ag, who replaced Mr. Frederick, but not until August.
Tr. 32,098-100, 33,321.
157.
Since the Committee itself states, in ef fect, that it did not attempt to answer this question, we conclude that it has not provided us with any useful information on it.
Th e Committee's discussion of the qualifications of the various individuals is not significant to our decision on this point (j) because they are not what is seriously at issue.
The question is the propriety of these individuals being in these positions in light of their involvement in connection with the cheating incidents.
The Committee has not helped us on that point.
iv.
Adequacy of communications 158.
The Appeal Board expressed concern about the adequacy of communication between management and the reactor operators.
cite.
Unfortunately, the Committee did little to address this E
issue.
It did not, for example, attend any of the management interface meetings on which Licensee now relies as evidence of its improved communications.
Tr. 31,975-977.
Nor did it review any documentation of those meetings. I d,.
We also have no evidence that the Committee sought, in any of its interviews, to ggg determine whether particular communications from management had E
E effectively reached the operators or vice versa.
Thus, once
()
again, the Committee has not done what was necessary to assist us on this subject.
3.
Conclusions with respect to the OARP Committee 159.
As we have already said, we are disappointed in the Committee's work.
We believe that they are experts in their fields, but we are concerned that they may have been seduced by their own expertise.
In order to reach conclusions of value to this Board, the Committee needed to undertake a substantial amount of independent empirical work.
Whether or not they conducted their investigation precisely as recommended by Dr.
Regan or by the NRC Staf f, they had to know the facts before they could reach acceptable conclusions.
160.
When it issued the Special Report, the Committee'had
()
not done the necessary work.
Since many of the' issues in this hearing are plagued with subjectivity unless. care is taken to maximize objective information, it was incumbent upon the '
Committee, a f ter the Special Report, ' to minimize the subjective nature of its investigation.
Only then could we be confident that its conclusions are based upon the facts-and are not
- influenced, subconsciously or otherwise, by.the fact that the Committee had already published the Special-Report and presented it to the Commission to support a decision 1 favorable to restart.
161.
The Committee obtained virtually no objective information,'and it accepted and made assertions'that appear favorable to Licensee, but -that 'are either unsupportable or inappropriate.
The Committee testified, for example, that the GPUN training budget for 1984 was $7.6 million.
That seems impressive, but it covers the entire Training and Education Department and includes training at Oyster Creek and other facilities.
Tr. 32,056.
The Committee did not even know what i
the figure is for the TMI-1 licensed operator training program that is the subject of these hearings.
Id.
The particular point i
may be minor, but it is symptomatic of the Committee's carelessness in preparing its testimony.
162.
We cannot, on the basis of the Committee's testimony, determine the current attitudes of TMI-1 operators or trainees.
We cannot determine the impact of the cheating incidents or whether they were caused in part by an aspect of the training program at that time.
Thus, we cannot determine whether the
()
current program may be similarly flawed.
We cannot determine the adequacy of current communications between management and the operators, and we cannot determine the. propriety of management's various personnel decisions.
163.
Mo s t important, we cannot determine whether the training program adequately prepares operators to run the plant safely.
The Committee simply did not address this issue to any significant extent.
It came closest in its reviews of the wr itten-examinations, but its failure to review any other examinations, to review job performance itself, or to review the crucial job-task analyses render its favorable conclusions utterly without foundation.
O
l 88-l 164.
The Committee's response to various criticisms of its work is that it believes that the conclusions that it reached v
based upon briefings were validated by later information.
Tr.
32,022-024.
This is simply not an acceptable answer.
As we have j
discussed throughout this section, this approach taints the later investigatory effort and renders later conclusions useless unless the later investigation is highly objective, which this one demonstrably wa s not.
i 1
1 i
- O 4
l.
~
b l
r.-
-(.'..
a y
- - y r
s+
.ie g
y
--p n
--y f
w-u f
e
+he' O
B.
The Licensed Operator Training Program at TMI-l 1.
Or ganiza tion 165.
The TMI-l licensed operator training program that is J
the subject of this proceeding is part of the TMI Plant Training organiza tion,2 4 which is one of four sections of the GPUN Training and Education ("T&E") Department.
Th e T& E Depa r tmen t, which is headquartered in Parsippany, New Jersey, and not at Three bule Island, is, in turn, one of four departments of the NLclear Assurance Division.
The Nuclear Assurance Division
-is one of six divisions of the-corporation that provide support to(TMI-1, TMI-2,.and Oyster Creek.26 Th us, a s we would expect, the licensed operator training program itself is several layers-removed-from the_ corporate division that.is ultimately.
O responsible for its performance.
148.
In addition, the licensed operator' training program is theoretically independent of Plant Operations,- just 'as quality -
1 assurance must be independent of;the construction and operations I4 management under-Appendix B' to 10 C.F.R. Part = 50. - We consider L
i'
. ~.
24 - See Long' & Coe, ' f f. Tr..-32,202, J Attachment 5, for an l:
organiza tion. chart of the TMI Plant : Training Organization.
See id.,-. Attachment 4, ;for an organization chart of _the T&E :
. De pa r tme n t. -
i 6
.See id.,1 Attachment 3, f for ' an organiza tion ' char't ofl the.
Nuclear Assurance Division.:
F E
1
-J J) this independence to be important for the same reasons.
The licensed operator training program serves the same function with respect to the quality of personnel that the quality assurance department serves with respect to the quality of design, construction, and operation.
No doubt this is the reason that the training program is one part of the Nuclear Assurance Division.
167.
Dr. Robert L.
Long is a Vice President of GPU Nuclear and Director of the Nuclear Assurance Division.
The Board described Dr. Long's credentials in its initial management decision of August 1981, when Dr. Long was Director of Training and Education.
See LBP-81-32, supra, 14 N. R.C. a t 444 (4171);
see also Long & Coe ff. Tr. 32,202 a t 27-29.
His credentials reveal extensive experience in a variety of aspects of nuclear O'
energy, reactor operations, and education and training.
168.
As the Director of the Nuclear Assurance Division,-Dr.
Long spende approximately one third of his time with the Training _
and Education Department.
He is responsible for overall-4 management direction, guidance, and _ policy setting for. that.
department, as he is for the other three departments.that report to him.
Long, Tt. 32,203.
Not surprisingly, in that_ position he does not review licensed operator training programs!in' detail.
-Based on his testimony, and in light of his other responsibilities, we conclude that he, reviews these : programs :in -
O u
(}
only a relatively general way.27 Dr. Long has no direct involvement in the development of Licensee's examinations, and to his knowledge no examination questions have ever been changed as a result of his comments.
Tr. 32,205.
Dr. Long relies upon those who work for him for the development of the training programs, and necessarily also for the development of examinations.
Id.
169.
Richard P.
Coe.
Richard P.
Coe is Director of the Training and Education Department.
In that role, he joins three other departments in reporting to Dr. Lo ng.
Dr. Coe has extensive experience in the field of education.
When he joined Licensee, however, he had no experience in the field of nuclear energy.
Long & Coe, ff. Tr. 32,202, Statement of Qualifications.
He has no direct involvement in the development p),
of the course content for the licensed operator training program.
Coe, Tr. 32,218.
l 170.
Samuel L.
Newton.
Samuel L.
Newton is the Manager of Plant Training at TMI-1.
He is one of four managers who report to Dr. Coe.
He has twelve years of experience' in the nuclear navy and approximately 4 1/2 years in the TMI Training 27 Dr. Long disagreed with this' characterization when first asked about the matter on cross-examination.
Tr. 3 2,20 4.
He l
' agreed with it in his deposition, however, Tr. 32,205, and we are inclined to credit his ' statement at ' that time that he reviews the training programs "to basically verify.for [himself] that they are consistent with what [he] understand [s] the guidance has been to [ Licensee] and what [ Licensee's] commitments are."
Id..It is clear from this discussion that he does not independentlyfreview the accuracy or. completeness of the programs,.for= example, but' only their consistency with various guidelines.
)
()
Department.
He has most recently managed the installation and integration into the operator training program of the Ba sic Principles Training Simulator ("BPTS") and has prepared portions of the TMI training program for the application for INPO accreditation.
Lo ng & Coe, f f. Tr. 3 2,202 a t 33-34.
He performs an oversight function with respect to all training programs at Three Mile Island, including the licensed operator training program.
Tr. 32,412.
According to his own characterization, he is responsible, along with Mr. Ross, the bbnager of Plant Ope r a tions, for ultimate approval of the licensed operator training program.
Id.
Mr. Newton has not, however, had any experience at an operating commercial nuclear reactor.
Tr.
32,214-215.
171.
Bruce P.
Le ona rd.
Bruce P.
Leonard is the Operator Training Manager.
He is responsible for all operator training at TMI-1, including licensed operator training. - He - ha s -
t approximately six years of experience in the nuclear navy and two years in the TMI Training Department.
He does not, however, have-any experience at an operating commercial nuclear power plant.
Tr. 32,215.
172.
Ronald H. Ma ag.. Ronald H. Bhag wa s o recentlyf appo'inted to the position of Supervisor, Licensed Operator Training af ter-serving as Acting Supervisor since August 1984. Long, Tr.,
32,215-217.
Thus, he is the manager with direct -responsiblity for the program that is the subject of.this hearing.
For reasons.
[w Y:
()
of which we are not aware, Licensee chose not to present Mr. Maag as a witness, and we have had no opportunity to hear from him.
Moreover, Licensee relied primarily upon Dr. Long to testify to Mr. re ag's qualifica tions, although Dr. Long did not have the personal knowledge to do so and based his opinions on those of Mr. N wton and Mr. Le ona rd.
Tr. 32,217.
e 173.
Mr. Maag spent approximately eight and one-half years in the Navy, some of which was nuclear experience, and about three years in the TMI-l Operations Department.
He has been in the-operator Training subsection for about a half a year.
Long &
Coe, f f. Tr. 3 2,2 02 a t 3 6.
He has no experience at an operating commercial nuclear reactor.
Tr. 32,215.
174.
Dennis J.
Bo lt z.
Ef fective January 7, - 1985, Mr. _ Dennis :J. Boltz, previously Supervisor, Simulation Instr action, assumed the position of j
.Simula tory Development Manager. ~ Mr. Boltz has _approximately eight yea rs iof experience l in.the TMI-1(Operations, Department '
as an instructor and1 Supervisor.
His experience..inciddes aix s
7 years as a TMI control ! room' operator and ~18 months}as a TMI'
~
i-38;-
Operations shift foreman.- Long &. Coe,. f f.'; Tr. ' 3 2,20 2 a t l
Leona rd, - Tr. 3 2,4 88. ' Mr. Boltz has-been' involved with'the L
specifications, design,lacceptance.: testing', andl instructor :
training. for. the Ba sic.' Principles ? TrainingbSimulator, including
'~
- - approx'imately ?nine months ispen.t;in-esNentia' ly f ull-time!
l residenceLat the simulator manufacturer.
1 1 Longa &.Coe,;ff.?Tr..
/.
'3 2 '20 2 a t' 3 8 l(~l-s-
i-s w
I i <
(~}
175.
He r be r t J.
La pp, J r.
Mr. He rbert J.
Lapp began serving as Manager of Educational Development for GPU Nuclear on October 1, 1984.
He has approximately fifteen years of experience in public school and industrial education and training.
He holds the degrees of B.S.
in-Physics and M.A.
in Secondary Education.
From 1969 to 1980 he served as a high school science instructor, advancing to department chairman in science and math.
From 1980 until joining GPU Nuclear, he served in training supervisory positions with Commonwealth Edison's nuclear plant training programs.
Long &
Coe, f f. Tr. 3 2, 2 0 2 a t _ 3 9.
Mr. Lapp has experience in developing' performance based training and instructor development programs.
He has. also served as a peer evaluator on an INPO accreditation team.
176.
Michael J.
Ro s s.
Michael J.
Ross has served as Manager, Plant Operations, since January ~1978.
Although the Training Department is theoretically. organized independent of Opera tions, Mr. Ross 'is heavily _ involved 'in operator training.
- The Board sunmarized his credentials in its. initial management '
decision in August 1981~.
LBP-81-32, supra, 14 N.R.C. at-'439:
(4154).
He has over twenty years of experience Lin' nuclear power-f plant operations.and' supervision.
In our!. initial decision, we observed.that~mr. Rhoss: may. be the mostlimportant person - on _.the:
TMI-1 operating team as far as theLppblic; health and safetyfis, concerned.
Id. a t 4 3 9-4 0 - ( #15 5).- We'are now certain'.of that' conclusion,EalthoughT as discussed. below,1we derive e littleicomfort.
.from:it.
a 177.
Mr. Ross is a man of extraordinary influence at Three Mile Island.
He has by f ar the most commercial nuclear experience of any of the management personnel most relied upon by Licensee, more than anyone involved in licensed operator training.
According to Dr. Long, he is involved in essentially all decisions as to whether someone remains in the training program at a particular stage.
Indeed, whether or not someone is allowed to continue in the program is essentially a decision of the operations department, which makes it largely bk. Ross's decision.
Long, Tr. 32,214.
Mr. Hu k ill i s Mr. Ross's superior, but, with one er.traordinary exception, he has never overruled Mr.
Ross on such a question.
The exception is Mr. Hukill's decision not to certify Mr. Frederick to sit for another NRC SRO instructor examination after his failure.
UCS Training Exhibits O
1 and 3 4, Ro ss, Tr. 34,624.
As revealed by Mr. Hukill's wr itten recommendation, he reached that decision largely as a result of the fear that because Mr. Frederick's change of testimony on a separate issue had' raised questions about his veracity, his presence as head of licensed operator _ training might pose an obstacle to restart from the Staff's point of view.
UCS Training Exhibit 1.
Thus, it is clear that Mr. Ross' position is essentially controlling with respect to one of the most important' decisions the training department can make - whether tof dismiss a trainee for inadequate performance.' In other wo rds, Mr. Ross ef fectively establishes the minimum standards for performance.
If, as in the case of Mr. Olive,- Mr. Ross considers that the-company has made too great an investment to dismiss an individual qw
(
,m
I despite poor performance, Mr. Ross prevails.
Ro s s, Tr.
31,449-450.
178.
Despite his pervasive influence over licensed operator training, Fr. Ross has never taken the Instructor Development Program.
Nor has he taken any of the courses in that program.
He has no formal training in the administration of oral examinations, although he plays the major role in that area.
He has no training in the construction of written examinations.
Tr.
32,601.
179.
Board Concerns.
The Board believes that, as a general proposition, these key management personnel have an appropriate range of education.
For a number of reasons, however, we are not confident that these managers will assure the proper functioning of the licensed operator training program or, therefore, the n
safety of TMI-1.
180.
First, as with the OARP Committee, we are concerned about a question of carelessness and a willingness to view everything as favorable to plant opera' tion regardless of the facts.
The clearest example of this involves the OARP Committee's Special Report references to Mr. Frederick, which were seriously misleading as they were stated.
Dr. Long, Dr.
Coe, Mr. New ton, and Mr. Leonard all reviewed draf ts of the Special Report that must have contained the misleading language.
Their role was to assure the accuracy of the Co.mmittee's statements.
Although Dr. Coe believes he had informed the.
Committee of the status of Mr. Frederick, neither The nor' any of these other individuals recognized the. language as misleading or
!O
\\_ J informed the Committee of its error.
The result was that the Commission was presented with a report that, in the
'}
view of the Committee itself, contained a significant error.
For reasons that we have stated earlier in this decision, the provision of misleading information by licensees is absolutely intolerable.
In l'ight of this f ailing, we are constrained to view these individuals as developing their testimony with blinders on.
We do not believe that they are intentionally deceptive, but we are concerned that they see only what is favorable to their program unless they are presented with negatives in a way that they cannot ignore.
181.
We are also concerned with the imbalance of commercial nuclear experience among these managers.
The training chain of command runs from Long to Coe to Newton to Leonard to Faag.
Boltz and Lapp are involved, but only in specialized or secondary t'
ways.
In the chain of command, only Dr. Long has significant commercial experience, but he is clearly too removed from the daily operation and details.to incorporate his experience into the program effectively.
The man with the most experience is Mr.
Ross, but he is in operations, not training.
It-is, therefore, not surprising tha t Mr. Ross wields so much influence.
As discussed below -in greater detail, we believe Mr. Ross may1 wear even larger blinders than these gentlemen, and we seriously question his' judgment.
At this point, however, it suffices to note that this situation viol,ates the ' principle.tha t training, Jaa an aspect of quality assurance, must befindependent of operational control.
Tha t is not the case at TMI-1.
rM
. ()
2.
Impact of Cheating 182.
The specific issue of this remand is whether the cheating incidents indicate unacceptable weaknesses in the TMI-l licensed operator training program.
Were the deficiencies in operator testing symptomatic of more widespread deficiencies in the the training program?
ALAB-77 2, c ite.
If we could determine the precise causes of the cheating incidents, we might simply answer the question of whether those causes exist within the current licensed operator training program.
183.
As we have seen, the OARP Committee testified that cheating is a complex matter that is highly situational.
The cheating may arise from examinee concerns about the adequacy of' their knowledge, the relevance of the examination, the fairness of the examination, and presumably many other causes.
It would be necessary to review each particular case to determine what l
aspects of the training program, if any, contributed to the i
cheating.
Findings No. 119-122.
184.
Licensee presented Drs. Long-and Coe as its witness on.
-the impacts of and response to the cheating.
Long.& Coe, ff. Tr.
32,202.
Dr. Long testified that the Licensee began attempting.to determine the root causes of the cheating incidents as soon as it learned Lthey had occurred, and that the Licensee was reasonably confident in its findings by late 1982.
Tr. 3 2,2 0 6-207.
185.
It turns out, however,ithat Dr. Long.never ' interviewed the cheaters for the specific purpose' of determining whether any_-
aspect of the training program had' contributed to their cheating.
v
-. ~ _ _.
} ;
(}
Nor was he aware of whether anyone else had conducted such interviews.. In addition, he testified that the Licensee had h
conducted no specific study to determine the impact of the cheating on the non-cheaters.
Tr. 32,206-211.
Th us, we have no i
evidence either from those who actually cheated or from the 1
non-cheaters, who could at least have evaluated the situation f_
from the point of view of the trainees, as to what any aspects, 1
{
if any,.of the TMI-l training program contributed to the cheating l
incidents.
1.
186.
Dr. Long presented testimony as to his own sense of responsibility for the cheating incidents and as to the-management response.
Long, f f. Tr. 3 2,2 0 2 a t 2-12. - As - t'o Dr.
I Long's sense of responsibility, we can say only that he said what-i any-reasonably intelligent person would say about his involvement and responsibility.
His testimony could easily reflect his understanding.of what must be said in order to prevail in:this hearing as it could reflect a deep-felt sense of. personal 4
responsibility.
Indeed, he'may think he feels this-sense of t
t responsibility, when in fact he does not.
We a r e no t -
psychologists and are not trained to address these complex.
matters in any detail.
As judges,1m have some experience;in evaluating the. credibility Lof witnesses,-but this particu'lari
- testimony was not given orally.
We are unable on_this basis to. -
reach any conclusion in which we'can be. confident..
187.
Dr. Long's testimony as to management's response to the s
cheating establishes that managements primary goal-in - this regard-isitolattempt to improve ~communicatl,ns with the L
2
-p...s q
=me.
ge w
V
-100-(}
operators.
Dr. Coe added some details of improvements in examination security.
Id. at 19-25. These ef forts certainly I
should improve things.
We cannot tell, however, whether they are adequate, because we do not know what caused the cheating incidents in the first place.
188.
This situation strengthens our view, consistent with that of the OARP Committee, that the crucial question here must be the adequacy of the overall licensed operator training program.
We now turn to that issue.
3.
The Adequacy of the Licensed Operator Training Program a.
The Standard for Decision 189.
The fundamental issue in this case, as originally stated by this Board and af firmed by the Appeal Board, is whether
(
"the instruction [is] adequate to prepare the operators to operate the plant safely."
ALAB-772, 19 N. R.C. at 1232.
We have previously discussed the scope of this remand and' the f act that the Licensee has chosen to address the impact of cheating and the various specific issues identified by the. Appeal. Board by attempting to establish the adequacy of the training program as a whc.e.
We mu s t, therefore, identify and apply'the standards necessary to determine whether the training program adequately, prepares the operators to run the plant safely.
190.
Dr. Regan - presented the most comprehensive testimony on this1 point.
In orderito determine whether the training program prepares operators to perform successfully, we must assess the training prog' ram against the operational performance o
A v
=
-101-()
of the trainees.
Regan, ff. Tr. 33,532 a t 9.
To this end, we must determine (1) whether the reactor operator jobs have been adequately analyzed such that job performance can be measured against the necessary standards of performance, (2) what standards govern the determination of adequate performance, (3) whether adequate mechanisms exist to measure performance in the job and performance in training so that it is possible to compare the two.
Reg a n, ff. Tr. 33,532 at 6-9.
191.
The analysts of a job involves the preparation of job-task analyses, which analyze both the job as a whole and the various tasks and subtasks involved in the job in terms of the behaviors required of the incumbent.
Thus, not only must the analyses describe the actions required in the job, it must address the types of behavior that are involved in taking those actions.
The types of behavior include, for example, whether a task involves following procedures, perceptual-motor behavior, or cognitive behavior.
The task analysis must also address the difficulty of the task, the standards to which it must be done, the circumstances under which it must be performed,-its importance, and the stimulii and response that are involved in the task.
Regan, f f. Tr. 33,532 a t 6-7.
192.
Dr. Regan acknowledges that job performance can of ten-be dif ficult to measure, but he -emphasizes the need to do as welll as possible in order to assure that the job is properly designed.
and that -related training is ef fective. _ Measurement can-be as simple 'as'directly recording the number of products someone
. produces inJa day.
Regan, f f. Tr. 3 3, 5 3 2 a t : 9.
V' '
-102-(}
Unfortunately, it is clear that the job of reactor operator is far more complex than that.
193.
Dr. Regan identifies a number of measurement tools that apply to this situation, most of which he identifies as intermediate measures of the ef fectiveness of a training program.
They are intermediate measures because they lack the characteristic of allowing a direct comparison of training performance to job performance.
First, job performance may be measured through various tests, including job knowledge tests and performance tests using a simulator.
Second, the training program as a whole may be assessed through what Dr. Regan termed the process of formative evaluation.
In essence, this process involves the careful and continuous development _ and review of training through the use of job-task analyses.
Dr. Regan identified the Instructional Quality Inventory ("IQI") a s one example of a mechanism for applying this evaluation.
Third, Dr.
Regan suggested comparing the existing training program to the state-of-the-art, with particular reference to individualized teaching through computers and other mechanisms that provide an
~
understanding of a " mental model" of the entire system.
Fo ur th,
Dr. Regan identified the measurement of trainee attitudes as one-intermediate measure of a tra ining program, - although the question-here is not attitudes at a given time, but attitude trends over time, which'can be correlated with changes in the-training <
program.
Finally, Dr..Regan identified the use of performance-
~
ratings as one means of_ measuring training performance against job performance.- Rega n, _ f f. Tr. ' 3 3,5 3 2 a t 9-10, 13-14.
I'd v
Y
-103-194.
In addition to identifying each of the measurement tools discussed in the previous paragraph, Dr. Regan testified to a number of principles that must adhered to in applying those tools.
First, various performance measures, such as examinations, must be validated.
That is, one must determine whether the performance measures actually test what they are intended to test.
Where, as at Three Mile Island, the very purpose of the examinations is to determine whether trainees are capable of performing their intended. jobs, there is a premium on assuring that examinations serve as accurate predictors 'of job-performance.
Re g a n, f f. Tr. 3 3,53 2 a t ll,f l3, 15.
No one disputes this point.
Indeed, the OARP Committee approved the
. Licensee's use of four different types of job performance examinations because-the can serve to cross-check each other.
)-
Committee, Tr. 31, 8 6 2-8 6 3.
195.
Dr. Regan cautions,_ however, that it is necessary to instill' objectivity in the; examinations and in'the process of i
validating examinations..Thus, to the extent that_there may be i
practical _ limits on the use of statistical methods to validate examinations, it becomes that much more important to be: explicit-
~
~
and o'bjective in the use of other_ job performance measures such-as formative evaluation. : Rega n f f. 12'. 3 3,532 a t L 6. ' -In this i L
. connection, Dr. Regan also identified'many pitfallsii'n' reliance upon oral examination's.
.In essence,Lthey tend;to;1ack standardiza tion and specificity, thereby t allowing. egtraneous
~
influences such as personal: interaction Lto. influence the
, outcome.- Oral 1 examiners =must be.trainedLin thelart,fand'there -
~
t 9
4
-104-must be clear criteria for evaluating operator performance.
{}
Rega n, f f. Tr. 33,532 a t 15-16.
196.
Second, Dr. Regan characterizes the use of formative evaluation, or reliance upon careful development and continuous evaluation of the training program, as a useful tool, but one that does not serve to predict job performance.
It cannot substitute for an effective effort at comparing performance in training to performance on the job.
Tr. 32,785-786, 32,766-768, 32,823.
197.
Third, Dr. Regan testified that measurements of employee attitude must be undertaken through structured surveys, ideally with anonymous answers.
Regan f f. Tr. 3 3,5 3 2 a t 14.
Because statements of employee attitudes are strongly influenced by the questions that are asked, it is essential to use great O
care in developing those question.
Informal interviews or discussions provide no assurance that'the employees being interviewed are providing answers that can be compared to each other.
Regan, Tr. 32,823.
198.
Fourth, Dr. Regan emphasized that attempts to measure employee performance must involve more than simple ratings by superiors.
These ratings are inherently imprecise unless accompanied by specific behavioral' anchors, and.they are very unreliable measures of the adequacy of the training program..
Regan, f f. Tr. 3 3,53 2 a t 12. - He emphasized. further that. the small size of a program or the close. involvement of the various-participants with each other.does'not permit reliance on informal' j
ra tings. - To the contrary, this is precisely,the type xy IL L
i
-105-i
~
of situation in which it is vital to follow explicit, repeatable, state-of-the-art training procedures in order to assure that j
evaluations are objective rather than subjective.
He r e, there is an even greater danger than in large programs that extraneous considerations such as personal relationships will significantly 1
influence evaluations of trainees or job incumbents.
That is particularly true where those performing evaluations do so by i
virtue of their status as supervisory personnel and experts in the subject matter at issue.
These individuals are not well qualified to evaluate performance, which is why educational i
specialists and explicit, objective measures of performance must be used.
Re ga n, f f. Tr. 32,693.28
~
l 199.
We be.Lieve Dr. 'Regan has accurately-stated the 4
i standards that must govern a determination of whether the 'TMI-l O-training program adequately prepares operators to run the plant' f
safely.
With him, we do not apply.each principle rigidly, nor would we. be at a loss to reach a decision if praciticall limitations prevented us from applying some of the applicable tests.
Ra the r, the. question is whether, under all of these-principles, Licensee.has demonstrated that'its licensed operator training program is adequate.to protect the public health and~.
safety.
t 4
I
! ggc,3,,,. witnesses' Leonard and Knief ' argued L that t various; aspects of. Dr. Regan's Erecommendations are l impractical; due to : the ismall size of the program.;.Leonardj and Kn ief, 1Y., f f. 3 3,3 6 4 at.
'12-13.
While Dr. Reganf recognized that there may be - some
-practical 111mit'ations ito what.can be' done. with f a : small program, ^
(]
particularly in~ the use. ofistatisticalLtools, he emphasized _ that
\\-
this' enhances the:need for objectivity and;in the applicationJof-
~thoseimeasurement-tools that are available. -Regan, f f. Tr.
o 32,693 a t 6. '
-106-b.
Licensee Examinations
(}
200.
This remand arose directly from cheating on examinations.
Since they constitute Licensee's most direct attempt to measure performance in the training program, we turn first to whether they provide a suf ficient degree of assurance that operators are trained to run the plant safely.
201.
Licensee tests trainee performance through four different types of examinations.
First, at the end of the replacement program, and annually in the requalification' program, trainees are given comprehensive written examinationa.
- Second, at the same times trainees are given comprehensive oral examinations.
Third, each trainee must pass an on-the-job-training ("OJT") review.
Fourth, each trainee must pass a review of performance on the B&W simulator.
Leona rd, - f f.
Tr. 3 2,4 09 a t 8, 14, 20-25.
As we have noted, the: OARP Committee affirmed that a major purpose of having four-such examinations is so.that, if each-is properly constructed and-administered, they serve to cross-check each'other, and thus to provide some validation of the examination process.
Committee, Tr.
31,862-863.
202.
In7 essence, the basis for the Licensee's opinion that its examinations serve.to measure ' operator ability to run the plant is Licensee's view that.that the'examsfand questions are well constructed and. that the? content of the training program.is:
good.
Leonard, Tr. 3 2,551. 'We address-the 'latter. point. in :
e connection with Licensee's job-task analysis effort.
As-
~
Ediscussed below,'our review-of the issue of examination.
.(
,U r
/
V
-107-()
construction and other matters directly related to the written examinations leads us to conclude that we can have relatively little confidence in the validity of the wr itten examinations.
203.
Written examinations.
As Cperator Training Manager, Mr. Leonard has overall responsibility for the development of Licensee's written examinations.
His memorandum of January 27, 1984, governs the development of such examinations.
Leona rd, Tr.
32,491.
204.
As set out in that document, individuals designated as exam writers develop the questions based upon guidelines stated by 3k. Le ona rd.
Under the guideline memorandum, any licensed operator is qualified to be designated an exam writer, althoug h Mr. Leonard testified that it is his practice to assign only licensed operator instructors to that task.
Once assigned,. the exam writer is responsible to make the ' initial judgments, pursuant to the Leonard _ guidelines, as to the type and number ' of questions and the type of -skill or ability area that the questions will test (e.g., memoriza tion, as - opposed to. decision analysis), and:to assign point values to the questions.
- Leonard, Tr. 32,494-497,.499.
205.
Af ter the exam writer completes _his -terk, 'the examination is reviewed by an exam coordinator, whose fresponsibilities are primarily administrative.
HeLassures:that:
questions are submitted in time,.-that they are typed,.and,that; they are distributed up the: chain for. review.
The exam coordinator is responsible'for assuring that the questions have been coded for skill or' ability, but-he-is.not' responsible.for.
~
a
-108-(}
assuring that exam questions are coded correctly.
He also performs a quality assurance check.
The point of that check is not to assure the accuracy of answers to the exam questions, but simply to assure that the type of answer given by the exam writer is the type of answer called for by the question.
Thus, if a question calls for a one-line drawing of a system, the exam coordinator looks to see whether such a drawing is provided, but he does not check the accuracy of the drawing.
Similarly, the exam coordinator must determine whether the exam writer has assigned points to the questions, but he does not review whether the points are correct or incorrect.
Leona rd, Tr. 32,498-501.
206.' Following the exam coordinator, a technical reviewer is responsible for assuring that the questions are properly worded and technically correct.
He also must comply with all exam security requirements.
Le ona rd, Tr. 32,502-503.
207.
The final review of the examination is conducted by Mr.
Leonard.
Leonard, Tr. 32,503-504.
208.
As part of the exam construction process, exam writers may: rely upon an exam bank, which contains questions from previous examinations.
Since there is no process specifically _
intended to update and assure the accuracy of the questions in the exam bank, the propriety of'using these old questions is determined solely through the process of exam construction discussed in the preceeding paragraphs.
Leona rd, Tr.. 32,504-506.
209.
Although _the dif ficulty of an examination must have a bearing on its validity, Licensee does not code exam questions for difficulty.
Leonard, Tr. 32,506.
Thus, whether an v
1
-109-(}
examination is too easy, or perhaps even too difficult, depends upon whether it is correctly developed by the exam writer and is not the subject of a separate determination by the exam wr iter.
In addition, in light of the responsibilities of the technical reviewers as described by Mr. Le ona rd, there is no independent check of the overall difficulty of the examination beyond whatever the exam writer himself may do, but is apparently not required to do.
210.
We are concerned that a colloquy among Mr. Jordan, Mr.
Leonard, and Mr. Newton may reveal a serious deficiency in Licensee's written examination process.
In essence, Licensee strongly resists asking open-ended questions that seek the implications of particular events with respect to reactor safety.
According to Mr. Leonard, the problem with such a question is that, it "does not specify to the examinee a particular response."
Thus, the Licensee's question construction guidelines clearly are intended to result in questions that
" elicit a response" from the examinee.
Leonard and Newton, Tr.
32,506-513.
Since, in our view, one of the major contributors to the TMI-2 accident was operator f ailure to grasp everything ~ that
.might be happening, we find this emphasis upon narrow questions to be disturbing.
The examination should, to some extent at least, test operator ability to conceptualize broadly enough to address emergencies for which they have no specific procedures or-precedents.
211.
Another disturbing aspect of the exam construction process is the fact tha t FM. - Leonard appears to-serve as _ the v)
(
s-.-
}
i
-110-
{}
only significant reviewer.
Technical reviewers address only whether questions and answers are properly worded and technically correct.
Leonard, Tr. 32,502.
Thus, only Mr. Leonard reviews l
such questions as the proper balance of questions on the examination, the difficulty of the examination, and the point values assigned to particular questions.
212.
This situation concerns us for several reasons.
- First, Mr. Leonard's qualification statement reveals no significant education in educational psychology or other subject areas that might make him an expert in the field.
Le ona rd, ff. Tr. 32,409,.
As best we can tell, he has taken the examination i
construction seminars or materials offered by Licensee, and he has experience as an instructor.
Id.
Thus, while he has some competence in the area, we question whether he can adequately O
serve as the ultimate reviewer of Licensee examinations.
213.
Our concern is heightened by Mr. Leonard's testimony on cross-examination.
Under questioning by Mr. ; Jordan concerning recent Licensee examinations,12. Leonard testified that he had no problems with three examination questions ~ identified by Mr.
Jordan.
It turned out, however, that in-his earlier deposition l
he had criticized all of these questions for various reasons.
Leona rd, Tr. 32,513-514, 32,519-520, 32,523-524, 32,526-527.
With respect to another question, Mr. Leonard testified during the hearing that the question could be improved because the language of the question might well suggest the correct answer.
i this idea was As it turned out, however, a
f n.
-111-
{}
'not original with Mr. Leona r d.
To the contrary, it was suggested to him by Mr. Jordan during his deposition, in which Mr. Leont rd had testified that he had no problem with the question as
- wr itten.
Leonard, Tr. 32,518-519, 32,525-526.
In addition, during his deposition tk. Leonard incorrectly answered one of the questions that Mr. Jordan later raised during the hearing.
Tr.
32,514-515, 32,523-524, UCS training exhibit 31.
We consider this to be a relatively minor point, but it does indicate that we cannot rely heavily upon Mr. Leonard for technical review of the examinations.
214.
While Dr. Gardner and Mr. Kelly reviewed some examinations and reached favorable conclusions, it is not clear to us precisely what they reviewed or what issues are resolved by their conclusions.
We believe, for example, that their review O
does not assure that future examinations will be adequate if the examination construction process.is -inadequate.
Since the Committee did not conduct a quality assurance check of the exam construction process itself, we find that its-testimony provides only slight _ support for the adequacy of Licensee's written examinations.
~
-215.
Finally, we are taken aback that the Licensee has done nothing to address the specific--issue of_this remand as it relates to Licensee examinations.
Mr. ' Leonard could.not recall
~
any specific study' correlating the results on Licensee examinations with operator competence on-the job.- Le ona rd, Tr..
32,553.
O v
k
-112-4 216.
With respect to the Licensee's written examinations,
[}
therefore, we are left with very little on which to rely.
The examination construction process has few significant controls.
It has never even been checked by the Licensee to determine whether there is a correlation between exam grades and performance on the job.
We conclude, therefore, that the wr itten examinations may serve a useful purpose, but that they cannot be relied upon as predictors of operator performance unless some independent indicator, such as validation by.the other types of examinations, provides some basis for confidence that we do not now perceive.
217.
Oral Examinations.
Oral examinations are the second
_ major type of examination relied upon by Licensee-to test trainee knowledge and 'tio validate other types of examinations.-
Licensee O
gives a comprehensive oral examination-to each trainee at the end i
of each replacement program.
This consists of an examination on -
fundamentals given' by a Board of two to three people and a
~
one-on-one examination that involves a plant walk-through and covers plant systems operating; procedures.
Licensee gives a similar oral examination in each annual-requalification-cycle, except that theifundamentalsfare-incorporated into the one-on-one walk-through, and there is no Board examination. LLeona rd, Tr.
5' 32,532,
' 218.-
The oral Board examination given ~ under.the1 replacement t'ra'ining program is admininstered by representatives:of training-
- and. operations.
.Tr'.
32,535.
Th'e one-on-one Jora l f e xamina tion. may
- e
~
. q,
be administered by licensed operators or: licensed or,
.[
~
v y
F1 t
^
4_
+a g'.
-113-
{Jortifiedinstructors.
The mix of training personnel as opposed to operations personnel who administer the one-on-one examination may be anything from all of one to all of the other, so that operations personnel may administer many or even all of these examinations.
Leonard, Tr. 32,538.
In addition, Mr. Ross typically does another one-on-one walk-through examination, although this is not required by the applicable procedures. Ross, Tr. 32,535.
219.
The situation is different in the requalification program.
There, bk. Ross and other supervisory members of the operations department administer the oral walk-through, which, as noted above, now includes the fundamentals aspect of the examina tion.
Ro s s, Tr. 32,539.
We have no testimony as to whether training department 220.
s parsonnel who administer oral examinations have any special qualifications or training to do so.
We do know, however, that none of the operations personnel who administer ti ese examinations, including Mr. Ross, have any special training to do so.
Ra the r, Licensee relies upon their subject matter expertise in the areas in question.
- See, e.g.,
Ross, Tr. 3 2, 6 01-6 0 2, 32,615.
221.
Not only do the oral examiners have no particular training for.this complex task, but both the examinations themselves and the grading are highly subjective.: As.shown by UCS Training Exhibits 29 and 30, the oral examination check lists provide some rudimentary guidance as.to the subjects that are to bo covered in oral examinations.
They do'not, however
. f*)
\\_/
m.
A+om 4
-2,2e44JAa &
-u,
.-+A----ilm-'4----
AJ'-iAJ.-4A-,4
- m+u--4
_Jh
.4A.-4-.4
.a.
--._b,.-L.6A_ a D,,-.24a+.
+uhu_.-
a
-M J-+.+.44W14.s___-.
t
)
115-7didate could well achieve 60% correct answers, yet be found f
a j to have an 80% knowledge on the basis of the examiner's opinion of the situation.
In his deposition, MI. Ross characterized this
' as an " arbitrary judgment" on the part of the examiner.
On 4
i redirect examination, he sought to change that characterization
! to a " subjective" judgment Ro s s, Tr. 3 3,0 6 7.
W'e agree with his initial characterization.
The very fact that the judgment is so
, subjective, particularly when there is virtually no standardization of the examination itself, renders the ultimate j grading decision an extremely arbitrary judgment.
i J.
224.
This problem is compounded by the fact that there is no I, review of the oral examinations.
The supervisor of licensed
, operator. training checks the documentation, but that is nothing moes than a bare checklist.
Tr. 32,540-546, UCS training Exhibit 0
30.
Although Mr. Ross presented conflicting testimony on the subject, it appears that no one reviewed the questions or answers with either the examiner or the examinee.-.Ross Tr. 32,'602-608.
225.
Despite this subjectivity,12. Ross testified ethat he
- baliev'es that oral examinations'are'very useful11n predicting.
' leter performance \\on the job.. The1 basis.for this. opinion -is largalyftheLfactJthat there11s a unique opportunity to'go into
. tha plant and see if the candidate truely has an understanding off
.tht cquipment.
In additionfit;is_a unique' opportunity-.to intorrogate the candidate in depth.and determine the: full, extent
' of-his knowledge.. Ro s s, Tr. 133,067-068.
D N
a-
.n--
1
-118-Thus, Licensee does not know whether those who are graded O
marginal on oral examinations perform better or worse than those who pass with flying colors.
Le ona rd, Tr. 33,455.
Certainly if Licensee has never seen fit to determine whether there is any j
such correlation, we cannot do so here.
We conclude that Licensee's oral examinations provide no basis for a finding that i
operators are adequately trained to operate TMI-1.
230.
OJT Examinations.
The on-the-job-training examinations are administered during that phase of the training through oral checkouts by operations personnel and, possibly, by a very small number of training personnel.
Se e, e. g., Le ona rd, f f. Tr. 3 2, 4 09 a t 8, Tr. 3 2, 6 3 9-6 41.
The operations personnel include shift supervisors and shift foremen.
Leonard and Ross, Tr.
32,473-474.
These individuals are not required to go through the Instructor Development Program, and, as we have already seen, they have no specific training in the administration of oral-j examinations.
As Mr. Leonard testified, their qualification to administer the oral examinations in the OJT program.is their licensed status, their experience at the plant, and their familiarity with oral examinations.
Tr. 32,475-477.-
231.
We have relatively little information on this subject, but-what information we do have gives us no basis'-for confidence that the OJT checkcets provide the validation ~ that, a s ye t,- i s.
lacking from the Licensee's examination process.
Ag a in, the se are ora'-examinations by supervisory ~ personnel who are not trained in giving. oral examinations.' All of the problems
-119-identified by Dr. Regan exist here as well.
We suspect, but have no record evidence to support the proposition, that these may be somewhat narrower, more standardized examination than the comprehensive orals.
If so, that would alleviate some of the problem.
On the other hand, this would also indicate that the OJT examinations are considerably narrower in scope and could not serve to validate the others in any significant way.
In any event, we have no such evidence.
In addition, we have no evidence that Licensee has made any effort to correlate performance on the OJT examinations with job performance.
Certainly if the Licensee will not do it, we cannot do it.
Accordingly, we are forced to conclude that the OJT examinations do not provide any basis for concluding that the licensed operator training program adequately trains operators to run Os
\\-
TMI-1.
232.
Simulator Examinations.
Training at the B&W-simulator in the replacement program is supervised, on behalf of' Licensee, by a Senior Reactor Operator who is responsible for evaluating both the operators and the instructors on their performance.- The SRO could be a member of either the operations ore the ' training depar tments, but in the past.he has been from-Operations.
As a f
member of the operations staff, he would not have taken the
(*)%
L.
s
-120-30 instructor development course, and his qualifications to serve in this ro2e would be essentially his subject matter expertise, as well as his ability to communicte well with the operators.
Leonard and Ross, Tr. 32,477-479.
23'3.
The requalification simulator training program is different from the replacement program in that more senior operations personnel such as Mr. Ross are normally in the supervisory position.
Ro s s, Tr. 32,480-481.
In fact, Mr. Ross gives the majority of crew demonstrations and examinations at the simulator.
Ro s s, Tr. 32,619.
234.
The limitations that apply to Licensee's simulator examinations are quite simular to those that apply to the oral and OJT examinations.
These are, in essence, subjective evaluations of employee performance.
They are carried out by supervisory personnel whom we know to have no training in the administration of oral examinations, which appear to have some 30 The transcript reference to this point is Tr. 32,477, lines 19-24.
As the transcript appears, the question on lines 22-23 states, " And assumably that person would have had the ins tr uc tor,
development course?" The.answe r is "If it is not an f instructor, that may be the case."
This language does not make sense because it' is the instructors who would have had the course.. We conclude, therefore, that the transcript of the question must be in error, and the answer given was to the question, "And assumably that person would not have had the instructor development course?"
,r'r.
k-
1
-121-
{}
similarities with the reviews of simulator performance.31
)
Unfortuately, Licensee has not adequately explained precisely what happens during a simulator examination so that we can i
determine whether it is sufficiently standardized and controlled l
to avoid the pitfalls that afflict other Licensee examinations.
i i
235.
We conclude, therefore, that while the simulator evaluations could considerably improve the situation, Licensee i
{.
has not adequately shown that to be the case.
We cannot
^ determine whether those who administer these examinations, I
particularly Mr. Ross, are qualified as examiners, as opposed to qualified as subject matter experts.
We also cannot tell whether the examinations are adequately controlled and the results
. adequately recorded so that they can contribute to validating the examination process.as a whole.
236.
Conclusion.-
We conclude that Licensee. has established an examination procedure that could theoretically provide
~
assurance, in connection with others aspects of'a sound program, that the training program will adequately prepare operators to.
We note - tha t f Mr. Ross wa s. invited - by the NRC Sta f f to.
participate in a ' forum - on how to give simulator examinations. -
Tr. 33,064.
The letter to him from the Staff on the. subject.
referred to hisLexperience and' interest =in'theJaimulator examinations.
Tr. :.3 3,0 66.
.He 'was one among representatives ' f rom somewhat less than twenty utilities.
'Tr. - 33,449.
We can.
Jconclude from this incident only _that Mr. Ross may,have had some experience that-the. Staff thon9ht useful-to.the: discussion of simulator examinations. 1In' fact, however, ;we cannot even go that'.
z far because' the Staf f may have believed:it1 appropriate for: Mr. -
Ross to attend so that his ' knowledge of the area.might be-improved. L on - this record, we. cannotidetermine 'whether this
? incident establishes that: Mr. Ross has any' expertisenin the.
administration of. simulator. examinations.
+
E
+
m
(
,4 g
1w g f-4f'we' N ut 7
-*-r-pv9 P7W-t' 4
eW d-
- ' -w4--
-122-(}
run the plant safely.
We agree with the OARP Committee that Licensee's use of four types of examinations is appropriate.
The problem is that the various examinations, as they are currently implemented, are not reliable either individually or collectively and cannot serve to validate the program.
237.
We find that Licensee's controls over the construction of the written examinations are inadequate.
The construction process does not include either the quality controls necessary to assure that examinations include a proper balance of questions with respect to subject matter, type of question, or difficulty, nor does it include personnel with the training expertise necessary to implement such quality controls.
238.
We find that Licensee's oral examination process is wholly inadequate.
Those who administer the examinations, on l
l whose abilities the exams depend, have no training in the administration of oral examinations, and there is no effective i
review of their work.
There is no a ttempt to standardize the examinations or to address any of the other concerns raised by Dr. Re g a n..
239.
We find that similar inadequac.as exist with respect to the OJT and simulator examinations.
Those examinations do.not, i
l however, appear to be as seriously flawed as the oral examinations, primarily because the subjects of these examinations are by their nature more clearly defined and limited.
240. ' Finally, whatever the adequacy _of the process forfeach of these types of examinations, we have no evidence lof how-
+)
performanecion these-examinations correlates with performance on
~
'%/
-123-J
/~T the job.
The simulator examinations, except to the extent that V
they are compromised for reasons discussed above, may provide some job performance benchmark.
Once again, however, we have no reliable evidence that good performance on the other examinations correlates with good performance on the simulator, or that there is any other correlation that might be of use.
Accordingly, we must conclude that we have no basis for relying upon Licensee's examination process to support a decision that Licensee's licensed operator training program will adequately prepare operators to run the plant safely.
c.
Program Development and Implementation 241.
Licensee's argument in support of the adequacy of the licensed operator training program is based primarily on the protosition that Licensee has carefully developed and is now e
impl(menting a training program that is performance based, as l
reflected in the job-task analyses and learning objectives on which the program now depends.
- See, e.g.,
Leonard, ff. Tr.
32,409 at 29-36.
Three aspects of the program are central to its success ani reveal serious flaws in the-program.
They are the development of the job-task analyses themselves, the question of upda ting anc ' _ revising the training, and the adequacy of.
instruction ~echniques.
We address these in turn below.
242.
Jot-Task Analyses and Program Development.
Mr. Leona rd explained Lictnsee's reliance upon what it terms the Training
. System Develo1 ment -("TSD") model that Lic.asee itself has
=devaloped,'which involves, _in large part,. the development of l
% >)
-124-job-task analyses on which the training program will ultimately be based.
Leonard, ff. Tr. 32,409 at 29-36.
Licensee began implementing this model using informal generic job-task analyses that had been or were in the process of being developed by INPO and that had been published by the NRC in NUfEG/CR-1750.
Licensee also tried to follow INFO guidelines for the development of a training program.
Id,., Leonard & Knie f, ff. Tr. 33,364 at 6-7, Tr. 33,376-377.
243.
While these early efforts may have been helpful, they were clearly not as extensive as envisioned, for example, by Dr.
Regan in his direct testimony, Regan f f. Tr. 33,532 a t The table-top analyses were informal in nature, Tr. 32,457-459, while the NUREG-1750 job-task analyses were extremely general, having been developed to apply to several vendors and several designs.
O Tr. 32,460.
Thus,. they were of little use unless Licensee added the details of its own jobs and systems.
Finally, the INPO Guidelines on which Licensee now relies'were not specific at all, but appear to have been nothing more than a generalized format for a sound training program.
Tr. 33,461-62.
We have no. doubt
[
-that Licensee could produce a sound format for a training program.
The issue here is whether, assuming the format to be sound the program itself is also sound., That' is where we have our doubts.
244.
As of the time of this hearing, Licensee was stillL a:-
- 1ong way from having completed'its' specific job-task' analyses.
It had analyzed the various jobs, and it had broken the jobs down into lists of hundreds of tasks,; but-it had not yet undertaken
. (3 V
-125-
{}
the analysis of the tasks into subtasks.
Tr. 32,454-456.
245.
At this point, there appears to be a general schedule to complete the job-task analyses before the next replacement class in the summer of 1985.
We are not certain, however, whether this is a firm schedule to which Licensee is committed, or whether it is simply a hope.
When asked about the schedule in j
his deposition, FT. Leonard responded that there was no corporate schedule.
Mr. Newton later testified that such a schedule had been set in mid-October, which wa s before Mr. Leonard's deposition.
Leonard and Newton, Tr. 3 2,54 6-5 51.
Since Mr.
Leonard is clearly closely involved in this ef fort, we would expect him to have known of such a schedule.
Thus, we conclude either that this is not a particularly serious schedule, or that communication within the training department concerning such important matters is grossly inadequate.
l
'246.
At this point, we can say that Licensee appears to be on the right track.
Dr. Regan appears to agree that the Training System Development ~ model bein' g.used by Licensee,. although not a.
substitute for comparing _ job performance to training performance, could be usefull if properly implemented.
Re ga n, -- f f.1Tr.' 3 2, 6 9 5 '
I at 3-5, 32,823.
The question,. then, is implementation.
The OARP Committee's review, which did not even include examination of, the crucial job-task analyses, was 'too superficial to :be of :any. help-to us on this subject.
Dr. Long ano.Dr. Knief have:some.
expertise in the area, but without independentrevidence :of the' adequacy'of their. work, weL areinot convince'd that)their expertise
~
or involvement are sufficient to assure successful inplementation C
-126-of the TSD model.
As Dr. Regan explained, simply invoking such a procedure does not assure that the training program is adequate.
We need evidence both that those who use the model are adequately trained to do so and that the model is being implemented correctly.
We have no direct evidence of the latter beyond the assertions of Licensee personnel.
With respect to the question of Licensee expertise, we find that Dr. Long is too f ar removed from the training process to allow reliance upon his expertise to assure success in this crucial area.
247.
The question, then, is whether Dr. Knief provides adequate expertise to assure success.
In response to Dr. Knief's and Mr. Leonard's discussion of the implementation of the TSD model, Dr. Regan raised essentially that question.
The TSD model involves a linear process in which the adequacy of later steps O
cannot be determined if the 2arlier steps were not adequate.
Re g a n, ff. Tr. 32,693, 3-5.
His concern was based upon the fact that Dr. Knief and Mr. Leonard had testified that the development and implementation phases- (steps' 3 and 4)_ of the approach were conducted effectively, but that the analysis, design, and evaluation phases (steps 1, 2, and 5) needed wo rk.
Id,.
- Indeed, it seems obvious that one could not-determine the adequacy of the development and implementation phases without having been -through the' underlying analysis.and-design phases of the work.
248.
Dr. Knief sought' to respond to this. concern 'by
-explaining how he'h'ad gone about teaching <a reactor. theory courseL
-without having had the time toido a front-end analysis of the.
pb
% m_5
}
-127-course material.
He then compared this experience to implementation of the TSD model, emphasizing that the key element in both cases is the evaluation phase, which he could still do to some extent, and on which he relied, for example, in improving his reactor theory course the second time around.
Knie f, Tr.
33,380-381.
249.
We are not convinced that this answe rs Dr. Re g a n 's concerns.
Obviously, it may be necessary to continue a training program was well as one can without stopping for a thorough redesign.
That is practical reality.
To use Dr. Knief's example, it is possible that one might teach the course quite well despite this handicap, or one might not.
It seems clear, however, that without completing the analysis and design phases of the course first, one could not really form a sound opinion as to the adequacy of the development and implementation. phases.
Yet that is what Dr. Knief testifies Licensee did when it began attempting to apply these principles.
Leonard & Knief, ff. Tr.
33,364 at 6.
We a r e le f t, therefore, with the' impression that Licensee is willing to determine the adequacy of various aspects of.its training program without having-done the initial' analytical and design work on which such a determination must be based.
.250.
If all other aspects of the training program were well designed and well implemented, we might - be able to consider this -
to be a minor' problem.
We cannot do so.
In light of. the
(~)N
\\_
i
-128-deficiencies that we have found in other aspects of Licensee's progr.;a, Licensee's reliance upon the TSD model, the development i
of job-task analyses, and learning objectives are crucial to Licensee's case.
Particularly in the absence of direct, independent evidence of the adequacy of Licensee's job-task analyses and its implementation of the TSD model, we are unable l
to resolve our concern about Licensee's expertise in this area.
When this concern is coupled with the fact that, as Dr. Regan explained, a model such as the TSD cannot serve as a substitute for correlating training performance with job performance, Reg a n,
Tr. 32,823, we are unable to rely upon Licensee's development and implementation of its training program as a basis for concluding that operators are adequately trained to run the plant safely.
251.
Updating The Training.
We have two concerns with respect, to the updating of the material in the licensed operator training program.
First, is the material updated and are operators trained and tested on it in a timely f ashion?
- Second, what is the effect on reactor safety o'f the' fact that material is frequently updated so that operators must be frequently. retrained in the same areas?
T 252.
Licensee appears to have developed a mechanism that should assure that training material is updated to remain current with the plant design.
In essence, new developments _ are incorporated into the Operations Plant Manual, which then serves n
I b
LJ
.A
-129-as the source for training material.
Tr. 32,908.
Licensee seeks to assure that the Fbnual itself is updated by assigning various portions of the Funual and related procedures to an " owner," who is responsible for keeping the materials up to date.
Licensee has also established a 42 step checkoff process by which it seeks to assure that new material is appropriately handled.
In addition, the operations department provides updates to the operators when necessary, and those updates are provided to the training department, which then decides whether and when the material should be incorporated into the training program.
Tr.
32,907-923, 33,080-081.
253.
Despite these controls, we are unable to conclude that the training materials and the knowledge of the operators, are ef fectively updated in a timely manner..First, the _upda te O
procedure seems to be extremely cumbersome.
With 42 steps, it must take a considerable amount of time to implement.
Second, it appears from the testimony that changes may'either be incorporated into the plant or even_found in the plant before they have been incorporated into the training program or even explained to the operators.
Third, while the updates _ distributed by the operations department at least inform operators of ~ new.
l l
. developments, Licensee has failed'to describe any_ mechanism.
- through which it assures that operators understand: the changes' and will take appropriate actions.. We assume that such information would be included in the next requalification
! training program, in which L operators would. eventually be tested. -
The problem -is that, this testing could be months away,
~
-Q
\\<
~
-130-and Licensee has no interim mechanism to assure that operators are competent to act on the basis of the new information.
254.
Our second concern was identified by Dr. Regan and has been confirmed by Licensee's testimony.
Dr. Regan testified that interference problems may arise where incumbents and trainees are subject to frequent changes in procedures and requirements.
In these situations, prior learning and experience may significantly inhibit both initial retention and learning of new material.
Th e problem can become particularly acute in emergency situations, when an operator may tend to revert to previous procedures.
Regan, ff. Tr. 3 3,532 a t 20.
It goes without saying that this could result in disaster at a nuclear reactor.
255.
The record of this proceeding provides us with an excellent example of the situation in which this problem may arise.
The steam generator procedures at TMI-l have undergone considerable change over the past several years, a process that continues even today.
The greatest changes appear to have occurred prior to January 1983.
At that point, Licensee decided to start incorporating the changes in to.the training program.
The changes continued, however, and it was.necessary to train and retrain the operators several. times as these changes occurred.
The' first~ major period of continuous training and retraining l
l appears to have been January t May or June 1983, which is a -
x_/
. {-
Up l' i
\\
-131-
)
short period of time in which to be trained and retrained several times on changing material.
Leonard & Ro s s, Tr. 32,855-859, 33,415-419.
256.
We have no evidence that Licensee undertakes any effort to address the problem of interference that may be caused by training and retraining as procedures change.
We suspect that use of the simulator may help in this area since it can address responses to emergency situations, but we have no basis for concluding that Licensee attempts to use it specifically for that i
purpose or that the use Licensee makes of the simulator will address the problem.
We conclude, therefore, that the licensed operator training program is inadequate to the extent that it does not include provisions to assure that the need for retraining based upon changes in the plant and procedures does not result in learning-interference that inhibits the operators' t
ability to respond correctly in emergencies.
257.
Adequacy of instruction techniques.
One of the means that Dr. Regan identified to help evaluate the adequacy of a training program is to compare the program to the state-of-the-art.
He emphasized, Lfor example, that a program of this sort and this size, particularly one.that involves training adults, should include ' individualized instruction both through the use of computers and through the use of instructors for thet-purpose.
He cautioned.against the use of what he termed
" platform instructors," which are essentially classroom teachers in the' familiar time-worn mold.
Re g a n, Tr. a t ' 32,787-788.
~
bl s_
.j
-132-258.
This training program appears to rely very heavily on platform instruction.
Even after Dr. Regan's testimony, Licensee gave us no details of the use of instructors for individualized training, although we expect there may be some such training done in upgrade programs prepared for candidates or operators who are not performing well in the training.
There seems to be some use of computers, particularly if we include use of the simulator,
~
but Licensee has provided us with no information on how extensively they are used or how they provide individualized instruction at the level of knowledge and learning of each operator, which Dr. Regan identified as an important consideration.
Regan, ff. Tr. 33,532 a t 5-6.
259.
This situation does not, by-itself, lead us to conclude that the training program is inadequate.
At the same time,
(
however, it is clear that the program is hardly'at the 1
state-of-the-art, and thus this information is of little help to us in addressing the basic question of whether the program adequately trains operators to run the' plant safely.
W U
-133-O d.
Trainee and Operator Selection and Evaluation.
260.
As Dr. Regan's testimony demonstrates, the evaluation of trainees and operators is crucial to the success of a training program.
It is also crucial to our charge, since we are to determine whether this training program adequately prepares the operators to operate the plant safely.
As we have said, the i
training program serves as the quality control for personnel selection.
Thus, the adequacy of the various evaluations carried out by the training program are vital to its success.
That is why, for example, all parties have placed such emphasis on Licensee's examination process.
Similarly, the adequacy of evaluations of later job performance is essential to the ability to correlate performance in training with performance on'the job, and thus to answer the very question 'now before us.
261.
One of the issues in this remand is whether the training program teaches trainees to operate the facility, or whether instead.it teaches them to pass examinations.
ALAB-772,
.19 N. R.C. a t 123 3.
The most significant evide'nce' we have' on this issue relates not so much to precisely how the material isitaught.
or how the examinations are constructed,- but to the question of:
'whether the Licensee insists upon continuing to; train; people with bad records until,. at long last, they are able to passuthe
~
~
examinations, which we : suspect most people could do if given.
.enough time and prompting.
4 u
-. -. =
-134-l 262.
Mr. Leonard testified that the candidates for the
(
4 replacement program for both ROs and SROs are selected by the Manager of Plant Operations, bk. Ro s s.
Leonard, f f. Tr. 32,409 at 3, 11, Tr, 32,471-472.
Mr. Leonard also explained that Mr.
.l Ross is involved in what appears to be every decision concerning the actions to be taken with respect to a candidate in the requalification program who fails one of the final examinations.
Le ona rd, ff. Tr. 32,409 at 7, 8, 14.
Mr. Ross himself confirmed his involvement in decisions concerning what should be done about trainees who perform poorly in the training program.
Ro s s, Tr.
32,593-596.
In that testimony, Mr. Ross also explained the criteria that he applies to these decisions, which essent'ially involve his subjective judgment as to the ability and desire of the individual in question.
Id.
263.
We believe that Mr. Ross was too modest when-he protested it would be. too strong a statement. to suggest. that his views in these matters'are controlling. 1Ro s s, Tr. 32,'595.
Mr.
.Ross himself admits that he can think of no ~ situations in which q
he would.not control the decision.
Ross,~ Tr. 3 2,5 9 4.
Mr. ? Hu kill :
testified to-the great extent to which he -relied on Mr. : Ross.
' Although he could overrule Mr. Ross aon practically 'any decision, =
he - has never Jdone so with respect. tan a decision to admit ;a-candidate to the training program,lnorfwith. respect.to aJdecision 4
that.someone should~be1 removed from the program or.that~an
~
' individual should-be allowed to continue 1in the program. ;Mr...
Hukill noted that Mr. Newton or Mr. Leonard wouldi bel involved in r~N k_/
1 P
A
~
g r
s e
g
-135-recommendations to him on these subjects, and that his review of O
these matters involved discussions rather than simply a decision f
to accept or reject a decision.
Nonetheless, the bottom line is that Mr. Hukill identified no situations in which he had 1
position on an issue of this sort.32 UCS I
overruled Mr. Ro s s '
Training Exhibit 34.
264.
The handling of hk. Frederick, as well as of other to Licensee's process individuals, is instructive with respect l
and standards for deciding when candidates should remain in the 1
l training program.
UCS Training Exhibits 2-5 are various evaluations of Mr. Frederick's performance.
For the most part, they are quite positive.
They include comments by Mr. Newton,
for example, tha t Mr. Frederick is " technically superb."
UCS Training Exhibit 5 at 3.
In a telling turn of phrase, Mr. Newton i
also described Mr. Frederick as "a lawyer's favorite."
Id. at 4.
Apparently much of the reason that bk. Frederick was considered to be "a lawyer's favorite" was his-cooperation and support for the company whenever he was called upon to J-32 An arguable exception is Mr. Hukill's decision not to certify Mr. Frederick to take the NRC SRO instructor examination, as revealed by UCS Training Exhibit 1.
There, however, the-question -was not whether Mr. Frederick should be continued 'in 'the training program, and it was not really whether he should be.
permitted to take-the examination.
Ra ther, as Mr. J Hukill's notes reveal, the true reason for his overruling Mr. Ross' recommendation. that -Mr. Frederick be allowed to sit for the
- examination was Mr. Hukill's view that.Mr. Frederick should ' not continue as Supervisor of Licensed Operator Training, particularly because the NRC Staff had:various unresolved
. concerns about Mr. Frederick's veracity arising f rom his testimony -in the GPU v. B&W civil suit.
Mr. Hukill wa s also.
somewhat concerned _ that Mr. Frederick-might be1placed in=a very bad position if he took the examination _and. failed it foria:
n second time..UCS Tra ining Exhibit '. l.
V
-136-
{ '}
participate in legal proceedings such as the litigation with Babcock and Wilcox.
Ne wton, Tr. 33,446-447.
Th u s, FT. Frederick was a good soldier.
265.
UCS Training Exhibit 1 reveals that Mr. Frederick f ailed an NRC SRO instructor examination in the spring of 1984, while he was Supervisor of Licensed Operator Training.
It also 1
reveals that Mr. Frederick subsequently barely passed an oral examination administered by Mr. Ro s s.
On their face, these developments belie the Mr. Newton's earlier conclusion that Mr.
1.aderick was " technically superb."
Despite these developments, both FT. Ross and the training department recommended that FT.
Frederick be certified to sit once again for the NRC examination.
This did not occur only because Mr. Hukill intervened, and he intervened more.out of concern for NRC Staff s
reactions in another context than out of concern for the quality i
l of Mr. Frederick's work.
UCS Training Exhibit 1.
266.
We cannot reach any significant conclusions about Mr.
Frederick himself f rom this testimony, but we do.begin to develop some troubling understandings about the licensed operator training program.
Fi rst,. it wa s not the Training Department, f
which should have taken the initiative, but Operations, in the l
_ person of Mr.' Hukill, who decided Mr. Frederick had not performed -
adequately to remain as Supervisor of Licensed Operator.
Training.
- Second, r^s, -
\\,)
-137-there is a problem here of too much subjective judgment playing a major role in these decisions.
Mr. Frederick, who was clearly a f avorite of his superiors, " barely passed" an oral examination.
The concerns raised by Dr. Regan about oral examinations, Regan 4
ff. Tr. 3 3,532 a t 12, 15, 16, suggest tha t bk. Frederick might well have f ailed a more objective examination because he would l
not have benefited from the positive personal interaction.
Our view of the situation is that Mr. Frederick, the good soldier who had stuck with the company through the bad times, may well have been treated lightly by both Mr. Ross and training management for just that reason.
Only concerra expressed by the NRC Staf f with respect-to Mr. Frederick served to result in his removal from the significant position of Supervisor of Licensed Operator Training.
267.
We cannot know with certainty whether the scenario O
described above is what actually occurred.- None of the evidence l
j significantly contradicts that possibility, however, and.we.are i
left with a serious concern about the-subjective nature of Licensee's judgments with respect to questions such-as these.
268.
Our concerns are only heightened when we' consider other.
examples discussed during the-hearing., UCS brought.out on cross-examination the poor. examination performance record of-a:
I Senior Reactor Operator, Mr. Olive, and two Reactor Operators,-
Me ssrs. ' Wa'ish and Moore'., All three of these individyals' were allowed to. continue after failures.
The details _are revealing.
269.
UCS Treining Exhibits 8-16, 27 and 28 are various e
performance evaluations and exam'ination results. related. to. Mr.=
Olive.
They f reveal' consistent dif ficulties,with 'Mr.10 live's '
l' 1
6
-138-performance in 1982 in the position Shif t Foreman, which he holds today.
Newton, Tr. 32,422.
Mr. Olive needed improvement in the way he gave instructions to subordinates and needed to get along better with personnel before he could be a good leader.
Supervisors noted that he needed to improve his attitude to a better foreman, and that he had already failed an attempt to 4
obtain an SRO license.
UCS Training Exhibit 11.
In addition to failing in that attempt, Mr. Olive had also f ailed a comprehensive written examination given in the reactor operator replacement program.
That f ailure occurred on January 25, 1982.
UCS Training Exhibit 27.
270.
By 1983, a year af ter the evaluation in UCS Training Exhibit 11, Mr. Olive's attitude problem were believed to have improved, but he still needed guidance due to inexperience in the
.O plant.
Despite that deficiency af ter a year in the. plant,. and apparently despite Mr. Olive's failure of one of the weekly quizzes during 1983, Tr. 32,447, the evaluators considered that his technical knowledge level was good.
UCS Training Exhibit 12.
In Ma rch of 1984, however, Mr. Olive failed the oral examination in the annual requalification program.
UCS Training; Exhibit 28.
As a result of that failure, he was placed in a specially designed upgrade training program.
Despite1that program, he failed another otal examination,lthis one given by a panel of members from the Operations'and: Training' departments.
Tr. 32,427-429.
271.
Af ter these1two f ailures, Licensee faced the question
~
of what to do.about Mr'. Olive.
Mr. Newton testified during the
()
N)
-139-hearing that Training discussed the matter with Mr. Ross and that s
Training also discussed whether the special effort to help Mr.
Olive would be detrimental to the training department.
- Newton, Tr. 32,429-430.
It turned out, however, that in his deposition Mr. Newton had testified that Training had not questioned whether to make a special effort for Mr. Olive.
To the contrary, Training went ahead because Mr. Ross, whose job it is ours to support, indicated that this is what he wanted us to do, and since we were capable of doing it or felt we could do it, we did it.
Newton, Tr. 3 2,4 31.
Thus, Training developed another upgrade program for Mr. Olive, this time including, among other things, extensive on-the-job-training (on a non-operational reactor).
-272.
This upgrade program was followed, at long last, by Mr.
Olive's passing an oral examination.
The situation had become so serious that Mr. Hukill had had to wa rn Mr. Olive that it wa s essential that he pass this one or.he might well lose his SRO license.
UCS Training Exhibit 9.
He did not. pass with flying colors, however.
In fact, his performance was so poor that that he was placed in yet another ' upgrade program in order to. address-recurring weaknesses in several areas.
UCS Training Exhibit 16, Ro ss, Tr. 32,445.
Despite this situation, he was returned to duty as a Senior Reactor' Operator and a Shift Foreman.
Id,.
'His.
upgrade program is being signed of f by. a Shif t Supervisor, who-is a member of the Operations Department.
Ross, Tr. 32,447-448.
273.
The discussion thus far~ raises'several concerns.
- First, FT.~ Olive had a consistent history.of inadequate
%d 1
i
- . _.m
-140-a
}
knowledge, even after he had been at TMI-l for several years.
Second, this is another example of the barely passed oral examination.
Once again, Dr. Regan's concerns apply.
Regan, ff.
Tr. 33,532 at 12, 15, 16.
Wi th bk. Hukill strongly trying to have Mr. Olive pass the examination, and with practically everyone having decided that he could do the job with a little more training, the natural inclination of the examiners would have been to find that he passed.
That is particularly true if, as Mr. Hukill wa rned, a failure might result in loss of his license.
We in no way suggest chicanery or any conscious wrongdoing.
Ra the r, the problem is lack of controls and the intrusion of extraneous concerns into the subjective judgments made in oral examinations.
We have no confidence that similar performance by another operator in a less threatening situation O
would also have been considered a marginal pass, rather than failure.
274.
We are also concerned about the wisdom of Licensee's judgment in this case in light of other 6evelopments with respect to Mr. Olive's performance.
We have seen that as early as 1982, Mr. Olive had attitude problems and dif ficulties dealing with other. operators.
- See, e.g., UCS Training Exhibit 11.
In early 1984, Mr. Olive wa s assigned by Mr. Ross to assist the Training
~
Depa r tment in connection with work on the simulator.
Mr. Ross hoped,'in part,.that this exposure would help him _ grow, which Mr.
Ross believed to be ~ important since Fk. Ross _ considered Mr. Olive to be fairly new to the supervisory ~ ranks at that point.
The
. result of this assignment, however,
(~~N V
.1
-141-
'(}
was that Mr. Olive could not get along well with his immediate supervisor in the Training Department assignment, and Training asked that he be returned to Operations.
UCS Training Exhibit 9, Tr. 32,436-439.
275.
Mr. Ross later explained that }2. Olive's attitude 4
l problems and other such difficulties may have been caused by strains in his personal life.
Ross, Tr. 32,968.
We appreciate that concern, and we believe that an employer should do so as well.
We believe, however, that the history of Mr. Olive's performance, as revealed by this record, goes beyond the reasons for his particular problems in 1984.
He has been consistently weak on various aspects of technical knowledge, and he has had problems dealing with other operators and in performing as a supervisor for more than two years.
We are hard pressed-to see a substantial' justification for - keeping Mr. Olive and expending
' training resources on him after'this poor record.
We have now had ' testimony to improvements, Leona rd, Tr. 3 2,967. 989-990, but question the value of.that information in light of the we concerns that we have expressed elsewhere about evaluating and examination aspects of the training program.-
276.
More important, we note with concern what we believe to be the real reason that Licensee has-been so lenient.in its trea tment of Mr. Olive.
As ' Mr. Ross explained,-
We have an investment in an employee that is very'hard to replace.
So, we would like to keep anl employee long term.
We like to keep.a guy in-the operations-staff'for fifteen years.
X k,) ~
~
l
-142-Ro s s, Tr. 32,449.
Fundamentally, the decision is strongly
{}
influenced by the desire to minimize costs and maximize convenience for the company.
277.
Mr. Ro s s ' other explanations of the decision to retain Mr. Olive ring quite hollow.
He testified, for example, tha t Mr.
Olive had "an extensive background in proven operation, and proven supervision."
Ross, Tr. 32,449.
This just is not true.
His background as a supervisor, as revealed by this record is, indeed, proven, but it is proven to be troubled and negative in many respects.
His extensive background is also questionable.
Although he served in-the nuclear navy, Tr. 32/450-451, he joined Licensee directly out of the navy, and he has no experience at an operating _ nuclear reactor.
Ro s s, Tr. 32,451.
278.
Licensee's decisions with respect to Messrs. Walsh and-O Moore similarly concern us.
As reflected in UCS Training Exhibits 21-23, _ Mr. - Walsh - f ailed a mock NRC e xam,1which is the comprehensive written examination at the end of the. replacement program.
He then passed. such a mock. exam, but proceeded. to fail an annual requalification exam soon thereafter.. Tr. 32,624-627 Mr. Leonard testified 'that Mr. :Walsh's performance. improve'd a f ter these failures, Tr. 32,958-960, but.we -find cold comfort in this' development for several~ reasons.
First, as we have. already' said,:
- we - have serious doubts about. the - adequacy of the examination process in the first place.- _ Second, HMr.. Leonard - did ~ not know: that :
Mr. Wa lsh 'had - f ailed. earlier,. and 'he 'had' made ; no.ef fort ; to follow
~Mr. Wa 1sh's _ per formance 'on.the' subjects - that he had / f ailed. - Tr.
33,434-5.
Th u s, ' Mr.. Leonard L had n'o way of knowing :wheth'er Mr. <
(-) -
Wa lsh had, in fact,. improved,:or_whether;his more'
. M
r u__
m
-143-recent performance had simply been on subjects on which he had performed well on the examinations that he had f ailed.
279.
Mr. Bbore's situation is more disturbing.
Fi rst, he f ailed two mock NRC exams in a row, ye t wa s allowed to continue in the program and take a third such exam, which he managed to pass.
UCS Training Exhibits 24-26, Tr. 32,627-630.
Those facts are troublesome on their face.
More troublesome, however, wa s Mr.
i -
Ro s s ' erplanation of Mr. Moore's dif ficulties.
In his deposition, Mr. Ross stated, with respect to Mr.- Moore, that "if you asked him to describe the white board,
. he would describe the black one to you.
He just didn't read the questions very well."
Mr. Ross agreed that his problem was that.he did not pay attention to directions.
Tr. 32,631.
Mr. Ross sought to minimize the significance of this information during the hearing by suggesting
(
that Mr. ~ Moore did well on oral examinations because the examiner t.
could stop hjm from going off on the wrong track.
Tr.
1 ;
32,631-632.
Th-is explanation hardly instills confidence in the oral examinations process.
Mr. Leona rd and Mr. Ross also-later
. testified to Mr. Moore's improved performance on Licensee examinations and argued that Mr. Moore is a competent. operator.
Tr. 32,960-962.- Even that testimony troubles us, however,;because Mr. Ross reiterated that Mr. Moore _goes off on tangents Uhat he,
g i
s shou'ld avoid when answerkhg questions.
Ro ss, Tr. 3 2,' 9 6 3.
In
.<w
~,
addition, as with Mr. Walsh, Mr. Leonard. had no direct. knowledge.
aof the areas in which he had performed-poorly, so the comparison toflater: performance.is' suspect.; Tr.'33,435-439.
m
- 'i
!Q
'. d ;h d.
D tau _ '*>
-144-280.
Our reading of the evidence is that Mr. Moore does, indeed, have difficulty paying careful attention to what is asked of him, particularly if it is asked in writing.
We cannot determine whether the problem is so serious that he should be dismissed as a reactor operator, but we would not want to have an accident handled by someone who went off on tangents in reading the procedures or otherwise in responding to the emergency.
281.
We are lef t,' on this record, with a serious concern
.about Licensee's priorities and its judgment in deciding how to handle those who do poorly-in its training program.
The evidence t
establishes that licensee tends to err in favor of its economic and convenience interests, and perhaps in favor of the personal interests of the employee, rather thancin favor of safety.
Given
-the~ inherent'and excessively subjective nature of the decision ~
- O making process in this training program, we believe. this situation is intolerable.
I -
282.
As in - any complex 'organiza tion, Licensee's evaluations 6
of the, job performance of its employees-is important to the
. ef fective functioning of the -organiza tion.: Licensee's evaluations of the. competence: of ?its reactor L operators is certainly. important
'to assuring that'the reactor is' operated' safely.
Licensee $s L
~ re. qualification' training l program serves "tio some : extent asisuch a-
- check,;although the preceeding discussion 11ndicatesj thati there is -
- no assurance thatt the ; program excludes unqualified ! personnel 5-from
~
~
-rsactor' operator positions.~ In. addition,cas Dr.JREgan-explained,.
~
i Regan, L f f. Tr. 3 3,'532 La t - 9,-fit fis f nec.essaryL to l assess performance ;
'l
~
):in the;:traiding-programiagainst perforinance.on theXjob in' _
s
~
~
(
-145-order to determine the adequacy of the training.
Licensee rebuttal witnesses sought to use a statement in Dr. Re g a n ' s deposition to suggest that such an assessment is not done all that often, and thus, presumably, that it should not necessarily be expected at TMI.
Leonard & Knief, f f. Tr. 33,364 at 3-4.
Dr.
Reg an - responded, in essence, that the Licensee witnesses had misunderstood his statement and that they were wrong.
While
' direct comparisons of job performance to training performance-may not be common when considering the f ull range' of educational and training activities, the situation is different in industrial settings such as Three Mile Island.
He r e, the relationship'of training to job perfornance is much closer, and the two are much more frequently assessed together.
Dr. Regan emphasized that in this-type-of setting, " assessment of the training program'against
. O the operational performance of the. individuals, teams, and systems Jinvolved in.the program is the'onlyl reliable means.of measuring the effectivenessLof training."
Regan, ff. Tr. 3 2,6 9 3 'a t 3..
283.
We address,-therefore, the' question of how: Licensee attempts to evaluate job _ performance and to compare job 1
'l
. performance to performance in the training' program.
284. -This turns out to be 'a distressingly Jatraightforward I
--review on.our part.
Other than an initial-performance. review
_af ter an employee has' been on' the job.for 90. days, there ~-are: no
-formal evaluations of operat'or performance.
Beyond that point,.
then unik.i contract prohibits --formal" written evaluations.-. Ross,
.Tr. 32,897._ Aspfar as-we'can tell,: evaluations'outsidefthe!'
~
??
c
-146-training program itself must be done on an informal basis between the employee and the supervisor, with perhaps some sketchy notes involved, but nothing that would constitute a careful review of job performance.
Tr. 32,897-898.
285.
We are aware of the fact that union contracts may prohibit forma 1 ' written job evaluations that might be used as the basis for discipline or dismissal.
Mr. Jordan raised the question of whether Licensee's union contract prevents Licensee from doing such evaluations if the purpose is to improve the training program, rather than to determine how an operator should be treated.
We think Mr. Ro s s ' response is instructive.. He testified that he could not see how an operator's performance would have a lot to do with what was taught in the training program.
He went on to-suggest that he viewed an evaluation as I
largely a personal matter, but his discussion establishes that a l
performance evaluation would cover how an operator does his job and his knowledge of the job.
Ro s s, Tr. 33,419-421.
286.
Mr. Ro s s ' testimony on this point confirms that he does not understand or would prefer to ignore one of the most significant principles in industrial training and job i
' performance.
There is a crucial need for objective information on I
which to make determinations of the adequacy of the training program and of the. adequacy of performance on the job.1 It is'not-enough to rely upon a technical expert such as Mr. Ross,, no matter how competent he may be. 'This is not an athletic team where the coach's intuitive judgments may lead to great-victories, nor:is it an educational institution where~there~is no clear relationship.
.i -
-147-between the classes and eventual job performance.
This is a nuclear reactor, where there are specific, definable tasks, where it is possible to assess the relationship between training and job i
performance, and where reliance on the coach's intuitive judgment i.
could lead to far worse than a losing season.
In lig h t o f Mr.
Ro s s ' considerable, indeed unique, influence at TMI-l and in the are greatly troubled by his failure to training program, we understand this fundamental point.
287.
Equally impor?. ant, we are constrained to find that Licensee is at this point incapable of directly comparing job perfomance with performance in the training program.
It makes no efrort to do so and seeks to defend its failure by blaming the union.
This in no excuse.
Licensee has made no effort to measure job performance'for this purpose.
e.
Employee _ Attitudes 288.
The Appeal Board expressed considerable. concern with the question of employee attitudes toward the training program or employee attitudes that may af fect the effectiveness of the training program.
ALAB-772, 19 N.R.C.'at 1234.
We have
~
previously discussed testimony from Dr. - Regan and the Staf f, which establishes that a sound ef fort to evaluate employee attitudes requires carefully structured ' interviews," anonymity, and the like.
We have also :previously discussed the point made by Dr.:
o Gardner and Dr. Regan that attitudes Lare transient,' and that the most significant point -is attitude trends: over time. =We now
~
discuss testimony purporting'to address-this, issue,fwhich we have received only from the ~ OARP: Committee and f rom Mr. Ross.
/^)
u
-148-289.
With respect to the testimony from the OARP Committee, we reiterate what we said about their methodology.
They conducted various interviews, but they did not employ. structured interviews, and their testimony is useless for the purpose of determining attitude trends over time.
In addition, at least some of the Committee members explicitly stated to those whom they interviewed not only that they were with this Committee, but that they were preparing testimony in the restart proceeding.
We have no doubt that this may significantly have affected the attitudes of interviewees who are strongly in favor of restarting the reactor.
Accordingly, we are unable to base any conclusions with respect to employee attitudes on the testimony of the OARP Committee.
290.
In addition to the testimony of the OARP Committee and Mr. Ross, we have the so-called RHR Report, UCS Training Exhibit O
6, and the raw data on which the report was based, UCS Training Exhibit 7.
The report, which was based on a survey of operators at TMI-1 and Oyster' Creek, reached a number of disturbing conclusions, including the following:
1.
A majority of trainees at TMI disagreed with the proposition that top-management is more concerned about public safety than it is about generating electricity.
UCS. Training' Exhibit 6 at 21.
-2.
Only a, slight majority of operators agree that despite cumbersome procedures, the Licensee-policy on-compliance isifollowed.- Foremen are said.to push operators to' keep things-moving,'and this requires deviating from written' procedures.. Id.. a t '
24.
3.
. Only. 60% of.those who, responded agreed ~ that the content of the last exams was relevant to their
' jobs, and only.one-third-agreedithat the oral portion of the' exam tested how one would act inian emergency.
Id. ~ a t 27.
,m
-149-4.
Three-quarters of the operators were dissatisfied with the training for licensing, and an even s,
greater proportion were strongly dissatisfied with requalification training.
bbst considered that the training department was not oriented to the needs of the operators.
Trainees and SROs at TMI disagreed that the quality of the training staff was good.
Id,. at 27-28.
5.
Three-quarters of the operators denied that training orepared them for what they actually do.
In their perception, training prepared individuals to pass examinations, but does not adequately prepare them to operate the plant.
Th is is particularly true at TMI.
Id. a t 28.
291.
We recognize that the substantive validity of the RHR I
Report was not litigated at length in this proceeding and that the survey was conducted in 1982, soon after the cheating incidents and hearings.
We also recognize that the significance of attitudes is not so much what they sre at any particular time, but i-how they change.
Despite these limitations, we are lef t with the RHR Report as the only significant evidence of attitudes done through a structured and reliable process.
The report raises
~
l concerns, and we have had no detailed evidence of later. structured surveys to inform us as to trends in the-findings in the RHR Re por t.
i 292.
Whatever the significance of the substantive findings l
in the RHR Report,.we are-extremely disturbed by Mr. Ross' reactions to those findings.. When he first read the -RHR Report :
finding that a majority of TMI trainees and a substantialcminority:
of other; personnel thought management was more concerned about generating electricity than-about safety, he was :not - concerned:
about the : finding.
He complained first that he had-some
~
difficulty understanding what-the: Report!was?saying.and then
((
/'T -
l
\\J l
-150-so he argued that he was aware of what attitudes actually were, l
was not concerned about the findings.
We disagree that it is difficult to understand what the report was saying on this point.
While the wr iting is not perfect, it clearly indicates a perception on the part of TMI trainees and many others that management is more concerned with generation than with safety.
Mo reover, Mr. noss could have checked with RHR or reviewed the raw data if he was unable to understand the report, but he failed to do either of these.
Ro s s, Tr. 32,565-567.
293.
On questions of the adequacy and relevance of training, Mr. Ross once again had no concerns about the RER findings.
To the contrary, his rather creative reading of the report emphasized the fact that training was improving, so the program was doing we ll.
He even managed in his deposition.to interpret the -language
(:)
at the bottom of page 28 of UCS Training Exhibit 6 as meaning that forty-nine out of fif ty of those who revealed these negative attitudes could have been f rom Oyster Creek, although that is simply not a fair reading of the language.
When confronted.with this'unsupportable interpretation, he retreated into his complaint.
that he has always had trouble reading.the RHR
/~T V
I d
-151-Re po r t.
Ro s s, Tr. 32,577-589.33 294.
Mr. Ro s s ' reactions to the RHR report were not reasonable.
He consistently read unfavorable language in a favorable light, and he failed to seek explanations for patently unfavorable statements.
Instead, he simply decided he could not understand the report in those areas.
Incredibly, he concluded that the RHR Report "was very good and very positive about TMI."
Ro s s, Tr. 3 2, 5 7 3.
Whatever the validity of the report, it certainly was not that.
295.
We come now to judge Mr. Ross' " testimony on the current-attitudes of operators at Three Mile Island.
Not' surprisingly, he concluded that those attitudes were " absolutely good."
His opinion is based upon operators' performance in training and his general interactions with them.
He has not done any particular-O the purpose of ascertaining operator attitudes.
interviews for:
33 Mr. Ross also testified.that the RHR conclusion with respect-to compliance with procedures was of no concern to him when he first read it.
In. essence, he had no concern about the matter both because.he thought he knew' what operator ~ attitudes were on the. subject and because tur read the RHR' report as indicating that-
~
a majority of operators agreed -that that procedures were-complied
- with. ' Although RHR r eported that ' only ' a slight majority had thisL f avorable attitude,.Mr. Ross 'made no ef fort to determine -whether -
this meant, as it easily.could'have, that.a substantial} minority:
believed that procedures : were l not being complied 'with.c. Ross, Tr..
32,567-572.
Had Mr. Ross investiga ted the matter, tut would have
' discovered that a large ' majority of operators atiTMI-believed that.l procedures were being followed,a so that. this iparticular Lissue. did-not. appear to be aLproblem.
UCS Training Exhibit 7.
- Oursproblem,.
therefore, is not with this particular RHR~ finding, but. with Lthe f act: thatf Mr. Ross seems incapab}e of f believingJthat anythingJcan reflect badly on TMI, even.when theJ1anguageiof a report clearlyc seems to do so.
Heisimply interprets ' the clanguage ina f avorable,.
if distorted, way, and :her does not make any ef fortito: determine -
~
whether or. not his. optimistic ? interpretation, is : correct.
a E
-152-Ro s s, Tr. 32,561-563.
In light of the advice we have from experts in the field, both the NRC Staff and Dr. Regan, and in light of 4
Mr. Ro s s ' blind optimism about attitudes at TMI-1, we are unable to give any weight to the highly subjective judgments that Mr.
Ross has made about operator attitudes.
He has made no systematic effort to develop the information that we need, and it is clear l
that anything but the most structured survey would be so colored by his perceptions and his obvious self-interest in reporting favorably about the attitudes of the men who work under his supervision that it is more likely to reflect his favorable views rather that whatever views operators may actually have.
296.
On this record, therefore, we have no reliable information about current operator attitudes.
We do have the RHR
~
i Re po rt, which reveals several disturbingly negative attitudes as of the date the survey was done in 1982, but we are unable to i
assess trends.
We are unable to respond to -the Appeal Board's questions in this area.
e 4
r e
4
' d g
'f
_g
-153-O III.
Conclusions of Law 297.
The Board has considered all documentary and oral evidence presented by the parties on the issues remanded by the Appeal Board in ALAB-772.
We have also considered the j
significance of the evidence with respect to the issues raised in the Commission's order of August 9, 1979, specifically the adequacy of the retraining and retesting of the operations staff and the adequacy of Licensee's managerial capability and resources in this area.
CLI-79-8, 10 N.R.C. 141, 143-146 (1979).
298.
Under ALAB-772, our ' charge is to address the question of whether, in light of the cheating incidents at TMI-1, the licensed operator training program adequately prepared the operators to run the plant safely. -We have done so, as the Appeal Board. directed, by considering the views of the Reconstituted OARP Committee, as well as the testimony of other f
witnesses with respect'to the; methodology employed by the-Committee.
We have also considered the testimony of 'all p
witnesses on the-fundamental issue of the adequacy of;the licensed operator training program.
Since both.-the Committee :and Licensee's other witnesses chose to respond to ALAB-772 :by.
testifying to the quality l of the ' program' as a-whole, we must
' decide that issue.
y 299. - We have found. as a matter of.f act that the' methodology-i employed by the OARP l Committee' to -respond to ALAB-772 swas inadequate to the task.' The Committee'did_not_providie;us1with:
(_/
5 i
^
v e
y
-154-testimony on which we could reasonably evaluate either the Committee's response to the cheating incidents or the quality of the training program as a whole.
We hold, therefore, that we have been unable to comply with the Appeal Board's directive to obtain the considered views of the Committee or of other consultants to Licensee.
300.
Turning to the substantive issues raised by the Appeal Board, we-reach the'following conclusions:
The management of the THI-l licensed operator a.
training program is not qualified to assure that the program adequately trains'the operators to run the plant safely.
The-only management personnel with significant expertise in the area of training are Dr. Long and Dr. Knief, both of 'whom are too O.
~
removed from operation of the-program to assure that it achieves this goal.
Similarly most of the management personnel have.no experience ln' the operation of a commercial ~ nuclear reactor.
b.
As a result of these deficiencies, the^ program
~
depends very heavily upon ' the - experience :and
. expertise ' of Mr. Ross.
Mr. Ross,. howe ver, : is Ma::ager.of Plant Operations.
This violates the' need for ' independence in.the actions s taken by:.thef training department.
In addition,.the evidence) establishes that :Mr.- Boss ihas an, unalterably-
.f.
optimisitic view'of conditions.;atATMI-1, such' that 10
^
4
\\..
t k
I
-155-he cannot be depended upon to recognize weaknesses O..
or deficiencies.
Thus, we hold that the licensed operator training program lacks the independence necessary to assure the safety of plant operation, c.
We hold - that Licensee has f ailed. to determine the causes of the cheating incidents, and thus has failed to show that the current licensed operator training program does not suffer from the same deficiencies that may have caused the cheating in the first place.
d.
Licensee's examination process is inadequate to measure an operator's ability to run the plant.
Controls on the construction of the written examinations are inadequate to assure that the examinations have a proper balance'of questions.-
The other-three types of examinations, oral, 'OJT, and simulator, which are essential'to validate the written examinations and each - other,.are inadequately controlled and too subjective to serve.
that purpose..
e.
The licensed operator ' training program depends;upon the adequacy -of its implemention of the TSD model-and its - development of job-task analyses L and learning objectives._' Licensee has not completed-theseLefforts.- In-addition,-Licensee:has. failed to demonstrate-that-the training; program adequately; addressesithe interference problemsi n
v is I
E
-156-that arise when job incumbents are frequently trained and retrained as procedures are changed.
Since this problem is most serious in dealing with wmergencies, this is of particular concern to the Board.
f.
We have been provided with no reliable evidence showing that communications between management and the operators are ef fective or that employee attitudes have improved since the developments that gave rise to the Appeal Board's concerns.
301.
Based upon our evidentiary findings and upon the conclusions set out above, we hold that Licensee has failed to demonstrate that the TMI-l licensed operator training program adequately prepares the operators to operate the plant safely.
O Thus, Licensee has f ailed to comply with the substance of the Commission's order that it retrain and re-examine the operators.
Based upon this evidence, we also conclude that Licensee does not have the managerial capability and resources in the: area of licensed operator training-to assure the safe operation:of' TMI-1.
Our conclusion.is buttressed'by the fact-that even after six years of.trying, Licensee has been unable to reach a minimially! acceptable standard of performance.
b'
1
-157-302.
For these reasons, we hold that Licensee has failed to O
4 demonstrate that the Commission chould authorize the restart of TMI-1.
TMI-l shall remain in a cold shutdown condition.
l Respectf ully submitted,
$dd
\\
A r
ptsyrda n, 'III i
William S i
f lf'f/ l^!' y i
Ellyyt R. We iss
//
4 l
HARMON, WEISS, & JORDAN 2001 S Street, N.W.
Suite 430 Wa sh ing ton,
D.C.
20009 (202) 328-3500 Da ted:
Fe brua ry 25, 1985 O
t
+
6 0
.w..p
==
O USMC February 25, 1985 Y FEB 25 P5:38 UNITED STATES OF AMERICA 0F S i'UCLEAR REGULATORY COMMISSION f-
)
"*:fe In the Ma tter of
)
)
- us. w,.
METROPOLITAN EDISON COMPANY
)
Docket No. 50-289,
)
(Restart Remand on (Three Mile Island Nuclear
)
Ma nag ement)
Station, Unit No. 1)
)
)
CERTIFICATE OF SERVICE I hereby certify that a copy of the UNION OF CONCERNED SCIENTISTS' PROPOSED FINDINGS OF FACT AND CONCLUSIONS OF LAW ON THE ISSUE OF LICENSED OPERATOR TRAINING AT TMI-1, and two letters to the Atomic Safety and Licensing Boar,d dated February 25, 1985,-
was served on those indicated on the accompanying Service List.
Service was made by deposit in The United States mail, first class, postage-prepaid, on January 31, 1985..
D-s#Yb William rdan III-g
T UNITED STATES OF AMERICA J
NUCLEAR REGULATORY COMMISSION BEFORE THE ATOMIC SAFETY AND LICENSING BOARD In the Matter of
)
)
METROPOLITAN EDISON COMPANY
)
Docket No. 50-289
)
(Restart Remand on (Three Mile Island Nuclear
)
Management)
Station, Unit No. 1)
)
)
SERVICE LIST Administrative Judge Gary J.
Edles, Chairman Jack R.
Goldberg, Esq.
Atomic Safety & Licensing Appeal Bd.
Office of the Executive Legal Dir.
U.S. Nuclear Regulatory Commission U.S.
Nuclear Regulatory Commiss'on Washington, D.C.
20555 Washington, D.C.
20555 Administrative Judge John H.
Buck Deborah Bauser, Esquire Atomic Safety & Licensing Appeal Bd.
Shaw, Pittman, Potts & Trowbridge U.S.
Nuclear Regulatory Commission 1800 M Street, N.W.
U
- nington, D.C.
20555 Washington, D.C.
20036 A...laistrative Judge Christine N.
Kohl Ms. Louise Bradford Atomic Safety & Licensing Appeal Bd.
TMI Alert U.S.
Nuclear Regulatory Commission 1011 Green Street Wasnington, D.C.
20555 Harrisburg, PA 17102 Administrative Judge Ivtn W.
Smith, Chairman Joanne Doroshaw, Esquire Atomic Safety & Licensing Board The Christic Institute U.S. Nuclear Regulatory Commission 1324 North Capitol Street Wcshington, D.C.
20555 Washington, D.C.
20002 Administrative Judge Sh ldon J. Wolfe Mr. and Mrs. Norman Aamodt Atomic Safety & Licensing Appeal Bd.
R.D.
5 U.S.
Nuclear Regulatory Commission Coatesville, PA 19320 Wn hington, D.C.
20555 Administrative Judge Lynne Bernabei, Esq.
Gustcve A.
Linenberger, Jr.
Government Accountability Project Atomic Safety & Licensing Board 1555 Connecticut Ave.
U.S. Nuclear Regulatory Commission Washington, D.C.
20009 Wschington, D.C.
20555 sting and Service Section Michael F.
McBride, Esq.
O ce of the Secretary LeBoeuf, Lamb, Leiby & MacRae U.S. Nuclear Regulatory Commission 1333 New Hampsnire Ave, N.W.
41100 wachington, D.C.
20555 Washington, D.C.
20036
Michael W.
Maupin, Esq.
Hunton & Williams 707 East Main Street L
P.'O. Box 1535 Richmond, VA 23212
-Thomas Y. Au, Esq.
Office of Chief Counsel Department of Environmental Resources 505 Executive Houses P.O.
Box 2357 Harrisburg, PA 17120 e ee 0
0
.