ML20126L008
| ML20126L008 | |
| Person / Time | |
|---|---|
| Site: | Crane  |
| Issue date: | 06/17/1985 |
| From: | Jordan W HARMON & WEISS, UNION OF CONCERNED SCIENTISTS |
| To: | Atomic Safety and Licensing Board Panel |
| References | |
| CON-#285-479 SP, NUDOCS 8506190390 | |
| Download: ML20126L008 (15) | |
Text
,
(7 !
UCS - June 17, 1985 UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION BEFORE THE ATOMIC SAFETY AND LICENSING BOARD
+-
)
D0 In the Matter of
)
gTED C
)
Docket No. 50-289 METROPOLITAN EDISON COMPANY
)
(Restart Remand on
)
Management)sP TS JUN 18 A10 52 (Three Mile Island Nuclear
)
Station, Unit No. 1)
)
)
CFFICE OF SECREW '
DGC4EilhG & SE?vir E ERANC)!
UCS' COMMENTS ON LICENSEE'S PLAN FOR POST-TRAINING EVALUATION UCS has reviewed the Proposed Evaluation Plan filed by Licensee on May 28, 1985.
The plan purports to respond to the Board's order that Licensee develop a plan for the evaluation of post-training performance of reactor operators in the job setting.
For the reasons stated below, Licensee's plan is inadequate and does not comply with the Board's order.
The Board's Order The Board held as follows with respect to the issue of post-training evaluation:
The Board concludes that the TMI licensed operator training program needs improvement because it does not provide for the evaluation of its trained personnel in the job setting for the purpose of validating and revising its training program.
The Board will impose a license condition that formal written on-the-job evaluations of operator performance both during normal and abnormal operation be performed.
PID, Paragraph 258, Slip, op. at 154.
Based upon this finding, the Board issued the following Order:
The Licensee shall implement a plan for the evaluation, after training, of the performance of its trained reactor operators and senior reactor operators 8506190390 850617 PDR ADOCK 05000289 0
i PDR
'5))
i l in the job setting, under both normal and abnormal operation, for revision of the TMI-l licensed-operator training program in accordance with the Board's finding and conclusions, particularly Section III.C.4 supra.
PID at 216.
Under the Board's decision, therefore, Licensee's evaluation plan must meet the following criteria:
1.
It must provide for evaluation of the licensed operators in the job setting.
2.
It must provide for evaluation of operator performance under both normal and abnormal operation.
3.
The evaluations called for by the plan must be designed to provide the information that is necessary to identify appropriate revisions of the TMI-l licensed operator training program, if any revisions are indicated by operator performance.
UCS' COMMENTS 1.
General Comments The major flaw in Licensee's proposed plan is that it appears to be more of a paper exercise than a program to evaluate operator performance in any useful way.
At several points, the l
plan states what Licensee intends to do, but it fails to detail I
how Licensee will do these things.
For example, in Attachment 1, i
which addresses evaluation of performance in abnormal events, Sections 4.4.2, 4.7, and 4.7.1 would now include language l
requiring review of operator performance, but Licensee does not explain how that review will be performed or on what basis i
judgments will be made.
In the absence of any detail as to how the review is to be performed and the standards against which performance is to be jndged, it is impossible for either UCS or the Board to determine l
. whether the evaluation will serve any useful purpose.
It could be anything from entirely subjective to adequately detailed, depending upon who does the evaluation.
Unless the plan itself and Licensee procedures spell out the required information, there will be no uniformity of approach, and there can be no assurance that the evaluations will be adequate.
This general problem plagues Licensee's plan throughout, more seriously at some times than at others.
During the meeting of June 11 among the parties to the hearing. UCS requested an explanation of how evaluations would be performed and hog Licensee would assure that the information was taken into account in the training program.
Dr. Long explained that he and others are responsible for assuring that these actions take place.
He explained, for example, that Licensco personr el, and particularly quality assurance personnel, undertake cumulative analysis of abnormal event data and look for trends, although he stated that this is outside of the training program itself.
He or other training managers are then responsible for integrating information into the training program.
This is part of the responsibility of their positions, but on the basis of this conversation, UCS understands that there is no specific formelized process to assure that these functions are performed and that the results of trending and other performance evaluations are properly taken into account in the training program.
. In summary, the plan is inadequate for failure to establish the requirements necessary to assure both that adequate performance evaluations will be performed and that the results of those evaluations will be used to improve the training program.
2.
The plan is inappropriate for this small training program.
Evaluations must be sooner, more frequent and more individualized.
The TMI-l licensed operator training program involves a relatively small number of people in a setting in which trainers, trainees, and supervisors all know each other relatively well.
This is not a program intended to train or evaluate 10,000 students.
Yet the evaluation plan fails to provide for the type of individual attention that is both necessary and possible in such a'ama31 program.
To the contrary, it could be applied to an enormous industrial facility with thousands of employees.
This approach results in two specific deficiencies.
- First, the plan does not adequately address whether operator performance is related to the aptitude of the operator in question or to the adequacy of the training program.
There is nothing in the plan that provides for distinguishing betwecn those two.
Unless such a distinction is made, however, the company cannot determine whether the training program should be changed to oddress poor operator performance, and it may make changes th'at hamper, rather than inprove the prograu if the program is not at fault.
Second, the progran includes a rigid schedule of evaluating operators first after six months on the job and then on an annual basis.
This schedule does not' meet the Board's objective of providing information that will be useful in evaluating and
l
. improving the training program itself.
The problem is that six months is such a long time on the Job that even an untrained person with great enough aptitude for the work might be performing as well as a trained operator simply by virtue of learning while doing.
Thus a performance evaluation at that point may not provide any information about the training program because performance will be determined as much or more by what was learned on the job as by what was learned in training.
}
An adequate plan would address these deficiencies in at least two ways.
First, it would provide for initial testing and determination of the aptitude of each individual operator for the type of work in question.
The operator's aptitude would then be taken into account when reviewing performance on the job, and it would be possible to determine whether or not poor performance l
reflected some failure of the training program in general, or whether the individual in question either needed specific attention or was not suited for the job.
In addition, this would l
allow any improvement in the training program to be tailored to the specific needs of the operators as revealed by the job performance evaluations.
That is precisely the approach that is both possible with such a small number of trainees and employees and necessary to assure that on the job evaluations result in appropriate changes to the training program.
Second, an adequate plan would provide for frequent evaluation relatively soon after the operator starts on the job.
The precise frequency of evaluation can best be determined by
t consultation between a training expert and a subject matter expert.
However, the initial evaluations should be within the first days or weeks on the job, and they should be repeated frequently for a substantial period of time, a minimum of several months.
After the initial several month period, the frequency can be decreased depending upon the operator's performance and the nature of the tasks being evaluated.
To the extent that the performance is consistently good and the tasks are relatively straightforward, frequency can be reduced, perhaps ultimately to a semi-annual or annual basis as proposed in the plan.
- However, for operators who do not perform consistently well, and for more complicated tasks, frequency should not be reduced, at least not as much.
Again, the frequency of the evaluations would be determined by the performance of the individual operator, and the training program would then be tailored and improved accordingly.
This mechanism assures that the performance evaluation, particularly early in the job, will reflect the training program and reveal deficiencies or areas of potential improvement.
This contrasts with Licensee's six-month evaluation, which is more likely to reveal an individual's ability to learn on the job rather than how well he was trained.
3.
The Evaluation of Performance During Normal Operation is Inadequate.
It was not clear to UCS from the materials provided by Licensee just how Licensee intends to perform the semi-anr.ual and annual evaluations of operator performance during normal
V
~7-operating conditions.
Dr. Long explained in the June 11 meeting, however, that the evaluation consists of completion of Exhibit 2 to Licensee's Attachment 5 for each operator by the operator's supervisor.
In completing this evaluation, the supervisor is to consider the areas identified in Licensee's Attachments 6 and 7.
This evaluation, particularly if performed only on an annual basis, will result in an extremely subjective description of the operators' performance.
The problem is twofold.
First, over the course of an entire year, an operator simply does too many things for the supervisor to be able to recall anything other than the grossest deficiencies.
The information from this evaluation would not be useful in identifying weak areas of the training program or specific areas that the training program needs to address.
Memory simply would not serve well enough.
Second, a supervisor's judgment in completing this type of evaluation is significantly influenced by the nature of the supervisor's interactions and relationships with the operators.
The
^
evaluation is at least as likely to reflect personality differences or conflicts as it is to reflect differences in performance.
This is precisely the type of evaluation that Dr.
l Regan demonstrated is subject to these extraneous influences.
Regan, Tr. ff. 33,532 at 15-16.
l What is needed is both more frequent evaluations, as 4
discussed earlier, and more specific evaluations.
Attachments 6 J
J und 7 identify general areas that supervisors are to consider, but each of those areas, in turn, breaks down into many specific
. ~
i.,
tasks.
To perform a complete evaluation, a supervisor would have to evaluate performance on each of the tasks, which would mean, in essence, using a checklist made up of the hundreds of tasks and subtasks set out in the job task analyses.
UCS does not believe that an evaluation of that sort would be reasonable simply because it would overwhelm the evaluator.
However, that does not mean that Licensee can avoid specific evaluation.
The appropriate middle ground is to require Licensee to have its supervisors undertake evaluations using checklists of items drawn from the job task analyses.
This approach should include repeated use of the same items to reveal trends over time, as well as changes in the items used in order to cover a broad enough subject area.
These evaluations need not be unduly burdensome if clearly organized and integrated into the supervisor's responsibilities.
Without this degree of specificity, the supervisor evaluations provide information that may be useful for such things as promotions and overall performance review, but that is virtually useless in reliably identifying weaknesses in the training program or areas that should be emphasized in training.
UCS proposed this approach in its discussion with Licensee.
Licensee rejected it as a
'*ain of its existing supervisory j
resources.
Licensee is rect.
Done correctly and efficiently, it will enhance Licensee's resources and performance.
4.
Evaluation of Performance During Abnormal Events is Inadequate.
Licensee relies upon review of abnormal event reporting forms to evaluate operator performance during such events.
The concept 4
is useful as far as it goes, but it's implementation is inadequate, and even if implemented correctly it is not adequate to the task.
The implementation of this concept is flawed for several reasons.
First, as discussed earlier, Licensee's plan does not detail how Licensee is going to assess operator performance during abnormal events.
Thus, it is not possible to judge that any particular evaluation will be done correctly, and there is no assurance that all evaluations will be done well.
Second, for many of the events that should be evaluated, Licensee will not necessarily record enough information.
In the discussion on June 11, Dr. Long stated that the report that appears as Attachment I to Licensee Attachment 4 would not necessarily show all of the operator actions taken with respect to the incident reported on the form.
Thus, any evaluation of operator performance will be a subjective one influenced by extraneous factors, rather than one based on what actually occurred.
Third, there is no specific assurance in the program itself that this information will be reviewed by training when it is significant to training.
Based upon Dr. Long's explanation, this may occur as part of the general job performance of Dr. Long and other training and operations managers, but there is no specific checkoff or other
r
~
aspect of the program that will assure that Training reviews the appropriate documents.
For the same reason, there is no checklist or similar mechanism by which it will be possible to review whether the information was correctly handled.
Even if its current proposal were implemented correctly, Licensee's evaluation of performance during abnormal events would l
still be inadequate.
In his testimony, Dr. Regan discussed the difficulty of measuring job performance for this sort of job, where there are not enough abnormal events or emergencies to f
permit reliable evaluations of performance during such events.
Regan, Tr. ff. 33,532 at 9-10.
In reviewing Licensee's plan, Dr.
Regan considered the use of the full scale simulator to be a necessary means of testing performance in these situations.
It is important to distinguish, however, between the use of the simulator for this purpose and the use of the simulator in Licensee's existing requalification training program.
Licensee's current use of the simulator is related to behavioral learning objectives that are part of training.
Thus, the simulator would test operator performance in areas in which the operator had been recently trained.
By contrast, an actual emergency will not take into account what has been most recently included in the training program.
Neither should a simulated evaluation of job performance.
These should be administered separately from tratning, although they could be scheduled during the training i
cycles for practical reasons.
As with evaluations of performance during normal operation, the frequency and content of the i
1
__,.._...~__...__.,m__.__
_11_
simulations should be tailored to individual performance.
They should be frequent soon after operators start on the job, and the frequency can be changed for individual operators as they demonstrate consistent good performance.
5.
Reviews of Performance Evaluations Must Consider Job Design as Well as Impacts on Training.
The adequacy of training and the ability to improve training are directly related to the design of the job.
It may be that poor performance in a particular area cannot be improved by further training because the problem is with the requirements of the job itself.
Unless Licensee addresses this issue in its review of performance evaluations, it cannot determine correctly whether something needs to be done about the training program.
Thus, Licensee may waste effort trying to improve training when the problem lies elsewhere.
Worse, Licensee may rely on attempts to improve training to the detriment of safety if training will I
not solve the problem.
Licensee has responded to UCS' concern by arguing that this is beyond the scope of the Board's order.
UCS disagrees.
These issues are so interrelated that they cannot be separated in this context.
Licensee's response is the type of myopic thinking that has gotten this company, this industry, and this agency into so much difficulty in recent years.
Licensee's plan is inadequate unless it includes consideration of job design in response to performance evaluations.
Conclusion For these reasons, Licensee's proposed post-training e
evaluation plan is inadequate to protect the public health and safety.
UCS urges the Board to reject it and to order that Licensee may not restart TMI-l until it has submitted and implemented a plan that meets UCS' objections.
Respectfully submitted, Md' I
/
William Jordan, III HARMON, WEISS & JORDAN 2001 S Street, N.W.
Suite 430 Washington, D.C.
20009 (202) 328-3500 Date:
June 17, 1985
- - - - - - ~ - -
a
.\\
UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION BEFORE THE ATOMIC SAFETY AND LICENSING BOARD In the Matter of
)
METROPOLITAN EDISON COMPANY
)
Docket No. 50-289 U
C
)
(Restart Remand on (Three Mile Island Nuclear
)
Management)
Station, Unit No. 1)
)
'85 JlN 18 A10:42
)
'C F F'C E 0 F S E P T '" ~
CERTIFICATE OF SERVICE C ' I['U ^ $
I hereby certify that copies of "UCS' COMMENTS ON LICENSEE'S PLAN FOR POST-TRAINING EVALUATION" in the above-captioned proceeding have been served on the parties identified in the attached Service List by deposit in the United States mail, first class this 17th day of June, 1985.
' A b ~S D~e William S g ordan; ~ III
e UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION BEFORE THE ATOMIC SAFETY AND LICENSING BOARD In the Matter of
)
)
METROPOLITAN EDISON COMPANY
)
Docket No. 50-289
)
(Restart Remand on (Three Mile Island Nuclear
)
Management)
Station, Unit No. 1)
)
)
SERVICE LIST Administrative Judge Docketing and Service Section Gary J.
Edles, Chairman Office of the Secretary Atomic Safety & Licensing Appeal Bd.
U.S.
Nuclear Regulatory Commission U.S.. Nuclear Regulatory Commission Washington, DC 20555 Washington, DC 20555 Administrative Judge Jack R.
Goldberg, Esq.
John H.
Buck Office of the Executive Legal Dir.
Atomic Safety & Licensing Appeal Bd.
U.S.
Nuclear Regulatory Commission U.S.
Nuclear Regulatory Commission Washington, DC 20555 Washington, DC 20555 Administrative Judge Deborah Bauser Christine N.
Kohl Shaw, Pittman, Potts & Trowbridge Atomic Safety & Licensing Appeal Bd.
1800 M Street, N.W.
U.S.
Nuclear Regulatory Commission Washington, DC 20036 Washington, DC 20555 Administrative Judge Mrs. Louise Bradford Ivan W.
Smith, Chairman TMI Alert Atomic Safety & Licensing Appeal Bd.
1011 Green Street U.S.
Nuclear Regulatory Commission Harrisburg, PA 17102 Washington, DC 20555 Administrative Judge Joanne Loroshaw, Esquire Sheldon J.
Wolfe The Christic Institute Atomic Safety & Licensing Appeal Bd.
1324 North Capitol Street U.S.
Nuclear Regulatory Commission Washington, DC 20002 Washington, DC 20555 Administrative Judge Mr. and Mrs. Norman Aamodt Gustave A.
Linenberger, Jr.
200 North Church Street Atomic Safety & Licensing Appeal Bd.
Parkesburg, PA 19365 U.S.
Nuclear Regulatory Commission Washington, DC 20555
.s Lynne Bernabei, Esq.
Government Accountability Project 1555 Connecticut Ave.
Washington, DC 20009 Michael F.
McBride, Esq.
LeBoeuf. Lamb, Leiby & MacRae 1333 New Hampshire Ave.,
N.W.
- 1100 Washington, DC 20036 Michael W.
Maupin, Esq.
Hunton & Williams 707 East Main Street P.O.
Box 1535 Richmond, VA 23212 Thomas Y.
Au, Esq.
Office of Chief Counsel Dept. of Environmental Resources 5050 Executive Houses P.O.
Box 2357 Harrisburg, PA 17120 Bud Ris, Executive Director Union of Concerned Scientists 26 Church Street Cambridge, Massachusetts 02238 Bob Pollard Union of Concerned Scientists 1346 Connecticut Ave.,
N.W.
Suite 1101 Washington, DC 20036
-