ML20127E565
| ML20127E565 | |
| Person / Time | |
|---|---|
| Issue date: | 06/18/1985 |
| From: | Advisory Committee on Reactor Safeguards |
| To: | |
| References | |
| ACRS-T-1420, NUDOCS 8506240587 | |
| Download: ML20127E565 (103) | |
Text
.
ORIGINAL d**eo' UNITED STATES OF AMERICA f
NUCLEAR REGULATORY COMMISSION QJ In the matter of:
ADVISORY COMMITTEE ON REACTOR SAFEGUARDS Joint Meeting of the Subcommittees on Human Factors and Maintenance Practices and Procedures Dccket No.
O Location: Washington, D.
C.
Date: Tuesday, June 18, 1985 Pages: 1
.99
- . r
,c,v.
f.
s0 \\,0",18ti10V8 f 0m A, L,!u b,,eA
~
ANN RILEY & ASSOCIATES Court Reporters Q,
'fS OL l
1625 I St., N.W.
Suite 921 dj g Washington, D.C.
20006 (202) 293-3950 i
8506240587 850618 PDR ACRS PDR T-1420
1 1
UNITED STATES OF AMERICA b
t 2
NUCLEAR REGULATORY COMMISSION L
3 ADUISORY COMMITTEE ON REACTOR SAFEGUARDS 4
JOINT MEETING OF THE SUBCOMMITTEES ON 5
HUMAN FACTORS AND MAINTENANCE PRACTICES AND PROCEDURES 6
7 Room 1046 8
1717 H Street, N.W.
9 Washington, D.C.
10 Tuesday, June 18, 1985 11 The Subcommittee on Human Factors and the 12 Subcommittee on Maintenance Practices and Procedures met, 13 pursuant to notice, at 1:00 p.m.,
David Ward presiding.
s 14 PRESENT:
15 D.A.
WARD, Chairman 16 W.
KERR, Member 17 C.
WYLIE, Member 18 G.A.
REED, Member 19 K.
GIMMY, ACRS Consultant 20 K.
EATON, Consultant 21 22 23 24
\\~
25
l i
2 1
ALSO PRESENT:
2 J.O.
Schiffgens, Staff Engineer 8
D.
KLEINKE 4
A.
MASCITTI J
5 6
I 7
8 9
10 11 l
12 l
13 14 a
15 16 17 18 19 20 i
21 22 23 24 25 1
i
,,.,,.-o_.m,----.
,m-
- - - -,, - - - -, -. ~, - -. -.
- - - - -, - - - +, -
--w--
S 1
P ROC EED i NGS t
i 2
(1:05 p.m.)
3 MR. WARD:
The meeting will now come to order.
This 4
is an open joint meeting of the ACRS Subcommittees on Human 5
Factors and Maintenance Practices and Procedures.
6 i cm David Ward, the subcommittee chairman of the 7
Human Factors Subcommittee.
8 Other ACRS members present are William Kerr, Glenn 9
Reed, and Charles Wylie, and we expect Kris Gimmy, a 10 consultant for the ACRS to be here presently.
11 The purpose of the meeting is to explore the use of 12 natural aptitude selection procedures, tests, and
(})
/
13 evaluations.
Presentations will be made by Dr. Alfred 14 Mascitti of the Wisconsin Electric Power Company and Dr. David 15 Kleinke of the Edison Electric Institute.
He will discuss the 16 history and objectives of selection testing and development, 17 validation, implementation of eel employment selection testing 18 projects, in particul'ar the Plant Operator Selection System or 19 POSS project and the Power Plant Maintenance Positions 20 Project He will also present data on the current use of eel 21 tests in the nuclear industry.
22 Herman Alderman, on my right, is temporarily the 23 assigned ACRS staff member for the meeting.
We expect John
[ h 24 Schiffgans shortly.
\\
%J 25 A transcript of the meeting is being kept, and I i
4 1
request that each speaker first identify himself or herself
!N V) 2 and speak with sufficient clarity and volume, so that he or 8
she can be readily heard.
We have received no written 4
statements from members of the public, nor have we received 5
any requests from members of the public for time to make oral 6
statements.
7 Unless any of the other members would like to 8
comment at this time, we will go ahead with the agenda, and I 9
will ask Dr. Kleinke to start off.
10 MR. KLEINKE:
Thank you, Mr. Ward.
11 I am David Kleinke, Manager of psychological 12 Services at Edison Electric Institute.
Thank you for the Ok.-
18 opportunity to discuss employee selection and testing.
As a 14 measurement psychologist, I welcome the opportunity to telt 15 the testing story, to relate the benefits of testing for 16 employee selection, as well as the limitations of tests.
17 Dr. Mascitti and I will describe selection testing 18 with special emphasis on the Edison Electric Institute testing 19 program.
Edison Electric institute or eel is the association 20 of investor-owned electric utilities in the United States.
21 eel member companies generate approximately three-quarters of 22 the nation's electricity.
28 I will describe the beginnings and organization of 24 eel testing and then move on to the description of how 25 employee selection tests are developed and put into use.
5 t
1 Dr. Nascitti will describe the eel plant operator
,s g
- I J
~#
2 selection system ni great detail, drawing on his experiences 8
at (disconsin Electric Company and as a member and now Vice
)
4 Chairman of the eel personnel Research Task Force, which has 4
5 guided these projects.
6 As you will see, we both have prepared material for 7-this aft.ernoon's presentations, but we expect that you will 4
8 ask questions as they come up.
stage with some hi& gory.
9 I must first set the 10 During the early to mid-1970s, a number of electric utilities 11 and other companies in o t h ee-Industries were growingly 12 dissatisfied with their employee selection procedures.
Many 18 of these employers were using only interviews and reviews cf 14 applications to decide which applicarits among several should 15 be hired.
Some companies had been using tests, but had 4
16 abandoned them.
Others had no recent history of testing for 17 selection, transfer, or promotion.
18 The dissatisfaction with existing selection i
19 procedures was probably felt most strongly in nuclear 20 operations.
As you well know, the training of a nuclear 21 operator is a long and expensive process.
If a procedure 22 could increase the probability that an Individual would l
28 complete the training, it would mean that fewer people would i
(
24 be needed at the outset.
25 An early attempt initiated by a prospective
6 1
contractor at organizing a consortium of utilities to develop j_
\\
2 a selection battery for nuclear operations failed.
It failed 3
because what was proposed would have been very expensive and 4
because the contractor was vague as to the project's outcome.
5 However, a group of electric utility company 6
psychologists and human resource generalists persisted in 7
pushing the r.o t i on that consortium-sponsored enployee 8
selection projects would be cost-effective.
9 Before going on, I am going to establish some basic 10 definitions.
For many of you, this covers familiar ground, 11 and I beg your indulgence, but we are discussing mwployee 12 selection tests.
13 First, a test we define as a systematic procedure 14 for quantifying some aspect of a person's makeup.
It is 15 systematic in the sense that its conditions of administration 16 and scoring are s t andardi =ed,
insofar as is humanly possible, 17 all examinees are presented with the same tasks.
They are 18 given the same or equivalent questions and have the same time 19 and opportunity to respond to them.
These responses will 20 differ, of course, but the same response will yield the same 21 score, no matter who scores them or when.
22 These scores are expressed numerically.
This 28 quantification allows us to make comparisons along examinee p
24 performances.
25 The subject matter of tests varies greatly.
I
7 1
define a test as measuring some aspect of a person's makeup.
\\\\
2 These aspects could be subject matter which is learned 8
through formal or informal instruction.
We dub such aspects 4
of these as achievements.
Examples would be physics or 5
accounting or principles of supervision.
6 Other aspects underlie these learnings and are 7
considered to be aptitudes.
Verbal ability, mechanical 8
comprehension, and spacial visualization are examples of these 9
aptitudes which underlie the eventual achievements.
10 Selection testing also measures personal traits, 11 such as adaptability to working alone, mechanical interests, 12 and self-control.
eel tests emphasize aptitude, with some D)
(J 18 measures of personal characteristics where appropriate.
14 Two or more tests given together for the same 15 purpose constitute a test battery, just as a storage battery 16 is a collection of cells.
17 Dep l oyee selection tests are used for hiring people 18 into the company, for transfers, and for promotions.
They are 19 not properly used as and-of-training tests, in most cases, 20 the achievement tests which we would administer at the end of 21 trainin0 could not be used for initial selection into the job.
22 Two concepts which are central to the understanding 23 of tests and their use are those of reliability and validity, t
24 I have already alluded to the consistency of test results, no 25 matter what the conditions of scoring and administration.
8
,es 1
This consistency of results, as well as their stability across (d
i
\\
2 time, is called reliability.
3 Validity is the accuracy of the test for its 4
intended purpose.
A useful synonym for validity when 5
discussing employee selection tests is job-relatedness.
A 6
tect is valid or job related when it accurately predicts job 7
performance or when it measures the personal traits needed for 8
the job and does so in proper proportions.
9 CSilda.]
10 MR. WARD:
I will welcome Mr. Gimmy to the table.
l 11' MR. GIMMY:
Thank you.
12 Epause.]
(
18 MR. KLEINKE:
Thank you for bearing with us.
14 Let me continue by describing the eel testing 1
15 program.
The leadership of eel in the 1970s became convinced 16 that they should sponsor the development and validation of 17 employee selection tests on a voluntary basis.
Member 18 companies were surveyed to discover which job families could 19 be best served by these tests.
(
20 CSlide.]
j i
21 The results of the survey confirmed that company 22 management felt the need for such testing projects and would 23 contribute to them.
i 24 We have seven projects.
They are here arranged in 25 roughly descending order of interest for nuclear power
F 9
1 generatlon.
b l
2 At the top of the list is the eldest and largest of 8
the eel testing projects, called the Plant Operator Selection 4
System, with the convenient acronym of POSS.
Dr. Mascitti 5
will be describing POSS in great detail I will only state 6
that it involves 70 eel companies and is intended for the 7
selection of operators for fossil and hydroelectric plants as 8
well as for nuclear plants.
9 MR. WARD:
When you say it involves 70 companies, 10 you mean 70 companies are using POSS, is that what that means?
11 MR. KLEINKE:
Seventy-three companies contributed to 12 the initial development of pOSS, of which two have caused 18 membership in eel, and one has merged with another.
So they 14 were original participants.
15 Approximately 55 to 57 are using it currently, 16 41though of these only 20 are using them for nuclear 17 p >s i t i ons.
,10 MR. WARD:
I see, okay.
So when you talk about 19 pOSS, it's described as a system, and I guess 1 thought of it 20 in tnrms of a test.
But it's more than that apparentlys is 21 that it?
22 M3, KLEINKE:
It is really a set of test batteries.
23 There are three different batteries of tests, each of which l ()
24 can be used interchengeably for fosall, nuclear, or 25 hydroelectrie, using different weightings of the scores, and
10 1
is called a system, principally to provide a convenient 2
8 MR. WARD:
I see.
Okay.
4 MR. KLEINKE:
It was originally POST, but the 5
contractor that plant Operator Selection Tests was trivial, 6
and thus dubbed it pOSS at the last minute.
7 MR. WYLIE:
Would you repeat how many utilities are O
using that?
9 MR. KLEINKE:
Approximately 55 to 57 utilities are 10 using it for one fuel or another.
Twenty in 1984 for nuclear.
11 MR. REED:
Since our focus is nuclear, you said 12 twenty nuclear companies are using it.
Are you saying --
18 well, what percentage is that of the nuclear utilities?
Is 14 twenty a large percentage?
15 MR. KLEINKE:
Of the nuclear utilities who are eel 16 members, it's roughly half.
There are 39.
17 MR. REED:
And you think they are actively using the 18 tests?
For what?
New hires or all transfers or what?
first of all, 19 MR. KLEINKE:
For the most part 20 they are actively using them.
I did not, for instance, 21 include in the twenty Wisconsin Electr6c, because in 1984, as 22 1 understand, Ulsconsin Electric did not use POSG, simply 23 because they didn't hire anyone.
They did in 1983 and 1992,
(
(
24 however.
So they didn't use the '84 list.
25 They are using it mostly for new hires s in some
11 1
cases, for internal transfers, opening the nuclear plant 2
operator trainee job to internal either bidders or appliers, 8
depending on their situation.
4 MR. MASCITTI:
Typically the pOSS is used for 5
entry-level selection.
And most of the companies are using it 6
for hiring individuals off the street or coming from previous 7
experience.
It can be used, and in some cases it is used in a 8
promotional sense.
For example, if a plant operator or 9
auxiliary operator is being considered for a control room 10 operator position.
11 MR. REED:
If you go back to your history you 12 discussed, this 50 percent that you're saying is perhaps the 18 case for new hires for nuclear utilities.
Has that been on a 14 steep, rising curve of percentage?
Could we say that in maybe 15 1977 that was 10 percent or no percent?
16 MR. KLEINKE: *Using pOSS?
17 MR. REFDs Using that or a near equivalent.
18 MR. KLEINKE:
I can't speak to what was in use in 19 1977, because my direct experience with the industry began in 20 1981.
I know some stories.
21 MR. REED:
Well, my impression would be that it was 22 a very small percentage of utilities in nuclear that were 23 using any kind of selection testing in, say, the late '70s.
/~'\\
24 la that your feeling, Dr. Nascitti?
25 MR. MASCITTir Yes, although I don't know for sure.
l l
l f
l 12 l
l 1
MR. WARD:
Well, how long has pOSS been available?
l 2
MR. KLEINKE:
Since early 1982, it was first given l
8 in three companies in January and February of
'82.
1 4
MR. WARD:
Were there some predecessors to pOSS?
5 MR. KLEINKE:
No.
6 MR. WARD:
This was an eel program that just 7
started?
8 MR. KLEINKE Started'from scratch, it had its l
9 official implementation in September through December of
'91, 10 but materials were not available and in use until February of 11
'82.
12 MR. WYLIE:
You mean as far as eel is concerned?
13 Individual utilities have had programs.
14 MR. KLEINKE Had programs for selection.
And there 15 were other programs, notably those that met the state and that 16 of general physics where tests were available.
17 MR. WARD:
Well, maybe you will get to this later, 18 but the other half of Edison Electric's nuclear utilities, do 19 you have any idea of what they are using?
Or maybe you are 20 going to get to this later.
I will just walt.
Why are they 21 not using, pOSS, for example?
I want to get at that sort of 22 thing.
.!S MR. KLEINKE Earlier today, Dr. Nascitti and I went I believe 1 24 through it on a case-by-case basis, and I have
(
25 have a good understanding of what they are using.
n_-_____---_____________._
r 13 1
MR. WARD:
All right.
Well, I'll wait until that 2
fits in somewhere, if you want.
3 MR. KLEINKE:
- Okay, in most cases, they arn using 4
some alternative form or simply not hiring people at the 5
moment, or were not in
'84.
6 MR. WARD:
So what you are saying, then, is that 7
most utilities are using something or other along the lines of 8
POSS7 9
MR. KLEINKE:
Most of our member utilities are using, 10 some form of selection testing, and I just have no way of 11 speaking for non-member utilities.
12 MR. WARD:
But is all of this of fairly recent 13 origin?
Is something of the '80s, or does this go back to the 14
'70s?
15 MR. KLEINKE:
I would imagine to the late '70s, i
16 don't know, for instance, when the program at Memphis State 17 Canter for Nuclear Studies began, a program that, as i 18 understand it, effectively ended last December.
19 The power plant Maintenance positions Project, 20 interestingly for acronym hunters, originally was called MAST, 21 and the same contractor decided at the last minute that he 22 wanted to call that MAGS, but to a psychologist, mass testing 23 is rather different from MAST testing, so l*ve resisted that.
24 The Power Plant Maintenance Positions Project is 25 relatively now.
We formally implemented that in October of
14 1
1984.
The positions covered by this project include 7
t 2
electricians, machinists, mechanics, instrument and control S
repairers and others in nuclear, fossil, hydroelectric plants 4
and on traveling crews maintaining these plants.
5 Now in this project, contrary to our pOSS 6
experience, we found that we could use the same weighting 7
of component tests across fumis, across types of pIants.
For 8
pOSS, we used difforent weightings for the components, for the 9
operators, but for the maintenance workers we found 10 essentially an electrician is an electrician is an 11 electrician, no matter where he or she works.
12 The maintenance test battery does use essentlally i
k 18 the same aptitude tests as the plant operator test, but has a 14 different measure of personal characteristics.
15 MM. WARD:
Well, how do you decide that it's 16 adequate to use the same weightings?
I mean, there is 17 validation testing?
10 MR. KLEINKE Yes.
All derived empirically, based 19 on tryouts with incumbents, and found in the job and task 20 analysis phase that similarity in duties across fuels and in 21 the emperimental phase, the similarities in performance across 22
- jobs, l
23 MR, REED:
- Well, I'm a little surprised by that i
24 statement.
I certainly thought that you could say, well. 00 25 percent of the skills, maintenance skills, are transferable
. ~. _ - - - _ - -
.,n-
l 1S l
1 and the same.
But isn't there a personality aspect that's
)
2 more important in nuclear?
~
8 I have always been 100 percent for honesty as the 4
most important characteristic of a person working in a nuclear that is, forthright honesty to come forward with 5
plant 6
things that he or she may have done wrong or anything like 7
that.
And it seems to me that there might be some personality 8
differences, and you said they're essentially the same, g
The major personality differences we found were 10 those which distinguished workers from operators, and they 11 could not use the same personal characteristics battery.
But 12 we did not find, in terms of job duties and in terms of the 13 desirable characteristics, differences in pattern across 14 fuels.
15 Now, it could well be and I don't have the data it could well be that for 16 that close to the top of my head 17 these characteristics, people in nuclear might be a half-step 10 higher or conceivably a half-step lower than their 1g counterparts in fossil.
What I was looking at and what 20 genorated that statement was examining tables of correlation 21 coefficients to find similarities in job tasks.
22 We found, for instance, 14C repairers most nearly i
23 resemble electricians, and that emerged looking at correlation 24 matrices which were unlabeled with respect to -- you know,
{
25 where is job 8 and job 11 rather than going in with the i
f 10 1
preconceived notion of, oh, yes, we are going to look at that.
,s
)
2 MR. REED:
How about paper?
Did you try to use a 3
pencil?
Certainly in nuclear, people are stuck with more 4
proceduralization on paper and desire to use a pencil than 5
they would be in fossil.
6 MR. KLEINKE I don't have data to support that, it 7
could be a weakness in the data.
8 MH. WARD:
Now, when you say that in nuctvar plants, 9
Individuals might be a half-step higher or lower in some 10 particular thing, I don *t quite understand what that means.
11 Somehow, a different slice of the population is attracted to a i
12 nuclear plant, or what is the significance of that?
(
18 MR. KLEINKEt I was referring to the job analysis 14 phase of the maintenance project in which with a questionnaire 15 we asked people essentially: What do you do for a living?
16 What are the important activities --
17 MR. WARD:
Oh, So you are saying those are the 18 characteristics of the job, not of the people.
19 MR. KLEINKE:
That's quite true.
The 20 characteristics of the job.
And find that.
21 MR. WAND:
There are some modest differences across 22 fuels, you are saying, maybe.
23 MR. KLEINKE The differences across occupation are 24 far more striking than those across fuel.
25 MR. WAND Okay.
Job characteristics, not people.
I
i l
17 1
MR. KLEINKE:
Right. Back to an electrician is an l
l 2
electrician is an electrician.
And the contractor did perform 3
the analysis to try to tease out the fuel differences while r
4 holding occupational differences constant, and it didn't 5
work.
The crafts do differ that much.
That is one of those 6
things that 1,
at least, looked -- after I was done, 1 7
scratched my head and said that's the way it should be and i
8 felt satisfied.
I l
l 9
Now, in addition, the power plants maintenance jobs 1
l l
10 are among three job groupings of jobs for which companies use 11 the physical abilities test.
The physical abilities test is 12 the major product of the physically demanding jobs projec'.
l
(
13 The other two groups of jobs, for your information, are 1
14 overhead and underground line work and substation construction 15 and maintenance.
I l
16 The physical abilities test measures arm strength, j
17 arm speed, equilibrium and flexibility, e
l I
18 There are two projects for selecting first line 19 supervisor, one for blue collar jobs and the other for office 20 and technical jobs. They are unique among the eel projects in 21 that we adopted existing programs for these projects while 22 other eel projects are custom developed.
I 23 Clerical positions and meter reader were grouped I
24 Into one project because we initially felt that meter readers l
25 and clerical workers needed about the same knowledge and
(
18 1
skilis and abiiitles.
That turned out to be only partlally m
i
)i t
2 true, so there are overlapping test batteries for the two a
types of positions.
4 MR. WARD:
Meter readers have to know how to get 5
along with dogs, don't they?
6 CLaughter]
7 MR. KLEINKE:
We did not have a get-along-with-dogs 8
subscale, but if we can develop one, I'll be willing to trot 9
down the street and sell it to the postal Service.
l i
10 That was interesting because in some cases the meter 11 reader has also scoe clerical duties, so they and up having to 12 take both batteries.
But that has a parallel in that an b
18 applicant for either a maintenance job or an operations job, i
14 someone cceing in fece the street, from outside the ccepany, 15 who says I want to work in a power plant, and he or she is 16 asked do you want to be a maintenance worker or an operator 17 and they say yes, well we have the mechanism through that 19 ccamon aptitude battery to be able to administer a shorter 10 battery that, had we had to ccebine two separate ones -- so we that's not econcey of scale s that's J
20 do get some econcey of l
f 2i econcey of ceabination, I guess, thereby.
22 F* i n a l l y, the system operator / power dispatching i
28 positions project developed a battery for the selection of i
I P4 energy control center employees.
Each of those projects t
A l
25 etands as a separate ontIty.
Each was a diiferant consortlum
_,____.._,-___.___,._.,,-.,-r,.
,____.___-_._.x.._____
19 p
1 of companles developed by differant contractors, aithough N
2 system operator and physica1 abilitles test were developed by 3
one contractor, and plant operator and maintenance by another 4
contractor.
5 The i ndustry Testing implementation project brings 6
all of those specific projects together.
I have already noted l
7 that consistency of administration and scoring is a e
fundamental requirement for test rollability.
eel tests are 9
administered in ccepanies by company staff.
These companies 10 vary widely in size and geographic location and in their 11 exporlance with testing.
12 Therefore, we needed some mechanisms to ensure the l
\\
13 uniformity of testing and scoring procedures as well as of the 14 security of the tests themselves as well as the test scores 1
15 and results.
16 The Industry Testing implementation project provides 17 these mechanisms eel trains and certifies a manager in each 10 company to be todting coordinator. eel standards also require i
19 that each company name a policy level official to bear overall 20 responsibility for eel testing within the ccepany and a 21 testing speciallet, a psychologist to advise the others on 22 technical and ethical issues.
28 This structure was devised under the Industry I
l 24 Testing implementation project, as well as the training j
25 outiIne for the testing coordinators and the test
{
l i
l
.--n-,_
--m.m-_
_.-_.-__-._.-.---.-.,-..-w_
20 1
administrators.
2 MR. WARD:
Could I ask you a question about that?
8 So you require that if a company is going to use a test, they 4
have a resident psychologist on the staff?
5 MR. KLE I NKE :
That they have a psychologist 6
available to them.
The person may be a consultant.
7 MR. WARD:
What sort of professional level e
psychologist are you talking about?
Would a ph.D. be required g
for that job?
10 MR. KLEINKE:
No.
And L should, strictly speaking, 11 be saying a person with psychological training because wo 12 recommend a minimum of a master's level training in either
(
18 Industrial organizational psychology or in measurement and 14 statistics, and two years in employee selection testing.
15 Now, on one hand, this does not meet many state 16 requirements for the right to call oneself a psychologist in 17 that it doesn't require a doctorate, but on the other hand, it 10 le more specific in requiring specific training and experience lg with employee selection testing as opposed to counseling or 20 clinical psychology or learning.
21 If I may, I know that we are unique as an 22 association project in that we do require these actions of 23 companies.
That was, frankly, somewhat revolutionary, that 24 here companies had contributed money and a great deal of time 25 and staff effort to develop these projects, and we turned
21 1
around and said, now, if you were willing -- or now you must 7_
I
/
2 do this, and then we will let you buy the test you have 3
already paid to develop.
4 That was unique.
Another industry had developed 5
clerleal tests for its employees, and legend has it that 6
within a year they were worthless because they had all been 7
copied and distributed very widely among potential examinees.
O We had no intention of going that route.
9 The implementation project developed the training 10 outline for test administrators as well as f or testing 11 coordinators.
These test administrators are the people who 12 actually give the tests, in my part of the country where I'm
("'\\
( j 18 originally frem, we called them monitors, but they are also j
14 called test proctors or test administrators, who do the actual 4
it:
testing and scoring.
16 These people are trained and supervised by the 17 testing coordinator, the person who has received the week-long 10 training from eel.
19 We also provide additional services to participating 20 ccepanies under the testing implementation project.
First was i
21 the coordination of test distribution. We have one central 22 supplier for secure testing materials, and only the certified 23 testing coordinator can obtain the necessary supplies.
We 24 also coordinated the format design of the tests themamlves to l
25 allow for scoring using electronic scanning equipment.
22 1
Essential data storage facility at eel is currently f
(
2 being developed to permit continuing research into the a
characteristics of the tests and of their results.
e 4
Finally, the Industry Testing implementation project 5
provides legal assistance to companies.
6 All of this is financed by the participating 7
companies.
They initially contributed to the development and 8
validation of the projects. eel also receives royalties from 9
the sale of testing materials.
These royalties support the 10 continuing research and administrative efforts needed to 11 maintain the projects.
12 MR. WYLIE:
Let me ask a question.
Where would la engineering support personnel fall in these tests?
Would that 14 be under NAST, I suppose?
15 MR. KLEINKE I'm sorry.
I'm not sure i know what l
16 you mean by engineering support personnel.
17 MR. WYLIE Well, all utilities either have 10 employees or engineering department support or the plants have 19 engineering --
well, they are degreed engineers.
20 MM. KLEINKE Engineering toch draftspeople?
They l
l 21 would not be covered by any of these.
22 MM. W4MDs Which, the professional engineers? What 23 about draftsmen or technicians?
24 HM. KLEINKEr No.
/
25 MM. WYLIEi For support of the plants?
l
l l
23 i
(
1 MM. MASCITTI:
Only the Instrument control 7g
()
2 technicians would fall under the maintenance project.
3 MM. WYLIE:
Only in l&C?
Mechanics would not?
4 MM. MASCITTl:
Oh, plant mechanics would.
i 1
l 5
MM. KLEINKE:
Of the people who can be called 6
engineering support, and you know, again, I am personally l
7 fuzzy on who would be -- what titles would be engineering 8
support, but Dr. Mascitti makes the point that instrument i
9 control repairers are covered under the maintenance, but not, 10 for instance, health physics technicians or chemical 11 technicians.
{
12 MM. WYLIE Okay.
Well, some utilities give a O_j 1
18 battery of tests to all technical personnel when they screen 14 them.
l 15 MM. WAMD:
Charley, you mean degreed personnel?
16 MM. WYLIE:
No, all technical personnel, 1
1 17 MM. WAMD:
Including degreed personnel?
IG MM. WYLIE Oh, yes, including degreed, i
tg MM. WAMD' I think specifically they are excluding 20 degreed personnel j
l 21 MM. WYLIE:
Okay, but basically these tests here are 22 not aimed at other technical personnel such as design people.
)
23 MM. KLEINKE:
That's correct.
There are, in fact.
24 many jobs within the electrical utility industry that are not j
i 25 covered here.
Note tnat overhead and underground line l
I i
-. - -. _... _. _.. _ ~,... -. _ _ - _ _ - -. _ _. _. - _ -. _ _,, _ _..
,,____.m,,--___.
l 24 l
1 workers, except for the physically demanding aspects of that g-~
2 job, there is no knowledge or abifities test other than the 8
physical for line workers, 4
Let me next turn to the steps in the development and 5
validation of employee selection tests.
Even a simpilfied 6
overview of these steps appears to be complex, 7
CSilde]
8 I must say the main purpose of this slide, other 9
than to give an overview, is to say it i s complex.
The goal 10 of every test developer is to produce the most valid test or 11 battery possible given the practical limitations of the 12 project The outline you see is basic for the development and b)
(_,/
18 validation of a large-scale test battery for employee 14 selection.
15 MR. KERR If you were to put this on a test and if he said yes, you would 16 say, "Can you read that?"
17 know that you should question his truthfulness.
18 MR. KLEINKEr My experience is that i expect the 19 medical departments in the companies would suggest that I was 20 encroaching on their territory.
I think of it, actually, as 21 the schematic for R202.
22 HR. KERR In any slide presentation, one should 23 always have one slide that i s c omp l e t e l y illegible.
()
24 MH. KLEINKE I understand that principle, which is v
25 why four alldes from now we are repeating this one.
--T 1
- m--
r-me---
-~
w a
'~--+-m
m---------r----'w
- - ' - - ^ - - - - - - - -
25 i
1
[Silde]
(\\~
2 is that more legible?
Okay.
3 The first step is to identify the target jobs.
This 4
la simple in most cases.
The project sponsors know exactly 5
which are the target jobs for which tests are needed, power 6
plant operator jobs constitute one such well-defined set of 7
positions.
Other instances are not as simple.
The physically 8
Demanding Jobs project limited itself to the most populous 9
physically-demanding jobs within the electric utility industry, 10 rather than to cover them all.
11 Having identified target jobs, the necessary next 13 step is to review the literature on other selection procedures 13 for those and for similar jobs.
For instance, the contractor 14 for the plant operator selection system reviewed the 15 literature not only for power plant operators but also for all 16 continuous process manufacturing jobs, to find out what 17 approaches had worked or what approaches had failed to work to for others in the past.
19 We got into an interesting bind here in that 20 generally psychologists, particularly if they are 21 academicians, have a compelling need to pubilsh results and 22 share with others.
On the other hand, testing people are, of 23 their nature, secretive and semi-paranoid about the security
'}
24 of our tests, particularly when those tests and projects are x'
25 sponsored by industries.
i 26 1
So therefore, we have this balance, and it is
/
t 2
sometimes difficult to get past information, and one finds 8
himself reinventing the wheel.
Fortunately in these projects 4
we had a good tradition of projects done on a 5
company-by-company basis and we were able to get free member 6
companies, particularly those with psychologists, we were able 7
to get from these member companies their previous studies, 8
something normally we don *t have tFe ability to do.
9 Review of these previous studies helps shape the 10 next step, the Job and Task Analyses. These analyses are 11 formal investigations of, first of all, which tasks are 12 frequently performed in and/or are very important to the jobs
(
19 frequent performance and important. And secondly, what are the 14 kncwledges, skills, abilities and other characteristics 15 necessary for entrance to the job.
16 These analyses when done for employee selection 17 testing are simpler than the same job and task analyses that to are done for training purposes.
Test developers need only 10 know the entrance skills, not the skills necessary for 20 independent functioning on the job. We have an instance in the 21 development of the Maintenance Project, for instance, where we 22 were about to put out on the street or in the industry, if you 29 will, a job and task analysis questionnaire to maintenance 24 workere.
s 25 We knew through the ongoing communication we have
27 1
with INPO that their training people were going to put a job l,_ 't sl 2
analysis out with target population in about the same month, 3
and there was a real danger that a given individual would be 4
caught in both samples and respond to similar-appearing 5
questionnaires, probably ones with the same questions.
6 We were thus able to coordinate with INp0 and sample 7
on an exclusive basis so that no one individual got hit with O
both questionnaires.
Also, INPO hadn't selected a contractor 9
at that point and felt that there was great advantage to 10 selecting as their contractor for this the same one we were 11 using so that they wouldn't have to pay for the duplication of 12 skills or for cccciun i ca t i on.
13 1 think that is a fine example of how cooperation sJ 14 among groups such as eel and INPO can cut down on extraneous, 15 unnecessary demands on employees.
10 MR. REED:
You made a very significant statement 17 just back a little bit, and I think I agree with it, but I was 10 sort of surprised.
You said that you didn't need great detall 19 in job task analyses that had been rehashed and rehashed and 20 that relate to training more than something else, but you can 21 do it from a lesser analysis where it involves attributes and 20 skills. It's a very significant statement.
I appenclate the 29 statement,
/}
C4 MM. KLEINKEi Thank you.
~J 25 Frequently we are askedi May we see your
28 i
1 information?
The other side of that coin is people say: We 73 i
\\
\\,j 2
want to use your job analysis report so we can design a 3
training program.
Now, our response to that is that's fine if 4
you want to know what the person looks like the first day you 5
begin training, but it's not fine if you want to use this to 6
say here are the things someone needs to come out of training 7
with if it's the normal beginning of the job training.
So 8
that, you know, that's both good and bad for its own uses.
9 Typical job and task analyses use questionnaires --
10 MR. WARD:
I'm sorry.
Could we go back to that a 11 little bit?
I'm still kind of struggling with this idea that i
12 the job and task analysis is -- I mean I can see where you
/~r) 18 would probably have for this purpose something simpler than 14 what you would want for training.
You don't have as much 15 time.
But they say it is pointed toward what the person is 16 going to be required to have when he enters training.
17 i don't quite follow that. I guess what you are 2
18 -
doing is separating out the trainable features or somethings st 19 is that it?
I am having trouble with the concept of a job and 20 task analysis for'somebody entering training.
I mean he is 21 not go i r.g to do any work; he is going into training. It's not 22 a different job.
28 MR. KLEINKE:
But we find that, for instance, a 24 maintenance worker needs to be able -- one of the things a
/','
Nu 25 maintenance worker does is read exploded diagrams, and on the
29 1
basis of the initial job analysis, which says " reads exploded 2
diagrams," then a task analysis reveals "needs to be able to 3
visualize three dimensions represented in two dimensional 4
space."
Okay, fine.
Then we know for testing purposes we are 5
probably going to end up with a measu e of spatial relations 6
in which some three-dimensional figures are represented in two 7
space.
S For the training person, though, that becomes a need 9
at the end of the course, "must be able to read an exploded to diagram or blueprint, perhaps with specific notations, 11 lettering and labeling conventions on them."
Then, if I am 12 not drawing the analogy too thinly, the purpose of training is (D
\\ )
18 to take the person who has the spatial ability to start with 14 and to say, okay, on the standard diagram, this is where you 15 look for the scale.
This is how you tell what is brass and 16 what is cast iron.
17 MF:
WARD:
Okay. So the job and task analysis you 18 use is more to pick someone who is going to ultimately be 19 trainable, to go to the other job and task analysis, I guess.
20 Okay.
21 MR. KLEINKE:
Yes.
22 MR. MASCITTI What the job and task analysis does 28 use is the participants, the people at the various levels
)
24 right on up through the progression in the family of jobs, so (d
25 we did do a task analysis of all of the represented positions
30 es 1
in the job family, but the focus is to get at the underlying 2
abilities, the function in that job, as opposed to determining S
what are the knowledges that must be taught.
4 MR. WARD:
Okay.
I think i understand.
Maybe the 5
term " job and task analysis," maybe there would be a better 6
description of it or something.
I won't worry about that.
7 Forget that now.
8 MR. KLEINKE:
I could recommend four tests of job 9
and task analysis if I could find two of them to agree with 10 each other.
There is a re!iability issue there, testing 11 reliability, not machine reliability; but the procedures are 12 essentially the same whether we yse it for training or for x
(m, 13 employee selection and testing, and the only way that i know 14 of to go about it reasonably when you have got a large group 15 of people is you first of all sit down in committees with 16 subject matter experts, trainers, supervisors, master level 17 workers and say: Hey, what do you do on this job?
You start 18 hashing them through to develop the initial questionnaires.
which again is a 19 Then the written questionnaire 20 response to what is the job, not what are you as a person, but is almost always responded to on 21 what do you do on the job 22 the basis of both frequency of performance and importance.
23 You know, how often do you do this thing and how important is (n) 24 it to your job?
25 Then the person conducting the analysis must somehow
31 1
reconcile the fact that frequency and importance are obviously 2
not always perfectly related.
Some of the extremely important 8
things we do on our jobs are fairly infrequently done, and 4
some of the very frequently performed skills, of course, are 5
not really all that important or else don't have that serious 6
a consequence of error.
7
[ Slide]
8 The information obtained on the job and task 9
analysis phase is used in the next three, which are 10 approximately simultaneous steps.
These steps prepare the 11 material and the personnel for the tryout of the tests and the 12 gathering of the data for their statistical validation.
The
(~N
()
13 experimental tests in a typical employee selection testing 14 project are either developed from scratch or are adapted from 15 existing test materials or are adopted wholly from existing 16 materials.
17 The first and foremost consideration is that they 18 measure the knowledge, skills, abilities or other 19 characteristics i d e n t i f.i e d at the job and task analysis stage 20 a s. lie i ng important to the job.
Responsible experimental 21 testing uses a variety of instruments.
The appearance of the 22 test is not an infallible guide to the test's usefulness. We 28 need hard data on examining performance on the test.
I have a
[
24 personal graveyard of bad guesses I have made about tests, and 25 especially about specific test questions, over the years.
I
82 1
have got some beauts that I have really gotten very involved
~
2 with and thought they were wonderful that just fell to the 8
ground.
4 in addition to hard data on examinee performance on 5
the test, we also need hard data on their work performance, or 6
at least as hard data as we can get.
This work performance 7
information is the criterion.
Thea criterion is the measure we 8
would really prefer to have for applicants. I would really 9
rather have an applicant come to me with his work performance 10 rating from his first two years on the job.
Of course, that's 11 impossible.
12 But in testing jargon, we talk about
)
\\,/
18 criterion-related validity.
The criterion is the thing we 14 would really rather have, but it is either too expensive to 15 get right now or it doesn't exist right now.
16 Company performance appraisals are poor criterion 17 measures even in a one-company study.
They seldom provide the 18 variability in scores that we need.
I have got to point out 19 that the problems of performance appraisals are by no means 20 limited to the electric utility industry.
Non-variation in everybody is satisfactory, everybody 21 performance appraisal is a problem that is found in all employment 22 is excellent 28 situations.
24 Is this step we selact a sample of examinees who 25 take the experimental tests. This somewhat automatically
33 1
selects the supervisors to rate the examinees' work
_s 2
performance. One of the great advantages of consortium 3
projects is realized right at this stage.
The number of 4
incumbents across all companies is so large that only a sample 5
of these incumbents is needed in each company.
Because much 6
experimental testing is done on an overtime basis, using a 7
smaller sample of incumbents means that obviously you are 8
going to have to pay fewer people overtime to take the 9
experimental tests, so that is a very tangible savings.
10 A not-so-readily-apparent advantage of consortium 11 testing is also available at this point of selecting tha 12 sample.
There is, of course, great social interest in j
13 providing opportunities for ethnic and racial minorities, as s
14 well as for females and non-conditional jobs.
Very few, if 15 any, companies have sufficiently large numbers of members of 16 these protected groups so that we can investigate how the 17 tests work for members of these groups.
18 By pooling 'across companies, we are able to identify 19 sufficiently large numbers in most cases of ethnic or racial 20 minorities or of women for non-conditional jobs to look at how 21 the tests are working for these people.
22 Now, most psychologists and most people experienced 23 in measurement for selecting employees know what the answer to
[~N 24 that is going to be.
It's that the tests work as well for the
(
25 protected groups as they do for whites or for males.
- However, i
84 1
there is continuing strong interest in demonstrating this s
[G'
~
2 validity for the protected groups.
3 it is an interest that has at least certain 3
4 intuitive appeal, although, as I say, the experience generally 5
is that the test works as well for members of protected groups 6
as it does for others.
7 MR. WARD:
But does this become part of a legal 8
justification under Equal Employment Opportunity statutes?
9 MR. KLEINKE:
The Uniform Guidelines in employee 10 selection recommends that this investigation be carried 11 forth.
In testing jargon, that is called unblocked validity 12
-- or differential validity, and the Guidelines recommend that O(,)
18 it be gathered, although the section on Division 14 of the 14 American Psychological Association principles for selection 15 and testing specifically says, hey, it's going to be a waste 16 of time because we know what the results are going to be, 17 So we have somewhat a cross purpose between the 7,7 18 published guidelines and the professional stamp.
We did 19 it. You know, I can't recommend professionally that it is 20 necessary to do because we are going to find, once again,'-that 4
1 21 first-class levers work like this, or some other principly a
gsohld 22 that we knew, but for potential response to queries, I l
N $
23 recommend 5
givenhser,
[V
\\
24 MR. WARD:
Okay.
But I take it that a a
25 utility who is using these tests has to justify tHeir use
35 1
under Equal Employment Opportunity laws, I guess.
Don't
,g
\\h 2
they? And is your testing in this area designed or does it S
help to support that?
Or maybe that's not an issue.
I 4
thought it was.
5 MR. KLEINKE:
Well, you are getting me on the thin 6
ice of practicing law without a license.
However, my 7
understanding, my non-lawyer's understanding of the law is and it's really an ethical issue for a psychologist as 8
that is that Federal statute and case law requires that a 9
well 10 test be professionally validated if there is what's called 11 adverse impact in selection.
12 Now, the other way to say that is if members of 13 protected groups do not have markedly lower rates of hiring 14 than do whites or males, then the test doesn't have to be 15 valid.
That is one way I read Federal law.
On the other 16 hand, as a psychometrician or an industrial psychologist 17
-- I wear the hats interchangeably -- I have a professional 18 responsibility to see that for sure the test is valid, to 19 assure that it is job related, that the people being selected 20 have a better than chance probability of being good employees, 21 if you will, of measuring high.
22 So that I think our professional respons ib i l i ty 23 exceeds that of the requirements, and indeed, current EEO 24 guidelines I'm sorry, current Federal guidelines 25 essentially require what until this month was the American
36 1
psychological Association Standards be followed.
The American
.s U
2 psychological Association revised their standards and 3
published them this month, and I can no longer say that the 4
guidelines the meat of the guidelines is essentially a 5
reprint of the validity section of the ABA standards, and i 6
have not been able to do a comparative reading of the ApA 7
standards with the guidelines to see how they jibe.
8 After suitable training and orientation of the 9
necessary company personnel, we proceed to the simultaneous 10 experimental testing and performance appraisal 11
[ Slide 3 12 Several analyses of the data are performed. The most p
(,
18 basic one is that of the correlation between job performance 14 and test performance. We are looking for tests which correlate 15 most highly with work performance.
The final battery is 16 selected to maximize t h i's correlation and to reflect the 17 knowledges, skills, abilities and other characteristics known 18 to be essential for the job.
19 Preparation of the reports which document all of the 20 foregoing and the testing or the preparation of the testing 21 material itself,omplete the task.
22 CSildal 23 Dr. Mascitti will next tell you about the plant 24 operators license.
25 MR. WARD:
Let me just repeat.
So you don't use
87 1
on-the-job supervisory appraisals at all They are just not i
2 accurate enough to make any use of them?
8 MR. KLEINKE:
We do, but we develop them 4
ourselves. I am delighted you asked that question. I failed to 5
make it clear.
part of each of the eel projects -- and I 6
expect if I were a contractor developing materials for someone 7
else, part of that project would be the development of 8
a performance appraisal.
That performance appraisal is given 9
to at least one supervisor for each emp l oyee.
So there is an 10 on-the-job appraisal, but it is custom-designed for the 11 project rather than using the company form.
12 MR. MASCITTI And typically the amount of work and (Oj 18 development that goes into a performance appraisal that comes 14 out of a validity study is far more extensive than that which 15 is used in the company.
16 MR. WARD:
Routinely in the company, yes.
Okay.
So I guess i didn't really -- I can't see 17 you use that plus 18 the blocks, but you use experimental testing plus those?
So 19 these are the special performance appraisals that you have 20 designed and supervised in this area.
21 MR. KLEINKE:
That's correct.
Which are based on 22 the tasks identified in the job analysis, and in fact, lists 28 tasks and are typical performances of these tasks.
They can
/}
24 be very specific to the job.
G 25 MR. WARD:
Okay.
Well, how many of each of those
i 38 1
things was involved?
Is this a procedure for the pOSS system
~
2 or are there several subelements of the POSS system?
Is this 8
a procedure for each of those?
4 MR. KLEINKE:
For the five projects that were 5
developed specifically for eel, a fast glance down there, 1 6
would say it was a process for every one of them.
7 MR. WARD:
Okay.
Now, what I'm trying to get at, 8
how many pieces of data do you have in each of those boxes, 9
experimental testing and performance appraisal, for each of 10 these validations?
11 MR. KLEINKE:
For pOSS there were 8400 operators, of 12 whom 800 were nuclear.
About 95 percent of those, there were f~h (v) 18 two performance appraisals.
For approximately 4.5 percent, 14 there was one performance appraisal, and for only about 15 one-half of 1 percent, there was no performance appraisal So 16 the vast majority had an operator who took, by the way, an 8 17 to 12-hour experimental battery, and two supervisors were 18 rating the perfvemance.
19 MR.' REED:
Before you turn it over to Dr. Mascitti, 20 you bring forward a few issues which I would like to focus on 21 because our business is nuclear reactor safety, and more and 22 more ncw, the human element, human error aspects come into 23 focus, and particularly in maintenance and other things, we 24 are looking to find key indicators to try to assist this 25 safety aspect.
39 1
Now, you said that even today in 1985 on this 2
perhaps steep curve of utilization, only about 50 percent of 8
the companies are using the tests, and I guess you don't have 4
an idea of how conscientiously they are using the tests. But 1 5
have this intuitive feeling that the tests haven't been used and maybe they are starting to be used in 1935, 6
very much 7
and of course, they weren't available until
'82, and that 8
other testing that might have been similar to it was not used and you have pointed out the 9
very much and that there was I have the feeling that 10 validation issue and EED aspects 11 there was not much of a selection activity going on, and we 12 could almost say at most companies prior to 1980.
(,/
18 That says to me that there may be people of zero 14 mechanical comprehension trying to function in maintenance 15 jobs, and that certainly is going to impact nuclear reactor 16 safety.
How would you shotgun the picture out there among the 17 utilities?
How many of them are effective?
I'm not going to 18 ask you to identify those that have been effective in doing 19 this, but I also have this intuitive feeling that those 20 that have been effectively selecting over a period of time 21 will also show a much better track record with respect to 22 performance in the nuclear business, 23 MR. KLEINKE:
I can't respond to that with any O
24 competence or, therefore, confidence.
I just plain don't
(
25 know.
I don't know how one would measure the effectiveness of i
40 l
1 the company, and then how one would relate that effectivness 2
to other measures.
8 MR. REED:
Let me go with my own personal experience 4
for a bit.
I was involved in three nuclear reactor 5
companies.
I know that all three practiced some kind of 6
selection testing right from the outset.
I know that two --
7 one practiced semisophisticated selection testing, and that O
was the Haddam Neck plant, and I know that Wisconsin point 9
Beach practiced pretty sophisticated selection testing for a 10 long period of time, like 20 or SO years.
Dr. Mascitti shou l t) 11 know better than 1.
12 But if you look at the records of those plants, 13 there seems to be a parallel between Ynnkee Rowe, Haddam Neck 14 and Point Beach.
Well, that is my own input to your data 15 bank.
16 MR. KERR:
The sample is so small that it could have 17 been a coincidence.
18 MR. WARD:
It might have just been because Glen was 19 there.
You never know.
20
[Laughtert 21 MR. REED:
I have tried to dispel that myth by just 22 this kind of work activity.
l 23 MR. KLEINKE:
It could well be that the company or h
24 that point Beach was of such high quality that (a) It used l
[
25 tests, and (b) employed you.
41 1
MR. REED:
I said I was trying to dispel that myth.
',,,s
\\
2 MR. KLEINKE:
Well, as a behavioral scientist, 8
whatever that is, I have to explore all the competing 4
hypotheses, and very seriously, when we look at, say, use of 5
tests or, for that matter, performance of tests, we have to 6
look at all of the possible explanations for some phenomena 7
happening other than the one you would like to see.
8 Now, I have belief that it is probable that you are 9
less likely to get into problems with a well-selected group of 10 employees, and as a matter of fact, for both the plant 11 operator selection project and for the maintenance selection t
12 project, we have got something that approaches hard data on 18 that for both projects.
14 One of the subanalyses was using a technique called 15 utility analysis.
Now, that unfortunate word has nothing to 16 do with the fact that we*are in the electric utility industry, 17 but rather can be best described as a combination of some of 18 the techniques of econometrics with those of psychometrics.
19 We look at the relationship between test performance 20 and success on the job.
We also relate success on the job to 21 the likelihood of having taken certain actions either to save 22 money or to cost money, you know, and such actions can be 23 Identified.
24 We then can cross over using some of these 25 statistical techniques and thereby and up with numbers
42 1
estimating the typical value, the typical saving per employee 7_
i I
2 for a year.
Now, we have those data, which we have made 8
incredibly conservative estimates.
4 The contractor on the Operator Project, for 5
instance, said that when you consider the outlier costs, the 6
extremely high cos*4 of certain kinds of mistakes on the part 7
of a nuclear control operator, literally the zeroes run off 8
the blackboard, so we had to trim outliers, very high savings, 9
very deep costs off, and then make the estimates even more 10 conservative, litarally to make them believable.
11 MR. REED:
I wish you could have given me a 12 judgmental opinion on my question because, you know, nuclear ss
)
18 safety waits for no man to do research forever or studies 14 forever or validations forever, so --
15 MR. KLEINKE:
I cannot give a judgmental one because I have only hand to applaud with.
I can 16 you are asking me 17 suggest which companies are making good use of tests, but i 1
18 have no way of knowing which companies are effective in their 19 operations and which are not.
20 MR. KERR Glen, it seems to me it's quite possible, 21 for example, that a good management looking for ways to 22 improve things would be the sort of management that would look 23 for tests that seem to have promise. Now, the tests may 24 improve the selection process, but the mere fact that there is 25 good management in place may have as much or more to do with
43 1
the success of the operation than the use of the tests.
2 MR. REED:
Well, let's say --
3 MR. WARD:
The variation on the Hawthorne effect, 1 3
4 guess.
5 MR. REED:
Well, a good management might be 6
frightened of the tests and not use them even though they are 7
still good management.
8 There have been good companies that have been i
9 involved in legal action along this line, and I guess, 10 Charlie, is it the case that they backed away from testing?
11 MR. WYLIE:
Oh.no.
Well, that goes back some years 12 where tests were developed for fossil plant operating 18 personnel, and they had a set of tests that they used for some 14 coal-handling operators that were not validated, and they got 15 into trouble.
So they then proceeded to develop validated 1
16 tests with the University of North Carolina psychological 17 Department, and they developed tests for all their plants and 18 all their personnel.
19 MR. MASCITTl+
1 believe that a good selection 20 system which measures ability or aptitude produces a higher 21 caliber employee, which in turn has many, many benefits, i
l 22 including better safety performance and lower turnover and so 28 forth.
Now again, I can't give you the statistics of research I I 24 that support this, looking at the efficiency and effectiveness N,/
25 of nuclear plants as compared to those who are using our
44 m
1 pOSS.
In my experience at Wisconsin Electric has been that 2
where we select an individual of extremely high aptitude and 3
for manual-type jobs concentrating on mechanical 4
cceprehension, that that individual has a far greater 5
capability to function on the job and is less prone to 6
accidents and overall builds a quality workforce.
7 MR. KERR:
But you are convinced that there is as mechanical aptitude, that one somethlyg which you describe 8
9 cannot,b=, trained to have this characteristic, that it's 10 inherent.
11 MR. MASCITTl:
Well, I don't know if I want to get 12 into a philosophical discussion of whether or not some things 18
~~
i 14 MR. KERR:
Well, it seems t o me that this is very 15 important because if you can train a person to have this, then 16 you may have to spend longer training the individual but you 17 can't really say that I select him because he can never learn d
10 this.
19 MR. MASCITTI:
Let me get into the description of 20 the tests, and I think we can get to a point where you might 21 get a better understanding of the types of tests that we use i
22 in the mechanical comprehension.
23 MR. KLEINKE:
But I would also like to respond to
[
24 the question of whether or not mechanical aptitude is 25 trainable, it is almost tautological because we see in effect
45 1
a continuum going from aptitude to achievement rather than
-s okay, spelling is an 2
binary points that, you know S
achievement, music is an aptitude.
To a degree what we call 4
aptitudes are probably susceptible to training.
We know they 5
are probably susceptible to atrophying through disuse.
6 I think a number of longitudinal studies, especially 7
of mathematical skills, have demonstrated this, that in some 8
things like mathematics, if you don't use it, you are going to 9
lose it.
On the other hand, we are able to deal with in a 10 very real fashion a notion called mechanical aptitude and deal 11 with it as though it is relatively untrainable, as opposed to 12 running a turret lathe, which is a very trainable skill given f-x k
13 that one h a:s threshold levels of certain aptitudes.
14 MR. REED:
I am thinking of the many experiences 15 that I have had with selection testing for basic aptitudes and 16 how these people were either hired or not hired, and how these 17 people, if they were hired in the marginal category, how they 18 failed in the workplace.
And if you have been in the 19 workplace for over 30 years, as I was, involved with these 20 kinds of things, my answer to Dr. Kerr is that mechanical 21 aptitude is not trainable, and if a person doesn't have it, he 22 is not a very good performer in a power plant.
23 MR. KERRt I was aware of your opinion. I was just 7m 24 trying to get another second opinion.
25 MR. MASCITTI:
For example, with the mechanical
40 1
comprehension, it is possible that children who do fix-it type
,_s 2
things all through their childhood and take an interest in 3
repairing the toaster or tinkering with their first car could 4
develop a mechanical comprehension over the course of life.
5 Whether or not they are born with an aptitude called 6
mechanical comprehension, I simply don't know, it is 7
something, though, that in my eyes I feel is not quickly 8
trainable.
It may be an accumulation of experiences over life 9
and raw ability.
10 MR. KERR:
But you see
.I have the same intuitive 11 feeling that Glen has, but I have learned over the years that 12 sometimes I can trust my intuition.
I was just trying to get g j IS some feel for whether you people have thought about it, and i 14 wondered what you thought.
15 MR. WARD:
Well, you know, I think they are telling 16 you that when you get candidates for a job who are 22 years 17 old or something, there are definite differences among them.
10 Now, they are not prepared to say how mucn of this difference 19 is due to their genes and how much is due to what they did in 20 the first 22 years of their life, but there is a difference at 21 that point.
22 MR. KERRt I could say the same thing about typing, 23 Someone who has taken typing is a much better typist than
[N 24 someone who didn't.
But you can learn typing in s i x mon tles,
25 This is what I'm trying to find out is this mechanical
47 1
aptitude something that, if it is teachable, has to be p._
2 teachable over 20 years, or is this something that with a 3
well-developed training program could be taught within some 4
reasonable period of time.
5 This is not a question that expresses an opinion.
I 6
don't have a strong opinion.
7 MR. MASCITTI:
My opinion is that it is something 8
that cannot be learned.
9 MR. WYLIE:
Well, obviously the test for plant 10 maintenance position, there obviously is a score at which you 11 don't feel the probability is sufficient that a company should 12 hire that person, so there is some level set for that reason.
(~%
( )
IS MR. KERRt But this says where the person is at this t
14 point, it doesn't tell you anything, I don't believe, about 15 how long it would take to get him to another point.
16 MR. WYLIE:
But it is a probability that he would i
17 succeed, isn't it?
18 MR. MASCITTI:
Yes.
19 MR. WYLIE It is a probability that he would ever 20 succeed at that job.
21 MR. KLEINKE:
It's a probability of success as 22 related to the observation of the actual incumbents, but in 28 many employment situations, it is not really an issue of
(
24 whether or not this person could ever with proper training do 25 the job. It is simply a matter of which K number of people are
t' 40 1
we going to select and, in doing so, make the most
,_x 2
cost-effective decision.
You know, we are going to hire K S
people from L applicants, and, you know, there is some group K 4
that maximizes our payoff. It's not going to include people 5
whom we would 6
MR. REED We are not supposed to consider the 7
financial aspects.
We are supposed to consider the safety 8
aspects.
And if you are saying the person is trainable in SO 9
years and in the 30-year period he might have achieved a level 10 of satisfactory, starting out with a zero mechanical 11 comprehension rating, that is not going to be in the interest 12 of safety, I am sure, because, first of all, things being what 18 they are in the workplace, the organization will allow those g
f 14 people of lesser mechanical comprehension to function on a 15 par, and there will be pressures from unions to put them on a 16 par and to salary rate them on a par, and they will come to 17 believe that they are on a par, and yet their dexterity and 18 real aptitude might not be sufficient to be safe.
19 MR. KERR:
When you validate what I call an i
20 invalidated test -- validating is an oversimplification, 21 probably, but looking at what has happened to people who made 22 various scores after they had gone through a training program 28 of some defined characteristics and after they have been on 24 the job.
So it is in that sense that one can say, well, maybe 25 he was trainable, but he or she was certainly not trainable up
49 1
to some average or norm which would be characteristic of the 73
.s 2
population that had scored higher.
3 is it that sort of thing?
I mean I'm sure this is 4
oversimplified.
5 MR. KLEINKE:
How about when we look at people who 6
are on the job, we are still looking at their fundamental 7
aptitudes and their training and experience on the job have 8
not probably done a very great deal to alter their underlying 9
aptitudes.
10 MR. KERRt Well, this is if there is any such 11 thing.
You see, that's what I'm looking for.
12 MR. KLEINKE All right. If you will accept an O(,)
13 illustration from our clerical series, I can provide one.
In 14 the clerical series we have both clerical aptitude battery and 15 a clerical skills battery.
We also have three levels, we i
16 identify three levels of clerical workerar entry in the first 17 six months, journey level and senior level.
Senior had to 1
6m 18 have been promoted.
19 We found no systematic differences when we went from l
20 entry to journey to senior in the aptitudes, and you would 21 expect that the entry level would be the lowest and the senior 22 would be the highest.
That did not happen.
We have had about 23 12 or 14 ex per i me.i t a l battery tests, and when you look at the 24 means across the three, you see every one of the six possible i
25 permutations of 1,
2 and 3 there.
i i
50
<w 1
However, in the clerical skills, typing and (d
\\
2 stenography, we found that seniors did best and entry did 3
worst, despite, frankly, a lot of people had trepidations.
4 The myth is that in organizations, people are hired as 5
stenographers and never use stenography. That was not 6
confirmed in our case.
7 Here I would say the same thing, that if we are testing aptitudes and other personal characteristics, these a
9 aren't going to change by being on the job.
10 MR. KERR Of course, by definition, an aptitude 11 doesn't 12 MR. KLEINKE:
Well, what we care to define as s
)
,,/
13 aptitude -- it's a very pragmatic definition.
14 MR. WARD:
Yes, but an aptitude may include early 15 childhood training and experience or something like that. I 16 think that's all. It isn't necessarily the total package of 17 what an individual is born with, but perhaps his earliest 18 experiences or something like that.
[
19 MR. NASCITTl it could also be prenatal in terms of f
20 how the mother -- you know, what her diet was like.
I 21 MR. WARD:
Yes, she was frightened by a nuclear 22 reactor or something.
23 MR. MASCITTI:
Well, if she had adequate nutrients I
(
24 or something.
There are a lot of things that could 25 contribute.
I
51 1
MR. KLEINKE:
I know enough abou t-the genetic versus 2
environment versus the interaction of tho two question in 3
human traits to know that I do not qualify as an expert,'
i 4
although I have read the writings of people who do qualify as 5
experts, both self-qualified and other qualified, and I have 6
to say we are very pragmatic here. We have to deal with people j
7 as they come to us. We are not in the intervention 8
business. So whether the eight-year old could have been 9
changed to a successful 18-year old applicant for a nuclear 10 trainee, I don't know.
i 11 MR. KERR Well, I asked because you know better 12 than I the controversy that exists about intelligence w
1 13 tests. Do they really measure intelligence, which is supposed s.
14 to be something inherent, or do they measure cultural I
15 background or training or education?
At least from my own 16 inexpert reading, it seems t o me there is a difference of i
17 opinion about whether they measure intelligence, among people l
18 who should know, 19 MR. KLEINKE Yes.
l 20 MR. MASCITTI:
There is a difference of I
21 opinion. Some prefer to say that it's a general ability, the 22 ability to function with their environment.
And as 29 Dr. Kleinke mentioned, the same thing holds true with j
[
24 aptitudes. The aptitudes are a measure of this person's 25 functioning at this particular point in time.
Our experience
-w-~_
_m
52 1
has shown us that they don't significantly change with time.
,w 2
MR. WARD:
As adults.
8 MR. WYLIE:
Well, let's go back.
I don't know 4
whether you have the experience, but say at an earlier age, 5
such as 16 or so, when measured at 16 and measured as an 6
adult, would that change?
Aptitude.
7 MR. KLEINKE I believe there is a body of research 8
that suggests that we get locked into our relative positions a 9
whole lot earlier than that. There is a book that is about 21 10 years old now called " Stability of Change in Human 11 Characteristics," in which Ben Blume reviewed a great many 12 studies, and looking at the correlations between the
(,
18 characteristic measured early and the characteristic measured 14 la*er, and looked for the magic point where the correlation 15 hit.71 because that's the square root of 5,
made statements 16 likes half the variability of adult intelligence is accounted 17 for by age 5, 18 Now, that book, appearing in the mid-sixties during 19 a time when there was a lot of movement at educational 20 intervention in elementary and secondary schools, caused quite 21 a furor, but if it is to be believed, and it was a review of 22 other studies, it suggests that by the time we are 16, most of 28 the differences among us that will be there for the rest of 24 our lives are well established.
25 l'm not suggesting, by the way, that we are half as
I 53 f,
1 smart at age 5 as we are ever going to be.
I'm talking about
(
2 the differences among people rather than, you know, sheer 9
number of cc's of brain power or whatever 4
MR. WARD:
We probably ought to go on, except I want 5
to ask one more question.
I think what we have just been 6
saying makes me feel better about it, but if we look at these 7
two boxes, performance appraisal and experimental testing, you 8
explained what the performance appraisal was, and that is the 9
special on-the-job appraisals, not the routine ones.
- Now, 10 experimental testing is people taking your battery of tests.
11 Now, these weren't taken when they were hired.
You 12 had 3500 subjects in this one case, so those had to be tests 18 taken not when they were hired but on the job, and at the same 14 time they were being evaluated.
The way you are really using 15 these tests is to take the test scores in the left-hand box 16 and using that to predict what you will get in the right-hand 17 box five or ten years later.
Somehow you have justified this 18 validation procedure anyway, and I guess it is because of what 19 has been discussed here for the last 20 minutes.
20 MR. KLEINKE:
And it's only on the basis of logic 21 that if we find the relationship -- we find the relationship 22 between the test performance and the job performance on the 23 part of incumbents, people who have been on the job for a 24 while.
In the case of nuclear operators, of course, they have 25 been company map loyees for quite a while or else been in a
54 1
nuclear plant somehow or in nuclear training for quite some 7s I
)
2 time.
S We must make the logical jump that that same 4
relationship will obtain for applicants who are hired, and 5
then sometime later be appraised on their jobs.
6 MR. WARD:
And the basis for that is this discussion 7
which we were just having wher e you said there doesn't seem to 8
be much change in this thing called aptitude of -- well, your 9
example of the clerical employees. I mean that sort of data 10 must be part of the basis for enabling you to claim that this 11 is a validation.
l 12 MR. KLEINKE:
Also the fact that that has worked 18 that way in many, many such studies in a variety of 14 occupations.
Also, I am aware that somewhere buried in here 15 is a sentence that says we are establishing a central data i
10 storage facility for continuing research and tests.
- Clearly, l
it is a stronger demonstration if 17 a stronger demonstration 18 you can glue a test to applicants and then wait and look at 19 their performance as employees.
1 20 MR. WARD:
But you haven't done that yet.
21 MR. KLEINKE:
Not as yet.
We are still too now.
22 Unfortunately, ideally -- if I were to do an experimental 23 design, I would run counter to your interests completely and j
'/
\\
24 say give the test to the applicants and then hire the ones t
25 with even social security numbers or social security numbers t
r
55 1
that are multiples of 3,
and then look at their test scores s
\\
2 because we are acting against our statistical selves by 3
cutting off the bottom scores, and that does mess up the 4
situation somewhat.
5 MR. WARD:
Yes.
Okay. It is imperfect but 6
pragmatic.
7 MR. WARD:
Why don't we take a break for ten 8
minutes.
9
[ Recess.3 10 MR. WARD:
Before we go.to Dr. Maschitti, let me 11 introduce to the committee Dr. Kent Eaton, who is right here.
I guess he is serving as a consultant 12 Kent is
. tending as 13 to the NRC Staff, the Human Factors Division.
Dan Jones asked 14 him to come to the meeting, and I would like to welcome Kent.
15 He doesn't plan to give a presentation, but 1 invite him to 16 comment.
As I said, if you get inspired on some particular 17 point, we would be delighted to hear what you have to say.
18 Let's go ahead then, please.
19 MR. MASCITTI:
My name is Alfred p.
Mascitti.
I am 20 supervisor of supervisory and professional placement at 21 Wisconsin Electric power.
22 I will give you a more in-depth explanation of the 23 plant operator selection system, or POSS, first by explaining 24 the components of the tests themselves, and then I will talk 25 about the specifics involved in the validation of this
l 56
~~
. particular project, and finally, our company experience with 1
(
2
'this battery of tests.
8 As Dr. Kleinke previously mentioned, pOSS is used to 4
select power plant operators. it has been in use since 1981.
s 5'
Since that time, we also engaged in a second project for power
~
6 plant maintenance workers, resulting in a test battery called
(
' 7 MAST.
The tests used for maintenance are almost identical to 8
those used for operations.
9 As I go through an explanation of the pOSS battery, i
s 10 I'will point out the similarities and differences between the 11 pDSS and the MAST tests.
POSS, Ilke MAST, can be used to 12 select personnel for nuclear, fossil and hydropower plants. 1 13 will start off by describing the tests.
14 CSlide]
15 The tests are paper and pencil tests in a multiple 16 choice format. They are machine scorable or they can be hand s
17 scored with plastic overlay stencils.
The operators' test, w
18 pOSS, are comprised of three components: aptitude index, 19 experience index, and personnel index.
20 The first component is comprised of five 21 aptitudes reading comprehension, spatial relations, 22 mechanical concepts, mathematical usage, and perceptual speed 23 and accuracy.
The first test in the aptitude component is a
[ \\
24 test of reading comprehension. This test measures a person's 25' ability to read and understand the type of material found in
57 1
power plant training manuals.
OV 2
The reading comprehension test consists of five 8
reading passages, each followed by several multiple choice 4
questions about the passage.
The examinee is instructed to 3
5 read the passage and understand the questions by choosing 6
among a number of multiple alternatives. This test not only 7
measures a person's recall of specific facts presented in the 8
materia! but also evaluates a person's ability to apply a 9
technical or scientific concept which is explained in the 10 material 11 This test is very similar to the type of task a new 12 employee encounters in training.
In fact, the reading
("
(,
18 passages are actual subject content taken from power plant 14 training manuals, including material on instrumentation, 15 measurement and physical and mechanical principles.
16 This test is included in both the operations pOSS 17 and the maintenance MAST test batteries.
18 The second test, spatial relations, measures the 19 ability of an individua.I to visualize the proper assembled 20 form of an object. This test has 20 problems.
At the 21 beginning of each problem is a picture showing the component 22 parts of an object. Each component part is marked to show how 23 the object is to be assembled.
This is followed by five 24 pictures showing five different ways the parts can be 25 aswambled. The examinem must ou t er tri 6 tem wh i cle fera. the Ocjact 1
58 1
would take if it were properly assembled,
-3 t
2 The spatial relations test is a crucial measure of 8
an applicant's mechanical ability since many 4
mechanically-related tasks require individuals to follow 5
instructions in assembling and disassembling parts of 6
machinery, tools, et cetera.
7 Spatial relations is also impottant for employees O
who must understand the inner workings of a system, to read 9
instruments and gauges and conceptually visualize what is 10 happening in a system. This test is used in both the POSS and 11 MAST batteries.
12 The third test, mechanical concepts, is also a
("%
(
)
18 measurement of mechanical comprehension.
This test measures 14 the ability of an examinee to understand mechanical 15 principles. Each item contains a pictorial description of a 16 mechanical situation, followed by a question and three 17 possible answers. The test is intended to measure the ability 10 of a person to perceive and understand the relationship of 19 physical forces and mechanical elements in practical 20 situations.
21 The test was constructed based on an analysis of the 1
22 tasks performed on the job.
The pictures deal with gears, 28 levers, pulley systems, valves, centrifugal force, water flow,
[')
24 gravity, pressurized air, et cetera.
It has 44 problems.
j D'
25 Tests of mechanical comprehension have a long and
~
i 59 1
very successful history when used for selection for manual p-2 jobs. previous research has shown that manual workers who have 3
a good understanding of mechanical principles have fewer 4
accidents and are less prone to error.
5 The mechanical concepts test is used in both the 6
pOSS and the MAST batteries.
7 The next test, mathematical usage, measures skill in 8
solving and manipulating mathematical relationships.
There 9
are three sections covering formula conversion problems, 10 algebra problems and story problems.
11 The mathematical usage test was developed based on 12 an analysis of job duties in training material. Therefore, it
(,,
18 measures the level of mathematical aptitude required to 14 complete training successfully and effectively function on the 15 job.
The test contains 46 multiple choice items.
16 The maintenance battery also includes a math test.
17 The last test in the aptitude series is a perceptual 18 speed and accuracy test.
It measures the speed and accuracy 19 in reading tables and graphs.
The first part contains a table 20 of numbers, which is used to answer 60 multiple choice items.
21 part 2 contains a graph which is used to answer 24 multiple 22 choice items. This test is not used in the maintenance 23 battery.
24 MR. REED Since you mention mechanical aptitude, 25 and I know that (J i scons i n E l ec t r i c has used a test of that
00 es 1
sort for many years, and I know of an incident, a very serious V
2 human error incident that took place where the issue we were 8
talking about this. morning of wrong train, wrong error took 4
place where a person went to the wrong unit and did things 5
wrong in a sequence of about six or eight things.
6 Was there ever a reevaluation by your department of 7
that particular incident and that person who had any 8
particular score that was observed?
9 MR. MASCITTI.
We took a look, as we do with any 10 situation where there are problems, and found that he was a 11 marginal, at best, candidate in terms of aptitudes, and 12 primarily in the area of mechanical comprehensive.
We also do 18 similar things, for example, if there is a fatality or other 14 severe accident that might be in the overhead line 15 construction.
We found in one case that a line mechanic who 16 was working on the pole was performing an act where he was 17 leaning over and exerting pressure in the wrong direction, the 18 screwdriver slipped, he fell forward into some electrical 19 wires and killed himself.
20 in going back and looking over the test materials, 21 we found that in fact he had a low or lower score in the 22 mechanical comprehension test, and it would be my test that 28 this person was violating the mechanical principle in that 1
l 24 particular situation.
25 This series of aptitude tests measures the important
61 1
abilities that are needed to perform power plant work. They f-w
?
2 mea:sur e an individual's capability to learn the technical 3
aspects of operations and maintenance work, therefore, they 4
can be used for applicants who have previous power plant 5
experience, and they are particularly useful in selecting 6
among applicants who do not have pr6cr power plant experience.
7 The aptitude tests focus heavily on mechanical 8
c on.prehen s i on and spatial relations. The research validation 9
found that these abilities are the most significant requisites, 10 of successful performance for operating and maintenance 11 personnel 12 The next component of POSS --
(
18 MR. REED:
Let me ask a question.
Supposing you 14 were to give this experience test and find that a person had 15 lots of experience but low aptitude for the work activity, say 16 low mechanical comprehension.
What would be your feeling?
17 Would that person be more desirable in a maintenance group, 18 let's say, in a nuclear plant because of his experience, or 19 would you go with a person with higher mechanical 20 comprehension and lesser experience?
21 MR. MASCITTI' Our selection strategy at Wisconsity 22 Electric, and one that I agree with in concept, is that there 23 has to be a minimum level of aptitude.
24 MR. REED:
You have a cut-off point, then, on 25 mechanical aptitude.
02 1
MR. MASCITTI Yes.
O 2
MR. GIMMY:
I had a question on mechanical aptitude 8
and mathematical.
Does the mechanical aptitude include any 4
valving-type tests where it might show three or four tanks 5
scattered around with valves where the applicant has to 6
indicate which valves to open and which valves to close to.
7 say, pump from Tank A to Tank C without pumping to Tank B7 Do 8
you have any of those kind of things?
9 MR. MASCITTl Not specifically as you have 10 MR. GIMMYr That is heavily used in testing for 11 chemical operators in the chemical industry, the valving-type 1
12 thing, which seems to be rather important in the nuclear 13 business, too, of not taking out all your feedwater pumps at i
14 once, or something like that.
You know, the sequence-type 15 thing.
You don't do that kind of testing?
16 MR. MASCITTit *Well, the mechanical comprehension 17 test does get into water flow and the concepts behind 4
18 pressurized air.
19 MR. GIMMY:
1, ' m talking about figuring out the maze 20 of pipes to get it from here to there without going over El there.
1 i
22 l1R. MASCITTI.
It has a number of diagrams that show l
t 23 piping arrangements, and if you were to turn this valve, in 24 what direction would it come out.
It would show different 25 diameter pipes and would ask the candidate to explain what the i
03 1
pressure might exert from one-size pipe versus the other, to 2
get at the underlying understanding of the physical properties 3
principle.
4 MR. GIMMY:
The second question would be on 5
MR. KLEINKE:
Well, could follow up?
One of the 6
other subtests that we list under perceptual speed and 7
accuracy is called visual pursuit and is a diagram that looks 8
to a layman very much like an electrical schematic, which 9
requires following a very complex pattern from left to right.
10 This, I think, would get at the same thing as the valving.
11 The only valving things per se in mechanical concepts have to 12 do with principles of hydraulics.
f 13 MR. GIMMY:
Another question I had is on the V
14 mathematical one.
Do you include any scale reading there 15 wherein you show pictures of gauges and dials, logarithmic 16 scales, et cetera with a pointer and ask the person to read 17 what that thing indicates and do some interpolation, which is 18 particularly different on log scales?
19 MR. MASCITTI:
We do not have on that particular 20 test a gauge or a dial; however, we have conversion problems 21 that require an individual to take one set of measurements 22 which are converted to another set of measurements, and there 23 are questions and mathematical problems to solve using the 24 conversion key.
\\
25 MR. GIMMY One thing i have seen on that in test
4 64 1
results is wherever you show a log scale, and even though it's p-w 2
quite evident that the spacing is changing, a number of 8
applicants won't take that into account when interpolating.
4 ff it's halfway between, they call it
.5 even though it's 5
really only about
.2.
I 6
MR. MASCITTI-Our test, which is called tables and 7
graph, the graph may get at that type of ability that you are 8
talking about because they must interpret the graph in i
9 different dimensions.
10 MR. GIMMY:
Do you have any graphs that are not 11 linear?
Do you have any that are round or that look like 12 ribbon meters?
\\
IS MR. MASCITTl No.
The next component of the pOSS i
14 is the previous experience questionnaire, it contains 92 15 questions related to a candidate's previous experience in 16 school, work and recreational situations.
The questionnaire 17 covers educational achievement, previous work and recreational 18 activities.
19 A candidate's answers to the experience 20 questionnaire are scored according to a number of scales that i
21 measure past patterns of overall effectiveness in areas of 22 vocational interest, work orientation, stability and 29 tendencies toward potential weaknesses in coping with stress.
24 The third component of pOSS is called a personnel 25 questionnaire, it consists of 139 statements requiring true j
i I
~, -. _.
65 1
or falso answers.
Applicants indicate whether they agree or
(]
2 disagree with statements that describe an opinion or personal 3
circumstance.
The personnel questionnaire measures aspects of 4
temperament found to be related to emotional stability.
5 I should note that the personnel questionnaire did 6
not statistically validate for the nuclear jobs.
This is t
7 probably due to the fact that nuclear plant personnel O
participating in this research study were already closely 9
screened for emotional stability.
The personnel questionnaire 10 is still administered to nuclear applicants for the purpose of 11 research to determine its actual effectiveness in ongoing 12 selection.
O)
IS Many utilities use an additional test which is not
(
14 part of the Edison Electric institute battery, such as the 15 MMpi, to measure emotional stability.
16 MR, WARD:
I want to try to understand what you just 17 said.
You said that the emotional stability tests did not 18 correlate.
What did that mean, first of all?
Do you mean you 19 didn't get a range or something that was significant?
20 MR. MASCITTI:
Exactly. One possible reason was that 21 we didn't get sufficient range in terms of emotional 22 stability, in other words, we didn't get a lot of individuals 23 who were low in emotional stability.
[~'\\
24 MR. WARD:
Okay.
Because of this 3500-person sample 25 or whatever it was?
There was preselection?
..%--.-m..,.,.v.wm._,-_,,,
__m,m_
66 1
MR. MASCITTI in this case it would have been 7g 2
approximately 800 nuclear, which were probably already closely 3
screened.
4 MR, WARD:
But there was also screening for all 5
these other attributes tested, too.
I mean in all cases you know, the total index or 6
well, I mean in any of these 7
the individual ones, you also were taking a population that I mean every nuclear plant or 8
had been screened by both 9
overy power plant has some sort of selection criteria.
10 MR. MASCITTl:
I believe there was more variability 11 in the aptitudes than there was in terms of emotional 12 stability, and maybe this supports Mr. Reed's earlier comment 18 that there wasn't a lot of selection testing going on.
14 MR. WARD:
Well, but there was for emotional 15 stability somehow. I mean there is always some screening.
I 16 mean the guy who does the hiring, who decides to hire certain 17 people, and people who stay in the program are screened as 18 they go along.
I mean even all these earlier tests.
You are 19 having to take them, even where there was a correlation, you 20 have sort of a narrow piece of the population, and you 21 are having to extrapolate that and decide that down here you 22 are going to cut people off and not hire people who score 29 below something.
[}
24 What you are saying, I guess, for the personality v
25 index, you didn't even get a line to extrapolate, and so you l
i
67 1
1 are not you don't have a cut-off score to recommend to the
\\
a 2
utilities using the personal stability index, I guess a is that 8
right?
4 MR. MASCITTI Right. We did not use it for nuclear 5
selection.
It is only administered to determine whether or 6
not it may prove useful in the future.
7 MR. WARD:
So you are still counting on the sharp 8
eye or experience of whoever it is that does the hiring?
9 MR. MASCITTI.
I mentioned also that apart from the 10 pOSS selection project, many companies are doing other kinds 11 of things associated with access authorization.
They may use 12 a test like the MMpi, a background investigation, clinical
(~%
4 13 interview.
(
14 MR. WARD:
Okay. So you think there really is a 15 reason for there being less of a correlation or less of a 16 lever arm on this one.
i 17 MR. REED:
How long has it been that companies have 10 been required by the NRC to use something like MMpl?
It has 19 been a while.
Most people have moved to MMpl and have been 20 having to use it, and that battery of some 600 questions 21 certainly has been a forerunner to your personnel index, isn't 22 that the case?
23 MR. MASCITTI' Yes.
4 24 The maintenance battery, MAST, also has an 25 additional component to the aptitude test It is called the
CS t
1 Background and Opinion Questionnaire.
The Background and 2
Opinion Questionnaire for ma6ntenance is similar to the 3
experience and personnel index for operations.
It measures 4
personal characteristics which are shown to be related to 5
effective power plant maintenance work.
It is an untimed test 6
which takes approximately 30 to 45 minutes to administer.
7 For both the pOSS and the MAST, composite scores are I
O calculated for each component and for the overall test 9
battery.
Thus, the tests previde a measure of ability or 10 aptitude and a separate measure of adaptability.
The 11 candidate's standing on an aptitude test is a measurement of 4
4 12 the person's mental abilities that are important to learn
( j 13 power plant work and to effectively function on the job.
j 14 Candidates with high aptitude acores should be expected to i
15 understand mechanical principles, comprehend written 16 materials, use and understand mathematical relationships, and 17 perceive details quickly and accurately.
18 The experience and personnel questionnaire for the 19 plant operator battery and the background and opinion 20 questionnaire for the maintenance battery provide an account 21 of a person's history and personal characteristics. The i
22 scoring of these questionnaires is based upon the relevant 23 history and personal characteristics that were found in l
{}
24 successful power plant workers, whereas the aptitude tests
%./
25 measure the ability of a candidate to learn and perform the l
__y9mm_.
-m,
---.-c
,,.-__+r.w7,
..,y%----e-,--,.+,-_yy.,---
_,--_,--,m._..,7g,,,+,,,,,,,,,y
.,y,p__--y-----
m_,.p.
,-4y.-
60 1
technical aspects of the job.
2 The experience and background questionnaires measure 3
the personal characteristics of the candidate to determine 4
whether the person can effectively adapt to the job demands of 5
power plant work.
6 CSilde3 7
The POSS project was initiated in mid-1978 and was 8
completed in September of 1981. The project was sponsored by 9
the Edison Electric Institute.
The research was conducted by 10 a very prominent industrial psychologist at personnel 11 Decisions Research Institute.
A total of 70 investor-owned 12 electric utility companies participated in the project.
18 representing fossil, nuclear and hydropower plants.
(
14 Research information was obtained and analyzed from 15 thousands of company officials, supervisors and plant 16 operations personnel working at hundreds plants.
There were 17 3400 operators tested with a battery of experimental tests, 18 and measures of perfo'rmance were collected for these 19 i n :l l v i dua l s.
20 Statistical analyses revealed that the POSS test 21 significantly correlated with performance.
This validity was 22 found for all race and gender groups.
Thus, the POSS tests 23 are valid for minorities and females.
We obtained similar 24 findings for the MAST.
(
25 As part of the research project, the cost-benefit of
t 70 l
1 using the test was analyzed.
Supervisors and managers
\\'
2 estimated the annual value in dollars of services of S
operators.
These dollar estimates were based on productivity I
4 estimates of effective and efficient operations as opposed to r
5 the costs associated with human error, such as machinery j
i, 6
breakdown resulting from poor operations.
J 7
It was found that the financial advantage to a 8
company using the 9093 to select a nuclear plant level 9
employee such as an auxillary operator would result in a
)
10 savings of $4,840 per year for each operator placed in the 11 position.
For an employee who was selected for a nuclear 12 control rocci pos i t i on, the savings would result in $26,660 per
\\
13 year per employee.
14 Thus, if a company used the pOSS to select ten 15 nuclear control room operators, the company would save over 16 92.6 million over a ten-year period for those employees.
17 These projections were based on extremely conservative 18 estimates, and they do not include the additional savings l
19 which can result from reduced turnover.
20 1 did not feel that the fact that we did a validity n
21 study on current employees is a disadvantage, or hesitancy or gg 23 MM. WAND Welle it worries me a little that you say I can see where it is a 24 that.
I mean, if you said 25 disadvantage, but I think there is enough evidence to 1
i i
71 l
1 compensate for it, I guess l'd feel better about what you're gs 2
saying.
3 MR. MASCITTI.
The vast majority of validation l
l 4
studies that are done, where measures of test scores and I
l 5
performance are collected, are done in this format.
Current l
l 6
employees are r
7 MM. WHRD:
1 can see that, because it's very l
l O
possible to do it that way, but it's still not necessarily i
9 getting a handle on the predictive thing, which is what you're.
l 10 trying to get a handle on.
il MM. MASCITTI:
And there is more to just learning 12 the job.
There is functioning in the job.
And in functioning O)
(
19 in the job, many individuals have to make decisions.
They i
14 have to make choices, and their current job performance is a 15 very important determinant in something that you wish to 16 predict.
17 So going to current employees who have already been 10 trained and have already received all of the preparation to 19 bring them at what should be their maximum point of 20 performance, that is the ideal point at which you would 21 collect information on performance, and that is the criterion I
i 22 that you want to predict.
You want to predict with your tests 23 individuals who will function at their most capable level 24 MM. WAND Well, strictly speaking, what you want to 25 do is screen out the people who wouldn't -- who would function l
l l
l l
i I
i 72 j
1 poorly in the plant.
l gs
\\~ )
2 How what I'm afraid of, with the procedure you have l
i 3
taken, la that those people have already been screened out, l
t i
4 and in the existing non-pOSS process, has already screened out 5
all those people anyway.
So wha t do you need POSS for?
L l
6 MR. NASCITTit Well, surprisingly enough, when you 7
do a very good job of developing your rating form and i
8 administering it, you will find there's a percentage --
l 9
MM. WARD:
There are some real dogs in there.
i l
10 MM. MEED:
Dave, it seems to me, you touched on a i
11 point that is very important, and that is your second point, l
l l-12 and that is, should the regulatory consider imposing selection t
l 18 testing requirements on the industry?
[
l r
i 14 We have imposed all kinds of training requirements, i
15 quite frankly, that in my personal judgment, we should have j
i l
16 Imposed selection first 'and training second.
We got the cart l
17 before the horse.
l 18 All right, now let's ask this question of these two l
l l
19 people in this wayi i
i i
20 If selection is important in the interest of safety 21 in nuclear power plant for, particularly, natural ability, can i
22 the industry, as INDO does things, can the industry cause it 23 to happen themselves, or are their hands tied and they I
l t
24 couldn't cause it to happen?
And do they need a regulatory l
25 requirement to cause it to happen, if we were to judge that it i
i
_ - - - -. - _. - _ _ _ _. ~ _, _.
_.____m.-_,_
I 78 t
is most important to safety?
l
\\
2 MR. KLE I NK E :
I don't see how a requirement for l
3 competent employee selection testing would differ in its 4
effect from the existing uniform guidelines on employee 5
selection procedures, the principles of Division 14 of the 6
American psychological Association, or my understanding of the i
7 current version of the American psychological Association l
8 principles, together with a very large body of state and 9
federal law.
10 We have i lot of laws and guidelines that require 11 certain actions on the part moployee selection tests in the 12 event of adverse impact.
13 MR. REED:
0 think you misunderstood my point.
My
(
14 point is, should it be a requirement for the nuclear industry j
i 15 to have and use selection testing?
And then, if it is a 16 requirement, will it happen?
17 it won't happen -- I don't think it will happen --
10 you.say fifty percent may be using it, but you really don't 19 know, and you don't know the extent to which they are using 20 it.
21 MR. KLEINKE:
Oh, I do know to what extent twenty 22 companies are using it in the nuclear
~~
29 MR. REED:
And is that to a satisfactory extent?
24 MR. KLElHKE:
I don't understand what "to a I
25 satisfactory extent" means.
l
74 1
HR. REED:
In the interest of the safety that we
,_s i\\
2 ought to have.
You know all about INPO's training 3
requirements that the regulatory has laid down.
4 le it on a par is utilization on a par?
The 5
selection is on a par with training?
6 MR. KLEINKE:
I don't know the level of compliance, 7
If that's the word, with the INPO regulations, so i can't say 8
whether or not this on a par with that, g
MR. WARD:
Do you mean the INp0 accreditation 10 program?
I don't know what you're talking about.
11 MR. KLEINKE:
I don't know.
Mr. Reed is citing the j
12 INPO training
'(s j
18 MR. WARD:
No.
INDO is going to accredit utility s
I 14 training programs.
15 MR. KERRs But he's also assuming, I think, that the 16 specificity of the NRC training programs is a good thing.
And I
17 l'm not sure i share his enthusiasm for that.
'here are regulatory requirements with 1G MR. REED:
T ig respect to training.
There are no regulatory requirements 20 with respect to selection.
21 MR. KERRr Why is it, if you have a good program --
22 and I think you do -- that more utilities aren't using it?
23 MR. KLEINKE In some cases, they are not hiring 24 anyone.
in others, they are using procedures that were i
25 metablished by consultants, alther before pOSS started being 1
75 I
developed or while POSS was being developed, l
s l
2 MR. KEHR:
Well, all of the utilities or virtually 3
all of the utilities who participated in financing this are 4
also using it?
5 MR. KLEINKE:
No.
l 6
MR. KEHHt is there some reason that one can give 7
for their not using it, other than the orneriness of human 8
nature?
9 NM. KLEINKEt Not necessarily ornery, but they are 10 comfortable with whatever they are using new and don't wish to 11 change.
12 Of the nuclear companies in pOSS, right now we have
(,/
13 10 companies not using pOSS, and I would rather not look at 14 the maintenance project, simply because it is so new.
Of 15 them, one, two, three, four, five, six of the ten definitely 16 will be using it at the next opportunity.
One or two -- one 17 la moving very, very slowly because they have very deepseated 10 fears expressed to me about instituting an/ procedure that 10 would result in an adverse impact.
20 in another couple of cases, there is one very large 21 company that I am told never really intended to use it.
They 22 were happy with what they had, but they felt that their 23 position in the industry required them to support the project,
)
24 which they did.
25 Moving down, I have got three additional companies
l 70 I
I'm sorry -- those ten companies are g-'s I
who participated (v)
}
2 certified to conduct eel tests, They have testing 3
coordinators, and they are, as far as we're concerned, ready 4
to go.
They are just not using them.
5 IJe have three companies where we have people trained I
6 to be certified and who have participated, but they haven't i
7 oven bothered filling out the certification forms, l
e In all cases, they are companies that have a history j
l 9
of close association with one or more -- with a consultant per, 10 ccmpany.
11 There are six additional eel member companies who 12 didn't participate at all in pOSS, and again, we find close O(s,)
IS association with consultants, in fact, I am look ing her e at 14 the namus of the companies, and I am almost salivating, in 15 each case, the name of the consultant that they are linked 16 with.
17 MH, REED:
l'Il make you a but, you wouldn't want to 10 give us a list of companies that do or don't, partial or 19 otherwise.
I can understand that, because we would 20 immediately want to correlate that to their performance 21
- records, 22 You do that.
23 MH, KLEINKE I will tell you one bit of informal
[}
24 information.
I was thinking of this, You know, my one-handed
%/
I 25 applause remark, that I don't have the criterion measure.
I L
(
77 l
I have no way of ranking the companies in quality.
That doesn't 2
work on my shift.
3 However, about three or four personnel research task 4
force meetings ago, there is an article in the Wall Street I
i j
5 Journal that mentioned about three or four or five companies 6
by name as being especially competent at getting generating 7
facilities built close to budget and close to schedule, and 8
mentioned other companies in trouble at the time.
9 NR. REED:
Do you remember the article?
10 NR. KLE l NKE I would have to tell you, there was an 11 amazing relationship between participation in eel testing and l
12 favorable or unfavorable mention in that article.
13 Now if you accept that reporter's judgment as to, l
14 you know, that a ccepany should be on the " good list" or on 15 the " bad list" 16 MM. WARD:
Wa li, I think the problem is, there 17 t'eally isn't any definitive judgment about ranking.
I mean, t
le we're struggling with that in several different areas, not lg just this one.
Everybody has their opinions, and there are 20 some kind of obvious tales in each direction, but still 21 they're just opinions.
22 MR. REED:
Well, I would like the record to show 28 what he said, not what David Ward said.
24 MM. KLE I NKE s Well, I would hope the record would 25 say both, I would hope that I could also underline my
e I
70 1
1 l
t concern that that was, I presume, a reporter's opinion.
f 2
MR. KERMs in the back of all this, there is, it I
I have an 1
seems to me, strong evidence that good management
[
l 4
Idea that with poor management, you could have excellent t
i 5
testing for maintenance and operating people, but still have a i
(
6 lousy operation.
t i
7 Now l'm not sure that if you have good management, e
you can have good operations without testing.
I have an idea l
1 j
g that you might have good testing because good management would, 10 require it.
11 So even though I have confidence that this testing
[
12 probably works, I'm not sure that mandating testing is going i
18 to solve the problem of plant safety.
I just think the l
I 14 responsibility lies elsewhere.
15 MM. MEED:
tty position would be different.
Let me i
16 make a pointa
(
j 17 You are not going to get immediate safety from
[
d
{
18 selection testing.
It takes a long time.
You hear of one I
ig company that's been doing it for thirty years I know of other I
20 companlee that have been doing it for a long time.
Good l
i; 21 management comes from well-selected people, be they at the r
22 bottom and go up to management and go to the top or whatever,
[
i 28 but in the long pull, good management le from good seenction, 24 and not irresponalble no selection for appropriate natural 1
25 ability in the workplace, f
1 i
l t
I l
79 1
NR. KEMMi But, Glenne i have not heard anything i
l s'
2 from these gentlemen that would suggest that their tests are 3
designed to select good managers.
Maybe they think they are.
h 4
If so, I would like for them to tell me.
l 5
MM. MEED:
I think that in the personnel evaluation.
6 there is probably at some point a management evaluation.
t 7
NM. NASCITTli Yes.
I mentioned earlier that my job l
l l
4 la Supervisor of professional placement.
j l
g NM. KEMMi l'm talking about the tests that we are 10 discussing today.
11 MM. NASCITTit The POSS tests are used for i
i 12 operations, plant operatoe* going up to control room operator, t
\\
I 18 There is one other EKI project which is for first-line
[
l i
l 14 supervisors, 3
l i
15 MM. KEMMi If you tell me that you have a good test F
i l
16 that will tell us whether a company prealdent is a good l
l l
17 praeldent or a lousy president, I would be in favor, maybe, of l
18 requiring that test.
Have you got one?
Ig MM. NASCITTli He far ss the eel projecte go, we j
i 20 started off with the areas that we felt were most importante I
21 and we did start off with power plant operators.
We have 22 since looked into other projects, and we are conaldering l
l l
28 projects for higher levels, b
24 At Wisconsin Electric, we have been using testing i
i i
25 for all levela, eeght on up to onet.ustwo seleution.
t i-l I
h
i l
CO I
f l
-w 1
NR. REED:
Dr. Kerr apparently doesn't believe in
\\_ /
2 Ash All to president of the company.
Quite frankly, it's a l
[
S Horatio Alger story.
But I really think i
4 MM. KERRi Well, I am not convinced that one has to
[
S take a test to be a good company president.
What I am trying i
6 to say is that I think that good management comes down from l
7 the top, and if you have lousy management, you can test your
{
operators and test your maintenance people, and it doesn't g
improve things a lot.
i l
10 NR. REED:
Solng back to the thing that I raised l
l 11 about should there be regulatory injection of selection l
l 12 testing, there is another aspect that we haven't hit, and yet
[
[
(
18 the word la on the board, and it says " union."
f l
i 14 Many companlea are blocked from selection testing, I l
15 suspect, by union agreements, and that is all fine and good, i
16 but if you==
17 MM. WARDi is that true?
I mean, you have alleged
{
i 18 that now.
How do I know statistically what the situation is.
l l
\\
f Ig MR. REED:
I alleged that -- well, I have the 20 feeling that there are union agreements which would make it j
21 very difficult to impose selection testing, even though it j
22 might be in the best interest of nuclear power plant testing.
24 How take that on faith for a moment.
Here, again, 24 let's go back to the issue for you people.
EL Do you think it would improve and he in the internet
)
l i
j Cl 1
of safety for the regulatory to impose the same kind of i
(
l 2
requirements on selection that they have done for training, i
l
?
3 because that would, I guess, override union agreements?
i 4
MM. MASCITTl My position, speaking for Wisconsin 5
Electric, is that selection is a prime element of the workings i
6 of our company, and we support it.
And we would support 7
guidelines along the lines of the NMC being involved with --
e MM. KEMMi Would the NMC guidelines change anything 9
that you are doing?
10 MM. MASCITTl it depends --
11 MM. KEMMi Well, that's the reason I wonder if you l
l l
12 would go along with NMC guidelines, when you don't know what j
18 they are.
l 14 MM. MASCITTli Well, I make that statement with the l
l 15 assumption that the guidelines will be done well, and that i
16 they will give companies the adequate flexibility to use the 17 appropriate techniques, t
19 MM. KEHMr Well, that's a charitable attitude.
19 MM. MASCITTli And selection testing is a changing 20
- field, Jobs change, situations change, and they have to be 21 looked at, and this requires judgment on the part of the 22 organization.
l 29 MM, KEMHi The issue that Mr. Mead is raising, f l
)
24 think, is not an issue of whether selection testing should be 25 used, but whether it should be required by the NMC.
l
. ~. _ _.. - -. _
C2 1
MM. GINNY:
One good thing about requiring some 2
selection testing would be as a defense against poor r
8 management, it has been my experience, and possibly yours 4
too, if you have seen good management and poor management come 5
and go, that one characteristic of poor management is that 6
they are absolutely convinced that they can pick people "I
7 know a good man when i see one" sort of thing.
And F
e incompetents tend to surround themselves with incompetents, I
9 NM. KEMMr i think an incompetent manager can beat i
10 any system.
11 NM. SINNY:
Well, he could, if he cheated, I guess, f
12 MM. KEMMs He doesn't have to cheat.
/
I
(_,\\
/
13 MM. GINNY:
I have seen companies visit places that 14 I thought were very competent technically, but the personrel 15 Department would be very weak.
The personnel guy had the 16 leaders' favor, but he wes one of these "I know a good man 17 when i see one" sort of thing, and I think testing would have l
l 18 helped, particularly if he had to do it, you know, if it 19 didn't come from above but from without, t
20 MM. WAMD:
Well, you know, Glenn you suggested that e
21 maybe if there are requirements there to start with with the
[
i 22 enlection ~~ but i guese the way I see it is, you know, by the 23 simplest of models, there are certain people in the plant that
(
24 the NMC grants a license -- I mean, an SMO or an MO license.
25 Lgt8s just talk about that for the moment, i
03 1
Dut the simplest model is, if the person can earn I
I t\\'
2 the license, you don't care how he got there, how he was 3
trained, or what sort of natural selection he had or 4
anything.
If he could pass the licon.se exam, that was good 5
anough.
And that was originally the approach.
6 Then people came to realize that, well, examinations 7
just aren't that good.
Our university professors don't grant O
degrees just on the basis that it used to be this way.
They 9
don't grant degrees on the basis of examinations alone.
So 10 the NRC decided that it ought to have, in addition to that, 11 some requirements on training, but they are sort of limited.
12 Now fairly recently, in the last couple of years, f%
)
(,/
13 the NHC began poking around with the idea of beefing up those 14 training requiramunts, and there was a lot of protest against 15 that The NUMnHC organization was created in defense of that, In and they, in essence, said, well, instead of having more 17 requirement, let's let the industry develop a systematic 10 training approach.
What I am referring to is the INPO 19 accreditation program.
20 Now if you go beyond that now to selection of even 21 those who can go into the training program, I think you are 22 butIding another base under the pyramid.
23 MH R ElID i The NRC, Dave, has butIt a great base for 24 what they have refnered to as achievement, achievnment
()
v 25 testing, achievement qualification.
84 1
I think the issue goes deeper than the ability to (s\\
\\
l 2
achieve and pass an SRO exam or an RO exam or these kinds of 8
things, and Charlie mentioned this earlier, and maybe he will 4
come to support this.
~
5 When you get into emergency situations, where you 6
have a learned response without natural ability, you are going 7
to do things wrong.
And I could go into my own 80-some-year O
history of people that I have seen rejected and one I'm 9
thinking of right now, rejected by Wisconsin Electric, who 10 went on -- ex-nuke-navy, had certainly lots of training, lots 11 of qualification -- went on to be involved in a major i
12 incident.
And I now of a Haddam Neck person, the same way, m
18 rejected by Haddam Neck and went on to be involved in a major names will never be 14 incident.
And this is in my own databank 3 15 spoken.
16 But when you get down to --
17 MR. WARD:
Were they the root cause of these 18 incidents?
When you say " involved,"
19 MR. REED:
I would say they were key figures in 20 major emergency incidents, all right, 21 1 have this judgmental feeling and learned 22 information battery from many years of seeinD people who were 28 selected for natural ability in their performance, that if
['N 24 you're going to have reliable performance in safety 25 emergencies at nuclear plants, you have got to have the basic
85 1
ability to play the fiddle, be it the musician's natural
,_s t\\
2 ability or a valve handwheel turner's natural ability, and if 8
you go the other route of learned response for everything, 4
it's not going to come cut correctly in the tight situation.
5 MR. WYLIE:
Let me comment.
I went back and talked 6
to the ex-superintendent, who have been superintendent of the 7
Oconee Nuclear Station since it was put into service, and 1 8
asked his opinion, and he was thoroughly convinced that you 9
should use selection of employees.
10 Now Duke power Company does use selection, and i 11 don't know if they are one of the ones using -- they probably 12 are not, because they developed a whole series of tests of 18 their own, jointly with the-University of North Carolina, many g
14 years ago, and have, through the years, for their nuclear 15 plant operators, and they are probably satisfied with what 16 they've got going.
17 But in talking to him, he cited several instances and how they got by the 18 where they had had employees that but that just didn't have the 19
- process, I'm not real sure 20 natural abilities in their jobs and, you know, created and maybe you 21 situations.
But he cited several that 22 gentlemen would like to comment on this -- that eventually 23 had emotional problems and requested transfers, and they 24 transferred them, and they are happy in the job someplace else
[
j v
25 where they are more qualified for what they are doing and have
86 1
the ability.
7-2 So his response was, you know, he was thoroughly 8
convinced that selection was the thing to do.
4 MR. WARD:
Except his company is doing it already, 5
so he doesn't have any problem, except a few lemons who squeak 6
through.
7 MR. WYLIE:
Well, his experience goes back to other 8
companies that he worked for before he came to Duke also.
t 9
MR. MASCITTI-Dave, you mentioned earlier that one 10 other natural screening would be to have the employees go 11 through a training program and have the training program weed 12 people out.
~
[T (m,/
18 We have always been against that type of approach, 14 because we feel that it's an injustice to two parties -- one, 15 to the company who invests in all that time and effort, and 16 second, to the individual, an individual who may have had a 17 good job somewhere else and decided to quit to come into your 18 nuclear facility and now finds themselves in a situation where 19 they cannot perform the job.
20 MR. KERR:
But Dave's point -- I don't think he was 21 disagreeing with you.
His point was that the NRC shouldn't be 22 regulating that sort of thing.
It may be unjust and 28 inefficient, but if in the final analysis the plant safety is 24 not affected, then probably the NRC shouldn't get involved.
25 MR. WYLIE:
Well, wouldn't plant safety be affected,
87 1
though, if a person, because of the pressures of his job, is 2
emotionally disturbed.
3 MR. KERR:
But the people we were talking about 4
would be screened out.
They wouldn't be left, if the 5
screening produces the same results.
I mean, that's a big 6
assumption.
7 MR. REED:
In my early experience with unscreened 8
personnel I had a person go mentally upset twice in a control 9
room Job activity as a control operator.
10 MR. KERR:
But Charlie has just told us about the 11 superintendent who also had some people who had problems, and 12 they had been screened.
'N
\\
18 MR. REED:
Well, no system is perfect.
And I will ss 14 make a rough judgment on the Wisconsin Electric system prior 15 to pOSS, because I saw it in action with two or three hundred 16 people.
17 MR. KERR:
- Glenn, I'm not trying to argue against 18 using tests.
The question in my mind is whether it should be 19 an NRC requirement.
20 MR. REED:
Let me continue on this.
In my opinion, 21 they probably had about a five to ten percent bad selection in 22 their test.
But let me point this out:
If there had not been 28 selection testing, I think that these people who are not
)
24 harmonious in that work activity might be forty or fifty 25 percent, and that begins to impact safety.
88 1
MR. LJYL I E :
Let me ask a question that maybe the 2
gentlemen could respond to.
S It is my opinion that if you have an intelligent 4
person, you can teach him by rote to do a job, and he probably 5
could become a plant operator.
6 Do you want to respond?
7 MR. KLEINKE:
No, I don't, but I'd like to tell you 8
why I don't want to respond.
9 i don't consider myself competent in human factors.
10 I know of many tests that have -.that can demonstrate that 11 you can take a lot of tasks and by simplifying them, maku them 12 accessible to anyone, you know -- anyone can do this, if you 18 break it down into its elements and teach the person, train 14 the person, to perform each of the elements in sequence.
15 But whether or not the job of nuclear power operator 16 or maintenance worker is amendable to this kind of breakdown, 17 i just plain don't know.
18 MR. WYLIE:
tJell, what I really was going to get 19 around to is, you know, assuming that you could teach a 20 person, and it may be a fairly highly intelligent person but 21 no mechanical ability to speak of, that you would teach him to 22 pass the test, and he becomes a plant operator.
And then how 28 would he respond in a situation that required mechanical
[^\\
24 aptitude to respond to that situation?
25 The chances are, he wouldn't respond very well
89 a
1 MR. KLEINKE:
That's my opinion as a layman.
I 2
really expect that if I worked at it diligently -- one note, 3
one chord, one measure at a time -- I could play the entire 4
Moonlight Sonata, but I also don't expect you'd want to listen 5
to it.
6 MR. WYLIE:
True.
7
[ Laughter.3 I
8 l play just like you do.
9 MR. EATON:
When you did your performance appraisals 10 for your criteria to validate the. tests, were there portions 11 of those appraisals that spoke more or less directly to 12 safety, or would you infer some gain in safety kind of gs( )
ilplicitly from overall general performance?
18 m
14 MR. MASCITTl:
I believe one of the performance 15 factors was a safety factor.
16 MR. EATON:
So the supervisor was rating the 17 individuals on a factor of safety as part of your' criterion, 18 and your tests were positively correlated to that performance 19 appraisal rating of safety?
20 MR. KLEINKE:
That is correct 21 MR. EATON:
That seems like a relatively significant 22 point.
23 MR. KLEINKE:
I'm very glad you asked it, yes,
/N 24 because these were specific behaviors.
There was a section 25 that included on the low end, " Ignores safety rules," or,
90 1
" Takes unsafe bypasses."
And performance on the test was fs\\
V 2
positively related to that factor, that people who do well on 3
the batteries tend not to be the people who bypass.
4 MR. REED:
Let me throw something at Dr. Mascitti 5
I happen to know that there is a person at your 6
point Beach plant, who has more industrial accidents, ten 7
tilmes more, than any other person maybe twenty times and 8
he scores more per year than all other personnel combined.
9 Have you ever done a review of that person and his 10 evaluations?
i 11 MR. MASCITTI l'm not sure he's still there.
12 MR. WARD:
Well, let me ask the subcommittee members
/m k,
18 what you think we ought to do to this?
14 I guess I have heard an explanation of, you know, 15 employment testing, and it sounds like something useful i
16 think if I was running a utility, I'd certainly use some sort 17 of selection criteria.
On the other hand, I suspect everybody 18 does.
I mean, by definition, everybody uses some form of i
19 selection criteria.
20 The question here is, what does that have to do with 21 us on the ACRS, and what does this have to do with the NRC?
22 is anybody of a mind that this thing should be pursued, so 23 that the ACRS should end up making a recommendation to the 24 Commission that they develop a program of requirements for 25 performance testing or --
91 1
MR. KERR:
If INp0 is involved -- INpO probably
.s U
2 wouldn't be involved, since eel is already doing it.
8 MR. WARD:
That's right.
I don't think INp0 --
4 MR. KLEINKE:
May I respond?
INp0 has appointed a 5
subcommittee to look at selection testing and to review 6
selection testing procedures, review the EEI general physics, 7
Memphis State, and one company project done by Lopez, a 8
consultant from the New York City metropolitan area.
9 They have incorporated the findings from that 10 subcommittee in, I believe, an audit or review procedure, and 11 the results of that review are probably available to this 1
12 group in whole from INpO.
So they have looked at it.
)
\\,/
18 Obviously, I have worked with them in presenting 14 pOSS on an ongoing basis.
15 MR. KERR:
It seems to me that if the initiative to 16 do this could come from the industry, it would be much more 17 likely to succeed, and I would be interested to know what, if 18 anything, INp0 is doing to encourage it.
19 MR. WYLIE:
Well, that may be one approach.
I was 20 just thinking, maybe get some of the utilities and talk to 21 them, some of the larger, more nuclear involved utilities, and 22 see how they feel about it.
28 MR. KERR:
I think that would be helpful.
24 MR. WYLIE:
Well, one thing that bothers me, you 25 know, we have asked that question of most that come up for
92 1
licenses, and in a couple of cases, I was surprised that they 7s 2
were doing nothing.
8 MR. WARD:
Is that right, really?
Well, they are 4
doing something.
They select people semehow 5
MR. WYLIE:
Well, in one or two csses, they had no 6
selection process involved in the testing, and that sort of 7
surprised me for the nuclear plants.
8 MR. REED:
Well, I think you know my position.
My 9
position is that INPO has, in my opinion, tried to do it by 10 suggestion, volunteers.
The utillties, some companies do it, 11 and some companies don't.
Some people do it perhaps to a low 12 standard or a low plateau, and others don't do it all, and 18 scoe do it at a very high plateau.
14 Quite frankly, I think it's more important than 15 training, and I really think it should be a regulatory 16 requirement, be it a one-liner.
I don't care.
A one-liner:
17 Thou shalt do selection testing for natural ability to a 18 satisfactory -- in a satisfactory manner.
19 1 would like.to say to the POSS tests, but --
20 MR. KERR:
You don't know how'many companies do or 21 do not use it?
22 MR. REED:
In a satisfactory manner.
I'm going to 23 make a prediction that not twenty percent do.
24 MR. KERR:
So you don't know how many do it and how 25 many don't.
93 1
MR. REED:
No.
I just know by going around to
(
2 meetings.
8 MR. WARD:
Well, I think that's the thing.
This is 4
a solution to some problem, but we are not sure what the 5
problem is or if there is a problem.
You are convinced there 6
is.
7 MR. REED:
The issue is safety and performance.
8 MR. tJARD:
Well, I know what the issue is, a
9 MR. WYLIE:
Let me ask, has the Staff made any 10 analysis that you know of as to the companies that do use 11 selection in their hiring of employees as related to the 12 things we talked about this morning in numbers of LERs or O
13 incidents?
j 14 MR. CWALINA:
No, we haven't.
We are intending to 15 include this as part of our questionnaire and part of the 16 protocol when we go on the site surveys, but we don't have any 17 of this data available now to make that kind of a correlation.
18 MR. WYLIE:
It would be helpful i
19 MR. CWALINA:
I agree, and we intend to include that 20 in the protocol 21 MR. REED:
I'm going to say it one more time, 22 because I keep struggling with this issue.
I have been 28 struggling with it on an INPO basis and a company basis and
["N 24 industry basis for a long time.
25 I sometimes think that I am hearing party line:
94 1
Yeah, we're doing a great job.
This Company X says we're fs e
n
(
/
~'
2 doing a great job.
Everybody else is doing a great job.
We 8
have selection, and if you bore into it, you will find that 4
they really don't have good selection at all And it's like 5
bad training.
The two things go hand in hand.
And if you 6
don't set standards for the selection and conscientiously do 7
it, then the product and the safety of the business is not S
going to be there from a human aspects point of view.
9 MR. KERR:
I agree with you.
But I don't think the 10 solution comes at this level I think it comes at the 11 management level And if you have a recipe for us to choose 12 good managers, then I am in favor of that.
18 MR. WYLIE:
Well, assuming we have bad managers, 14 then what do you do?
15 MR. WARD:
This ain't going to help.
16 MR. KERR:
You have got to set criteria for plant 17 performance and put teeth in them and shut the plants down if f
18 performance is not satisfactory.
We have not done that in 19 this country.
20 MR. WYLIE:
.That's right, we haven't.
21 MR. REED:
It's much more difficult to set criteria, 22 make rules and regulations for good management --- in fact, 28 l'm not so sure anyone can define that -- than it is to set (U
24 criteria for natural ability.
25 MR. KERR:
That's right.
But setting criteria for
95
'l natural ability may not -- I mean, there's no demonstration
,y _
J r
2 that it will accomplish your objective if the management is S
lousy.
4 1 agree it's something you can do, so maybe you 5
should do it, even though it may not accomplish anything.
But 6
what I'm trying to see us do is to accomplish increased 7
safety.
8 MR. WARD:
Okay.
Well, does anybody have any 9
suggestions for whether we want the full committee to hear 10 something about this?
11 Cpause.]
12 MR. KERR:
I guess I don't see too much necessity.
13 And that's not a criticism of what we have heard.
I think my 14 point of view of what we heard, it's been very well presented 15 and convincing.
So perhaps it would be interesting to the 16 committee to see what exists.
I don't know.
17 MR. WARD:
Any other?
18 MR. WYLIE:
Yes.
I think that the committee would 19 be interested in what's been presented.
Now the outcome of 20 it, the results of it, I'm not so sure that the committee 21 would support without more substantiating data.
22 MR. KERR:
Would it be an imposition on you 28 gentlemen to make a presentation to the full committee, or 24 would you consider it worth your while to do that at all?
25 MR. KLEINKE:
No problem.
~
_ _ _ =
96
~
1 MR. MASCITTl No problem.
f\\
2 MR. KLEINKE:
Depending on the date.
8 MR. WARD:
Well, it will probably be August.
We 4
have got some --
5 MR. KLEINKE:
We could adjust it.
6 MR. WARD:
Okay.
Well, I think it's an interesting 7
topic, and obviously we aren't comfortable with any particular 8
bottom line on it at the present time.
9 i will talk to John, and perhaps I would like, then,.
10 to arrange perhaps an hour, hour and a half total for the full 11 committee, probably at the August meeting, let's tentatively 12 say, which is the first or second week in August.
But we f^.)
- (_/
18 would have to get something compatible with your schedules.
14 MR. REED:
I would like to recommend, if you're 15 coming in in August, it's fine and dandy, the eel tests, the 16 pOSS tests, but anything you can do, in my opinion, to talk 17 about case histories or examples of where things have gone 18 sour and that relate to testing that has been done, where you 19 have correlations -- if you had any case histories that are you can see for 20 significant and important, I would like 21 some people on the ACRS, it's going to take a killing blow I
22 like Three Mile Island to get recognition of the issue of
)
i 28 selection for natural ability.
And I don't think you're going O1 24 to quote a Three Mile Island, but in my own experience, I Ab non-professional, as a layman, with respect to this 25 coulo
l 97 1
issue -- I have a few recollections and opinions.
fs 2
MR. WARD:
Well, there are different perspectives.
3 But in addition, I mean, one thing that I thought was missing 4
today was a little more of an overall perspective.
You know, 5
there are different kinds of people.
Some are more impressed 6
by anecdotal case histories with a lot of pizzazz.
Others are 7
more impressed by a better understanding of the statistics of 8
the situation, for example.
And I thought we were missing 9
some factual information about how many nuclear utilities are 10 using pOSS or something equivalent, and maybe it is hard to 11 get those data.
12 But we've heard some reports that some new utilities
('~\\
8
(,/
18 apparently are not using any sort of testing, but I'd like to 14 find out how big the problem is.
You know, if 95 percent of 15 the utilities are using seine t h i ng that's as good or almost as 16 good as pOSS, then I can't get very excited about the need for a
17 regulation, even if I thought that regulation was appropriate.
18 MR. REED:
If you put in the time factor, how long 10 have they been using it?
Has it been since Three Mile 20 Island?
How long does it take to realize good results from 21 this?
22 MR. WARD:
Okay.
You see what I mean, though.
23 MR. KLEINKE:
Yes.
(\\
24 MR. WARD:
Well, any other questions?
A 25 Well, I really want to thank you two gentlemen.
-. -=.
98 1
This has been -- you have gone into an area that we are very
(,s.i 2
interested in, and as you can tell, we don't know much about, 8
but we are open to learning, and we think it's been very 4
useful We really appreciate your coming.
5 MR. KLEINKE:
I would like to thank you for the 6
opportunity, as I said at the outset, to tell the story.
1 7
feel like the Carlo Rossi of employee selection testing.
I O
love to talk about it.
9 MR. WYLIE:
Let me ask one more question.
What do you talked about minorities, say, 10 you do about 11 Spanish-speaking people that may not have mastered the English 12 language when you give tests.
18 MR. MASCITTl?
Well, when we were talking about 14 minorities, we were talking about all minorities, in 15 particular Hispanics who have difficulty with the English 16 language.
We do not have an alternate form of the POSS tests 17 in Spanish.
18 MR. GIMMY:
Nor should you have, unless the plant 19 has procedures in Spanish.
20 MR. WYLIE:
Well, I don't know.
Maybe that's a 21 requirement, that he has to master the English language.
I 22 don't know.
28 MR. WARD:
Well, I haven't seen many control rooms 24 in Spanish, so --
25 MR. WYLIE:
Well, I don't know, it's just a
99 1
question.
s 2
I know many years ago, there was a fellow that 8
applied for a job, and Duke had a requirement that they had to 4
take the mechanical aptitude, and he failed both of them.
I 5
happen to know the guy was a very good engineer, and I was 6
able to get him hired, and he turned out to be a very 7
successful employee.
But he was a German immigrant and hadn't 8
mastered the English language, is the reason he didn't do well 9
on the test.
But he turned out to be an excellent employee.
10 MR. WARD:
Okay.
Well.-thank you very much.
11
[Whereupon, at 4:20 o' clock, p.m.,
the joint meeting 12 of the subcommittees was adjourned.]
18 14 15 16 17 18 19 20 21 22 23 24 25 l
,m
(
1 CERTIFICATE OF OFFIClAL REPORTER 2
3 4
5 This is to certify that the attached proceedings 6
before the United States Nuclear Regulatory Commission in the 7
matter of: Advisory Committee on Reactor Safeguards S
9 Name of Proceeding: Joint Subcommittee Meeting on Human Factors and Maintenance Practices and Procedures I
10 11 Cocket No.:
12 PIace:
Washington, D. C.
\\J l
13 Date:
Tuesday, June 18, 1985 14 15 were held as herein appears and that this is the original 16 transcript thereof for the file of the United States Nuclear 17 Regulatory Commission.
13 l
(Signature) ig (TypedName'ofReprter) guzarge B. G ng 20 21 22 23 Ann Riley & Associates. Ltd.
24 25 w-
--ww-g
-_-_m e
._.,w.,,___
--.r-m
--w-.+r-e
- - - - + -
oV EEI Employee Selection Testing For Nuclear Power Plants 1.
Development and Organization of EEI Employee Selection Training 2.
EEI Employee Selection Testing Projects Plant Operator Selection System (POSS)
Power Plant Maintenance Positions (MAST)
Physically Demanding Jobs Project First Line Supervisor Selection Project Blue Collar White Collar Clerical Positions and Meter Reader System Operator / Power Dispatching Positions Industry Testing Implementation Project 3.
Steps in Development and Validation Identify Target Jobs 4
Review Literature Job and Task Analysis 4.
Steps in Development and Validation (continued) p Select Examinee Sample Construct / Select Performance Appraisal Materials Construct / Select Experimental Tests Experimental Testing Performance Appraisal 5.
Steps in Development and Validation (continued)
Data Analysis Select Final Tests Prepare Reports Prepare Testing Material 6.
Plant Operator Selection System (POSS)
Components Validation Company Experience 7.
Components Aptitude Index Reading Comprehension Spatial Relations Mechanical Ox Mathematical Perceptual f.;2eed and Accuracy Experience Index Personnel Index
~
8.
Validation Correlates with Performance:
3,400 Operators Valid for Minorities, Females Savings Per Year /Per Operator:
Fossil: Plant Level
$ 6,270 Control Room 23.080 Hydroelectric or Switchboard 2,955 Nuclear: Plant Level 4,840 Control Room 26,660 9.
Company Experience Improved Training Fever Dropouts Revised Curriculum Feedback of Results Cost Savings Minimal Union, Legal Difficulties
- 10. Data on Current Use of EEI Tests in Nuclear Industry OO O