ML20237B578
| ML20237B578 | |
| Person / Time | |
|---|---|
| Issue date: | 11/23/1987 |
| From: | NRC COMMISSION (OCM) |
| To: | |
| Shared Package | |
| ML20237B573 | List: |
| References | |
| NUDOCS 8712160360 | |
| Download: ML20237B578 (108) | |
Text
--
ORT 31gi O.
UNnED STATES NUCLEAIt REGULATORY COMMISSION
- ========================================================.==================,===_==;l IN THE MATTER OF:
DOCKET NO:
PUBLIC MEETING ON OPERATOR REQUALIFICATION EXAMINATIONS O
LOCATION:
BETHESDA, MARYLAND PAGES: 1 - 106 DATE:
NOVEMBER 23, 1987
- ===================================================================================-
I i
Heritage Reporting Corporation O(licial Reporters 1220 L Street. N.W.
pd Wuhingson. D.C. 20005 (202a284ss 8712160360 871123 PDR ORG NRRDPDR L__-__-_-__-
l 1
1 UNITED STATES NUCLEAR REGULATORY COMMISSION Sec50T&I 2
i 3
In the Matter of:
.)
)
4 PUBLIC MEETING ON OPERATOR
)
)
5 REQUALIFICATION EXAMINATIONS
)
)
6 7
8 Monday November 23, 1987 9
Room 118 10 7920 Norfolk Avenue Bethesda, Maryland 11 12 The above-entitled matter came on for hearing, 13 pursuant to notice, at 1:00 p.m.
,sL),
14 15 PRESENT:
16 RICH STAROSTECKI, Associate Director l
Inspection and Technical Assessment Programs
(
17 Office of Nuclear Reactor Research Nuclear Regulatory Commission 18 JACK ROE, Director 19 Division of License Performance and Quality Evaluation 20 Nuclear Regulatory Commission 21 JOHN HANNON, Branch Chief Operator Licensing Branch 22 Nuclear Regulatory Commission j
23 LYNN WIENS, Section Leader Operator Licensing Branch 24 Nuclear Regulatory Commission 25 (Continued on next page.)
O o
Heritage Reporting Corporation (202) 628-4888
2 1
PRESENT:
(Continued) i 2
RONALD EATON, Senior Reactor Engineer Nuclear Regulatory Commission 3
JERRY WACHTEL 4
Training and Assessment Specialist Nuclear Regulatory Commission 5
6 7
8 9
10 11 12 f-s 13 14 15 16 17 18 19 20 21 22 23 24 25 (O
Heritage Reporting Corporation (202) 628-4888 t.
_------o
3 Q
(/
1 PROCEEDINGS 2
MR. STAROSTECKI:
I am Richard Starostecki, and I am 3
the Associate Director within NRR for Inspection and Technical 4
Assessment Programs.
The purpose of this public meeting is to 5
have a forum where we have some dialogue with representatives 6
of industry, licensed operators, and the public, if there are 7
any such people present.
We will be transcribing this meeting, i
8 and a copy will be available within about three days.
9 Consequently, when you have questions or comments to l'0 make, please identify yourself and your affiliation so that we 11 can follow up on any action items.
So if you have questions, 12 please come up and feel free to use the microphone, so that we 13 can get it recorded.
14 I think that it is worthwhile to reflect on the fact 15 that the suspension and the requalification examination process 16 was instituted by NRR as a result of concerns that we have had 17 developing over several months, and it was reinforced when we 18 had the meeting with NUMARC representatives in September.
So 19 this reexamination of the program is not just sparked by one 1
20 meeting.
I I
21 We have in fact put forth a Commission information I
22 paper that deals with the whole subject dated October 20th, and 23 copies should be available.
That spells out what our concerns 24 were specifically; the inputs, and concerns, and thoughts
)
i 25 provided by a number of licensed operators and representatives
(
Heritage Reporting Corporation j
(202) 628-4888 i
-._.-.---.-----A.___.
4
/~)
(,_/
1 of industry; and our reactions to some of those comments.
2 As I said, the purpose today is to have dialogue.
So 3
we are going to have John Hannon, the responsible branch chief 4
involved, lead you through the process to where we see 5
ourselves going.
6 However, I have to caution you in that I believe that 7
the approach that we are taking is sufficiently different that 8
we are not going to achieve what we hope to achieve in a very 9
short period of time.
I personally believe that it may take at 10 least a year to develop some of these specific banks of 11 questions and other information that is needed to support this 12 process.
13 So we are in fact looking for suggestions, comments, 14 ideas, and criticisms.
And so as the presentation goes 15 forward, please feel free to raise your hand, come up and use 16 the microphone, and raise the issue as it comes up.
We can 17 spend as much time as is necessary to deal with the question.
18 In a nutshell, I am talking about the fact that we 19 need to develop a transition plan from where we were to where 20 we are going.
We are in fact expanding on a pilot examination 21 process.
We have a pilot examination schedule for a plant in 22 Region 2.
That is H.B.
Robinson, and we hope to be there in 23 early December.
I expect that we will do a pilot in each NRC 24 Region.
25 What needs to be recognized is that our examination b
J Heritage Reporting CorporatAan (202) 628-4888 a
5
()
1 process should not be the controlling factor.
What we are 2
saying is the utilities are responsible for coming up with a 3
requalification program for the licensed operators that 4
reinforces the approach that we are taking.
That is we are 5
looking at examination banks that the utilities develop.
So 6
written exams should not be the sole responsibility of the NRC.
7 The operators are going to be allowed in our exams 8
open book exams.
We are going to be checking them on the tech 9
specs and their knowledge of the procedures.
Therefore, the 10 questions that we will be asking in written exams are going to I
11-be less reliant on memorialization and more of a testing of 12 comprehension.
That clearly represents a change in the way 13 that we are doing business.
7-V 14 That also says that each utility ought to have a bank 15 or an exam bank subject to our review, such that when we NRC 16 develop a written examination that we want to rely on that exam 17 bank and draw on those questions to write up our own exam.
18 This should alleviate a lot of fears of individual operators.
19 The thrust of our process cleariy is to train and 20 improve on the operator's skills and knowledge.
So we also 21 recognize that major changes are needed in the simulator 22 portion of the requalification program.
So scenarios need to 23 be developed for simulators similarly as you have a bank of l
24 questions for the written exam.
j 25 We also will be needing to talk with you all as to O
l l
Heritage Reporting Corporation i
(202) 628-4888 l
L _---- _-- - _
1 6
]
l 1
how we;NRC can.go in and review your written exam bank l
2
' questionsand review your simulator scenario banks to judge 1
3 acceptability and completeness. 'Now this needs to be
' ntegrated with the INPO initiative as to' accreditation in the i
4.
5 requalification area.
So clearly, that is an area where we 6
need to have a little bit more dialogue.
'7' With that as an introduction, I just want to 8
highlight the fact that what John Hannon is going to be 9
presenting to you represents the' ultimate goal of what'we hope 10 to achieve.
And if there are any lessons to be learned along 11 the way, we will make the necessary adjustments.
12 We do not yet have a transition plan developed, and I 13 think that is the very next thing that we need to be doing.
So 14 any thoughts and suggestions that you have on the transition 15 period, how to make the transition work better, and ultimately 16 on the final product, it will be appreciated.
17 So with.that as an introduction, John, would you want 18 to start.
19 MR. HANNON:
Let me introduce a few folks here at the 20 front of the table before we begin.
Jack Roe at my right is 21 the Division Director responsible for the area including 22 operator licensing.
Lynn Wiens is a team leader who was tasked 23 to put together the plan that I will be discussing with you 24 that we are invoking down at H.B. Robinson.
And Ron Eaton and 25 Jerry Wachtel.
Ron has been helping as an assistant team TO l
Heritage Reporting Corporation (202) 628-4888
7
)
1 leader working'with Lynn.
And Jerry is our simulator expert, 1
2 and has been assisting in'the details of-how we. implement a 3
simulator in a plant that does not have a simulator.
4 If you will permit, I will-go up here so that I can-5 reach these slides.
6 Does everyone have a copy of the slide package?
'7 MR. STAROSTECKI:
No.
8 MR. HANNON:
Okay.
Well, in that event, I am forced 9
to use this.
I do not know if you can all see this.
10 Is anyone here, by the way, from CP&L?
11 (No response.)
12 MR. HANNON:
Good.
We can be a little relaxed about 13 what-we say about'H.B. Robinson.
Actually, we have had a O:
14
-couple of meetings with CP&L management.
They were very 15 cooperative.
I think that Lynn went down a couple of weeks ago 16 and met with two crews that have been selected to participate 17 in this trial; program.
And there were a few folks there that 18 were a little bit apprehensive about it.
But when they had an 19 opportunity to discuss what we were planning to do and 20 understood what the process involved, I think that most of I
21 their apprehension was dispensed.
t I
22 MR. STAROSTECKI:
Let me interrupt right now, John, 23 and just raise the point that during the transition plan that 24 it is clearly our intent to have the responsible managers and e
25 examiners from NRC meet with the candidates to explain the O
Heritage Reporting Corporation (202) 628-4888
I L
I 8
)
I process to them in advance of the examinations themselves.
2 We are also considering that long term as a standing 3
requirement to avoid I think a lot of apprehension as to how l
4 the process will work even after we are through with the 5
transition.
6 (Slide.)
7 MR. HANNON:
Okay, the first slide.
This is the 8
first-time that either Jack I think or Rich may have seen this, 9
so it is subject to scrutiny by not only them but you as well.
10 It pretty much puts out what I can get from the grassroots in 11 terms of what we think this program is going to take to 12 implement.
13 Phase 1 will consist of the H.B. Robinson trial, and O
14 then we have scheduled to evaluate the results in January.
And 15 then we want to go and continue this trial program into what we 16 will call Phase 2 which will be a look at another facility in 17 each of the other four regions.
What we would anticipate is 18 that the individuals that are presently involved in the 19 examination at H.B.
Robinson would sort of fan out into the 20 other regions, so that we can bring along some expertise to the 21 other regions to help them implement.
22 And what we are talking about doing in the other 23 regions, we will be talking sometime in the spring or summer 24 time frame, depending upon each individual region's individual 25 needs and schedules.
Heritage Reporting Corporation (202) 628-4888
9
(
1 For example, in Region 4, we know that Fort Calhoun 2
right now is scheduling a requal examination during the 3
February and March time frame.
And we would intend to 4
piggyback and to participate with them as the first facility in 5
Region 4.
6 There are a number of reasons for that.
And for 7
example, I will just give you that one to indicate the kind of 8
thinking that goes into the selection here.
It is that 9
unfortunately Fort Calhoun possesses an unsatisfactory program, 10 and they have a number of licensed operators whose licenses 11 have expired, and we are not permitted to reissue those 12 licenses until they are at a satisfactory program.
So there is 13 an interest on both their part and ours to get their program V
14 reviewed again.
15.
It just so happens that Fort Calhoun also is a 7
16 facility without a simulator.
So we will be achieving a dual 17 purpose there.
We will be having an exam administered under 18 this new format at a facility without a simulator.
And you 19 will see why that is important in a minute as we go through the 20 details here.
21 MR. STAROSTECKI:
Let me just interrupt again.
This 22 phasing looks good to me from the standpoint of how do we NRC 23 train our people and our examiners.
But what we need to 24 reflect on in this phasing is where from the utility point of 25 view is the opportunity to develop the exam banks for the O
Heritage Reporting Corporation (202) 628-4888
10
()
I written questions and the simulator scenarios for the simulator 2
banks of questions.
3 If we can get half of the utilities to have this done 4
by September 1988, and that is just an arbitrary set of numbers 5
with an arbitrary fraction, it would give us a bench mark.
But 6
we need to develop coincident with this the progress with which.
7 you the industry will work towards developing the capability 8
for us to do this kind of exam process.
9 And right now, we do not have any such input.
And 10 rathe-than hava a mandate, I think that it would be preferable 11 to get some appreciation very quickly as to what fraction of 12 the industry will be ready by region.
13 MR. HANNON:
As a planning goal, we have scheduled rs I
14 Phase 3 which would be the resumption across the board full 15 scope of all of the NRC administered requal exams to begin next 16 fiscal year in October 1988.
And we intend to have whatever 17 examiner standard revision necessary to implement this thing 18 codified by that time.
And I anticipate that there will be 19 some major changes to the examiner standards, NUREG 1021.
20 MR. STAROSTECKI:
So the first question that I have 21 is will the entire industry be ready October 1988 to undergo 22 this examination this way, which means exam bank questions, 23 simulator scenarios, and revised requalification training i
24 programs in place?
25 MR. GIBSON:
This is Al Gibson from Region 2.
O Heritage Reporting Corporation (202) 628-4888
\\
11
' (N
'(,)
1 MR. STAROSTECKI:
Come on up, Al.
2 MR. GIBSON:
I did not intend to come up and speak.
3 But I think that maybe in responding to Rich's question that
)
4 you should understand that the type of exam that we are i
5 proposing is different from what you are accustomed to.
And in
)
i 6
particular, the written exam will be an open book exam.
So l
7 while most of you may have exam question banks, I think that 8
you are going to have to take a look at'the nature of those 9
questions to see if they are appropriate questions for open 10 book exams.
11 MR. STAROSTECKI:
I agree, a very good point.
12 MR. RANNON:
Al, would you identify yourself for the 13 record, please?
O 14 MR. GIBSON:
Al Gibson, NRC Region 2.
15 MR. WIENS:
John, let me add to what Al said, too.
j 1
16 This is Lynn Wiens, the team leader.
And the Robinson folks
)
17 down there in preparing for the trial exam, they found that one 18 of the most difficult parts that they had to do was preparing 19 the written exam questions.
So it is a big effort, and is 20 going to be a big effort on your part in this program.
21 MR. STAROSTECKI:
What I am trying to say very 22 politely is that October 1988 may not be a realistic date for 23 full implementation of this effort.
But we need some feedback 24 and input from you as to come up with a realistic milestone.
25 MR. MOLDEN:
This is Jim Molden from Diablo Canyon.
O Heritage Reporting Corporation (202) 628-4888
12
()/
r 1
MR. HANNON:
Let me just listen and I will repeat 2
his question.
Go ahead.
3 MR. OLDEN:
I would find it very difficult to make an 4
assessment without your providing us an example.
And the 5
sooner that you could do that, the better off we would be to 6
make that assessment.
7 MR. HANNON:
Jim Molden from Diablo Canyon has asked 8
about the apparent need from where he sits of getting an 9
example so that they can respond to the scheduling issue.
I 10 think that if you will bear with me that I hope to give you a 11 better feel for what we are talking about as we go through 12 this, so that you will understand what the implications are.
13 And for your own particular facility, you might be in a better 14 position at the end of the discussion here in terms of 15 assessing when you might be able to be prepared to conduct such 16 an examination or to participate with us.
17 So let's hold that question for now, because I think 18 that it will become easier to answer as we get into some of the 19 details here.
20 MR. ROE:
Jim, we also have an expectation that we 21 are going to try to frequently brief the industry on what we 1
22 found at each stage, so that you have an opportunity to do some 23 planning, to get things in place that look like are going to 24 stay in the NRC's program.
We have provided you the Commission l
l 25 paper just to give you an idea about our initial thought.
And l
l C) l Heritage Reporting Corporation (202) 628-4888 l
i
_ _ _ ___ __U
13
()
I this particular meeting to tell you where we believe that we 2
are going, to inform you about the approach that we are going 3
to use at H.B.
Robinson.
4 When we finish that, we will probably brief you about 5
the results of it and what we see as long-term.
So that you 6
can start now, and you do not have to start later.
Because we 7
do intend to restart the requalification program fairly soon 8
after that October time frame.
And we just have to apply our 9
resources.
10 Our problem is that obviously the regulation, which 11 we have chose not to attempt to change, requires that those 12 individuals who can get that six year license that we can 13 relicense them.
They have got to get an exam again, and have O
14 to pass an exam.
And we are faced with that.
And obviously, 15 you know that with the NRC's resources that we have got to 16 level out our program over the next four or five years.
So 17 that we do not have people who are getting into a very 18 difficult situation in their fifth and sixth year.
19 (Slide.)
20 MR. HANNON:
Let me just go over some of the 21 administrative changes that were made in this new approach.
22 First, we intend to use a ninety day advance notification.
So 23 that you will be informed ninety days in advance that the NRC 24 is planning to conduct this requalification examination.
And i
25 we would ask for a thirty day turn-around on the crew O
Heritage Reporting Corporation 3
I (202) 628-4888
)
-_______D
14 g
It,/
1 identification.
2 At least the way that I see is shaping up is that the 3
first time that we go to each facility that the crew selection 4
is pretty much in your court.
And what we are looking for is i
5 at least two crews, one that would be comprised of the folks l
6 who are in the requal training cycle at the time that the exam 7
will be administered again for lessening the impact on your 8
training staff and your operation staff, and another crew made 9
up generally of people who are non-shift watch standers.
10 And then if you have a large complement of operators, 11 then there might be a third crew of other people.
But at least 12 the first time through, the burden will shift to you as who 13 those folks would be.
And the idea is to have the same people O
14 in the crews that you train and operate the plant with.
If you 15 have four people on your crews, then they would be examined as 16 those four people.
So there would not be any mixing of people 17 on crews.
18 Now as we go further downstream to the next cycle of 19 the examination, there is going to be a need for more 20 interaction, because there are other factors coming into play.
21 We have to make sure that we are going to meet the Commission's 22 mandate on the six year licenses, for example.
You start 23 talking about splitting crews them, because the numbers do not 24 work out any longer.
25 So there are some logistic details that we will have O
Heritage Reporting Corporation (202) 628-4888
15 r^
1 1
to face eventually.
But at least for the first round here, we 2
are talking about asking the utilities to assign the crews that 3
would be examined.
Of course, we have the opportunity, you 4
know, we have another sixty days.
And if that is not 5
acceptable to us for whatever reason, we will interact with you i
6 and ask to discuss it, if we have a need to change it.
7 MR. STAROSTECKI:
But clearly, the intent is one 8
shift that will be examined will be the shift that trains as a 9
shift together, that we will examine them together.
The second 10 shift that we will want as a minimum to examine will be most 11 probably long-term made up of engineers who are staff engineers 12 or staff supervisors, and we will reserve the right to make up 13 that shift.
You can propose, but clearly NRC will have the
('"
14 final call on that second and third shift.
And I would expect 15 that the second and third shift for examination will not be 16 part of the operating crew.
17 Now maybe in terms of logistics and making sure that 18 an individual operator gets his license and gets an opportunity 19 to pass through a requalification to get his license renewed, 20 he may in fact have to participate as part of shift number two 21 or three.
And that is clearly what John is saying.
But the 22 intent is as a minimum t*.at the routine training cycle for the 23 on-shift will not be perturbed by introducing new people.
24 MR. HANNON:
Let me just stop here a minute and ask.
25 Lynn, pretty much give me your experience down at H.B. Robinson O
IIeritage Reporting Corporation (202) 628-4888
.-_.____-__._m__mm
~
16
'(
1 in terms of the crews that were selected.
2 Do they conform pretty much to the guidance that we 3
have just discussed?
4 MR. WIENS:
They conform exactly to that.
We took 5_
one on-shift crew.
The crew that will be in training during 6
the week of our visit which is.Decemb'er 14th, that is one of 7
the crews that will be examined.
The second crew is made up of 8
staff licenses and instructors who are licensed to make up'the 9-second crew.
10 MR. HANNON:
Okay.
The next thing is -- I am sorry, 11 I had a question.
Would you. identify yourself.
12 MR. SMITH:
John Smith, Florida Power Corporation.
13 Are you saying here that you will give us a ninety O
14
' day notice, and then thirty days later identify the crews, so 15 that the crew is identified sixty days before the exam?
16 MR. HANNON:
That is the intent, yes.
17 The next is.that the letter notifying you of the l
18 examination will request that you supply us the material in 19 order to begin the construction of the examination.
And that 20 would include what we have already indicated is the most 21 difficult part, the proposed examination written questions and 22 the simulator scenarios.
And we will get into some more detail 23 of what that has to be as we move ahead.
l 24 MR. STAROSTECKI:
Point of clarification, John.
)
l 25 Do we not expect to issue a letter thirty days prior f
O l
Heritage Reporting Corporation (202) 628-4888 i
o:_____________
17
.D fs_/
1 to the examination identifying who the individuals are who will 2
be the subject of.the simulator; written'and walk-through?
3 I think that one could misunderstand that it would 4
occur sixty days' prior.
Clearly, our intent is to get out a 5
. letter. thirty days in advance of a test saying who the 6
Individuals are.
7 MR. HANNON:
Actually, we also intend to meet with 8
the individuals on-site with the exam team approximately thirty 9
days prior-to-the conduct of the examination.
So there will be.
10 plenty of time for those folks to interact with the examination 11 team before the exam.'
A question in the back.
12 MR. VAN HOUTE:
I am Herb Van Houte of the Rochester 13 Gas & Electric.
According to this paper that we got, it says O
14 that the facility will know sixty days in advance which 15 operators will be evaluated, sixty days in advance, the 16-requirements and procedure.
17 MR. STAROSTECKI:
Let me say again that our intent is 18 that thirty days prior to the examination that we will identify 19 who the candidates are and who the individuals are, and how we 1
20 will do that.
If you hear from somebody else, that is nice, j
21 and you can tell them it is thirty days.
22 MR. WIENS:
This is Lynn Wiens, and let me clarify 23 that again.
As Rich said, what happens is that sixty days 24 prior to the exam that the facility will identify those people 25 that they propose to take the exam.
Thirty days prior to the O
l Heritage Reporting Corporation (202) 628-4888 i
i.
f, 18 1
exam, the NRC will confirm who will be taking the exam.
And it 2
may not be the same as what you propose.
4 i
31 MR. HANNON:
All right.
Another big change that we.
l 4
will discuss'in more detail is the fact that the facility is 1
5 being asked to provide a technical assistant,.someone who would 6
actually become a part of the exam team.- And we have asked 7
that this individual preferably be an individual in the 8
operations department, not a training individual.
9 There are a couple of reasons for that.
One is that 10 we want them to somewhat be a little divorced from the training 11 function.
Because one of the things that we are going to be 12 asking them to do is to sign a statement saying that they have 13 not corroborated in any way with either the people who are 14 taking the exam or the people who are training the folks to 15 take the exam in terms of the compromise issue.
16 Another thing is that if an individual is an operator 17 that we view that as a credibility enhancement.
And we desire, 18 the NRC would desire that the individual have maximum 19 credibility, the person that we would be using to assist the 20 exam team.
Now you might say that we have people in our 21 training department who have a license.
Yes, but they are not 22 serving that same role and function, and they actualtly maybe do 23 not keep their license current even.
24 So we can discuss that if there are individual i
25 differences.
But as a goal, we would like to have the O
Heritage Reporting Corporation (202) 628-4888 1
1
19
()
1 individual who assists the exam team be an operator at the 2
facility.
3 The next is that we intend to have facility 4
management observing the simulation process, the simulator exam 5
portion.
That could mean that you would have an operations 6
supervisor in the control booth with the chief examiner or it 7
might mean that you would want to have a training manager, but 8
someone from management who understands the operations of the 9
facility will be invited to observe what is going on.
10 I am sorry, I want to get a question before I go on; 11 does someone have a question back here?
12 MR. HARRIS:
I just want to make sure that I am 13 following along with you.
My name is Ken Harris, Florida Power 14
& Light.
15 I guess that my first question is in a normal crew 16 make-up, can we use as an example the STA in his normal role as 17 we use him, even though he may not be licensed?
18 MR. WIENS:
The question is when you train in your 19 regular requal training program do you use the STA when you are 20 training?
21 MR. HARRIS:
And the answer would be yes under a 22 training such as this.
23 MR. WIENS:
Our response so far has been if you use a 24 STA in your regular training and he is part of your shift crew, 25 then we will examine with the STA.
j Heritage Reporting Corporation (202) 628-4888
20
)
1 MR. HARRIS:
Okay, good.
2 MR. ROE:
Ken, what I think that our objective is 3
twofold.
Our preference is to see you train the crew the way 4
that they operate.
We want to therefore examine the crew the 5
way that you. operate which is the way that you train.
That is 6
our preference.
There may be some modifications.
You may have 7
something in there.
There may be an administrative assistant 8
or somebody that you say has an important role, and we may 9
disagree with he or her on that particular shift.
10 But essentially, our objective is to examine that way 11 that you operate the facility with the crew composition.
One 12 of the questions that comes up is that my tech specs require 13 only so many operators, but I always augment the shift, can I O
14 have an examination with that augmented shift.
And the answer 15 is yes, because that is the way that you generally operate and 16 that is the way that we will do it.
17 MR. HARRIS:
And that is the way that we think it 18 snould be.
19 My next question is on the management representative, 20 what do you anticipate the accountability of him to be or his 21 accountability to be?
22 MR. WIENS:
Okay.
What we expect is that we if we I
23 observe something in the simulator examination that we are 24 saying is wrong, the accountability would be that the facility i
25 representative would be there and we can concur in real time
()
Heritage Reporting Corporation (202) 628-4888
21
()
I with the actual eyes observing what has occurred and get 2
concurrence or agreement, or at least there would'be no 1
3 misunderstanding between the facility'and us as to what i
4 occurred and an agreement as to whether this was an unstaffed i
5 action or resolve it right at the time, I guess.
6 MR. HANNON:
Let me supplement that to clarify.
If 7
we were, for example, to see the operators using hand signals 8
to communicate, we might want to ask that. operations supervisor
.9 or manager is that standard practice at your facility and that 10 kind of thing.
So that is the. role that we see.that this i
11 individual would be playing.
12 MR. ROE:
But I think if I could answer the question 13 a little more pointedly.
Again if you say accountability.
The 1
14 NRC will take the responsibility for making the regulatory 15
. judgement.
We may confer, but we will carry the burden of that 16 particular judgment on whether that person chould pass or fail 17 the particular exam.
18 There may be things that occur in that simulator exam 19 that the utility wants to take action on, and there may be 20 things that occur in the way that we approach the examination 21 that you are telling me that you want to provide feedback to 22 the NRC.
We do not exclude utility representation.
As a 23 matter of fact, we want to include them in there so that they 24 get the feedback on basically the things that our examiners see 1
25 so that they are able to enhance their own program and their l
Heritage Reporting Corporation (202) 628-4888 L
I l
._-__-____-_____-_-_______-_____a
22
(
l own people, and also able to provide us with feedback so that 2
we can enhance the next exams that we give at that facility and 3
others.
4 MR. HARRIS:
Just two more quick ones.
5 In stressing the need.for team training, do you 6
anticipate an. entire team failure in administering the exam, or 7
are you specifically looking at individuals?
8 MR. WIENS:
We do not anticipate -- hopefully, nobody 9
is going to fail, but the emphasis will be on the team.
10 MR. HARRIS:
I agree.
11 MR. WIENS:
The team will all be evaluated primarily, 12 and it will be.the team that passes or the team thct is shown 13 to be.
'O 14 MR. HARRIS:
The team'up or the team down?
15 MR. WIENS:
That is correct, in most cases.
And John 16 further into his presentation will provide a little bit of 17-amplification on that', but that is primarily it.
18 MR. STAROSTECKI:
Let me just say that I can envision 19 a team failing.
And if a team fails, we are going to follow it 20 up with plant walk-throughs as well.
So one of the things that 21 you will notice in our Commission paper is we are using the 22 simulator ae one of the initial exams to give us a better 23 background for follow-up questions on the walk-throughs.
24 MR. ROE:
I think that we have given you too crisp an 25 answer.
I think that you are going to find that you are going j
Heritage Reporting Corporation j
(202) 628-4888
23
()
1 to have team judgments, and there are also going to be times 2
that you are_ going to find that the individual in team 3
compositions shows real shortcomings.
So I think that you are 4
going to find that it is dependent upon both the team and the 5:
individual'.
6 MR. HARRIS:
So it is either/or, it can be the team 7
up or down, or the individual up or down?
8 MR. WIENS:
There will be a team evaluation in all 9
cases.
There will always be a team evaluation.
There could be 10 exceptionally poor performance, and there could end up being an 11 individual evaluation on the simulator exam also.
12 MR. HARRIS:
An indivjdual on a team?-
13 MR. WIENS:
On a team, right.
O 14 MR. STAROSTECKI:
I think that you have got'to draw 15 the distinction.
When you have an initial licensing exam, we 16 will continue on initial licensing to have an individual 17 examiner for each candidate in the simulator.
We do not intend 18 to do that for the requal.
The thrust is that we basically are 19 going to look at overall team performance.
And if team 20 performance is adversely affected by one or two key
)
21 individuals, we are going to take care of that, too.
But it-is
)
22 not going to be a-shadowing type of effort one examiner on each 23 individual in the simulator.
24 MR. HARRIS:
My last question addresses how do we 25 integrate this with classes that we already have in progress.
(}
Heritage Reporting Corporation (202) 628-4888
24
( )-
1 Is it going to be black and white in October 1988 if 2
we accept those dates?
3 MR. STAROSTECKI:
That is specifically the comment 4
that I had about the transition plan, because we need to 5
recognize that you do not change overnight.
And that is why 6
the things one, two, three that John Hannon put up reflects 7
really our NRC interest in training our examiners but does not 8
reflect your need to transition your exam banks, your 9
simulator, and your own training program.
And we Mo need 10 feedback from each individual utility as to how you do that.
11 Maybe the answer is we will put out a request asking you for 12 your individual milestones.
13 MR. HARRIS:
Thank you.
14 MR. HANNON:
Okay.
The final point on the 15 administration here is that we can accept some feedback from 16 the actual job incumbents who participate in the examination 17 process, and there will be a survey form at the end of the 18 written exam that we would like to get some feedback from on 19 that.
There was another question in the back, I think.
20 MR. VAN HOUTE:
Excuse me, I have another one along 21 the same line.
22 The paper that you just handed out is correct, right, 23 the paper that we just got?
It says right here that the 24 facility will know sixty days in advance which operators will 25 be evaluated.
This is the paper that you just gave us.
Heritage Reporting Corporation (202) 628-4888
t 25 L(
l MR, STAROSTECKI:
Consider it wrong,,and we will 2
revise it to thirty.
3 MR. VAN HOUTE:
Are we going to have it open for 4
questions at the end of this?
5 MR. STAROSTECKI:
Yes.
6 MR. SMITH:
John Smith again from Florida Power 7-Corporation.
8 What do you see as the involvement of this facility's 9
technical assistant, and.where will this involvement take 10 place, will it be at the utility or will he be going to the 11 Regional Offices?
12 MR. WIENS:
The facility tech assist will be doing 13 his assistance at the facility.
He will not be requested, as O~
14 we envision it right now, to go to the Regional Office.
What 15 he will do is the training department will be the one who 16 creates question banks for our simulator scenarios.
That will 17 not be the task for the technical assistant.
What he will do 18 is after the NRC examiners create an examination and create a-19 simulator scenario from the information provided to him by the 20 facility, he will review that information with the team and 21 give his advice as to wnether it is realistic or whether it 1; 22 important.to the operation of the shift, and provide the 23 insight and the viewpoint from the operations people as to the 24 adequacy of the examination.
25 MR. WACHTEL:
If I may, I just want to make one O
Heritage Reporting Corporation (202) 628-4888
26
,()
1 clarification on the last slide.
The last item that you.saw 2
said up to nine operators submit a post-exam process critique.
3 It.is our intent that all participants in the program will.
4 submit that critique.
The limitation on nine operators is just-5 for the initial: pilot test, because of OMB questionnaire 6
guidelines.
But ultimately, we want everyone to participate in-7 that critique.
8 (Slide.)
9 MR. HANNON:
Okay.
One thing that I would point out 10 again is that what you are.looking at here is what we would 11 have to call the closest to ideal that we could come with the 12 planning part-of this.
And actual implementation and 13 practicality might mean that we have to modify some of these O-14 things, and that'would go to working with the technical 15 assistant.
So there may be some deviations from what is being 16 discussed here on a case by case basis.
17 But basically, in terms of the examination itself and 18 the administration of it, the first thing is that we want to 19 give the simulator examination first.
We would intend that 20 that be set up so that it could begin at the beginning of the 21 week, if a week were set aside'for the exam.
We would conduct 22 the evaluation of the crew performance in that simulator exam.
23 We anticipate that it would be parallel graded, and 24 that our examiners would have the same evaluation sheets that 25 your graders would be using.
So there is obviously an O
Heritage Reporting Corporation (202) 628-4888
27
(
1 interaction process that has to go on here, so that.we are all
-2 operating from the same ground' rules in terms of what the 3
acceptable performance needs to be.
So the graders would be 4,
'using the same grading sheets.
5 Yes, a question in the back.
6
'MR.
EBBINGHAM:
Mike Ebbingham from Maine Yankee 7
Atomic.
8 Does that mean that we as parallel examiners will use 9.
'your forms, or that you will use our forms?
10 MR. HANNON~
Lynn, do you want to tell him how we did 11 it at H.B. Robinson?
12 MR. WIENS:
Well, at H.B.
Robinson, what we are 13 planning on doing right now is that we will use our forms to do
,O-
.14 the crew evaluations.
And Shelly Spilberg is actively 15 preparing and actually modifying some existing forms to use on 16 those exams.
17 In the future, I am anticipating that we probably 18 will end up using NRC forms normally.
However, particularly in 19 the development stage if a particular facility has a form that 20 they particularly think is very good, you know, we are most 21 receptive to taking it and looking at it and maybe incorporate 22 it as our own.
So we are not at this point locked into using L
L 23 the NRC form.
l 24 MR. STAROSTECKI:
I would like to propose that maybe L
25 this is the kind of action item that we ought to focus on LO Heritage Reporting Corporation (202) 628-4888
28
()
1 during the transition period.
It does not make sense to have 2
two separate forms.
We ought to have one universal form that 3
the industry and the NRC sees useful.
4 MR. HANNON:
You will see that that will become 5
imperative that we are all using the same grading scale, 6
because the results of this is we are going to be in a position 7
of evaluating your evaluators.
We have never done that before, 8
and this we see as a significant enhancement and in consonance 9
with the INPO accreditation process, the transition to the 10 systems approach to training, and the emphasis on your ability 11 to analyze your own training needs and evaluate your own 12 people.
13 So in terms of the NRC's perspective, this
%)
14 capability, that is the capability to not only observe the crew 15 performance, but also to evaluate how well your own training 16 staff is able to evaluate its crews, then you know, we have an 17 enhancement to the NRC evaluation process.
So it is imperative 18 that we are all using the same grading scales.
Yes.
19 MR. HAMPTON:
I am John Jim Hampton, the Duke Power 20 Company.
21 If we do have a disagreement between your evaluation 22 and the utilities, how will we resolve that disagreement?
23 MR. HANNON:
As I indicated, there was an obvious 24 interaction.
The exam team in the pilot phase will start off 25 the interaction with a meeting with the facility management.
(
(
Heritage Reporting Corporation (202) 628-4888
29
/"
(_)\\
l 1
When these difficulties are encountered, there will be a t
2 negotiation.
And if it is not resolved at the exam team level, 3
it will be flagged to the team leader's level and then to my j
4 attention, and we will get it resolved.
Unless someone has a 5
better way of dealing with these kind of difficulties, I see 6
that as a viable approach.
7 Okay.
The next thing that would be done after the 8
simulator phase of the examination is completed for the crew, 9
we would then move into the plant walk-through.
The 10 walk-through will be developed in conjunction with the 11 technical advisor and technical assistant.
It will be a 12 pre-formed examination.
The written questions would be written 13 out, and the expected responses would be identified.
And it s
(
14 would be focused on a small number of systems that we agree 15 upon in advance are those systems that are most important for 16 sale operations at that facility.
17 I believe that what happened at H.B.
Robinson was 18 that they gave us a list of ten systems, and we evaluated it i
19 and suggested two or three systems that we would like to use 20 instead of a couple that they had.
And that was negotiated, 21 and finally came to a final set of systems.
And then we used 22 that set of systems to produce the walk-throughs.
23 So this is a little bit different than what we had l
24 done in the past in a number of areas.
It is more confined and l
I 25 the scope if narrower, but the potential for more depth is l
l
("%
%/
l Heritage Reporting Corporation (202) 628-4888 l
{
1
30
()'
1:
there.
2-MR. STAROSTECKI:
But the. specific systems'that we.
i 3
. will take the people through on the walk-throughs._do not get 4
advertised until the day that we take the' people on the 5
' walk-throughs.
6 MR. HANNON:
That is correct'.
7 MR. STAROSTECKI:
So we look at the candidate list of 8-systems and we' decide at the last. minute which systems, and 9
leave it up to the examiner which systems he can take the 10.
candidate through.
11 MR. HANNON:
That is correct.
But the point is that 12 early on up front that the facility management and the NRC 13 agree on that sample' set, on what~are the most important O
14 systems for safe operation at the facility.
15-Okay.
Another important point to realize here is 16 that.when we create this walk-through portion of the i
17 examination that we want to use as the job performance measure,
-l 18 in other words the pass / fail criteria for the walk-through, 19-that'it will be derived directly from your training plan, from i
20 your learning objectives and your facility's training material.
21 So for example, if it says that the individual must 22 know three of the four ways to secure a valve in a system, and 23' you take that individual'out there and you ask the questions 24 and he only knows two, he did not meet your training objective, 25 your learning objective.
And so, you know, it would be l
(
Heritage Reporting Corporation (202) 628-4888
31
()
1 preestablished that that would not be a satisfactory grade for l
l I
2 that particular question.
3 So this is another significant thing that we want to 4
have the pass / fail criteria the walk-through tied to the 5
facility's learning objectives and training plan.
A question 6
in the back.
7 MR. TADISH:
My name is Russell Tadish from Cleveland 8
Electric Illuminating Company.
9 What consideration did you give to making this 10 portion of the exam also be performance based, that is from the 11 standpoint of the license operators typically in the control 12 room and there are other people who are typically outside of 13 the control room, and I know that we have had difficulties in 14 the past figuring out what the licensed operator is responsible 15 for compared to the folks outside of the control room, how did 16 you handle that, did you have a plan in that area?
17 MR. HANNON:
I believe it goes back again, and I will 18 let Ron and Lynn supplement this, I believe that it is 19 performance based.
Because we first of all are going to write 20 out the questions in advance, and we are going to be able to 21 Q/A them to make sure that they are tied to a KA knowledge and ability in the KA catalogue of a high importance rating.
And 22 23 we are keying it off your own facility training manual.
24 So if you have a performance based training program, 25 which I presume you will, then whatever we extract from that, O
Heritage Reporting Corporation (202) 628-4888
l
[
32
?
(_)
1 if we have gone to the high importance systems and we have gone 2
to the significant KAs, I think that it would be difficult to 3
see how you could come away with a non-performance based 4
examination.
5 Do you want to supplement that?
6 MR. WIENS:
I do not know if I could add much more to 7
uhat John just said.
As he said, the entire walk-through exam 8
is based on performance measures identified to us by the 9
facility.
So I mean that really is about as performance based 10 as you could possibly get, I would think.
l 11 MR. STAROSTECKI:
Let me try and answer the question.
12 It is clearly not our intent to take the operator and have him 13 redesign the protection system for us and tell us what happens
-3 I
m' 14 if individual components in the protection system like 15 resistors and transistors fail.
But when you are talking about 16 systems and particular mechanical systems, turbines, pumps, 17 valves, and major bistables, the operator is going to have to 18 have knowledge of those systems.
We are not going to be asking 19 him to redesign them.
20 MR. TADISH:
What I was referring to was that the 21 accreditation process has caused us to go through and ask 22 questions of the licensed operator what do you do for a living.
23 And we have identified a scope around his job, which is 24 primarily control room centered.
And we have similarly gone 25 through for non-licensed operators and asked them what do you Heritage Reporting Corporation (202) 628-4888
33 es
(_)
1 do for a living, and we find out the scope of his job is 2
outside of the control room.
3 Historically in the industry with the walk-through 4
exam, you take the operator in the control room and outside of 5
the control room and ask him all kinds of questions in both 6
areas.
The area outside of the control room through the 7
accreditation process is really being identified as a 8
non-licensed operator area primarily.
9 And what I am trying to say is that it is a 10 performance base on the non-licensed operator, but it is a 11 performance base for the licensed operator.
12 Do you understand my question?
13 MR. WIENS:
I understand it.
But what I would say 7s(')
14 though is that we would not going to ask your licensed 15 operators to demonstrate knowledge or abilities on something 16 tPat is not identified a licensed operator requirement.
17 MR. TADISH:
That is true, but I think that it is 18 going to require some special attention during the transition 19 period.
20 MR. WIENS:
Oh, yes, I understand.
We do not deny 21 that at all.
In fact, that was one of our areas that we had to 22 work on hard for the Robinson test, to identify those items 23 outside of the control room which were truly licensed operator 24 requirements.
There are items out there that they have to do.
25 MR. IIANNON :
Are there any more questions on that Heritage Reporting Corporation (202) 628-4888 i
(
l 34
()
1 point?
That is a good. subject.
2 (No response.)-
3 MR. HANNON:
Okay.
After the walk-through is 4
completed for this crew, then they would be given the written 5
exam.
And it is basically a two-part exam.
The first part i
6 would be given on a frozen simulator where the individual would 7
be given a set of questions and go into the simulator and 8
observe the indications that are there, and then' write the 9
answers to the exam.
10 Typically, we anticipate that it would be about an 11 hour1.273148e-4 days <br />0.00306 hours <br />1.818783e-5 weeks <br />4.1855e-6 months <br /> and a half long.
I do.not know how many questions yet.
I 12 guess that is coming out of the. trial experiment.
And again, 13 this is obviously performance based right there'where he works,
.O.
14 and it would be expected to be a good job performance i
15 indicator.
16 Is there a question?
Jim Molden.
17 MR. MOLDEN:
Is there going to be a strict time limit 18 placed on the exam, a deadline of say an hour to an hour and a 19 half?
20 MR. HANNON:
The question is will there be a time j
l
- 21 limit placed on this portion of the exam.
And again that is-22 just something that we are going to work out in the 23 experimental process.
Our anticipation is that maybe we will 24 need to freeze the simulator more than once to complete the j
25 exam, and everybody will be given an ample opportunity to O
L Heritage Reporting Corporation (202) 628-4888
I L
35 l ()
1 answer the questions for that freeze.
And then they come back 2
and maybe there is one more freeze, and they get another set of 3
questions and go in and work that.
When they are done, they 4
are just finished.
We are saying that in the ball park that it 5
is going to take somewhere in the neighborhood of an hour and a 6
half.
7 MR. WIENS:
On the first one, you know, we are going 8
to have time limits be flexible.
We recognize that we are 9
going to have to learn from our experience on this first exam, 10 and even probably on some follow-on exams.
But we are going to 11 do some testing ahead of time.
We intend to have the facility 12 technical assistant actually take this exam.
Not under us, 13 because we do not want him to feel nervous taking it, but to (v~h 14 take this exam.
Then we will get a little bit of a feel on the 15 length of it we hope from that experience.
16 MR. 11ANNON :
One of the comments that we got in the 17 previous meeting was that we do not have any validity check on 18 the time element of the exam.
So this is one of the efforts 19 that we are using to try and combat that.
Yes, in the back, 20 MR. FREY:
Steve Frey from Duke Power Company.
21 Are you going to allow any crew interaction time at 22 all, say fifteen minutes prior to freezing the machine, or are 23 you just going to have them come in to a frozen machine with no 24 beginning of operation?
25 MR. WIENS:
Our intent right now is that when they l
()
Heritage Reporting Corporation (202) 628-4888
l 36
()
I come in for that portion of the. written exam to give them a 2
familiarization period.
I am not certain right now of how 3
long, whether it be five minutes or something, but we are going 4
to allow them to have a familiarization time before we actually 5
start the written exam, and before they have seen their 6
questions on the exam also.
Obviously, you are not going to 7
give them the questions and then have them wander around and l
8 talk to one another, 9
MR. STAROSTECKI:
But the point clearly is that they 10 are going to be able to interact with each other and discuss 11 what it is that they have got on the boards.
12 MR. WIENS:
Prior to starting the written exam, that 13 is correct.
O 14 MR. FREY:
But my question is are they going to be 15 able to operate the controls for a fifteen minute of time or so 16 prior to the scenario?
17 MR. WIENS:
At least we were not planning that to 18 happen.
19 MR. FREY:
It is hard to diagnose something that is 20 in a frozen state, and do not know what evolved up to the time 21 that you got to that point.
22 MR. WIENS:
Well, it may be a transient that would be 23 frozen.
And if you allow it to run for fifteen minutes, thac 24 transient may not meet the conditions that your written exam is 25 based on.
So that could be very difficult for us to try and Heritage Reporting Corporation (202) 628-4888
37
(_/
1 do, if the simulator were not in a transient steady state 2
condition during that fifteen minute or five minute period.
3 MR. HANNON:
I understand the question and the 4
concern.
And let me just indicate that this again is a trial, 5
and it is an experiment really.
And what we will do is that we 6
will look at that and we will see what kind of results we get 7
from doing the way that we are talking about doing it here.
I 8
think that we have to develop an additional capability that we 9
do not presently have now if we were talking about running a 10 simulator and then freezing it, because you are talking about a 11 dynamic situation.
And you almost have to be able to create 12 the written exam on the spot, and we are not prepared to do 13 that.
14 MR. WIENS:
John, let me add, too, that on this 15 frozen simulator portion of the written exam, there are not 16 going to be questions that we are going to hide things in the 17 simulator and expect the operators to determine what switch was 18 malpositioned or something like that.
You know, the question 19 is going to direct the operators to the area that we want them 20 to answer, you know, so that they will be able to go and say 21 all right, these are the parameters, that we have got a. trace 22 on the recorder, you know, and direct them to evaluate why it 23 decreased or increased, or whatever it was.
24 You know, we are not planning on trying to hide 25 things from operators to check on awareness or something like
(,b Heritage Reporting Corporation (202) 628-4888 l
1
)
.--- -- ------ A
38
}
l that during this portion of the exam.
2 MR. STAROSTECKI:
It may be that the best solution is 3
not to use the frozen simulator and simply write a question 4
that initialized his knowledge vefore he sits down and takes 5
the written test.
6 MR. WIENS:
Well, as a matter of fact, that is what 7
we are thinking of doing for the non-simulator type of exam, 8
exactly that.
9 MR. STAROSTECKI:
So you know, during this pilot 10 phase, if it turns out that you do not gain much by freezing 11 the simulator and letting the operator look at it, maybe the 12 answer is just have a longer written question and let him work 13 from that with photographs or pictures of the simulator showing 14 the appropriate gauges and dials with the indications that he 15 would normally have in the control room.
We have done that in 16 the past, given the written exam and said here is what such and 17 such board looks like, and here is what the meters say, and the 18 written exam is taken miles away from the simulator.
That may 19 be the answer.
20 MR. WIENS:
If the simulator causes a problem, we may 21 have to do that.
But we are hoping that being in the simulator 22 would actually alleviate some of the shortcomings of a 23 photograph or something like that, but maybe that is not the i
24 case.
25 MR. FREY:
The major concern that I have is it going O
Heritage Reporting Corporation (202) 628-4888 i
39
)
1 to be a single malfunction or multiple malfunctions?
2 MR. STAROSTECKI:
This is a written exam.
3 MR. FREY:
I understand that.
4 MR. STAROSTECKI:
He is going to be faced with a 5
situation and he is going to be asked questions about it.
The 6
number of malfunctions is irrelevant.
7 MR. PREY:
I disagree.
If you are going to have a 8
simulator tube run from the LOCA and everything combined into 9
one examination like this, it is going to be hard to figure 10 that out.
11 MR. WIENS:
The same criteria is going to apply to a 12 simulator which John is going to demonstrate here, in that we 13 are not going to have a battleship in the desert type fs
.(J 14 situations in there to analyze.
15 MR. HANNON:
One other thing that I would be quick to 16 point out here is that your technical assistant is on the exam 17 team.
And one of the reasons for that is to help preclude the 18 kind of thing that you are talking about doing.
If this exam 19 becomes too complicated and the operator himself, your 20 representative, is not willing to put his name behind this 21 examination because of those kinds of concerns, then we are 22 certainly going to take that into account.
23 (Continued on next page.)
24 25 Heritage Reporting Corporation (202) 628-4888 l
40.
)
1 MR. HANNON:
That is one of our checks and balances.
2 MR. VAN HAUTE:
I have been up here before.
I.am a 3
Shift Supervisor at,Ginna Station.
I'have been through many 4
simulator scenarios, other ones, all over the country and at 5
Ginna.
We have a pretty good simulator.
6 The idea of being able to take an exam, without first c
7' having the simulator at steady state, and also to see how the 8
controls:are operating, would not be very beneficial.
The idea 9
of. freezing it along the way may be okay once everybody's got a 10 handle on what is going on.
11 You can have multiple failures.
You can have 12 parameters indicating things that can be caused by various 13 things.
It can be a valve failure, it could be a channel O
14 failure.
15 I:f you freeze it halfway through, you can't tell, you 16 may not be able to tell exactly what that is.
It is a very 17 integrated system.
i 18 It may be more beneficial to take the examination in 19 a dynamic process, all right, while the system is being tested, f
20 while the operators are going through a transient, from a 21 steady state through a transient.
I think this freezing might i
22 get you in a little trouble.
23 MR-WIENS:
I think what we would say is we would i
1 24 certainly recognize your concern and maybe there is a 25 possibility we could do that, start the simulator running, and O
Heritage Reporting Corporation
)
(202) 628-4888
I 41
()
1 stop it at a point and ask questions.
2 MR. STAROSTECKI:
Let me just make the initial 3
observation, we are going to need a written examination.
I 4
think your point is well taken.
If we need to put the 5
additional emphasis on writing a better question and 6
establishing the initial conditions, maybe that would be less 7
confusing.
And if I had to guess, I would say right now we 8
would be more inclined to write a more comprehensive set of 9
initial conditions and give you photographs of the boards with
.0 all the appropriate setpoints, because it is going to be 11 difficult for those plants that don't even have a simulator.
12 MR. VAN HAUTE:
Well, the ones that have good 13 simulators, as ours does, and I think as anybody who has been O
14 there has seen, it is a very good simulator.
I might suggest 15 that you maybe go to some of these plants that have good 16 simulators and experience what they can do, to be able to judge 17 possibly from that.
18 I get the impression that this, that you are trying 19 to get a, we a e trying to work together here to try to come up 20 with the best product.
And hopefully I think that is what we j
i 21 all want to see.
22 MR. STAROSTECKI:
The problem with the written 23 examination is we are trying to understand the operator's j
i 24 knowledge of his procedures and his technical specifications j
25 and his sense of integrated system operation and not to quiz j
i O
Heritage Reporting Corporation (202) 628-4888
_ _________________ a
42
(
1 him on what he forgot or didn't look at on the simulator.
2 And we see that as a problem with somebody forgetting 3
what they saw on the boards sitting down and writing an exam 4
question then wanting to get up and look at the simulator.
5 Maybe the answer is, give them more information in writing.
6 MR. HANNON:
One of the reasons we went to this 7
format, and again, this is an experiment, it's a trial.
If it 8
doesn't work, we are not wedded to it.
But one of the reasons 9
we went to this approach is to make the examination more of a 10 job performance measure, more operationally oriented.
And what 11 better could you have than to have an individual actually in 12 the control room environment?
13 So, it's part of the open book approach, the O
14 simulator being there in a frozen condition.
15 So, if it doesn't work, we're not wedded to it, we'll 16 try something different.
17 MR. MOLDEN:
John, is it your intent that the written 18 portion of the exam is administered in a simulator?
)
I 19 MR. HANNON:
The first portion of the written exam 20 will be administered in the simulator, that is correct.
4 21 MR. EATON:
John, she didn't hear the question.
22 MR. HANNON:
The question was, will the first part of j
23 the two-part written examination be administered in the 24 simulator.
That's our present approach.
That's what we're 25 planning to do.
/'~T
's]
Heritage Reporting Corporation 3
(202) 628-4888
)
J
l 1
I 43
()
1 MR. McCOY:
Ken McCoy from INPO, just a comment on i
2 this subject.
We had our people review the proposal and one of 3
their comments was that prior to the first part of the exam a
4 would be useful to have the shift turnover checklist provided 5
as a way to get a background on what had been going on.
l l
6 MR. HANNON:
Thank you, Ken.
That is a constructive 7
comment.
Thank you.
One more in the back.
8 MR. HELD:
John Held, Calvert Cliffs.
9 I'm also wondering, in a simulator we don't have 10 6,000 different procedures for each individual that's going to 11 be examined like we're supposed to have for classrooms.
Does 12 that mean we're supposed to bring them down there for the 13 simulator?
O 14 MR. WIENS:
What we are planning on doing on this 15 first part of the thing is that we are going to have at least 16 at Robinson, split the exam in two parts.
We'll bring one crew 17 in, take the exam, then take the second crew in and take the 18 exam.
19 The crew questions, although they are all going to 20 have the same questions, they may not be in the same order.
So 21 the people won't have to be necessarily going to the same 22 reference document at the same time and not have enough 23 documents.
24 So we are hoping or planning on only needing the 25 regular set of documents in the simulator for this portion of O
Heritage Reporting Corporation (202) 628-4888
1 44 1
the exam.
I 2
If that doesn't work out, we'll learn from the trial i
)
i 3
case here.
4 MR. HANNON:
There are obviously going to be some 5
logistical difficulties in conducting this examination and we 6
are going to work each one of them individually with each 7
facility so that we may have to make some special provisions, 8
may even have to lengthen the time for the exam.
9 The next thing to note here on this written exam is i
10 that it will be parallel graded.
And this is a carryover from 11 the old previous pilot test we' started back in May of 1986 12 where we tried to do an alternative to the requalification 13 program and we found that there is some benefit to, as there is O
14 in the simulator, to be able to evaluate your graders, to also 15 have that same element brought into the evaluation here for the 16 overall program.
17 So we intend to have the written examination parallel 18 graded.
Some more questions?
19 MR. JESSUP:
John, Bill Jessup from Union Electric.
20 This parallel grading, are you doing it as a part of the recall 21 program evaluation and if that is true, for the written 22 examination, is it also part of the simulator examination? You 23 talked a little bit earlier about there being a difference of 24 opinion for the simulator.
If there is, is that going to be a 25 problem with the recall program evaluation?
O Heritage Reporting Corporation (202) 628-4888
i 45 l
(,)j r
1 MR. WIENS:
The answer to both those questions very 2
simply is yes.
The evaluation and the parallel grading will be 3
a factor on your recall program evaluation and on the simulator 4
parallel grading that will also have an input into the program 5
evaluation.
6 I think, John, you have a talk about that a little 7
later on in more detail than what I'just said.
8 MR. HANNON:
In fact, we'll get into that next.
9 (Slide) 10 MR. ROE:
John, one of the things I want to remark 11 about, I think you'll find out that as we get into the open 12 book process that the practical implications are going to be 13 multiple copies of the reference material.
I know what is 14 going to happen is though we don't want to you're going to have 15 people crawling all over each other to get to the tech. specs, 16 to the procedures, to the key documents and the particular 17 facility.
It's going to frustrate them.
18 So I think what we're going to have to do is see a 19 reasonable amount of reference documents there that are going 20 to be obviously much more than you have available just in the 21 control room.
You can go to the extreme of having a whole set 22 of documents for each one of the candidates.
23 But to me, reproduction costs are much less than 24 difficulties and frustrations of the candidates taking the 25 examination and having a much more fluid and effective process.
O Heritage Reporting Corporation (202) 628-4888
46
)
1 I don't think we're going to require multiple copies, but --
2 MR. WIENS:
Well, in the simulator portion I don't 3
think we are, at least not right now.
But on the Part 2 of the l
4 exam which is in the classroom, there you will be required to 5
have multiple copies of the reference documents.
6 So you're not going to get out, or away from it.
7 There is going to be a requirement to copy certain references.
8 Not everything that you have, but certain tech. specs, 9
emergency procedures, things like that, there will be that 10 requirement to have multiple copies of those.
11 MR. RIETZ:
Guy Rietz from Nebraska Public Power 12 District.
What about the plants without a simulator?
How do 13 you intend on giving a written simulator examination?
7-.
14 MR. HANNON:
Well, we have a proposal in from Region 15 4 on that because they're talking about doing it at Fort 16 Calhoun in February, March time frame.
17 It's a little premature right now but what it 18 basically contemplates is sort of what Rich was just mentioning 19 where you would have some photographs, maybe some mockups, so 20 we obviously can't use a simulator but we can approximate that.
21 So to the extent that we could approximate it, then that is 22 what we would try to achieve.
23 MR. STAROSTECKI:
Exams like there were given about i
24 four years ago in Region 1 and in fact what was given were l
25 pictures and snapshots of the boards and the appropriate meters l
t
()
V l
Heritage Reporting Corporation (202) 628-4888 9
i 47
()
1 and then the examiners would pencil in or ink in the needle 2
setpoints that they wanted the candidate to focus in on and 3
then ask questions about it.
And at the time, appropriate 4
tech. specs and procedures were available to the candidates to 5
answer the questions about what that meant to them.
6 MR. HANNON:
Another question?
7 MR. CARROL:
Bob Carrol from RG&E.
On the parallel 8
grading, I had trouble with it in the pilot program and the 9
reason is it causes a great deal of time length between the 10 people taking the exam at the time you finalize grades.
11 There were numerous phone calls back and forth to the 12 point that the NRC finally just said if we are outside the 13 margin, then we'll take the other comment.
14 So in fact we didn't even grade off the same key.
15 I also think for the one exam what basically happens 16 is the utility, whether they do it consciously or not, grades 17 harder, making sure that they fall within.
It would be far 18 more, to me, reasonable for you to just take a look at other 19 examinations that we have given, bounce it against a key, make 20 sure that they're all, that the grade point value was adhered 21 to.
I don't understand this one thing.
You almost tell us, 22 grade hard on this one, and that's the only one we are going to i
23 look at, boys, so just grade tough on this one.
24 I don't understand the motive there at all.
Maybe 25 you can enlighten me.
But to me it just caused numerous things Heritage Reporting Corporation (202) 628-4888
48
(
I 1
where we tried to get a' key straight because we don't want 2-anybody to be unfairly cheated,. lot of time.
3 MR. EATON:
John, I think one of the advantages of 4.
having that technical expert from the utility is that again he 5
is working on the whole exam, including the key and the 6
questions.
7
.If the question is one and one only answer that 8
everybody would' agree is the correct answer, then it should be 9
graded evenly by both.
Hopefully that is all resolved before 10 the exam even goes down.
Then it's just a question of bouncing 11 it against what everybody agrees upon.
12 I understand the problem you're talking about from 13 before.
O
.14 MR. CARROLL:
The way you're limiting the expert to a 15 staff operations person, he is technically competent, I'll 16 grant you that.
But in terms of breakdown in points, how-17 people.were trained, et cetera, he doesn't possess the 18 expertise. That is in the training room.
19 MR. WIENS:
It's going to be the same department that 20 grades the exam, too.
21 MR. CARROLL:
True.
But you prepared the exam.
I'm 22 saying, in the pilot program, the facility prepared the 23 examination, and we still had problems.
So here you go one 24 step beyond and you're taking responsibility, and I think 25 you're going to run into more conflicts.
I think it is O
Ileritage Reporting Corporation (202) 628-4888 l
l
i 49:
()
1 unnecessary.
2 MR. STAROSTECKI:
I guess I have to answer you in the 3
following way.
I have seen an awful lot of successes with i
4 parallel. grading and maybe when you have this kind of problem, 5
it's something that'needs to be resolved in'your case 6
specifically because I think I've seen more successes than the 7
kind of problems you're eliciting.
It is artificial to grade 8
the tougher, because the NRC is looking at you.
We would 9
rather have you grade the way you always grade and if then 10 there is a difference of opinion, then we can resolve it.
11 The problem is not one of "gotcha."
The problem is 12 one of are we doing the operators justice by a good training 13 program. And if we're not grading the exam properly for O
14 whatever reason, that needs to be corrected.
If in fact you 15 are grading too severely, and forcing memorization, then that 16 is.a problem in and of itself.
17 Part of the parallel grading is just to uncover those 18 differences and if they fall within those few percentage 19 points, it doesn't concern us.
But if there are big 20 differences and we are failing several people and you're not, 21 clearly that is unacceptable.
22 I know recently we have seen the opposite where a 23 utility has failed several people and the NRC didn't.
If you 24 look into it, you find out there are some artificialities.
25 MR. HANNON:
Okay.
Let's move on.
That is a l
)
Heritage Reporting Corporation (202) 628-4888
50 O\\
(_)
1 concern, and we understand it, and we are going to be 2
looking at that area as we execute the trial program here and 3
see if our experience is similar to what we've had in the past i
4 trials.
5 Let me talk a little about the simulator examination.
6 That is the first part of the series and again, we are going to 7
be looking to get some scenarios, proposed scenarios from the 8
facility.
We would expect them to be as realistic /kg/ possible, 9
have some, based on some actual events at the facility or like j
10 facilities and call out the emergency and abnormal procedures.
11 We anticipate a scenario would take somewhere in the 12 neighborhood of around 50 minutes to an hour.
Again, we have i
13 already told you that the crew size and composition would be O
14 the same as what you normally train and operate the facility.
15 Again, no interaction would take place between the 16 examiners and the crew.
It is a passive observation, same as 17 what we would be observing your facility graders conduct.
18 The crew would be evaluated against the 19 team-dependent time critical behaviors that we basically have.
20 I guess there are two different things we look for, the 21 supervisory skills and the crew interactions, the 22 communications.
Those two competence areas would be evaluated 23 and we would have grading scales that we would be using to do 24 that and we would have agreement between the utility and the i
25 exam team as to what those grading scales were.
(
CE) l Heritage Reporting Corporation (202) 628-4888 l
i t
)
51 (O
._/
1 That would be achieved early on as we work with the 2
facility.
The same as what we touched on earlier was can an 3
individual fall the simulator portion of the exam. And the 1
4 answer is yes, but it would be upon pre-conceived flaws in 5
execution.
6 In other words, we would agree in advance on what 7
that individual would have to do before he could be identified 8
for a needed remedial training.
9 Lynn, how did that work at H.P. Robinson?
Was there 10 a specific -- ?
q l
11 MR. WIENS:
Well, it hasn't worked yet because the 4
12 simulator scenarios are being prepared this week and so next 13 week we intend to identify the items ahead of time which would V
14 lead to failure if they were done unsatisfactory by any member 15 of the crew, by the team.
And one thing I think we ought to 16 add on that, we do have to recognize that an operator, 17 theoretically anyway, could do something clear out in left 18 field where we just didn't anticipate he would do it and is by 19 all measures, by anybody looking at it, would be 20 unsatisfactory.
I 21 So obviously, that wouldn't be identified ahead of 22 time, but if it is serious enough, it could lead to failure.
23 We are talking about severe, serious shortcomings,
)
l 24 incompetencies by an operator, not just -- I don't know what l
25 you'd call them -- routine or less serious deficiencies, I l
Heritage Reporting Corporation (202) 628-4888 f
l 52 1
guess.
O 2
MR. HANNON:
Okay.
We also indicated that this part 3
of the examination would have an input to the overall program 4
evaluation and since we will have observers in the simulator,.
5 at least in the booth, or on the floor, back away watching, if i
L 6
we identify what appears to be some individual weaknesses in 7
the operators' performance, that can be noted and followed up 8
on the walkthrough.
And there will be a prescribed way as to 9
how that transition would take place and the follow-up 10 questioning would be done.
11 One of the things is that whatever we question the 12 individual on has to be relatable to a high safety-significant 13 function, either from the KA catalog or from the utility's 14 training program, and there shouldn' t be any disagreement over 15 what the job performance measure for that individual is.
16 So if we do observe weaknesses in the simulator, they 17 will be geared to the scenario that was run and can be followed 18 u p.
19 We indicated that this portion of the exam would 20 take, the walkthrough could take a couple hours.
If there are 21 significant weaknesses noted, we might have to run another hour 22 on the followup, for an individual on the plant walkthrough.
23 One question in the back.
24 MR. CARLSON:
Lars Carlson, Yankee, What about the
)
25 similar exam for a regular crew?
Will all the members of the Heritage Reporting Corporation caee> 6ee-4ees a
i 53 O
in a normal position or do you intend to rotate the
~1 crew stand 2
shifts to the board, or really be examining in their normal job 3-functions?
-4 M R.
WIENS:
They will be examined in normal job 5
function. However, i f, for example, you take.the shift 6
supervisor and the shift foreman, and normally on your shift 7
you rotate them week by week or something like that, then we
.8 would anticipate that we will rotate them on our exams also.
9 But if you don' t rotate them, we' re not going to 10 rotate them on the exam.
So it's however you operate your 11 shift that's the same way we' re going to examine them.
12 MR. GOUTH:
Ray Gouth, Commonwealth Edison.
.13 You have indicated in your writeup on the simulator
(
14 evaluation that if you identify an unsatisfactory area for team
-15 performance and the utility does not, that the program is 16 unsat.
Could you explain that a little more, discuss that a 17 little further?
18 MR. WIENS:
Do you want me to, John?
19 MR. HANNON:
Yes, go ahead.
20 MR. WIENS:
Well, yes, that needs a little 21 amplifying.
If there is just a difference, if the utility 22 identifies the crew performance in one area and we identify it, 23 let's say they identify it sat.,
we identify it as unsat., and 24 yet all the deficiencies are pretty much trackinD right down 25 the line and so there is no real disagreement between the
)
Heritage Reporting Corporation (202) 628-4888 L
54 1
ability of the facility evaluators to detect deficiencies on jes V'
2 their crew and ours, it's Just the Judgmental thing of whether l
3 the overall performance was sat. or unsat., that would not 1
4
'necessarily lead to an unsat. evaluation on the program.
5 We are looking once again in this kind of subjective 6
area of significant difference in evaluation on the crews.
7 MR. GOUTH:
Okay.
I am also having a little 8
difficulty in trying to identify how many utility people will 9
be involved in this critique. We have one who is a technical 10 assistant that assists with the exam.
Is he also an evaluator 11 overall?
12 MR. WIENS:
No.
13 MR. GOUTH:
He is Just a technical assistant?
-Q 14 MR. WIENS:
He's going to be a technical assistant to v
15 our team that will assist in. preparations for the exam and I
16 review of exam materials.
The number of people in the critique i
17 after -- are you talking about after simulator exam?
18 MR. GOUTH:
Well, both.
The simulator examination I 19 assume is going to be involvement from the utility plant site, 20 or are you looking for people from the simulator who operate 21 the console?
22 23 24 l
l 25
(
Heritage Reporting Corporation l
(202) 628-4888 le-___--____-_________-._-______.
55
(~N l
(,)
1 MR. WIENS:
No, the people that will be evaluating 2
your crew from the facility will be from your training 3
department, and we don't set the number of people that you'll 4
use.
It will be the number that you normally use to evaluate 5
your people.
6 So if you use two people for a four-man crew, that's 7
what you would use during the crew.
8 MR. GOUTH:
From either the site or the simulator?
9 MR. EATON:
Well, however you do it.
In other words, 10 in your requalification program, part of your job is to 11 evaluate your requalification program through written and 12 operating tests, so there must be some way that you evaluate 13 your people for feedback into your program to make it better.
(
14 MR. GOUTH:
But you have no concern whether it's 15 offsite people evaluating --
16 MR. EATON:
However you do it.
17 MR. GOUTH:
-- onsite people?
18 MR. WIENS:
It's whoever you normally use, that's 19 what we expect to see.
20 MR. HARRIS:
I have one question.
Ken Harris, 21 Florida Power and Light.
Do you have any intent of changing 22 the scenarios that are written by the utility?
23 MR. WIENS:
If they need to be changed we would 24 change them.
But we will, if we change them, the facility will i
25 know it ahead of time and we will talk to them about it, O
Heritage Reporting Corporation (202) 628-4888
56
()
1 discuss it with them and reach agreement on the need for the 2
changes.
We don't intend to pop surprises on you.
3 MR. HARRIS:
There will be a V&V program then, a 4
validation and verification run on any changes you make in the 5
scenario?
6 MR. WIENS:
Yes.
7 MR. HARRIS:
Okay.
Thank you.
8 MR. WIENS:
By the team.
By the exam team.
-9 MR. HANNON:
One other thing I would point out herf-10 on the simulator evaluation, is after we're done with the 11 scenario, we want to see the crew evaluated, in other words, 12 we'd like to have an exchange.
Whatever you do at your 13 facility, if you take them into another room and have a self C
\\~
14 critique, then we would like to have that conducted.
So that's 15 all, again, part of our observation of your evaluation process.
16 And I think that'll be a significant contributor to the overall 17 evaluation, j
l 18 If you don't, if you don't presently use the self-19 critique technique after your scenarios, we would like to 20 encourage you to do one and we'll be attempting to do that in j
21 the up front work, because we'd like to see that done.
l 22 Question?
23 MR. MOLLEN:
At what point does the training i
24 department get involved in the actual scenarios that you have I
25 picked to be run?
(
Heritage Reporting Corporation (202) 628-4888 1
l
-___________________________________J
57
()
1 MR. EATON:
I hope you're already involved.
As I 2
understand it --
3 MR. STAROSTECKI:
No, no.
When they're picked.
4 MR. HANNON:
The question was when does the training 5
department become involved in the scenarios after the NRC has 6
picked them?
I would indicate again as we have already heard 7
there is a validation process that has to take place.
As soon 8
as those scenarios are identified, we expect to interchange 9
with the training department to try them out and verify that 10 they work, see if there are any changes that need to be made.
11 All this has to be done before the exam is administered.
12 MR. STAROSTECKI:
Yes, but John, let me clarify 13 something.
We may want to run six scenarios in advance and 14 work with the training department, but not necessarily 15 advertise the scenarios you're going to use for the simulator 16 until you tell the instructor to run them for the test. So the 17 Interaction will occur on the simulator bank, or the scenarios, 18 but actual consultation prior to selection for administration 19 of candidates, I don't expect any.
20 MR. EATON:
Other than with the technical
{
21 representative right, Rich?
The team technical representative 22 from the facility will of course be involved.
23 MR. MOLDEN:
The reason I asked the question is I 24 think I would like to see us reconsider having at least one of 25 each type of people involved in this process. At some point, I
e^s k) l Heritage Reporting Corporation (202) 628-4888
.__ _ ___ __L
j 58 i
)
I the simulator people have to get involved down to which exams 2
we-are going to run and'which scenarios we are. going to run, l
3 and we'll be involved much earlier than that?in the development j
4 process.
I think it is advantageous, both short term and'long l
l 5
term, to make sure that it is adequately represcaced from the l
l 6
operation side and the training side.
7 MR. STAROSTECKI:
I guess we're not communicating.
8 It is our desire that you have a bank of scenarios that you run l
l 9
on the simulator that your training department is intimately 10 familiar with.
We don't expect to perturb the process 11 radically beyond what you already have.
Where we do perturb 12 those scenarios we would hope to have it worked out with your 13 department, with'your training department.
O' 14 So it is not the NRC making up a casualty of multiple 15 failures for something you've never seen.
Hopefully, the bank 16 on scenarios you have is sufficiently comprehensive that we can r
17 take and select the few we need.
18 MR. MOLDEN:
I think in the short term, I mean our 19 long term goals are that we have this person, this operator 20 representative.
The signal that sends me is that the training 21 department is not equipped to perform that same process, and we 22 don't have the same level of integrity and quality.
'23 MR. WIENS:
No, that's not the intent, Jim.
24 MR. MOLDEN:
Some of the words that I heard --
25 MR. WIENS:
No, the training department will prepare I-i
)
Heritage Reporting Corporation j
(202) 628-4888
59 I )-
1 and submit the simulator scenarios to the NRC. We'll select.
2 they're going to submit more scenarios than what we're going to.
3 use on our exam. But we will use from that' submitted by the 4
-training. department of your facil'ity the scenarios.
Our 5
technical assist is' going to review them to look for the 6
operational nature and any comments on them and if a change, if 7
he recommends a change, we will possibly make that change.
But 8
we'll go back to the training department and make sure it's 9
feasible to run on your simulator why we made~it.
We're not-10 going to exclude the training department.
11 MR. STAROSTECKI:
-I also-have to tell you, I envision 12 the NRC potentially putting in single failures.
And we need to.
13 discuss those single failures with a technical assistant or O'
14 your' representative, based on plant experience.
15 MR. MOLDEN:
And also I hope, based on the fact that
' 16 -
he is intimately familiar with the capabilities of the 17 simulator.
18 MR. WIENS:
We'll get the capabilities of the 19 simulator from your training department.
20 MR. MOLDEN:
If this guy then puts this failure in.
21 MR. WIENS:
We're going to, you know, if we make 22 changes or something, we're not going to do it and leave the 23 training department in the dark.
We're going to try it out the 24 week before.
We're going to be there, on your site, 14 days 25 before the thing, and we're going to test out these scenarios, 1
Heritage Reporting Corporation i
(202) 628-4888 i
l 1
i 60
-(
1 we're going'to find out if.they work and it's going to' involve
]
l 2-somebody from the training department, they're probably going 1
3 to run the simulator for us.
]
4 MR. MOLDEN:
Why not involve them up front?
1 1
5 MR.,WIENS:
But they are involved up' front, Jim.
6 MR. MOLDENr. To the. degree that he is able to tell
]
7 them whatLthe-program design does, what'the simulator 8
capabilities are at the beginning of this process, and'he would j
9 also sign the came statement that says I will not divulge any 10 of this information.
q 11_
MR. WIENS:
But he is doing that.
They are creating 12 and submitting the simulator scenarios to us.
I don't see 13 where we're not involving him.
14 MR. EVRINGHAM:
This is Mike Evringham from Maine 15 Yankee.
I'd like to carry on just a little bit more I think 16 from what Jim is talking about.
And I agree with him that I l
17 would suggest that to make this program workable and it would i
18 actually iron out some of the problems I think are going to 19 happen in the transition period, that if you could include a 20 training department representative with this operations 21 technical representative, both to review the exam, the 22 scenarios for technical content, and also your program relies 23 heavily on examining to the utility's learning objectives, 24 which I think the training department is in a much better shape 25 to interpret whether or not the examination or any changes you O
Heritage Reporting Corporation (202) 628-4888 l
_.____._________________________________u
61
<-' _g) s 1
want'to make to the examination affect those learning 2
objectives.
3 So I don't think it would be subtracting anything, 4
especially during the transition period, and I think it would 5
help iron out some of the earlier problems.
l l
6 MR. WIENS:
I still don't think we're communicating.
I l
7 We are not excluding the training department from reviewing the j
8 scenarios. As I say, they prepare the scenarios up --
9 MR. EVRINGHAM:
Up front.
10 MR. WIENS:
-- up front.
And if we change them, 11 we're going to go back to the training department and discuss 12 changes that may be made to it.
13 MR. EVRINGHAM:
I think it would be easier if it was
('-)
14 one person involved with your team from the training department 15 who --
16 MR. STAROSTECKI:
Well, let me offer you an 17 observation.
Part of this process is to test the training 18 department. Make~no bones about it.
I'm not convinced you have 19 solid, 100 percent good training programs out there, plain and 20 simple.
We're going to find out which ones are lousy and we're 21 going to have to fix them. And part of that is to find them.
22 And this is, I think, a very good way, not the only way, but I 23 think it is an innovative way of getting at reflecting 24 operations experience into our examiners because we may very 25 well decide that what is being done for training is not Heritage Reporting Corporation (202) 628-4888
62
()
1 sufficient and more needs to be done.
And this is one way of i
2 finding out.
3 So I am not sold that all training departments have 4
100 percent handle on what is going on at the plant.
5 MR. MOLDEN:
I would like to suggest there may be 6
other ways that don't. alienate the Training Department from the 7
plant.
And I think that that is one of the concerns as well as 8
making sure the exam is appropriate, in consideration of all 9
the aspects of how you evaluate people, how you develop exams, 0 that the operations people don't get involved with.
that's not 11 their job.
And I know what Lynn is saying is true, that you 12 are alienating the training department from the plant by saying 13 that we believe this person out here should be our O
14 representative because.
15 MR. HANNON:
Okay.
Thank you.
That is informative.
16 We'll take that comment under consideration.
It wasn't our 17 intent to do that at all.
18 MR. WIENS:
We kind of felt like we were cooperating 19 with the training department all the time, I mean, even more 20 so.
So if that feeling is there maybe we have to do something 21 to correct that feeling.
It is certainly not our intent, 22 because as you can probably see from the development of this, 23 we intend to work hand in hand with the training department to 24 the complete development, I mean more so than we ever have 25 previously.
So that certainly was not our intent.
O Heritage Reporting Corporation (202) 628-4888
63
()
l'
.MR. VAN HAUTE: ' Gentlemen, I don't know if I'm the 2.
only one;here that is actually in operations..
Is anybody else 3-here in operations?
Okay.
4 Over the last couple years, we've'had, training 5
obviously has made a big. change, and part of that training 6.
- change has been the accreditation process.
I can assure you 7
that coming from operations, as far as I'm concerned, and I 8
hope'I'm not shooting myself with this, accreditation has been 9
quite an experience,.and the people who know accreditation are
~
10 trainers.
And I would have to agree, I'd rather see, I'd-like.
11 to see somebody from operations, definitely representing the 12 operators, but I also do believe you would be very beneficial 13 to have somebody from training, because they know what.the O.
v 14 accreditation requirements are.
15 If we get a question --
16 MR. STAROSTECKI:
Let me just stop.
I hope you 17 understand that NRC is not locked into 100 percent saying we 18 believe accreditation has worked and we find your program 1
19 acceptable solely on that basis.
We are looking behind the l
20 accreditation process and we audit the INPO accreditation 21 process.
And I think you have to understand our perspective
'22
.from the regulatory point of view, we think it is a great 23 improvement and I agree with you.
It has been very successful.
24 But we can say that on the basis of our audits and looking 25 behind it through techniques such as being discussed here.
.O l
Heritage Reporting Corporation (202) 628-4888 l
L
64
'( )
1 So it's not necessarily the fact that because it's 2
accredited we don't look behind it.
3 MR. VAN 11AUTE:
I understand what you are saying and 4
I personally have felt that right along, that the accreditation 5
program is something that the utility is trying to, is 6
implementing to try to appease everybody, including the NRC, 7
and also that we have to be able to take an exam from your 8
point of view.
Because I've been around for a long time, so I 9
kind of know what that's coming from, I believe.
10 But we also have a team effort at my plant, and I 11 imagine most of the plants, and part of that team effort is 12 training.
If it's going to cause a problem, for these other 13 units, I think, and I don't know, maybe it would cause it with 14 us, I think you should reconsider it.
I'd like to think that 15 we're talking about honorable people everywhere here, in 16 operations and in training.
And I think we should build on 17 mutual respect.
And if this is one way of doing it, I think 18 you should do it.
19 MR. WIENS:
Let me respond to that, in that I, as it j
20 finally penetrated to me what the concern was, that I don't
{
l 21 disagree.
What I've seen at Robinson, as I said, what our 22 intent is, the training people and the exam team are working 23 hand in hand.
I mean there is no question that the training j
I 24 department is an integral part of the development of the thing 25 and down there it hasn't been a concern to become involved in.
O Heritage Reporting Corporation (202) 628-4888
_a
l l
65
(
.(~y
(_)
1 But if the perception is there that we're lacking in trust or 2
confidence in the training department up front, you know, maybe 3
we can do something to the wording or something to the program, i
4 MR. STAROSTECKI:
Yes.
But be careful.
We are still i
5 going out there auditing the requalification program and we are 6
going in with a very skeptical outlook and some training 1
7 programs may need to be augmented.
Plain and simple.
8 MR. VAN HAUTE:
I understand that.
We went through 9
a drill with INPO here recently in which my ship was subjected 10 to it at a short notice, like within a couple hours, couple 11 days.
Couple hours, actually.
And during that time we were 12 audited, the training department was audited along with us. The l
13 people who were on our simulator, they audited, the INPO group
\\#
14 audited our performance and they audited the training people's l
15 performance as far as their feedback to us, as far as our 16 critique after the scenarios and everything else.
I thought 17 that worked our pretty well.
18 I'm just saying I think you ought to involve 19 training. If you're going to have one person from operations, i
20 you should have somebody that's knowledgeable from training, 21 especially from the accreditation side of this thing.
22 MR. WIENS:
I feel like we can do what Rich said, and 23 that is, for example, as I say, we are working hand in hand 24 with the training department in developing our things, our 25 exams and everything.
O Heritage Reporting Corporation (202) 628-4888
66 r^T t, j 1
However, on the other side, in order that we feel we 2
do an adequate program evaluation, those people that would be 3
involved in the program evaluation, such as the graders of the 4
crew and the simulator exam, we don't allow those to be the 5
same people that are wocking with our team in development and 6
think that we don't want to feel a bias towards those people.
7 So we're trying to do both.
We're trying to work with them, 8
and we're trying to remain so we can do an objective evaluation 9
of the program.
l 10 MR. ROE:
If we could summarize, I think I. understand 11 the point that's been discussed now for about ten minutes.
We 12 should consider adding to the team, if you will, an advisor 13 from the training department who receives the same oath of
-s 14 confidentiality to the team and does not provide information.
15 MR. VAN HAUTE:
Or is independent, just like the 16 other guys.
17 MR. ROE:
And we will consider that, we will evaluate 18 that, as we evaluate the role of the operations department 19 technical advisor and see the advisability of adding a second 20 person on there.
21 MR. HANNON:
Okay?
Sort of lost track of where we 22 were, :are, though.
It's a good discuss an.
23 I 1.hink the final point on the simulator exam is that 24 we would take aorantage of any capability that might exist in 25 the simulator te have it videotaped provided we got the consent O
Heritage Reporting Corporation j
(202) 628-4888 i
l I
67 p
(_f 1
of all parties.
2 If everybody agrees we can use the videotapes, we'll 3
use them, if they're available.
4 MR. HELD:
John Held, Calvert Cliffs.
Are simulator 5
exams for the actual crew team evaluation, are there going to 1
6 be a set of criteria that we judge against and you judge 1
l 7
against, the same criteria, or how is that going to be 8
established?
9 MR. IIANNON:
Yes.
Let me let Shelly answer that.
10 MS. SPILBERG:
I'm in the process of developing 11 rating scales that will be used by both sets of graders, that 12 will be, that are intended to be performance-based and that 13 will bring us the highest reliability between those two groups O
\\#
14 as possible.
15 MR. HANNON:
Why don't you identify yourself?
16 MR. SPILDERG:
Shelly Spilberg, NRC.
17 MR. HANNON:
Okay.
Was that answer okay?
One more 18 question.
19 MR. CATELL:
Fred Catell of New York Power Authority.
20 Along that line, in terms of the crew evaluation, since I 21 submit probably all of us in this room are aspiring to meet the 22 info guidelines H6026 for evaluation forms, I assume you're 23 doing the same?
24 MR. HANNON:
That is correct, and if we have a need 25 to change our forms, to conform to guidance that is out that is O
Heritage Reporting Corporation (202) 628-4888 i
l l
60 1
1
_O 1
better then whet we greeenur heve, we obvioeur w m texe l
2 advantage of that.
L 3
(Slide) l 4
MR. HANNON:
Let me move on.
The next part of the 5
exam is a walkthrough.
I mentioned earlier that the facility
)
1 6
needs to help us identify what systems we would focus on in the 7
plant walkthrough portion of the. exam.
We would anticipate 8
getting that decision early on.
.9 It ideally would be based on some risk analysis for like.
10 the f acility or some, maybe a 14 hd plant has a risk analysis,-
j 9
11 but some systematic way for us to assure ourselves that the.
12 content of this exam.is important from a safe operation 13 perspective.
O 14 We will select a couple of those systems, maybe a 15 third one, to get comprehensive examination.
We anticipate 16 about a two-hour length and if there is need for follow-up from 17 the simulator, we could extend it for perhaps another hour.
18 We will have the question plan reviewed by the 19 technical advisor from the facility, the team member and have 20 an exchange interaction there to make sure that it is indeed a 21 performance base and it does have the right job performance 22 measures that are geared to the facility's objectives for your 23 requalification training program. And we also indicated that 24 whatever we're doing on that exam has to meet the test 25 specification which says it has a high KA rating.
O Heritage Reporting Corporation (202) 628-4888
69 l
(()
1 MR. McCOY:
Ken McCoy, INPO.
Another comment.
When 2
we reviewed this program we noted that in the simulator exam, 3
having a facility person there address the concerns that have 4
been raised previously about the quality of the exam and in the 5
written exam, having the dual grading does the same thing.
But i
6 in the oral exam, in the walkthrough, since this is an oral i
7 exam and is, has in the past been the subject of a number of 8
complaints on the quality of the exam, that there is not a peer 9
or someone accompanying the exam team.
10 MR. HANNON:
That's a good question.
And we have 1
11 carefully looked at how we would resolve those concerns.
And 12 we don't have an answer right now, but what we intend to do at 13 least for the initial trial is have each one of the plant
(
14 walkthroughs observed by an NRC supervisor and they will be, as 15 part of our quality control, quality assurance validity i
16 process, we will have a supervisor accompany each examiner 17 during the walkthrough and will provide a quality check on the 18 process. I'm not sure if that will need to be continued, or if 19 it can be done on a random sampling basis as we presently do.
20 But that is our initial approach, that's what we intend to 21 apply for the first exam.
22 MR. STAROSTECKI:
Let me just add something, Ken.
I I
I 23 don't know if this is going to be a sore point or not.
My f
24 experience is there have been some complaints in this area but l
j 25 not an overwhelming amount. I think this is an area where we
/*
1 ku l
Heritage Reporting Corporation l
(202) 628-4888
70 i
( (])
I will give the NRC examiners more training.
Our intent is 2
rather than have our examiners spending more time reviewing 3
your training materials, preparing the exam, our examiners will 4
be able to spend more time with hands-on time on the simulator 5
with a technical assistant, spend more time at the plant t
6 walking systems down.
7 By giving better training to our examiners, better 8
familiarizing them with the plant and better familiarizing them 9
with the simulator, I believe we'll have better quality 10 walkthroughs.
This is one area where I can see a lot of 11 improvements coming about, because we will have relieved the 12 examiner of some of the administrative burdens they 13 traditionally have had.
()
14 I think you will always have some potential for 15 personality conflict, and quite frankly, I don't know how we 16 get around that.
We just might have to deal with it when it 17 occurs.
I 18 MR. WIENS:
There is one additional measure of 19 quality assurance on this walkthrough that we do also.
The 20 examinars will have the test outline for the plant walkthroughs 21 which would include the topic areas to be covered and the job 22 performance measures to be evaluated.
Those will be prepared 23 ahead of time and they will be reviewed by the facility 24 technical assistant and if we go to a training department 25 technical assistant in the future, they'll be reviewed ahead of O
Heritage Reporting Corporation (202) 628-4888
71 1
time and any concerns will-be discussed and agreed to prior to 2
the examination.
So there is a measure anyway of peer review 3
so to speak on this exam also..
4 MR. STAROSTECKI:
Yes.
But I almost have to add that 5
I am not going to hamstring any examiner, if he goes out there l
6 and he starts doing walkdown and as we all know, when you pull 7
one string and it leads to another set of questions, and if 8
these questions are not on this outline, I expect the NRC 9:
examiner to follow up and see where they lead.
And that 10 doesn't mean they weren't pre-approved.
This is an area where
.11 we are going.to give an awful attitude to the examiners to find 12 out if the knowledge is there.
So this is an area we will all 13 pay attention to.
O 14 MR. WIENS:
We agree, Rich, and as a matter of fact, 15 on the forms we developed for this area, for those follow-on 16 questions that are asked during the walkthrough, we not only 17 anticipate but we expect those to occur.
What we expect our 18 examiners to do, however, is to document those questions and 19 then after the fact, they can get reviewed.
So they are still 20-going to get reviewed.
It's just like Rich said, we're not 21 going to hamstring the examiner.
He not only should but is i
22 expected to follow up.
I 23 MR. ROE:
Maybe I could give Ken a little bit more l
l 24-straightforward answer.
Ken, we have reviewed this particular j
25 problem, and we have discussed it a couple levels. We do not O
Heritage Reporting Corporation J
(202) 628-4888 l
L
i 72
()
I have-any strong-intention to adding to each one of these 2
walkthroughs a peer evaluator from the utility to watch what 3
his candidate says and to have some sort of an integral 4
process.
We expect this is going to be conducted by the NRC 5
person, and if there is going to be an evaluation, it will be
]
6 NRC people evaluating other NRC people, and'that the evaluation 7
that is conducted by the candidate is pretty much by that one 8
license examiner and it's not two people, one from the utility l
9 and one from the NRC,.that does it.
But I think you should j
10 know that we will continue to have an open mind. However, we j
11 have looked.at it very hard and we desire to have it as the l
.12 burden is on the NRC examiner, with input from the facility on 13 the-approach, but when it gets out there on the floor, that guy
()
14 has to make the judgment without any conversation with a peer.
15 MR. HANNON:
Another thing I'm looking to get I
16 calibrated on here is, and the reason I want to have the 17 supervisor along, at least on the first few trials, is if an 18 individual feels that he was mishandled, he was either 19 intimidated somehow by the examiner or was asked questions that 20 really weren't operationally oriented, or you know, some 21 problem he may have experienced, during the conduct of the 22 plant walkthrough, I intend to, I want to hear about that.
I 23 want that to come back through the critique process and then I 24 would like to discuss that with the examiner and his supervisor 25.
that was with him and let's see if there really is a disconnect O
Heritage Reporting Corporation (202) 628-4888 1
m
73
("3
,/
1 and maybe we need to look at how we're conducting these 2
individual walkthroughs, on an individual basis.
I 3
So we are sensitive to this area, we're trying to 4
improve it and we really do need some feedback in the exam 5
critique process so we can evaluate how well we're doing.
6 MR. EVRINGHAM:
I've got two questions.
First of 7
all, one on the written exam.
There are two parts to the 8
written exam, one on the frozen simulator, one in the 9
classroom.
Is there one grade, simply 80 percent overall?
10 MR. WIENS:
Yes.
11 MR. EVRINGHAM:
There's no section, you have to get 12 70 percent in either section?
i 13 MR. WIENS:
No.
14 MR. EVRINGHAM:
Okay, just one grade.
15 MR. WIENS:
One grade.
16 MR. EVRINGHAM:
The other question would go to this 17 operating exam or the second part of that question, this 18 operating exam again, similar to what we're doing now, and 19 licensing exams, there is one grade?
Correct?
The simulator 20 and the walkthrough as you're calling it here, are one exam?
21 MR. WIENS:
Yes.
Operating test.
l l
22 MR. EVRINGHAM:
Okay.
With the operating test, J
23 something that has I guess confused me for some time is, if you j
i 24 look at the things in the code that are required to be examined 25 over, I guess we're talking more of a licensing situation here,
(
l l
Heritage Reporting Corporation (202) 628-4888 E-_-__--_---
74 1
that it applies to requal.
I guess my question is what really 2
is the purpose for the plant walkthrough, other than to follow 3
up on weaknesses found on a_ simulator exam?
4 MR. WIENS:
To identify familiarity with plant 5
systems, controls.
It's one thing just to have a knowledge or 6
so to speak, a book knowledge or something.
It's another thing l
t 7
to demonstrate that they actually have the ability to locate 1
8 and identify different controls and things, particularly i
9 outside the control room which we cannot evaluate in the 10 simulator.
But that is reason that our plant walkthrough, 11 number one, is short, because we recognize there's not that 12 many things outside the control room, that an operator may be-13 directly. responsible for,.but that there are some things, and O
14 we cen't edeeeete1r eve 1uete thet, either, in the written exem 15 or in the simulator, so we have the plant walkthrough to
'16 evaluate those systems and controls.
17 MR. EVRINGHAM:
So these job performance measures 18 that.you are talking about checking four of, I believe was the 19 number, these jobs performance measures we are to choose from 20 the list of tasks basically, tasks that licensed operators
)
21 perform outside the control room.
Is that true?
22 MR. EATON:
Yes.
Let me give you an example.
I was i
23 on the IIT Team for the Rancho Seco over-cooling Event, and 24 they had licensed operators out there trying to override relief 25 valves, trying to operate manually auxiliary feedback valves.
O Heritage Reporting Corporation (202) 628-4888
75 (k
1-They didn't do a very good job of it.
And.I assume that that's 2
within their job description, because they were certainly out i
3 doing it and also trying to direct non-licensed operators to 4
doing that, which is their responsibility.
5 I assume that Maine Yankee, last time I was there, 6
that the. licensed operators had the responsibility of directing 7
the non-licensed ~ operators if they had questions.
8 MR. EVRINGHAM:
Well, I'think that the licensed 9
operators, especially depending on which plant and'how.they're 10 perhaps set up, there is always a responsibility for giving 11 direction, giving order to open such and such a valve or 12 perform such and such a procedure.
But I guess we may have a 13 little bit of a disconnect problem in the transition here.
If O
14 you look at what most accredited programs would call a job 15 performance measure today, these job performance measures are 16 going to be based on that person's job in the control room.
17 There are, and I can only speak for my. plant, there are very 18 few operator, licensed operator tasks outside the control room.
19 MR. EVRINGHAM:
Would this be the type of job 20 performance measures you want to see?
'21 MR. HANNON:
I think the answer is we are going to 22 have to be careful about what we use, but the requirement 23 exists for a walkthrough, as part of the operational test and 24 it has now been focused for the requalification examination, i
25 and if it indeed turns out to be a disconnect, when we go to O
Heritage Reporting Corporation L
(202) 628-4888
_________-_-__________a
76
()
1 the facility, and that we cannot identify either the number of 2
systems that are necessary or that when we have the systems we 3
can't connect that with the job, with the control room 4
operator's task, I personally don't see how that can be, but if 5
we do reach that impasse, we would obviously have a problem and 6
it might be a problem with your training material, and it might 7
be a problem with our exam approach.
I don't know.
But there 8
definitely would be a problem.
We haven't crossed that bridge 9
yet.
I don't think we've encountered that difficulty at H.B.
10 Robinson.
11 MR. STAROSTECKI:
I think you have to keep in mind, 12 when we issue somebody a license we're doing it in accordance 13 with our regulations and we expect a certain level of knowledge
(}
14 about the plant, the systems, and what's really out in the 15 plant, and the same goes true with our requalification.
Part 16 55 says we expect a working knowledge.
And I appreciate what 17 people are saying about accreditation, but there comes a point 18 in time when we say we still want that knowledge of that plant 19 exhibited and our approach has been that the best way to do 20 that is to do a plant walkthrough.
And if that causes somebody 21 problems, I would offer that maybe we ought to revisit the 22 accreditation process.
23 MR. HARRIS:
I'll take a different position and agree 24 with what you are saying.
I think we in the utilities feel 25 that he needs to know what is outside of control and O
Heritage Reporting Corporation (202) 628-4888
77
,m) 1 development.
(
2 MR. WIENS:
Yes.
Robinson, they identified, they did 3
not have a problem identifying systems outside the control room 4
for which to demonstrate during the walkthrough.
l 5
MR. HANNON:
John, do you have some questions?
6 MR. CROSS:
Jim Cross from Grand Gulf.
7 One of the reasons I think for coming up with this 8
new program is to identify the training needs and try to get 9
the examination needs to coincide.
That's b in a problem in 10 the past.
11 This program, the walkthrough portion, is probably 12 the least identified program that we talked about today and I 13 think the one we need to be most careful about.
The standards
(
14 are not defined.
In fact, in some of the training programs 15 under INPO accreditation, there's a lot of leeway in the plant 16 knowledge of reactor operators.
17 We believe that they should be examined in the plant 18 and that's very appropriate.
But the standards are not 19 identified.
20 The objectives that the NRC is going to examine on 21 are not identified.
You take the simulator training, that was 22 very identified.
The examination program was very identified.
23 The walkthrough is not very identified.
And the feedback l
24 portion is missing in the walkthrough area where the examinees I
i 25 are allowed to feed back to the NRC what they thought about Heritage Reporting Corporation (202) 628-4888 i
E__ _
l 78
()
1 this process, l
2 I would recommend that the walkthrough portion 3
probably needs the feedback more than the other two.
4 MR. STAROSTECKI:
I don't want to sound defensive, 5
but walkthroughs and written exams were administered long 6
before there was accreditation.
Okay?
Before there was an 7
INPO we had walkthroughs.
And quite frankly, the feedback, and 8
I've only been in the operator licensing business since,hout 9
1981 and I have not heard much feedback in complaining about 10 the walkthrough as much as I have about the lack of quality in 11 the written exams and the simulator exams. We are saying that 12 we are refining the requalification program as we administer 13 it.
You are proposing a radical overhaul.
O
\\
14 Yes, there may be some problems with our 15 walkthroughs, but they pale by the problems we've had with the 16 simulator and the written.
And I have to tell you, from my 17 perspective, we have more serious issues to deal with, and I 18 think we can maybe have some more questions and answers on this 19 and share points of view, but I'd like to refine what we're 20 doing first with the simulator and written before we tackle the 21 broader question, because quite frankly, we're not throwing out 22 the NRC examination process, we're trying to refine it and make 23 it better.
24 MR. ROE:
I think you have uncovered a point that 25 we're all aware of is that we consider the operator's normal O
Heritage Reporting Corporation (202) 628-4888 s
79
()
I work station the control room, and we're focused on that, and 2
we know that he has got responsibilities associated with the 3
direction he gives to other licensed operators and unlicensed 4
operators in a normal and abnormal situation to operate things 5
in that plant.
He has got to know what he has told them, where 6
to do it and how to do it, and what the impact is.
7 I think we have seen in a lot of utility approaches 8
you don't get in the control room without going through the 9
whole program in the facility, and I understand, it
's like 10 every other command situation, is that when you tell somebody 11 to do it, it's most important that you understand what you have 12 told them to do and many people believe that you have got to 13 have done it several times yourself, so that you understand
(~h s/
14 what you have directed be done.
15 And I notice that we all seem to take a look now at 16 our programs.
There may be a weakness there and we're going to 17 have to focus in on that as we go through this and see 6f 18 there's some of the other things that we've laid on the top 19 that we haven't really done the job in that particular area.
20 MR. HANNON:
I would also point out that there is a 21 feedback process, and that the last part of the written exam 22 which is the last thing the examinee will experience will be an 23 opportunity for him to give us some feedback on the entire 24 process, including the walkthrough portion.
25 Any other questions we can take right now?
A couple O
l Heritage Reporting Corporation (202) 628-4888
80 (G
1 over here?
l
_j 2
MR. CATELL:
Fred Catell of New York Power.
With 3
regard to licensed operator tasks outside the control room, we 4
have identified quite a few and in most cases, they are 5
procedurally addressed.
One, I think, that you would be 6
looking at particularly would be safe shutdowns from outside 7
the control room,-which there is a procedure for.
Will this 8
portion of the exam be open book as it would be normally for i
9 the people in the plant operating this way?
Will they get to
~
10 carry procedures around with them?
11 MR. WIENS:
If, you know, for example, you bring out 12 other safe shutdown from outside the control room, at least 13 every plant that I've been to they have a procedure located at 14 that station. And he would be allowed to use that procedure, 15 yes.
16 But anything, as we say, this whole exam series, the 17 written, the simulator and the plant walkthrough, is designed 18 to allow the operators to use those reference materials that 19 would normally be available to them in their actual operating 20 conditions.
l 21 MR. STAROSTECKI:
Be careful.
If you're doing a 22 walkthrough, the procedures for the remote shutdown panel 23 should be in the vicinity. The FSAR most probably is not. So 24 although the FSAR, the procedures may be available to them in 25 the control room, when you're doing a plant walkthrough I.would O
Heritage Reporting Corporation (202) 628-4888
81
()
1 not expect that to be a readily available reference.
2 MR. WIENS:
That's right.
It is readily available 3
under normal conditions.
So you can't say we're going down to 4
a certain station and realize he wouldn't have that but he 5
knows they are there so he picks up a whole bunch of references 6
and brings with him.
It's what is normally available in those 7
stations and those situations.
8 MR. HANNON:
Can we take some more questions?
9 MR. SMITH:
John Smith, Florida Power Corporation.
10 Would a walkthrough be the same for those of us who do not yet 11 have a simulator? And what will you do about the simulator 12 portion of the exam?
13 MR. HANNON:
Well, as again mentioned, we do have a 14 proposal in. It's under study right now.
It's a little 15 premature for us to comment on that.
We briefly mentioned, one 16 of the things we are contemplating to do for the simulator 17 freeze portion of the written, for the walkthrough, will be 18 necessarily expanded to include a more thorough scenario 19 discussion, so there will be some differences that we'll have 20 to accommodate.
21 We're going to work those details out when we conduct i
22 the exam in Region 4 at Fort Calhoun.
23 MR. STAROSTECKI:
I think that's another good topic.
24 I think we would appreciate some comments and views from you 25 all on what we ought to do to get a better understanding of Heritage Reporting Corporation (202) 628-4888
)
82
(
1; integrated system operation.
That's :really what the simulator 1
2 allows us and for those. utilities that don't have a simulator,-
3 I would hope.that they do have some ideas and approaches, and j
4 let'us know about them.
5 MR. HANNON:
There is a meeting at Fort Calhoun with-l l
6' Fort Calhoun, I think, in Region 4 during the first week.in l
7 December and we're going to get their views and their thoughts 8
'at that meeting _and of course we'd be open to any other 9
comments you might have.
10 MR.'GASPAR:
Joe Gaspar,. Fort Calhoun station.
11 Yes, Joe, I think we have given it some thought and 12 we're struggling with the issue right now. And we don't have a 13 lot of good guidance.
The only one we have looked at a little 14 bit is we have modified the CE simulator to better represent 15 Fort Calhoun.
That's been our only real good idea at that 16 right now.
17 MR. EATON:
Do you use your procedures there, Joe?
18 MR. GASPAR:
Yes, we use our own procedures.
But 19 doing that in two months or three months from now we haven't 20 validated those changes on the CE simulator yet.
We were 21 alming them something.like at April, requalification cycle. So 22 it gets into plant specifics here.
But I don't have a good 23 way. Walking through scenarios is the only other way that we've 24 thought of and I think you've already discussed that.
25 MR. EATON:
Are you going to be down in Region 4 O
Heritage Reporting Corporation (202) 628-4888
83
()
1 meeting?
2 MR. GASPAR:
Yes.
We will be definitely meeting down 3
there en masse probably, now, to compete with you.
4 MR. HANNON:
Don't get too constrained by the 5
tentative schedule that I mentioned, too.
We're not under that 6
kind of pressure that we would have to conduct this exam before 7
you would be prepared to do it.
8 Do we have any more questions on this area before we 9
move on to the written?
10 MR. CARROL:
I just want to make sure I understand.
11 It's not your intention that anything around the job 12 performance measure is actually performed?
13 MR. WIENS:
That's correct. It's a walkthrough or a, 14 you know, just as we do it now.
We do not expect the operators 15 to actually manipulate the controls.
16 MR. CARROLL:
It may be just the semantics. At least 17 on my point, where you're talking about job performance 18 measures, that really details the actual job.
It would 19 probably be simulated, and the past failure criteria, the man 20 would have to do it or he'd have to control the flow rate, 21 something to that effect.
Maybe it would be more appropriate 22 for you to ask for the task list that are done outside the 1
23 control room.
If you are all looking for the knowledges and
)
24 skills that are derived from that task, for most cases the job 25 performance measurement won't do that.
They would be imbedded
)
l Heritage Reporting Corporation (202) 628-4888 l
i
..,,,,,,u,,u,,,,,,,,,,.,,,
,,.,..c, I
84
()
1 in the systems, at least in our place.
So maybe it is just a 2
question of semantics.
Bob Carroll.
I'm sorry.
3 MR. WIENS:
Thank you, Bob.
4 MR. GOUTH:
Ray Gouth, Commonwealth Edison.
The 5
simulator exam and the written exam that will be parallel 6
graded, is there any consideration for parallel grading on the 7
walkthroughs?
8 MR. ROE:
The answer is there is no consideration for 9
parallel grading on the walkthroughs.
10 MR. HANNON:
Okay.
If there are no further questions 11 on that area let's move and talk quickly about the written 12 examination.
13 (Slide) 14 MR. HANNON:
Again, we have indicated that we desire 15 it to be tied to a high safety significance and operational-16 oriented issues, and we identify with our KA catalog as being 17 our method for assuring content validitf and overall integrity 18 of the process.
So that is one of the ways that we're going to 19 be Q/Aing the written exam, making sure that it is tied to our 20 KA catalog.
21 We also would of course need to have the facility 22 propose the questions, and at least in the first few rounds l
23 we're going to be working closely with you on this because none 24 of us have a whole lot of experience in creating good, open 25 book type questions.
And we have some guidance.
And Shelly
/~l 1
\\_)
f Heritage Reporting Corporation (202) 628-4888
{
l l
1 l
85 l)-
1 has helped produce some guidance on how to go about doing that.
l 2
So it is going to be a learning process for all of 3
us.
4 MR. STAROSTECKI:
It is clearly, though, this is 5
where a major industry effort would be helpful to us in that i
6 ideally each utility ought to have its own exam bank, and 7
questions oriented specifically for this kind of open format 8
and use of procedures.
9 If the INPO accreditation process really works and 10 goes in the right direction, this would be a big start in my 11 view.
12 MR. HANNON:
Do we have any questions, now?
13 MR. CATELL:
Ili.
Fred Catell.
Just a general big 14 picture question on the written.
15 Are we looking at the same categories as we have in 16 the past but a different type of question and also in terms of 17 exam bank, if you wanted to come in to my plant and give an 18 exam, big picture, ballpark number, how many questions would 19 you like from me ahead of time?
20 MR. HANNON:
It's a good question.
Let me start and 21 then I'll let Lynn supplement it.
The test specification that 22 we created for requalifcation goes to two basic, fundamental 23 areas.
One is proficiency on the job and the other is currency l
24 on the actual plant mods and changes in the control room and in 25 the plant.
O lleritage Reporting Corporation l
(202) 628-4888 j
86
()
1 We looked, in terms of how do we decide that this
-2 written exam meets our needs.
We have to rely on the 3
regulation.
It identifies the 13 areas of coverage.
And then 4
we rely on your requalification training program, what you have 5
identified the needs for your people are at that time in their 6
experience.
7 Somehow we have to meld these two and perhaps we may 8
be starting from a wide discrepancy, and come to some final 9
conclusion that we have a comprehensive written exam.
It has 10 to deal in the areas of infrequent, normal occurrences at the 11 facility.
Perhaps if you've had a long operating history, a 12 lot of time, a high availability, it might be plant startup and 13 shutdown, that would be an area of interest for the 14 examination.
Abnormal and emergency operations.
High 15 consequence, low frequency type of events would be an area that 16 we would be interested in focusing on in the requalification 17 examination.
I 18 Lynn may want to supplement that.
l l
19 MR. WIENS:
I think I'll find out whether that 20 answered your question or not because it seemed fairly decent.
l 21 The only thing I might add is that we'll look at the exam to 22 ensure it covers the necessary areas required by regulation in 23 the 13 areas that we are required to examine against in Part 24 55, 25 MR. STAROSTECKI:
But it is clear we are not going to l
r l
Heritage Reporting Corporation (202) 628-4888
87
()
I have the traditional four-part written exam.
2 MR. WIENS:
No. It won't even be similar.
3 MR. STAROSTECKI:
We are departing from that.
4 MR. HANNON:
That's correct.
There won't be any 5
resemblance to the initial exam with this requalification exam.
6 As you can see already, it's a different format.
7 With regard to our ability to determine that the exam 8
is comprehensive, we have left some " wiggle room" here and 9
allowed that up to 20 percent of the examination may be 10 sampling areas that are outside the scope of your present i
11 requalification learning objectives, for example.
12 The reason we do that his we're taking, you know, as 13 part of the INPO accreditation process, that you have 14 successfully identified your needs for' training for your 15 facility for your job incumbents and this 20 percent 16 availability here that we have allowed would have, give us a 17 chance to sample that, to verify it.
18 If, for example, there is an area in the 13 that are 19 listed in the regulation you all have not trained on and you 20 have no learning objectives or you have not been using those 21 learning objectives for the last cycle, we might want to sample 22 a couple of questions in that area, as a way of verifying your 23 determination.
And again, we also mention that it's going to 24 be with the learning materials or the operational materials 25 that the job incumbents have available to them during their O
Heritage Reporting Corporation (202) 628-4888
88 O
1 vertermemce ce ene Joe.
2 Two questions.
Either one.
3 MR. CATELL:
Number of questions?
4 MR. HANNON:
The number of questions, we haven't 5
decided yet.
I think it's still up in the air.
We're going to 6
have to, it's part of the learning process here to decide how 7
much time the exam takes and we don't, I don't know, do you 8
have a feel for that?
9 MR. WIENS:
No, I don't. I was hoping Ron Eaton would 10 have. gotten back, because he may have a better feel for the 11 number they have developed so far.
You know, for the number of 12 questions on the exam, though, we're not looking at a large i
13 number of questions, at least in our initial thinking on it.
14 Maybe down in the 20 to 30 questions category.
We'd like to 15 see more questions than that prepared so we can select from 16 among those proposed.
17 MR. HANNON:
The question came up at HB Robinson, and 18 we were going in with a certain request for a number of 19 scenarios, a number of written questions, because of the 20 experimental nature of the process here, in fact, we don't have 1
21 a lot of experience from anybody, for doing this, we have to be i
22 flexible.
If we ask for 30 questions and we can only produce 23 20, then 20 is 20.
So it's not cut and dried right now, and 24 we're going to have to get some experience.
25 MR. VAN HAUTE:
On that, will it be time-validated, O
Heritage Reporting Corporation I
(202) 628-4888 1
89
()
1 that test?
2 MR. HANNON:
The intent is to have the technical 3
advisor take the examination and validate the time.
4 MR. VAN HAUTE:
What if you find a deficiency from an 5
operator's standpoint?
If you find a deficiency in the 6
training program, in the question, and the operator does not 7
answer the question and it is because he hasn't had that 8
training in the last year, does the operator fail or does the 9
training department fail?
10 MR. WIENS:
Both.
11 MR. STAROSTECKI:
The nature of the question ought to 12 be determined by what the nature of the individual's duties are 13 and what his license is for.
He is not licensed to take
()
14 training once a year.
He's licensed to operate that plant 15 safely.
16 MR. VAN HAUTE:
All right.
It's a continuous 17 process.
I mean if you find a problem with the training, 18 though, because I think we've all gone through this where we're 19 coming in to check training and it ends up being the operator 20 that takes the heat.
The operator gets the brunt of it.
21 MR. ROE:
There are really two major objectives to 22 our program.
Sometimes we lose sight of this.
We have two 23 things we're going to accomplish.
The first thing we 24 accomplish is that we want to make an evaluation of the 25 training program at the facility.
That is number one.
Number O
Heritage Reporting Corporation (202) 628-4888
90
()
1 two, which is a co-equal, is that we want to make an evaluation 2
of the individual's capability to safely operate the plant.
So 3
we're going to do both things for them.
We lose sight of that 4
sometimes and forget that those are the two accomplishments, 5
and we come out, we want to make it a licensing decision on 6
that individual, continued operation of the facility, and then, 7
the six-year license, the capability of relicensing them, and 8
to meet that regulatory requirement and also to make a 9
determination of the acceptability of the training program at 10 that facility for not only that operator but all the ones that 11 we didn't check.
12 MR. WIENS:
For example, if the training department 13 dropped the ball, and so some of the operators are not safe and
(~%
'~#
14 competent operators, it may be the training department's fault 15 that they didn't train you properly, but at NRC,it is still our 16 obligation to ensure that until you have been retrained and 17 brought up to a safe and competent level, that you are removed 18 from the ship.
19 MR. VAN HAUTE:
If the impression is for safety, that 20 is certainly true.
21 MR. STAROSTECKI:
Let me just amplify on that.
You 22 can't answer that question black or white here, because you 23 have to know a little bit more, because one of the questions l
24 I'll have is if that operator hasn't attended any 25 requalification training lectures for the last year because O
Heritage Reporting Corporation (202) 628-4888 L
91 O
(_/
1.
he's taken it all home and raid he's done'it on his own time, I 2
have a problem with both the operator and the training program.
l 3
MR. VAN HAUTE:
That's understandable.
l l
4 MR. STAROSTECKI:
It's not a simple black and white l
l 5
issue, because you have to look at the kind of training that 6
the individual is getting, whether it's one week out of seven, 7
or whether it's a two-week shot once a year.
And I recognize 8
you don't see much of the latter.
It's also the nature of the 9
question.
10 MR. VAN HAUTE:
There's a large spectrum of various 11 training programs, that's obvious. I mean, that's what the 12 requirements are. Coming from ours, we have to maintain at 13 least 80 percent attendance so that wouldn't be the case.
14 My' final question I think is during this interim 15 period, which you say would probably be a year, and I'll tell 16 you, I think it's going to be longer than that, in order to 17 make it a quality program, especially with the questions from 18 our side, if you are subjected to the pilot program and you 19 pass the pilot program, will your license be renewed for six 20 years and if everything is all right, will you get that?
21 MR. HANNON:
Yes.
The position that we have taken 22 here is although these requalification examinations in the 23 transition period are considered trial, experimental 24 approaches, that their results will be valid.
So if you are a 25 proud possessor of a six-year license and you pass this O
Heritage Reporting Corporation i
(202) 628-4888 I
92
(~)
1 examination, you will have met the wicket for your renewal.
V 2
MR. VAN HAUTE:
Do you know who the representatives i
3 will be in other regions at this time?
You mentioned Region 4 l
4 but you haven't said anything about 1 or 3.
i 5
MR. HANNON:
We have not identified the exam team 6
members yet. Is that your question?
The only other one that we 7
have an inkling on now is Fort Calhoun.
0 Okay.
The rest of this slide, I'll point out that 9
the written exam basically has two parts and we might want to 10 touch on that briefly, is that why we have two parts. One is 11 given in the simulator and that's to check proficiency; 12 Remember that I said that that was one of the major components 13 that we were looking for in the requalification area when we A)
(_
14 went back and started from zero and said what should be the 15 NRC's objective and test specification for requalification, and 16 we feel that at least you've heard our reasons for why we think 17 that the frozen simulator is one way we can get a plant 18 proficiency check in a reasonable job performance measure.
19 The other area is the plant's limits and controls 20 sections. And we believe that on is appropriately done in a 21 classroom type of setting and we would anticipate having 22 whatever operational material, written material that would be 23 available to the operator in the control room available to him 24 in that classroom environment when the exam is administered.
25 And again, if there's eight people, I don't know quite how this
()
l I!eritage Reporting Corporation (202) 628-4888 1
1 u___----_-__--_-----------
3 93
()
1 is going to work. We may need to have eight different copies of 2
stuff to make sure that there's time.
There may be another 3
way for us to do it.
We might have to run people in in shifts, l
4 do four, if you only have four copies.
So those kind of 5
details we'll have to work out on an individual plant basis.
6 MR. EATON:
You've got all kinds of questions.
7 MR. HANNON:
Yes, let's take a few questions.
8 MR. WILLIAMS:
One related to drawings, you obviously 9
have all the plant drawings available in the control room, at 10 least on microfiche or something. Are we talking just 11 procedures or are we also looking at the drawings.
Lee 12 Williams, Alabama Power.
13 MR. HANNON:
Ron, do you want to answer that?
They 14 want to know if we're going to need to look at drawings.
15 MR. EATON:
I think it's going to have to be whatever 16 the test, when we make up the test we'll have to identify that j
17 at that time.
There may be a necessity to have some drawings, 18 but not necessarily all of them.
You're not going to want to 19 have all the P& ids in there, because we're just not going to 20 cover them all.
But we'll probably identify the ones we need 21 plus a few more.
I mean, we're not also going to necessarily 22 focus the guy in on the specific problem.
23 MR. 11ANNON :
Let's take another question.
24 MR. CARROLL:
I was wondering how you justified 25 eliminating the area of recall totally from the written exam.
Heritage Reporting Corporation (202) 628-4888
94
( ])
1 Through our analysis, the operators have said this, you need to
.2
.know this, and we've written an objective and now you basically 3
are taking that away.
4 MR. EATON:
No, we haven't, not really. If you take a 5
look at the classroom one that includes the emergency operating 6
procedures, there is a necessity for' recall on immediate 7
actions'and we've been hammered pretty well on that by all the 8
utilities on requal, or recall in areas where it's not desired 9
and where you don't teach to it.
But there is definitely a 10 necessity for it particularly in the emergency operating 11 procedures, and that would be addressed in that Part B.
12 MR. CARROLL:
What if your objective is beyond the 13 emergency procedures requal.?
()
14 MR. EATON:
If it's identified by your learning 15 objectives as an area where he must memorize it, then it 16 certainly is fair game from a recall standpoint, but don't 17 forget the majority of the test is open book.
And so while you 18 may require him to requal, you have the material available to 19 him in the classroom, he still has that material available to 20 him.
21 MR. CARROLL:
Well, that I think is the problem 22 there.
It's just the question, if the subject matter excerpt 23 identified this as something that needed to be known, something 24 that although he could look it up and maybe that the transients 25 that occur are so rapid that he really has to know the level of O
Heritage Reporting Corporation (202) 628-4888
1 95
()
I this tank in terms of gallons, because if you go back, by the 2
time you look at the book, you've already lost it. How do you j
3 preclude that, from going to the book and looking it up?
Seems 4
to me that if you're fa.thful to the objective, --
5 MR. ROE:
One of the points is don't forget that i
6 there's the walkthrough portion and my understanding is in some 7
of those walkthroughs we intend to ask questions on immediate 8
actions.
And that person standing there is going to have to 9
answer those questions with reference material he normally 10 would have which is only himself.
11 And that will be addressed in some of the, they won't 12 be just systems things, they will be immediate actions and 13 those types of things that are imiaediately recall.
However, we
[}
14 do not want to get into the approach that we had before is to 15 have a tremendous amount we expect for requal -- recall, when 16 we expected actually, in a normal operating environment, that a 17 person would have checked the reference material, such as 18 one-hours and tech. specs and things.
19 MR. CARROLL:
We have no argument there.
And that's 20 why up front we did an analysis, would the guy do it from the 21 book, and if so, then it said, given the procedure, follow it.
22 It seems to ine that as long as you are faithful to the 23 objective, you wouldn't have a problem.
24 MR. WIENS:
We plan to stay faithful to objectives.
I 25 MR. HANNON:
One of the things you have to keep in O
f Heritage Reporting Corporation (202) 628-4888 i
i 3
r 96
- O.
1 mied here 1 there e ectue11y three different modee of p
2 examination that we'll be experiencing, maybe even four now 3-with the two-part written exam.
So I. don't, I think that.the 4
appropriate mode for testing the requal -- recallabilityLis 5
perhaps on a walkthrough and that's where we have it.
And also l
l 6
in the simulator, because if the " gent" or the' operator forgets 7
.something that he should have known on. recall, that is.one of 8
the things that's going to be able to be observed and we'll 9
follow up with it on the walkthrough.
10 MR. SELLMAN:
Mike Sellman, Northern States Power 11 Company.
Is there any difference in a written exam between i
12 the ROs' and SR0s'. exams?
13 MR. HANNON:
There is a difference.
I'm not sure-
, 0 14 exactly how that will play itself out.
Do you have a comment 15 based on your experience?
16 MR. WIENS:
At Robinson we had an exam -- although I 17 haven't seen the two different exams -- there is going to be.an 18 RO-and an SRO exam, which will not be the same exam.
The 19 facility is preparing the questions, and we're reviewing those 20 questions based on the learning objectives of the facility and 21 our own KA catalogue evaluating between RO and SRO required 22 knowledge.
We are doing a QA check on what is proposed to us 23-in that area.
24 But I haven't seen the exams yet to see how 25 successful we are in having truly different exams.
I O
L Heritage Reporting Corporation (202) 628-4888
97
('\\
(_/
1 MR. HANNON:
Another thing that we might find is 2
that when you try to do this exam, it depends on what your own 3
delineations are in your own training program.
If you train 4
your reactor operator to the same level that the SROs are 5
trained to, then perhaps the only distinctions would be in the 6
simulator.
Where we are doing the crew evaluation, we are 7
looking at a different job performance measure for the senior 1
8 reactor operators.
)
i-9 So, I think that's something that we're sensitive to.
10 We're going to be flexible in how we deal with that.
11 Any other questions on the written?
12 Yes?
13 MR. JOHNSON:
Joe Johnson, TVA.
Did you say that the 14 technical advisor would take the exam to validate the estimated 15 time on it -- the written?
Did you say that?
1' 16 MR. HANNON:
Yes, I said that.
That is our intent.
17 We need to have some feedback to verify how long this thing is 18 going to take.
19 MR. WIENS:
That's not during the exam, though.
I 20 mean, he takes this two weeks ahead of time, or when the exam 21 is prepared.
22 MR. JOHNSON:
So when we're recommending the 23 technical advisor from the plant, is this is an individual who i
24 will be taking the examination prior to the candidates taking 25 it to establish some type of validity on the time?
(
Heritty.
Reporting Corporation (202) 628-4888 i
L_______.
98
()
1 MR. HANNON:
That's correct.
2 MR. MOLDEN:
Won't that person have already reviewed 3
the answers?
4 MR. HANNON:
He will have seen parts of the exams.
5 Don't forget -- he's going to be working with us, and will have 6
done some QA on the exams.
But he might have done a broader 7
scope than what actually finally appears on the written exam 8
itself.
So what we're simply trying to get in is not to write 9
out, necessarily, all of the answers.
What we're trying to get 10 him to do is to tell us, " Hey, is an hour and a half adequate 11 for the operators here?"
"Did we come up with something that 12 is exorbitantly longer?"
13 He's going to give us a QA check on how long it is 14 probably going to take the operators to do this thing.
That's 15 all.
16 Another question?
17 MR. JOHNSON:
Yes.
Joe Johnson, TVA.
Certainly 18 recognizing that an open-book format, in my estimation, is 180 19 degrees from what we have been doing, surely the NRC would 20 allow some degree of flexibility in the administration of the 21 exam such that you have some leeway on that time within the 22 discretion of the individual administering the examination.
23 Four hours plus or minus thirty minutes -- or something like 24 that.
This is dramatically different.
We have no historical 25 frame of reference to indeed support the fact that your em
'b f
Heritage Reporting Corporation l
(202) 628-4888 I
__________________a
\\
99 r
l 1
estimations are in fact sound.
2 MR. HANNON:
Good point.
3 MR. EATON:
I would-hope that we could come within 4
the ball park on the individual items, and then maybe l
5 cumulatively come up with some reasonable amount of time.
And l
6 it may be that the examinero will have to allow an additional 7
twenty minutes or an additional half an hour.
8 MR. JOHNSON:
It just needs to be respected.
9 MR. EATON; Yes.
10 MR. JOHNSON:
It has not been within the authority of 11 the individuals administering the examination in times past.
12 MR. EATON:
This is a new system.
13 MR. WIENS:
We plan to be, as John almost called it, 14 almost infinitely flexible on the time for this examination.
15 (I aughter )
16 MR. HANNON:
Other questions?
17 I think I have one more slide here.
18 (Slide projected) 19 MR. HANNON:
We talked briefly just about the program 20 criteria that we will be using.
There is not a whole lot of 21 change from what we have been doing in the past.
A lot of this 22 will look familiar.
23 We're doing parallel grading.
We'd like to see, for 24 a satisfactory program, 90 percent agreement on the pass / fail I
I 25 between your graders and ours.
We'd like to see the scores O
Heritage Reporting Corporation (202) 628-4888
s.
l
-100 l
l' comeEcut~within five percent on at least 80 percent of the 2
exams, and that we have an overall pass rate from the 3
individuals who.take the exam of 80 percent.
That's comparable 4
to what we're doing now.
l 5
Basically, this came from tne pilot program that we.
6 did earlier.
The'only additional thing here is that there is.
7 no major deficiency.in the simulator grading.
Your evaluators 8
come to the -- when we observe the critique, we would.say, 9
" Yeah, they did a good job in evaluating the performance of i
10 that crew.
They came up with the same things we came up with."
11 If'there are major discrepancies, they will have to 12' he discussed, and a determination made on the' impact of that.
13 That's for the program.- For the individuals, they
()
14 need to make a grade of 80 on the written, and be satisfactory i
15 on plant walk-through.
In other words, meet the job 16 performance measures of.the training program and shew that 17 there was no gross error committed during the simulator portion 18 of the exam.
19 Yes?
Let's take questions.
20 MR. MOLDEN:
Is there a minimum number of people that i
21 you require to participate?
Say you have to give program 22 evaluations --
23 MR. HANNON:
Well, this goes to a broader question of 24 how we keep running commentary on the status at an individual
'25 plant.
We have talked about a number of different things here.
O Heritage Reporting Corporation (202) 628-4888
101
/q-
\\j 1
One is a rolling sample, where you keep the lightest, minimum 2
sample size being the dominant or significant point.
With 3
the last twelve folks that were examined, you look at those 4
people and their grades to do your program evaluation.
5 We may only talk to about eight different people if 6
we are just looking at two crews -- for example, at H.B.
7 Robinson.
So, we'll need a couple of site visits before we'll 8
be able to make a program evaluation.
We might want to combine 9
these eight folks that'we took this time with four of the ones 10 we had before.
11 You may notice that we have identified that a minimum 12 of twelve people is necessary for us to have an adequate sample 13 size to evaluate any facility's training program.
So when we n
k-14 accumulate a number of twelve people, then we are in a position 15 to make an evaluation.
16 Another question?
'17 MR. JOHNSON:
Do you appreciate that if you don't go 18 ten, fifteen, or twenty, then 80 percent of any number gives 19 the edge to the NRC and penalizes the utility?
20 MR. HANNON:
I appreciate that, although I don't know 21 that it has been major determinant in our past practice.
22 MR. JOHNSON:
Only to those of us who have lost their i
23 jobs.
24 MR. HANNON:
We'll consider that.
I hope that that 25 doesn't become a major concern with this new program.
O Heritage Reporting Corporation (202) 628-4888 1
102
.()
1 Any other questions?
2 MR. GENCIUS:
Terry Gencius, Wisconsin Public 3
Service.
Are the consequences of a fail on the individual 4
' portion of it the same as what they were before?
5 MR. HANNON:
The question is:
Are the consequences 6
of a fail on the part of an individual the same'as what they 7
were before -- "before" meaning, I presume, before the new rule 8
became effective.
The answer is that what we'll do -- as we 9
have in the past -- if there is an individual who fails the 10 requal. examination, we would expect the individual to be taken 11 off duty and be given remedial training and re-examined either 12 by the utility or by the NRC and found satisfactory before 13 being placed back-in the watch-standing role.
-( )
14 If he has failed a portion of the examination, we may' 15 determine that we will have to re-examine him again before he 1
i 16 can meet the requirement for renewal.
If we do, we will tell 17 the individual what part of the. exam he has to take -- either
]
l 18 part of it, or the whole thing.
Those decisions will be made 1
I
)
19 on a case by case basis.
20 MR. STAROSTECKI:
What you're saying, John, is before 21 a license can be renewed at the end of six years, he has had to 22 have demonstrated satisfactory written, simulator, and walk-23 through.
24 MR. ROE:
Right.
25 MR. HANNON:
Another thing to be aware of is that we j
O Heritage Reporting Corporation (202) 628-4888 l
l l
L______________
103
()
1 have been taking the position -- I think this is fairly firm --
2 that if you fail part of an operating test, because they are 3
inter-related -- the walk-through portion and the simulators --
4 even if you only fail part of it -- say an individual made a 5
gross error on the simulator but everything else was okay.
We 6
would want him to take the entire exam over again because they 7
are so closely connected.
8 MR. HARRIS:
Ken Harris, Florida Power and Light.
9 Is there any statistical basis for the 5 percent on 80 percent, 10 or do you all have any experience with that?
I don't know what 11 the make-up of the exams are, but if you have a lot of essay 12 questions, that's difficult, from -- let me say -- my very 13 limited experience in grading these exams.
\\./
14 I worry how we're going to get around that if we have 15 a difference.
And the other two points on that graph -- we 16 need to define " major" and " gross".
We need some positive 17 definition there.
18 MR. ROES We agree with that.
19 MR. EATON:
We agree with that.
20 MR. HARRIS: How about the first question?
21 MR. EATON: We have already identified with the 22 technical advisor of the plant when the simulator scenarios are 23 made up that there would be certain areas and if they did this 24 as opposed to what would be the normally expected corrective I
25 action for the evolution, they would be identified as failures.
)
Heritage Reporting Corporation l
(202) 628-4888 l
l
104 h
1 MR. IIARRIS:
I think you're looking at core 2
degradation -- significant core damage -- on this type of 3
thing.
I don't think you can leave this to each utility to 4
define.
You are going to have definitions all over the place.
l 5
I think you need a more positive definition of " major" and l
l 6
" gross", and leave that at any individual facility.
l 7
MR. WIENS:
We recognize that, and we haven't arrived 8
there yet.
But we tend to agree.
9 MR. ROE:
John, correct me if I'm wrong, but I think 10 we used our judgement to establish those quantitative measures, 11 and we found that over our short experience of date, they have 12 served us relatively well.
13 MR. HANNON:
Yes, and it's our intent to have an 14 objective examination such that we wouldn't expect to have a 1
l 15 lot of dispute over what the right answer to any written 16 question is.
We are going to be getting a fair amount of 17 rigorous review both on the part of the facility and the NRC.
18 Yes?
19 MR. WILLIAMS:
Lee Williams, from Alabama Power.
I 20 am reading a handout I evidently was given by the people at 21 H.B.
Robinson.
For the technical advisor that will assist you, 22 as statement here says:
"The individual must not be involved 23 in any way in requalification program activities between the 24 time of the exam review and administration."
25 He still has to participate in our retraining O
Heritage Reporting Corporation (202) 628-4888
I 105
?(
1 requalification process.
I don't think that was the intent l
2 there, was it?
I' 3
MR. WIENS:
No.
I understand the integrity question.
4 MR. WILLIAMS:
Was it that he not be teaching any of 5
the court e.s, and that he is allowed to retain his own 6
requalification currency.
7 MR. STAROSTECKI:
I think the room's a little hot, 8
and it's not conducive to a lot more creative thinking.
If you 9
go out and hit the cold air and you come up with some more i
10 questions, let me encourage you to put them down in writing and 11 send them to John Hannon, who is the branch chief.
12 (Laughter) 13 MR. STAROSTECKI:
Questions that will come into the l
14 Agency will be directed to him.
If you want to hava an impact 15 on where we are going, I think the faster you get them to John, 16 the better it's going to be, and avoid the administrative 17 delays.
18 MR. ROE:
One other point is:
Let's be aware that 19 we've got programs going and accreditation with respect to l
20 requal by INPO, and we want to be sure that we communicate back l
21 and forth about anything that impacts on that particular 22 program, because the NRC does have a policy statement out on 23 it.
24 The other thing is that I would urge you all to work 25 carefully with INPO and NUMARC, because there is a lot of Heritage Reporting Corporation (202) 628-4888 1
L 1
106
-().
1 opportunity for you all to reinvent the wheel time after time 2
after time after somebody's already done it.
Minding together 3
here might be very, very useful to make some of these 4
accomplishments.
5 MR. STAROSTECKI:
Thank you for your time.
6 (Whereupon, the matter was concluded at 3:37 p.m.)
7 8
9 10 11 12 13
[^)
N>
14 15 16 17 18 19 20 21 22 23 24 25 1
Heritage Reporting Corporation i
(202) 628-4888
1 CERTIFICATE 2
l
\\
3 This is to certify that the attached proceedings before the 4
United States Nuclear Regulatory Commission in the matter of:
5 Name:
PUBLIC MEETING ON OPERATION REQUALIFICATION 6
7 Docket Number:
NRC-342.ll7 8
Place:
Bethesda, Maryland 9
Date:
November 23, 1987 10 were held as herein appears, and that this is the original 11 transcript thereof for the file of the United States Nuclear 12 Regulatory Commission taken stenographically by me and, 13 thereafter reduced to typewriting by me or under the direction 14 of the court reporting company, and that the transcript is a 15 true and accurate recor of the oregoing proceedings.
)
I f
AAW f
~
16
/S/
Iv l4 r 17 (Signature typed):
18 Official Reporter 19 Heritage Reporting Corporation 20 21 22 23 24 25
{
l
()
Heritage Reporting Corporation (202) 628-4888