ML20245J429

Requalification Program Audit Rept 50-445/OL-89-01 on 890703.Audit Results:All Individuals & Crews Passed All Exams

ML20245J429
Person / Time
Site:	Comanche Peak
Issue date:	08/09/1989
From:	Pellet J, Whittemore J NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV)
To:
Shared Package
ML20245J424	List: ML20245J419 ML20245J429
References
50-445-OL-89-01, 50-445-OL-89-1, NUDOCS 8908180028
Download: ML20245J429 (8)

Text

y m7

_ x.

i i> glp J.

s

+

.J-i

,g.;

Q..

, b. g, m

3 h! (

• A

i;

• (

l1

g 1

1.

o f, 4

' APPENDIX

,o,

b.I' 4h.,.*

1 B g$ ~

(U;S.NOCLEARREGULATORYCOMMISSION' g

REGION-IV.

Lh p

NRCOperator'Ocensing'ExaminationReport:.50-445/0L;89-01;

,c y,

S

'e : Construction Permit:~ CPPR-126,

"n a_.-

m 7": Docketb 50-445.-

s e

f ( lLicenseei:

TU Electric' (TV):.

400 North Olive, L.B.'81

Dallas,'TX' 75201<

'a y u

Facility Name',

Comanche l Peak Steam Electric-Station (CPSES).

s

... ~

N, Examination:at:

Comanche Peak Steam Electric Station Glen Roser Texas 3-t w

.1 s

Examination conducted:' Week 'of. July 3,1989 x

, p

[

Chief Exeminer:

9 29

~

w

(/ E. Whittemore. Examiner Date

-o V

t

Orperator Licensing Section -

Division.of. Reactor Safety

+

~

a f

Pi iApproved:'

o 54 u

J. '._. ; Pellet,

hief Date '

X

-Operator Licensing Section-Division Reactor Safety 1

-Inspection Summary NRC-Administered Examinations Conducted During the Week of July 3, 1989

+

(Repor t 50-445/0L 89-01)

'NRCiadministered written, walk 2 hrough, and simulator requalification examinations t

a

to.two unit crews consisting of two reactor operators plus tio senior reactor operators each and a-staff crew consisting of four senior reactor operators.

,9 JAll' individuals and crews passed all of the examinations.

[....

i' 8908180028 090009 PDR V

ADOCK 05000445 PNV i

as mm__m.m.__._ax_

.r y

c.

x DETAILS-

1. -

Persons Examined CREW SRO Fg TOTAL Requalification Examinations: Pass-3 8-4 12 Fail-0 0

0 0

"2..

Examiners J. E. Whittemore, Chief Examiner D. N. Graves K. M.-Kennedy J. L.' Pellet F. W.. Victor 3.-

Examination Report Performance results for individual examinees are not included in this report since it will be placed in the NRC Public Document Room and these results are not subject to public disclosure.

.a, Examination Material Development The licensee initially submitted developed material late in 1988.

NRC revierters.found the following problems with new written and walk-through examination items:

(1) Th5 facility licensee staff assigned'to test preparation did not understand which test items were appropriate for Sections A,

" Plant Operations and B. " Limits and Controls" of the written examination. Several questions designated for Section A required the examinee to use the Emergency Plan Implementing Procedures or to interpret specific Technical Specification LCO's and determine operability. The focus of Section A should be on plant systems and should not require the use of emergency preparedness ~. procedures.

Further, it is acceptable to require the operator to recognize Technical Specification LCO's, but questions requiring interpretation or operability determination should be reserved for Section B of the written examination.

(2) Several written items placed examinees in " double jeopardy.

Most often these items required a "yes/no" or "true/ false" answer and then required an explanation to support the initial answer with partial credit being allowed for each correct partial answer..The disadvantage is that the examinee ccn 1

rarely receive partial credit by correctly supporting a wrong initial answer. The licensee initially proposed to require the entire correct answer for full credit. When the Chief Examiner l

L L

L:

m_ __1__

%g l %;v _

,s

- - - -- m w,

u

^

um wx V

,e

-34 g

f,

,A-

.c-

/

p

.c

[Y

. pointed out that th'is solution was also unfair to the examinee, 7'

'most 6f the; items were left as-they were initially. ;It is preferable not to use questions that can be answered "yes/no" or "true/ false.'

s,

.(3).Many ~questiorisicould be answered by simple recallfof knowledge.

s Comprehension or synthesis level knowledge was not required to E,

correctly answer-the question, or it was merely necessary to

'4

'look' up the correct answer, and no decisionmaking responsibility was required of the examinee.

(4) _ The majority of.the Job Performance Measures (JPM)Iquestions-were look up or recall ~,.and required no analysis of synthesis on the part of the examinee to answer'the question. Frequently,

^

the, answers to the questions were contained in-the procedure

~

that was used to complete the task. Questions used with a JPM are subject to,the same. content and construction requirements as those submitted for written items.

j 1

'(5) The. performance-standards for JPM steps were often inadequate because of > an overt.se of the phrase "Same as Element." The element often contained detailed information that should have been'in the' performance standard for a particular step.

For future development, the licensee should consider performance elements that' state what the step is, and the performance standard should provide the evaluator with a detailed description of how the step may be successfully accomplished.

In April"1989, the Chief Examiner spent a day at the CPSES facility, The above results 'of the material review were discussed with.the a

licensee' Also at this time logistics of examination administration were' addressed, and several problems were resolved.

In addition, the licensee committed to have' completed examination material to the'NRC 60 days beforecthe scheduled examination.'

~ The entire sample pool of the finalized material was received in the NRC Region-IV office more than 60 days prior to the, examination week.

.During review of the additional material, it was found that problems noted in the initial review phase still existed cnd only marginal improvement had been made. NRC did not discuss these problems with facility personnel until the examination preparation week when the examination was assembled. Additional problems with material were j

discussed with the licensee in early June. The specific problems i

uncovered at this time'and noted to the licensee were'the following:

The dynamic simulator scenarios contained steps that were

' considered Individua? U mulator Critical Tasks (ISCT) but

,i these were not so idencified in the operator evaluation criteria. One example of this v:as that plant conditions L

i l

1 i

_._______m___

7:

7 W

u 4

required a manual scram to prevent a challenge to plant safety, but the scenario contained no evaluation criteria as required

'by ES-601.

It was noted that often there were questions slated for a particular written Section A static simulator that could not be used together on the same examination. This' occurred because the text of one question would give away the answer to another question. These questions that were mutually exclusive and could have negatively impacted examination validity if used together, were not identified by the facility licensee..

b.

Examination Development During the week of June 19, 1989, the Chief Examiner traveled to the CPSES site. The Chief Examiner and one CPSES contractor, who had signed an ES-601 security agreement form, assembled Sections A and B written examinations considered to be satisfactory by both parties.

These examinations were time validated by having the designated facility representatives take the examinations. These examinations were graded by NRC and both designated licensee representatives passed the written examinations.. At this time, the representatives were informed that these examinations could be counted as their annual written recualification examinations if the CPSES prgram permitted this. Also at this time, security agreements t:en signed by the designated facility representatives. After some modification of the examinations, the facility representatives and the Chief Examiner agreed on the final written examination versions.

After finalizing the written examinations, JPMs were selected and assembled into walk-through examinations. This task required considerable effort since examinees could not be examined on JPMs to 4

which they had already been exposed either for validation or practice. The team was able to assemble examinations meeting this requirement because the licensee had developed and instituted a tracking system which indicated the individuals that had been exposed to specific JPMs. More problems were discovered when the selected 1

JPMs were subjected to a final extensive review before administering examinations. The most significant problem involved the JPM followup questions. Many of these questions were deleted or rewritten.

Another area that had to be corrected was the misidentification of

{

critical steps within the JPMs. The majority of the action required was redesignating previously designated " Critical" steps as "Non-cri tical." A third area requiring correction was to improve i

individual step' performance standards.

Simulator scenarios were selected and agreed to by the facility representatives and NRC. There were no major improvements or corrections required after resolving the critical item evaluation issue noted previously.

In attempting to evaluate the facility l

gG t

j

'i e

m

-5 licensee's relative strengths.in' material development for the three examination areas, NRC considers simulator scenario development to' be the strongest.

When examination _ assembly was complete, the Chief Examiner met with R

-the designated facility evaluators at the request of the licensee.

During this meeting, NRC answered questions about administering walk-through examinations.

In' response to a specific question, the evaluators were told that it was always acceptable to stop an examination and question or consult with the NRC examiner. The four facility evaluators and a designated administrative assistant signed pre-examination security agreements.

c.

Examination Administration (1) On July 3, 1989, written examination Section B was administered

.to two groups of six operators separately in back-to-back sessions. Simultaneously with the first written examination session, walk-through examinations consisting of the five common JPMs were conducted with four of the six operators scheduled to take the second session of the written examination. Precautions..

were taken so that individuals not involved in the written or -

walk-through examinations were separated and monitored to preclude any compromise. Twelve individuals were administered written and walk-through examinations in this manner. At the conclusion of the day's examinations, the NRC examiners debriefed individual facility evaluators to obtain immediate results and findings of the three partial walk-through examinations conducted.

The facility licensee demonstrated a lack of attention to detail because the written examinations failed to provide alternate format or rearrangement of the order of questions within the individual written examinations. This action had been previously requested by NRC to facilitate examinees use of common material with minimum risk of compromise.

(2) On July 5. 1989, written examination Section A was administered to two groups of six operators separately in back-to-back sessions in the CPSES simulator. Simultaneously with the first written examination session, walk-through examinations consisting of the five uncommon JPMs were conducted with four of the six operators scheduled to take the second session of the written examination. Precautions were taken so that individuals not involved in the written or walk-through examination were separated and monitored to preclude any chance of compromise.

Twelve individuals were administered written and walk-through examinations in this manner. At the conclusion of the day's examinations, the NRC en:ainec debriefed individual facility evaluators to obtain immediate results and findings of the three walk-through examinations completed that day.

h

.]

T i

y

+ The facility licensee demonstrated a lack of attention to detail-by failing to page check the Section A examinations before they were-administered. Consequently, one of the questions that should have been included in the examination was missing. This error was not discovered until the examination was actually being administered to the sbtond group. Fortunately, the included questions were sufficient to. serve as a valid examination. A further' lack of attention to detail was demonstrated by the licensee's failure to provide alternate

. orders for the.. individual examinations as discussed in c.(1) above'.

' {3) The morning of July 6, 1989, was set aside for the facility licensee and the NRC to grade written examinations and to finalize grading of the walk'-through examinations. Subsequent comparison of grading results between the two parties revealed that there was 100% agreement and that all examinees had passed all phases of the examination to this point. During the afternoon, the dynamic simulator examination was administered to one of the three crews using a facility prepared scenario. Again, the facility licensee and NRC agreed that all examinees and the crew passed this phase of the examination.

(4) On July 8, 1989, the two remaining crews were evaluated using a common set of scenarios, but different from the set used on the previous day. The two crews were kept separate and monitored to preclude any risk of compromising the scenarios. Again, tiie facility licensee and the NRC were in total agreement that all individuals ano both crews had passed the examinations.

d.

Examination Results All examinees passed all phases of the examination as evaluated by the facility licensee and the NRC. The written examination results indicated (1) lack of knowledge of the excess letdown system and its operation; and (2) deficiencies in diagnostic ability for RCS instrument failure.

A total of six examinees failed seven individual JPMs (one person failedtwo). JPM 9012, entitled " Responding to Anticipated Transient Without Trip," was failed by four examinees. The licensee is encouraged to determine whether the root cause of this was a performance problem, a problem with the evaluation tool, or both and to correct it. Among the examinees, a total of 227 questions out of a possible 240 JPN followup questions were answered correctly. One individual answered 85% of the followup questions. All others performed better.

There were no critical performance errors observed during administration of the simulator examinations. There were obvious

l($%5N(lp

.. y O

pg M ' ;

- p V,,

i "J

f_

,' w l, t. h....X i.R i 3

^ 4 b

%ff &p' '

.'4

.-7 9

+.

Er g j

ggWy, 3 y

'7 y

y

.h.

l Y

4 comunidation weaknesses noted in the performance of: one crew.s The-

,Qf g

-remaining two~ crews ~ demonstrated this problem to-a lesser degree.

~

'z.

,s,

. ; Site Visit ~Sumary -

y g.y.q e

fThe'NRCmetwith'thefacilitylicenseetrainingstafftoprovidethem l preliminary results and findings. At this time, all significant

-weaknesses were noted.. TheseL weaknesses were mostly' development, not

^

examinee performance oriented. :Throughout the process, specific

-problems relating'to material had been relayed to the facility evaluators and! designated technical representatives.

JAn exit meeting wasLheld at the CPSES site at the conclusion of.the

' examinations. The following. personnel were present:

~NRC Tl[

CASE'

.S. Burris.

.G. Bell E. Ottney J. Pellet' H. Bruner M. Thero R.:Warnick-J. Donahue J.'Whittemore J. Kelley, Jr.

.c

.J. Wiebe J. McMahon J. Walker A. Scott It was noted that the facility representatives had proved to be. adept at solving complex logistic problems that had to be overcome to preserve examination integrity.

Further,' facility representatives had demonstrated technical expertise and a willingness to work long V

_ hours; these are considered to be major factors in the successful' outcome of the examination process.

It was al'so noted that the performance of the shift Reactor Operator assigned to. Unit I during the walk-through examinations was very-helpful.

The following items were. communicated to the licensee representatives as :coments, observations, suggestions, or deficiencies:

. m(1)' All 12 operators and all three crews ' passed the examinations. :

~

Based on this and in spite of the programmatic weaknesses

. encountered, the CPSES licensed operator requalification program is' evaluated to be satisfactory.

w-

?(2) The facility evaluators performed well; they were considered to V

W be competent and thorough.

!m

,j L

Ii

-(3)"Therewereminimumofexamineeperformanceproblems.- Nearly all weaknesses were in the material development area.

J d

p A

ya i

s

.i

]

v

~-

, p ';a y -

f 1 7j jL;

[f? pi r,

y lTO,. :

?4 Q f n'

e g,y;e. g,

4 e

y a

,.L.

• {3 7 a

g.-

c a

v (i @

(4).The, facility demonstrated resistance to NRC comments.and.

l

.' suggestions' for improving the initial-material submitted. This 1was. difficult to understand since most' comments were made to

~

Y[F[,

Limprove examination fairness and examinee's ability to pass the examination.

r

-R

. (5)~ More attention to detail is;needed to preclude the problems 4

ca encountered during the' preparation and administration.~of the 4

written examinations,:1.e., failure to alter examination format i

-or to" include all' questions that were intended for the examination.-

~

13 W

.(6)1 The NRC encourages the facility licensee to use the simulator s

for adinini.stering walk-through examinations whenever possible s"

y for the purpose ~ oflimproving examination validity.

' (7h Simulator scenario development!'is considered to be the strongest area;of~the facility's ability to develop valid. examination

' material.

. (8) During simulator examinations, crew members consistently failed to assure that communications were received or acknowledged.

(9){ Comand and control responsibilities were often unclear to the

~

-- point.where. it was difficult to determine which crew member was in charge.

'(10) Crews' consistently overfed steam generators when responding to

' transients using the emergency procedures.

t (11)' Crews did not effectively judge or estimate how long a time was needed in order to see the reactivity. effects of normal boration.

)

L4:'

Y.

1 b_

u_

-