ML030410321

Written Examination Review Worksheet, Es - 401-9 with NRC Comments for the Monticello Initial Examination - Oct. 2002

ML030410321
Person / Time
Site:	Monticello
Issue date:	10/14/2002
From:	Mcneil D NRC/RGN-III/DRS/OLB
To:
References
50-263/02-301, ES-401, ES-401-9 50-263/02-301
Download: ML030410321 (8)

Text

WRITTEN EXAMINATION REVIEW WORKSHEET, ES-401-9 WITH NRC COMMENTS FOR THE MONTICELLO INITIAL EXAMINATION - OCT 2002

ES-401 Written Examination Form ES-401-9 Review Worksheet The first 30 questions were used for the sample. All 125 questions were reviewed for accuracy and K/A match.

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD I

-7 (F/H)

(1-5)

Stem Cues T/F Cred. 'Partial Job-Minutia

1. /

Back-Q= SRO U/E/S Explanation FocusI Dist.

Link units ward K/A Only II 1

F 2

x S

2 H

2.5 x

U K/A asks ability to monitor, Q asks what will cause a rod block.

3 H

2.5 x

U Applicants will perform this operation during the operating test (overlap).

4 F

2.0 S

5 H

2.5 x

U Not SRO ONLY. ROs required to know this material 7

H 2.5 E

Put stem in past tense, put distractors in columns to highlight 2-parts 8

H 2.0 x

U effecting should be affecting - K/A ask for recirc effects, Q asks who?

9 (F)H 2.5 S

10 H

2.5 E

Reformat distractors 11 H

2.5 E

annuciator is annunciator Instructions

• Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question using a 1 -5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6.

Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7.

At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOIK LOD (F/H)

(1-5)

Stem Cues T/F ICred. IPartial Job-MinutiaI #/

Back-Q=

SRO U/E/S Explanation Focus j

[Dist.

Link junits ward K/A Only 12 H

2.5 S

13 F

2.0 S

14 H

2.0 S

15 (F)H 2.0 U

Requires some analysis. With info given, can't rule out distractor b. how I (was this taught?

16 H

2.5 E

inseration is spelled insertion 17 (H)F 2.0 U

If 18 is fundamental, so is this one. Put plant conditions in past tense.

ask how taught.

18 F

2.0 S

19 H

3.0 x

S 20 F

2.0 S

21 (H)F 2.5 U

Requires only memory of power supplies to the APRMs.

22 (F)H 2.5 U

Requires some analysis of given conditions to arrive at a conclusion.

23 (H)F 2.5 x

U Requires only memory of basis and when a T.S. is applicable.

24 (F)H 3.0 U

Requires some analysis of given conditions to arrive at a conclusion.

25 (F)H 2.5 U

Requires some analysis of given conditions to arrive at a conclusion.

26 (F)H 2.0

?

U Requires some analysis, do you require ROs to make operability calls?

27 (F)H 2.0 U

Requires some analysis of given conditions to arrive at a conclusion.

28 H

3.5 S

29 (F)H 2.5 U

Requires some analysis of given conditions to arrive at a conclusion.

30 F

2.0 S

ES-401 3

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD

-M7ntau/I~c-Q R

//

(F/H) (1-5)

Stem Cues[ TF red. Partial Job-iMinutia ui I~acka

/]

o u/Es Explanation Focus D

ist. Link units ward K/A Ony 31T HI

]

E Put "The Containment Spray Valves will" in the stem. in distractor C.,

F make Valve plural like the other distractors (Valves).

Z[.1*1 I I IZII II I lIi Is I_______

1___ 2.5

?

I II UI I

] 1 I 1;

u Requires some analysis of given conditions to arrive at a conclusion.

I 31 1+/- 01 a

I I_ I JI I

__I ] I I__

I

-]

Is I_______________________

F13 I 2.0 1 I71 I II I I _I I

[7 I....1.0 I

I I

I 0

I II IU Requires some analysis of given conditions to arrive at a conclusion.

[9ITF)lI I

I 1

I 1 III I

I I I

U fRequires some analysis of given conditions to arrive at aconclusion.

2.40 I[J Is S

41 " *1 HI I I E Put in column format.

42 1 1

• H 2.

1 IJ I I I E IPut in column format.

43 1_

1__

1 I I

I 2.5 I 1 I I 1 I I I

E IPut in column format 44 H1.0 1 x

42 1 211 II I I I I2 E j Put in column format Requires an RO to make an operability determination. Describe the H2.0 U

monthly test somewhat Requires some analysis of given conditions to arrive at a conclusion.

48 (H)F I[2.0 UII I1I I

I 1I1I I

[

U Requires recall of memory items, no analysis.

49 H T 1 s I

ES-401 4

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOKILOD

-T t

I (F/H) (1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q= SRO U/E/S Explanation

[-Focu

2.

I DT, I ]_ L ILLII I

Focus

]

Dist.

Link units ward K/A Only 50i712.01_ I II I

2.

1__I WI I I sI l

52

__o II II

_I I

=I LL J I

IPut in column format.

I11=-0 1 11hZ II I

.II 7

L EI Put in column format F54, 1I ol

- II I

I I2. E [Put in column format; put stem in past tense.

55.o l. I. I IZ I I I III I

E Re-word the stem to reduce confusion.

56 F

2.0 I

I E

Put in column format 57 FI 2.0 _ 1 I

I III_

__IIsI 1

x E

Remove 'Postulated Scenario, quiet" answer doesn't fully answer

-x E

2.5 question, reword d. to answer question, not make statement.

59ITH 2.0 1 Put 11 1

_ i IIITI e

incolumn format. Do you really have 300 cps with the head o11.o HIZIZII IZIZII IZIII I IsJ________

61t F 1.1I I.I 1I I I I I I I I u U K/A is generic valve lineup. Q is specific to the RClC system.

_1__f l

I I

I I

I f I

I I

I I U IIf tech specs are provided, directlook-up. Ifnotechspecs, not required I J H

2jJ_

[U I to be memorized.

F164 IF 2.0 1

I 1 I

III Isl Wzil.* a I I I1 1 I IL. I I I IV _ ___sl_

[66FI1 2.0 1 [I11[

I

[

Ixls I]

F 16 I

I I1II I

I_

I I

ksm t

168 1F 12.5 I

I u

IxROs would be required toknowthismaterial frommemory.

ES-401 6

Form ES-401-9

1. ] 2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

(F/H) (1-5)

Stem Cues I T/F Cred. Partial Job-Minutia

1. /

Back-a= SRO U/E/S Explanation

__8 I

2 Focus Dist.

Link units ward K/A Only I 8 1 12.5 1 I.

j I

L.I

_ Ij I

x Ix E

Put in column format

[91J 1

]HF 2.0 I

I I

I 11111 U I

Answer is memorized information 2T(lFF2I I III III1[ I Material is not required to be memorized (steps from a procedure). Put

__U

[incolumn format.

iI Il.lIZ]ZI1....IIII I__

l.I.[I I I o Answer is memorized information 94 1 1F -1 2.5 E

I I I I IL I -

I I_

Put in column format I2-.Io 1.13.0 I IEI I II Ix l Put in column format 96 I(lIl 2.o0 1 I

I I I7 7 I IxI jx E IPut in column format 96.l1 IIF I I II

.I0 IIIsl

_71.*

ZIIfIzI__LI J1[V71______s___I

[98 1,,

] (F 2.0 x

I II Answer is memorized information and not SRO ONLY (system info, not x_

U tech spec LCO Basis) Put in column format L99 1i

1. o l I__

II I

I 20I

[I I

I s

L ~ I~o.

1 1. l I

___ LL J......I __ ]i...j __ I I.....L..

I I x l I 1*101 1 I

1 11 I

I III 1

I5 S

.o1*102I

.ol I [I..I..

I I.IZ II I....

..... ZIIls J________________________

[Lo

.0 I[If____ 1][

I T

E]Put in column format 1.04 II I.I.I I

I I F

I 2.0_s

-0*

10II5I I I

([

I 1 1 1 I0 Analysis required to arrive at conclusion

ES-401 7

Form ES-401-9 1.1

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD 1

T1 t~

(F/H) (1-5)

Stem Cues T/F ICred. Partial Job-IMoinutia

1. / Back-Q= SRo U/E/S Explanation Focus Dist.

Link units ward K/A On2yJ o61.

I I

_I 1 2.5

[

11111 U [Analysis required to arrive at conclusion F°7THII*5I IT IIIJ I

' JI I ] U Not SRO ONLY, ROs are responsible for EOP Basis. Put in column I

o108 II I

I I

I _

I FI I

Is. __

o1i o l I -Iii

-I i I2.0 1

1 1] Ix u Not SRO ONLY, ROs are responsible for EOP Basis.

M1OF 12.0 1111111 I

I I

Ix]

E iPlant conditions are unnecessary.

L,Ii1]. o l

2.

I1 IL I.I

_ 11 1 I__ I... I..... I I s____________________I_____

L,1

• 1 oIII__ 11 1 I I_

11 1 I__

1 i-L.L....]_ I

_________s l______________

F1131F[ 2.0 [

I I I I I.I.I I I u

Put in column format. Tests group5 info-See Q #29.

1141/2/[I3.

1 IIIII II 3.Ix x

U Answer is memorized information J

"[115 lF[ 2.0 II

[111111 I

I U

Analysis required to arrive at conclusion 11611o.1 I IIZIJII F

1 I21 S I1 1171Fl I I I III I

7 Il I

x E JPut in column format 11Z8.I1II I

I I

HI1_2.5_1_S "119 1F[ 2.0 x

I.

I I TIx I

I TI I

_U fROs not required to be knowledgeable of LCO time limitations.

2o12 1[30 [(I

[

III_

I 3.0 1I1 1

121 1H 2.5 x

I U

I I I

x U

NotSRO ONLY, ROs are responsible for EOP Basis.

,.1.

5121.1 I I I I.

1 1 [

I Is

[ 1221H 2.01 [

[

[

1 I _

I S

I IPl 1

=

l l=o 123 1 FI 2. 1 1

1 1E IPut incolumn format

ES-401 8

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD

[

[

(F/H)

(1-5)

Stem Cues T/F ICred. Partial Job-Minutia

1. / Back-Q=

SRO ULEIs Explanation Focus Dist.

Link units ward K/A Only]

,=,1Ii.0*

I-IZ

~I -~ =I 1.II ZI I I I Is l __________________________

,2SF ocus uII Inits I

I I s F1263LII7I2.0 I

_J I¶IILIIZI I

_sl Ii71WT.L I[IILI ZIZ]ZZI IIUA _________

IL 2L.II_

[IZ ITIII LIIIII Ir Is oo x

N

-RO ONLY, ROs are required to know EOP Basis. Put in column

=8 1 I o l __ IZ [ I

_ I I Z I ZI I I i I x l s I ______