ML20260H163

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML20260H163
Person / Time
Site:	Peach Bottom
Issue date:	06/30/2020
From:	Exelon Generation Co
To:	Todd Fish Operations Branch I
Fish T
Shared Package
ML19105A151	List: ML20184A247 ML20184A248 ML20260H163 ML20260H164 ML20260H165 ML20260H166 ML20260H167 ML21056A220 ML21056A221
References
CAC 000500
Download: ML20260H163 (6)

Text

ES401 Written Examination ReviewWorksheet (PBAPS)

FormES-401-9 1.

2.

3.PsychometricFlaws 4.JobContent Flaws 5.Other 6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues T/F Cred.Partial Job-Minutia#/ Back-Q= SROB/M/NU/E/S Explanation Focus Dist.

Link unitsward K/A Only 1

H 3

Y B

S 2

F 3.5 Y

N S

3 F

2.7 Y

B S

4 F

3 Y

B S

5 H

2.7 Y

B S

6 H

3.3 Y

N S

7 H

3.0 Y

M S

8 F

3.5 Y

N S

Instructions

[Refer toSection DofES-401 andAppendix Bfor additional information regarding eachofthefollowing concepts.)

1.

Enterthelevel ofknowledge (LOK) ofeachquestion aseither(F)undamental or(H)igher cognitive level.

2.

Enter thelevel ofdifficulty (LOD) ofeachquestion using a1

- 5(easy

- difficult) rating scale (questions inthe2 - 4range areacceptable).

3.

Checktheappropriate boxifapsychometric flawisidentified:

Thestemlacks sufficient focus toelicit thecorrect answer (e.g.,unclear intent,moreinformation

isneeded, ortoomuchneedless information).

Thestemordistractors contain cues(i.e.,

clues, specific determiners,

phrasing, length, etc).

Theanswer choices areacollection ofunrelated true/false statements.

Thedistractors arenotcredible; single implausible distractors should berepaired, morethanoneisunacceptable.

Oneormoredistractors is(are) partially correct (e.g.,

iftheapplicant canmake unstated assumptionsthat arenotcontradicted bystem).

4.

Checktheappropriate boxifajobcontent error isidentified:

Thequestion isnotlinked tothejobrequirements (i.e.,

thequestion hasavalid K/A but, aswritten,isnotoperational incontent).

Thequestion requires therecall ofknowledge that istoospecific fortheclosed reference testmode(i.e.,

itisnotrequired tobeknownfrommemory).

Thequestion contains datawithanunrealistic level ofaccuracy orinconsistent units (e.g., panel meterinpercent withquestion ingallons).

Thequestion requires reverse logic orapplication compared tothejobrequirements.

5.

W forconformance withtheapproved K/Aandthosethataredesignated SRO-only (K/A andlicense level mismatches areunacceptable).

6.

Enterquestion source:

(B)ank, (M)odified, or(N)ew.

Checkthat(M)odified questions meetcriteria ofES-401 Section D.2.f.

7.

Basedonthereviewer's

judgment, isthequestion aswritten (U)nsatisfactory (requiring repair orreplacement), in need of(E)ditorialenhancement, or(S)atisfactory?

8.

Ataminimum, explain any"U"ratings (e.g.

howtheAppendix Bpsychometric attributes arenotbeingmet).

ES401 2

FormES-401-9 1.

2.

3.PsychometricFlaws 4.JobContent Flaws 5.Other 6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

Stem CuesT/F Cred.Partial Job-Minutia

1. / Back-Q= SROB/M/NU/E/S Explanation Focus Dist.

Link unitsward K/AOnly 9

F 3

Y B

S 10 F

3.5 Y

N S

11 F

3 Y

B S

12 H

4 Y

B S

13 H

3.3 Y

N S

14 H

4 Y

N S

15 H

2.5 Y

N S

16 H

3.5 Y

N S

17 H

3.3 Y

B S

18 H

3.3 Y

B S

19

. F 3.0 Y

B S

20 H

3.0 Y

B S

21 H

2.8 Y

B S

22 F

2.8 Y

N S

23 H

2.8 Y

B S

24 F

3.5 Y

B S

25 H

3.5 Y

N S

26 F

2.8 Y

B S

27 H

2.8 Y

N S

28 F

2.7 Y

N S

29 H

3.0 Y

B S

1.

2.

3.Psychometric Flaws 4.JobContent Flaws 5.Other 6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

StemCuesT/F Cred.Partial Job-Minutia

1. / Back-Q= SROB/M/NU/E/S Explanation Focus Dist.

Link unitsward K/A Only 30 F

1 X

X Y

N U

LOD=1;Nooperational significance.

Selected newK/Aandwrote newquestion.

31 F

3.7 Y

N S

32 F

3.3 Y

B S

33 F

3.0 Y

N S

34 H

3.8 Y

N S

35 H

4.0 Y

N S

36 H

2.5 Y

M S

37 H

3.3 Y

N S

38 F

4.0 Y

N S

39 F

3.0 Y

N S

40 F

3.0 Y

B S

41 H

3.0 Y

B S

42 F

4.0 Y

N S

43 H

3.5 Y

N S

44 F

3.0 Y

M S

45 H

2.5 Y

N S

46 F

3.3 Y

N S

47 F

3.3 Y

N S

48 F

3.0 Y

M S

49 H

3.3 Y

N S

50 F

2.8 Y

N S

51 F

1 Y

N U

LOD=1;B,C,Dall are

switches, question asksfortemp.jndicatjons.

Selected newK/Aand wrote anewquestion.

52 H

3.5 Y

N S

1.

2.

3.Psychometric Flaws 4.JobContent Flaws 5.Other 6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

StemCuesT/F Cred.Partial Job-Minutia#/ Back-Q= SROB/M/NU/E/S Explanation Focus Dist.

Link unitsward K/A Only 53 H

3.5 N

N U

K/Am/mbecause "cause" phrase ofK/Anotaddressed.

Revised the question.

54 H

3.0 Y

B S

55 H

2.8 Y

M S

56 H

3.0 Y

N S

57 H

3.0 Y

N S

58 H

3.0 Y

N S

59 F

4.0 Y

B S

60 H

3.3 Y

M S

61 H

3.3 Y

N S

62 F

2.3 Y

M S

63 F

3.0 Y

N S

64 H

3.0 Y

N S

65 H

3.0 Y

N S

66 M

3.0 Y

N S

67 F

3.3 Y

B S

68 F

3.0 Y

B S

69 H

3.7 Y

N U

Question toosystem-specific forTier3.Wrotenewquestion.

70 F

3.0 Y

N S

71 F

3.5 Y

N S

72 F

3.0 Y

B S

73 F

3.0 Y

B S

74 H

3.0 Y

M S

75 H

3.8 N

N U

K/Am/mbecause question doesnotaddressm asp.ect oftheK/A.

1.

2.

3.Psychometric Flaws 4.JobContent Flaws 5.Other 6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

StemCuesT/F Cred.Partial Job-Minutia

1. / Back-Q= SROB/M/NU/E/S Explanation Focus Dist.

Link unitsward K/A Only 76 H

3.0 Y

Y B

S 77 H

2.7 Y

Y B

S 78 H

3.0 Y

Y B

S 79 H

3.0 Y

Y B

S 80 H

3.3 Y

Y B

S 81 H

3.0 Y

Y B

S 82 H

3.0 X

Y Y

N U

C&Dimplausible:

Operators wouldnever doeither action.

Revised question.

83 H

3.3 Y

Y B

S 84 H

3.3 Y

Y N

S 85 H

2.0 Y

Y B

S 86 H

3.0 Y

Y B

S 87 H

3.0 Y

Y M

S 88 F

4.0 Y

Y N

S 89 H

3.5 Y

Y N

S 90 H

2.3 Y

Y N

S 91 H

4.0 Y

Y N

S 92 H

4.0 Y

Y B

S 93 H

2.7 Y

Y N

S 94 F

2.3 Y

Y B

S 95 H

2.7 Y

Y N

S 96 H

3.7 Y

Y N

S 97 H

2.5 Y

Y M

S 98 H

3.0 Y

Y B

S

1.

2.

3.Psychometric Flaws 4.JobContent Flaws 5.Other 6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

StemCuesT/F Cred.Partial Job-Minutia#/ Back-Q= SROB/M/NU/E/S Explanation Focus Dist.

Link unitsward K/A Only 99 H

4.0 Y

Y N

S 100 H

4.0 Y

Y M

S

.