ML19052A378

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML19052A378
Person / Time
Site:	FitzPatrick
Issue date:	02/21/2019
From:	Brian Fuller Operations Branch I
To:	Entergy Nuclear Operations
Shared Package
ML18163A275	List: ML19052A357 ML19052A378
References
CAC 0500
Download: ML19052A378 (7)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9 Q

1. LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /Units Backward Q - K/A SRO Only 1

F 3

Y B

S 2

F 2

Y N

S 3

F 3

Y N

S 4

H 3

Y N

S 5

F 2

Y B

S 6

F 2

Y B

E Change distractor C to improve LOD 7

H 3

Y N

S 8

H 3

Y N

S Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3.

Check the appropriate box if a psychometric flaw is identified:

Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

Cues: The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).

T/F: The answer choices are a collection of unrelated true/false statements.

Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.

Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).

4.

Check the appropriate box if a job content flaw is identified:

Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).

1. /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

Backward: The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only. (K/A and license-level mismatches are unacceptable.)

6.

Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.

7.

Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. / Units Backward Q - K/A SRO Only 9

H 3

Y B

S 10 H

3 Y

B S

11 F

2 Y

B E

Editorial 12 H

3 Y

N S

13 H

3 Y

B S

14 H

3 Y

B S

15 H

3 Y

M E

Stem does not give reason for torus lowering, add seismic event to stem 16 H

3 Y

N E

Need irrefutable evidence that level will not be restored. Change bullets 3 and 4 in stem to say no injection sources are available 17 F

2 Y

B S

18 F

2 Y

B E

Changed stem to improve LOD and the plausibility of distracters B and D 19 F

2 Y

B S

20 H

3 Y

M S

21 H

2 Y

B S

22 F

2 Y

B S

23 H

3 Y

M S

24 H

3 Y

N E

Removed values for max safes in stem

ES-401 3

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. / Units Backward Q - K/A SRO Only 25 H

2 Y

N S

26 H

3 Y

N S

27 H

3 Y

B S

28 F

2 Y

B S

29 H

3 Y

B S

30 F

3 Y

M S

31 H

2 Y

B S

32 H

3 Y

M S

33 H

3 Y

B S

34 H

3 Y

N E

Changed wording of distractors to be consistent throughout 35 H

2 Y

B S

36 H

2 Y

B E

Reworded Q(1) and Q(2).

Rearranged answers to improve plausibility 37 H

3 Y

M E

Editorial 38 H

3 Y

B S

39 H

2 Y

M S

40 H

3 Y

B S

41 F

2 Y

N E

Editorial

ES-401 4

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. / Units Backward Q - K/A SRO Only 42 F

2 Y

N S

43 H

3 Y

M S

44 F

2 Y

M S

45 H

3 Y

B S

46 F

3 Y

B S

47 F

2 Y

N S

48 F

2 Y

B S

49 H

2 Y

M S

50 F

3 Y

N S

51 H

3 Y

B S

52 H

2 Y

M S

53 H

2 Y

N E

Editorial 54 F

2 Y

B S

55 H

3 Y

B S

56 H

3 Y

N S

57 H

3 Y

M S

58 H

3 Y

B S

ES-401 5

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. / Units Backward Q - K/A SRO Only 59 F

3 Y

N S

60 H

3 Y

N S

61 F

2 Y

B S

62 H

2 Y

M S

63 H

3 Y

M S

64 H

2 Y

N S

65 F

2 Y

B S

66 F

2 Y

B S

67 F

3 Y

N S

68 F

2 Y

M S

69 H

3 Y

B S

70 F

3 Y

M S

71 F

2 Y

B S

72 F

2 Y

B S

73 F

2 Y

B S

74 H

2 Y

B S

75 F

2 Y

B S

ES-401 6

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. / Units Backward Q - K/A SRO Only 76 H

3 Y

Y B

S 77 F

2 Y

Y M

S 78 H

3 Y

Y B

S 79 H

3 Y

Y B

S 80 H

2 Y

Y B

S 81 H

3 Y

Y N

S 82 H

3 Y

Y B

S 83 H

3 Y

Y N

S 84 H

3 Y

Y B

S 85 H

3 Y

Y B

S 86 H

3 Y

Y N

S 87 H

3 Y

Y B

S 88 H

3 Y

Y N

S 89 H

2 Y

Y N

S 90 H

3 Y

Y N

S 91 H

3 Y

Y N

S 92 H

2 Y

Y M

S

ES-401 7

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. / Units Backward Q - K/A SRO Only 93 F

2 Y

Y B

S 94 F

2 Y

Y B

S 95 H

3 Y

Y N

S 96 H

3 Y

Y B

S 97 H

3 Y

Y B

S 98 H

3 Y

Y B

S 99 H

2 Y

Y B

S 100 F

2 Y

Y B

S