ML23307A060

FY24 NMP2 Form 2.3-5 Written Examination Review Worksheet

ML23307A060
Person / Time
Site:	Nine Mile Point
Issue date:	11/03/2023
From:	Brian Fuller NRC/RGN-I/DORS/OB
To:	Constellation Energy Company
References
EPID L-2024-OLL-0003
Download: ML23307A060 (1)

Text

Form 2.3-5 Written Examination Review Worksheet

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 1

F N

S 2

F B

S 3

H B

S 4

H N

S 5

H B

S 6

H N

S 7

H B

E Added break between small and loss 8

H N

S 9

F B

S 10 H

N E

Added explanation in stem explaining why moving irradiated fuel 11 F

B S

12 H

N S

13 H

N E

Corrected Typo 14 F

M S

15 F

N S

16 H

N S

17 H

B S

18 F

B S

19 F

B S

20 F

B S

21 H

N S

22 F

B S

23 F

M S

24 F

2 B

E Reformatted Q to increase LOD 25 F

B S

26 H

M S

27 H

M S

28 H

M S

29 H

B S

30 H

M S

31 F

N S

32 H

B S

33 H

2 N

E Reworded Q to increase LOD 34 H

N S

35 H

B S

36 F

1 2 M B U S Modified Q was Low LOD, used the parent Q in its place 37 F

B S

38 H

B S

39 H

N S

40 F

X M

U S Distractors C & D are implausible for the stem conditions. Relaced system for (3) 41 H

N S

42 H

N S

43 H

N S

44 H

B S

45 H

M S

46 H

2 N

S 47 F

B S

48 F

N E

Changed Distracter A to a different system, reordered the answer choices 49 F

N S

50 H

N S

51 H

N S

52 H

N S

53 F

2 X

N E

Distractor D implausible Replaced D 54 F

B S

55 H

B S

56 H

2 N

S 57 F

B S

58 F

1 2 N

U S Low LOD Changed answers to specific power supplies

59 F

M E

Added time values to bulleted information 60 H

B S

61 H

X N

U S Minutia Resampled generic K/A 62 H

B S

63 H

N S

64 F

B S

65 F

1 B

U S Low LOD Wrote new Q 66 F

X X

N U S Distractor A is implausible. Third bullet in stem is ambiguous.

Rewrote stem and distractor A 67 F

B S

68 H

X M

U S K/A MM Rewrote Q to make radiological safety 69 H

N S

70 H

B S

71 F

1 M B U S Low LOD Used the parent QUSE THE PARENT Q 72 H

B E

Enhanced titles on graph axes 73 H

N S

74 H

N S

75 H

N S

76 F

SRO=Y N

S 77 H

SRO=Y B

E Added SRO Justification statement 78 H

SRO=Y M

S

79 H

SRO=Y N

S 80 H

SRO=Y B

S 81 H

SRO=Y N

S 82 H

SRO=Y N

S 83 F

X SRO=Y N

U S Distractors C&D implausible.

Rewrote all distractors 84 H

SRO=Y M

S 85 H

SRO=Y N

S 86 F

SRO=Y M

S 87 H

SRO=Y B

E Changed Answer to read verify closed vs direct closing.

88 H

SRO=Y B

S 89 H

SRO=Y B

S 90 H

SRO=Y N

U S DIRECT LOOK-UP WITH THE PROVIDED REF Changed Q require to determine operability and that work is unplanned 91 H

SRO=Y N

S 92 H

SRO=Y M

S 93 H

SRO=Y B

U S Oversampling the topic with Q94 Wrote new Q 94 H

SRO=Y B

S 95 F

SRO=Y B

S 96 F

1 SRO=Y N B U

LOW LOD Selected another Q from bank that meets K/A 97 H

X SRO=Y B

S 98 F

SRO=Y B

S

99 F

SRO=Y B

S 100 H

SRO=Y B

S

Form 2.3-5 Instructions for Written Examination Review Worksheet Refer to ES-4.2 for the definitions of terms used in this worksheet for the written examination.

Review each question (Q) as submitted and as subsequently revised and document the following in the associated worksheet columns:

1. Enter the level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) from 1 (easy) to 5 (difficult); mark direct lookup questions (applicant can directly determine the answer from the provided reference) as LOD 1. A question is (U)nsatisfactory if it is LOD 1 or LOD 5.

3. Check the appropriate box if a psychometric flaw is identified:

Stem Focus: The stem lacks enough focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information). This is an (U)nsatisfactory question.

Cues: The stem or one or more answer choices contains cues (e.g., clues, specific determiners, phrasing, length). This is an (U)nsatisfactory question.

T/F: All of the answer choices are a collection of unrelated true/false statements. This is an (U)nsatisfactory question.

Cred. Dist.: The distractors are not credible; single implausible distractors require (E)nhancement, and more than one noncredible distractor in the same question results in an (U)satisfactory question.

Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem). This is an (U)nsatisfactory question.

4. Check the appropriate box if a job content flaw is identified:

Job Link: The question is not linked to the job requirements (i.e., the question has a valid knowledge or ability (K/A) but, as written, is not operational in content). This is an (U)nsatisfactory question.

Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory). This is an (U)nsatisfactory question.

1. /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). This is an (U)nsatisfactory question.

Logic: The question requires backward or reverse logic or application compared to the job requirements. This is an (U)nsatisfactory question.

5. Check the first box if a K/A mismatch flaw exists. Check the second box if the question is flawed because it is written at the wrong license level. Either condition results in an (U)nsatisfactory question.

6. Enter the questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-4.2.

7. Based on the review performed in steps 2-5, mark the question as (U)nsatisfactory, in need of (E)nhancement, or (S)atisfactory.

8. Fully explain the reason for any (U) in column 7 (e.g., how the psychometric attributes are not being met).

9. Save the initial review comments and detail subsequent comment resolution so that each exam-bound question is marked by an (S) on this form.