ML14059A254

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML14059A254
Person / Time
Site:	Nine Mile Point
Issue date:	02/26/2014
From:	Exelon Generation Co
To:	Todd Fish, Brian Fuller Operations Branch I
Jackson D
Shared Package
ML13295A030	List: ML14059A010 ML14059A254 ML14059A255 ML14059A256 ML14059A258 ML14076A120 ML14076A121 ML14076A122 ML14076A123
References
TAC U01875
Download: ML14059A254 (6)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 2 y M s 2 H 3 y M s 3 F 2 y N s 4 M 3 y B s 5 H 3 y N s 6 F 3 y N s 7 F 2 y B s 8 F 2 y B s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 -5 (easy- difficult) rating scale (questions in the 2-4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

.. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 H 3 y B s 10 H 3 y M s 11 F 3 y N s 12 H 4 y N s 13 H 2 y B s 14 H 2 y N s 15 H 3 y M s 16 F 3 y M s 17 H 4 y N s I 18 H 3 y N s 19 H 3 y N s H 2 y N E Modified stem to focus on operation of system at NMP, not GFE level 1 20 I

21 H 2 y M s 22 H 3 y N s 23 F 2 y B s 24 H 3 y N s 25 H 3 y B s 26 F 2 N N u KIA MM, Resampled for new KIA,wrote new Q 27 H 3 y N s 28 F 3 y N s

ES-401 3 Form ES-401-9 I

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N UIE/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 y N s 30 F 2 y B s 31 F 3 y N s 32 F 3 y N s 33 H 2 y N s 34 H 2 y N s 35 H 3 y B s 36 H 3 y N s 37 H 4 y N s 38 H 4 y N s 39 F 3 y B s Restructured Q to fill in blank(s) format 40 F 2 y N s 41 H 2 y B s 42 M 2 y B s 43 M 3 y B s 44 F 3 y B s 45 F 2 y M s 46 F 2 y N s 47 F 3 y N s 48 F 3 y N s

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 H 3 y M s y

50 F 2 M s y

51 H 2 B s y

52 H 3 M s y

53 H 4 M s y

54 F 3 N s y

55 F 2 N s y

56 F 2 N s y

57 F 2 N s y

58 F 3 N s y

59 H 3 B s y

60 H 3 X N E Distractor D implausible. Fixed Q to make Distractor D plausible y

61 H 2 N s y

62 H 2 N s y

63 H 3 M s y

64 H 2 M s y

65 F 3 M s y

66 F 2 N s y

67 H 2 N s y

68 M 2 B s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only y

69 F 2 N s y

70 F 3 B s y

71 H 2 X B E Distractor B implausible, wrote new stem and distractors y

72 F 2 N s y

73 F 1 B u LOD=1; Selected new KIA, wrote new Q y

74 F 2 X N E Distractor B implausible, wrote new distractor y

75 F 2 N s y

76 F 3 y N s y

77 H 3 y N s y

I I

78 H 3 y rv1 s y

79 H 3 y N s y

80 H 2 y N s y

81 H 3 y B s y

82 F 3 y B s y

83 H 2 y M s y

84 H 3 y N E Deleted procedure reference in stem to retain 55.43(5) aspect of Q y

85 H 2 y N s y

86 H 2 y M s y

87 F 1 N N u LOD=1, column 1 is system knowledge, col 2 is immediate actions Resampled, New KIA, wrote new Q y

88 F 2 y N s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only y

89 H 1 y N u LOD=1, wrote new Q y

90 F 3 X y N E Subset issue; constructed new answer matrix y

91 H 3 y N s y

92 H 2 y B s y

93 H 2 y N s y

94 F 2 y B s y

95 F 2 y B s y

96 H 2 y B s y

97 F 2 y N s y

98 H 2 y B s y

99 H 2 y M s y

100 F 3

-- L_

y N s ---**-----*-