ML13310A618
| ML13310A618 | |
| Person / Time | |
|---|---|
| Site: | South Texas  |
| Issue date: | 10/04/2013 |
| From: | Clyde Osterholtz Operations Branch IV |
| To: | South Texas |
| laura hurley | |
| References | |
| Download: ML13310A618 (5) | |
Text
ES-401 STP-2013-09 Written Examination Review Worksheet Form ES-401-9 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
H 3
X N
U/S Correct answer too obvious/Replaced 2
F 3
N S
3 F
3 N
S 4
F 3
X M
E/S Add Specific system to A/Added 5
H 3
X B
E/S Add actual ctmt press in stem/Added 6
H 3
N S
7 F
3 N
S 8
H 3
X X
M E/S Change stem to 28/make D correct answer/MIMIMUM in caps/Revised 9
F 3
N S
10 H
3 B
S
ES-401 STP-2013-09 2
Form ES-401-9 Page 2 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 11 F
3 B
S 12 H
3 B
S 13 H
3 X
M E/S In stem change 56.5 to 57.5 & make C correct answer/Revised 14 H
3 B
S 15 F
3 X
B E/S Capitalize NEXT in stem/Revised 16 F
2 X
B U/S B,C.D not credible/Replaced 17 H
3 X
X N
E/S Get rid of unnecessary info (cueing) in all distracters/ Revised 18 F
3 X
N U/S Q does not match K/A/Replaced 19 F
2 N
S 20 F
3 N
S 21 H
3 X
N E/S Abbreviate RCFC in distracters/Revised 22 F
3 X
X X
N U/S Two correct answers/Replaced 23 H
3 N
S 24 H
3 X
X N
E/S Add Procedure in stem; Change A/Revised 25 F
3 X
N E/S Add procedure in stem; enhance distracters/ Revised 26 F
3 X
X N
E/S Change slowly lowering to lowering in stem/Revised 27 F
2 X
N E/S Delete unnecessary info in B/Revised 28 H
3 X
X N
U/S A & B not credible/ Replaced 29 F
3 X
X N
E/S Get rid of unnecessary info in all distracters/ Revised 30 H
3 X
N E/S A not credible. Revise to make 2X2/ Revised 31 F
3 X
N E/S Delete first half of stem and add procedure title in stem/Revised 32 H
3 X
B E/S In stem put MINIMUM in caps/Revised 33 H
3 X
B U/S Two correct answers (A and B)/Replaced 34 H
4 B
S 35 F
3 B
S
ES-401 STP-2013-09 3
Form ES-401-9 Page 3 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 36 F
3 B
S 37 H
3 X
B E/S Underline all the ORs in the distracters/Revised 38 H
3 X
B E/S Change could to should in stem/ Revised 39 F
3 X
B E/S Put MINIMUM in caps in stem/ Revised 40 F
3 X
X N
E/S Delete first sentence in stem and add EDG in the remainder of stem/Revised 41 F
2 N
S 42 F
3 N
U/S A & C not credible/Replaced 43 F
3 X
B U/S B & D not credible/Revised distracters 44 H
3 X
X B
S 45 H
3 X
N U/S A & B not credible/Revised distracters 46 H
3 B
S 47 H
3 X
X B
U/S Q does not match K/A/Replaced 48 H
3 X
N U/S B & D not credible/Revised distracters 49 F
3 X
N U/S Q not credible (not being able to manually initiate fire protection)/Replaced 50 H
3 X
N E/S Add procedure in stem/Revised 51 H
3 X
X N
S 52 H
3 X
M E/S Change instantly to immediately in A& B and add times in C & D/ Revised 53 H
2 N
S 54 H
3 X
N E/S Put times in A & C/ Revised 55 H
3 X
X X
N E/S Get rid of second half on all distracters/ Revised 56 H
3 X
N E/S Add to AFW pump 14 in C & D/ Revised 57 F
2 B
U/S Too similar to question 43/Replace 58 H
3 N
S 59 F
3 X
N E/S Change would to should in all distracters/ Revised 60 F
2 N
E/S Revise stem to make more clear/Revised
ES-401 STP-2013-09 4
Form ES-401-9 Page 4 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 61 H
2 X
N U/S Only correct answer is credible/Revised 62 F
2 X
N E/S C not credible/ Revised 63 F
2 N
S 64 H
3 N
S 65 F
3 X
N E/S Change stem values so that all pumps have different temperatures/Revised 66 H
3 X
N U/S A & C not credible/ Revised 67 F
3 X
N E/S Add procedure in stem/ Revised 68 F
3 X
N E/S Chang each and selected to 25 and 50 and ask for TOTAL in stem/ Revised 69 H
3 N
E/S Remove equipment numbers from B/Revised 70 H
3 N
S 71 H
3 N
S 72 H
3 X
X N
E/S Add procedure in stem; eliminate unnecessary info in distracters/ Revised 73 H
3 X
N E/S D not credible/Revised 74 F
2 N
S 75 F
3 X
X B
U/S Two correct answers/Restructured Q to eliminate problem 76 F
3 X
N S
77 F
3 X
N S
78 H
3 X
B S
79 H
3 X
N S
80 H
3 X
N S
81 H
3 X
N S
82 H
3 X
N S
83 H
3 X
N S
84 H
3 X
X N
U/S Direct lookup - get rid of references/Deleted references 85 H
3 X
X N
E/S Add ONLY to A & D/Revised
ES-401 STP-2013-09 5
Form ES-401-9 Page 5 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 86 H
3 X
X X
N E/S Add valve name in stem and in A/Revised 87 H
3 X
X N
U/S Direct lookup - get rid of references/References deleted 88 F
3 X
N S
89 F
2 X
N S
90 F
3 X
N U/S B & C not credible /Replaced 91 H
3 X
X N
U/S B & D not credible/ Revised 92 F
2 X
N S
93 H
2 X
N S
94 F
3 X
X N
E/S Add procedures in stem//Revised 95 H
3 X
N U/S A & B not credible/Replaced 96 H
3 X
N U/S Too similar to Admin JPM/Replaced 97 H
3 X
X X
N E/S Minimum in caps in stem; Change C to acting rad mgr ONLY/ Revised & Changed 98 H
3 X
N S
99 H
3 X
N S
100 H
3 X
N S