ML14336A361
| ML14336A361 | |
| Person / Time | |
|---|---|
| Site: | Arkansas Nuclear  |
| Issue date: | 11/21/2014 |
| From: | Vincent Gaddy Operations Branch IV |
| To: | |
| laura hurley | |
| References | |
| Download: ML14336A361 (5) | |
Text
ES-401 ANO1-2014-11 Written Examination Review Worksheet Form ES-401-9 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
F 2
N S
2 H
3 X
N E/S A not credible; SCM already rising with temperature dropping/Replaced A 3
F 2
N S
4 H
3 B
S 5
H 2
N S
6 H
2 B
S 7
H 2
N S
8 H
2 X
X N
U/S Same as Q 34/Replaced 9
F 2
N S
10 H
3 X
B U/S Correct answer only credible distractor/Replaced
Form ES-401-9 Page 2 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 11 H
3 M
S 12 H
3 X
N E/S C not credible with no reactor trip scenario/Replaced C 13 H
3 X
B U/S A and D are the same answer/Revised A 14 F
3 X
X B
E/S Change would to should**. D not credible/Revised 15 F
2 X
X N
E/S **; Change time increments in distractors as to not hit setpoint exactly/Revised 16 F
2 X
X B
E/S Cap LOWEST in stem. Stagger numbers in distractors to not hit stpt exactly/Revised 17 F
3 X
N E/S Add procedure in stem/Added 18 H
3 X
N E/S D not credible/Replaced D 19 H
3 M
S 20 F
2 X
N E/S Add procedure in stem/Added 21 H
3 N
S 22 H
2 X
B E/S Add procedure in stem/Added 23 H
4 X
N E/S Add event in stem/Added 24 H
3 X
B E/S D not plausible/Replaced D 25 H
3 X
N E/S Change distractor flow rates to be more realistic/Revised 26 F
3 B
S 27 H
2 X
B U/S C and D not credible/Replaced Question 28 H
3 X
N E/S **/Revised 29 H
3 X
N E/S Add continuous flow to A/Added 30 F
2 N
S 31 F
2 X
N U/S Adjust all distractors to make more credible/Adjusted 32 H
3 B
S 33 H
3 X
M E/S **/Revised 34 F
2 N
U/S Two correct answers as written/Revised stem to prevent multiple correct answers 35 F
2 X
M E/S **/Revised
Form ES-401-9 Page 3 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 36 F
2 X
N U/S Distractors not credible as written/Revised distractors to include channels 1 through 8 37 F
2 M
S 38 H
3 X
B E/S Add procedure in stem/Added 39 F
3 N
S 40 H
3 X
N U/S A and B not credible/Replaced Question 41 F
2 X
N E/S Capitalize BOTH in stem/Done 42 F
2 N
S 43 F
2 X
N U/S B and D not credible/Revised B and D to make credible 44 H
3 X
X N
E/S **; Revise A and B to make more consistent/Revised 45 H
4 X
N E/S Change shall to should in stem/Revised 46 H
3 N
S 47 H
3 B
S 48 H
3 X
X N
U/S A & B both correct as written; D is none of the above/Revised stem and distractors 49 H
3 X
B E/S **/Revised 50 H
3 M
S 51 H
3 X
X N
E/S **; D is none of the above/Revised 52 H
2 B
S 53 F
2 B
S 54 H
3 B
S 55 H
3 X
B E/S **/Revised 56 H
3 B
S 57 F
2 B
S 58 F
2 X
N E/S Revise distractors such that all four dont have the same info in them/Revised 59 H
3 B
S 60 H
3 X
B U/S B and D not credible/Revised B and D
Form ES-401-9 Page 4 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 61 H
3 X
X B
E/S Revise stem and distractors to make more credible/Revised 62 H
3 X
B E/S Add SCM and procedure in stem/Added 63 H
3 B
S 64 H
2 N
S 65 H
2 N
S 66 F
2 X
B E/S Add at a MINIMUM in stem/Added 67 F
3 X
B E/S Add procedure in stem/Added 68 F
2 X
X B
E/S Add procedure in stem; revise C to make more credible/Added and revised 69 F
2 N
S 70 F
2 B
S 71 F
2 M
S 72 H
3 B
S 73 F
2 X
N E/S Revise stem to remove unnecessary info/Revised 74 F
2 B
S 75 H
2 N
S 76 H
4 X
N S
77 H
3 X
X X
N E/S Revise question to be consistent with reference/Revised 78 H
3 X
X N
E/S Capitalize SPECIFICALLY in stem/Revised 79 H
4 X
N S
80 H
3 X
X N
U/S Revise A and B to avoid multiple correct answers/Revised 81 H
4 X
X B
U/S Direct lookup Tech Spec question/Replace 82 F
2 X
N S
83 H
4 X
X X
N E/S A not credible; add FIRST in stem; eliminate first half of all distractors/Revised 84 H
2 X
X X
B U/S Only correct answer credible + same as Q 93/Replaced 85 H
3 X
M S
Form ES-401-9 Page 5 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 86 H
3 B
S 87 H
3 X
B E/S Capitalize SPECIFIC in stem/Revised 88 F
2 B
S 89 H
4 X
N E/S Place noun names of valves in distractors/Revised 90 H
3 X
X B
U/S Only correct answer credible/Replaced 91 H
3 X
B U/S Q duplicates scenario instrument failure/Replaced 92 H
3 B
S 93 H
3 X
B U/S Only correct answer credible/Replaced 94 H
2 X
M E/S Revise stem to make different from original bank Q/Revised 95 H
3 X
B E/S A not credible/Replaced A 96 F
3 M
S 97 F
3 X
M E/S Add ONLY in A; capitalize AND in distractors/Revised 98 F
3 X
B E/S Eliminate unnecessary info in stem/Revised stem 99 F
2 N
S 100 H
2 X
N U/S Same Q as an Admin JPM/Replaced
- comment generic to entire test (change would to should)