ML112700882
| ML112700882 | |
| Person / Time | |
|---|---|
| Site: | Wolf Creek  |
| Issue date: | 08/26/2011 |
| From: | Clyde Osterholtz Operations Branch IV |
| To: | Wolf Creek |
| References | |
| 50-482/11-301 | |
| Download: ML112700882 (5) | |
Text
ES-401 WC-2011-08 Written Examination Review Worksheet Form ES-401-9 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
H 3
N S
2 F
2 X
B ES Make stem more restrictive Changed 85 to 80 to make C more plausible 3
F 2
X X
N US D gives away correct answer Replaced D 4
H 3
N S
5 F
2 N
ES Enhance C and D Enhanced 6
H 3
N US No correct answer as written Revised distracters to properly indicate correct answer 7
H 3
X B
ES C and D not credible Revised distracters 8
F 2
X N
ES Add temp and press info to stem Added 9
F 2
N S
10 F
2 N
S
ES-401 WC-2011-08 2
Form ES-401-9 Page 2 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 11 H
2 X
N ES Replace A Replaced 12 H
2 X
X N
US C and D not credible Revised 13 F
3 X
N ES Add stem info that verifies PR hi flux locks in Revised 14 H
3 N
S 15 H
3 X
B ES Replaced D Replaced 16 H
3 X
X N
US Bad stem and bad distracters Question rewritten 17 H
4 N
S 18 H
3 N
S 19 F
3 X
N ES Modify B and C to make more credible Revised 20 H
3 X
B ES Modify distracters for consistency Revised 21 H
2 N
S 22 H
3 B
S 23 F
3 N
S 24 F
3 X
B ES Modify stem + justification s dont back up distracters Modified; justifications clarified 25 H
3 X
N ES Fix typo in C Fixed 26 H
2 X
X N
US Answer in stem; D not credible Revised 27 H
2 M
S 28 F
3 N
ES Modify distracters to make more credible Modified 29 F
3 X
X N
ES Modify stem and distracters for consistency Modified 30 H
2 X
N S
31 F
2 X
X N
ES D not plausible; modify stem info Revised stem and D 32 H
2 X
N US B and D not credible Revised all distracters 33 F
2 X
B ES Delete unnecessary info in stem Deleted 34 F
3 X
N ES Fix typo in stem Fixed 35 H
2 X
N ES Revise stem - upper to 100% Program Revised
ES-401 WC-2011-08 3
Form ES-401-9 Page 3 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 36 H
2 N
S 37 H
3 X
N ES Need to say something about B RHR in stem Added 38 H
3 N
S 39 H
3 X
X B
US K/A mismatch - need new K/A and Question Added new K/A and Question 40 H
3 N
S 41 H
2 X
X N
US Got correct ans w/o reading question Revised distracters so correct ans not obvious 42 F
3 N
S 43 H
2 N
S 44 H
2 N
S 45 H
2 N
S 46 H
2 X
N ES Amend A and B to make more believable Amended 47 F
2 X
N ES Change A so that it is not so similar to D A changed 48 H
3 X
N ES B too close to correct answer Replaced B 49 H
2 X
X N
US A cues answer; C & D not credible Question rewritten to eliminate problems 50 H
3 N
ES Stem has wrong info Stem fixed 51 H
3 X
B ES Enhance A to make more credible A enhanced 52 F
3 X
B ES Modify D to make more believable D modified 53 F
2 X
X X
M US Got correct ans w/o reading question Revised distracters so correct ans not obvious 54 F
2 X
N US Only credible distracter is correct answer Question rewritten 55 F
3 X
M ES B not credible B replaced 56 H
4 N
S 57 F
3 M
S 58 H
3 X
N ES Delete PF from stem PF deleted 59 H
3 B
S 60 F
3 N
S
ES-401 WC-2011-08 4
Form ES-401-9 Page 4 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 61 H
2 X
X X
N US C cues answer, D not credible, B doesnt address stem Question rewritten 62 F
2 N
S 63 F
3 N
ES Revise C to make more clear Revised 64 H
2 N
US Got correct ans w/o reading question Revised distracters so correct ans not obvious 65 H
3 X
B US Only one credible distracter Question rewritten 66 F
2 B
S 67 F
2 X
N ES Remove unnecessary info from stem Removed 68 F
2 B
S 69 H
3 X
B ES Make B orange path to be consistent with C Revised 70 H
3 B
S 71 H
3 X
B ES Revise A to make more credible Revised 72 H
3 N
S 73 F
2 B
S 74 H
2 B
S 75 F
2 N
US B & C not credible Question rewritten 76 H
3 X
B S
77 H
3 X
N S
78 H
2 X
X N
US Only one credible distracter Question rewritten 79 H
2 X
X N
US Only one credible distracter Question rewritten 80 H
2 X
X X
N US No correct answer as written Question rewritten 81 H
2 X
X N
ES Change would to should in stem Changed 82 H
2 X
X M
ES Modify C to keep from giving away correct answer Modified 83 H
2 X
B S
84 H
2 X
N S
85 H
3 X
X N
ES Change PF info to Mvars Changed
ES-401 WC-2011-08 5
Form ES-401-9 Page 5 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 86 H
2 X
X N
ES Revise C and D to make more credible Revised 87 H 2
X X
N ES B not credible Replaced 88 F
2 X
X X
N ES Change A and C times; add procedurally to stem Changed and added 89 H
2 X
X N
ES Delete unnecessary info in stem Deleted unnecessary info 90 H
2 X
X N
US Answer obtained w/o reading question/too close to Question 59 Question replaced 91 F
2 X
X N
ES Modify stem to make distracters more plausible Modified 92 F
2 X
X X
N ES Clarify stem, replace non-credible D, make B & C more clear Changes made 93 H
2 X
X X
B US Entire question not credible Question replaced 94 F
3 X
B S
95 H
3 X
X X
M ES Modify stem and distracters to enhance question Modified 96 H
3 X
N S
97 H
2 X
X X
B US Answer is in question Question rewritten 98 H
3 X
X N
ES Enhance distracters C & D Enhanced 99 H
3 X
N S
100 H
3 X
N S