ML103210263
| ML103210263 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 10/15/2010 |
| From: | Clyde Osterholtz Operations Branch IV |
| To: | Southern California Edison Co |
| References | |
| Download: ML103210263 (5) | |
Text
ES-401 SONGS 2010-10 Written Examination Review Worksheet Form ES-401-9 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
H 3
X B
US Two correct answers MODIFIED DISTRACTERS 2
F 2
N ES ADDED AUTOMATICALLY 3
F 2
B S
4 H
3 X
B ES B and C same answer REVISED B 5
H 3
B ES Define Stem DEFINED STEM 6
H 3
X N
ES B not credible REVISED B 7
F 2
N ES Add level ADDED 8
F 2
N S
9 F
2 B
S 10 F
2 B
ES Change A to one in A and B - fix margin REVISED
ES-401 SONGS-2010-10 2
Form ES-401-9 Page 2 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 11 H
2 N
S 12 H
3 X
N ES Change D to be consistent with C REVISED 13 F
2 B
S 14 H
2 N
ES D-an to a REVISED 15 H
4 B
ES Modify stem from <19% to actual level MODIFIED 16 H
3 N
ES Fix typo FIXED 17 H
3 N
ES Fix typo FIXED 18 H
3 B
S 19 F
2 X
B ES Change to 2 140s and 2 504s for consistency REVISED 20 H
4 X
X N
US Two correct answers REPLACED 21 H
3 X
X B
US Answer in stem REVISED 22 H
2 X
N ES Change correct answer to be less obvious CHANGED 23 F
2 B
S 24 F
2 B
S 25 H
2 N
S 26 H
3 N
S 27 H
3 N
S 28 F
2 N
S 29 F
4 X
M US Unnecessary info in stem makes question 50/50 - REMOVED UNNECESSARY INFO 30 H
3 X
B US Correct answer obvious REVISED 31 F
3 X
N ES D not plausible REVISED 32 H
3 X
B ES Modify A for consistency MODIFIED 33 F
2 N
ES Fix typo FIXED 34 F
2 N
S
ES-401 SONGS-2010-10 3
Form ES-401-9 Page 3 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 35 H
2 B
S 36 H
3 B
S 37 H
3 N
2 B
ES Get rid of modulate in C REVISED 39 H
3 B
S 40 H
2 B
S 41 H
3 B
S 42 H
2 B
ES Get rid of underline in stem REVISED 43 H
2 X
B US B and D not plausible MODIFIED 44 H
2 N
ES D not plausible MODIFIED 45 H
2 X
N ES Change C to be consistent with others MODIFIED 46 H
2 B
S 47 H
2 B
ES Clarify ESDE CLARIFIED 48 H
4 B
S 49 H
3 B
S 50 H
3 B
S 51 H
3 N
S 52 H
3 N
ES Fix typo FIXED 53 H
3 N
S 54 F
3 X
M ES Modify distracters to use actual values MODIFIED 55 F
2 X
B ES Modify distracters to remove unnecessary info MODIFIED 56 H
3 N
S 57 F
2 B
S 58 H
3 X
N ES D mentions a deviation not described in stem MODIFIED 59 F
2 X
N S
ES-401 SONGS-2010-10 4
Form ES-401-9 Page 4 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 60 H
2 X
N ES Modify D to make credible MODIFIED 61 F
3 X
X M
US No correct answer MODIFIED 62 H
2 N
ES Add fully to dropped ADDED 63 H
2 M
S 64 H
3 X
N US A and C do not address question REPLACED 65 H
2 X
B US Only one credible distracter REPLACED 66 F
2 X
N US No correct answer REPLACED 67 F
3 B
S 68 F
2 X
B ES D not credible MODIFIED 69 F
2 B
ES Add in service to temporary power ADDED 70 F
3 X
B US A and D correct MODIFIED A 71 F
2 X
B ES Add MINIMUM to stem ADDED 72 F
2 N
S 73 F
2 X
B ES Modify C and D for consistency MODIFIED 74 F
2 X
X B
ES Modify stem and distracters for consistency MODIFIED 75 F
2 B
S 76 H
3 X
N ES Add clarifying info ADDED 77 H
3 X
X N
US Question/K/A mismatch REPLACED 78 H
3 X
N S
79 H
3 X
X B
ES D not plausible REVISED 80 F
3 X
N S
81 F
4 X
B S
82 F
3 X
B S
83 H
4 X
N S
84 H
2 X
B S
ES-401 SONGS-2010-10 5
Form ES-401-9 Page 5 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 85 H
2 X
X B
ES B not plausible REVISED 86 H
3 X
N ES Modify stem - add power level, CDM temperature, remove delay and rising MODIFIED 87 H 2
X N
ES Change psia to psig CHANGED 88 H
4 X
B S
89 H
3 X
X N
ES Change Mode 5 to temperature in D MODIFIED 90 H
2 X
X N
ES Change 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> to 6 and 10 in A and D respectively CHANGED 91 H
3 X
X N
ES Stem needs a rad level to properly evaluate ADDED 92 H
3 X
X N
US Question/ K/A mismatch REPLACED 93 H
3 X
X X
N US Three correct answers REPLACED 94 H
2 X
X B
ES B not credible REVISED 95 H
3 X
X M
ES Modify stem to ensure only one correct answer MODIFIED 96 F
2 X
B S
97 H
2 X
X B
ES A not credible MODIFIED 98 F
2 X
X B
US No correct answer ALL DISTRACTERS MODIFIED EXCEPT B 99 F
2 X
B S
100 F
2 X
B S