ML101240982
| ML101240982 | |
| Person / Time | |
|---|---|
| Site: | Comanche Peak  |
| Issue date: | 03/29/2010 |
| From: | Clyde Osterholtz Operations Branch IV |
| To: | Luminant Generation Co |
| References | |
| 50-445/10-301, 50-446/10-301 | |
| Download: ML101240982 (6) | |
Text
ES-401 CP-2010-03 Written Examination Review Worksheet Form ES-401-9 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 1 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
H 3
X N
US Two possible correct answers (a & b) REPLACED 2
F 2
B S
3 H
2 X
N S
4 H
3 N
S 5
H 2
B ES 6
H 3
N ES Whats an escutcheon? CHANGE TO HAND SWITCH 7
F 2
B S
8 H
3 X
N US Two possible correct answers MODIFIED B 9
H 3
N S
10 F
2 N
S
Form ES-401-9 Page 2 of 2 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 11 H
2 X
B US 2 (or more) correct answers if A, B, or C is correct MODIFIED STEM AND DISTRACTERS 12 H
3 B
S 13 F
2 X
B S
14 H
3 N
US 2 correct answers + wrong K/A REVISED K/A/ - REVISED D 15 H
2 B
ES MODIFIED A, C & D for BALANCE 16 H
2 N
2 X
N US 2.) not necessary - REPLACED; TOPIC COVERED IN JPM 18 H
3 M
ES Superfluous info in stem - MODIFIED 19 H
3 B
ES 20 F
2 B
ES Get rid of unnecessary info STEM AND DISTRACTERS MODIFIED 21 H
3 N
ES 22 H
3 B
ES K/A mismatch? CHANGED K/A AND ADDED ADDITIONAL REFERENCE 23 H
4 N
ES B and C need enhanced DISTRACTERS ENHANCED 24 H
3 N
S 25 H
3 X
N ES B not credible REPLACED 26 H
3 X
N ES Added MSODE 1 in stem 27 H
3 N
ES 28 H
3 X
B ES C nSot credible - has nothing to do with mid loop REVISED C 29 H
2 X
MB ES A not credible - closer to point source yet 1/4 dose rate (?) - REPLACED QUESTION 30 F
2 N
S 31 H
4 N
ES 32 H
3 N
ES 33 H
2 X
B US B is also correct MODIFIED STEM TO MAKE B INCORRECT 34 H
3 B
S
Form ES-401-9 Page 3 of 3 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 35 H
3 N
S 36 H
2 N
US C could also be correct (primary concern) - REVISED STEM TO MAKE C INCORRECT 37 H
3 N
S 38 H
2 N
S 39 F
2 X
B S
40 H
3 X
N ES Get rid of 2.s - EOS-1.1A transition by procedure E-0 DISTRACTERS REVISED 41 H
3 X
B ES Revise to make distracter C more consistent DISTRACTER C REVISED 42 F
2 X
B ES 43 F
3 B
ES Modify A for consistency - C should have no s A & C MODIFIED 44 F
3 B
ES Modify distracters for consistency DISTRACTERS MODIFIED 45 F
3 X
B ES Remove stem cue STEM MODIFIED 46 H
3 N
ES Modify D unnecessary info D MODIFIED 47 H
3 N
S 48 H
3 X
N US A and D not credible - no reason to believe ECA condition exists MODIFIED TO GET RID OF ECA CHOICES 49 H
4 N
ES is the last bullet in stem necessary? STEM MODIFIED 50 H
3 N
S 51 H
3 X
N ES A not credible - change D for consistency (?) A & D MODIFIED 52 H
3 X
X N
ES Stem not consistent with distracter D - Why is B wrong? STEM & B & D MODIFIED 53 H
3 X
M ES C not consistent with stem C MODIFIED 54 H
3 MB US Nearly identical to original question - OK if change from M to B CHANGED TO B 55 H
3 N
US Two correct answers MODIFIED DISTRACTERS 56 H
3 M
S 57 F
2 B
S
Form ES-401-9 Page 4 of 4 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 58 H
2 X
MB US Identical to bank question + A and C not credible CHANGED TO BANK 59 F
2 B
S 60 H
2 N
ES 61 F
2 X
N ES D credible (?) - MODIFIED D 62 H
3 N
ES 63 H
3 N
ES 64 H
3 N
S 65 H
2 X
N ES MODIFIED DISTRACTERS 66 F
1 X
N US Q not credible - When would following procedures be considered non conservative?
CHANGED TO A DIFFERENT CORRECT ANSWER 67 F
2 B
S 68 F
2 B
S 69 F
3 B
S 70 F
2 N
S 71 F
2 M
ES Change 2009 to 2010 -CHANGE MADE 72 F
2 X
M ES Add minimum -CHANGE MADE 73 F
2 X
N ES Add FRP Bases to stem -CHANGE MADE 74 F
2 B
S 75 F
2 X
X N
ES If A were correct, so would C (A not credible) - MODIFIED STEM TO MAKE C INCORRECT 76 H
3 X
X N
ES 77 H
3 X
B S
78 H
3 X
X N
S 79 H
2 X
N S
80 F
2 X
N ES
Form ES-401-9 Page 5 of 5 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 81 F
2 X
B ES If RV head not fully tensioned, isnt that Mode 6 (doesnt that limit temp?) CHANGE TO MODE 5 82 F
2 X
N ES Conditions in stem match question? MODIFIED STEM 83 H
3 X
B ES Add parameters to stem -ADDED 84 H
3 X
X X
N US A looks possibly correct and B and D dont appear credible - STEM REWORDED 85 H
2 X
N ES Only correct answer has a time limit -REMOVED TIME LIMIT FROM CORRECT ANSWER 86 H
2 X
X N
ES Why is B wrong -MODIFIED STEM AND DISTRACTERS 87 H 2
X X
N US Answer in stem (last bullet) - REMOVED CUE FROM STEM 88 H
2 X
N ES Modified stem 89 H
2 X
X N
US What does the diesel (just the diesel - no designation as to which one) have to do with Bus 1ECc1? - STEM AND DISTRACTERS MODIFIED 90 H
3 X
X N
ES B not credible - clarify stem info - why is ref material needed? - B REVISED; PROPOSED REFERENCE REMOVED 91 H
2 X
X N
US Why is D wrong? - nothing mentioned about potential curie content in stem D MODIFIED 92 F
2 X
B ES Provide reference for C -REFERENCE ADDED FOR C 93 F
2 X
N ES Simplify stem; explain FP terms and references - MODIFIED 94 F
3 X
X B
ES If C were correct, so would A - makes C not credible - DISTRACTERS MODIFIED 95 F
2 X
B ES provide references - REFERENCES ADDED 96 F
2 X
B S
97 F
2 X
N ES C - change 14 to 7 based on TS(?) - MODIFIED C 98 F
3 X
B S
99 H
3 X
N ES Change to higher EAL - MODIFIED QUESTION TO MAKE CORRECT ANSWER A HIGHER EAL 100 H
2 X
B=
F=
E=
Additional Notes:
Form ES-401-9 Page 6 of 6 OBDI 202 - IOLE Process M=
H=
U=
N=
SRO TOTALS:
B=
F=
E=
Additional Notes:
M=
H=
U=
N=
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
- 2.
Chief Examiner comments are indicated in blue.
- 3.
Average difficulty is __ ___ on the RO exam and ___ on the SRO exam.
- 4.
The 10CFR55.41/43 distribution is: RO / SRO 41.1 =
43.1 =
41.2 =
43.2 =
41.3 =
43.3 =
41.4 =
43.4 =
41.5 =
43.5 =
41.6 =
43.6 =
41.7 =
43.7 =
41.8 =
41.9 =
41.10 =
41.11 =
41.12 =
41.13 =
41.14 =
- 5.
The answer distribution is: RO / SRO A = 13 (17%)
/
6 (24%)
B = 16 (22%)
/
7 (28%)
C = 21 (28%)
/
5 (20%)
D = 25 (33%)
/
7 (28%)
- 6.
There are questions with attachments provided.