ML021070284
| ML021070284 | |
| Person / Time | |
|---|---|
| Site: | Hope Creek  |
| Issue date: | 03/18/2002 |
| From: | Diane Jackson NRC/RGN-I/DRS/OSB |
| To: | Conte R Public Service Enterprise Group |
| Conte R | |
| References | |
| 50-354/2002-301 50-354/2002-301 | |
| Download: ML021070284 (14) | |
Text
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LCD (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-Q=
SRO U/E/S Explanation Focus Dist.
Link units ward K/A IOnlyII
~(
/<
L Pp 2
P, f-i1 2-Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not lhnked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD
-r-i-T11 (F/H)
(1-5)
Stem Cues T/F Cred. IPartial Job-Minutia
- /
]Back-Q=
SRO U/E/S Explanation Focus Dist.
Link units ward K/A
__Only
~/L 2
(* /-(
2-("L( k(
tA
/ (
Lj,.
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a I - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The aquestion is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 44 of 46
-'4 NUREG-1021, Revision 8, Supplement 1 ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1 -5) Stem ~Cues TIF Cred. ~Partial Job-Minutia V1 Back-Q=
SRO IUIEIS Explanation Focus IDist.
Link units ward K/A Onlt 2.t 1m3 i 2-
,/
" A!"-
7_
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (ie., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (ie., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are desiqnated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
I 44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem ICues IT/F Cred. Partial Job-Minutia ~#/~ Back-Q=I SRO U/E/S Explanation Focusl I
I Dist.
Link I units ward K/A Only 3"/
/4 2
71 r-3r Z
V-5" C.'1 A
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks Sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
S4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 V
Up $1 44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD I
I T
I (F/H)
(1-5)
Stem Cuesl T/F ICred. IPartial Job-I Minutia I #/
Back-Q=
SRO U/E/S Explanation Focus jI I IDist. Link units ward K/A I Only
£1
/* {
Z "I
l
-v--
o~
.o
-,?-'
.,l* -
l,*LT "If
-('
- 1.
-c! /
2_
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the approprate box if a job content error is identified:
The question is not linked to the job requirements (i.e,, the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem ~Cues T/ýFjCred. Partial Job-Minutia
- 1 Back-Q=
SRO IUIEIS Explanation Focus 0Dist.
Link unitslward K/A Only_
L( 11 2
r-'
p-(
2_
Z-Ad 2-Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 44 of 46
- i) e i." e CC i*.,*'_.
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F7/H-1)
(1-5)
Stem ~Cues TIF Cred. Partial Job-Minutia
- /~ Back-Q I SRO UIJEIS Explanation IFH j -
Focuste ue Dist.
Link units ward K/A=s Only Explanation
&&1
- .(
- 1 A
2 "7) H z
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question ingallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are desi~Qnated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain an' "U" ratinls (e.l., how the Appendix B psychometric attributes are not being met.
NUREG-1021, Revision 8, Supplement 1 44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
L0K LOD r
T-Tt i
I(F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /1 Back-Q= SRO U/E/S Explanation Focus IDist.
Link units ward K/A Ony
? L
/ (
2 2 1, L (
L
$ I -(
L
-/"
W ~ ~
~ ' ~ 4 ~ & r ~
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
S1.
Enter the level of knowledge (L0K) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are desicjnated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratingls (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 A""& 7t 44 of 46
2*
(0 ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia NI/ Back-Q= ISRO U/E/S Explanation I
__ j Focus J~ JDist.
Link
___units ward~ KIA JOnly S
-;-7/% (J V5 (zr'9c',
75U 3
/
S 2o I 3
/
,5_
j 3
iti S
- 3 IP 3
_S Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1
- I 44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 11.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
ILOK I LOD ~
r
-r
- r~~~~
(F/H)
(1-5)
FcStem ICues I T/F Cred. Partial Job-Minutia
- 1 Back-Q=
SRO U/E/S Explanation 11 J~Focus[
I I Dist, I Link [
I units [ ward_
K/A JOnly 1I 3 1
1 1__6_d__SK Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 2- ý awpý 44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-Q=
SRO U/E/S Explanation Focus Dist.
Link units ward K/A Only
- to,,
d
- L S
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws 4, Job Content Flaws
- 5. Other
- 6.
- 7.
Q# LOK LOD T
I ir 1
(F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-Q=
SRO U/E/S Explanation Focus I
JDist.
Link units ward K/A IOnly o -
5
__/
in +/-.3
)1131J t4 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
S The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q# LOK LOD T
(F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-Q=
SRO U/E/S Explanation IFocust I
IDist.
Link units ward K/A Only 0 7 b
? 3
/
v Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (Flundamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 -4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
Th Ustem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appopiate box if a job content error is identified:
The quperstion is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are desigjnated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 I
44 of 46
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (1.5)
Stem Cues TI Cred. Partial Job-Minutia
- / I Back-Q=
SRO U/E/S Explanation Focus Dist.
Link unit ward K/A Only r/
S 15us I
1C. I I
.I5 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NUREG-1021, Revision 8, Supplement 1 I
44 of 46