ML030640309
ML030640309 | |
Person / Time | |
---|---|
Site: | Arkansas Nuclear ![]() |
Issue date: | 02/25/2002 |
From: | NRC Region 4 |
To: | Entergy Operations |
References | |
50-313/02-302 50-313/02-302 | |
Download: ML030640309 (10) | |
Text
ES-401 Written Examination Form ES-401-9 (R8, S1)
Review Worksheet FINAL QUESTION RESOLUTION
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0461 H 3 X X S (SRO) (MODIFIED) REPLACED WITH Q 467..
0452 H 2 X X S (SRO) (NEW) CFR REFERENCE ADDED 0432 H 2 X X S (SRO) (NEW) CFR REFERENCE ADDED 0454 H 3 X X S (SRO) (NEW) 0455 F 2 X X S (SRO) (NEW) CFR REFERENCE ADDED 0456 H 3 X X S (SRO) (NEW) CFR REFERENCE ADDED 0342 F 3 X X S (SRO) 0368 H 3 X X S (SRO) CFR REFERENCE ADDED 0278 H 3 X X S (SRO) CFR REFERENCE ADDED 0032 H 3 X X S (SRO) (MODIFIED) Q CHANGED 0367 F 2 X X S (SRO) 0062 H 4 X X S (SRO) CFR REFERENCE ADDED 0182 H 3 X X S (SRO) CFR REFERENCE ADDED 0347 H 2 X X S (SRO) CFR REFERENCE ADDED 0169 H 3 X X S (SRO) CFR REFERENCE ADDED 0458 F 2 X X S (SRO) 0407 F 2 X X S (SRO) 0457 H 2 X X S (SRO) 0459 F 2 X X S (SRO) 0331 H 4 X X S (SRO) 0460 H 3 X X S (SRO) 0411 H 2 X X S (SRO) 0354 F 2 X X S (SRO) 0117 F 3 X X S (SRO)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0129 F 3 X X S (SRO) 0194 H 3 X S (RO) (MODIFIED) CORRECTED ANSWER & NOTES TO AGREE.
0436 H 2 X S (RO) (MODIFIED) 0095 H 3 X S (BOTH) (MODIFIED) 0417 H 3 X S (BOTH) (NEW) 0344 H 2 X S (BOTH) 0229 H 3 X S (BOTH) 0174 F 3 X S (BOTH) 0414 F 3 X S (BOTH) (NEW) 0418 F 2 X S (BOTH) 0014 F 2 X S (BOTH) 0420 H 3 X S (BOTH) 0422 H 3 X S (BOTH) (MODIFIED) 0426 H 2 X S (BOTH) (NEW) 0423 H 3 X S (BOTH) (NEW) 0415 H 3 X S (BOTH) (NEW) 0427 H 3 X S (BOTH) (NEW) 0429 H 2 X S (BOTH) 0052 H 4 X S (BOTH) (MODIFIED) 0002 H 3 X S (BOTH) (MODIFIED) 0189 H 4 X S (BOTH) (MODIFIED) 0430 F 2 X S (BOTH) 0431 H 3 X S (BOTH) CHANGED QUESTION 0416 F 3 X S (BOTH) (NEW) 0063 H 3 X S (BOTH) 0435 H 3 X S (BOTH) (NEW)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0439 H 2 X S (BOTH) (NEW) 0197 H 3 X S (BOTH) 0441 F 2 X S (BOTH) (NEW) 0443 H 2 X S (BOTH) (NEW) 0447 F 3 X S (BOTH) (NEW) 0448 H 3 X S (BOTH) (NEW) 0151 H 3 X S (BOTH) (MODIFIED) 0463 H 3 X S (BOTH) (NEW) 0104 F 2 X S (BOTH) 0462 H 3 X S (BOTH) (NEW) CFR REFERENCE ADDED 0027 H 3 X S (RO) 0425 F 2 X S (RO) 0428 F 2 X S (RO) 0191 F 2 X S (RO) 0299 H 3 X S (RO) 0261 H 3 X S (RO) 0255 H 2 X S (RO) 0437 H 3 X S (RO) 0073 H 3 X S (RO) 0444 F 2 X S (RO) 0311 H 3 X S (RO) 0446 H 2 X S (RO) 0205 H 3 X S (RO) 0450 H 2 X S (RO) (NEW) 0046 H 3 X S (RO) 0170 H 3 X S (RO)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0389 F 2 X S (RO) CFR REFERENCE ADDED 0451 F 2 X S (RO) (NEW) 0245 F 2 X S (RO) 0232 H 2 X S (RO) CFR REFERENCE ADDED & QUESTION CLARIFIED 0233 H 2 X S (RO) 0126 H 3 X S (RO) 0161 H 2 X S (BOTH) 0419 H 3 X S (BOTH) 0020 H 4 X S (BOTH) 0015 H 2 X S (BOTH) 0421 H 3 X S (BOTH) 0285 H 2 X S (BOTH) 0363 H 3 X S (BOTH) 0372 H 4 X S (BOTH) 0337 H 3 X S (BOTH) 0164 H 2 X S (BOTH) 0035 H 2 X S (BOTH) 0184 H 3 X S (BOTH) 0364 H 2 X S (BOTH) CFR REFERENCE ADDED 0424 F 2 X S (BOTH) IMPORTANCE < 2.5. JUSTIFICATION ADDED TO OUTLINE.
0365 H 3 X S (BOTH) 0187 H 3 X S (BOTH) 0167 F 2 X S (BOTH) 0186 H 2 X S (BOTH) CFR REFERENCE ADDED 0137 F 2 X S (BOTH) 0264 F 2 X S (BOTH)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0464 H 2 X S (BOTH) MODIFIED Q 0222 H 3 X S (BOTH) 0434 H 3 X S (BOTH) 0379 H 2 X S (BOTH) 0440 F 2 X S (BOTH) 0085 F 2 X S (BOTH) 0442 H 3 X S (BOTH) 0077 H 3 X S (BOTH) 0200 H 3 X S (BOTH) 0445 F 2 X S (BOTH) 0140 H 3 X S (BOTH) REPLACED Q WITH Q468 0224 H 3 X S (BOTH) 0102 H 3 X S (BOTH) MODIFIED DISTRACTOR D 0449 H 3 X S (BOTH) 0210 F 2 X S (BOTH) 0108 H 3 X S (BOTH) 0244 F 2 X S (BOTH) 0113 H 4 X S (BOTH) CFR REFERENCE ADDED 0247 F 2 X S (BOTH) 0122 F 2 X S (BOTH) CFR REFERENCE ADDED 0121 F 2 X S (BOTH) 0234 H 3 X S (BOTH) CFR REFERENCE ADDED 0433 F 3 X S (RO) (N)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
@ The answer choices are a collection of unrelated true/false statements.
@ More than one distractor is not credible.
@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
@ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Form ES-401-9 (R8, S1)
Review Worksheet
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0461 H 3 X X X ? U (SRO) (MODIFIED) DISTRACTORS B & D NOT CREDIBLE.
DISTRACTOR A PROVIDES A CUE. CFR REFERENCE? WILL WORK ON DISTRACTORS OR REPLACE.
0452 H 2 X ? E (SRO) (NEW) CFR REFERENCE?
0432 H 2 X ? E (SRO) (NEW) CFR REFERENCE?
0454 H 3 X X S (SRO) (NEW) 0455 F 2 X ? E (SRO) (NEW) CFR REFERENCE?
0456 H 3 X ? E (SRO) (NEW) CFR REFERENCE?
0342 F 3 X X S (SRO) 0368 H 3 X ? E (SRO) CFR REFERENCE?
0278 H 3 X ? E (SRO) CFR REFERENCE?
0032 H 3 X ? U (SRO) (MODIFIED) NOT SRO ONLY. WILL REPLACE.
0367 F 2 X X S (SRO) 0062 H 4 X ? E (SRO) CFR REFERENCE?
0182 H 3 X ? E (SRO) CFR REFERENCE?
0347 H 2 X ? E (SRO) CFR REFERENCE?
0169 H 3 X X S (SRO) CFR REFERENCE?
0458 F 2 X X S (SRO) 0407 F 2 X X S (SRO) 0457 H 2 X X S (SRO) 0459 F 2 X X S (SRO) 0331 H 4 X X S (SRO) 0460 H 3 X X S (SRO) 0411 H 2 X X S (SRO) 0354 F 2 X X S (SRO)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0117 F 3 X X S (SRO) 0129 F 3 X X S (SRO) 0194 H 3 X E (RO) (MODIFIED) ANSWER & NOTES DO NOT AGREE. WILL CHANGE TO AGREE.
0436 H 2 X X S (RO) (MODIFIED) 0095 H 3 X S (BOTH) (MODIFIED) 0417 H 3 X S (BOTH) (NEW) 0344 H 2 X S (BOTH) 0229 H 3 X S (BOTH) 0174 F 3 X S (BOTH) 0414 F 3 X S (BOTH) (NEW) 0418 F 2 X S (BOTH) 0014 F 2 X S (BOTH) 0420 H 3 X S (BOTH) 0422 H 3 X S (BOTH) (MODIFIED) 0426 H 2 X S (BOTH) (NEW) 0423 H 3 X S (BOTH) (NEW) 0415 H 3 X S (BOTH) (NEW) 0427 H 3 X S (BOTH) (NEW) 0429 H 2 X S (BOTH) 0052 H 4 X S (BOTH) (MODIFIED) 0002 H 3 X S (BOTH) (MODIFIED) 0189 H 4 X S (BOTH) (MODIFIED) 0430 F 2 X S (BOTH) 0431 H 3 NO U (BOTH) QUESTION DOES NOT MATCH K/A. WILL REVIEW AND/OR CHANGE.
0416 F 3 X S (BOTH) (NEW)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0063 H 3 X S (BOTH) 0435 H 3 X S (BOTH) (NEW) 0439 H 2 X S (BOTH) (NEW) 0197 H 3 X S (BOTH) 0441 F 2 X S (BOTH) (NEW) 0443 H 2 X S (BOTH) (NEW) 0447 F 3 X S (BOTH) (NEW) 0448 H 3 X S (BOTH) (NEW) 0151 H 3 X S (BOTH) (MODIFIED) 0463 H 3 X S (BOTH) (NEW) 0104 F 2 X S (BOTH) 0462 H 3 X S (BOTH) (NEW) CFR REFERENCE 0027 H 3 X S (RO) 0425 F 2 X S (RO) 0428 F 2 X S (RO) 0191 F 2 X S (RO) 0299 H 3 X S (RO) 0261 H 3 X S (RO) 0255 H 2 X S (RO) 0437 H 3 X S (RO) 0073 H 3 X S (RO) 0444 F 2 X S (RO) 0311 H 3 X S (RO) 0446 H 2 X S (RO) 0205 H 3 X S (RO) 0450 H 2 X S (RO) (NEW)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0046 H 3 X S (RO) 0170 H 3 X S (RO) 0389 F 2 X S (RO) CFR REFERENCE?
0451 F 2 X S (RO) (NEW) 0245 F 2 X S (RO) 0232 H 2 X X E (RO) CFR REFERENCE? QUESTION NOT CLEAR - IS STEM ANSWER BEFORE OR AFTER SHIFT RELIEF? WILL CLARIFY.
0233 H 2 X S (RO) 0126 H 3 X S (RO) 0161 H 2 X S (BOTH) 0419 H 3 X S (BOTH) 0020 H 4 X S (BOTH) 0015 H 2 X S (BOTH) 0421 H 3 X S (BOTH) 0285 H 2 X S (BOTH) 0363 H 3 X S (BOTH) 0372 H 4 X S (BOTH) 0337 H 3 X S (BOTH) 0164 H 2 X S (BOTH) 0035 H 2 X S (BOTH) 0184 H 3 X S (BOTH) 0364 H 2 X S (BOTH) CFR REFERENCE?
0424 F 2 X E (BOTH) IMPORTANCE < 2.5. ADD JUSTIFICATION TO OUTLINE.
WILL ADD.
0365 H 3 X S (BOTH) 0187 H 3 X S (BOTH) 0167 F 2 X S (BOTH)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0186 H 2 X S (BOTH) CFR REFERENCE?
0137 F 2 X S (BOTH) 0264 F 2 X S (BOTH) 0464 H 2 X X E (BOTH) DISTRACTOR D NOT PLAUSIBLE. WILL MODIFY.
0222 H 3 X S (BOTH) 0434 H 3 X S (BOTH) 0379 H 2 X S (BOTH) 0440 F 2 X S (BOTH) 0085 F 2 X S (BOTH) 0442 H 3 X S (BOTH) 0077 H 3 X S (BOTH) 0200 H 3 X S (BOTH) 0445 F 2 X S (BOTH) 0140 H 3 X X X U (BOTH) DISTRACTORS B, C, D NOT CREDIBLE. STEM PROVIDES A ANSWER CUE. WILL REVIEW.
0224 H 3 X S (BOTH) 0102 H 3 X E (BOTH) DISTRACTOR D SHOULD BE REWORDED TO BE CONSISTENT WITH THE FORMAT OF THE OTHER DISTRACTORS.
WILL REVIEW.
0449 H 3 X S (BOTH) 0210 F 2 X S (BOTH) 0108 H 3 X S (BOTH) 0244 F 2 X S (BOTH) 0113 H 4 X S (BOTH) CFR REFERENCE 0247 F 2 X S (BOTH) 0122 F 2 X S (BOTH) CFR REFERENCE 0121 F 2 X S (BOTH) 0234 H 3 X S (BOTH) CFR REFERENCE
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 0433 F 3 X S (RO) (N)
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
@ The answer choices are a collection of unrelated true/false statements.
@ More than one distractor is not credible.
@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
@ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2 Form ES-401-9 (R8, S1)
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only