ML19093A999
| ML19093A999 | |
| Person / Time | |
|---|---|
| Site: | Indian Point  |
| Issue date: | 03/25/2019 |
| From: | Entergy Nuclear Indian Point 2 |
| To: | Peter Presby Operations Branch I |
| Shared Package | |
| ML18130A647 | List: |
| References | |
| CAC 00500, EPID L-2019-OLL-0012 | |
| Download: ML19093A999 (7) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q LOK LOD Source Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /Units Backward Q-SRO (B/M/N)
(U/E/S)
Focus Dist Link KIA Only 1
F 2
N E
Reworded Q stem to improve clarity.
2 H
2 N
s 3
F 2
X B
u KA mismatch. Replaced Q.
Low LOD, possible issue with B distractor, possibly second correct 4
F 2
X X
N u
answer.
Added procedure Rewrote Q
to improve KA match.reference to stem and rewrote distractors.
5 H
3.5 B
E Added units/symbols for all parameters to prevent cueing.
6 F
2.5 X
B E
Choice B implausible. Replaced Choice B. Minor stem wording change.
7 H
2.5 B
s Low LOD. Poor alignment between Q and choices.
Choices 8
F 2
N E
related to AMSAC have low discriminatory value. Reworded stem and choices.
Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.
- 3.
Check the appropriate box if a psychometric flaw is identified:
"Stem Focus": The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
"Cues": The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).
'T/F": The answer choices are a collection of unrelated true/false statements.
"Cred. Dist.": The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.
"Partial": One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).
- 4.
Check the appropriate box if a job content flaw is identified:
"Job Link": The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
"Minutia": The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).
#/Units": The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
"Backward": The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those KIAs that are designated "SRO-only." (KIA and license-level mismatches are unacceptable.)
- 6.
Enter question's source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.
- 7.
Based on the reviewer's judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any "U" status ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q LOK LOO Source Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /Units Backward Q-SRO (B/M/N)
(U/E/S)
Focus Dist Link KIA Only 9
F 2
B s
Possible overlap with question 3?
10 H
2.5 N
E Minor editorial change to stem. Elaborated on plausibility of distractors B and D in justification section.
11 F
2.5 B
s 12 F
3 N
E Resolved inconsistency in key as to the correct answer.
Minor editorial and typographical changes.
13 F
3.5 N
s 14 H
3 B
E Improved plausibility of distractors by lowering initial power level and changing distractor A 15 F
2.5 B
E Modified Choice A to eliminate possibility of cueing.
16 H
2 B
s 17 F
2.5 N
s 18 H
2.5 B
s 19 F
2.5 B
E Added information to distractor Choice A to improve plausibility.
20 H
2.5 N
s 21 H
3 N
s 22 H
3 N
E Minor typographical changes.
23 H
3 M
s 24 H
2.5 B
s 25 H
2.5 B
s
ES-401 3
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q LOK LOD Source Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /Units Backward Q-SRO (B/M/N)
(U/E/S)
Focus Dist Link KIA Only 26 H
4 N
u Overlap with 020. Replaced the Q.
27 F
3 B
E Made wording consistent in choices.
28 H
2.5 M
s 29 H
2.5 B
E Deleted leading 3rd bullet. Deleted unnecessary information in question.
30 H
2 B
s 31 F
2 N
E Recategorized Q as higher order. Revised stem to raise LOO.
Recategorized Q as higher order. Enhanced clarity by 32 F
3 N
E removing the assumption statement from the Q and revising last stem bullet.
33 F
3.5 B
s 34 H
3.5 B
E Modified times in distractors to align with reasonable misconceptions as stated in distractor justifications.
35 F
3.5 N
s 36 H
2 X
N u
Low LOO. Does not match KA. Replaced Q.
37 H
3 B
s 38 H
3 B
E Removed unnecessary information from the stem.
39 F
2.5 N
E Added additional information to stem to support distractor plausibility.
40 F
3 B
s 41 F
2.5 B
E Removed some information from distractors A and D to improve plausibility.
42 H
3.5 M
s
ES-401 4
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws 5.0ther
- 6.
- 7.
Q LOK LOO Source Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /Units Backward Q-SRO (B/M/N)
(U/E/S)
Focus Dist Link KIA Only 43 H
3.5 B
E Modified stem and distractors to improve clarity and ensure sound basis for key answer.
44 F
2.5 B
E Minor editorial changes.
45 H
3.5 B
E Modified Q ('could be' vice 'is the') to ensure answer technical accuracy.
46 H
4 N
s In stem, changed 'event' to 'fluctuations in grid voltage' for clarity.
47 H
3.5 B
s Addeded statement to stem to ensure technical accuracy.
48 F
3.5 N
E Reworded choices and stem to move repeated words from choices to the stem.
49 H
3 B
E Minor stem rewording to eliminate potential cueing.
Added specific value and limit to stem to improve plausibility 50 F
3 B
s of Choice A. Referenced procedure in stem to ensure only one correct answer. Minor formatting changes for consistencv.
51 H
2 X
N u
Rewrote Q to match KA.
52 F
3 B
s 53 F
3.5 N
E Modified Choices C and D to address possibility that a distractor could be eliminated based on another distractor.
54 H
2.5 B
s Recategoried as fundamental level Q.
55 H
2.5 B
E Modified choices to improve discriminatory value.
56 H
3 M
s Recategorized as a bank Q.
57 H
3.5 B
E Rephrased key answer to match style of distractor for psychometric balance.
58 H
3 M
s
ES-401 5
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q LOK LOO Source Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /Units Backward Q-SRO (B/M/N)
(U/EIS)
Focus Dist Link KIA Only 59 F
3 X
N u
Added initial power level to stem. Restructured Q so no longerT/F.
60 F
3 N
E Added power level to stem. Reworded Q to make specific to situation. Reworded choices to improve balance.
61 H
3 B
s 62 H
3.5 N
E Added an initial power level for clarity. Removed unnecessary underlining emphasis in the stem.
63 H
3 B
E Modified distractors to improve plausibility.
64 H
3 X
B u
Collection of T/F statements. Rewrote Q andfhoices.
65 H
3 B
u Overlap with operating test. Replaced the Q.
66 F
3 B
E Minor grammatical changes to stem.
67 H
3 X
B u
Reselected KA and replaced Q.
68 F
2.5 N
E Changed pedigree from N to B. Moved words from choices to stem.
69 F
2.5 B
E Inserted portion of drawing into Q stem rather than provided a handout.
70 H
2.5 N
E 2nd part of Q to closely linked to 1st part. Modified Q and choices to ensure one part didn't answer the other part.
71 H
2 X
B u
Replaced Q with one that meets plant-wide generic KA requirements.
72 H
3 X
B u
Multiple implausible distractors. Revised Q and choices.
73 F
3.5 B
E Reworded key answer to ensure it is not cueing.
74 F
3 N
s Changed pedigree from N to B.
75 H
2.5 B
s
ES-401 6
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws 5.0ther
- 6.
- 7.
Q LOK LOD Source Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /Units Backward Q-SRO (B/M/N)
(U/E/S)
Focus Dist Link KIA Only 76 H
3 B
E Modified Q slightly to improve alignment with choices.
77 H
3 N
s 78 H
3 N
E Changed PIGS data to screen captures to ensure alignment with KA.
79 H
3 N
E Revised choice B to make more distinct from choice A.
80 H
3 N
s 81 H
3 N
s 82 H
3 B
s 83 H
3 B
s 84 H
3.5 N
s 85 H
3 M
E Added additional information to stem for clarity.
86 H
3 N
E Reordered stem bullets for clarity.
87 H
3 N
E Reworded Q and choices to improve clarity.
88 H
3 N
E Reworded Q for clarity.
89 F
2.5 B
E Minor typographical changes.
90 H
3 N
u Flawed 2x2 structures. Able to answer Q with only 2 of 3 items in choice. Restructured Q and choices.
91 H
3 B
s 92 H
3 N
E Revised stem and slight modification to each distractor to ensure key answer correct and distractors wrong.
ES-401 7
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q LOK LOD Source Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /Units Backward Q-SRO (B/M/N)
(U/E/S)
Focus Dist Link KIA Only 93 H
3 N
s 94 F
3 B
s 95 F
3 B
s 96 H
3 B
E Clarified only 1 reference handout to be provided.
97 H
3 N
E Modified Choice D to improve plausibility.
98 F
3 B
s 99 H
3 N
s 100 H
3 B
s