ML16081A285: Difference between revisions
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
||
(2 intermediate revisions by the same user not shown) | |||
Line 3: | Line 3: | ||
| issue date = 11/20/2015 | | issue date = 11/20/2015 | ||
| title = NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2) | | title = NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2) | ||
| author name = Fish T | | author name = Fish T | ||
| author affiliation = NRC/RGN-I/DRS/OB | | author affiliation = NRC/RGN-I/DRS/OB | ||
| addressee name = | | addressee name = | ||
Line 19: | Line 19: | ||
=Text= | =Text= | ||
{{#Wiki_filter: | {{#Wiki_filter:ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9 | ||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions | |||
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] | [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] | ||
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOO) of each question using a 1 -5 (easy -difficult) rating scale (questions in the 2 -4 range are acceptable). | : 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. | ||
: 3. Check the appropriate box if a psychometric flaw is identified: . The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) . | : 2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable). | ||
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f. 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? | : 3. Check the appropriate box if a psychometric flaw is identified: | ||
: 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met). | .. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) . | ||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.-7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s | The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) . | ||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s ES-401 4 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s ES-401 5 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s | The answer choices are a collection of unrelated true/false statements . | ||
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable . | |||
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) . | |||
4. | |||
..Check the appropriate box if a job content error is identified: | |||
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) . | |||
.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) . | |||
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) . | |||
The question requires reverse logic or application compared to the job requirements . | |||
: 5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable). | |||
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f. | |||
: 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? | |||
: 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met). | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.- 7. 8. | |||
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s | |||
ES-401 4 Form ES-401-9 | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s | |||
ES-401 5 Form ES-401-9 | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q | |||
67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s 71 F 3 | |||
- -N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s | |||
ES-401 6 Form ES-401-9 | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC - EVALUATED ON OP TEST. | |||
SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s | |||
ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9 | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions | |||
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] | [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] | ||
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOO) of each question using a 1 -5 (easy -difficult) rating scale (questions in the 2 -4 range are acceptable). | : 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. | ||
: 3. Check the appropriate box if a psychometric flaw is identified: . The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) . | : 2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable). | ||
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f. 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? | : 3. Check the appropriate box if a psychometric flaw is identified: | ||
: 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met). | .. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) . | ||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.-7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s | The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) . | ||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s ES-401 4 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s ES-401 5 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s | The answer choices are a collection of unrelated true/false statements . | ||
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable . | |||
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) . | |||
4. | |||
..Check the appropriate box if a job content error is identified: | |||
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) . | |||
.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) . | |||
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) . | |||
The question requires reverse logic or application compared to the job requirements . | |||
: 5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable). | |||
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f. | |||
: 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? | |||
: 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met). | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.- 7. 8. | |||
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s | |||
ES-401 4 Form ES-401-9 | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s | |||
ES-401 5 Form ES-401-9 | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q | |||
67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s 71 F 3 | |||
- -N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s | |||
ES-401 6 Form ES-401-9 | |||
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. | |||
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC - EVALUATED ON OP TEST. | |||
SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s}} |
Latest revision as of 05:09, 5 February 2020
ML16081A285 | |
Person / Time | |
---|---|
Site: | Salem ![]() |
Issue date: | 11/20/2015 |
From: | Todd Fish Operations Branch I |
To: | Public Service Enterprise Group |
Shared Package | |
ML15135A303 | List: |
References | |
TAC U01910 | |
Download: ML16081A285 (6) | |
Text
ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .
The answer choices are a collection of unrelated true/false statements .
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .
4.
..Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .
.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .
The question requires reverse logic or application compared to the job requirements .
- 5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
- 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.- 7. 8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q
67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s 71 F 3
- -N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC - EVALUATED ON OP TEST.
SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s
ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .
The answer choices are a collection of unrelated true/false statements .
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .
4.
..Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .
.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .
The question requires reverse logic or application compared to the job requirements .
- 5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
- 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.- 7. 8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q
67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s 71 F 3
- -N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC - EVALUATED ON OP TEST.
SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s