ML16081A285: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
Line 19: Line 19:


=Text=
=Text=
{{#Wiki_filter:s 0/;Cf )DfS ES-401 Written Examination Review Worksheet Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #I Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions
{{#Wiki_filter:ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9
: 1.           2.         3. Psychometric Flaws               4. Job Content Flaws         5. Other   6.     7.                                     8.
Q#   LOK         LOO (F/H)       (1-5) Stem Cues         T/F   Cred. Partial Job- Minutia #I           Back-   Q= SRO B/M/N U/E/S                                     Explanation Focus                     Dist.         Link           units ward       KIA Only 1   H           3                                                                                           N     s 2     F           2                                                                                           B     s 3   H           1                                                                                           N     u     GFE; LOD=1; REPLACED KA, WROTE NEW Q 4     F           2                                                                                           B     s 5   F           2                                                                                           N     s 6     F           3                                                                                           N     s 7   H           3                                                                                           B     s 8     H           2                                                                                         M       s Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOO) of each question using a 1 -5 (easy -difficult) rating scale (questions in the 2 -4 range are acceptable).
: 1.           Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
: 3. Check the appropriate box if a psychometric flaw is identified: . The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) . . The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) . . The answer choices are a collection of unrelated true/false statements . . The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable . . One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) . 4. Check the appropriate box if a job content error is identified: . The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) . . The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) . . The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) . . The question requires reverse logic or application compared to the job requirements . 5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
: 2.           Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f. 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 3.           Check the appropriate box if a psychometric flaw is identified:
: 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
          ..            The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.-7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s
            ...
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s ES-401 4 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s ES-401 5 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s --71 F 3 N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s ES-401 6 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC -EVALUATED ON OP TEST. SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s s 0/;Cf )DfS ES-401 Written Examination Review Worksheet Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #I Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .
The answer choices are a collection of unrelated true/false statements .
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .
4.
            ..Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .
              ..          The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .
The question requires reverse logic or application compared to the job requirements .
: 5.           Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
: 6.           Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
: 7.           Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 8.           At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other 6.- 7.     8.
Q# LOK   LOO (F/H) (1-5) Stem Cues   T/F   Cred. Partial Job- Minutia #/     Back- Q= SRO B/M/N U/E/S Explanation Focus             Dist.         Link         units ward   KIA Only 9   F     2                                                                         N   s 10   H     3                                                                         N   s 11   H     3                                                                         B   s 12   H     3                                                                         B   s 13   F     2                                                                         N   s 14   H     3                                                                         N   s 15   H     3                                                                         B   s 16   F     2                                                                         B   s 17   F     2                                                                         N   s 18   F     3                                                                         N   s 19   F     3                                                                         B   s 20   H     3                                                                         M   s 21   F     3                                                                         B   s 22   H     3                                                                         B   s 23   F     3                                                                         B   s 24   F     3                                                                         B   s 25   H     3                                                                         B   s 26   F     2                                                                         B   s 27   F     2                                                                         M   s 28   F     2                                                                         M   s
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other 6. 7.                             8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/     Back- Q= SRO B/M/N U/E/S                       Explanation Focus             Dist.         Link         units ward   KIA Only 29   H     3                                                                         B   s 30   F     2                                                                         M   s 31   F     2                                                                         M   s 32   F     2                                                                         M   s 33   F     2                                                                         N   s 34   H     3                             x                                         B   u   NON-DISCRIMINATING Q; WROTE NEW Q 35   H     3                                                                         N   s 36   H     3                                                                         B   s 37   H     3                                                                         N   s 38   H     2                                                                         N'   s 39   F     2                                                                         N   s 40   H     3                                                                         B   s 41   F     2                                                                         N   s 42   H     3                                                                         B   s 43   H     3                                                                         M   s 44   F     2                             x                                         N   u   NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45   F     2                                                                         B   s
 
ES-401                                                                         4                             Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.     8.
Q# LOK   LOO (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/     Back- Q=   SRO B/M/N U/E/S Explanation Focus             Dist.         Link         units ward   KIA Only 46   H     3                                                                           N     s 47   H     3                                                                           N     s 48   F     2                                                                           B     s 49   H     3                                                                           M     s 50   F     3                                                                           B     s 51   H     3                                                                           N     s 52   H     3                                                                           N     s 53   F     2                                                                           B     s 54   H     3                                                                           N     s 55   H     3                                                                           B     s 56   H     3                                                                           B     s 57   F     2                                                                           B     s 58   H     3                                                                           B     s 59   H                                                                                 N     s 60   H                                                                                 B     s 61   H                                                                                 B     s 62   H                                                                                 B     s 63   F                                                                                 B     s 64   H                                                                                 N     s 65   H                                                                                 N_   s
 
ES-401                                                                         5                                                     Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other 6. 7.                           8.
Q#   LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/     Back- Q= SRO B/M/N U/E/S                       Explanation Focus             Dist.         Link         units ward   KIA Only 66   F     1                             x                                           N   u   LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q
67   H     3                                                                         B   s 68   F     2                                             x                           B   u   MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69   F     2                                             x                           N   u   MINUTIA; WROTE NEW Q 70   H     2                                                                         M   s 71   F     3
                                                    -                                -N   s 72   F     2                                                                         B   s 73   F     2                                                                         B   s 74   H     3                                                                         N   s 75   F     2                                                                         B   s 76   H     3                                                                         N   s 77   F     4                                                                         N   s 78   H     4                                                                         M   s 79   H     4                                                                         N   s 80   H     3                                                                         M   s 81   H     3                                                                         N   s 82   H     3                                                                         B   s 83   H     4                                                                         B   s 84   F     3                                                                         B   s 85   F     3                                                                         B   s
 
ES-401                                                                         6                                                   Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other 6. 7.                           8.
Q#   LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/     Back- Q= SRO B/M/N U/E/S                     Explanation Focus             Dist.         Link         units ward   KIA Only 86   H     4                                                                         M   s 87   H     4                                                                         B   s 88   H     3                                                                         B   s 89   H     4                                                                         N   s 90   H     4                                                                         N   s 91   H     4                                                                         N   s 92   H     4                                                                         N   s 93   H     3                                                                         N   s 94   F     3                                                                         B   s 95   H     3                                                                         B   s 96   F     3                                                                         N   s 97   H     3                                                                         B   u   OVER-SAMPLED TOPIC - EVALUATED ON OP TEST.
SELECTED NEW KA, WROTE NEW Q 98   F     3                                                                         B   s 99   H     4                                                                         N   s 100   F     3                                                                         N   s
 
ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9
: 1.           2.         3. Psychometric Flaws               4. Job Content Flaws         5. Other   6.     7.                                     8.
Q#   LOK         LOO (F/H)       (1-5) Stem Cues         T/F   Cred. Partial Job- Minutia #I           Back-   Q= SRO B/M/N U/E/S                                     Explanation Focus                     Dist.         Link           units ward       KIA Only 1   H           3                                                                                           N     s 2     F           2                                                                                           B     s 3   H           1                                                                                           N     u     GFE; LOD=1; REPLACED KA, WROTE NEW Q 4     F           2                                                                                           B     s 5   F           2                                                                                           N     s 6     F           3                                                                                           N     s 7   H           3                                                                                           B     s 8     H           2                                                                                         M       s Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOO) of each question using a 1 -5 (easy -difficult) rating scale (questions in the 2 -4 range are acceptable).
: 1.           Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
: 3. Check the appropriate box if a psychometric flaw is identified: . The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) . . The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) . . The answer choices are a collection of unrelated true/false statements . . The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable . . One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) . 4. Check the appropriate box if a job content error is identified: . The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) . . The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) . . The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) . . The question requires reverse logic or application compared to the job requirements . 5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
: 2.           Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f. 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 3.           Check the appropriate box if a psychometric flaw is identified:
: 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
          ..            The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.-7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s
            ...
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s ES-401 4 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s ES-401 5 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s --71 F 3 N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s ES-401 6 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC -EVALUATED ON OP TEST. SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s}}
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .
The answer choices are a collection of unrelated true/false statements .
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .
4.
            ..Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .
              ..          The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .
The question requires reverse logic or application compared to the job requirements .
: 5.           Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
: 6.           Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
: 7.           Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 8.           At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other 6.- 7.     8.
Q# LOK   LOO (F/H) (1-5) Stem Cues   T/F   Cred. Partial Job- Minutia #/     Back- Q= SRO B/M/N U/E/S Explanation Focus             Dist.         Link         units ward   KIA Only 9   F     2                                                                         N   s 10   H     3                                                                         N   s 11   H     3                                                                         B   s 12   H     3                                                                         B   s 13   F     2                                                                         N   s 14   H     3                                                                         N   s 15   H     3                                                                         B   s 16   F     2                                                                         B   s 17   F     2                                                                         N   s 18   F     3                                                                         N   s 19   F     3                                                                         B   s 20   H     3                                                                         M   s 21   F     3                                                                         B   s 22   H     3                                                                         B   s 23   F     3                                                                         B   s 24   F     3                                                                         B   s 25   H     3                                                                         B   s 26   F     2                                                                         B   s 27   F     2                                                                         M   s 28   F     2                                                                         M   s
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other 6. 7.                             8.
Q# LOK LOO (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/     Back- Q= SRO B/M/N U/E/S                       Explanation Focus             Dist.         Link         units ward   KIA Only 29   H     3                                                                         B   s 30   F     2                                                                         M   s 31   F     2                                                                         M   s 32   F     2                                                                         M   s 33   F     2                                                                         N   s 34   H     3                             x                                         B   u   NON-DISCRIMINATING Q; WROTE NEW Q 35   H     3                                                                         N   s 36   H     3                                                                         B   s 37   H     3                                                                         N   s 38   H     2                                                                         N'   s 39   F     2                                                                         N   s 40   H     3                                                                         B   s 41   F     2                                                                         N   s 42   H     3                                                                         B   s 43   H     3                                                                         M   s 44   F     2                             x                                         N   u   NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45   F     2                                                                         B   s
 
ES-401                                                                         4                             Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.     8.
Q# LOK   LOO (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/     Back- Q=   SRO B/M/N U/E/S Explanation Focus             Dist.         Link         units ward   KIA Only 46   H     3                                                                           N     s 47   H     3                                                                           N     s 48   F     2                                                                           B     s 49   H     3                                                                           M     s 50   F     3                                                                           B     s 51   H     3                                                                           N     s 52   H     3                                                                           N     s 53   F     2                                                                           B     s 54   H     3                                                                           N     s 55   H     3                                                                           B     s 56   H     3                                                                           B     s 57   F     2                                                                           B     s 58   H     3                                                                           B     s 59   H                                                                                 N     s 60   H                                                                                 B     s 61   H                                                                                 B     s 62   H                                                                                 B     s 63   F                                                                                 B     s 64   H                                                                                 N     s 65   H                                                                                 N_   s
 
ES-401                                                                         5                                                     Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other 6. 7.                           8.
Q#   LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/     Back- Q= SRO B/M/N U/E/S                       Explanation Focus             Dist.         Link         units ward   KIA Only 66   F     1                             x                                           N   u   LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q
67   H     3                                                                         B   s 68   F     2                                             x                           B   u   MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69   F     2                                             x                           N   u   MINUTIA; WROTE NEW Q 70   H     2                                                                         M   s 71   F     3
                                                    -                                -N   s 72   F     2                                                                         B   s 73   F     2                                                                         B   s 74   H     3                                                                         N   s 75   F     2                                                                         B   s 76   H     3                                                                         N   s 77   F     4                                                                         N   s 78   H     4                                                                         M   s 79   H     4                                                                         N   s 80   H     3                                                                         M   s 81   H     3                                                                         N   s 82   H     3                                                                         B   s 83   H     4                                                                         B   s 84   F     3                                                                         B   s 85   F     3                                                                         B   s
 
ES-401                                                                         6                                                   Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other 6. 7.                           8.
Q#   LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/     Back- Q= SRO B/M/N U/E/S                     Explanation Focus             Dist.         Link         units ward   KIA Only 86   H     4                                                                         M   s 87   H     4                                                                         B   s 88   H     3                                                                         B   s 89   H     4                                                                         N   s 90   H     4                                                                         N   s 91   H     4                                                                         N   s 92   H     4                                                                         N   s 93   H     3                                                                         N   s 94   F     3                                                                         B   s 95   H     3                                                                         B   s 96   F     3                                                                         N   s 97   H     3                                                                         B   u   OVER-SAMPLED TOPIC - EVALUATED ON OP TEST.
SELECTED NEW KA, WROTE NEW Q 98   F     3                                                                         B   s 99   H     4                                                                         N   s 100   F     3                                                                         N   s}}

Revision as of 22:43, 30 October 2019

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)
ML16081A285
Person / Time
Site: Salem  PSEG icon.png
Issue date: 11/20/2015
From: Todd Fish
Operations Branch I
To:
Public Service Enterprise Group
Shared Package
ML15135A303 List:
References
TAC U01910
Download: ML16081A285 (6)
v • d • e


Text

ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

...

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.- 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q

67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s 71 F 3

- -N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC - EVALUATED ON OP TEST.

SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s

ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

...

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.- 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q

67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s 71 F 3

- -N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC - EVALUATED ON OP TEST.

SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s

Retrieved from "https://kanterella.com/w/index.php?title=ML16081A285&oldid=2022071"