ML15110A382: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
 
Line 19: Line 19:


=Text=
=Text=
{{#Wiki_filter:ES-401 Written Examination Review Worksheet Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 2 B s 2 H 2 N s 3 F 2 M s 4 F 2 N s 5 H 3 B s 6 H 3 B s 7 H 2 M E pump failing is window dressing; deleted bullet 8 H 3 B s Instructions
{{#Wiki_filter:ES-401                                                             Written Examination Review Worksheet                                                                     Form ES-401-9
: 1.         2.           3. Psychometric Flaws               4. Job Content Flaws         5. Other     6. 7.                                     8.
Q#   LOK       LOO (F/H)       (1-5)   Stem Cues         T/F   Cred. Partial Job- Minutia #/         Back-   Q= SRO B/M/N U/E/S                                     Explanation Focus                     Dist.         Link           units ward       KIA Only 1   H           2                                                                                           B     s 2     H           2                                                                                           N     s 3     F           2                                                                                           M     s 4     F           2                                                                                           N     s 5     H           3                                                                                           B     s 6     H           3                                                                                           B     s 7     H           2                                                                                           M     E     pump failing is window dressing; deleted bullet 8     H           3                                                                                           B     s Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOO) of each question using a 1 -5 (easy -difficult) rating scale (questions in the 2 -4 range are acceptable).
: 1.           Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
: 3. Check the appropriate box if a psychometric flaw is identified: . The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) . . The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) . . The answer choices are a collection of unrelated true/false statements . . The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable . . One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) . 4. Check the appropriate box if a job content error is identified: . The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) . . The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) . . The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) . . The question requires reverse logic or application compared to the job requirements . 5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
: 2.           Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 3.           Check the appropriate box if a psychometric flaw is identified:
: 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
            ...         The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .
ES-401 2 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #I Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 H 3 N s 10 H 3 B s 11 H 3 N s 12 H 3 M s 13 F 2 N s 14 H 3 N s 15 H 3 N s 16 H 3 N E Changed 'continues to degrade' to 'continues to rise' 17 H 3 M s 18 H 3 M s 19 F 1 N u LOD=1 ; Q replaced 20 F 2 N s 21 F 3 M s 22 H 3 B s 23 H 4 N s 24 H 3 N s 25 H 3 M s 26 H 3 N s 27 H 3 B s 28 F 2 N s 29 H 3 M s 30 F 2 M s
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 31 H 3 M s 32 F 3 N s 33 H 3 B s 34 H 3 N s 35 F 3 N s 36 H 3 N s 37 H 3 N s 38 F 2 N s 39 H 3 B s 40 F 3 B s 41 H 3 B s 42 F 3 M s 43 F 2 B s 44 H 3 B s 45 F 2 N s 46 F 3 N s 47 F 2 B s 48 H 3 N s 49 H 3 B s 50 F 3 B s 51 H 3 N s 52 H 3 M s 53 F 2 N s 54 F 2 N s
          ..            The answer choices are a collection of unrelated true/false statements .
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #I Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 55 H 4 B s 56 F 3 B s 57 H 3 M s 58 H 4 N s 59 H 3 M s 60 H 3 B s 61 F 2 N s 62 H 3 N s 63 F 3 M s 64 H 3 N s 65 H 3 B s 66 F 2 N u Better suited for op test venue; also overlaps with op test. Replaced Q. 67 F 1 N u Replaced KA and wrote new Q 68 F 2 N s 69 F 2 B s 70 F 2 N s 71 F 2 M s 72 F 2 N s 73 F 2 N s 74 F 2 B s 75 H 1 B u LOD = 1; license level knowledge not needed; replaced Q 76 H 2 B s 77 H 3 M s
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .
: 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 78 H 3 B s 79 H 3 B s 80 F 3 N s 81 H 3 N s 82 H 3 B s 83 H 3 N s 84 H 3 N s 85 H 4 N s 86 H 3 M s 87 H 3 N s 88 H 2 M s 89 H 3 M s 90 H 3 x M E Choice A not clearly wrong. Changed Q to ask, "What is the most restrictive TS action requirement?" 91 H 3 M s 92 H 3 N s 93 H 3 N s 94 F 3 N s 95 F 2 N s 96 H 2 B s 97 H 4 M s 98 F 3 N s 99 F 2 B s 100 H 2 N s}}
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4.
          .. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .
            ..           The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .
The question requires reverse logic or application compared to the job requirements .
: 5.         Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
: 6.           Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
: 7.           Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 8.       At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
 
ES-401                                                                       2                                                                   Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                   8.
Q#   LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #I Back-   Q=   SRO B/M/N U/E/S                           Explanation Focus             Dist.         Link         units ward KIA Only 9   H     3                                                                         N     s 10   H     3                                                                         B     s 11   H     3                                                                         N     s 12   H     3                                                                         M     s 13   F     2                                                                         N     s 14   H     3                                                                         N     s 15   H     3                                                                         N     s 16   H     3                                                                         N     E   Changed 'continues to degrade' to 'continues to rise' 17   H     3                                                                         M     s 18   H     3                                                                         M     s 19   F     1                                                                         N     u   LOD=1 ; Q replaced 20   F     2                                                                         N     s 21   F     3                                                                         M     s 22   H     3                                                                         B   s 23   H     4                                                                         N     s 24   H     3                                                                         N     s 25   H     3                                                                         M     s 26   H     3                                                                         N   s 27   H     3                                                                         B     s 28   F     2                                                                         N     s 29   H     3                                                                         M   s 30   F     2                                                                         M     s
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other 6. 7.     8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S Explanation Focus             Dist.         Link         units ward KIA Only 31   H     3                                                                       M   s 32   F     3                                                                       N   s 33   H     3                                                                       B   s 34   H     3                                                                       N   s 35   F     3                                                                       N   s 36   H     3                                                                       N   s 37   H     3                                                                       N   s 38   F     2                                                                       N   s 39   H     3                                                                       B   s 40   F     3                                                                       B   s 41   H     3                                                                       B   s 42   F     3                                                                       M   s 43   F     2                                                                       B   s 44   H     3                                                                       B   s 45   F     2                                                                       N   s 46   F     3                                                                       N   s 47   F     2                                                                       B   s 48   H     3                                                                       N   s 49   H     3                                                                       B   s 50   F     3                                                                       B   s 51   H     3                                                                       N   s 52   H     3                                                                       M   s 53   F     2                                                                       N   s 54   F     2                                                                       N   s
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.                                     8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #I     Back- Q=   SRO B/M/N U/E/S                               Explanation Focus             Dist.         Link         units ward   KIA Only 55   H     4                                                                           B     s 56   F     3                                                                           B     s 57   H     3                                                                           M     s 58   H     4                                                                           N     s 59   H     3                                                                           M     s 60   H     3                                                                           B     s 61   F     2                                                                           N     s 62   H     3                                                                           N     s 63   F     3                                                                           M     s 64   H     3                                                                           N     s 65   H     3                                                                           B     s 66   F     2                                                                           N     u   Better suited for op test venue; also overlaps with op test. Replaced Q.
67   F     1                                                                           N   u   Replaced KA and wrote new Q 68   F     2                                                                           N   s 69   F     2                                                                           B   s 70   F     2                                                                           N   s 71   F     2                                                                           M   s 72   F     2                                                                           N   s 73   F     2                                                                           N   s 74   F     2                                                                           B   s 75   H     1                                                                           B   u   LOD = 1; license level knowledge not needed; replaced Q 76   H     2                                                                           B   s 77   H     3                                                                           M   s
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws     5. Other 6. 7.                                   8.
Q#   LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/     Back- Q= SRO B/M/N U/E/S                               Explanation Focus             Dist.         Link         units ward   KIA Only 78   H     3                                                                         B   s 79   H     3                                                                         B   s 80   F     3                                                                         N   s 81   H     3                                                                         N   s 82   H     3                                                                         B   s 83   H     3                                                                         N   s 84   H     3                                                                         N   s 85   H     4                                                                         N   s 86   H     3                                                                         M   s 87   H     3                                                                         N   s 88   H     2                                                                         M   s 89   H     3                                                                         M   s 90   H     3                       x                                               M   E   Choice A not clearly wrong. Changed Q to ask, "What is the most restrictive TS action requirement?"
91   H     3                                                                         M   s 92   H     3                                                                         N   s 93   H     3                                                                         N   s 94   F     3                                                                         N   s 95   F     2                                                                         N   s 96   H     2                                                                         B   s 97   H     4                                                                         M   s 98   F     3                                                                         N   s 99   F     2                                                                         B   s 100   H     2                                                                         N   s}}

Latest revision as of 13:48, 31 October 2019

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)
ML15110A382
Person / Time
Site: Nine Mile Point Constellation icon.png
Issue date: 03/01/2015
From: Todd Fish
Operations Branch I
To: Isham P
Exelon Generation Co
Shared Package
ML14238A301 List:
References
TAC U01905
Download: ML15110A382 (5)
v • d • e


Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 2 B s 2 H 2 N s 3 F 2 M s 4 F 2 N s 5 H 3 B s 6 H 3 B s 7 H 2 M E pump failing is window dressing; deleted bullet 8 H 3 B s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

... The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

.. The answer choices are a collection of unrelated true/false statements .

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

.. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 H 3 N s 10 H 3 B s 11 H 3 N s 12 H 3 M s 13 F 2 N s 14 H 3 N s 15 H 3 N s 16 H 3 N E Changed 'continues to degrade' to 'continues to rise' 17 H 3 M s 18 H 3 M s 19 F 1 N u LOD=1 ; Q replaced 20 F 2 N s 21 F 3 M s 22 H 3 B s 23 H 4 N s 24 H 3 N s 25 H 3 M s 26 H 3 N s 27 H 3 B s 28 F 2 N s 29 H 3 M s 30 F 2 M s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 31 H 3 M s 32 F 3 N s 33 H 3 B s 34 H 3 N s 35 F 3 N s 36 H 3 N s 37 H 3 N s 38 F 2 N s 39 H 3 B s 40 F 3 B s 41 H 3 B s 42 F 3 M s 43 F 2 B s 44 H 3 B s 45 F 2 N s 46 F 3 N s 47 F 2 B s 48 H 3 N s 49 H 3 B s 50 F 3 B s 51 H 3 N s 52 H 3 M s 53 F 2 N s 54 F 2 N s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 55 H 4 B s 56 F 3 B s 57 H 3 M s 58 H 4 N s 59 H 3 M s 60 H 3 B s 61 F 2 N s 62 H 3 N s 63 F 3 M s 64 H 3 N s 65 H 3 B s 66 F 2 N u Better suited for op test venue; also overlaps with op test. Replaced Q.

67 F 1 N u Replaced KA and wrote new Q 68 F 2 N s 69 F 2 B s 70 F 2 N s 71 F 2 M s 72 F 2 N s 73 F 2 N s 74 F 2 B s 75 H 1 B u LOD = 1; license level knowledge not needed; replaced Q 76 H 2 B s 77 H 3 M s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 78 H 3 B s 79 H 3 B s 80 F 3 N s 81 H 3 N s 82 H 3 B s 83 H 3 N s 84 H 3 N s 85 H 4 N s 86 H 3 M s 87 H 3 N s 88 H 2 M s 89 H 3 M s 90 H 3 x M E Choice A not clearly wrong. Changed Q to ask, "What is the most restrictive TS action requirement?"

91 H 3 M s 92 H 3 N s 93 H 3 N s 94 F 3 N s 95 F 2 N s 96 H 2 B s 97 H 4 M s 98 F 3 N s 99 F 2 B s 100 H 2 N s

Retrieved from "https://kanterella.com/w/index.php?title=ML15110A382&oldid=2034644"