ML16081A285

From kanterella
Revision as of 22:43, 30 October 2019 by StriderTol (talk | contribs) (Created page by program invented by StriderTol)
Jump to navigation Jump to search

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)
ML16081A285
Person / Time
Site: Salem  PSEG icon.png
Issue date: 11/20/2015
From: Todd Fish
Operations Branch I
To:
Public Service Enterprise Group
Shared Package
ML15135A303 List:
References
TAC U01910
Download: ML16081A285 (6)
v • d • e


Text

ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

...

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.- 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q

67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s 71 F 3

- -N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC - EVALUATED ON OP TEST.

SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s

ES-401 sAL~ ~l Written Examination Review Worksheet 0/;Cf t"\'&~R )DfS Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 F 2 B s 3 H 1 N u GFE; LOD=1; REPLACED KA, WROTE NEW Q 4 F 2 B s 5 F 2 N s 6 F 3 N s 7 H 3 B s 8 H 2 M s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

...

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.- 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 N s 10 H 3 N s 11 H 3 B s 12 H 3 B s 13 F 2 N s 14 H 3 N s 15 H 3 B s 16 F 2 B s 17 F 2 N s 18 F 3 N s 19 F 3 B s 20 H 3 M s 21 F 3 B s 22 H 3 B s 23 F 3 B s 24 F 3 B s 25 H 3 B s 26 F 2 B s 27 F 2 M s 28 F 2 M s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 B s 30 F 2 M s 31 F 2 M s 32 F 2 M s 33 F 2 N s 34 H 3 x B u NON-DISCRIMINATING Q; WROTE NEW Q 35 H 3 N s 36 H 3 B s 37 H 3 N s 38 H 2 N' s 39 F 2 N s 40 H 3 B s 41 F 2 N s 42 H 3 B s 43 H 3 M s 44 F 2 x N u NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q 45 F 2 B s

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 H 3 N s 47 H 3 N s 48 F 2 B s 49 H 3 M s 50 F 3 B s 51 H 3 N s 52 H 3 N s 53 F 2 B s 54 H 3 N s 55 H 3 B s 56 H 3 B s 57 F 2 B s 58 H 3 B s 59 H N s 60 H B s 61 H B s 62 H B s 63 F B s 64 H N s 65 H N_ s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 1 x N u LOD=1; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEW Q

67 H 3 B s 68 F 2 x B u MINUTIA; NON-DISCRIMINATING Q; REPLACED KA, WROTE NEWQ 69 F 2 x N u MINUTIA; WROTE NEW Q 70 H 2 M s 71 F 3

- -N s 72 F 2 B s 73 F 2 B s 74 H 3 N s 75 F 2 B s 76 H 3 N s 77 F 4 N s 78 H 4 M s 79 H 4 N s 80 H 3 M s 81 H 3 N s 82 H 3 B s 83 H 4 B s 84 F 3 B s 85 F 3 B s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 4 M s 87 H 4 B s 88 H 3 B s 89 H 4 N s 90 H 4 N s 91 H 4 N s 92 H 4 N s 93 H 3 N s 94 F 3 B s 95 H 3 B s 96 F 3 N s 97 H 3 B u OVER-SAMPLED TOPIC - EVALUATED ON OP TEST.

SELECTED NEW KA, WROTE NEW Q 98 F 3 B s 99 H 4 N s 100 F 3 N s

Retrieved from "https://kanterella.com/w/index.php?title=ML16081A285&oldid=2022071"