ML15293A556

2015-10 Draft Written Exam Comments

ML15293A556
Person / Time
Site:	South Texas
Issue date:	10/02/2015
From:	Vincent Gaddy Operations Branch IV
To:
References
Download: ML15293A556 (7)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 1 H 3 N S 2 H 3 B S Used on 1999 NRC exam 3 H 2 B S 4 H 3 N S 5 H 3 N S 6 H 2 B S 7 F 2 N S 8 F 2 B S Instructions (Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question a 1(easy) to 5 ( difficult); (questions with a difficulty between 2 and 4 are acceptable)

3. Check the appropriate box if a psychometric flaw is identified:

• Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc.).

• T/F: The answer choices are a collection of unrelated true/false statements.

• Cred. Dist: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

• Partial: One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

• Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

• Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

• # / Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• Backward: The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only (K/A and license level mismatches are unacceptable)

6. Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" Status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 9 H 3 N S 10 H 3 N S 11 F 2 N S 12 H 3 B S Used on 1999 NRC exam 13 H 3 N S 14 H 3 B S 15 H 3 N S 16 H 3 N S 17 H 3 B S 18 F 2 N S 19 H 3 N S 20 H 3 N S 21 H 3 N S 22 H 3 N S 23 F 2 B S 24 H 3 N S 25 F 2 B S

ES-401 3 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 26 F 3 N S 27 H 3 B S Used on 2010 NRC exam 28 F 2 N S 29 H 3 N S 30 H 3 M S Used on 2009 NRC exam 31 H 3 B S Used on 2009 NRC exam 32 H 3 B S Used on 2003 NRC exam 33 H 3 N S 34 F 2 N S 35 F 2 N S 36 H 3 B S Used on 2009 NRC exam 37 H 3 N S 38 H 2 B S 39 H 3 B S Used on 2007 NRC exam 40 F 3 B S 41 H 3 N S 42 H 4 N S

ES-401 4 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 43 F 3 N S 44 F 3 N S 45 F 2 B S Used on 2005 NRC exam 46 F 3 B S Used on 2010 NRC exam 47 H 3 N S 48 H 3 N S 49 H 3 B S 50 H 3 N S 51 F 3 B S Used on 2007 NRC exam 52 H 3 N S 53 F 2 M S 54 H 4 N S 55 F 3 N S 56 F 2 N S 57 F 2 N S 58 H 3 B S Used on 2005 NRC exam 59 F 2 N S

ES-401 5 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 60 H 3 N S 61 H 3 B S Used on 2009 NRC exam 62 F 2 N S 63 F 2 B S 64 H 3 N S 65 H 4 N S Used on 1995 NRC exam 66 F 3 N S 67 F 2 N S 68 F 2 N S 69 F 2 N S 70 F 4 N S 71 F 2 N S 72 H 3 M S 73 F 2 N S 74 H 3 N S 75 F 2 N S 76 H 4 B S Used on 2001 NRC exam

ES-401 6 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 77 F 3 N S 78 H 3 B S Used on 2011 NRC exam 79 F 3 B S 80 F 4 N S 81 H 3 N S 82 H 3 N S 83 F 3 N S 84 H 3 B S 85 H 3 N S 86 F 3 N S 87 H 3 B S Used on 2005 NRC exam 88 F 4 N S 89 H 3 B S Used on 2007 NRC exam 90 H 3 B S Used on 2007 NRC exam 91 H 3 N S 92 H 3 N S 93 F 2 N S

ES-401 7 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 94 H 3 B S 95 H 3 N S 96 F 2 N S 97 F 2 N S 98 F 3 N S 99 H 3 N S 100 F 3 B S Used on the 1997 NRC exam

ES-401 Written Examination Review Worksheet Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 1 H 3 N S 2 H 3 B S Used on 1999 NRC exam 3 H 2 B S 4 H 3 N S 5 H 3 N S 6 H 2 B S 7 F 2 N S 8 F 2 B S Instructions (Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question a 1(easy) to 5 ( difficult); (questions with a difficulty between 2 and 4 are acceptable)

3. Check the appropriate box if a psychometric flaw is identified:

• Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc.).

• T/F: The answer choices are a collection of unrelated true/false statements.

• Cred. Dist: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

• Partial: One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

• Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

• Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

• # / Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• Backward: The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only (K/A and license level mismatches are unacceptable)

6. Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" Status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 9 H 3 N S 10 H 3 N S 11 F 2 N S 12 H 3 B S Used on 1999 NRC exam 13 H 3 N S 14 H 3 B S 15 H 3 N S 16 H 3 N S 17 H 3 B S 18 F 2 N S 19 H 3 N S 20 H 3 N S 21 H 3 N S 22 H 3 N S 23 F 2 B S 24 H 3 N S 25 F 2 B S

ES-401 3 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 26 F 3 N S 27 H 3 B S Used on 2010 NRC exam 28 F 2 N S 29 H 3 N S 30 H 3 M S Used on 2009 NRC exam 31 H 3 B S Used on 2009 NRC exam 32 H 3 B S Used on 2003 NRC exam 33 H 3 N S 34 F 2 N S 35 F 2 N S 36 H 3 B S Used on 2009 NRC exam 37 H 3 N S 38 H 2 B S 39 H 3 B S Used on 2007 NRC exam 40 F 3 B S 41 H 3 N S 42 H 4 N S

ES-401 4 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 43 F 3 N S 44 F 3 N S 45 F 2 B S Used on 2005 NRC exam 46 F 3 B S Used on 2010 NRC exam 47 H 3 N S 48 H 3 N S 49 H 3 B S 50 H 3 N S 51 F 3 B S Used on 2007 NRC exam 52 H 3 N S 53 F 2 M S 54 H 4 N S 55 F 3 N S 56 F 2 N S 57 F 2 N S 58 H 3 B S Used on 2005 NRC exam 59 F 2 N S

ES-401 5 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 60 H 3 N S 61 H 3 B S Used on 2009 NRC exam 62 F 2 N S 63 F 2 B S 64 H 3 N S 65 H 4 N S Used on 1995 NRC exam 66 F 3 N S 67 F 2 N S 68 F 2 N S 69 F 2 N S 70 F 4 N S 71 F 2 N S 72 H 3 M S 73 F 2 N S 74 H 3 N S 75 F 2 N S 76 H 4 B S Used on 2001 NRC exam

ES-401 6 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 77 F 3 N S 78 H 3 B S Used on 2011 NRC exam 79 F 3 B S 80 F 4 N S 81 H 3 N S 82 H 3 N S 83 F 3 N S 84 H 3 B S 85 H 3 N S 86 F 3 N S 87 H 3 B S Used on 2005 NRC exam 88 F 4 N S 89 H 3 B S Used on 2007 NRC exam 90 H 3 B S Used on 2007 NRC exam 91 H 3 N S 92 H 3 N S 93 F 2 N S

ES-401 7 Form ES-401-9

3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. 2.

6. Source 7. Status Q LOK LOD SR 8. Explanation Stem Cred. Job- #/ Back Q- (B/ M / N) (U /E /S)

(F/H) (1-5) Cues T/F Partial Minutia O Focus Dist Link Units ward K/A Only 94 H 3 B S 95 H 3 N S 96 F 2 N S 97 F 2 N S 98 F 3 N S 99 H 3 N S 100 F 3 B S Used on the 1997 NRC exam