ML15293A556

2015-10 Draft Written Exam Comments

ML15293A556
Person / Time
Site:	South Texas
Issue date:	10/02/2015
From:	Vincent Gaddy Operations Branch IV
To:
References
Download: ML15293A556 (7)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 1

H 3

N S

2 H

3 B

S Used on 1999 NRC exam 3

H 2

B S

4 H

3 N

S 5

H 3

N S

6 H

2 B

S 7

F 2

N S

8 F

2 B

S Instructions (Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question a 1(easy) to 5 ( difficult); (questions with a difficulty between 2 and 4 are acceptable)

3.

Check the appropriate box if a psychometric flaw is identified:

Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc.).

T/F: The answer choices are a collection of unrelated true/false statements.

Cred. Dist: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

Partial: One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

1. / Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

Backward: The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only (K/A and license level mismatches are unacceptable)

6.

Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-401 Section D.2.f.

7.

Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any "U" Status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 9

H 3

N S

10 H

3 N

S 11 F

2 N

S 12 H

3 B

S Used on 1999 NRC exam 13 H

3 N

S 14 H

3 B

S 15 H

3 N

S 16 H

3 N

S 17 H

3 B

S 18 F

2 N

S 19 H

3 N

S 20 H

3 N

S 21 H

3 N

S 22 H

3 N

S 23 F

2 B

S 24 H

3 N

S 25 F

2 B

S

ES-401 3

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 26 F

3 N

S 27 H

3 B

S Used on 2010 NRC exam 28 F

2 N

S 29 H

3 N

S 30 H

3 M

S Used on 2009 NRC exam 31 H

3 B

S Used on 2009 NRC exam 32 H

3 B

S Used on 2003 NRC exam 33 H

3 N

S 34 F

2 N

S 35 F

2 N

S 36 H

3 B

S Used on 2009 NRC exam 37 H

3 N

S 38 H

2 B

S 39 H

3 B

S Used on 2007 NRC exam 40 F

3 B

S 41 H

3 N

S 42 H

4 N

S

ES-401 4

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 43 F

3 N

S 44 F

3 N

S 45 F

2 B

S Used on 2005 NRC exam 46 F

3 B

S Used on 2010 NRC exam 47 H

3 N

S 48 H

3 N

S 49 H

3 B

S 50 H

3 N

S 51 F

3 B

S Used on 2007 NRC exam 52 H

3 N

S 53 F

2 M

S 54 H

4 N

S 55 F

3 N

S 56 F

2 N

S 57 F

2 N

S 58 H

3 B

S Used on 2005 NRC exam 59 F

2 N

S

ES-401 5

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 60 H

3 N

S 61 H

3 B

S Used on 2009 NRC exam 62 F

2 N

S 63 F

2 B

S 64 H

3 N

S 65 H

4 N

S Used on 1995 NRC exam 66 F

3 N

S 67 F

2 N

S 68 F

2 N

S 69 F

2 N

S 70 F

4 N

S 71 F

2 N

S 72 H

3 M

S 73 F

2 N

S 74 H

3 N

S 75 F

2 N

S 76 H

4 B

S Used on 2001 NRC exam

ES-401 6

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 77 F

3 N

S 78 H

3 B

S Used on 2011 NRC exam 79 F

3 B

S 80 F

4 N

S 81 H

3 N

S 82 H

3 N

S 83 F

3 N

S 84 H

3 B

S 85 H

3 N

S 86 F

3 N

S 87 H

3 B

S Used on 2005 NRC exam 88 F

4 N

S 89 H

3 B

S Used on 2007 NRC exam 90 H

3 B

S Used on 2007 NRC exam 91 H

3 N

S 92 H

3 N

S 93 F

2 N

S

ES-401 7

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 94 H

3 B

S 95 H

3 N

S 96 F

2 N

S 97 F

2 N

S 98 F

3 N

S 99 H

3 N

S 100 F

3 B

S Used on the 1997 NRC exam

ES-401 Written Examination Review Worksheet Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 1

H 3

N S

2 H

3 B

S Used on 1999 NRC exam 3

H 2

B S

4 H

3 N

S 5

H 3

N S

6 H

2 B

S 7

F 2

N S

8 F

2 B

S Instructions (Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question a 1(easy) to 5 ( difficult); (questions with a difficulty between 2 and 4 are acceptable)

3.

Check the appropriate box if a psychometric flaw is identified:

Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc.).

T/F: The answer choices are a collection of unrelated true/false statements.

Cred. Dist: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

Partial: One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

1. / Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

Backward: The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only (K/A and license level mismatches are unacceptable)

6.

Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-401 Section D.2.f.

7.

Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any "U" Status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 9

H 3

N S

10 H

3 N

S 11 F

2 N

S 12 H

3 B

S Used on 1999 NRC exam 13 H

3 N

S 14 H

3 B

S 15 H

3 N

S 16 H

3 N

S 17 H

3 B

S 18 F

2 N

S 19 H

3 N

S 20 H

3 N

S 21 H

3 N

S 22 H

3 N

S 23 F

2 B

S 24 H

3 N

S 25 F

2 B

S

ES-401 3

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 26 F

3 N

S 27 H

3 B

S Used on 2010 NRC exam 28 F

2 N

S 29 H

3 N

S 30 H

3 M

S Used on 2009 NRC exam 31 H

3 B

S Used on 2009 NRC exam 32 H

3 B

S Used on 2003 NRC exam 33 H

3 N

S 34 F

2 N

S 35 F

2 N

S 36 H

3 B

S Used on 2009 NRC exam 37 H

3 N

S 38 H

2 B

S 39 H

3 B

S Used on 2007 NRC exam 40 F

3 B

S 41 H

3 N

S 42 H

4 N

S

ES-401 4

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 43 F

3 N

S 44 F

3 N

S 45 F

2 B

S Used on 2005 NRC exam 46 F

3 B

S Used on 2010 NRC exam 47 H

3 N

S 48 H

3 N

S 49 H

3 B

S 50 H

3 N

S 51 F

3 B

S Used on 2007 NRC exam 52 H

3 N

S 53 F

2 M

S 54 H

4 N

S 55 F

3 N

S 56 F

2 N

S 57 F

2 N

S 58 H

3 B

S Used on 2005 NRC exam 59 F

2 N

S

ES-401 5

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 60 H

3 N

S 61 H

3 B

S Used on 2009 NRC exam 62 F

2 N

S 63 F

2 B

S 64 H

3 N

S 65 H

4 N

S Used on 1995 NRC exam 66 F

3 N

S 67 F

2 N

S 68 F

2 N

S 69 F

2 N

S 70 F

4 N

S 71 F

2 N

S 72 H

3 M

S 73 F

2 N

S 74 H

3 N

S 75 F

2 N

S 76 H

4 B

S Used on 2001 NRC exam

ES-401 6

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 77 F

3 N

S 78 H

3 B

S Used on 2011 NRC exam 79 F

3 B

S 80 F

4 N

S 81 H

3 N

S 82 H

3 N

S 83 F

3 N

S 84 H

3 B

S 85 H

3 N

S 86 F

3 N

S 87 H

3 B

S Used on 2005 NRC exam 88 F

4 N

S 89 H

3 B

S Used on 2007 NRC exam 90 H

3 B

S Used on 2007 NRC exam 91 H

3 N

S 92 H

3 N

S 93 F

2 N

S

ES-401 7

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6. Source (B/ M / N)

7. Status (U /E /S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /

Units Back ward Q -

K/A SR O

Only 94 H

3 B

S 95 H

3 N

S 96 F

2 N

S 97 F

2 N

S 98 F

3 N

S 99 H

3 N

S 100 F

3 B

S Used on the 1997 NRC exam