ML022590294

From kanterella
Revision as of 14:28, 25 March 2020 by StriderTol (talk | contribs) (StriderTol Bot insert)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
Draft - NRC Comments
ML022590294
Person / Time
Site: Millstone Dominion icon.png
Issue date: 08/12/2002
From: Wilson M
Dominion Nuclear Connecticut
To: Conte R
NRC/RGN-I/DRS/OSB
Conte R
References
05000423/02-301 05000423/02-301
Download: ML022590294 (12)
v • d • e


Text

ES-401 Written Examination Form ES-401-9 Review Worksheet Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Flaws Stem Cues 1

T/F

4. Job Content Flaws Cred. Partial Job- Minutia #/ B
5. Other Q=

6.

SROU/E/S 1 7.

Explanation

-2 Focus _ _ Dist. _ Link units ward K/AOny _

I F 2 S 2 H 3 _ E USE WOULD"AND "WOULD NOT", BOLD F 2 S H 2  ? S NAT CIRC/REFLUX BOILING?

H 2 X S H 2 X S H 2 X Is H 2S H 12 S 0H 12 X S1 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onlv (K/A and license level mismatches are unacceptable).

I

6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met). I NUREG-1021, Revision 8, Supplement 1 44 of 46

ES-401 Written Examination Form ES-401-9 Review Worksheet J

Q# LOK 1 2.

LODJj

3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation LFocus __Il Dist. l Link l units ward K/A I OnlyI 11 F 2 1 s 12 H 2l 13 F 2 S 4 F 2 E WHY IS " IMPLAUSIBLE?

15 H 2 X S 6 F 2 X S 7 F 2 = iS 8 F 2 X X S 19 F 2 X S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.
  • More than one distractor is not credible.
  • One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4. Check the appropriate box if ajob content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onyv (K/A and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met). I NUREG-1021, Revision 8, Supplement 1 44 of 46

ES-401 E0Written Examination Form ES-401-9 Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOI (F/H) (1-5) Stem lCues T/F Cred. lPartial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus j [j Dist. j Link j units ward K/A JOnly 21 2 _ S 22 2 _ E REWORD "CREW HAS VERIFIED" 23 2  ? S THERE IS PROBABLY A RULES OF USEAGE KA 24 _ = = is 25 2 X S/E DISTRACTOR 'D" 26 2 S 27 2 S 82 S 29 2 mm U 302 S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

  • One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onlv (K/A and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46 Es4O Wrte Exmnto For _S40 -9.. _..

ES-401 Written Examination Form ES-401-9

Review Worksheet

[ Q#

LOK 2.

LOD

3. Psychometric Flaws
4. Job Content Flaws 5. Other 6. 7.

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO UIEIS Explanation Focus _ Dist. Link _ units ward K/A Only 1 H 2 x E HOW DO YOU KNOW CHANGES BIG ENOUGH FOR TRIP 2 H 2 X S 33 F 2 X S 4 H 2 x X lE NO REFERENCES? FROM MEMORY 5 H 2 S 6 F 2 E REWORD D - MINIMIZE LOSS, F&B IS A SMALL LOCA i7 F 2 x S 8 H 2 X IS 9 F 2 _5 X l l F 2 X S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a I - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

  • More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46 ES-401 Written Examination Form ES-401-9 Review Worksheet

11. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

Q# LOK LOD 1 1 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation

_ Focus j j Dist. _ Link Junits ward K/A OnlyI I 11 H 2 = S 2 F 2 X S 3 F 2 MM U EK1.2,NOT1.1 l IH 2 is 1H 12 X E B NOT CREDIBLE t6 H 2 S

~7 H 2 S 18 H 2 X S

~9 IF 2 X S

~O IF 12 S Instructions (Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

I The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc). I The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if ajob content error is identified:

. The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46 ES-401 Written Examination Form ES-401-9 Review Worksheet

11. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 16. 7.

Q# LOK LOD _ [

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO UIEIS Explanation I Focus Dist. L ink nits ward K/A Only 1 F 1 U TOO BASIC, NON DISC 2 F 2 S 3 H 2 X S 54 H 2 _ X S 55 F 2 S 56 H 2 S 57 H 2 S 8 H 2 X E D- SD IS PROHIBITED 59 H 2 X S 0H 2 X S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

I* The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

I* The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

I* The answer choices are a collection of unrelated true/false statements.

I* More than one distractor is not credible.

. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

I* The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

I . The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onlv (K/A and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U"ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46 I ES6-401 Written Examination Form ES-401-9 I Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LODl (F/H) (1-5) Stem Cues T/IF Cred. Partial Job- Minutia #/ Back- Q= SRO UIEIS Explanation 1 Focus Dist. Link units ward K/A Only 31 F 2_____l 32 H 2 X X E AOP 3575 MANDATORY?

3 H 2 NO U 55.43?

4 H 2 _U/E SIMILAR TO ANOTHER Q
5 H 2 X S 6 H 2 =X I X X U MODE 3 IS HSBY 37 H 2 38 H 2 39 F 2 X 70 F 2 _

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a I - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements. I More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if ajob content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onl, (K/A and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any GU" ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46 ES-401 Written Examination Form ES-401-9 I Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD _

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #1/ Back- 0= SROU/ES Explanation Focus Dist. LiunitadK/ Only 71 2 _ _ S 7222 s 732 X S 74 2 = =S 75 12 1 s r62 S 772 S 682 S 9 2E 2 MEN ENTER, 1 STANDS BY AS SAFETY MAN Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (Flundamrental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

  • One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4. Check the ap~propriate box if ajob content error is identified:
  • Te question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1 021, Revision 8, Supplement 1 44 of 46 LE-401 Written Examination Form ES-401-9 I Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD l (F/H) (1-5) Stem Cues T/FlCred. Partial Job- IMinutia#/ #SRO Back- U/E/S Explanation Focus Dist. _ Link units ward K/AOnly l 311 XK U CONFUSING STEM, 1ST STAGE PA ISNORMAL FOR A POWER 32 3 X =

332 5 34 12 1 5 2 X X S 36 2 X X S TS 972 X S T Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

I* The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

I More than one distractor is not credible.

. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if ajob content error is identified:

I

  • The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U"ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46

ES-401 Written Examination Form ES-401-9 Review Worksheet

11. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK0 _ -- l (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation i Focus Dist. l Link unitsI ward K/AI Onlyl 31 F 2 S 2 F 2 X s S

3 F 2 _ x S 4 F 2 S 5 F 2 x x E? SIMILAR TO EARLIER Q 6 F 2 x S 7 F 2 X U C,D 8 F 2 = X lS Is 9 F 2 X X E A NOT CREDIBLE 100 F_]2 XS Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

  • The answer choices are a collection of unrelated true/false statements.
  • More than one distractor is not credible.
  • One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met). I NUREG-1021, Revision 8, Supplement 1 44 of 46

ES-401 Written Examination Form ES-401-9 Review Worksheet 1 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK lLOD_

(F/H) (1-5) Stem Cues T/F Cred. JPartial Job- Minutia #1 Back- Q SRO UIEIS Explanation

[ I Focus Dist. Link units ward K/A On y Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable). I
3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

  • The answer choices are a collection of unrelated true/false statements.

I More than one distractor is not credible.

. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

I The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

I. The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

  • . The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onlv (K/A and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

Q#(FH)(1-5) Stem Cues T/F l Cred. Partial Job- Minutia #/ l Back- Q= ISRO UIEIS Explanation 1- Focus j __ ZDist. l Link - [units ward K/A I Only n_ _ _ _ _ _ _ _ _

45 of 46 NUREG-1021, Revision 8, Supplement 1

Retrieved from "https://kanterella.com/w/index.php?title=ML022590294&oldid=2691970"