ML17047A597

From kanterella
Jump to navigation Jump to search
NRC Comments on Draft Facility-Developed Written Exam (Folder 2)
ML17047A597
Person / Time
Site: Indian Point Entergy icon.png
Issue date: 10/03/2016
From: D'Antonio J
Operations Branch I
To:
Entergy Nuclear Indian Point 2
Shared Package
ML16076A436 List:
References
U01926
Download: ML17047A597 (13)
v • d • e


Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOO ~

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 1 3 s 2 3 s 3 3 x E Reasons for flowpath, not just what is it -fixed 4 3 s 5 3 s 6 3 s 7 3 E D implausible? Max speed, no cooling? - correct ans 8 3 s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Based on the reviewer's judgment, is the question as written (U)nacceptable , in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES=-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOO ~

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 9 3 x u "NTE" not credible, use wrong numbers, 2 vs 3 pumps -fixed 10 3 s Muu orny ma, c -nxea 11 3 E 12 3 s 13 3 s 14 3 x u "Reasons for" ka; ask about effect of air loss -ok 15 3 s 16 3 s Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
f. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOO -

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only -

17 x u A, C can cause the stated condition. Maybe "how do you depress, max or controlled and why?

18 3 s 19 3 s 20 3 s 21 3 E ALARM?

22 3 s 23 3 s 24 3 s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-4()1 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD .---

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 25 H 3 E Add SG levels 11.;orrect ans should be "c based on given power level; also contlict with 26 H 3 E AOP typo 27 F 3 s Explanation A?; Dis implausible 28 H 3 E stagger RCP numbers 29 H 2 s Effect of boration? How did it validate? - OK 30 H 3 s 31 F 3 s 32 H 3 s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified: -
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
f. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOO -

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 33 H 3 s 34 F 2 s Numbers in the stem? fixed 35 H 3 s 36 H 3 s 37 E 3 s 38 H 3 s 39 H 3 s 40 F 2 s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
f. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOO ~

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 41 F 2 s 42 H 2 E "an" addidtional 60 min dist a 43 H 3 s 44 H 3 x u You say "run back defeat in stem. Make IC 82%, man v auto, 2 v 3 AFW 45 H 3 s 46 H 3 s 47 H 3 s 48 H 3 s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
f. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD -

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 49 F 2 s 50 F 2 s 51 H 2 s 52 F 2 s 53 F 3 s B decreases 54 H 3 E Hi and Hi-Hi are subsets a, b say Hi AND Hi-Hi 55 F 2 E 56 H 3 s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
f. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Exam1nat1on Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOO ..---

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only ILOS "l,jl:l KV 57 F 3 E 58 H 3 s 59 F 2 s 60 H 2 s 61 F 2 loose E Better ka match 62 H 3 E Remove 2"d bullet, make one list 63 F 2 E B implausible, H2 02 in a, b reversed 64 H 3 no u replaced Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
f. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Exam1nat1on Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK (F/H)

LOO (1-5)

- Explanation Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO U/E/S Focus Dist. Link units ward KIA Only 65 H 2 s L

67 F 3 s 67 H 3 s 68 H 2 s 69 E Change to SI O/S based on validation 70 H 2 s 71 u Rewrite, either DLO or too hard, stem cues 72 H 2 s 73 H 3 s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other - 6. 7.

Q# LOK (F/H)

LOD (1-5)

Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 74 F 2 s 75 F 2 s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
f. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES401 Written Exam1nat1on Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK (F/H)

LOD -

(1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 76 u Forgot to replace Q fixed 77 H 3 s 78 H 3 s 79 u u Should be 76 fixed 80 H 3 s 81 H 3 E Procedure step/ procedure 82 H 2 83 H 3 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
f. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES=-401 Written Exam1nat1on Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD -

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 84 H 2 u RO level KA - fixed-85 H 3 s 86 H 3 s 87 H 3 s 88 H 3 s 89 H 3 s 90 H 3 s 90 H 3 s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • The answer choices are a collection of unrelated true/false statements.

The One or more distractors are is not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
  • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
  • The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
g. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD -

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only 91 H 3 s 92 H 3 s 92 H 3 s 93 H 3 s 94 F 1 u Improve LOD - replaced 95 H 2 s 96 H 3 s 97 F 3 s 98 H 2 s 99 F 2 s 1..::

100 H s

Retrieved from "https://kanterella.com/w/index.php?title=ML17047A597&oldid=2440520"