ML12059A370

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML12059A370
Person / Time
Site:	Indian Point
Issue date:	02/13/2012
From:	Division of Reactor Safety I
To:	NRC Region 1
Jackson D
Shared Package
ML112350689	List: IR 05000247/2012301 ML112411212 ML120300321 ML12059A360 ML12059A366 ML12059A370 ML12059A434 ML12059A435 ML12059A443 ML12059A447 ML12059A449 ML12072A227 ML12072A249 ML12172A061
References
U01844, 50-247/12-301, ES-401-9
Download: ML12059A370 (13)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9 Q#

1.

LOK I 2.

LOO

3. Psychometric Flaws 4. Job Content Flaws 8.

(FfH) (1-5) Back Q= Explanation ward KIA I Only F 2 2 F 2 S 3 F 2 S 4 H 2 S 5 H 3 S 6 H 2 S 7 H 2 S H I I 2 Instructions I IS

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

10. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

11. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

12. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (I.e., clues. specific determiners. phrasing. length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired. more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (I.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g .. panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are deSignated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter qUestion source: (B)ank. (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.

7. Based on the reviewer'S judgment, is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum. explain any *U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. B.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back:- Q= SRO B/M/N UtE/S Explanation Focus Oist Link units ward KIA Only 9 F 2 no U KAmm 10 H 2 S 11 F 2 S 12 F 2 - S' Back: asswards ka sort of mm 13 H 2 S 14 F 2 S 15 H 2 S 16 H 4 S Fair RO level from memory?-Y Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the follOwing concepts.]

13. Enter the level of knowledge (LOK) of each question as either {F)undamental or (H)igher cognitive level.

14. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy difficult) rating scale (questions in the 2 4 range are acceptable).

15. Check: the appropriate box if a psychometric flaw is identified:

• • The stem lack:s sufficient focus to elicit the correct answer (e.g .* unclear intent. more information is needed. or too much needless information).

The stem or distractors contain cues (Le.* clues. specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated truelfalse statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

4.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

Check: the appropriate box if a job content error is identified:

• • The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

• • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and tnose that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank. (M)odified. or (N)ew. Check: that (M)odified questions meet criteria of ES-401 Section O.2.f.

7. Based on the reviewer'S judgment, is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E)ditorial enhancement. or (S)atisfactory?

B. At a minimum. explain any "U" ratings (e.g .. how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Oist. Link units ward KIA Only 17 H 2 S 18 F 2 S 19 H 3 S 20 H 2 S 21 F 2 S 22 F 2 S 23 H 3 S RO level?

24 F 2 S Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

16. Enter the level of knowledge (LOK) of each question as either (F}undamental or (H}igher cognitive level.

17. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

18. Check the appropriate box if a psychometric flaw is identified:

• The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• • The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

• • The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B}ank, (M)odified, or (N}ew. Check that (M}odified questions meet criteria of ES-401 Section O.2.f.

7. Based on the reviewer'S judgment, is the question as written (U}nsatisfactory (requiring repair or replacement), in need of (E}ditorial enhancement, or (S}atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LaD ---

(F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #I Back- Q= SRO B/MIN UlE/S Explanation Focus Dist. Link units ward KIA Only 25 H 3 S RO fair?

26 H 3 S 27 H 3 S RO?

28 F 2 S 29 F 3 S 30 F 2 S 31 H 3 E Add bullet - plant parameters on program prior to trip Move 520 deg bullet to after turbine trip for clarity 32 F 2 S Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

19. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

20. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy difficult) rating scale (questions in the 2 - 4 range are acceptable).

21. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

I The stem or distractors contain cues (i.e., clues. specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

• The question is not linked to the job requirements (I.e.* the question has a valid KIA but, as written, is not operational in content).

• The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).

• • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-onfy (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment. is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (FtH) (1-5) Stem Cues TtF Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 33 H 2 S 34 H 2 U Technically incorrect. Lvi down unless head bubble.

35 F 2 S 36 F 2 X E Where did pump trip come from?

37 H 2 S 38 H 3 S 39 H 2 S 4Q F 2 S Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

22. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

23. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

24. Check the appropriate box if a psychometric flaw is identified:

• • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).

• • The answer choices are a collection of unrelated true/false statements.

The distraclors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

4.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

Check the appropriate box if a job content error is identified:

• • The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

• The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

5.

6.

• The question requires reverse logic or application compared to the job requirements.

Check g,..,o", thaI"" ",mp!"" Ie< ""'Ionn~", ,.;th the appro"'" I<JA and thooe thaI a'" do,,,nato" Soo"nly (I<JA eo" I;"""e le,.1 mf""a""" ace Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

'",""PtaJ

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U' ratings (e.g., how the Appendix B psychometric attributes are not being met).

- .... -~ ..... _. ... --.-~.-.- ..-.----.. ..... -~.-.--.- ... - . -..-- ...... -~ ... _._ _.

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

00 LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 41 F 2 S 42 H 4 S More specific than KA calls for 43 H 3 S 44 H 2 S 45 F 3 S 46 H 3 S 47 F 2 S 48 H 2 S Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

25. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

26. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

27. Check the appropriate box if a psychometric flaw is identified:

• • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (I.e., clues, specific determiners, phrasing, length, etc).

• • The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

4.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

Check the appropriate box if a job content error is identified:

• The question is not linked to the job requirements (I.e., the question has a valid KIA but, as written, is not operational in content).

• • The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

5.

• The question requires reverse logic or application compared to the job requirements.

Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)dltorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U' ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cree!. Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Oist. Link units ward KIA Only 49 F 2 S 50 H 2 E o not plausible - "droop model iso mode" maybe?

51 F 2 S 52 F 2 S 53 H 2 S 54 F 2 S 55 H 2 S 56 H 3 S Instructions

[Refer to Section 0 of ES401 and Appendix B for additional information regarding each of the following concepts.]

28. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

29. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

30. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed. or too much needless information).

The stem or distractors contain cues (Le.* clues. specific determiners. phrasing. length, etc).

The answer choices are a collection of unrelated true/false statements.

• • The distractors are not credible; single implausible distractors should be repaired. more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (Le .. the question has a valid KIA but, as written. is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le.* it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are deSignated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank. (M)odified. or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.

7. Based on the reviewer's jUdgment, is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E)ditoriaJ enhancement, or (S)atisfactory?

n

8. At a minimum, explain any 'U ratings (e.g .* how the Appendix B psychometric attributes are not being met).

~ ... - . - . - . -... --.-~ ... -.----.--.~ ...... -- ..... - . - -_._---- - - -

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 57 F 2 S 58 H 2 U KA mm - this ka is looking for RGHX temp 59 H 2 S 60 F 2 S 61 H 2 S 62 F 2 E 2809 vs 3300 - 3300 seems too obviously high 63 H 2 S

"-~

64 H 2 no no U This is just guessing at cause of alarm. Reselect KA Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional infonmation regarding each of the following concepts.]

31. Enter the level of knowiedge (lOK) of each question as either (F)undamental or (H)igher cognitive level.

32. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

33. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more infonmation is needed, or too much needless infonmation).

The stem or distractors contain cues (i.e., clues, specific detenminers, phrasing, length, etc).

The answer choices are a collection of unrelated truelfalse statements.

• • The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for confonmance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (Blank, (M)odifjed, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E)ditorial enhancement, or (S)atisfactory?

n B. At a minimum, explain any *U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (FtH) (1-5) Stem Cues TfF Credo Partial Job- Minutia #1 Back- Q= SRO B/MtN UtEfS Explanation Focus Dist. Link units ward KIA Only 65 F 2 S 66 F 2 S 67 F 2 no U KA mm locate and operate?

68 H 2 S 69 H 2 S 70 H 2 E Stem basically says you violated MTC. Fix last sentence 71 F 2 S 72 H 2 no U C,D not water actions Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

34. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

35. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

36. Check the appropriate box if a psychometric flaw is identified:

• • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

• • The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

4.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

• The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "un ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 8acl<- Q= SRO BIMIN UlE/S Explanation Focus Oist. Link units ward KIA Only 73 F 2 E " most probable" not acceptable 74 H 2 S 75 F 2 S Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

37. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

38. Enter the level of difficulty (LaD) of each question USing a 1 - 5 (easy - difficult) rating scale (questions in the 2 4 range are acceptable).

39. Check the appropriate box if a psychometric flaw is identified:

• • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

• The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

• • The question is not linked to the job requirements (I.e., the question has a valid KIA but. as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (Bjank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section O.2.f.

7. Based on the reviewer's judgment. is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any *U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES*401 Written Examination Review Worksheet Form ES-401*9 Q#

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other I 6. 7. 8.

LOK LOD (FIH) (1-5) Explanation I Focus I 76 H 3 5 ~move the "not" in c, d; answer based on operational concem 77 H 3 5 S I 78 F 2 5 S 79 H 2 5 S 80 H 3 5 S 81 H 2 5 S 82 H 3 5 S I

83 H 2 5 I Instructions

[Refer to Section 0 of ES-401 and Appendix 8 for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g .. unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (I.e.* clues, specific determiners. phrasing. length, etc).

The answer choices are a collection of unrelated truelfalse statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (I.e.* the question has a valid KIA but, as written. is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g .* panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (8lank, (M)odified, or {N)ew. Check that {M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix 8 psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #/ Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 84 H 3 S 85 H 3 S 86 H 1 X U 87 F 3 S 88 F 2 S 89 H 2 X E Improve stem 90 H 2 S 91 F 2 S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

4. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

5. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

6. Check the appropriate box if a psychometric flaw is identified:

• • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).

• • The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

• • The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

• • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-on/y (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer'S judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement. or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/MIN utEtS Explanation Focus Dist. Link units ward KIA Only 92 F 2 S 93 H 2 X U No permission not plausible a, b 94 F 2 X U C, d not plausible. Ask TS basis for Ivl,: low level wrong distractors 95 H 3 S 96 F 2 S 97 H 2 S 98 F 2 S 99 H 2 S 100 F 2 S

7. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

B. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

9. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent. more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners. phraSing, length, etc).

The answer choices are a collection of unrelated truelfalse statements.

• • The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le .. it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

5.

• The question requires reverse logic or application compared to the job requirements.

Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank. (M)odified. or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.1.

7. Based on the reviewer's jUdgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U* ratings (e.g., how the Appendix B psychometric attributes are not being met).