ML13310A618

STP 2013-09 Draft Written Examination Comments

ML13310A618
Person / Time
Site:	South Texas
Issue date:	10/04/2013
From:	Clyde Osterholtz Operations Branch IV
To:	South Texas
laura hurley
References
Download: ML13310A618 (5)

Text

ES-401 STP-2013-09 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 X N U/S Correct answer too obvious/Replaced 2 F 3 N S 3 F 3 N S 4 F 3 X M E/S Add Specific system to A/Added 5 H 3 X B E/S Add actual ctmt press in stem/Added 6 H 3 N S 7 F 3 N S 8 H 3 X X M E/S Change stem to 28/make D correct answer/MIMIMUM in caps/Revised 9 F 3 N S 10 H 3 B S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 5 OBDI 202 - IOLE Process

ES-401 STP-2013-09 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 11 F 3 B S 12 H 3 B S 13 H 3 X M E/S In stem change 56.5 to 57.5 & make C correct answer/Revised 14 H 3 B S 15 F 3 X B E/S Capitalize NEXT in stem/Revised 16 F 2 X B U/S B,C.D not credible/Replaced 17 H 3 X X N E/S Get rid of unnecessary info (cueing) in all distracters/ Revised 18 F 3 X N U/S Q does not match K/A/Replaced 19 F 2 N S 20 F 3 N S 21 H 3 X N E/S Abbreviate RCFC in distracters/Revised 22 F 3 X X X N U/S Two correct answers/Replaced 23 H 3 N S 24 H 3 X X N E/S Add Procedure in stem; Change A/Revised 25 F 3 X N E/S Add procedure in stem; enhance distracters/ Revised 26 F 3 X X N E/S Change slowly lowering to lowering in stem/Revised 27 F 2 X N E/S Delete unnecessary info in B/Revised 28 H 3 X X N U/S A & B not credible/ Replaced 29 F 3 X X N E/S Get rid of unnecessary info in all distracters/ Revised 30 H 3 X N E/S A not credible. Revise to make 2X2/ Revised 31 F 3 X N E/S Delete first half of stem and add procedure title in stem/Revised 32 H 3 X B E/S In stem put MINIMUM in caps/Revised 33 H 3 X B U/S Two correct answers (A and B)/Replaced 34 H 4 B S 35 F 3 B S Page 2 of 5 OBDI 202 - IOLE Process

ES-401 STP-2013-09 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 36 F 3 B S 37 H 3 X B E/S Underline all the ORs in the distracters/Revised 38 H 3 X B E/S Change could to should in stem/ Revised 39 F 3 X B E/S Put MINIMUM in caps in stem/ Revised 40 F 3 X X N E/S Delete first sentence in stem and add EDG in the remainder of stem/Revised 41 F 2 N S 42 F 3 N U/S A & C not credible/Replaced 43 F 3 X B U/S B & D not credible/Revised distracters 44 H 3 X X B S 45 H 3 X N U/S A & B not credible/Revised distracters 46 H 3 B S 47 H 3 X X B U/S Q does not match K/A/Replaced 48 H 3 X N U/S B & D not credible/Revised distracters 49 F 3 X N U/S Q not credible (not being able to manually initiate fire protection)/Replaced 50 H 3 X N E/S Add procedure in stem/Revised 51 H 3 X X N S 52 H 3 X M E/S Change instantly to immediately in A& B and add times in C & D/ Revised 53 H 2 N S 54 H 3 X N E/S Put times in A & C/ Revised 55 H 3 X X X N E/S Get rid of second half on all distracters/ Revised 56 H 3 X N E/S Add to AFW pump 14 in C & D/ Revised 57 F 2 B U/S Too similar to question 43/Replace 58 H 3 N S 59 F 3 X N E/S Change would to should in all distracters/ Revised 60 F 2 N E/S Revise stem to make more clear/Revised Page 3 of 5 OBDI 202 - IOLE Process

ES-401 STP-2013-09 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 61 H 2 X N U/S Only correct answer is credible/Revised 62 F 2 X N E/S C not credible/ Revised 63 F 2 N S 64 H 3 N S 65 F 3 X N E/S Change stem values so that all pumps have different temperatures/Revised 66 H 3 X N U/S A & C not credible/ Revised 67 F 3 X N E/S Add procedure in stem/ Revised 68 F 3 X N E/S Chang each and selected to 25 and 50 and ask for TOTAL in stem/ Revised 69 H 3 N E/S Remove equipment numbers from B/Revised 70 H 3 N S 71 H 3 N S 72 H 3 X X N E/S Add procedure in stem; eliminate unnecessary info in distracters/ Revised 73 H 3 X N E/S D not credible/Revised 74 F 2 N S 75 F 3 X X B U/S Two correct answers/Restructured Q to eliminate problem 76 F 3 X N S 77 F 3 X N S 78 H 3 X B S 79 H 3 X N S 80 H 3 X N S 81 H 3 X N S 82 H 3 X N S 83 H 3 X N S 84 H 3 X X N U/S Direct lookup - get rid of references/Deleted references 85 H 3 X X N E/S Add ONLY to A & D/Revised Page 4 of 5 OBDI 202 - IOLE Process

ES-401 STP-2013-09 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 86 H 3 X X X N E/S Add valve name in stem and in A/Revised 87 H 3 X X N U/S Direct lookup - get rid of references/References deleted 88 F 3 X N S 89 F 2 X N S 90 F 3 X N U/S B & C not credible /Replaced 91 H 3 X X N U/S B & D not credible/ Revised 92 F 2 X N S 93 H 2 X N S 94 F 3 X X N E/S Add procedures in stem//Revised 95 H 3 X N U/S A & B not credible/Replaced 96 H 3 X N U/S Too similar to Admin JPM/Replaced 97 H 3 X X X N E/S Minimum in caps in stem; Change C to acting rad mgr ONLY/ Revised & Changed 98 H 3 X N S 99 H 3 X N S 100 H 3 X N S Page 5 of 5 OBDI 202 - IOLE Process