ML112700882

2011-08-Written Examination Comments

ML112700882
Person / Time
Site:	Wolf Creek
Issue date:	08/26/2011
From:	Clyde Osterholtz Operations Branch IV
To:	Wolf Creek
References
50-482/11-301
Download: ML112700882 (5)

Text

ES-401 WC-2011-08 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 N S 2 F 2 X B ES Make stem more restrictive Changed 85 to 80 to make C more plausible 3 F 2 X X N US D gives away correct answer Replaced D 4 H 3 N S 5 F 2 N ES Enhance C and D Enhanced 6 H 3 N US No correct answer as written Revised distracters to properly indicate correct answer 7 H 3 X B ES C and D not credible Revised distracters 8 F 2 X N ES Add temp and press info to stem Added 9 F 2 N S 10 F 2 N S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 5 OBDI 202 - IOLE Process

ES-401 WC-2011-08 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 2 X N ES Replace A Replaced 12 H 2 X X N US C and D not credible Revised 13 F 3 X N ES Add stem info that verifies PR hi flux locks in Revised 14 H 3 N S 15 H 3 X B ES Replaced D Replaced 16 H 3 X X N US Bad stem and bad distracters Question rewritten 17 H 4 N S 18 H 3 N S 19 F 3 X N ES Modify B and C to make more credible Revised 20 H 3 X B ES Modify distracters for consistency Revised 21 H 2 N S 22 H 3 B S 23 F 3 N S 24 F 3 X B ES Modify stem + justification s dont back up distracters Modified; justifications clarified 25 H 3 X N ES Fix typo in C Fixed 26 H 2 X X N US Answer in stem; D not credible Revised 27 H 2 M S 28 F 3 N ES Modify distracters to make more credible Modified 29 F 3 X X N ES Modify stem and distracters for consistency Modified 30 H 2 X N S 31 F 2 X X N ES D not plausible; modify stem info Revised stem and D 32 H 2 X N US B and D not credible Revised all distracters 33 F 2 X B ES Delete unnecessary info in stem Deleted 34 F 3 X N ES Fix typo in stem Fixed 35 H 2 X N ES Revise stem - upper to 100% Program Revised Page 2 of 5 OBDI 202 - IOLE Process

ES-401 WC-2011-08 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 36 H 2 N S 37 H 3 X N ES Need to say something about B RHR in stem Added 38 H 3 N S 39 H 3 X X B US K/A mismatch - need new K/A and Question Added new K/A and Question 40 H 3 N S 41 H 2 X X N US Got correct ans w/o reading question Revised distracters so correct ans not obvious 42 F 3 N S 43 H 2 N S 44 H 2 N S 45 H 2 N S 46 H 2 X N ES Amend A and B to make more believable Amended 47 F 2 X N ES Change A so that it is not so similar to D A changed 48 H 3 X N ES B too close to correct answer Replaced B 49 H 2 X X N US A cues answer; C & D not credible Question rewritten to eliminate problems 50 H 3 N ES Stem has wrong info Stem fixed 51 H 3 X B ES Enhance A to make more credible A enhanced 52 F 3 X B ES Modify D to make more believable D modified 53 F 2 X X X M US Got correct ans w/o reading question Revised distracters so correct ans not obvious 54 F 2 X N US Only credible distracter is correct answer Question rewritten 55 F 3 X M ES B not credible B replaced 56 H 4 N S 57 F 3 M S 58 H 3 X N ES Delete PF from stem PF deleted 59 H 3 B S 60 F 3 N S Page 3 of 5 OBDI 202 - IOLE Process

ES-401 WC-2011-08 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 61 H 2 X X X N US C cues answer, D not credible, B doesnt address stem Question rewritten 62 F 2 N S 63 F 3 N ES Revise C to make more clear Revised 64 H 2 N US Got correct ans w/o reading question Revised distracters so correct ans not obvious 65 H 3 X B US Only one credible distracter Question rewritten 66 F 2 B S 67 F 2 X N ES Remove unnecessary info from stem Removed 68 F 2 B S 69 H 3 X B ES Make B orange path to be consistent with C Revised 70 H 3 B S 71 H 3 X B ES Revise A to make more credible Revised 72 H 3 N S 73 F 2 B S 74 H 2 B S 75 F 2 N US B & C not credible Question rewritten 76 H 3 X B S 77 H 3 X N S 78 H 2 X X N US Only one credible distracter Question rewritten 79 H 2 X X N US Only one credible distracter Question rewritten 80 H 2 X X X N US No correct answer as written Question rewritten 81 H 2 X X N ES Change would to should in stem Changed 82 H 2 X X M ES Modify C to keep from giving away correct answer Modified 83 H 2 X B S 84 H 2 X N S 85 H 3 X X N ES Change PF info to Mvars Changed Page 4 of 5 OBDI 202 - IOLE Process

ES-401 WC-2011-08 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 86 H 2 X X N ES Revise C and D to make more credible Revised 87 H 2 X X N ES B not credible Replaced 88 F 2 X X X N ES Change A and C times; add procedurally to stem Changed and added 89 H 2 X X N ES Delete unnecessary info in stem Deleted unnecessary info 90 H 2 X X N US Answer obtained w/o reading question/too close to Question 59 Question replaced 91 F 2 X X N ES Modify stem to make distracters more plausible Modified 92 F 2 X X X N ES Clarify stem, replace non-credible D, make B & C more clear Changes made 93 H 2 X X X B US Entire question not credible Question replaced 94 F 3 X B S 95 H 3 X X X M ES Modify stem and distracters to enhance question Modified 96 H 3 X N S 97 H 2 X X X B US Answer is in question Question rewritten 98 H 3 X X N ES Enhance distracters C & D Enhanced 99 H 3 X N S 100 H 3 X N S Page 5 of 5 OBDI 202 - IOLE Process