ML14336A361

2014-11 Draft Written Exam Comments

ML14336A361
Person / Time
Site:	Arkansas Nuclear
Issue date:	11/21/2014
From:	Vincent Gaddy Operations Branch IV
To:
laura hurley
References
Download: ML14336A361 (5)

Text

ES-401 ANO1-2014-11 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 2 N S 2 H 3 X N E/S A not credible; SCM already rising with temperature dropping/Replaced A 3 F 2 N S 4 H 3 B S 5 H 2 N S 6 H 2 B S 7 H 2 N S 8 H 2 X X N U/S Same as Q 34/Replaced 9 F 2 N S 10 H 3 X B U/S Correct answer only credible distractor/Replaced Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 5 OBDI 202 - IOLE Process

ES-401 ANO1-2014-11 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 3 M S 12 H 3 X N E/S C not credible with no reactor trip scenario/Replaced C 13 H 3 X B U/S A and D are the same answer/Revised A 14 F 3 X X B E/S Change would to should**. D not credible/Revised 15 F 2 X X N E/S **; Change time increments in distractors as to not hit setpoint exactly/Revised 16 F 2 X X B E/S Cap LOWEST in stem. Stagger numbers in distractors to not hit stpt exactly/Revised 17 F 3 X N E/S Add procedure in stem/Added 18 H 3 X N E/S D not credible/Replaced D 19 H 3 M S 20 F 2 X N E/S Add procedure in stem/Added 21 H 3 N S 22 H 2 X B E/S Add procedure in stem/Added 23 H 4 X N E/S Add event in stem/Added 24 H 3 X B E/S D not plausible/Replaced D 25 H 3 X N E/S Change distractor flow rates to be more realistic/Revised 26 F 3 B S 27 H 2 X B U/S C and D not credible/Replaced Question 28 H 3 X N E/S **/Revised 29 H 3 X N E/S Add continuous flow to A/Added 30 F 2 N S 31 F 2 X N U/S Adjust all distractors to make more credible/Adjusted 32 H 3 B S 33 H 3 X M E/S **/Revised 34 F 2 N U/S Two correct answers as written/Revised stem to prevent multiple correct answers 35 F 2 X M E/S **/Revised Page 2 of 5 OBDI 202 - IOLE Process

ES-401 ANO1-2014-11 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 36 F 2 X N U/S Distractors not credible as written/Revised distractors to include channels 1 through 8 37 F 2 M S 38 H 3 X B E/S Add procedure in stem/Added 39 F 3 N S 40 H 3 X N U/S A and B not credible/Replaced Question 41 F 2 X N E/S Capitalize BOTH in stem/Done 42 F 2 N S 43 F 2 X N U/S B and D not credible/Revised B and D to make credible 44 H 3 X X N E/S **; Revise A and B to make more consistent/Revised 45 H 4 X N E/S Change shall to should in stem/Revised 46 H 3 N S 47 H 3 B S 48 H 3 X X N U/S A & B both correct as written; D is none of the above/Revised stem and distractors 49 H 3 X B E/S **/Revised 50 H 3 M S 51 H 3 X X N E/S **; D is none of the above/Revised 52 H 2 B S 53 F 2 B S 54 H 3 B S 55 H 3 X B E/S **/Revised 56 H 3 B S 57 F 2 B S 58 F 2 X N E/S Revise distractors such that all four dont have the same info in them/Revised 59 H 3 B S 60 H 3 X B U/S B and D not credible/Revised B and D Page 3 of 5 OBDI 202 - IOLE Process

ES-401 ANO1-2014-11 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 61 H 3 X X B E/S Revise stem and distractors to make more credible/Revised 62 H 3 X B E/S Add SCM and procedure in stem/Added 63 H 3 B S 64 H 2 N S 65 H 2 N S 66 F 2 X B E/S Add at a MINIMUM in stem/Added 67 F 3 X B E/S Add procedure in stem/Added 68 F 2 X X B E/S Add procedure in stem; revise C to make more credible/Added and revised 69 F 2 N S 70 F 2 B S 71 F 2 M S 72 H 3 B S 73 F 2 X N E/S Revise stem to remove unnecessary info/Revised 74 F 2 B S 75 H 2 N S 76 H 4 X N S 77 H 3 X X X N E/S Revise question to be consistent with reference/Revised 78 H 3 X X N E/S Capitalize SPECIFICALLY in stem/Revised 79 H 4 X N S 80 H 3 X X N U/S Revise A and B to avoid multiple correct answers/Revised 81 H 4 X X B U/S Direct lookup Tech Spec question/Replace 82 F 2 X N S 83 H 4 X X X N E/S A not credible; add FIRST in stem; eliminate first half of all distractors/Revised 84 H 2 X X X B U/S Only correct answer credible + same as Q 93/Replaced 85 H 3 X M S Page 4 of 5 OBDI 202 - IOLE Process

ES-401 ANO1-2014-11 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 86 H 3 B S 87 H 3 X B E/S Capitalize SPECIFIC in stem/Revised 88 F 2 B S 89 H 4 X N E/S Place noun names of valves in distractors/Revised 90 H 3 X X B U/S Only correct answer credible/Replaced 91 H 3 X B U/S Q duplicates scenario instrument failure/Replaced 92 H 3 B S 93 H 3 X B U/S Only correct answer credible/Replaced 94 H 2 X M E/S Revise stem to make different from original bank Q/Revised 95 H 3 X B E/S A not credible/Replaced A 96 F 3 M S 97 F 3 X M E/S Add ONLY in A; capitalize AND in distractors/Revised 98 F 3 X B E/S Eliminate unnecessary info in stem/Revised stem 99 F 2 N S 100 H 2 X N U/S Same Q as an Admin JPM/Replaced

• • comment generic to entire test (change would to should) Page 5 of 5 OBDI 202 - IOLE Process