ML15244B448
| ML15244B448 | |
| Person / Time | |
|---|---|
| Site: | Duane Arnold  |
| Issue date: | 08/20/2015 |
| From: | NRC/RGN-III/DNMS/MLB |
| To: | Duane Arnold |
| Zoia, C D | |
| Shared Package | |
| ML15118A355 | List: |
| References | |
| Download: ML15244B448 (6) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 2.0 B E Changed stem wording from Recirc loop flow based on facility validation comment to improve clarity (EDITORIAL ENHANCEMENT) 2 H 3.0 N S 3 H 2.0 N S 4 H 2.0 N S Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
- The answer choices are a collection of unrelated true/false statements.
- The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- The question requires reverse logic or application compared to the job requirements.
4.Check questions that are sampled for conformance with the 5.approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 5 H 3.0 X N U Facility identified change added a pressure condition to the stem in order to prevent the possibility of a second answer (PARTIAL).
Modified a stem condition to remove wording related to an APRM high flux scram (EDITORIAL ENHANCEMENT).
6 H 3.0 N S 7 H 3.0 N S 8 H 2.0 N S 9 H 3.0 N S Original question was replaced at facility request. Replacement question stem was revised based upon validation comments to 10 H 2.0 N S prevent confusion related to rod blocks and annunciators (EDITORIAL ENHANCEMENT).
11 F 2.0 B S Stem wording changed for clarity; replaced reference to main 12 H 3.0 N E generator MW with reactor power. Answer and distractors updated accordingly (EDITORIAL ENHANCEMENT) 13 H 3.0 M S 14 F 2.0 B S 15 F 3.0 B S Order of conditions in stem modified based upon facility feedback 16 H 3.0 N E (EDITORIAL ENHANCEMENT).
Wording in stem changed from prior to exceeding to to avoid 17 H 3.0 N E exceeding at facility request (EDITORIAL ENHANCEMENT).
18 F 2.0 N S Fire alarm added to initial conditions to prevent an NRC-identified 19 F 2.0 X N U condition where the stem conditions would not lead to the correct answer (STEM FOCUS).
20 F 2.0 N S 21 H 2.0 N S 22 H 2.0 B S Removed stem wording of assuming no operator action 23 H 2.0 N E (EDITORIAL ENHANCEMENT).
24 F 3.0 M S 25 F 2.0 N S 26 F 2.0 B S Stem condition added based upon NRC review to clarify that 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> warmup is met to prevent distractor D from being a second correct 27 F 2.0 X X N U answer (PARTIAL). Stem condition provided based upon NRC review to clarify that Defeat 9 physically cannot work to prevent distractor A from being a second correct answer (STEM FOCUS).
28 H 2.0 B S 29 H 3.0 N S 30 F 2.0 B S Changed the value of -24 to -25 in the answer based upon facility review (EDITORIAL ENHANCEMENT). Changed the second part of 31 H 2.0 X N E distractor D to include prior to the low pressure isolation setpoint to make the distractor plausible (CREDIBLE DISTRACTOR).
32 H 2.0 N S Added a table containing valve numbers and names for clarity based upon facility review (EDITORIAL ENHANCEMENT). Answer and 33 H 3.0 N E distractors reformatted for clarity based upon NRC review (EDITORIAL ENHANCEMENT).
34 H 3.0 B S 35 H 3.0 N S 36 H 2.0 B S 37 H 3.0 M S
38 F 3.0 N S 39 F 2.0 M S Condition added to stem based upon facility review to prevent D 40 H 2.0 X B U from being a second correct answer (STEM FOCUS).
41 H 3.0 N S 42 F 2.0 N S 43 H 2.0 N S Removed additional wording from stem for clarity based upon NRC 44 F 2.0 N E review (EDITORIAL ENHANCEMENT).
45 F 3.0 N S 46 H 3.0 B S 47 F 2.0 M S Capitalized the name of the equipment in the stem based upon 48 H 2.0 B E facility review (EDITORIAL ENHANCEMENT).
49 H 3.0 B S 50 F 2.0 N S 51 H 2.0 N S Unit (psig) moved from stem to individual answers for clarity based 52 F 2.0 N E upon facility review (EDITORIAL ENHANCEMENT).
53 F 2.0 N S 54 H 2.0 N S 55 F 3.0 N S Wording in stem modified for clarity based upon facility review 56 F 2.0 N E (EDITORIAL ENHANCEMENT).
57 F 2.0 B S 58 H 2.0 B S 59 F 2.0 M S 60 H 3.0 N S 61 F 2.0 N S
62 H 2.0 B S Additional alarm (Hi Rad) added to stem based upon facility review.
63 F 2.0 M E Additional wording (PCIS and MSIVs) added to distractor/answer for clarity (EDITORIAL ENHANCEMENT).
Corrected typo where the word alarm was shown in the plural form 64 H 3.0 B E based upon facility review (EDITORIAL ENHANCEMENT).
Fan designators modified for better readability based on facility 65 F 2.0 N E review (EDITORIAL ENHANCEMENT).
Turbocharger inlet temperature value in stem lowered to better align 66 F 3.0 N E with system precautions and limitations based upon facility comment (EDITORIAL ENHANCEMENT).
Tense of the word observes changed for clarity based upon facility 67 F 2.0 N S review (EDITORIAL ENHANCEMENT).
68 F 3.0 N S 69 H 2.0 B S 70 F 2.0 N S 71 F 2.0 N S 72 F 2.0 B S Additional information added to stem based upon NRC review to 73 H 3.0 X M U provide the information necessary for the answer to be determined (STEM FOCUS).
74 F 2.0 B S Additional EOP information removed from flowchart symbol graphic 75 F 2.0 B E in order to removed unnecessary information from the question (EDITORIAL ENHANCEMENT).
Removed unneeded information from the stem and reformatted the 76 H 2.0 X N E structure of other stem information for clarity based upon facility review (EDIOTRIAL ENHANCEMENT).
77 H 2.0 X N S 78 H 3.0 X N S 79 F 2.0 X N S 80 H 2.0 X M S 81 H 2.0 X B S
82 H 2.0 X N S Question replaced with a new question for the same K/A at facility 83 H 2.0 X N S request.
84 H 3.0 X M S 85 H 3.0 X B S 86 H 3.0 X N S Wording in answers and distractors adjusted for clarity based upon 87 H 2.0 X N E facility review (EDITORIAL ENHANCEMENT).
88 H 2.0 X M S 89 H 2.0 X N S Stem condition modified and additional wording added to stem for 90 H 2.0 X B E clarification based upon facility review (EDITORIAL ENHANCEMENT).
91 H 2.0 X N S 92 H 3.0 X B S 93 H 3.0 X B S 94 F 3.0 X N S Additional wording regarding SRO direction added to the stem and 95 F 3.0 X N E RPV water level band added to the answer for clarity based upon facility review (EDITORIAL ENHANCEMENT).
96 H 3.0 X B S Question replaced with a new question for the same K/A at facility 97 H 2.0 X N S request.
98 F 2.0 X B S 99 F 2.0 X B S 100 H 2.0 X B S