ML22243A219
| ML22243A219 | |
| Person / Time | |
|---|---|
| Site: | Prairie Island  |
| Issue date: | 08/26/2022 |
| From: | Jim Nance NRC/RGN-III/DRS/OLB |
| To: | Xcel Energy |
| Nance J | |
| Shared Package | |
| ML21188A222 | List: |
| References | |
| Download: ML22243A219 (13) | |
Text
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 1
Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level General Comment:
Ensure that all references provided to the applicants meet the acceptance criteria given in ES 4.2.B.6.d.
1 F
2 N
S Question is SAT as written.
2 H
3 M
E S
NRC: Embedded Reference Significantly Modified from 2020 ILT NRC Exam R01.
Update Pedigree to reflect question is Modified.
Response
Updated pedigree to reflect modified from 2020 ILT.
Question is now SAT.
3 H
2 N
S Question is SAT as written.
4 F
2 N
E S
NRC: Embedded Reference The meanings of the abbreviations in the distractors need to be included in the distractor pedigrees (ERR, DBAD, HWRN, etc.).
Response
Revised justifications to Include definitions are requested. Question is now SAT.
5 H
2 N
S NRC: Embedded Reference 6
H 2
B E
S NRC: Distractor D is not plausible. Move the Bus 15 is LOCKED OUT bullet to a subsequent statement in the Stem question to enhance Distractor D plausibility.
Response
Moved bullet as requested. Question is now SAT.
7 F
2 N
S Question is SAT as written.
8 H
2 X
B E
S EARLY SUBMITTAL NRC: Embedded Reference PI 2016 ILT Audit This question does not meet the K/A. What is the loss or malfunction of the RHR system? This is also not a high cog. question.
Response
Malfunction inserted into the stem of the question making this a high cog. Question and a K/A match.
Question is now SAT.
9 H
2 N
S Question is SAT as written.
10 H
2 N
E S
NRC: Embedded Reference A labeled, embedded reference is not operationally valid, and unnecessarily simplifies this question.
Response
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 2
Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level Remove words from picture. Use arrows to label potions of picture A through D. Question is now SAT.
11 F
2 N
S EARLY SUBMITTAL. Question is SAT as written.
12 F
2 X
N E
S NRC: Distractor D is implausible as LMS is not related to troubleshooting. Also, explain more in the pedigree what SumTotal and Xpress Request are and why they are plausible but not correct in this question.
Response
Changed D to CHAMPS Update Request. Added amplifying information to Dist. A & B analysis.
Question is now SAT.
13 H
2 X
N U
S NRC: This question does not meet the K/A, this is system-specific, not a unit-specific difference.
Response
Recommend replace KA.-(New K/A selected G2.2.1)
Replacement Question SAT.
14 H
2 X
X M
U S
NRC: Embedded Reference Significantly Modified 2018 ILT NRC Exam R34 Contains only partially correct answer and Distractor B is not plausible per conditions given in the Stem.
Reword the distractor to elicit a single correct answer and enhance plausibility.
Response
Dist. B explanation changed; remaining distractors reworded. Question is now SAT.
15 H
2 X
M E
S NRC: Embedded Reference Significantly Modified 2020 ILT NRC Exam R40 The stem states that Unit 1 Safety Injection is actuated. Clarify Stem conditions to test knowledge of SI Actuation logic. Reword distractors as needed.
Response
Revised Stem question to focus on SI Actuation logic, reworded distractors, and updated the pedigree.
Question is now SAT.
16 H
2 N
E S
Reference provided NRC: Remove Reference FR H-1 to enhance question discriminatory validity.
Response
Removed reference provided. Question is now SAT.
17 F
3 B
E S
NRC: Replace the word intermittently in Distractors.
Response
Changed intermittently to shiftly. Question is now SAT.
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 3
Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level 18 F
2 N
S Question is SAT as written.
19 H
2 X
X X
N E
S EARLY SUBMITTAL Reference provided NRC: Embedded Reference
- 1) Cuing. The SWO Tag # SIM: LI-41018 (in the embedded reference for the question) states that Meter Missing Units of Inches Rflg Canal Level, which is the answer. 2) All others are implausible distractors. 3) LOD of distractor D is 1. Since 2.53 cm equals one inch, 400 cm equals 158 inches (or ~13 feet) and fuel assemblies are approximately that tall.
- 4) Does not meet K/A - it involves simply reading the gauge. 5) This is not a high cog. Question 6) Direct Look-up. 7) Actuating SI during a refueling is implausible
Response
- 1) through 7) have been addressed. Changed question to a 2 X 2 format. Question is now SAT.
20 F
2 N
E S
NRC: The correct answer is the only option with a procedural reference; remove the procedure.
Response
Removed the procedure reference from the answer, and reworded the distractors. Question is now SAT.
21 F
2 X
N E
S EARLY SUBMITTAL NRC: Distractors C and D can be easily eliminated and are implausible due to Stem Initial Conditions.
Response
Changed Initial Conditions in Stem to BOL and MOL to enhance Distractors C & D plausibility. Question is now SAT.
22 H
2 X
N E
S NRC: Distractor A is implausible due to Initial Condition given in the Stem. Change Distractor A.
Response
Changed distractor A to Condenser hood spray.
Question is now SAT.
23 H
3 B
S Question is SAT as written.
24 H
2 M
S NRC: Embedded Reference Significantly modified from 2018 ILT NRC Exam R26.
Question is SAT as written.
25 F
2 X
B E
S NRC: Distractor A is implausible as this parameter is not assessed for this system. Replace Distractor A.
Response
Changed distractor A to Contamination and reordered the distractors. Question is now SAT.
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 4
Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level 26 F
2 N
S Question is SAT as written.
27 H
2 X
N U
S Reference provided NRC: This does not meet the K/A. Replace question to focus on Aux Spray Valve operation.
Response
Question has been replaced. Replacement question is SAT.
28 F
2 M
E S
NRC: For Distractor A pedigree, change dilution to boration.
Significantly Modified from 2018 ILT NRC Exam R07.
Response
Distractor Analysis changed. Question is now SAT.
29 F
2 N
S Question is SAT as written.
30 F
2 B
S Question is SAT as written.
31 H
3 X
B U
S NRC: Embedded Reference K/A mismatch - What automatic features of the CVCS VCT pressure control are addressed?
Response
There is no automatic VCT pressure control system.
Recommend replace KA.-(New K/A selected A3.09)
Replacement Question is Bank and SAT.
32 H
3 N
E S
NRC: Embedded Reference Add Plant Initial Condition to Stem-MODE.
Response
Added Mode 1 to Stem Initial Conditions. Question is now SAT.
33 H
3 N
S Question is SAT as written.
34 H
2 N
S Unit 1 Indicators DO NOT have Red Lines in the Control Room. Sim Work Request processed to remove lines from the simulator indicators.
Question is SAT as written.
35 F
2 X
N E
S NRC: Distractor C is implausible as SG Levels are not impacted by RCS inventory. Replace Distractor C.
Response
Replaced Distractor C and updated the pedigree.
Question is now SAT.
36 F
2 N
S Question is SAT as written.
37 F
2 X
N U
S NRC:K/A mismatch for steam generator water level.
Replace the K/A.
Response
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 5
Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level Recommend replace KA.-(New K/A selected K5.18)
Replacement Question SAT.
38 H
2 N
S Question is SAT as written.
39 F
2 N
E S
NRC: Common, basic knowledge as tied to an Immediate Action item. Remove Stem tie to a specific IA Step to enhance question discriminatory validity.
Response
Removed the specific reference to a step from the Stem. Question is now SAT.
40 F
2 N
E S
EARLY SUBMITTAL NRC: Why are DC MOVs plausible? Change LOK to memory.
Response
Some DC MOVs are used at Prairie Island. Changed LOK to F. Question is now SAT.
41 H
2 M
E S
NRC: Embedded Reference Significantly Modified from 2018 ILT NRC Exam R52.
Change Stem Initial Conditions to Unit 2 to enhance discriminatory validity.
Response
Changed Stem Initial Conditions as requested.
Question is now SAT.
42 H
2 B
S NRC: Embedded Reference PI 2018 ILT NRC Exam R16 Question is SAT as written.
43 F
2 N
E S
NRC: This is not a high cog. question.
Response
Changed LOK to F. Question is now SAT.
44 H
2 M
E S
EARLY SUBMITTAL NRC: Embedded Reference Significantly Modified from 2020 ILT NRC Exam R39.
No meter is below 1830 psig. Difficult to determine if below/at setpoints; adjust images to clarify.
Response
Modified embedded reference. Question is now SAT.
45 H
2 N
S NRC: Embedded Reference Question is SAT as written.
46 F
2 M
B E
S NRC: This is not a significantly modified bank question as it does not change 1 Stem condition and 1 distractor. Update the pedigree to reflect Bank.
Response
Agree. Not Significantly Modified; pedigree updated.
Question is now SAT.
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 6
Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level 47 H
2 X
N E
S NRC: Embedded Reference Distractor D is not plausible for DC Voltage. Replace distractor and reorder in ascending/descending order.
Response
Replaced Distractor D and reordered in descending order. Question is now SAT.
48 F
2 X
N U
S NRC: Distractor B and C are implausible based on the Stem condition given. Replace distractors and edit distractor wording for clarity.
Response
Replaced Distractor C and edited B and D for clarity.
Question is now SAT.
49 H
2 M
S NRC: Embedded Reference Significantly Modified from 2020 ILT NRC Exam R33.
Question is SAT as written.
50 H
2 X
N U
S NRC: Embedded Reference The question meets the K/A. However, Distractors C
& D, are implausible because they do not meet the Initial Conditions given in the Stem. Reword to focus on the cause of the malfunction which matches the given result. Add SG levels to the Initial Conditions in the Stem to enhance plausibility.
Response
Added SG Initial Conditions to the Stem, modified Stem question, and changed distractors to focus on the cause of the malfunction. Question is now SAT.
51 H
2 X
M U
S NRC: Embedded Reference Distractor A and C are not plausible for overspeed condition given. Add expected annunciator to the Stem and replace Distractors A and C.
Response
Changed Stem to include item noted and replaced distractors. Question is now SAT.
52 H
2 B
E S
Reference Provided NRC: Embedded Reference Provided references are not needed to support the question when given the embedded reference.
Remove the provided references.
Response
Removed provided references. Question is now SAT.
53 H
2 N
S NRC: Embedded Reference Question is SAT as written.
54 F
2 B
S Question is SAT as written.
55 F
2 B
S 2016 PI ILT NRC Exam Question is SAT as written.
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 7
Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level 56 F
2 N
S Question is SAT as written.
57 H
2 X
N E
S NRC: Embedded Reference Distractor D is implausible - the embedded reference shows Unit 1. Modify or remove the image header.
Response
Modified header to reflect in-plant image, and replaced Distractor D with something tied to Unit 1.
Question is now SAT.
58 F
2 B
S PI 2016 ILT NRC Exam Question is SAT as written.
59 H
3 B
S NRC: Embedded Reference PI 2018 ILT NRC Exam R04 Question is SAT as written.
60 F
2 N
E S
NRC: Clarify Stem Initial Condition to reflect the Tier 4 fundamental concept.
Response
Changed Stem Initial Condition to reflect an approach to criticality. Question is now SAT.
61 H
3 M
S EARLY SUBMITTAL NRC: Embedded Reference Significantly Modified 2020 ILT NRC Exam R24 Question is SAT as written.
62 F
2 N
S Question is SAT as written.
63 H
2 N
S Question is SAT as written.
64 H
3 B
S NRC: Embedded Reference 2016 PI ILT NRC Exam Question is SAT as written.
65 F
2 X
N U
S NRC: Distractors C and D are implausible since Recombiners will not raise hydrogen levels. Rewrite question to not use a 2 X 2 format. This is also a memory level question.
Response
Agree. changed to low cog., removed the first part of the distractors/answer, and changed second parts to four unique answers. Question is now SAT.
66 F
2 B
S PI 2014 ILT NRC Exam Question is SAT as written.
67 F
2 N
S Question is SAT as written.
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 8
Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level 68 H
3 N
S NRC: Embedded Reference Question is SAT as written.
69 F
2 X
N E
S NRC: Distractors D is not plausible for the Initial Conditions provided in the Stem.
Response
Replaced Distractor D with Recover rod G-11 to 75 steps. Question is now SAT.
70 F
2 N
E S
NRC: Enhance the Distractor B justification in the Pedigree.
Response
Distractor Analysis updated for Distractor B. Question is now SAT.
71 H
2 X
M U
S NRC: Embedded Reference Distractors A, B, and D values are not on the embedded reference, and are not credible/plausible.
Response
Replace distractor values with plausible selections which are listed in the reference table provided in the Stem. Question is now SAT.
72 H
2 M
S NRC: Embedded Reference Significantly Modified from 2018 ILT NRC Exam R55.
Question is SAT as written.
73 H
2 N
E S
NRC: The explanation for the correct answer does not clearly address the solid plant condition.
Response
Clarified the Correct Answer justification. Question is now SAT.
74 H
2 X
N E
S NRC: Embedded Reference Distractors D is not plausible as it implies no system response to stated malfunction.
Response
Changed Distractor D and reworded another distractor to enhance plausibility. Question is now SAT.
75 H
3 N
S Question is SAT as written.
76 /
1 H
3 N
E S
References provided (used for question 94 as well)
NRC: Distractor D could also be considered correct because the applicant is currently in step 28 and transitioning to step 29.
Response
Replaced distractor D with step 5 of FR-H.2. Question is now SAT.
77 /
2 H
2 M
E S
Reference provided NRC: Embedded Reference PI 2014 ILT NRC Exam
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 9
Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level Provide entire reference (not just 1 page). Question is SAT as written (Not T/F; SRO level Task) 78 /
3 H
3 N
S Reference provided Question is SAT as written.
79 /
4 H
2 N
E S
Reference provided NRC: Embedded Reference The provided reference (1 page) implies information the applicant may know from memory and has low discriminatory validity. Provide all TS 3.0 LCOs and SRs to enhance discriminatory validity.(Not Direct Look Up).
Response
Provided reference as stated. Question is now SAT.
80 /
5 H
2 N
S Question is SAT as written.
81 /
6 H
3 X
N E
S References provided NRC: Provide the initial temperature of the Spent Fuel Pool in the Stem to enhance the plausibility of Distractors C & D. Enhance the distractor analysis for why 32 hours3.703704e-4 days <br />0.00889 hours <br />5.291005e-5 weeks <br />1.2176e-5 months <br /> is plausible.
Response
Added 130 deg F to stem and updated pedigree.
Question is now SAT.
82 /
7 H
2 N
E S
Reference provided NRC: Distractor D is not plausible as there are not 3 pumps per Unit. Replace Distractor D.
Use of the EOP Table to complete the Step is an SRO task, and is not LOD 1.
Response
Rewrote the question to affect both Units, changing distractors and correct answer. Question is now SAT.
83 /
8 H
1 2
X N
E S
EARLY SUBMITTAL NRC: Embedded Reference
- 1) Level of difficulty = 1. Red path would always be first. 2) Since this is a MCB indication, ROs would be expected to properly respond to it. Even though ROs may not select the procedure to be entered (which was not asked), they would be expected to recognize high priority color CSFs and report them.
Response
- 1) and 2) have been addressed by modifying the embedded reference and changing the question format to 2X2 to select an FR procedure. Question is now SAT.
84 /
9 H
2 X
B U
S NRC: This is a K/A mismatch to vital AC Instrument power. Change K/A.
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 10 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
Response
Recommend replace KA.-(New K/A selected AA2.08)
Replacement Question SAT and is a Bank from 2018 ILT NRC Exam.
85 /
10 F
2 X
N U
S NRC: Distractors A, B, and D, are implausible.
Response
Changed distractors to following plausible entities:
Design Engineering Manager Regulatory Affairs Manager Plant Operations Review Committee Question is now SAT.
86 /
11 F
2 X
N E
S NRC: This is an SRO task per the Fleet Conduct of Operation procedure. Modify the Stem to address an activity required by an SRO.
Response
Revised Stem and SRO justification for clarity.
Question is now SAT.
87 /
12 H
3 N
E S
NRC: Correct the typo (step 22 vs step 21) in the correct answer.
Question is SAT as written.
88 /
13 H
1 2
X N
E S
EARLY SUBMITTAL Reference Provided NRC: 1) This is a direct lookup question, LOD with the first page of the provided reference (flowchart),
and given that containment pressure is 29 PSIG, FR-Z.1 is the only choice. 2) Distractors B, C, and D-are implausible. 3) Consider that overlap with S100 exists, which could be avoided by asking for more than just F-05 Flowchart use.
Response
- 1) and 2) have been addressed by adding additional Initial Conditions to the Stem which clarify the need to determine which 1FR response is required.
(See S100/25 for response to 3). Question is now SAT.
89 /
14 F
2 N
S Question is SAT as written.
90 /
15 H
2 X
N E
S EARLY SUBMITTAL Reference provided NRC: Distractors C and D are implausible and have the same result. C requires the crew to continue the procedure at step 9; and D requires the operator to return to the procedure in effect.
Response
Changed Distractors A, B, and C and updated the Distractor Analysis as required. Question is now SAT.
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 11 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level 91 /
16 F
3 B
E S
NRC: Omit BOTH in distractor D. Use AND for distractors C and D.
Response
Made changes as requested. Question is now SAT.
92 /
17 H
2 X
N U
S NRC: There are three implausible distractors. There are no possible reasons given in the question stem that would alter this well-known, immediate transition to 1ES-1.2, TRANSFER TO RECIRCULATION.
Replace question to prioritize annunciator actions.
PI - 2014 ILT EOP Exam #4 Grp 1
Response
Replaced the question with one listing four annunciators at 100% power and asking which annunciator the SS will direct the crew to address first Replacement question is New and is SAT..
93 /
18 H
2 N
S NRC: Embedded Reference Question is SAT as written.
94 /
19 H
2 X
N E
S Reference provided (used for questions 76 as well)
NRC: Distractors D is not plausible with the Initial Conditions provided in the Stem. Modify the 12 SG Initial Conditions in the Stem to enhance plausibility of Dist. D. Update Distractor Analysis.
Response
Changed Stem initial condition as requested and updated pedigree. Question is now SAT.
95 /
20 H
3 B
S Reference provided.
Question is SAT as written.
96 /
21 F
2 N
S Question is SAT as written.
97 /
22 H
3 B
S Question is SAT as written.
98 /
23 H
3 B
S NRC: Embedded Reference.
Question is SAT as written.
99 /
24 H
3 N
E S
NRC: Embedded Reference Update Distractor A Analysis.
Response
Revised pedigree as requested. Question is now SAT.
100
/ 25 H
2 M
E S
NRC: Embedded Reference Revise the embedded reference to label and permit selection of the correct path as the answer and update the distractors and the Analysis accordingly.
Response
Revised the embedded reference and updated the pedigree accordingly. Question is now SAT.
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 12 Form 2.3-5 Instructions for Written Examination Review Worksheet Refer to ES-4.2 for the definitions of terms used in this worksheet for the written examination. Review each question (Q) as submitted and as subsequently revised and document the following in the associated worksheet columns:
- 1. Enter the level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) from 1 (easy) to 5 (difficult); mark direct lookup questions (applicant can directly determine the answer from the provided reference) as LOD 1. A question is (U)nsatisfactory if it is LOD 1 or LOD 5.
- 3. Check the appropriate box if a psychometric flaw is identified:
Stem Focus: The stem lacks enough focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information). This is an (U)nsatisfactory question.
Cues: The stem or one or more answer choices contains cues (e.g., clues, specific determiners, phrasing, length). This is an (U)nsatisfactory question.
T/F: All of the answer choices are a collection of unrelated true/false statements. This is an (U)nsatisfactory question.
Cred. Dist.: The distractors are not credible; single implausible distractors require (E)nhancement, and more than one noncredible distractor in the same question results in an (U)satisfactory question.
Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem). This is an (U)nsatisfactory question.
- 4. Check the appropriate box if a job content flaw is identified:
Job Link: The question is not linked to the job requirements (i.e., the question has a valid knowledge or ability (K/A) but, as written, is not operational in content). This is an (U)nsatisfactory question.
Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory). This is an (U)nsatisfactory question.
- /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). This is an (U)nsatisfactory question.
Logic: The question requires backward or reverse logic or application compared to the job requirements. This is an (U)nsatisfactory question.
- 5. Check the first box if a K/A mismatch flaw exists. Check the second box if the question is flawed because it is written at the wrong license level. Either condition results in an (U)nsatisfactory question.
- 6. Enter the questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-4.2.
- 7. Based on the review performed in steps 2-5, mark the question as (U)nsatisfactory, in need of (E)nhancement, or (S)atisfactory.
- 8. Fully explain the reason for any (U) in column 7 (e.g., how the psychometric attributes are not being met).
- 9. Save the initial review comments and detail subsequent comment resolution so that each exam-bound question is marked by an (S) on this form.
Form 2.3-5 Written Examination Review Worksheet Prairie Island 2022 August 8 - 12, 2022 13 NOTES