ML16204A022
ML16204A022 | |
Person / Time | |
---|---|
Site: | Prairie Island ![]() |
Issue date: | 06/10/2016 |
From: | Mcneil D NRC/RGN-III/DRS/OLB |
To: | Northern States Power Company, Minnesota |
Shared Package | |
ML15274A272 | List: |
References | |
50-282/OL-16-301, 50-306/OL-16-301 | |
Download: ML16204A022 (6) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9
- questions reviewed in depth.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only
- 1 H 2 N S 2 H 2 N S
- 3 H 2 N S 4 H 2 B S
- 5 F 2 N S 6 H 2 N S
- 7 H 2 B S 8 H 2 N S Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
- The answer choices are a collection of unrelated true/false statements.
- The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A, and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 9 F 1 N U LOD = 1 Replaced Question 10 H 2 M S
- 11 M 2 B S 12 M 2 N S
- 13 M 2 N S 14 H 2 B S 15 F 2 N S 16 F 2 N S One implausible distractor. Establishing RCS bleed and feed wont
- 17 F 2 N E involve Aux Feed Replaced distractor.
18 F 2 N S
- 19 H 2 N S 20 H 2.5 N S Bullet #4: No operator actions have been taken. Moved bullet to 21 H 2 M E better location.
22 H 2 N S
- 23 H 2 N S 24 H 2.5 N S 25 F 2 N S 26 H 2 N S 27 H 2.5 N S 28 F 2 N S
- 29 H 2 N S 30 H 2 N S
- 31 F 2 B S
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 32 F 2 N S 33 F 1 B U LOD=1 Reworked question.
34 F 2 N S 35 H 2 N S 36 H 2 N S LOD=1. I was able to guess the correct answer without looking at
- 37 H 1 B UE OPT: Modified one distractor and made question satisfactory.
LOD-1. Doesnt completely test the K/A. (2X2) Reworked 38 H 1 N U distractors - matches K/A.
39 F 2 N S 40 H 2 N S
- 41 H 2 N S 42 H 2 N S Not fundamental. Requires applicant to determine a correct
- 43 H 2 N E response to an error.
QK/A. The Q does not accurately test the K/A. In order to use the 44 F 2 N US procedure to correct the condition, the impact has to be known.
Q does match the K/A.
45 H 2 N S 46 F 1 N U LOD=1. Reworked question.
- 47 H 2 N S 48 H 2 N S
- 49 F 2 B S 50 F 2 N S
- 51 H 2 N S 52 F 2 B S
- 53 F 2 N S 54 F 2 B S
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 55 H 2 B S 56 H 2 N S
- 57 H 2 N S 58 F 2 N S
- 59 H 2 N S 60 H 2 N S
- 61 F 2 N S 62 F 2 N S 63 F 1 N U LOD=1. Reworked question.
64 H 2 M U QK/A Reworked question.
65 F 1 N US LOD=1. Discussed w/ licensee, Q is LOD = 2.
66 F 1 N U LOD=1. Replaced K/A, wrote new question.
- 67 F 1 N US LOD=1 Discussed question - LOD = 2.
68 H 2 N S Two distractors contain Digital Rod Control IN positions when the 69 H 2 N U control rods have to be withdrawn. Implausible distractors.
Reworked distractors.
70 H 2 N S Do we have any immediately action statements in section 3? If not,
- 71 F 2 N E change to some time limit. Suggest going to a (1) ; (2) format. Many Tech Spec Immediately actions. Fixed format.
Distractors C & D are implausible. The picture shows that valve CV-72 F 2 N U 31415 is associated with B SG Blowdown, not SG 11. Replaced question.
Distractor c states the purpose is to minimize the cooldown capability of the non-faulted S/G. Why would we want to do that? Appears to
- 73 F 2 B E be implausible. See if we can replace. Replaced distractor -
changed one initial condition.
74 F 2 N S 75 H 1 N U LOD=1 Reworked question.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only
- 76 H 2.5 B S 77 H 1 N U LOD=1. Direct look-up. Replaced question.
78 H 1 M U LOD=1. Direct look-up. Replaced question.
79# H 2 N S 80 H 2 N S 81 H 2.5 N S LOD=1 or 2 implausible distractors - procedure is a give-away.
82 H 2 M S Procedure is not a give-away. Changing bullet one changes correct procedure. LOD = 2.
- 83 H 2 N S 84 F 2 N S Add the rad monitor annunciators to comply with the K/A Statement.
85 H 2 N E Added rad monitors.
LOD = 1. Discussed with licensee, determined the question was 86 H 2 B US LOD = 2.
- 87 H 2 N S 88 H 2 N S 89 H 2.5 B S 90 H 2 N S
- 91 H 2 M S Verify this is not a direct look-up. Not a direct look-up.
Q does not address a portion of the K/A. Discussed question -
92 H 2 N US determined question meets the K/A.
93 H 2 B S Modify question to more fully address the K/A.
94 H 2 B S 95 F 1 N US LOD=1. After discussion with licensee, determined LOD = 2 96 H 2 N S
- 97 F 2 B S 98 H 2 N S
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only What is the correct answer? The Answer key says one distractor, but 99 H 2 N S the word description for justifying the distractor states another distractor is the correct answer. Explanation corrected.
100 H 2 N S