ML23151A736
| ML23151A736 | |
| Person / Time | |
|---|---|
| Site: | Byron  |
| Issue date: | 02/27/2023 |
| From: | NRC/RGN-III/DORS/OB |
| To: | NRC/RGN-III |
| Bruce Bartlett | |
| Shared Package | |
| ML19121A194 | List: |
| References | |
| Download: ML23151A736 (1) | |
Text
3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation Generic Comments
- 1. The Level of Difficulty is missing. Not required to be completed.
- 2. Blue color are Free Look questions 1
H 3
N S
- 1. The distractors need additional information. More information in support of the plausibility would be helpful. Answer A and D need procedure references.
2 H
3 X
N U
S
- 1. Question appears to be SRO-only; requiring the applicant to diagnose plant conditions, determine the corrective action and the bases for that action.
Bases portion removed; question is now satisfactory.
3 F
2 N
E S
- 1. The question information states that it meets Tier 1 and then goes on to say that it is a Tier 3.
Which is it and why? Tier 3 box will be deleted for this and other Generic K/As.
The question was changed and the changed question is satisfactory.
4 F
2 N
S
- 1. Add In accordance with to 2BOA PRI-1, what actions will the RO take?
Otherwise, the question is satisfactory.
5 H
3 B
S 6
H 3
N E
S
- 1. Direct overlap with scenario 2 event 4. Recommend changing
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation failure mode in the question to fails high.
Scenario modified; question is satisfactory.
7 F
2 N
E S
- 1. Question stem states that the PORV is capable of being manually cycled, however, answer explanation C (supposedly the correct answer) states that the PORV is not capable of being manually cycled. Answer explanation D, as 3.4.11 Condition A appears to be the correct LCO to enter with a PORV that IS capable of being manually cycled, NOT Cond B. Revise the answer explanations as appropriate.
Corrected; question is now satisfactory.
8 F
2 B
E S
- 1. Which OWS graphic is correct to trip the turbine, 5502 or 5512?
Q8 references 5512, while scenario 1 event 9 references 5502.
Scenario event corrected; question is now satisfactory.
9 F
3 B
S 10 H
3 N
E S
- 1. What is the basis for the 25%
value? If it is because it is less than 27% Adverse Containment then why isnt it 26% or some other value? Is 8% selected
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation because it is less than 10%
(Non-Adverse Containment)?
Again, why this value instead of some other value? Normally a value is selected because it matches a value that the candidate is used to seeing it is just placed in a condition that is not appropriate. Question changed and corrected.
- 2. The answer explanation for A and B is exactly the same, they should be better differentiated.
Answer C explanation should reflect that it comes from 1BFR-H.1 Question changed and corrected.
(27% Adverse) AND no feed flow established I believe that it could be argued that neither answer B nor D is correct.
Multiple correct answers is UNSAT. Ask the question differently. When is dryout?
Question changed and corrected.
- 4. Will work on trying to rewrite the question.
The question was successfully revised and is now acceptable.
11 F
2 B
S
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation 12 F
3 B
S 13 H
3 B
S 14 H
3 M
S
- 1. What purpose does part 2 of the question add? Not required for the K/A. Is it just for making this a 2x2 question?
15 H
3 B
S 16 H
3 B
S 17 H
2 N
S
- 1. Add 1BOA ELEC-5 reference of Att. C item 1 for jumper point in the answer explanation.
- 2. The Tier 1 justification implies that the BOA will be given to the applicants where the K/A justification implies the drawing will be given to the applicants.
- 3. Give coordinates for the jumpers on the drawing in the answer explanation.
18 H
3 N
E S
Yes - question is satisfactory.
19 F
3 N
S 20 H
3 B
S 21 F
2 N
S
- 1. Consider adding initial conditions or stating that the
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation 1BOA SEC-8 was entered to address the tube leak, as the applicant can reasonably assume that the tube leak has propagated such that 1BEP-3 has been entered, and answer D is correct (no time duration, leak rate, etc. is given to invalidate this assumption).
22 F
2 B
S 23 F
2 B
S 24 F
2 X
N U
S
- 1. Not tier 1 question (most closely tier 3).
Question can be answered without providing the initial conditions - which makes it a tier 3 generic and not meet the K/A.
Question revised to better meet K/A; question is now satisfactory.
25 F
2 X
B U
S
- 1. Why are distractors A & B plausible? Eliminated by stating that CST/RWST level transfer to containment would be expected. The initial conditions state that ECCS is on sump suction. The only information available is that the crew has entered 1BFR-Z.2.
- 2. Place a space in the first stem bullet between from and 100%
- 3. Include the Orange Path entry conditions in the references.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation Corrected; question is now satisfactory.
26 H
2 B
S 27 F
2 N
S 28 F
3 M
S 29 H
3 N
S 30 H
3 B
S 31 H
3 N
S 32 F
2 N
S 33 F
2 B
S 34 H
3 N
S 35 H
3 B
S 36 H
3 B
E S
- 1. Answer explanation A - change loss in the first sentence to lose.
Corrected; question is now satisfactory.
37 H
3 B
S 38 H
3 B
S 39 H
2 B
S 40 H
3 N
S
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation 41 F
2 N
E S
- 1. Give the step reference in addition to the procedure reference when completing answer explanation D
- 2. Consider emphasizing when the unit reaches full power in part 2 of the stem, or rephrasing to state At 100% power...
Corrected; question is now satisfactory.
42 F
2 N
S 43 H
3 B
S
- 1. Also operating experience from an actual Byron plant event 44 F
2 B
E S
- 1. Memory needed to reach the correct answer no analysis or evaluation is required. It is not a High Cognitive level question.
Corrected; question is now satisfactory.
45 H
3 B
E S
- 1. Should the answer be given in minutes or seconds? The BAR and the print are in minutes.
Either; question is satisfactory.
46 F
3 B
E S
- 1. Memory needed to reach the correct answer no analysis or evaluation is required. It is not a High Cognitive level question.
- 2. Kv in the stem should be kV Corrected; question is now satisfactory.
47 F
2 N
S
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation 48 F
2 N
S 49 H
3 X
M U
S
- 1. More discussion will be needed as to whether the question has been modified sufficiently.
- 2. Mostly memory involved, very little to any assessment or evaluation. Why is this question at a High cognitive level?
Modified Stem different answer is now correct: Multiple decision points make question High cog; question is now satisfactory.
50 F
2 B
S 51 F
2 B
S 52 F
2 X
N U
S
- 1. For the question submitted the KA asks for Function AND Operation. Suggest rewriting the question in reverse. Ask what parameter controls the valve (pressure/temperature) and in which direction (open/closed). This will hit both function and operation and then would be a 2X2 style question.
- 2. Consider rephrasing stem for ease of reading: What parameter controls the valve position of 1SX147A,, during normal plant operations?
Corrected; question is now satisfactory.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation 53 H
3 B
S 54 H
3 N
E S
- 1. SI and MS isolation does not HAVE to actuate. This makes this distractors too easy to eliminate. Change to Safety Injection and MS Isolation will actuate to...
Question is now satisfactory.
55 F
2 X
N U
S
- 2. Why is D also not a correct answer? Could change the question stem to which Tech Spec is required first.
Corrected; question is now satisfactory.
56 F
2 B
S 57 H
3 N
E S
- 1. Answer explanations B and D do not explain why a rising SFP level indication is plausible.
because it is opposite the correct answer is not sufficient justification.
Distractor analysis modified; Corrected; question is now satisfactory.
58 H
3 N
S 59 H
3 N
S
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation 60 H
3 N
S 61 F
2 B
S 62 F
2 N
S 63 F
2 B
S 64 F
2 N
S 65 H
3 B
S 66 F
2 N
S 67 F
2 N
S 68 F
2 B
S 69 F
2 B
E S
- 1. What is the purpose of showing locations 2 or 6 when theyre not an answer choice? Delete them and show only the four answer choices.
- 2. Why is location 4 a different shading than the others?
Cueing? Make them all look the same.
Corrected; Corrected; question is now satisfactory.
70 H
3 M
S 71 H
3 M
S 72 H
3 B
S
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation 73 H
3 N
S 74 H
3 B
E S
- 1. The stem is written differently than the other stems when it comes to being in an EOP.
Change the first line to The crew entered 1BEP-1.
- 2. Show additional work on getting the correct answers.
Corrected; Corrected; question is now satisfactory.
75 F
2 B
S 76 H
3 N
S 77 H
3 B
E S
- 1.
Part 2 of the question is unnecessary to meet the K/A.
Part 2 also tests BAP 1450-T2 knowledge, where Q80 also tests knowledge of the T-sheet.
Recommend modifying part 2 of the question to minimize testing of the containment entry checklist.
Corrected; Corrected; question is now satisfactory.
78 H
3 B
S
- 1. Reorder answers from longest to shortest or from shortest to longest 79 H
3 B
S 80 F
2 B
S
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation 81 H
3 B
S 82 H
3 N
S 83 F
3 B
E S
- 1.
Question is acceptable as is but consider enhancing distractor A
& B explanations to include how pressure affects OTDT.
Distractor analysis modified; question is satisfactory.
84 H
3 N
S 85 F
3 M
S 86 H
2 N
S 87 F
3 B
S 88 H
3 X
N E
S
- 1. Changed to 1BGP 100-4 to have a larger change in power.
Will also change the stem to focus on the power reduction instead of the securing of AFW.
- 2. Corrected; question is now satisfactory.
89 H
3 N
S 90 F
2 N
S 91 F
3 B
E S
- 1. Recommend putting 1 in the stem as (1), as the applicant will likely read the 1 as 1 rem.
- 2. Corrected; question is now satisfactory.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia
- /Units Logic Q-K/A License Level
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8.
Explanation 92 F
2 X
N U
S
- 1. Per OP-AA-109-101-1002 Section 4.1.5, it appears that only 25% of the tagouts hanging longer than 2 months need to be validated. In which case, the applicant may argue that annually is a correct answer.
Reworded the stem question; question is now satisfactory.
93 H
3 M
E S
- 1. Why was a modified question used instead of the original question? Explain why this question meets the K/A as written. The question does not meet the K/A.
Question stem modified to better match the K/A; question is now satisfactory.
94 F
3 M
S 95 F
2 B
S 96 F
2 B
S 97 H
3 N
S 98 F
2 B
S Previous 2 exams (Byron 2021) 99 F
2 B
S 100 F
2 B
S
Form 2.3-5 Instructions for Written Examination Review Worksheet Refer to ES-4.2 for the definitions of terms used in this worksheet for the written examination.
Review each question (Q) as submitted and as subsequently revised and document the following in the associated worksheet columns:
- 1. Enter the level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) from 1 (easy) to 5 (difficult); mark direct lookup questions (applicant can directly determine the answer from the provided reference) as LOD 1. A question is (U)nsatisfactory if it is LOD 1 or LOD 5.
- 3. Check the appropriate box if a psychometric flaw is identified:
Stem Focus: The stem lacks enough focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information). This is an (U)nsatisfactory question.
Cues: The stem or one or more answer choices contains cues (e.g., clues, specific determiners, phrasing, length). This is an (U)nsatisfactory question.
T/F: All of the answer choices are a collection of unrelated true/false statements. This is an (U)nsatisfactory question.
Cred. Dist.: The distractors are not credible; single implausible distractors require (E)nhancement, and more than one noncredible distractor in the same question results in an (U)satisfactory question.
Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem). This is an (U)nsatisfactory question.
- 4. Check the appropriate box if a job content flaw is identified:
Job Link: The question is not linked to the job requirements (i.e., the question has a valid knowledge or ability (K/A) but, as written, is not operational in content). This is an (U)nsatisfactory question.
Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory). This is an (U)nsatisfactory question.
- /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). This is an (U)nsatisfactory question.
Logic: The question requires backward or reverse logic or application compared to the job requirements. This is an (U)nsatisfactory question.
- 5. Check the first box if a K/A mismatch flaw exists. Check the second box if the question is flawed because it is written at the wrong license level. Either condition results in an (U)nsatisfactory question.
- 6. Enter the questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-4.2.
- 7. Based on the review performed in steps 2-5, mark the question as (U)nsatisfactory, in need of (E)nhancement, or (S)atisfactory.
- 8. Fully explain the reason for any (U) in column 7 (e.g., how the psychometric attributes are not being met).
- 9. Save the initial review comments and detail subsequent comment resolution so that each exam-bound question is marked by an (S) on this form.