ML22147A185
| ML22147A185 | |
| Person / Time | |
|---|---|
| Site: | Arkansas Nuclear  |
| Issue date: | 04/13/2022 |
| From: | Heather Gepford Operations Branch IV |
| To: | Entergy Operations |
| References | |
| Download: ML22147A185 (19) | |
Text
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5
- 1. LOK 2. LOD 6
7 Ans Letter CFR 8
Q#
(F/H)
(1-5) stem focus cueing T/F cred dist partial job link minutia
- /
units back-ward K/A SRO-only B/M/N U/E/S A/B/C/
D 55.41x55
.43y Explanation REVISE TO ENSURE ANSWER DISTRIBUTUONS ARE MORE EVEN (at least 17 of each (ABCD) on RO; no more than 8 in any one of SRO) 1 H
3 X
N E
D 41.7 FIRST TEN. Change "wills" to "shoulds" in stem.
KS: Changed two "wills " to "shoulds " in the stem. Q now S 2
F 2
N S
C 41.10
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 to 5 (easy to difficult) rating scale (questions in the 2 to 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- a. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- b. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
- c. The answer choices are a collection of unrelated true/false statements.
- d. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- e. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- a. The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- b. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- c. The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- d. The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
Place the answer letter here Place the 55.41 or 55.43 item here
- 8. At a minimum, explain any Unsat ratings (e.g., how the Appendix B psychometric attributes are not being met). a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 3
H 3
X X
B E
A 41.7 ON 2017 NRC Exam. K/A description appears incomplete.P-36B appears in 2/4 distractors -
change B to say "Raise HPI flow until CET temperatures stablize"(?).
KS: corrected K/A description and edited distractor B. per discussion. Q now S 4
H 3
N S
C 41.10 5
H 3
N S
D 41.10 6
F 3
X X
M U
D 41.5 Correct answer appears obvious.
When would D ever be the wrong thing to do? A and B do not appear palusible given the initial conditions. While C may not be required by TS, it's still a prudent action that could be appealed as correct.
KS: deleted "Core offload in progress", added MODE 6, edited question sentence per discussion.
Q now S a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 7
H 4
X X
B E
A 41.6 FIRST TEN. ON 2016 NRC EXAM. Explanation for D suggests that it could be capable in some circumstances? Ensure we don't have a possible second correct answer. Why LOD=4?
KS: Updated Distractor "D" justification to include piping size (1/4 inch diameter) to prove a large leak could not exist in the PZR sample cooler. LOD 4 because very specific component knowledge (i.e. hx size and construction) is required to answer correctly. In stem, add OP-1203.039, Excess RCS leakage, has been entered.
KS: Added OP-1203.039 has been entered. Q now S 8
F 2
N S
C 41.1 9
F 2
B S
C 41.7 ON 2018 NRC EXAM.
10 F
3 X
B E
C 41.10 ON 1998 NRC EXAM. Add "Per 1202.006" in stem KS: added per OP-1202.006.Q now S 11 H
1 X
M U
A 41.7 Knowledge that only one tripped channel will not actuate ESAS equipment and that two tripped channels will appears to be LOD=1 knowledge and therefore not discriminating (B and C do not appear plausible)
KS: edited so that 1 channel from each parameter has tripped to make has NOT plausible. Q now S
12 H
3 N
S C
41.7 13 H
3 N
S D
41.7 a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 14 H
3 X
B U
A 41.10 B and D do not appear plausible as it's difficult to believe that wear on a machine is minimized by running it.
KS: modified distractor per discussion.Q now S 15 H
3 B
S A
41.10 ON 2001 NRC EXAM.
16 H
2 M
E C
41.7 ON 2013 NRC EXAM. Don't see much modification - probably still a bank question.
KS: Changed to Bank Q now S 17 H
3 B
S B
41.10 ON 2016 EXAM.
18 F
2 X
B E
C 41.10 ON 2007 NRC EXAM. C appears to have three reasons while A, B, and D only have one (modify distractors)? Is "maintaing" SCM the same thing as "extending the time period before SCM is lost?"
KS: modified correct answer per discussion.Q now S 19 H
3 X
B E
B 41.13 ON 2017 NRC EXAM. Ensure accident analysis for ejected rod doesn't encompass power peaking and shutdown margin design limits.
KS: edited distractor per discussion.Q now S 20 H
2 N
S A
41.10 21 H
3 N
S B
41.10 22 H
3 X
N E
D 41.7 Change D to "Fill from the BWST using gravity feed through idle P-34B Decay Heat Pump" to be more consistent with the other distractors?
KS: edited per discussion. Q now S a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 23 H
4 X
N E
C 41.10 Discuss as answer key appears to be incorrect(?). The time limit for exceeding 75 gpd and 30 gpm rate is 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> (not 2) if you account for the one hour to initiate shutdown. May need to modify distractors as if the 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> is mistakenly applied to the 75 GPD time the answer is also 1700.
KS: adjusted leakrate vs. time per discussion. Q now S 24 H
2 X
B E
D 41.7 ON 2004 NRC EXAM. C does not appear plausible (what's the potential integrity issue?) Make valve open instead of closed?
KS: deleted valve was closed per discussion. Q now S 25 H
2 X
N E
A 41.5 In stem, change "The most likely cause of these indications is" to "These indications are indicative of" KS: changed as suggested. Q now S 26 H
3 M
S A
41.7 27 H
3 X
N E
B 41.7 Discuss: would reactor power at 50% in stem make question more discriminating?
KS: Changed power to 50% Q S
28 H
2 X
N E
B 41.7 Add to before (2) in stem.
KS: changed as suggested. Q now S 29 F
2 N
S C
41.9 30 F
2 N
S A
41.7 31 F
3 M
S C
41.7 32 F
3 M
S A
41.7 ON 2016 NRC EXAM.
33 H
3 N
S D
41.7 ON 2017 NRC EXAM.
34 F
2 B
S B
41.7 a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 35 H
3 X
N E
C 43.5 In distractors C and D, RCS pressure would be 1650 psig and 1600 psig respectively. These don't seem plausible. Discuss possibility of changing indicated rates to increase distractor plausibility.
KS: edited rates as discussed.Q now S 36 F
3 X
B E
B 41.7 Discuss; ensure sufficiantly different than scenario 1 failure of PT-1021.
KS: failures are different as discussed. Q now S 37 F
2 X
B E
D 41.7 Discuss; would like to get a better idea of why distractors C and D are plausible.
KS: modified distractors as discussed. Q now S 38 F
2 B
S A
41.9 ON 2011 NRC EXAM.
39 F
3 N
S C
41.7 40 H
3 X
B U
D 41.5 ON 2016 NRC EXAM. Distractor A - don't see why anyone would think RB spray pumps would be required to be secured with higher than normal reactor building pressure following a LOCA?
Distractor B - requiring RB sprays to be secured with LPI flows above their minimum is not plausible. Distractor C - If one argued that C could cause D to happen, could it also be interpreted as a possible correct answer?
KS: replaced distractor A. Q now S a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 41 H
3 X
N E
B 41.5 Discuss; not sure anyone would think it was OK to attempt a pump start without at least looking at the breaker. Understanding that one train of ESF equioment satisfies a safety function is a pretty fundamental (near LOD=1) concept.
KS: Discussed and OK as is. Q now S 42 H
3 N
S B
41.5 43 H
3 N
S B
41.5 44 H
3 X
N E
D 41.7 Needs more of a reference than "ran on simulator" to justify (attach graph?)
KS: retained record of graph.Q now S 45 H
3 X
X N
E D
41.7 Two correct answers as written. If suction is required to be transferred at 5.1 ft, it will still be required to be transferred at 4 ft.
In stem, specify "below a MAXIMUM of" for level.
KS: edited stem as discussed. Q now S 46 H
3 X
M E
B 41.5 In stem place commas before and after Degraded Power KS: added commas Q now S 47 H
3 X
X B
E D
41.10 ON 2010 NRC EXAM. Change "would" to "should" in stem. Don't see how info in stem has anything to do with reactor coolant pump status (distractor C).
KS: modified distractor C. Q now S
48 F
3 X
B U
B 41.7 ON 2010 NRC EXAM. Discuss; taking "manual" control and relying on a "mechanical" trip with no DC control power appears obvious to me.
KS: Selected new K/A and different question. Q now S 49 F
3 N
S A
41.7 a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 50 H
3 X
X N
E B
41.5 Ensure knowledge is not SRO only (TS evaluation for PRMs)?
Discuss; not sure we are interested in testing knowledge of when TS don't apply.
KS: Discussed and OK as is. Q S
51 F
2 X
X B
U D
41.7 ON 2010 NRC EXAM. K/A mismatch - K/A is asking for radiation theory, including sources, types, units, and effects (not detectors). Don't see how either C or D could be interpreted as plausible.
KS: New question. On new Q change beta to CO-60 pulses. Q now S 52 H
2 N
S C
41.7 53 F
2 X
B U
B 41.9 ON 2017 NRC EXAM. Not plausible to believe that air is provided to MSIV's to primarily supply a source of air for ADV's (A and C). If D was correct, so would B (renders D implausible).
B only plausible correct answer.
KS: edited stem to make ADVs plausible. reformatted to 2x2. Q now S 54 F
3 X
N E
A 41.5 Change "will's" to "should's" in stem.
KS: modified stem as discussed.
Q now S 55 F
3 M
S D
41.7 56 H
3 X
N E
D 41.7 In stem add "a MAXIMUM of" after "below" KS: modified stem as discussed.
Q now S 57 F
3 N
S B
41.5 a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 58 H
3 X
X B
U D
41.5 In stem, what is the difference between "seized" and "failed?"
Doesn't appear reasonable that feedwater heating would remain stable given the conditions in the stem. This renders only A and C as plausible answers.
KS: Changed seized to failed.
Added bullet in stem about Main Turbine. Q now S 59 H
3 B
S B
41.5 ON 2016 NRC EXAM.
60 F
3 N
S A
41.10 TVGV - "throttle valve governor valve?"
KS: spelled out throttle valve governor valve in stem. Q now S 61 H
2 X
N E
B 41.7 D is essentially "none of the above" and should be avoided.
Delete "interlocked" in A and C (pump trips/recircs back).
KS: Modified distractor D. as discussed. Deleted "interlocked to" in other distractors. Q now S 62 F
2 X
B U
A 41.7 ON 2020 NRC EXAM. Discuss; if choice (2) was correct, how could choices (3) and (4) be possible? If not, eliminates C and D as plausible answers?
KS: Changed 2. to Waste Gas Compressor starts. Added word "immediately" in stem. Q now S 63 H
3 X
N E
B 41.5 In stem, change "will align in approximately" to "should automatically align in." Add "the" prior to the last "Instrument Air System."
KS: modified stem as discussed.
Q now S 64 F
3 X
B E
D 41.10 Capitalize the two MINIMUM's in the stem.
KS: capilatized minumum as suggested. Q now S a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 65 F
2 X
X B
U C
41.5 Question is system specific to RCPs (not generic as per K/A).
Distractor D describes an indication that is impossible?
KS: wrote new generic question Q now S 66 F
2 M
S C
41.13 67 F
2 X
B U
B 41.10 No correct answer as written.
Distractor A indicates to "replace" the tag without clearing it first, as specified in both distractors B and C (according to reference it should be cleared).. Three of four distractors appear to be "clear and replace" with the exception of distractor D (cues D not correct).Make D clear and replace with new/different danger tag upon replacement? Don't see the need for "immediately."(discuss)
KS: revised stem and distractor D. Q now S 68 F
3 X
X X
X B
U C
41.12 ON 2017 NRC EXAM. In stem, what is the reason to cue the appliant that the area is a locked high radiaiton area when the dose rate is given? Distractor A could easily be argued correct. B not plausible, as there are no contamination levels given in stem.
KS: Replaced Distractor A. Edited stem per discussion Q now S a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 69 F
2 X
N U
C 41.10 If one performs (1), then the lilelihood of (2) is minimized.I can think of no circumstance when (1) is ever wrong. No one can gurantee that a particular action WILL prevent equipment damage
((2) not plausible). The notion that SM/CRS approval is not necessary to peform a "simple routine" action that is outside of procedural guidance doesn't seem likely.
KS: New K/A and new question as discussed Q now S 70 F
2 B
S A
41.1 ON 2014 NRC EXAM.
71 F
3 X
X N
E A
41.1 Change "at approximately" to "nearest to." Dicuss; 3.0 DPM seems extremely large to be plausible.
KS: edited stem and lowered distractor from 3.0 dpm to 1.5 dpm. Q now S 72 H
3 X
M E
A 41.1 Change "will" to "should" in stem.
KS: Edited stem as requested. Q now S 73 H
3 B
S C
41.14 74 F
2 X
B E
D 41.14 Modify stem to "ANO1 Unit 1 reactor vessel" to make appear more plant specific KS: Edited stem as requested. Q now S 75 H
2 M
S A
41.14 a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 76 H
3 X
X N
E D
43.5 FIRST TEN. Probably need RCS temperature decreasing along with a SG pressure drop to make overcooling more plausible. "OR as determined by reactor engineering" kind of cues that either A or C is correct. Really LOD=4? (I got it right away)
KS: Added "RCS temeperature is LOWERING " in the stem.
Deleted "OR as determined by Reactor Engineering " in answer choices to remove queing.
Changed LOD to 3.
KS. Rev. 1: based on validation comments replaced 2nd part with EAL Classification. Question now appears too similar to operating test scenario 1 - discuss; may have to modify or replace.
KS: Different actions as discussed. No change made. Q now S 77 H
3 N
S A
43.5 FIRST TEN.
78 H
3 B
E D
43.5 FIRST TEN. Q not changed enough to be considered modified (same concept and answer). ON 2014 NRC EXAM KS: Changed from Modified to Bank.
KS: After validation changed from LOD 4 to LOD 3.Q now S 79 H
3 N
S B
43.5 80 H
3 N
S A
43.5 a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 81 H
3 X
X B
E B
43.5 ON 2017 NRC RETAKE EXAM.
The problem and answer are the same as original question (should remain as a bank question).
Discuss - if the SRO ordered both B and D performed, would they be "wrong?" May want to rephrase stem to ask what is required vice what is correct.
KS: asked what is required per discussion. Q now S 82 H
3 X
B U
B 43.5 ON 2011 NRC EXAM. Possible two correct answers. If one performs A, then isn't B satisfied?
KS: Replaced distractor A. Q now S
83 H
2 B
S D
43.1 ON 2016 NRC EXAM.
84 H
3 X
X B
E D
43.5 ON 2017 NRC RETAKE EXAM.
Change "will" to "should" in stem.
Discuss - can't distractor B be eliminated without reading the question (plausibility)?
KS: changed "will" to "should", left the rest as is per discussion. Q now S 85 H
3 X
N E
B 43.5 REFERENCE PROVIDED.
Where is the attached SPDS trend? Do you also intend to provide procedure 1203.25 (that would make it a direct lookup)?
KS: graph attached to question page that student receives. Q now S a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 86 H
3 X
N E
C 43.2 FIRST TEN. REFERENCE PROVIDED. Insert comma in 10,000 in stem. Change "ensure" to "provide" in stem (assured/ensure seems awkward). Why is a handout of TRM 3.5.1 necessary to answer this Q?
KS: Added comma in 5,000 and 10,000. Replaced the first "ensure " with "provide ".
Removed TRM 3.5.1 and figure from student handout.
KS: After validation changed from LOD 4 to LOD 3.Question still indicates a reference is provided.
KS: removed TRM 3.5.1 as reference, the others are correct.
Q now S 87 H
3 X
N U
B 43.5 Don't see how C and D are plausible with conditions given.
Was this intended to be closed reference?
KS: After discussion okay as is. Q now S a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 88 H
4 X
N E
A 43.5 FIRST TEN. Discuss plausibility of C(1) and D(1) i.e. why would one believe power would decrease when drawing more steam out of the plant given no reactor trip occurs (based on the procedure choices).
KS: changed stem to ask "Governor Valve Demand "
instead of "Reactor Power ". It is more plausible for governor valve demand to lower as a result of Steam Header pressure lowering.
KS: based on validation comments, removed bullet on SG "A" pressure and added bullet on MWe.
After validation changed from LOD 3 to LOD 4. Q now S 89 H
3 B
E A
43.5 FIRST TEN. Still a bank Q. Stem and distractors were just reworded, not "modified." (same concept and answer)
KS: Changed from Modified to Bank Q now S 90 H
3 X
X N
E C
43.5 Couldn't one argue that D is also correct, because it is actually required to be performed and could potentially solve the problem? (discuss; at a minimum ENSURE should be capitalized in the stem)
KS: capitalized "required" and "ensure". Q now S a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 91 H
2 X
N E
C 43.5 FIRST TEN. This Q is essentially covering two different concepts into one Q, one which matches the K/A (first part), and one which doesn't (second part). 2X2 questions don't necessarily have to have both parts directly addressing the K/A, but they need to be related to each other. Power supply loss or erratic operation of an NI is a totally different concept and unrelated to reactor trip bases on a rod withdrawal accident. Second half of question needs to be replaced with something that is related to the first half of the Q.
KS: Replaced question with New QID 1332.Q now S 92 F
3 B
S C
43.5 93 H
3 X
N E
B 43.5 In stem in (2), change "the initial procedure" to "OP-1203.008" KS: changed as requested. Q now S 94 F
3 B
S A
43.5 95 F
2 X
X B
U D
43.7 ON 2020 NRC EXAM. In stem change (1) to "What is the MINIMUM required time" and (2) to "Who is REQUIRED to" Since the SM is the ultimate authority on shift, why could refuleing commence without his permission? Possible two correct answers?
KS: edited stem as discussed. Q now S 96 F
3 X
N E
C 43.5 Discuss; Since the reference specifies that this activity is performed in conjunction with another individual, is distractor C really entirely correct?
KS: discussed, OK as is. Q now S 97 F
3 X
N E
C 43.3 Change will to should in stem.
Should "PAD" be defined?
KS: changed as requested. Q now S a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 98 F
2 X
X N
E D
43.4 FIRST TEN. Spell out "two" in stem. Isn't SM/CRS approval required for all releases? If so, this pretty much makes A or C the only plausible disctractors. Also, one need only pick the most conservative action in first half of Q to come up with the correct answer. Could Q be revised to ask if someone in addition to the SM/CRS is required and add a bogus requirement that makes the lesser conservative answer correct?
KS: Changed "2 " to "two ".
Reconstructed the question to select from a list the correct combination of actions that require two independent verifications. No longer a 2x2. Q now S 99 F
3 X
B E
D 43.1 Doesn't the Emergency Plant Manager act as the Emergency Director until the EOF is manned?
KS: No, SM acts then transfers directly to ED. Q now S 100 F
3 X
B E
A 43.5 ON 2017 RETAKE EXAM. In stem (2), PAR should be PARs KS: changed as requested. Q now S Results Table RO LOK -H 42 Avg RO LOD 2.65 Flaws 10 CFR Distribution RO LOK-F 33 AVG SRO LOD 2.88 Stem focus 28 41.1 20 43.1 2 SRO LOK - H 17 Overall LOD 2.71 Cues 3
41.2 0
43.2 1 SRO LOK - F 8
T/F 0
41.3 0
43.3 1 Cred Dist 33 41.4 0
43.4 1 RO Bank 29 38.67 SRO Bank 11 44 Partial 9
41.5 14 43.5 20 RO Mod 11 14.67 SRO Mod 0
0 job link 0
41.6 1
43.6 0 a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 RO New 35 46.67 SRO New 14 56 units 1
41.7 30 43.7 1 minutia 0
41.8 0
Total Bank 40 40 backward 0
41.9 3
Total Mod 11 11 KA 3
41.10 20 Total New 49 49 SRO-only 1
41.11 0
LOD = 1 1
41.12 1
24 41.13 2
12 Answer Dist ( in %)
41.14 3
RO Edit 30 40 SRO Edit 16 64 RO-A 19 SRO-A 6
RO-B 19 SRO-B 6
Total Sat 38 38 Total Unsat 16 16 RO-C 19 SRO-C 6
Total Edit 46 46 RO-D 18 SRO-D 7 a Rev 2
Form 2.3-5 ANO1-2022-04 Written Examination Review Worksheet Form 2.3-5 a Rev 2