ML21147A538
| ML21147A538 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 05/06/2021 |
| From: | Heather Gepford Operations Branch IV |
| To: | Arizona Public Service Co |
| References | |
| Download: ML21147A538 (15) | |
Text
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9
- 1. LOK 2. LOD 6
7 Ans Letter CFR 8
Q#
(F/H)
(1-5) stem focus cueing T/F cred dist partial job link minutia
- /
units back-ward K/A SRO-only B/M/N U/E/S A/B/C/
D 55.41x55
.43y Explanation SRO section has 10 questions where the answer is "A" and only 3 where the answer is "B" -
Discuss redistributing.
1 H
3 X
M E
B 41.10 ON 2016 EXAM. Currently in the syle of a "none of the above" question. Discuss restructuring to have something actually wrong and in a 2X2 format. This could also make this a modified bank question. I understand the comment. I dont like all questions like this to have something wrong but I will change to address the comment.
Revised now modified. Q now S
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 to 5 (easy to difficult) rating scale (questions in the 2 to 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- a. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- b. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
- c. The answer choices are a collection of unrelated true/false statements.
- d. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- e. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- a. The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- b. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- c. The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- d. The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
Place the answer letter here Place the 55.41 or 55.43 item here
- 8. At a minimum, explain any Unsat ratings (e.g., how the Appendix B psychometric attributes are not being met). a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 2
H 3
X N
U A
41.10 STEAM TABLES PROVIDED(? -
Q says no reference provided). A appears to be easily eliminated at it essentially says "do nothing." D appears to be easily eliminated because it is the only distractor that has "insufficient" in the first half + if it were correct one wouldn't need to know the second half information. a reference "not" provided since the steam tables are available for use on every question. I understand the comment on A, but D is a good distractor since there are really three things the operators look at for RCP operation - Is RCS pressure below the SIAS setpoint? If yes, secure one RCP in each loop. Is the RCS at least 24°F subcooled? If not, secure all 4 RCPs. Is there cooling to the RCPs (cooling is lost on a CSAS or on a SIAS if NC and EW are cross-connected). If no cooling is available, all 4 RCPs are stopped. So the question is asking them to evaluate all 3 items - subcooling, pressure above/below the SIAS setpoint, and the status of cooling to the RCPs. Based on this, I think all 4 answers are plausible. Q Revised. Q now S 3
H 2
N S
B 41.7 4
H 3
B S
B 41.3 5
H 3
N S
C 41.10 6
H 3
N S
D 41.8 7
H 2
N S
B 41.4 a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 8
H 3
X N
E A
41.3 Ensure not too similar to event 3 in scenario 1. The event referenced is an RPS Pressurizer Pressure transmitter failure which has zero impact to the PPCS system. No overlap here.
Ensured. Q now S 9
H 3
X N
E C
41.6 Change "will's" to "should's" in stem. Revised. Q now S 10 H
3 X
N E
B 41.10 Change "will's to "should's" in stem. Revised. Q now S 11 F
3 X
N E
D 41.10 Why use "rebound" when the term "dryout" is used in the reference? Dryout and rebound are used somewhat interchangibly at PVNGS, although when RCS temperature begins to rise it is most commonly referred to as rebound. The term rebound is in the same paragraph as dryout in the provided reference in the worksheet (first sentence references temperature rebound). Plant terminology OK, Q now S 12 H
3 X
N U
A 41.1 This question provides a cue to eliminate two distactors in Q 27.
(i.e. that condensate can be established on a LOAF without the use of a functional recovery procedure) Appendix 44, Feeding with Condensate Pumps, is directed from both the LOAF and Functional Recovery Procedures and Q12 doesnt specify which EOP the CRS is operating in when the direction to perform Appendix 44 is given so I dont really agree that this is cueing the correct answer to Q27.
Explanation OK, Q now S 13 F
3 B
S B
41.14 ON 2020 NRC EXAM.
14 H
3 N
S C
41.7 a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 15 H
3 N
S A
41.7 Change "will's" to "should's" in stem. Revised. Q now S 16 F
3 X
X N
U C
41.8 Doesn't appear plausible that an RCS/NC intersystem LOCA would be associated with a CIAS or a CSAS (distractors B and D) While I understand the comment, the NC CIVs auto close on a CSAS and closure of the NC CIVs is one of the major mitigating actions for an RCS to NC intersystem LOCA so this is a pretty plausible distractor. This question has actually been missed by 3 of 16 validators (19%),each of whom picked B or D. I could come up with a different distractor for B and D part 2, but I think that another distractor would be less plausible. Explanation OK.
Revised stem. Q now S 17 H
4 N
S D
41.4 18 H
3 N
S C
41.4 19 H
3 N
S B
41.6 a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 20 H
3 X
X B
E A
41.3 ON 2016 NRC EXAM. Delete "ONE" in the stem. Distractor D appears to need to be reworded (Makeup the CHRG PMPS?).
Why is it necessary to have the depressurization info in the stem as it tends to cue the correct answer (RWST is only distractor that has to do with HPSI injection)? Deleted ONE. Fixed Makeup to CHRG PMPS. The reason for saying that depressurization is not desired is that depressurizing using HPSI is a valid option for borating the RCS (although not desired). The level in the RWT locks out the Boric Acid Makeup Pumps which is what makes A the correct answer, it really has nothing to do with the HPSI Pumps normally taking a suction from the RWT.
And even at 65%, the RWT is a viable source for HPSI injection.
Revised. Q now S 21 H
3 X
N E
D 41.10 Discuss K/A applicability - not sure procedurally turning off SU NI channels constitues a loss of intermediate range instrumentatioan. I understand that taking a component out of service isnt exactly the same as a component failing, however in order the address the locate and operate components, including local controls portion of the KA, it made the most sense to lose the NI by being procedurally required to remove it from service. I think its the best balance matching both the system and text of the KA.
Explanation OK. Q now S a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 22 H
3 X
X N
U D
41.11 Acronymn in stem cues that it is a "purge" isolation - only C and D specifically refer to "purge,"
apparently rendering A and B implausible. Revised. Q now S 23 F
3 X
X X
N U
D 41.12 Q does not indicate if SG is dry or has inventory nor do the references support the reason for the flow rate. Since thermal stress of U-tubes is almost always a concern, it appears one could make an argument for any of the four distractors to be the correct answer. Also, K/A does not appear to match as there is no accidiental liquid radwast release indicated here. I dont believe that the status of inventory in the SG is needed in the question but I did not find anything that conclusively said if the 1000 gpm limit does or does not take precedence over feeding at 1360-1600 gpm during a
SGTR/EDS so I added that the affected SG is currently 10% WR and slowly lowering. Added a reference showing that the reason for the feedrate is based on minimizing release to the environment. The KA was matched, in our opinion, by asking about an accidental release mitigated in the EOPs. A radioactive release due to a SGTR/ESD is certainly accidental so we felt like this was the best match to the spirit of the KA.
Revised. Q now S 24 F
2 B
S D
41.11 ON 2019 NRC EXAM.
25 H
3 B
S C
41.10 ON 2016 NRC EXAM.
26 F
3 B
S A
41.10 ON 2018 NRC EXAM. a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 27 H
3 X
X N
E D
41.10 Ensure RO level of knowledge.
Also, this Q is cued by Q 12 (see Q12 comments). I think this is on the cusp of RO vs SRO knowledge, but feel like this question is fair for the RO exam.
See comments on Q12 regarding the cueing between Q12 and Q27. OK - Q now S 28 H
4 N
S C
41.3 29 H
3 N
S B
41.3 30 H
3 X
N E
D 41.3 Discuss plausibility of spray pond as a distractor (A and B) - i.e.
what would actually affect spray pond level. A leak in the EW Heat Exchanger would affect SP level (see EW system drawing in the pedigree). Plausible that the examinees would confuse which systems cool which heat exchangers in a SDC lineup. OK -
Q now S 31 F
3 X
N E
C 41.8 In stem, change "can" to "may be" Revised. Q now S 32 F
3 X
N U
C 41.3 Distactors A, C, and D appear to be simply unbelievable. Correct answer seems obvious.
Replaced. New Q now S 33 H
3 B
S A
41.4 ON 2016 EXAM.
34 F
3 N
S C
41.3 35 F
3 N
S D
41.7 36 H
3 N
S B
41.7 37 H
3 X
N E
C 41.7 Change "will" to "should" in stem.
OK, but in the future location of indications in the control room are much better tested in the simulator. Changed will to should.
I dont disagree but Im not sure how else to address this sort of KAhow do you locate control room switches on a written exam? Maybe this type of KA will change on 1122 Rev 3? Revised.
Q now S a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 38 F
2 N
S D
41.6 39 F
2 N
S D
41.7 40 H
3 N
S C
41.4 41 F
3 B
S B
41.4 ON 2018 NRC EXAM.
42 F
3 N
S A
41.7 43 H
3 N
S B
41.4 44 H
3 M
E C
41.7 ON 2019 EXAM. Q says modified, but original Q not included. Original question added.
Q now S 45 F
3 X
B U
B 4.11 ON 2016 EXAM. All distractors indicate specific actions except A)2 and C)2, which indicate implementation of a procedure.
Change A)2 and C)2 to be consistent with other distractors.As written, A and C do not appear credible. Revised. Q now SAT 46 F
2 N
S C
41.7 47 H
3 B
S D
41.4 ON 2019 NRC EXAM.
48 F
2 N
S C
41.8 49 H
3 X
N U
B 41.3 ON 2016 NRC EXAM. D does not appear plausible. If seal cooler inlet temp stabilized at 200 degrees, why would anyone think the other seal temperatures would exceed 200 degrees? Also, too similar to Q29 on 2020 Replaced K/A and Q. Q now S 50 H
3 M
S C
41.3 REFERENCE PROVIDED. Looks familiar. Was the original question on a previous NRC exam? Yes, modified from 2020. Original question included in pedigree.
51 H
3 X
N E
A 41.5 Change "will" to "should" in stem.Revised. Q now SAT 52 H
3 N
S C
41.10 53 H
3 X
N E
A 41.7 Change "will's" to "should's" in stem. Revised. Q now S 54 F
3 X
N E
B 41.4 Change "will's" to "should's" in stem. Revised. Q now S a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 55 H
3 N
S D
41.7 OK, but it seems that this knowledge would be better tested in the simulator 56 F
3 N
S A
41.3 57 H
3 N
S C
41.2 58 F
3 X
N E
D 41.7 Discuss plausibilty of "radiolysis of water" Radiolysis of water is the separation of oxygen and hydrogen due to the radiation field. This is plausible since it is a contributor, it just isnt a large a contributor as the Zirc-Steam reaction. The reference in the pedigree discusses this in the last paragraph, its just not highlighted. OK - Q now S 59 H
3 X
X B
U D
41.11 ON 2018 NRC EXAM. Delete ONE in stem. Change "actuated" to "actuate" in C explanation.
Possible two correct answers.
Discuss why A incorrect since RU-31 directly causes the cross trip.
Deleted ONE. Changed C explanation to actuate. The discussion of DIRECT actuation vs cross-trip is a common one at PV so I understand your comment but feel very confident that the students would not be at all confused about what is being asked here. That being said, Ill take a look and see if there is a more legally bulletproof way to ask the question. Revised. Q now S
60 H
3 N
S A
41.8 61 H
3 N
S B
41.4 62 H
4 X
N E
A 41.7 In D, replace "kicked out of" with "automatically removed from."
Revised. Q now S 63 H
3 X
N U
C 41.4 K/A mismatch. K/A talks about the use of procedures and question asks what would happen with no operator action. Replaced with new Q. Q now S a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 64 F
2 N
S B
41.7 65 F
3 N
S D
41.10 66 F
2 X
N E
C 41.7 Add "per procedureXXX" in stem.
Revised. Q now S 67 F
2 X
N E
C 41.10 "3" appears obviuosly correct, which renders distractor A implausible. Revised. Q now S 68 H
2 X
N E
D 41.10 Add "per procedureXXX" in stem.
Revised. Q now S 69 F
2 X
N U
A 41.10 Not sure that knowing initial tagouts have IV has any discriminating value (distactors B and D) While I agree that it is fairly well known that initial tagouts are done using two people, knowing whether that is done using CV or IV, especially since new operators require a peer check when performing the initial hanging of a tag, makes it plausible that the two person hanging of the tag would suffice.
OK. Q now S 70 F
3 N
S A
41.10 71 F
2 B
E A
41.12 ON 2019 NRC EXAM. "Does" should be "dose" in B explanation Revised. Q now S 72 F
3 N
S D
41.12 73 F
2 X
N U
B 41.12 Don't see how C and D are plausible, as PCM's genrally do not have a "CPM readout." Q revised. Q now S 74 H
3 N
S A
41.10 75 F
3 N
S D
41.10 76 H
3 N
S A
43.5 77 H
3 N
S A
43.5 STEAM TABLES PROVIDED. a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 78 H
3 X
N U
C 43.2 Needing a DNBR trip at Mode 2 or when when rods can be initially withdrawn doesn't seem credible.
C and D appear to be easily eliminated. The DNBR trip is required to be operable when MODE 2 is entered, so I would say that this is clearly plausible.
Since the trip is required in MODE 2, I think that operators will be uncertain as to the earliest time that the trip is needed.
Revised to make DNBR the correct answer. Q now S 79 H
4 N
S D
43.5 80 H
3 B
S D
43.4 ON 2020 NRC EXAM.
81 H
4 N
S B
43.2 82 H
3 X
N U
D 43.1 REFERENCE PROVIDED.
Discuss why this knowledge wouldn't be better tested as an Admin JPM. K/A not an EP K/A (mismatch) Q replaced. Q now S 83 F
3 B
S C
43.2 ON 2016 NRC EXAM.
84 H
3 N
S A
43.5 85 H
4 X
B E
B 43.2 Please explain "7 gpm rise in sump levels" in stem. Q Revised.
Q now S 86 H
3 N
S C
43.1 Beaver Valley event 87 F
3 N
S D
43.5 I was got on this one 88 H
4 X
X N
U A
43.2 Pressure would be AT 185 psig at 1045 (not less than). Also, wouldn't the associated air compressor for receiver B recharge it prior to reaching 185 psig? Revised. Q now S 89 F
3 X
B E
C 43.4 ON 2020 NRC EXAM. Does RU-12 failing cause an annunciator to come in? If so, please include in the stem. Revised to add alarm.
Q now S 90 H
4 N
S C
43.2 a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 91 F
2 B
E B
43.2 ON 2016 NRC EXAM.
Explanations missing for distractors C and D, Revised explanation. Q now S 92 F
3 X
B E
B 43.1 ON 2016 NRC EXAM. Discuss K/A applicability 93 F
3 B
S B
43.5 ON 2018 EXAM.
94 H
3 N
S D
43.1 95 H
3 X
N E
C 43.2 Discuss K/A applicability.
Discussed. Q now S 96 H
4 X
N E
D 43.2 Discuss K/A applicability.
Discussed. Q now S 97 F
3 N
S A
43.4 98 F
2 B
S C
43.4 ON 2018 NRC EXAM.
99 F
3 X
N E
A 43.1 Discuss K/A applicability.
Discussed. Q now S 100 H
3 X
M E
C 43.5 ON 2020 NRC EXAM. Change "will's" to "should's" in stem.
Please explain how question was substantially modified from the original? Discussed. Q now S Results Table RO LOK -H 45 Avg RO LOD 2.85 Flaws 10 CFR Distribution RO LOK-F 30 AVG SRO LOD 3.16 Stem focus 15 41.1 18 43.1 5 SRO LOK - H 16 Overall LOD 2.93 Cues 5
41.2 1
43.2 9 SRO LOK - F 9
T/F 0
41.3 11 43.3 0 Cred Dist 17 41.4 11 43.4 4 RO Bank 12 16 SRO Bank 8
32 Partial 3
41.5 1
43.5 7 RO Mod 3
4 SRO Mod 1
4 job link 0
41.6 3
43.6 0 RO New 60 80 SRO New 16 64 units 0
41.7 16 43.7 0 minutia 0
41.8 5
Total Bank 20 20 backward 0
41.9 0
Total Mod 4
4 KA 8
41.10 18 Total New 76 76 SRO-only 1
41.11 3
LOD = 1 0
41.12 4
RO Sat 42 56 SRO Sat 14 56 41.13 0
12 Answer Dist ( in %)
41.14 1
32 RO-A 17 SRO-A 6 a Rev 2
ES-401 PV-2021-04 Written Examination Review Worksheet Form ES-401-9 RO-B 17 SRO-B 5
Total Sat 56 56 Total Unsat 15 15 RO-C 21 SRO-C 8
Total Edit 29 29 RO-D 20 SRO-D 6 a Rev 2
Question Overlap with the Previous Two NRC Exams QA form ES-401-6 item 4 requires 4 or less questions on RO exam from previous two NRC exams, SRO portion two or less questions from previous two NRC exams*
- if this is exceeded, the region shall call NRR to explain why/reasons.
RO overlap from previous two NRC exams 4
SRO overlap from the previous two NRC exams 2
Questions Requiring References to Answer 6-10 questions rule of thumb on SRO portion of exam are allowed to be open reference, while the RO exam is generally none RO Questions that require hand-outs 2
SRO Questions that require hand-outs 2