ML18141A855
| ML18141A855 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 04/20/2018 |
| From: | Vincent Gaddy Operations Branch IV |
| To: | Arizona Public Service Co |
| References | |
| 50-528/18-04, 50-529/18-04, 50-530/18-04 50-528/OL-18, 50-529/OL-18, 50-530/OL-18 | |
| Download: ML18141A855 (9) | |
Text
ES401PV201804WrittenExaminationReviewWorksheetFormES4019
- 1. LOK 2. LOD 6
7 Ans Letter CFR 8
Q#
(F/H)
(1-5) stem focus cueing T/F cred dist partial job link minutia
- /
units back-ward K/A SRO-only B/M/N U/E/S A/B/C/
D 55.41x55
.43y Explanation 1
F 2
B S
C 41.6 ON 2012 NRC EXAM. Needs LOD (2?)/Made LOD=2 2
H 3
N S
B 41.5 3
H 3
B S
C 41.14 ON 2010 NRC EXAM.
4 H
3 N
E A
41.11 In explanation, 1-4 should be A-D/Revised 5
H 3
X N
U A
41.2 ON 2007 NRC EXAM. Hard to fathom being able to control RCS parameters such as temp, PZR level and pressure on a large break LOCA - makes correct answer the only one plausible./Replaced with new question not previously on NRC exam 6
F 3
N S
A 41.10
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 to 5 (easy to difficult) rating scale (questions in the 2 to 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- a. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- b. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
- c. The answer choices are a collection of unrelated true/false statements.
- d. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- e. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- a. The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- b. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- c. The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- d. The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? Place the answer letter here Place the 55.41 or 55.43 item here
- 8. At a minimum, explain any Unsat ratings (e.g., how the Appendix B psychometric attributes are not being met). a Rev2
ES401PV201804WrittenExaminationReviewWorksheetFormES4019 7
H 3
M E
B 41.7 Same concept on SESS blue alarm is being tested on question 52 (replace one or the other).
Blue alarm portion needs a reference. Original bank question doesn't appear to have correct answer indicated./Revised 52 to resolve conflict 8
H 3
N S
C 41.3 9
H 2
N S
C 41.10 Changed LOD from 3 to 2.
10 H
3 N
S D
41.10 REFERENCE PROVIDED (steam tables) 11 H
3 N
S B
41.4 12 H
3 B
E B
41.4 Need reference that MFW is preferred over AFW./Revised 13 F
3 N
S B
41.10 14 H
2 N
S A
41.7 Changed LOD from 3 to 2.
15 H
4 N
S C
41.5 16 F
3 B
S B
41.7 ON 2016 NRC EXAM.
17 F
3 X
B E
D 41.7 ON 2016 NRC EXAM. Stem should be broken up into two sentences - doesn't read well when answers inserted./Revised 18 H
3 N
E A
41.5 Reference a little weak. Beef up if possible./Revised 19 H
3 N
S D
41.1 20 H
3 N
S D
41.10 21 H
3 X
N E
D 41.3 Change WILL's to SHOULD's in distractors. Question OK, but "None of the above" type questions should be avoided/Revised 22 F
2 B
S A
41.9 ON 2008 NRC EXAM. Needs LOD (2?)/Revised 23 F
3 X
N E
A 41.5 Is there more than a lesson plan to justify correct answer? Stem would be better and question less appealable if we could add "Per EOP Background document XXX, why" /Revised a Rev2
ES401PV201804WrittenExaminationReviewWorksheetFormES4019 24 H
3 X
N E
B 41.4 Add "ONLY" after main turbine in C and D since tripping the reactor also trips the turbine. Please explain color scheme in indication./Revised 25 H
3 N
S D
41.9 26 F
2 N
S C
41.10 On previous NRC exam.
27 H
3 X
N E
B 41.10 REFERENCE PROVIDED.
Revise 2nd part of distractors to be consistent (use "stopping the cooldown" vice "maintaining RCS temperature")./Revised 28 H
3 N
S C
41.10 29 F
2 B
S D
41.3 30 H
3 N
S C
41.6 31 F
3 N
S A
41.8 32 H
3 N
S D
41.7 33 F
3 X
X N
U C
41.10 On previous NRC Exam? Second half of all four distractors different
- don't need knowledge of the first half to answer question (revise).
Please explain how a valve can be throttled if it is full open or full closed (stem says "when throttling HPSI injection valves").
Original bank question does not appear to indicate its correct answer./Replaced with new question 34 H
3 X
N U
D 41.3 Correct answer only credible distractor/Replaced with new question 35 H
3 N
E B
41.7 Explanation for A needs revised (will isolated)/Revised 36 F
3 N
S B
41.3 37 F
3 N
S A
41.7 38 H
3 N
S B
41.4 39 H
3 N
S D
41.7 40 H
2 N
S D
41.7 41 H
3 M
S B
41.10 MODIFIED FROM 2015 NRC EXAM. a Rev2
ES401PV201804WrittenExaminationReviewWorksheetFormES4019 42 H
3 X
N E
A 41.14 Only need to know second half of answer to get question correct.
Re-write to eliminate first half of distractors./Revised 43 F
3 X
M E
C 41.8 Previous NRC Exam? In stem, change "wills" to "shoulds."
/Revised 44 F
3 N
S B
41.8 45 F
3 N
S D
41.4 46 F
2 B
E D
41.10 MODIFIED FROM 2010 EXAM.
47 H
3 X
X N
U D
41.7 ON 2016 NRC EXAM. Delete ONE in stem. A not plausible, as it is "None of the Above." D not plausible as how can a battery charger be charging anything with zero amps? /Replaced with new question not previously on an NRC exam 48 F
2 N
S A
41.8 49 F
3 N
E C
41.11 In stem, change "will" to "should."
/Revised 50 H
3 N
S C
41.11 51 H
3 N
S B
41.7 52 H
3 N
E C
41.8 Add reference regarding expectation to manually start spray pond pump with blue alarm.
Same concept on SESS blue alarm being tested on question 7 (replace one or the other)/Revised 53 F
3 X
N E
A 41.4 In stem, change "will" and "can/will" to "should." /Revised 54 H
3 N
S B
41.4 Good question a Rev2
ES401PV201804WrittenExaminationReviewWorksheetFormES4019 55 H
3 X
X N
U B
41.7 Distracors inconsistent. First part of C and D are different, but explanation says they are both correct (i.e. "stopped" versus "be stopped" and "running" versus "start." D as correct answer could be challenged as how can fans that are "running" in stem "remain off?" Second part of distractors A and B have info about only one set of fans, but C and D have info on both sets, cueing that correct answer is probably C or D./Revised 56 H
3 N
E A
41.7 Need reference that explains how cicumstances in the stem are possible (i.e. one rod slips and only the A channel of CWP and DNB alarms) /Revised 57 H
2 X
X N
U C
41.5 Only knowlegde needed to determine correct answer is that the RCS uses a ramped Tave with a constant pressure. This appears to be LOD=1 knowledge.
Also, stem bullet has a typo (Unit 2 is was)/Revised 58 F
2 N
S C
41.7 59 F
2 N
S B
41.2 60 H
2 N
E B
41.1 Please explain why Mode 2 (keff greater than or equal to 0.99) is different than the reactor critcal -
don't see it in reference./Revised 61 F
3 B
S A
41.9 ON 2009 NRC EXAM 62 F
3 N
S C
41.4 63 F
2 N
S B
41.7 64 H
3 N
S A
41.4 65 H
3 B
S B
41.11 ON 2010 NRC EXAM.
66 F
2 N
S D
41.10 OK, but should avoid "all of the above" type questions 67 F
2 N
U D
41.10 LOD=1 No real discriminating value/Revised a Rev2
ES401PV201804WrittenExaminationReviewWorksheetFormES4019 68 F
3 X
X M
E B
41.4 ON 2016 NRC EXAM. Question has strucural problems. "in either steam generator" is in 3/4 distractors, cueing A is probably wrong; The correct answer seems to have the worst condition as well (PPM vs. PPB).
Restructure, make another answer correct, and make the question modified bank. /Revised 69 F
2 X
X N
U B
41.10 First part of answer is in question.
The stem essentially is asking for an interim (temporary) procedure action. TAPA is short for Temporay Approved Procedure Action. Applicant only needs to know what the TAPA acronym stands for to eliminate C and D./Replaced 70 F
3 N
S C
41.10 71 F
2 N
S B
41.10 In general, how often are annunciators jumpered out in the control room?(normally 2-5) 72 F
2 N
S D
41.9 Discuss why this meets Tier 3 criteria 73 F
2 X
N U
A 41.12 ON 2010 NRC EXAM Make tolerances tighter to give the question some discriminating value, i.e change 4 inch/sec to one inch/sec and change 1 inch to 3/4 inch in distractors/Replaced with a new question not previously on an NRC exam 74 F
3 N
S C
41.10 75 F
2 N
S C
41.10 76 H
3 N
S A
43.5 77 H
3 X
N E
D 43.2 In stem, change "will" to "should,"
and add MAXIMUM in front of completion time/Revised 78 F
3 N
S B
43.6 a Rev2
ES401PV201804WrittenExaminationReviewWorksheetFormES4019 79 H
3 X
N U
C 43.5 Discuss why this question has any discriminating value given that not re-energizing a faulted bus and not having increasing subcooling on a LOCA is pretty fundamental knowledge (LOD=1) -
certainly not SRO only
/Restructured question to make discriminating 80 H
3 N
S A
43.5 81 H
3 B
S C
43.2 ON 2016 SRO EXAM 82 H
3 N
S D
43.2 83 H
3 X
N U
A 43.2 Stem indicated a power reduction is in progress and that loss of vacuum AOP has been entered, which also directs boraton. Stem also says condenser shell pressure "continues to rise slowly." As written there is no way for the applicant to determine power level at the time vacuum reaches >7 inches. Cannot be correctly answered as written.
Revise to provide necessary info/Revised 84 F
3 M
S A
43.1 MODIFIED FROM 2016 EXAM 85 F
2 N
S B
43.2 86 F
3 N
S D
43.2 87 H
3 X
X N
U C
43.5 C and D not plausible. Second half of all four distractors have to do with restoring feedwater, cueing that restoring feed is the highest priority. Also, since electrical problems are described in the stem, this tends to cue that operation of the TDAFW pump is the probable success path.
Revise or replace? /Question reconstructed to eliminate problem. a Rev2
ES401PV201804WrittenExaminationReviewWorksheetFormES4019 88 F
3 X
B E
C 43.2 ON 2016 SRO EXAM. Don't see how A is plausible as the valve galleries would be considered "onsite" and wouldn't have anything to do with the Offsite manual. Revise A?/Revised 89 H
3 N
S A
43.2 Need reference that defines "useable feet" required to meet tech spec/Reference added 90 H
3 N
S C
43.5 91 F
3 N
S D
43.4 92 F
3 N
S C
43.2 93 H
3 X
N E
D 43.4 Change "will" to "should" in stem/Revised 94 H
3 N
S C
43.1 95 F
2 N
E B
43.6 Must have a 10 CFR Reference.
Discuss if 43.6 best fit/Revised 96 F
3 B
E D
43.5 97 H
3 N
S B
43.2 98 F
3 M
S C
43.2 99 F
2 X
N S
A 43.5 100 F
3 N
S B
43.5 Results Table RO LOK -H 40 Avg RO LOD 2.73 Flaws 10 CFR Distribution RO LOK-F 35 AVG SRO LOD 2.88 Stem focus 11 41.1 20 43.1 2 SRO LOK - H 13 Overall LOD 2.77 Cues 4
41.2 2
43.2 11 SRO LOK - F 12 T/F 0
41.3 5
43.3 0 Cred Dist 15 41.4 10 43.4 2 RO Bank 10 13.33 SRO Bank 3
12 Partial 0
41.5 5
43.5 8 RO Mod 4
5.333 SRO Mod 2
8 job link 0
41.6 2
43.6 2 RO New 61 81.33 SRO New 20 80 units 0
41.7 15 43.7 0 minutia 0
41.8 5
Total Bank 13 13 backward 0
41.9 4
Total Mod 6
6 KA 0
41.10 20 Total New 81 81 SRO-only 1
41.11 4
LOD = 1 0
41.12 1 a Rev2
ES401PV201804WrittenExaminationReviewWorksheetFormES4019 RO Sat 47 62.67 SRO Sat 17 68 41.13 0
RO Unsat 9
12 SRO Unsat 3
12 Answer Dist ( in %)
41.14 2
20 RO-A 16 SRO-A 6
RO-B 23 SRO-B 5
Total Sat 64 64 Total Unsat 12 12 RO-C 19 SRO-C 8
Total Edit 24 24 RO-D 17 SRO-D 6 a Rev2