ML023240232
| ML023240232 | |
| Person / Time | |
|---|---|
| Site: | Nine Mile Point  |
| Issue date: | 09/27/2002 |
| From: | Pisano L Constellation Nuclear |
| To: | Conte R NRC/RGN-I/DRS/OSB |
| Conte R | |
| References | |
| 50-220/02-303, ES-401, ES-401-9 50-220/02-303 | |
| Download: ML023240232 (8) | |
Text
ES-401 NMP #1 9/02 EXAM Written Examination Form ES-401-9 Note: Resolutions to comments in bold and italics Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F F
~red-j Partial Job-Minutia
- /
Back-Q i SRO UIEIS Explanation Focus Dist.
Link I units ward KA..A Ql I
F 2
E Add word "containment" to the stem-recommended changes made 2
H 2
S 3
H 3
E TYPO - fixed 4
H 2
E Bold, underline and cap the word "required" - - recommended changes made 5
H 2
S 6
F 2
S
- Z7/
F 2
X U
Explain why distractors"A&D" are credible also reword the stem, S 10 Question and K/A replaced.
8 F
1-2 S
Borderline level of difficulty okay in limited #s 9
H 2
E add the word "execute" to the "D" distractor - - recommended changes made 10 H
3 S
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check the appropriate box if the sampled question does not match the approved K/A or an SRO-only question is not at the SRO level.
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
For any "U" ratings, at a minimum, explain how the Appendix B psychometric attributes are not being met.
ES-401 2
Form ES-401-9
- 3. Psychometric Flaws Stem Cues Focus "T/FCred.
Dist.
Partial
- 4. Job Content Flaws
- 5. Other Job-Minutia
- / IBack-
.I S *I Link I units I ward 1K/A I Onl
- 6.
U/E/S 7.
Explanation 11 F
3 E
Change "may be" in the stem to "should be" - revised as recommended 12 H
2 S
13 F
1-2 S
Borderline LOD also don't agree it is higher order - okay in limited number. Licensee indicated that this was intended to be a memory level question.
14 H
3 S
15 H
3 S
16 F
2 S
17 H
2 U
Check to make sure battery rating is required knowledge from memory Replaced with a new question 18 F
2 X
S check K/A - replace K/A -Agreed K/A match is okay 19 H
2 E
Fig "B" - move title alter Fig, "B" in the stem - reworded the stem to clarify 20 H
2 E
reword stem to "may fail as a direct..." - recommended changes made 21 F
2 E
revise wording of stem change "must be" to "shall be" in service to agree with procedure - recommended changes made 22 F
2 E
add to stem SOP-9.1 - recommended changes made 23 F
2 S
24 F
2 S
25 H
3 E
- 25 appears to have the wrong proposed answer - see justification section - Answer verified correct and corrected justification.
26 F
2 S
27 H
3 S
28 H
3 S
29 H
2 S
Make sure that valid RO learning objectives exist for all EOP application type questions on this exam. Licensee verified valid RO learning objectives exist 30 H
2 S
same comment as #29.
Licensee verified valid RO learning I
I objectives exist
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. O r
- 6.
- 7.
Q#
LOK LOD 1
(F/H)
(1-5)
Stem ICues TIF ICred. Partial Job-Minutia
- 1 IBack-Q U/E/S Explanation Focus I Dist.
Link unitsI ward o nly 31 H
2 E
same comment as #29 &30. Licensee verified valid RO learning objectives exist. Deleted commas in each distractor and reorded distractors.
32 H
2 S
Give applicants entire reference not just Att 5 also don't tell appplicants which question reference will be used to answer.
33 F
2 E
modify stem - stem modified change ? To:
34 H
2 S
35 H
2 E
In stem add the word "temperature" after RWCU pump area and HTX 36 F
2 S
37 F
2 E
change verb in "A&B" from "may" to "will" to agree with "C&D" recommended changes made 38 F
2 S
39 H
2 S
40/
F 2
X S
"A&B" distractors verify credible. Licensee verified distractors 15 credible.
41 F
2 E
Add words in the stem "the core" in front of srpay pump min flow relief valve - recommended changes made 42/
F 2
X X
X U
12 be used to fill torus? Tighten stem focus Tightened stem focus also
$7 revised distractors.
43 H
3 S
44 H
2 S
45 F
2 S
- 44 stem may cue #45, changed #44 SRN#12 withdrawn to maintain on scale 46 F
2 S
47 F
3 S
46 H
2 S
49 H
2 S
50 H
2
]
S stem 2 indications for valve open and 2 for valve closed - revised stem
ES-401 4
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem CuesI T/F Cred. Partial Job-Minutia
- 1 IBack-Q-
U/E/S Explanation Focus Dist.
Link units ward 0...................
51 H
2 X
E stem unclear - rewrote stem 52 F
2 S
53 H
3 E
change stem: "may result" to "will result" - recommended changes made 54 H
3 E
reorder bullets in stem
- recommended changes made 55 H
2 S
56 H
2 E
Change "may" to "will" in each answer choice
- recommended changes made 57 H
2 S
58 H
2 S
59 F
2 S
60 H
2 S
61 H
2 S
62 F
2 E
Add to stem RBEVS in a normal "standby" line-up - recommended changes made 63 H
2 E
Modify the stem - recommended changes made 64 F
3 E
Add "EDG 103" in front of "output breaker" in choices "A&B" recommended changes made 65 F
2 S
66 F
2 S
67 F
2 E
Add the word "the" to choices "B,C, and D" 68/
F 3
X X
U Delete"before next use" In answer col. 2 also "B&D" implausible 71 distractors Revised stem and distractors 69 F
2 S
70 H
2 S
NUREG-1021, Revision 8, Supplement I
ES-401 5
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws fOthielr
- 6.
- 7.
Q#
LOK LOD i-I (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-SO U/E/
Explanation Focus I Dist.
Pa Link J
M units ward QWR IN S
71 F
2 S
72 F
2 S
73 H
2 S
74 F
2 S
75 H
2 S
76 H
2 S
77 F
2 S
78 F
3 S
79 F
2 S
80 F
2 S
81 H
2 S
82 H
2 E
change "dump" valve to" bypass" valve in stem - recommended changes made 83 H
2 S
84 H
2 S
85 H
2 S
86/
H 3
X U
Make sure "B" is incorrect - Distractor "B" was revised 83 87 H
2 X
U "B&D" distractors implausible with cavity drained also revise wording in "A" to add "pump". Added word and revised distractors.
88/
F 1-2 U
LOD Borderline - tested OPs test - Replaced question same K/A used 16 89 F
2 S
90 H
3 X
U Ability to locate not tested also better tested in Part C of the exam I -
-T-1 Modified question and distractors significantly.
NUREG-1021, Revision 8, Supplement 1
A Form ES-401-9
- _LAN4
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws 5.Oth
- 6.
- 7.
(F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- 1 Back-SRO UIEIS Explanation 91/
H 2
X U
Not tier 3 genetic (system type) replace queston Replaced K/A and o0 question 92 H
2 S
93 H
2 X
E Correct answer for #34 may be cued by reference (core map) - deleted reference as a handout 94/
H 2
X U
Not tier 3 generic (system type) replace question Replaced K/A and
)6 question 95/
F 2
X U
Distractors don't appear credible for a very high Red area Distractors
)5 were revised 96 F
1-2 X
X X
U Tighten stem and answer choices. As currently written, "first frisking" and "each area" doesn't make sense. Add minimum to frisking requirement. See proposed rewrite. Tightened stem and reworded choices.
97 H
2 X
X U
"D" isn't this a true statement regardless Not tier 3 generic (system type) replace question Replaced K/A and question 98 H
2 S
99 H
2 S
100 F
2 S
- RO 13 F
2 S
14 F
2 X
X U
"A, B & D" distractors credible? Stem cues Power ops to contiue Revised stem 15 H
1-2 X
U low LOD. Direct lookup with the reference. Better tested in dynamic scenario or JPM Wrote a new question that matches the K/A 20 H
3 S
21 F
1-2 U
low LOD. Direct lookup with the reference. Already tested on dynamic scenario Revised distractors 24 H
1-2 U
low LOD. Direct lookup with the reference. Already tested on dynamic scenario Used same K/A and wrote new question
I.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws 5,1 Oth*
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutiai
- /
Back-Qi SRO. U/E/S Explanation Focus j
Dist.
Link units ward __,
27 F
2 X
U K/A mismatch, low LOD test more than one EOP leg or EOP at the time for an SRO Direct lookup with the reference. Already tested on dynamic scenario Replaced K/A and question 33 F
1-2 E
low LOD. Rewrote question to calculate time to boiling given RPV operating history 6
H 1-2 U
low LOD, test more than one EOP leg or EOP at the time for an SRO.
Rewrote question 39 H
2 S
10 H
2 S
42 H
2 E
don't just give the one section 50.72 &73 give all I OCFR2-50. Add a comma to the fourth bullet, put "the" in front of NRC 48 H
3 S
55 H
1-2 U
low LOD, test more than one EOP leg or EOP at the time for an SRO Already tested on dynamic scenario?
Replaced K/A and question 56 H
1-2 X
X U
MSIVs isolated given in the stem clues the correct answer both vessel and containment isolations required.
Replaced question 57 H
3 U
Already tested on dynamic scenario Replaced K/A and question 39 F
2 E
Simplify answer choice wording 51 F
2 X
X U
Distractor "D" could be correct. If error discovered next day how could it be reported any earlier? Revised stem and distractor 34 H
3 S
35 H
3 E
Add "a" before refueling outage 38 F
1-2 X
U C&D don't appear credible also LOD 39 F
2 E
Add "power" alter "100%". 2n' bullet: add ' for "valve's" ; remove ' for "its".
)2 H
2 S
Change stem "what individual is responsible for final authorization to extend someone's annual Nine Mile dose limit to 1600 mrem"
)3 H
2 S
Need to reword stem - all answers could be correct (i.e securing "within" 35 hours4.050926e-4 days <br />0.00972 hours <br />5.787037e-5 weeks <br />1.33175e-5 months <br /> could mean securing in 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />). Proposed re-write:
"Which one of the following describes how much longer Drywell venting can continue? Drywell venting can continue for another..."
4 H
2 S
Distractor C and D too similar, come up with a replacement (maybe "raise RPV level to 100 inches"?)'. A bit detailed for a tier 3 generic.
I I
I I
I I
IReplaced "D" distractor
OPERATING EXAM COMMENTS:
0 The # of applicants have dropped from 16 to 8-9, therefore, the amount of exam material needed to conduct the exam has decreased. From the material submitted Scenarios #1, 3, 4 will be used with #5 to be used as a spare. Admin sets RO "B" and SRO "A" will be used.
0 Requested licensee to send list of references that will be presented to the applicants for the written portion of the exam. Tech Specs will be provided only sections 3 and 4.
0 In-plant JPM #9 will be replaced too simplistic also in same general area as the diesel fire pump JPM # 10.
0 General comment: All Scenarios need procedure details added for ARPs, SOPs, EOPs, etc. Also for TS calls need to list all affected TS and most limiting LCO.
0 Scenario #4 needs BOP actions added all board actions currently assigned to the RO. Also need to ensure for other scenarios that board assignments each watch station are more clearly delineated. Also for this scenario the licensee had to rework the leak rate several times during the validation week to get the scenario to the point that the pace was good.
0 Scenario #1 deleted APRM #14 OOS in the turnover to ensure the failure of APRM #13 event #2 would result in operator action in this case bypass the APRM and place in a half scram. Added a normal reactivity at the beginning of the scenario to increase power 5% so that the RO would have a reactivity in that position. Changed event #4 so that recirc seal failure could not be isolated resulting in the cause of the LOCA rather than a random failure LOCA.
Licensee suggested that we consider administering 3 scenarios each to the SROI candidates because of the way the RO position functions ( i.e., the BOP gets most of the actions other than reactivity manipulations using recirc and rods which is the RO responsibility and in most cases he stays at his panel). Consequently, the SROI applicants will get 3 scenarios one each at the SRO, RO and BOP positions to ensure all minimum exam requirements are satisfied.
SRO, A.1.a JPM, rather than telling the applicants that the APRMs were calibrated in error - have the applicant review the ST for acceptance, recognize the error.
SRO, A.l.b JPM, add at least another parameter that exceeds limits so that the JPM requires identification of what chemistry parameter exceeded and requires determination of mitigating actions and affect on the plant( i.e., TS) not just notifying personnel.
SRO, A.3, question 2, have applicant determine that the RWCU room is a high rad area and is in an emergency condition (rather than being told this) and have the applicants determine that the situation requires the use of an emergency RWP.
0 SRO, A.4 JPM, make time critical 15 min. to classify and 15 min. for notification.