ML12356A346
| ML12356A346 | |
| Person / Time | |
|---|---|
| Site: | Ginna  |
| Issue date: | 10/18/2012 |
| From: | Peter Presby Operations Branch I |
| To: | Geckle M Constellation Energy Nuclear Group |
| Jackson D | |
| Shared Package | |
| ML12137A368 | List: |
| References | |
| TAC U01857 | |
| Download: ML12356A346 (11) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (FtH)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- /
Back-Q=
SRO B/MtN UlEtS Explanation Focus Dist.
Link units ward KIA Only 1
H 3
X B
U References do not support the key answer. KIA mismatch.
Replaced the question.
2 F
2 B
E Distractors have low plausibility.
Modified the question.
3 F
3 X
B U
Choices A and C (2nd parts) not sufficiently discriminating.
condition in stem.
Changed 4
F 3
X B
U Overlap with another portion of exam. Limited operational validity.
Reselected KIA and replaced question with bank question.
Can be answered without plant-specific knowledge, solely through 5
H 1
X N
U application of generic fundamentals principles.
Rewrote the question.
6 H
3 B
E The 2nd part of Choice B is not distracting.
choices.
Reworded 2nd parts of 7
H 3
X X
N U
Multiple correct answers. Rewrote the stem to support a single correct answer.
KA match not obvious. Choice C is implausible.
8 H
3 X
B E
Explained KIA match on justification sheet. This question topic is appropriate for an RO because it tests general understanding of the procedure mitigation strategy. Rewrote the question.
Consulting with TSC is an SRO function and therefore inappropriate to test basis for the consult on the RO exam. However, knowledge of 9
H 3
X X
N E
instrument post-accident response is expected RO knowledge.
Reference to temp/pressure effects on dIp cell cues Part 1.
Rewrote stem.
Unnecessary "group 1" in stem. Choice D is a correct statement.
10 H
3 B
E Removed unnecessary information from stem and added "IAW [the procedure name]" to the question.
Page 1
ES*401 2
Form ES*401*9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
LOK LOD Q#
(F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/M/N utEtS Explanation Focus Dis!.
Link units ward KIA Only Psychometric flaw. Collection of TtF statements. KIA mismatch.
11 F
3 X
X X
B U
Two (or more) correct answers.
Reselected KA and replaced with a bank question.
12 F
2 B
E Choices A and D have low plausibility. Reworded question and replaced two distracters.
13 H
3 X
B U
KIA mismatch.
Selected a different bank question.
NBtD are implausible given lack of specificity of stem and absolute 14 F
2 X
B U
nature ("All") of choices.
Rewrote the question.
15 H
3 X
X N
U Multiple implausible distracters. KIA mismatch.
Reselected the KIA and wrote a new question.
16 H
3 B
E Added "and why" to the stem to align with both parts of each choice.
17 H
3 N
S Unnecessary stem wording. The 2nd parts of Choices A and D do not 18 F
2 N
E relate to the question asked in the stem. Need to move the common information into the stem.
Reworded stem and choices.
19 H
3 X
N U
Choices Band D implausible.
Rewrote the question and choices.
Need a procedure reference in the stem. Stem has leading 20 H
3 M
E information. Need additional information in justification.
Minor modifications made to address comments.
The choices are two-part but the stem question only asks for the first 21 H
3 B
E part.
Modified question to ask for both parts.
Page 2
ES-401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
LOK LOD Q# (FfH)
(1-5)
Stem Cues TfF Credo Partial Job-Minutia
- f Back-Q=
SRO BfMfN utEfS Explanation Focus Dist.
Link units ward KIA Only Question reduces to just 2 credible choices because of the reference provided for Q#31. Distinguishing between the choices becomes a 22 F
1 X
B U
direct lookup with the reference. LOD is too low. LOD is too low.
Topic is SRO level.
Reselected KIA. Replaced with a bank question.
Choices A and B have low plausibility. Choices Band Dare implausible. The stem is leading.
23 H
3 X
N U
Rewrote the question. New question matches most important part of the two-part KA.
Implausible Distracter Choices C and D.
24 H
2 X
N U
Wrote a new question.
Distracter Choice A implausible. Choice A justification incorrect.
Choice C needs clarification. Choice C needs clarification. Choice D 25 H
2 B
E is not clear as to which train.
Modified stem, choices and justifications to address comments.
Second part of the choices not clearly linked in stem to the first part.
26 F
3 N
E Stem wording slightly modified.
Somewhat confusing justification for one of the distracters.
27 F
3 N
E Clarified justification.
Not very operationally oriented, but meets the KIA. Choices A and B 28 H
3 N
E have low plausibility.
Question reworded to enhance distractor plausibility.
Psychometric flaw in exclusivity of choices.
29 H
3 B
E Corrected flaw with use of "only" in the choices.
Multiple implausible distracters.
30 F
3 X
B U
Reworded answer choices. Question matches KA as the referenced procedure is effectively the EOP for this event at this station.
Not a RO level question.
31 H
3 X
N U
I Reselected the Y~A. 'vJrote a new question.
I I
I I
I I
I I
I I
I I
I I
I I
Page 3
ES-401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
LOK LaD Q#
(F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/M/N utE/S Explanation Focus Dist.
Link units ward KIA Only 32 H
2 B
S 33 F
3 X
M U
KIA mismatch. Questionable operational validity.
Replaced question with a modified bank question.
34 F
1 B
U Question is not discriminating.
Wrote a new question to the same KA.
Move repetitive information to stem. Choice B answer non-specific.
35 H
3 B
E Choices C and D are "should" (optional) actions.
are not independent from Choice A.
Choices C and D Reworded choices.
36 H
3 N
E Provide noun name for numbered component in stem.
Added noun name.
37 F
3 M
E Typo in stem. Choices C and D say essentially the same thing.
Modified stem and choices to address the comments.
Only testing GFE knowledge, already tested on GFE. Need to adjust 38 H
3 N
E question to test this KA in conjunction with some site-specific system or procedure knowledge.
Modified question to require application of site-specific knowledge.
LaD too low. Distracters not plausible because correct answer so 39 F
1 X
B U
obvious.
Changed the correct answer.
Labeled as comprehension level, but question is actually memory 40 F
2 N
S level. Re-categorize as memory level.
Recategorized as memory ievel.
Distracter validity questionable given no time frame. Stem very 41 H
3 X
B U
leading. Arguments can be made for two correct answers (B or D).
Modified question and choices to establish validity and ensure I
I distracters are incorrect.
Page 4
ES-401 2
Form ES-401-9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws Stem Cues TIF Credo Partial Focus Dist.
- 4. Job Content Flaws Job-Minutia
- 1 Back-Link units ward
- 5. Other Q=
SRO KIA Only
- 6.
- 7.
B/M/N U/E/S
- 8.
Explanation 42 H
3 X
X N
U B is a subset of D. Choice C is another potentially correct answer.
Could Choice A also be a correct answer?
Modified question.
43 F
2 X
X N
U This question is only testing fundamentals concepts, already tested on the GFE. It doesn't test site-specific system knowledge. Choice C is a possible 2nd correct answer. Choice D could be argued as a wrong answer.
Modified question to require site-specific system knowledge to answer. Fixed distracters.
44 F
3 X
X B
U KIA mismatch. Choices A and C are implausible.
Reselected KIA. Replaced with a bank question.
45 H
3 X
B U
Question doesn't test the KA. Choice A is a subset of Choice D.
Replaced with a different bank question that matches the KIA.
46 H
3 N
E Choice C doesn't answer the question asked in the stem. Choice B is not plausible for one of the directions in which the instrument could fail. Stem should state the instrument failed high.
Modified question choices.
47 H
3 X
B U
KIA mismatch.
Question modified to match KIA.
48 H
2 X
X N
U KIA mismatch. Choices C and D are implausible. Choices Band D
,~oot d;O"09.
~
Wrote a new question.
49 F
3 M
U Choices A and C have low plausibility. Re-categonze this question as memory-level, not comprehension/analysis-level.
Reselected KIA. Replaced with a modified bank question.
50 L""",
F d
B S
Appears to relate more to SRO job responsibilities. However, the question exactly matches this relatively high RO importance rated KIA and tests EOP knowledge expected of an RO.
PageS
ES-401 2
Form ES-401-9
- 1. I 2.
LOK LOD
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other I
- 6.
- 7.
- 8.
Q# I (F/H)
(1-5)
Creel. Partial Job-UIE/S Explanation us Dist.
Link 3-part answers not good, reduces diSCriminatory value as too 51 F
3 N
E ways to eliminate wrong answers. Take off the "where" portion of the question and choices. Adjust justifications accordingly.
Changes made as indicated.
52 F
3 B
s The "(1)" and the "(2)" in the choices is confusing. Reformat to have them as column headings above all the choices, then space the parts 53 H
3 N
E Iof each choice under the appropriate heading number using tabs.
Part 2 wording is confusing. Confusing stem wording.
Modified stem and choice wording for clarity.
Need additional info in stem to support distractor plausibility. Adjust BEl wording of Choice D to make same length as other choices.
ti 55 H
3 U
Modified as indicated.
Choices A and B implausible.
Rewrote the question.
3 Minor justification typo.
Choice D is a subset of Choice A. Recommend modification to 57 H
2 BEl improve level of difficulty.
Revised question.
58 H
3 x
U KIA mismatch.
Modified the question.
59 H I 3
E Too similar to 0#41. Reselected KIA and wrote a new question.
60 F
I 3
E ChoiCe A implausible.
Added to stem. Reworded Choice A.
Choice A has low discriminatory value. Choice B implausible.
61
~~~L H
3_I x
M U I Choice D implausible.
Modified question.
Page 6
ES*401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
LOK LOD Q#
(F/H)
(1-5)
Stem Cues TIF Credo Partial Job-Minutia
- 1 Back 0= SRO B/M/N utE/S Explanation Focus Dist.
Link units ward KIA Only Selection of EOP transitions more of a SRO responsibility.
However, the question matches high RO importance KA. Please 62 H
3 B
E document why the facility thinks this question is testing RO level knowledge. Choice A is implausible. Choice D is the obvious choice, making this a question with low discriminatory value.
Modified question and justifications to address comments.
Some overlap with 0#61 and poor basis for licensing decision.
63 F
3 M
E Per Chief Examiner direction, reselected KIA. Replaced with modified bank question.
64 F
3 M
E Choices Band D have low plausibility.
Revised distractors to improve plausibility Stem wording lacks focus and missing alignment information to 65 M
3 N
E support choices.
Made changes to stem and choices.
Reword Choice A to better align with formatting in C and D as 66 F
3 B
E follows:
"Manually operate SI pumps as necessary if in a yellow, orange or red path FR."
Potential for stem confusion. The key answer may not be correct in 67 H
3 N
U all cases.
Could not fix problems. Replaced with new question.
68 F
3 B
E Modify wording of distractor choice to enhance plausibility.
Modified as indicated.
Part 1 of Choices A and B are implausible. Choice C doesn't appear 69 F
?
X N
U to be incorrect, just incomplete.
Revised question to address comments.
70 F
3 B
E Minor revision to stem, choices to clarify question and improve symmetry.
71 F
3 B
E Minor modification to stem and one distractor choice to ensure distractor is wrong in all cases.
Page?
ES-401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
LOK LOD Q# (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/M/N u/E/S Explanation Focus Dist.
Link units ward KIA Only Is wording of Choice A the correct reason per the basis document?
Is it more correct to say, "core uncovery may occur due to insufficient Siflow"?
72 F
3 B
E Choice B has low plausibility, given reactor is shutdown and primary safeties designed to maintain RCS <110% of design pressure (TS SL for pressure) on full turbine load reject. Reword distracter.
73 H
1 N
U Rewrote question.
Modify question and choices to ensure distractors more plausible.
74 F
3 N
E Modified as indicated.
75 F
3 N
S Not SRO-only. Possible to eliminate all distractor through application 76 H
3 X
N U
of RO system knowledge only.
Reselected KIA.. Wrote a new question.
Not SRO-only. Can be answered with RO knowledge only.
77 H
3 X
N U
Reselected KIA. Wrote a new question.
Choices C and 0 implausible and overall question LOD too low. This question is testing recall knowledge only, no analysis required. Re 78 H
1 X
N U
categorize as memory-level.
Wrote a new question.
Virtually a collection of T/F statements. Two distractor choices are not credible, given options in the other two choices. This question is testing recall knowledge only, no analysis required. Re-categorize as 79 F
1 X
N U
memory-level. Low LOD. Question wording doesn't preclude a distractor choice as an arguably 2nd correct ans\\'\\Ier. l', choice is a subset of the key answer.
Rewrote the question.
I I
I I I Page 8
ES-401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
LOK LOD Q#
(F/H)
(1-5)
Stem Cues TIF Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/M/N U/E/S Explanation Focus Dist Link units ward KIA Only Question is two-part whereas the choices are three-part. Doesn't look like the middle part is needed. Reword choices to align with question:
A.
Proc #1 - Mode X
- 8.
Proc #1 - Mode Y 80 H
3 M
E C.
Proc#2 Mode X D.
Proc#2 ModeY Modify formatting to improve readability. Use the "(1), (2)" format to break the question and the choices into distinct parts. Put the two parts on separate lines for each choice.
Recommend modify stem to have question align with desired answer.
Modified as indicated.
81 H
3 X
B U
Not SRO-only. Not tied to 55.43.
Rewrote the question.
82 H
3 N
E This question is a direct lookup with handout Handout will not be provided.
Not clear that selection of correct answer indicates a selection of mitigation procedure as expected for a question tied to 83 H
3 X
N E
10CFR55.43.b (5).
Modified choices to ensure answer choice demonstrates selection of a procedure.
First part of question appears to provide a poor basis on which to make a licensing decision. As written, question is memory-level, no analysis required. In general, common information in each answer 84 F
3 N
E choice should be fe-located to the stem to minimize word count and improve ease of reading f interpreting the question and answers.
Second part of two of the choices has low plausibility.
Modified stem and choices to address comments.
85 H
1 N
E Question too easy. One choice not plausible.
Modified stem and choices to address comments.
Page 9
ES-401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
LOK LOD Q# (F/H)
(1-5)
Stem Cues T/F Creel. Partial Job-Minutia
- /
Back-Q:::
SRO B/M/N UlE/S Explanation Focus Dist.
Link units ward KIA Only Not SRO-only. Can be answered with RO system knowledge only.
86 H
3 X
M U
No procedure selection required.
Wrote a new question.
Choice A is implausible.
87 H
3 B
E Rewrote Choice A.
88 H
3 N
S Memory-level question. Re-categorize as memory-level.
89 F
3 N
E Question re-categorized as memory level.
90 H
N S
KIA mismatch. Choices A and B implausible. The correct choice is the obvious choice. The "assessment of conditions" part of this question is not readily apparent in the choices given. Either document, on the question worksheet, the conditions being assessed 91 H
2 X
X N
U and the need for assessment in order to select the correct procedure or else fe-structure Question to more directly test assessment of conditions.
Rewrote question to address concerns.
92 F
3 N
E Minor rewording to address structural flaw.
Needs minor adjustment to stem to ensure requires SRO-only level 93 H
3 M
E knowledge of procedure to answer.
Stem adjusted to ensure SRO-only.
Minor choice rewording to improve symmetry and ensure correct 94 H
3 N
E answer is correct.
The distractor choices are implausible, and the key answer stands out as an obvious correct answer. None of the distracters are 95 H
2 M
U supported by normal operations or the stem conditions.
Revised question to address concerns.
Appears Question is tied to wrong item under 55.43 b. Correct H
3 N
E answer may not be entirely correct.
96 Modified Question to address comments.
I I
I I
I I
I Page 10
ES-401 2
Form ES-401*9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
LOK LOD Q#
(F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- 1 Back-Q= SRO B/M/N utE/S Explanation Focus Dist.
Link units ward KIA Only KIA mismatch. Question tests the generic KIA selected, but not in 97 F
3 X
M U
the context of the associated system.
Replaced with a modified bank question.
A distractor choice is a possible 2nd correct answer. Two of the 98 H
3 X
X N
U distracters are not discriminating.
Modified question and choices to ensure distracters are wrong and plausible.
99 H
3 M
E Minor modification to distracters to enhance plausibility.
100 H
3 X
X N
U Not SRO-only. Can be answered with RO system knowiedge.
Rewrote the question.
Page II