ML040130722
| ML040130722 | |
| Person / Time | |
|---|---|
| Site: | Monticello  |
| Issue date: | 09/23/2003 |
| From: | Lanksbury R NRC/RGN-III/DRS/OLB |
| To: | |
| References | |
| 50-263/03-301 50-263/03-301 | |
| Download: ML040130722 (8) | |
Text
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S
- 7. Explanation Monticello 2003 Exam Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1-R H
2 S
3 2-B F
1 U-1 3 {LOD <2 #1} Meets K/A, however due fundamental information and question construction this question exhibits very low discrimination.
LOD does not meet threshold of acceptability. Replace question.
Resolution, final submittal question based on new K/A {201002A1.03},
question is SAT 3-R H
2 3
4-R H
2 E-1 3 Change distractor D explaination. Resolution, final submittal contained additional explianation.
5-B H
2 E-2 3 {Low LOD #2} When the stem contains too large a series of conditions, there are inadvertently more clues provided, allowing the applicant to dismiss one or conditions embedded in the distractors; thus discrimination is lessened. The problem lies not in the format, but in the degree of verbiage that, in turn, provides hints. Each of the 5 conditions can be treated as a T/F statement against the events to determine the correct sequence. Stem says Div I out of service (oos), #4 only item with Div I in statement, eliminate Div I oos from stem and add to the first sentence only the following events occurred.
Resolution, final submittal question incorporated enhancements, question is SAT Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7.
At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 6-B H
3 U
U U-2
{DDS #1} Fundamental problem with the "double distractor set"
{DDS} question type, set too easily lends itself to low discrimination (e.g.
50/50) since the same distractor options are repeated twice alternatively within the set. Knowing one part of the answer allows 2 other distractors to be dismissed easily. See Appendix B.C 2.a. (Models C / D) by extending distractor quality via adding reasons (because, since, etc.) to distractor options. Distractor B and C option of starting an RHR pump and submitting a work order not plausible. Considering the typical time required to process a work order, would not expect a minimally competent operator to accept running RHR that long. Per the explanation given if PCV-2459 oos... bypass... & start RHRSW... A is partially correct and applicant can argue pressure irrelevant because of the procedure. Specific Determiners (C.2.m. (2)).
Resolution, final submittal question rewritten and is SAT 7-B H
1 U
U U-3 3 {DDS #2} {LOD <2 #3} Does not meet K/A. Given question construction, preventing pump damage fundamental GFES knowledge, thus LOD <2. {Question 10-B gives cue to pump min flow protection and 10 sec. time.} Resolution, final submittal question rewritten and is SAT, as rewritten question is H vs F.
8-B F
2 U
U E-3 3 Answer A change to RHR-62 and CST-183 has not been operated.
Specific Determiners (C.2.m. (1/9)). Resolution, final submittal question has enhancements incorporated and is SAT.
9-R H
2 S
3 Need technical proof that HPCI Aux Oil pump still required at 1500 rpm, otherwise applicants can make assumptions of normal ops &
possibly justify either A or C as correct. Resolution, final submittal includes additional references to justify the 1500 rpm.
10-B F
2 S
3 Provides cues to question 7-B, change distractor C to 5 sec.
Resolution, final submittal has enhancement incorporated plus question 7-B rewritten such that cue no longer exists.
11-S F
3 U
X E-4 3 Declaring the system inoperable per TS does not qualify for the 2nd part of the K/A use procedures to correct, control, or mitigate consequences of those abnormal conditions or operations. Resolution, final submittal rewrote 2nd part of each choice to enhance compliance with and part of the K/A.
12-B H
2 S
{DDS #3}
13-B F
2 S
14-B H
3 S
MODIFIED Previous MNGP NRC Exam 8/23/1999 15-B H
2 U
U U
U-4 Correct answer is like inform the Shift Supervisor, i.e. always correct.
Choices A, C, & D are physical operator actions whereas the answer is a communication. Specific Determiners (C.2.m.). Replace or significant rewrite. Resolution, final submittal is SAT.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 16-R H
2 E-5 Which of the following required to continue the reactor startup?
Resolution, final submittal question has enhancement incorporated.
17-B F
2 S
18-R F
2 S
19-R F
2 S
3 20-B H
2 E-6 Change sentence structure of C to remove distration of 6th bypassed LPRM INPO - Dresden 1998 Resolution, final submittal question has enhancement incorporated.
21-B H
2 E-7 Put CST level in stem and put HPCI suction does not transfer Resolution, final submittal question has enhancement incorporated.
22-B H
2 U
U U-5
{DDS #4} {LOD <2 #4} gas/boiling & filling/cooling + GFES level knowledge. 216000 Al. Ability to predict and/or monitor changes in parameters associated with operating the NUCLEAR BOILER INSTRUMENTATION controls including: A1.01, Recorders and meters Question as written does not match K/A, illicit reason for changes in recorders and meters not predicting/monitor those changes. REWRITE QUESTION. Resolution, final submittal question was rewritten to resolve issues and is SAT.
23-R H
2 S
24-B F
2 S
25-R F
1 U
U-6
{DDS #5} {LOD <2 #5} open/closed runout/dead head Cant do verbatim from procedure. REWRITE QUESTION Resolution, final submittal question was rewritten to resolve issues and is SAT, plus as rewritten question is Fundamental.
26-B H
1 U
U-7 3 {LOD <2 #6} Distractor A not credible because stem states it fails OPEN. Distractors B & D minimally credible due to explanation there is no backup pressure regulator might accept one but not both. **look at PID for A. REWRITE QUESTION. Resolution, final submittal question was rewritten to resolve issues and is SAT.
27-B F
3 E-8 Suggest changing distractors C and D from 190 to 200 degrees {alarm setpoint} to add to plausibility. Resolution, final submittal question has enhancement incorporated.
28-B F
2 Modified Clinton 2001 NRC Exam 29-B F
3 U
E-9 2002 NRC Exam - INAPPROPRIATE NEGATIVES (C.2.e; E.3)
Resolution, final submittal question stem reworded.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 30-B H
1 U
U U-8
{LOD <2 #7} Stem asks locations provide DW temp, item 4 states it provides DW temp, therefore has to be included. Asking for SPDS does not discriminate {fundamental knowledge repeatedly reinforce during simulator drills}. Applicant knows 2/4 correct, applicant can assume distractor C not correct since it does not contain 4. If distractor C not correct and it contains 2/3 then B which also contains 3 can not be correct. In the end a minimum of 50/50 choice between A and D.
REWRITE QUESTION Resolution, final submittal question was rewritten to resolve issues and is SAT. {New question is higher order.}
31-B H
2 S
32-S F
2 X
E-10 3 Enhance distractor B to decalre RHR SDC inop restore in 30 days and Change time frame on A to initiate shutdown. Resolution, final submittal question has enhancement incorporated.
33-B F
2 S
34-S H
2 X
S 3
35-B H
3 S
3 36-R H
3 E-11 Reword distractor D slightly, leave 15% power in stem. Resolution, final submittal question has enhancement incorporated.
37-B H
1 U
E-12
{DDS #6} {LOD <2 #8} 50/50 {12%/15% & CVs OPEN/CVs CLOSED}
Resolution, final submittal question enhancement to reduce usage of double distractor set questions.
38-B H
2 U
E-13 E vs U due to question used on 2002 NRC Exam. Good question, however, the question as written does not address the... as they apply to the Reactor/Turbine Pressure Regulating System. K/A = Knowledge of operational implications of {Turbine inlet press vs. reactor press} as they apply to the Reactor/Turbine Press Reg Sys. +level of procedural memory recall. Replace question. Resolution, final submittal included a replacement question and is SAT.
39-R H
2 S
{DDS #7}
40-R H
2 E-14
{DDS #8} Change 2nd part to all choices for procedure selection to operator actions. Resolution, final submittal question has enhancement incorporated.
41-R F
2 S
1999 NRC Exam 42-S F
2 X
S
{DDS #9}
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 43-R H
2 U
U U-9
{LOD <2 #9} Distractor choices other than answer are fundamental, given Reactor Scram statement in the stem first sentence, that the question does not meet the minimum level of difficulty for discrimination.
Could assume B and D are cause of scram therefore correct. Remove Reactor Scram statement from first sentence and replace answer with a choice closer to distractors. Resolution, final submittal question was rewritten to resolve issues and is SAT.
44-B H
2 E-15 Clarified the concern in choices B and D. Resolution, final submittal question has enhancement incorporated.
45-B H
4 S
{DDS #10}
46-R H
2 S
47-R F
2 S
{DDS #11}
48-R F
2 E-16 INPO Bank - LaSalle 2000 Change distractor c back to bank distractor.
Resolution, final submittal question has enhancement incorporated.
49-S H
2 X
E-17 3 Change stem statement from EDG 13 not available due to LC-107 lock-out, to EDG 13 unavailable. Applicants get a copy of EALs on exam. Resolution, final submittal question has enhancement incorporated.
50-B H
3 S
3 51-B F
2 S
3 52-B F
3 S
53-B F
2 S
54-B H
1 U
E-18
{DDS #12} Change distractors to four vent fan trips, eliminate DDS.
Resolution, final submittal question enhanced to reduce usage of double distractor set questions.
55-S H
3 U
X U-10 3 Distractors A and B not credible, rewrote distractor B Resolution, final submittal question has distractor B rewritten.
56-B F
2 S
57-B H
2 U
E-19 Take Immediately out of B. Resolution, final submittal question has enhancement incorporated.
58-B H
2 E-20 59-S H
2 X
S 3 {DDS #13}
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 60-S H
1 U
X U-11 3 {LOD <2 #10} Direct look-up. Remove 125/250 VDC from applicant TS references/handouts? Resolution, final submittal question has required changes incorporated 61-S H
2 X
S 3 {DDS #14}
62-B F
2 S
3 63-R H
2 S
64-B H
2 E-21 Remove performance of procedure 0255-07-IA-1 from distractor C.
Resolution, final submittal question has enhancement incorporated.
65-B F
2 S
66-B H
2 U
U-12
{LOD <2 #11} What supports the credibility of distractors B, C, or D Can applicant make assumptions and argued B is correct?} What is the concern in distractor C, stem ask for concern. REWRITE QUESTION Resolution, final submittal question was rewritten to resolve issues and is SAT.
67-B H
2 S
3 {DDS #15}
68-R F
2 E-22 3 {DDS #16} {LOD <2 #12} 50/50 B or C. Resolution, final submittal question enhanced to reduce usage of double distractor set questions.
69-B H
1 U
U-13
{DDS #17} {LOD <2 #13} 50/50 Resolution, final submittal question changed stem conditions sufficiently to raise difficulty to 2.
70-B H
2 S
71-S F
2 X
S
72-B F
1 U
E-23
{DDS #18} {LOD <2 #14} 50/50 either A or B. Distractors C and D less than minimally credible Torus cooling used in all modes.
Resolution, final submittal question enhanced to reduce usage of double distractor set questions.
73-B H
3 E-24 Rework stem for clarity. Resolution, final submittal question has enhancements incorporated.
74-S H
2 S
{DDS #19}
75-B F
2 S
76-B H
3 S
77-B H
2 S
3
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 78-S H
2 U
X E-25 Per Answer/Distractor Explanation classification is based on loss of 2 of 3 fission product barriers with a potiential loss of the 3rd barrier.
Applicant does not have to demonstrate knowledge of impact of High Off-site Release Rate on EAL. REWRITE QUESTION. Resolution, final submittal question was rewritten to resolve issues and is SAT.
79-R H
3 S
80-B F
2 S
81-B H
3 S
82-B F
2 S
83-B H
2 S
3 84-S F
2 X
E-26 3 {DDS #20} Use info from basis to change pressure on C to 1363 and on B to 1250. Resolution, final submittal question has enhancements incorporated.
85-S H
2 X
E-27 Stem statement It is required to... does not make since. Add actions to previous sentence. Resolution, final submittal question has enhancements incorporated.
86-B H
1 U
U-14
{DDS #21} {LOD <2 #15} a part of double distractor set is less than minimal level of difficulty thus question becomes 50/50 between B and D. {1021 App B page 21 item C.} Resolution, final submittal question was rewritten to resolve issues and is SAT, plus question as now written is higher order {H vs F}.
87-S H
2 X
E-28 Rework stem for clarity. Resolution, final submittal question has enhancements incorporated.
88-S F
2 X
S 89-R H
2 S
90-S H
2 X
S
{DDS #22}
91-S F
2 X
S 92-B H
2 S
{DDS #23}
93-B H
2 E-29 CHANGE stem, eliminate HPCI bullets, bullet other conditions and add torus temp @ 83F. Resolution, final submittal question has enhancements incorporated.
94-B H
2 S
{DDS #24}
95-B H
2 S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 96-R H
2 U
U-15
{DDS #25} Simplified stem and changed distactors A & B to open 3 ads valves because..... and changed C & D to shutdown the reactor because... Resolution, final submittal question was rewritten to resolve issues and is SAT.
97-B F
2 U-16 Removed word immediately from Distractor A. and change to Reactor Bldg area for rad condition to meet K/A. Resolution, final submittal question was rewritten to resolve issues and is SAT.
98-B H
2 S
99-B H
2 S
3 100-B F
1 S
101-R H
2 S
{DDS #26}
102-B F
2 E-30 Changed distractor pressures to increase plausibilty. Resolution, final submittal question has enhancements incorporated.
103-S H
2 X
S 104-B H
2 E-31 Build A & B like C & D, using equipment nomenclature. Resolution, final submittal question has enhancements incorporated.
105-B F
2 E-32 Leave out RR pump Anns and add slightly elevated RR seal temps.
Add distractor run RR to min & trip reactor. Resolution, final submittal question has enhancements incorporated.
106-R F
2 S
107-S H
2 X
S 108-B F
2 S
109-S F
2 X
E-34 Change C to Ensure two operable SRMs, with atleast one in each of the adjacent quadrants. Resolution, final submittal question has enhancements incorporated.
110-S H
2 S
111-B H
2 U-17
{LOD <2 #16} Change stem, 1) remove statement Stator Cooling Pumps can not be started 2) Added bullet immediate actions taken, 3) 2nd bypass valve full open. Change distractors C &D to operator actions to be taken ie scram push buttons & reduce RR flow. Made LOD 2 &
LOK H. Resolution, final submittal question was rewritten to resolve issues and is SAT.
112-R H
1 U
U U-18
{DDS #27} {LOD <2 #17} + change so does not use direct wording out of text definition. Resolution, final submittal question was rewritten to resolve issues and is SAT.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 113-S H
2 x
U-19 Question as written is SRO Only. Resolution, final submittal question was accepted as SRO Only.
114-B H
2 S
115-B F
2 S
3 116-R H
2 S
117-B F
2 S
118-B H
2 E-35 Slight wording change to distractor A INPO Bank Dresden2 2002.
Resolution, final submittal question has enhancement incorporated.
119-S H
2 X
S 120-R F
2 S
Modified from 2002 NRC Exam question.
121-S F
2 X
E-36 Fuel Handling Supervisor be substituted/included in one or more of the choices. Resolution, final submittal question has enhancement incorporated.
122-S H
2 X
E-37 INAPPROPRIATE NEGATIVES (C.2.e; E.3) Reword last sentence in stem for clarity. Resolution, final submittal question has enhancement incorporated.
123-B H
2 U-20 Reword stem to include reactor power level, and include a Core Spray Ob injection valve open in the stem. Reword A to make it the correct answer {MO-2398}. Coreect flaw in distractor C whichmakes it incorrect. Resolution, final submittal question was rewritten to resolve issues and is SAT.
124-S F
2 X
S 125-B F
2 U-21 As written the given correct answer B is not correct, add words to D about moving steam dryer with crane. D is correct answer.
Resolution, final submittal question was rewritten to resolve issues and is SAT.
126-B F
2 U
E-38 Reword stem for clarity. Resolution, final submittal question has enhancement incorporated.
127-S F
2 U-22 Conflicts with Admin JPM A.1.a Resolution, question was deleted 128-B F
2 U
U-23 K/As provide for Fueling Handling tasks, K/A 2.3.3 seeks knowledge of waste disposal and handling systems. REPLACE THE QUESTION.
Resolution, final submittal question was new and is SAT.
Symbol clearly distinguishes questions used for 30 question initial review.
Symbol clearly distinguishes those questions which received independent verification of indicated references/Sample verification for licensee generated exam/100 percent verification for NRC generated exam.
Symbol clearly distinguishes questions used for K/A conformance sample review/At least 30 questions with an emphasis on those taken directly form exam bank/100 percent review if more than 20 percent of sample does not match K/A statement.