ML030800094
| ML030800094 | |
| Person / Time | |
|---|---|
| Site: | Quad Cities  |
| Issue date: | 02/04/2003 |
| From: | Lanksbury R NRC/RGN-III/DRS/OLB |
| To: | Skolds J Exelon Generation Co, Exelon Nuclear |
| References | |
| 50-254/02-301, 50-265/02-301 | |
| Download: ML030800094 (9) | |
Text
NRC COMMENTS ON THE OPERATING TEST AND THE WRITTEN EXAMINATION, INCLUDING ES-401-9 FOR THE QUAD CITIES EXAMINATION - DEC 2002
NRC COMMENTS FOLLOWING MATERIAL VALIDATION & VERIFICATION Admin JPMs General: Changes noted in submitted copy during validation week.
RO JPM A.1.a, was replaced due to identical JPM selected and conducted during re evaluation on licensee's audit exam failure. New JPM developed, agreed upon, and verified during validation week. New JPM, based on Review Printout of Thermal Limits Condition, with modifications. Comments noted on JPMs.
Systems JPMs General: Changes noted in submitted copy during validation week.
Simulator Scenarios General: Changes noted in submitted copy during validation week.
Written Examination:
General: Review comments noted in ES 401-9.
NOTE: Licensee's written examination development program randomly re-arranges the four distractors, therefore, the correct answer choice from initial written exam submital to the as administered exam will be different. Correct answers were verified.
ES-401 Written Examination Review Worksheet Form ES-401-9 Al
//
A//
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD I-i i
it (F/H)
(1-5)
Stem ICuesl T/F jCred. IPartial Job-IMinutia
- /
Back-Q=
SRO U/E/S Explanation Focusj j
J Dist.
Link units ward K/A Only H
2 x
x U
Low higher level question. First part of question requires thought to know that CRD pressure must be increased as power increases, i.e, affect pressure. Therefore, 2nd part of question is answered by the first part of the stem, pressure is affected. So to affect CRD, must open valve.
Appears to simplistic. Required changing all distractors and part of stem to focus on how differential pressure will be affected. Typo "presssure".
2 F
2 E
This is a memory question of color coding. H-6 is not moved so same color the entire time.
3 H
2 S
4 H
2 S
5 H
2 X
U No effect on power supplies, so how does this meet K/A on power supplies? Added Bus 13-1 fault. Typo "ocurred" should be "occurred" 6
F 2
S 7
H 3
E If given the procedure as a handout, it appears to be a simple look up.
Proposed to blank out the auto action section.
8 H
3 E
Two question 8 sheets in tab, corrected. Not bank question, but modified.
9 F
2 S
2g
_S Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the apropriate box if a job content error is identified:
T e question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference lest mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?.
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
TE: Margin noted as "N" are new questions and includes the thirty questions reviewed.
NO
ES-401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD
-IT I
I (F/H)
(1-5)
Stem Cues T/F Cred. IPartial Job-Minutia
- /
Back-Q= [SRO U/E/S Explanation Focus u
n Dist. J Link units ward 1K/A JOnly I
101 H
2 E
Incorrect reference, lesson plan was updated. Correct ref for final submital.
12 H
2 S
13 F
2 S
14 F
2 X
E Added clarification to one distractor and answer to include ref to SRM 21.
Al 15 F
2 S
16 H
3 E
Grammar was not clear. Add wording "Rx wtr IvI inst" in stem. May want to remove 450 psig ref on title of procedure. Appears similar to SRO #108?
/
17 H
2 S
178 H
2 U
Two distractors not credable, B and C. Recommend adding RCIC low pressure trip for B, and add "except overspeed" for C, or similar.
19 F
2 E
Second part of question contains common "a keylock switch on the.."
Move this phrase into question stem.
20 H
3 E
May need procedure #, and remove "only" from stem.
Al 21 H
2 E
Typo on 4thh line, "palces" should read "places". Remove "Target Rock" in item 3.
~,'
2 H
2 E
Remove identifiers of "first two" and "Target Rock" in each distractor. OK, as higher level, due to understanding power supplies.
Al 23 H
2 E
Simplify distractors, by removing common verbiage and focus on actions.
24'2 H
2 S
25 F
2 S
/*/
F 1
U Simple, not sufficiently discriminatory. Simple answer from memory Il patterned response. Changed question, changes direction of sych scope.
27 F
2 S
A/
278 H
2 E
Place "Halon" in stem of question. This will better fit the selected K/A.
29 F
3 S
Al 0
H 2
E Clarify distractors, use "less or more Neg" rather than increase decrease.
ES-401 3
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F jCred. Partial Job-Minutia
- /
Back-Q=
SRO U/E/S Explanation Focus Dist.
Link unitsjward K/A Only 31 H
2 X
X X
U Distractor A is missing reference to torus area water level. Distractors B &
D appear correct. CUE: torus leaks, torus area water level can do nothing but increase! Rewrote question to better enlist the distractors.
32 F
2 S
373 H
1 X
X X
X U
Too simple, not high level. New question written to focus on more operationally significant K/A and type of question. New question written at a higher level knowledge and K/A, G. 2.1.11. Changed to H.
34 F
2 S
36 F
2 S
37 F
2 E
Term "purging/deinerting" used in stem and all distractors but C. Removed wording in distractors to better focus on question.
38 F
2 S
39 F
2 S
40 F
2 S
4 H
2 X
U Does not meet K/A. Question needs focus on K/A, i.e., use of response instruction. Also, ref given does not justify answer. Need new question written to higher level knowledge. Changed to H.
42 F
2 S
43 F
2 S
- 44)
F 2
S H
2 S
Two distractors with words restore and maintaining, appeared similar. But, in general context they are different, question accepted as written.
46 F
2 S
47 F
2 S
48 H
2 S
No supporting information on function of Disch. Vol. Hi Wtr Byp keylock.
...... _Info received from licensee. Question more of higher level, reclassified.
A/
Al Al A,/
Ai
ES-401 4
Form ES-401-9 A/
gl A/
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD t
(F/H)
(1-5)
Stem Cues T/F
'cred. Partial Job-Minutia
- 1 Back-Q=
SRO U/E/S Explanation Focus Dist.
Link units ward K/A Only I I
4 H
2 X
X U
Technically answer is incorrect. Timer of the valve is OK, but if actuation press or ADS signal is in, it will not open even after the 14.5 sec delay.
Also, the answer notes only 14 sec. Clarified distractors and stem. Also, initially appeared question 22 may give cues, but determined acceptable.
50 F
2 S
51 H
2 E
Correct one distractor, change Group 1 to Group 2.
52 F
2 X
X U
Change 2 distractors to be more credible, noting 2 other QGA curves.
5 F
2 X
U Remove QGA-200 in stem no as not to cue the answer - primary containment control. Use plant condition instead, i.e., increased drywell temperature.... Title provided in question 66.
54 F
2 S
0 1 H 2
E Added sensors instead of sensor. Wrong ref to objective, K29 not K18.
56 H
3 S
57 H
2 S
58 F
2 S
59 H
2 S
60 H
2 E
Typo, "receive" not "receive."
61 H
2 S
62 F
2 X
X E
Not higher level, changed to F, unsat if insufficient number of high level questions. One distractor not credable, not pertaining to criticality.
Reworded others for better focus.
63 F
2 S
64 H
2 S
65 F
2 E
Memory question, changed to F. Unsat if insufficient number of higher level questions. Changed one distractor and stem, better focus.
66 F
2 E
Missed noting specific K/A on reference info.
67 H
2 E
Better wording on two distractors, but original were correct.
68 F
2 S
Al t4/
Al
ES-401 5
Form ES-401-9 jA' A/
Al Al Al 1
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues TIP Cred.
Partial Job-Minutia
- /
IBack-Q=
SRO U/E/S Explanation FocusJ j Dist.
Link units ward K/A Only_
69 H
2 S
70 H
2 S
71 F
2 E
Changed one distractor to note d/p pressure vice drywell condition which is a repeat of the given stem condition. Better distractor.
72 F
2 X
X U
Why is 'B' CR HVAC room a creditable distractor? How about near EHC skids? This does not appear to meet K/A (high off-site release rate).
Required fixing stem to include second part of K/A requirement.
73 H
2 X
U Two distractors too similar and appeared not credible. Changed distractor I
II IIwording to be more credible.
74 F
2 S
- 75_,
F 2
S 76 H
2 S
77 F
2 S
78 H
2 S
H 2
X U
Clarified two distractors for better credibility for SBLC switch position, not "OFF" rather Sys 2 or Sys 1. For better discrimination added 2A SBLC was electrically OOS. Required changing the correct answer.
8 H
2 X
X U
Appeared to give cue or double jeopardy with # 79. Changed focus of question to pump instead of squib. Changed all distractors to reflect the question changes.
81 F
2 S
82 H
3 E
Changed one distractor, to focus on IRM 13.
83 F
2 S
84 H
2 S
-25.
H 2
S 86 H
2 S
87 H
2 X
E Added RPV pressure in stem, changed wording in stem to better relate to the distractors.
88 H
2 S
ES-401 5
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1 -5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /
Back-Q=
SRO UIEIS Explanation Focus Dist.
Link unitsward K/A ON 8
F 2
E Appeared not too discriminating, but after discussion, question noted as acceptable with some changes. Changed one distractor from "five" to "four", and changed stem condition to Mode 3 instead of Mode 1.
90 F
2 E
Meets A4.02 K/A but not A4.04! Typo-Wrong K/A was noted, updated.
(91)
F 2
E Enhanced stem and distractor, better wording.
92 H
2 S
93 F
2 S
94 F
2 S
95 F
3 E
Changed from high to low, a memory of just the emergency dose limits.
96 H
2 S
97 H
2 S
SF 2
E Changed one distractor to be more credible of ATWS level ind location.
99 H
2 S
1700j H
2 X
U Ref given was incomplete. Need new question, not appropriate for RO, IIchanged K/A to be more operationally important, to K/A AA2.17.
101 H
3 E
Fixed one distractor to be more credible. Other minor wording changes.
102 F
3 S
(103)
H 3
X U
Changed two distractors A and C to be credible.
104 F
2 S
105 H
2 X
U Each distractor is a stand alone question for Tech Spec, an apparent True/False statements without need for any info in stem. Require rewrite to have stem conditions and why it meet Tech Specs.
106 F
2 E
Add clarifying wording in stem, for better focus.
1 7 H
2 S
Why is this marked as a higher cognitive question?
108 H
2 E
Better focus, added RPV press of 100 psig and upper insturment to -30".
109 H
2 S
10 H
2 E
Changed to high level. Requires T/S and operability understanding.
110_ H Changed wording in distractors for better focus.
'A 4/
/V V
/V Av' iv A1
ES-401 6
Form ES-401-9 Al A/
,4/
A/
Al A/
4/
A/
A/'
1
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /
Back-Q=
SRO U/E/S Explanation Focus Dist.
Link unitslward K/A ONy 111 H
2 E
Changed stem from Auxiliary to Reactor Building.
"112 F
2 S
113 H
2 E
Grammatical change to two distractors.
1 H
2 S
1 H
2 E
Consider reword stem, to give condition to determine battery is inop.
116 H
2 S
Ensure QGA given have entry conditions blacked out.
117 H
2 E
Change 250 deg F to 300, closer to limit.
118 H
3 S
119 H
3 E
Change items in stem to clarify better. Instead of torus level "normal" give torus level value. Change one distractor better wording, more credible.
"120*
H 3
E Change one distractor to state Initiate SBLC instead of start cond pump.
121 H
2 S
122 F
2 S
ýr/123 H
2 E
Enhance one distractor more credible.
124 F
2 S
125 F
3 S
126 127 128 129 130