ML100880246
| ML100880246 | |
| Person / Time | |
|---|---|
| Site: | Cooper  |
| Issue date: | 09/21/2009 |
| From: | Garchow S Operations Branch IV |
| To: | Nebraska Public Power District (NPPD) |
| References | |
| 50-298/09-301 | |
| Download: ML100880246 (10) | |
Text
ES-401 CG-2009-01 Written Examination Review Worksheet Form ES-401-9 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 10 OBDI 202 - IOLE Process
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation 1
H 2
B S
A, CR, 41.7 2
F 2
B S
C, CR, 41.4 3
H 3
U B
U S
C, CR, 41.7 - This question is a KA mismatch because it does not test on the RCIC logic power supply.
Will rewrite question to address comment.
This question was revised.
4 H
3 N
S C, CR, 41.7 5
H 3
U B
U S
D, CR, 41.6 - This question was unsatisfactory because there were no references provided and the question appears to have more than one correct answer.
Will rewrite question to address comment.
This question was revised.
6 H
3 N
S C, CR, 41.7
ES-401 CN-2009-10 2
Form ES-401-9 Page 2 of 10 OBDI 202 - IOLE Process
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation 7
H 3
E B
E S
A, CR, 41.7 - Recommend changing distractor A to Both valves will begin opening at the same time.
Will make suggested change.
The distractor was rewritten.
8 F
3 U
N U
S D, CR, 41.7 - This is an unsat question because it does not test on a design feature or interlock.
OK as written.
9 H
3 B
S B, CR, 41.4 10 F
2 N
S A, CR, 41.6 11 H
3 B
S B, CR, 41.4 12 H
3 B
S D, CR, 41.9 13 H
3 N
S D, CR, 41.5 14 H
3 U
N U
S A, CR, 41.7 - This question is unsat because it does not involve operating the AC distribution control.
Will rewrite the question.
The question was rewritten.
15 H
3 U
B U
S A, CR, 41.10 - This is a KA mismatch because it does not test on the impact losing IA will have on ADS; it is testing on what action to take which is a different KA.
OK as written.
16 H
3 U
U B
U S
D, CR, 41.4 - This question is unsat because distractor C could also be considered correct. I.e., the TCV would open further and the B pump should be started.
Secondly, CCW and TEC are not equivalent systems and therefore it is a KA mismatch.
This question will be rewritten.
The question was rewritten.
17 H
3 B
S A, CR, 41.4 18 F
3 B
S A, CR, 41.4 19 H
3 N
S D, CR, 41.7 - This question is satisfactory assuming the original picture is of better quality than the black and white picture reviewed.
20 F
3 B
S D, CR, 41.6
ES-401 CN-2009-10 3
Form ES-401-9 Page 3 of 10 OBDI 202 - IOLE Process
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation 21 H
3 N
S C, CR, 41.10 22 H
3 B
S B, CR, 41.7 23 H
3 B
S A, CR, 41.7 24 H
3 B
S C, CR, 41.7 25 F
3 E
B E
S A, CR, 41.6 - Distractor B could be argued as correct since if the SRMs are fully inserted then it would be prior to the SRMs being fully withdrawn.
Will modify distractor B.
Distractor B was modified.
26 F
3 B
E S
D, CR, 41.6 - CAF to ensure there are IRM shorting links as no references are provide and some BWRs do not have IRM shorting links.
Question is ok as written.
27 H
3 B
S B, CR, 41.10 28 H
3 U
B U
S B, CR, 41.7 - This is a KA mismatch because it down not test on the power supply.
This question will be rewritten.
The question was revised.
29 H
3 B
S B, CR, 41.6 30 H
2 B
S B, CR, 41.4 31 H
3 E
B E
S A, CR, 41.5 - A reference is needed for verification of the correct distractor.
This question is ok as written.
32 H
3 B
S C, CR, 41.8 33 F
3 B
S D, CR, 41.6 34 H
4 B
S B, CR, 41.10 35 H
3 B
E S
D, CR, 41.6 - The question states to use TS values. CAF to see if these are different than the current setpoints.
The note will be deleted referencing TS.
The question was revised.
36 F
3 N
S D, CR, 41.11 37 F
3 B
S C, CR, 41.4
ES-401 CN-2009-10 4
Form ES-401-9 Page 4 of 10 OBDI 202 - IOLE Process
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation 38 F
3 E
B E
S B, CR, 41.7 - It could be argued distractor C is correct since both valves are sealed open for at least 2 minutes.
Distractor C will be revised.
Distractor C was revised.
39 H
2 N
S B, CR, 41.5 40 H
2 U
N U
S D, CR, 41.5 - This is a KA mismatch because the question does not involve reactor pressure effects on reactor power.
Will revise question.
This question was revised.
41 H
3 B
S B, CR, 41.7 42 F
3 B
E S
C, CR, 41.10 - CAF that TEC is equivalent to CCW.
This question will be revised.
This question was revised.
43 F
3 B
S C, CR, 41.10 44 H
3 N
E S
C, CR, 41.5 - Reword the stem and distractors so they are grammatically correct.
Will correct grammatical errors.
The grammatical errors were corrected.
45 F
3 U
B U
S B, CR, 41.10 - The distractors are a collection of True / False statements.
This question will be revised.
This question was revised.
46 H
3 E
U B
U S
D, CR, 41.4 - This is KA mismatch because it does not test on the reason for the pump downshift.
This question will be revised.
This question was revised.
47 H
3 U
B U
S C, CR, 41.5 - This is KA mismatch because it does not test on the reason for the idle loop flow.
This question will be revised.
This question was revised.
48 F
3 B
S B, CR, 41.7 49 F
3 B
S A, CR, 41.6
ES-401 CN-2009-10 5
Form ES-401-9 Page 5 of 10 OBDI 202 - IOLE Process
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation 50 F
3 B
S A, CR, 41.4 51 H
2 N
S D, CR, 41.9 52 H
2 E
B E
S A, CR, 41.4 - There was no reference provided. The distractors seem arbitrary and the question would probably be thrown out if it were appealed.
This question will be revised.
This question was revised.
53 F
3 U
B U
S B, CR, 41.10 - This is an SRO Only question since it tests on the implementation of the E Plan.
This question will be revised.
This question was revised.
54 H
2 B
S B, CR, 41.4 55 H
3 B
S C, CR, 41.10 56 F
3 U
B U
S C, CR, 43.2 - This is an SRO Only question since it tests on TS bases. Recommend rejecting this KA and selecting another.
This KA will be replaced.
A new KA was selected and a new question was written.
57 F
1 U
B U
S A, CR, 41.10 - The distractors are not plausible since the three incorrect distractors all have the operator continue inserting the fuel bundle with rapidly rising SRM counts. These actions are counter intuitive.
This question will be revised.
The question was revised.
58 H
3 B
S A, CR, 41.5 59 F
3 B
S A, CR, 41.10 60 H
3 N
S C, CR, 41.11 61 H
3 B
S A, CR, 41.6 62 H
3 N
E S
D, OR, 41.10 - Not all EOPs are to be provided; discuss with facility.
This question will be revised to remove the references.
This question was revised.
63 H
3 B
S B, CR, 41.10 - Is this a closed reference question?
This question will be rewritten.
ES-401 CN-2009-10 6
Form ES-401-9 Page 6 of 10 OBDI 202 - IOLE Process
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation This question was rewritten.
64 F
3 U
B U
S C, CR, 41.10 - The distractors are a collection of true / false statements.
This question is ok as written.
65 H
3 B
S B, CR, 41.10 66 H
3 U
B U
S A, CR, 41.10 - This is a KA mismatch because it does not address a generic operator responsibility as required but instead is focused on a specific operator action taken in response to an event.
This question will be rewritten.
This question was rewritten.
67 F
2 N
S C, CR, 41.10 68 F
2 B
S D, CR, 41.10 69 F
3 U
B U
S A, CR, 41.10 - The distractors are True / False statements.
This question will be revised.
This question was rewritten.
70 H
3 E
N E
S C, CR, 41.12 - Reference not provided and per the information provided, the longest time prior to exceeding the limit is 60 minutes, not 55.
This question will be revised.
This question was revised.
71 F
3 B
S A, CR, 41.12 72 H
3 E
B E
S A, CR, 41.10 - Is this a closed reference question?
This question will be revised.
This question was revised.
73 H
2 N
S B, CR, 41.10 74 F
3 E
B E
S A, OR, 41.6 - Provide switch noun name in the distractors. If this gives away the correct answer, revise the question. We dont expect operators to have switch numbers memorized.
Question was be revised to reference the attaché electrical print.
75 F
3 N
S C, CR, 41.7 - Ensure the RCIC isolation switch is operated by an AO.
76 H
3 N
S C, CR, 43.5
ES-401 CN-2009-10 7
Form ES-401-9 Page 7 of 10 OBDI 202 - IOLE Process
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation 77 H
3 N
S C, CR, 43.5 78 H
3 N
S D, CR, 43.5 - Provide noun name for PCB 3310 and 3312.
79 H
3 U
N U
S D, CR, 43.2 - This is a KA mismatch because it does not focus on determining if a system/component is operable or a refueling accident.
This question will be rewritten.
This question was rewritten.
80 H
3 U
N U
S C, OR, 43.2 - This is a KA mismatch because it does not focus on determining if a system/component is operable.
This question will be rewritten.
This question was rewritten.
81 H
3 B
S C, OR, 43.2 82 H
3 N
S C, CR, 43.5 83 H
3 N
S A, CR, 43.5 84 H
3 B
S D, OR, 43.2 85 H
3 N
S B, CR, 43.5 86 H
3 N
S C, CR, 43.5 87 H
3 N
S D, CR, 43.5 88 H
3 E
B E
S A, OR, 43.2 - Change EARLIEST in stem to LATEST This will be changed.
The wording was changed as recommended.
89 H
3 E
B E
S C, CR, 43.2 - CAF, not sure distractor D is incorrect.
Question is ok as written.
90 H
3 B
S C, CR, 43.5 91 H
3 N
S D, OR, 43.2 92 H
3 U
B U
S D, OR, 43.2 - This is a KA mismatch because it does not test on a TS entry condition; it is testing on a TS basis.
This question will be revised.
This question was revised.
ES-401 CN-2009-10 8
Form ES-401-9 Page 8 of 10 OBDI 202 - IOLE Process
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation 93 F
3 E
E B
E S
A, OR, 43.5 - From the references provided it appears the max limit must be exceeded in two areas and not one as described in the stem. Second, the correct distractor states a scram is required when the EOP states a reactor shutdown is required using OP 2.1.5.
This question will be reviewed for technical accuracy.
This question was revised.
94 F
3 E
B E
S A, OR, 43.5 - No reference was provided to assess the question.
Will review the reference during validation.
95 H
3 B
S C, OR, 43.2 - CAF on what sections of TS 5.5 to be provided.
96 H
3 B
S D, OR, 43.2 97 H
3 U
B U
S C, OR, 43.5 - Since the stem does not state the status of alternate injection systems, depending on how the candidate interprets this step of the EOP would determine if distractor B, C, or D is correct.
Will review during validation.
This question was revised.
98 F
3 E
E B
E S
C, CR, 45.7 - Revise stem to read how many bundles to what is the maximum number of Otherwise, distractors A, B, and C are all correct.
Will revise this question.
This question was revised.
99 H
3 E
B E
S D, OR, 43.2 - Change EARLIEST to LATEST in stem.
Will make the indicated change.
The wording was changed.
100 F
3 B
S B, CR, 43.5 B= 55 F= 28 (37%)
E= 13 M= 0 H= 47 (63%)
U= 17 - 13 RO TOTALS:
N= 20 S= 45 Additional Notes:
B= 14 F= 4 (16%)
E= 6 M= 0 H= 21 (84%)
U= 4 - 4 SRO TOTALS:
N= 11 S= 15 Additional Notes:
GENERAL COMMENT
S:
ES-401 CN-2009-10 9
Form ES-401-9 Page 9 of 10 OBDI 202 - IOLE Process
- 1.
Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
- 2.
Chief Examiner comments are indicated in blue.
- 3.
Average difficulty is __2.83_ on the RO exam and 3.00 on the SRO exam. Both these values are within the normal expected difficulty range for this reviewer.
- 4.
The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 0 41.2 = 1 43.2 = 11 41.3 = 0 43.3 = 0 41.4 = 12 43.4 = 0 41.5 = 7 43.5 = 13 41.6 = 11 43.6 = 0 41.7 = 15 43.7 = 1 41.8 = 0 41.9 = 2 41.10 = 21 41.11 = 2 41.12 = 2 41.13 = 0 41.14 = 0
- 5.
The answer distribution is: RO / SRO A = 22
/ 4 B = 18
/ 2 C = 18
/ 11 D = 17
/ 8
- 6.
There are 2 / 12 (RO/SRO) questions with attachments provided.
- 7.
This review was performed by S. Garchow the week of 8/10/2009.
- 8.
The SRO exam contains 84% higher cognitive questions. This is significantly higher than the recommended limit of 60%.
- 9.
The number of higher cognitive questions on the RO exam exceeds the limit of 45 questions (60%) by two questions.
- 10.
There are a number of formatting, grammatical, and typing errors that need corrected.
- 11.
Based on the information in the questions, it appears the candidates are to be given a complete set of EOPs without the entry conditions. This is not reasonable. Will need to discuss what attachments will be given out as part of the exam.
ES-401 CN-2009-10 10 Form ES-401-9 Page 10 of 10 OBDI 202 - IOLE Process
- 12.
The SRO exam is heavily skewed away from distractor B. Redistribute the answers so there are at least 5 correct answers that are distractor B.