ML031970786
| ML031970786 | |
| Person / Time | |
|---|---|
| Site: | Cooper  |
| Issue date: | 06/30/2003 |
| From: | Gody A NRC Region 4 |
| To: | Jason Christensen Omaha Public Power District |
| References | |
| 50-298/03-301 50-298/03-301 | |
| Download: ML031970786 (11) | |
Text
ES-401 RO Written Examination Form ES-401-9 (R8, S1)
Review Worksheet Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
F 2
S 2
H 2
S New 3
H 2
S 4
H 2
S New 5
F 3
S 6
H 3
S 7
F 3
S How does this match the K/A for control board indications? Licensee indicated that MO-74 is on the control board and must be located to determine if RWCU is properly aligned. Ok as is 8
F 2
S New 9
H 2
S 10 H
2 S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7.
At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9 (R8, S1)
RO Written Exam Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 11 F
3 S
delete NOTE from the stem. Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B.
Verified that answers are distributed between A-D. Ok as is.
12 F
2 S
13 H
2 S
New. Edit 2d sentence in stem (core spray pump flow is 5000 gpm). Licensee indicated they will modify. Verified change.
14 F
2 X
S New. Edit stem (power supply to the...Licensee indicated they will modify. Verified change.
15 H
2 S
16 F
2 S
17 F
2 S
New 18 H
2 X
S Delete note from stem Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B. Verified that answers are distributed between A-D. Ok as is.
19 H
2 S
20 H
2 X
S 21 H
2 S
New 22 H
2 S
23 H
2 S
24 F
2 S
25 H
3 S
New 26 H
2 S
27 H
2 S
Chg one of the distractors to NE Quad cooling fan units (Isthis possible? Different train of REC?). Not possible to change as requested - would have 2 correct answers - no change.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 28 F
2 S
New. Delete Note from Stem Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B.
Verified that answers are distributed between A-D. Ok as is.
29 F
3 S
30 F
2 S
ES-401 2
Form ES-401-9 (R8, S1)
RO Written Exam Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 31 F
3 S
Delete note from stem, new Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B.
Verified that answers are distributed between A-D. Ok as is.
32 H
2 X
S 33 H
2 S
New. Del note from stem; delete are before necessary. Change answers A and C to have C LPCI pump running?? More plausible?
Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B. Verified that answers are distributed between A-D. Based on licensee comments, leave as is - no changes recommended.
34 F
2 S
spell out Rod Worth Minimizer in stem. Will make change. Verified change.
35 H
2 X
S New 36 F
2 S
New 37 F
2 X
S 38 F
2 S
39 F
3 S
New. Spell out (MDEM) etc. Will make change. Verified change 40 F
2 X
S 41 H
2 S
42 H
2 X
S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 43 F
2 S
new SRO Only question - TS basis question. Answer D longer than other answers - psycometric. Reviewed the licensee comments and agree - a USAR questions. Verified change.
44 H
2 X
S New. SRO Only question - TS question. Misunderstood the licensees question, answer justification, and the need to provide T.S. reference.
This question is at the RO level. Ok as is.
45 H
3 X
S New. SRO only question - TS question. Licensee feel s this in an appropriate RO question. Discussed with licensee in more detail and they indicated, they would replace the question. Verified change.
46 H
2 S
SRO only question? The SRO would dispatch an operator. Agree with licensee comments - knowledge of entry conditions required for ROs.
47 H
2 S
Delete NOTE from stem Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B.
Verified that answers are distributed between A-D. Ok as is.
48 F
2 S
new 49 F
2 S
Needs operational validity. Reword to ask an operator action. K/A is an A not a K. Needs to test an ability not a knowledge. Licensee will replace this question. Verified change.
ES-401 RO Written Exam Form ES-401-9 (R8, S1)
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 50 H
2 S
51 H
2 X
S Direct lookup?? Recommend no reference. Licensee will remove reference from exam. Verified change.
52 H
2 S
53 H
2 S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 54 F
2 S
New. Needs operational validity. Reword to ask an operator action.
K/A is an A not a K. Needs to test an ability not a knowledge. Licensee will replace test item with different question. Verified change.
55 H
2 S
New.
56 H
2 X
S 57 H
2 S
Recommend no reference. Agree with licensee.
58 F
2 X
S 59 F
2 S
Shouldnt they know guidelines for returning to 1A w/o a ref.
Recommend no reference. Agree with licensee comments - no changes 60 H
2 X
S New. The way the ques is written, it appears obvious. Licensee indicated no change needed - disagree. Replaced with a new question.
61 F
2 S
SRO only question. K/A is Part 20 not CNS limits. Agree with licensee comments - based on CNS limits as per 2nd part of K/A statement.
Agree with licensee that NOT SRO only.
62 F
2 S
New. Basis Question. Recommend no reference. As written, appears to be an SRO question. Agree with licensee comments.
63 F
2 S
New. Delete (EOP-5A Step SC-12) Agree with licensee - no change.
64 H
2 S
65 F
2 S
66 F
2 S
New. Recommend no reference. Needs operational validity. Reword to ask an operator action. Ok as is per licensee comments.
67 H
3 S
Isnt the SS responsible for all shift surveillances? Licensee indicated no change necessary - disagree. Level of difficulty is low. Need a different question. Replaced question.
68 F
2 S
ES-401 2
Form ES-401-9 (R8, S1)
RO Written Exam
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 69 F
2 S
replace; too low in difficulty as written. Licensee disagrees. I reviewed and the candidate does have to identify the alarm is on the other division, so the question probably has a LOD of 2.
70 F
2 S
71 H
2 S
72 H
2 X
S 73 F
2 X
S Is the answer only a partially correct answer? Based on ans descr.
Licensee is modifying question. Verified change 74 F
2 S
75 F
2 X
S 76 F
2 S
77 H
2 X
S 78 H
3 S
79 F
3 S
New. Needs operational validity. Reword to ask an operator action.
Agree with licensee position, no change.
80 F
2 X
S 81 F
3 S
82 H
2 X
S New 83 H
3 S
New 84 H
2 S
85 F
2 S
New 86 H
3 S
87 F
2 S
New 88 F
2 S
ES-401 2
Form ES-401-9 (R8, S1)
RO Written Exam Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 89 F
2 S
New. Delete note from stem Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B.
Verified that answers are distributed between A-D. Ok as is.
90 H
2 S
edit stem Using Main Turbine Bypass... Question will be modified.
Verified change.
91 F
2 X
S Change Ans D to include: valve opening is teminated by torque switch Question will be modified. Verified change.
92 H
3 X
S Ans A or B need to be modified. Currently, neither one are valid distractors. Licensee stated that distractors cant be modified to improve plausibility - will replace question. Verified change 93 F
2 X
S 94 H
2 S
SRO only question - refueling and definition of core alteration?
Licensee has a learning objective and part of SAT program. OK 95 H
2 S
New. Direct LU?? SRO only question - TS action statement and LCO time. Licensee has a learning objective and part of SAT program. OK 96 H
2 S
New ???? Is this a dir LU??? SRO only question - TS action statement Licensee has a learning objective and part of SAT program. OK 97 F
2 X
S Needs operational validity. Reword to ask an operator action. Agree with licensee comments, leave as is.
98 H
2 S
New delete NOTE from stem Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B.
Verified that answers are distributed between A-D. Ok as is.
99 H
2 S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 100 H
2 X
S delete note from stem. Could Answer C also be correct, although not the minimum? Recommend changing answer. Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B. Verified that answers are distributed between A-D.
Ok as is.
ES-401 SRO Written Examination Form ES-401-9 (R8, S1)
Review Worksheet Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 75 F
2 X
S 76 F
3 S
77 F
2 X
X S
New. How can power lower to max stpt?? Licensee will modify Answer B. Verified change.
78 H
2 S
New. This is a sys ques not plt assessment and selection of norm, abn, or emerg proc. Not SRO only. Licensee will modify to change the stem to ask operability and link to 55.43(2). Verified change 79 F
2 S
New. How is this 43.5? appears just 41.5. Not SRO only. Licensee believes this question is linked to 55.43(5). I disagree - 55.41(5).
Licensee agreed to modify. Verified change 80 F
2 X
S New. Not SRO only. Leave as is - agree with Licensee comments.
81 H
4 S
New. Why is this SRO only? Similar questions on RO exam - what distinguishes this at the SRO level? Is this a T/S Limit? Agree with Licensee, no changes required, also probably 55.43(6 or 7).
82 H
2 S
Modify Answer B - reads like you are refueling, not operational.
Recommend no reference. The SROs should know this. Licensee is removing T.S reference for question. Verified change.
83 H
3 X
S 84 H
2 S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7.
At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9 (R8, S1)
SRO Written Exam Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 85 H
4 X
S Shouldnt they know by memory 1 hr TS conditons? Recommend no references. Licensee is removing T.S. 3.1.6 reference. Verified change.
86 F
2 S
New 87 H
2 S
88 F
2 S
New.
89 F
2 S
New. Not SRO only. Systems level question. Agree with licensee. No changes recommended.
90 H
2 X
S New. Delete note from stem. Not sure the note is necessary. But does specify arranged by order of magnitude as discussed in App B.
Verified that answers are distributed between A-D. Ok as is.
91 F
2 S
New 92 H
2 S
New.
93 H
2 S
New.
94 H
2 S
Should the correct answer be 7000 gpm vs 6500 gpm (per answer justification)? Topic is good, question appears to have more than one correct answer. Reword question. Agree with licensee comments and resolution. Verified change.
95 H
2 S
New. 43.5???? no procs involved. Not SRO only. Need to change something. Licensee will modify. Verified change.
96 H
2 X
S 97 H
2 S
98 F
2 S
99 F
2 S
As written, appears to be both an RO/SRO only question. Modify to make more operational valid. Reviewed licensee comments and discussed with licensee. The SM is responsible for both PSE and routine dose tracking of operators - leave as is.
100 H
2 X
S How does this match the K/A. Agree with licensee comments, no changes necessary.