ML040160693
| ML040160693 | |
| Person / Time | |
|---|---|
| Site: | Prairie Island  |
| Issue date: | 09/08/2003 |
| From: | Cooper P Nuclear Management Co |
| To: | NRC/RGN-III |
| References | |
| 50-282/03-301, 50-306/03-301 | |
| Download: ML040160693 (7) | |
Text
ES-401 Written Examination Form ES-401-9 Review Worksheet The first 30 questions were the initial 30 questions reviewed. All questions were ultimately reviewed including K/As.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only R1 H
3 S
S R2 H
4 S
S New Question R3 F
2 S
S Reference supplied doesnt support the answer. The reference supplied is for the basis for the minimum operable level in the refueling water storage tank. Licensee to supply reference to support the answer.
Discussion with Licensee demonstrated that the initial answer provided by the licensee was correct.
R4 F
2 S
S R5 H
3 S
S New Question R6 H
2 S
E Question stem has Unit 2 in Mode 5 with the head removed. Licensee to correct the text to have initial conditions in stem to be Mode 6. Done Sat.
R7 F
3 S
S R8 H
3 S
S New Question R9 H
4 S
S Are operators required to have all EOP procedure steps memorized?
Reference supplied doesnt support the answer. Licensee to supply learning objective to show applicability of the question and a reference to support the answer supplied. Licensee supplied the learning objective.
Question Sat. New Question R10 H
2 S
S New Question
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
One or more distractors is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7.
At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9 Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only R11 H
2 S
U Edited to a higher level question.
R12 H
4 X
S U
Reference did not support only one correct answer. Answers C and D were both correct. Licensee agreed there was more than one correct answer supplied with the question and will edit a distractor so that there is only one correct answer. Question edited so that there is only one correct answer. Licensee pointed out that this question was originally classified as a low level question but is a high level question. The examiner agreed. Question is Sat. New Question R13 F
2 S
S This is a fundamental question. No reference provided. Licensee to provide reference. Reference provided Sat.
R14 H
3 S
S The reference is not specific enough to support the answer. The reference says the control group is C. B can be powered from either a safety or non-safety bus. Where does it say that at 72% power that B wont be powered from bus 122? Licensee to supply better reference.
Reference provided Sat. Showed that at 72% power heater group B would not be powered from bus 122.
R15 F
2 S
S The reference is not specific enough to support the answer. This is a memory question. Licensee to supply better reference. Reference provided Sat.
R16 H
4 S
S New Question R17 H
4 S
S New Question R18 F
4 S
S Are ROs expected to memorize EOPs? Reference does not support answer. Question references step 4, 12, and 13, the reference supplied (ECA 1.1) wasstep 20. Licensee to supply learning objective to show applicability of the question and a reference to support the answer supplied. Licensee provided learning objective. Reviewed steps 4, 12, and 13 of reference provided answer Sat.
R19 F
2 S
S This is a memory question. Licensee agreed.
R20 H
4 S
S New Question R21 F
4 X
S U
Are ROs expected to memorize procedures and steam tables?
Licensee to provide learning objective and will provide steam tables with exam. Licensee replaced K/A and question. 068 AK3.18. Replacement question provided had two correct answers. Licensee edited one distractor and question is Sat.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only R22 F
4 S
S Are ROs expected to know procedure basis? Licensee to provide learning objective. Licensee provided learning objective. Question Sat.
R23 H
3 S
S No reference provided. Licensee to provide reference. Reference provided Sat.
R24 H
3 S
S No reference provided. Licensee to provide reference. Reference provided Sat. New Question R25 H
2 S
S R26 F
3 S
E Explanation gives different answer than that provided with the question.
Licensee to edit question. Done Sat R27 F
3 S
S The answer supplied disagrees with the explanation provided.
Reference supplied does not support answer completely. Licensee will edit question and supply better reference. Edited and reference provided. Question Sat.
R28 H
2 S
U Edited to a higher order question. New Question R29 F
2 S
S R30 H
4 S
S R31 H
2 S
S R32 H
3 S
S No reference provided. Licensee to provide reference. Reference provided Sat.
R33 H
3 S
S R34 H
4 S
S R35 H
3 S
S R36 F
2 X
X S
U Are operators required to know TS bases? High pzr level trip is not a plausible distractor for DNB. Recommend replacing it with Over power delta T trip. The basis for which is linear heat generation rate. Learning objective provided and changes made as recommended. Question Sat.
R37 H
3 S
U Questions 37 and 39 present double jeopardy in that the failure to know that bistables need to be energized for trip can result in a wrong answer for both question. The licensee will replace question 37. Question rewritten Sat and raised to High Cog.
R38 H
3 X
S S
No reference provided cant verify correct answer. The containment is most likely to reach 2 psig before it reaches 120 degrees which could make pressure most limiting assuming no venting. Reference provided, question Sat.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only R39 H
4 S
E Explanation disagrees with the answer given. This is double jeopardy with question 37. If the candidate doesnt know that the containment spray bistables need to be energized to work both questions will be missed. Licensee will edit question such that explanation does not disagree with the answer given and replace question 37. Done Sat.
R40 F
3 S
S R41 F
2 S
S R42 H
3 S
S No reference provided. Licensee will provide reference. Reference provided Sat.
R43 H
3 S
S Licensee to supply reference to support answer. Reference provided Sat. Edit to use MW instead of % power in distractors to match procedure. Done Sat.
R44 H
3 S
S R45 H
3 X
S U
No info provided with question. No K/A, Answer, or reference. Licensee provided question with reference material. Distractors b and d are not credible because they obviously add feedwater to the S/Gs. Licensee will edit. Edited distractors question Sat.
R46 F
2 X
S U
A is a non-credible answer since 1of the two pumps is steam driven.
This is a memory question. Licensee agreed and will edit or replace question. Changed question format and distractors question Sat.
R47 H
3 S
S R48 H
2 S
U Edited to a higher order question.
R49 F
2 S
S This is a memory question. No reference provided. Licensee will provide reference. Reference provided Sat.
R50 H
2 S
S R51 F
3 S
S R52 F
2 S
S R53 F
3 S
S No reference provided. Licensee will provide reference. Reference provided Sat.
R54 F
3 S
S R55 F
3 S
S No reference provided. Licensee will provide reference. Reference provided Sat.
R56 H
4 S
S Cant verify answer no reference provided. Licensee will provide reference. Reference provided Sat.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only R57 F
2 S
S Cant verify answer no reference provided. Licensee will provide reference. Reference provided Sat.
R58 F
3 S
S Cant verify answer no reference provided. Licensee will provide reference. Reference provided question Sat.
R59 F
2 X
S U
C is not a credible distractor if panel does not exist. Licensee will edit question. Edited question but not explanation. Question Sat.
R60 F
2 S
E Cant verify answer no reference provided. Licensee will provide reference. Question stem to be edited for clarity. Question not clear that the question asked is for rising atmospheric pressure. Edited done Sat.
R61 H
2 S
S R62 F
3 X
S U
B and C appear to both be correct per the reference provided. Licensee will change distractor B. Done Sat.
R63 F
4 S
S R64 F
2 S
S R65 F
2 X
S U
Series of T/F distractors. Licensee will edit or replace questions.
Question edited. Question edited, now explanation differs from answer provided licensee will edit answer. Done Sat.
R66 F
2 S
S R67 F
2 S
S R68 F
2 S
S R69 H
3 S
U Explanation disagrees with the Answer given. C is the correct answer if the reactor is critical outside the predicted reactivity +/-750 PCM more info is necessary to make that determination. Licensee will edit question so that explanation and correct answer match. Licensee will provide a reference to the candidate with information available to determine if reactivity at criticality is +/- 750 pcm. Edited but licensee did not provide a copy of the reference that is to be provided to the candidate.
Reference provided question Sat.
R70 F
3 S
S The reference provided doesnt support the answer. Licensee will provide reference. Reference provided Sat.
R71 F
3 S
S Cant verify answer no reference provided. Licensee will provide reference. Reference provided Sat.
R72 F
2 S
S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only R73 F
3 X
S U
C is also correct answer. Licensee will edit distractor C to make it incorrect. Edit performed the answer is still incorrect.
R74 F
3 S
S Is this question appropriate for an RO? Licensee will provide a learning objective. Learning objective provided. Question Sat.
R75 H
3 S
S S1 H
3 U
S U
Question does not meet K/A. Licensee will rewrite question. New question meets K/A Sat. New Question.
S2 H
4 S
S E
Question as submitted had a Unit 2 Stem with Unit 1 answer and distractors. Drawing submitted as reference had an error in the drawing.
Licensee will edit question so that Unit in stem and distractors match.
Question edited Sat. New Question.
S3 H
3 U
S U
The K/A refers to events which are reportable per Federal Regulations.
The question discusses internal reportability for entry into TS 3.0.3. The K/A also refers to the Emergency Plan. Question changed to meet K/A Sat. New Question.
S4 H
4 S
S U
Cant verify answer no reference provided. Licensee will provide reference. Reference provided does not support answer. Unsat as provided. When reference was produced the initial answer to the question was incorrect. Question was edited Sat.
S5 H
4 X
S S
U Cant verify answer with the reference provided. Licensee will provide reference. D is not a credible distractor, cant have feed and bleed with all AC gone. Licensee will change distractor. Edited Sat.
S6 H
4 S
S S
S7 H
3 S
S S
S8 H
3 S
S S
S9 H
3 S
S S
S10 F
3 S
S E
Recommend dropping the reference to the EOPs in the answer to make it more correct. Done Sat.
S11 H
2 X
S S
U This is a direct look up question. The reference supplied instructs the candidate to raise pressure. Edited Sat.
S12 H
4 S
S S
S13 H
2 X
S S
U This is a direct look up question on pages B3.3.1-3 and B3.3.1-7 of reference provided to the candidate. If the TS were supplied without the basis question would be OK. Edited Sat.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only S14 H
3 S
S S
S15 F
3 S
S S
This is a memory question. The question can be answered by memory from the EOP. Licensee agreed.
S16 F
3 S
S S
S17 F
3 S
S S
S18 H
2 S
S S
S19 H
3 S
S S
S20 H
3 S
S S
S21 H
3 S
S S
S22 H
3 S
S E
The reference provided doesnt support the answer given. Licensee to supply reference and edit distractors. Reference provided and distractors edited Sat.
S23 H
3 S
S S
S24 H
3 S
S S
S25 F
2 S
S S
With Changes made there are less than 50% higher level quesitons (36 of 75 is 46%) on the RO exam. Licensee edited questions so that at final count there were 38 higher order questions which slightly greater than 50%.