ML080700355
| ML080700355 | |
| Person / Time | |
|---|---|
| Site: | Arkansas Nuclear  |
| Issue date: | 02/29/2008 |
| From: | Garchow S Operations Branch IV |
| To: | Entergy Operations |
| References | |
| 50-368/08-301 | |
| Download: ML080700355 (8) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only B, CR, 41.5, NRC 2002 - Other distractors could be considered correct if SG U pressure is not specified and the condensate pumps are running. Also, D could be 1 F 3 U B argued to be correct in that 95% SG level would provide and adequate heat sink.
S The S/G pressure was be added.
D, CR, 41.10, NRC 2003 - Suggest changing B to lower PZR level for added H U credibility. KA mismatch in that the question does not test on a major component or 2 4 U B control.
F S This question was rewritten.
B, CR, 41.10, NRC 1998 - Also, should not ask a question for the primary reason, U this is like best answer. D could also be argued to be correct.
3 H 3 U B S Following a discussion with the facility, this question is ok as written. The word primary will be deleted from the stem.
A, CR, 41.3, NRC 1998 - KA mismatch. The answer to this question deals with U knowledge regarding medium or small break LOCAs, not LBLOCA. At least, the 4 F 2 U B knowledge does not deal with pumps DURING a LBLOCA.
S Following a discussion with the facility, this question is acceptable as is.
U B, CR, 41.10, NRC 1998/2005 - Notes need to be corrected: D is listed as B. This is 5 F 2 U B NOT a modified question because one condition in the stem was not changed. That S
makes it a bank question from the last two NRC exams. This is a KA mismatch Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only because the question does not test on the reason for the response but tests on the expected response itself.
This question was rewritten.
6 H 2 B S B, CR, 41.4 7 H 3 N S A, CR, 41.5 C, CR, 41.4 - This is a KA mismatch because the question does not test on the U
8 H 4 U N reasons for the response but the response itself.
S This question was revised.
9 H 4 B S A, CR, 41.7, NRC Exam 2007 10 H 2 B S D, CR, 41.7 11 H 3 B S A, CR, 41.5 12 H 3 B S C, CR, 41.10, NRC 2002 C, CR, 41.5, NRC 1998 - Add the SG pressure to the stem so that the condensate U system could not be considered as a SG feed option making more than one answer 13 H 2 U B correct.
S Upon further review of the question, it is ok as written.
D, CR, 41.8, NRC 1998 - This question does not test on the effect of battery U
14 L 2 U B discharge rates on capacity. Instead it tests on procedural requirements.
S This question was revised.
B, CR, 41.10 - This question does not test on the function or purpose of major U
15 H 2 U B components or controls but instead tests on operator actions.
S This question was revised.
16 H 3 N S B, CR, 41.4 17 H 3 B S C, CR, 41.6 B, CR, 41.4 - This question does not test on the reasons for the response but the H U 18 3 U B response itself.
F S This question was revised.
19 H 3 B S C, CR, 41.7, NRC 2002 20 F 2 N S A, CR, 41.2 U C, CR, 41.10, NRC 1998 - Answer A is also correct since 5.3 Hg exceeds 7.0 Hg.
21 F 2 U B S Upon further review of the question, it is ok as is.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 22 F 3 B S B, CR, 41.9, NRC 2002 23 F 3 N S B, CR, 41.9 A, CR, 41.10, NRC 2002 - This question does not test on the availability of main or U
24 H 3 U U B aux feed as required by the KA. Also, answer B could be argued as being correct.
S Distractor B was revised.
25 H 3 N S D, OR, 41.10 26 F 2 B S D, CR, 41.5, NRC 1998 27 H 3 N S D, CR, 41.10 28 H 2 N S D, CR, 41.7 C, CR, 41.10 - This does not match the KA because the question does not address U
29 H 2 U B the predict the impact part of the KA.
S This question was revised.
30 H 3 N S A, CR, 41.4 31 F 2 B S A, CR, 41.7 32 H 2 N S C, OR, 41.5 33 H 3 N S A, CR, 41.4 34 H 3 B S D, CR, 41.4 35 H 3 N S C, CR, 41.10 36 H 3 B S B, CR, 41.7 37 H 3 N S C, CR, 41.10 C, CR, 41.5 - This is a KA mismatch because the question does not involve the RPS H U 38 3 U N as required by the KA.
F S This question was revised.
39 H 2 B S A, CR, 41.7 40 H 3 B S D, CR, 41.7 A, CR, 41.8 - This KA is focused on the cooling for the containment spray pump itself U
41 H 3 U B and not the containment spray process flow.
S This question was reivised.
42 H 3 N S D, CR, 41.5
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 43 H 3 B S B, CR, 41.4 44 H 3 B S D, CR, 41.4 45 F 3 N S A, CR, 41.4 46 H 3 M S B, CR, 41.4 47 H 3 N S C, CR, 41.7 48 H 4 M S C, CR, 41.7 49 F 2 N S C, CR, 41.7 B, CR, 41.7, NRC 2002 - This is a KA mismatch because it does not test on the U
50 H 3 U B purpose of the load sequencer.
S After discussing with the facility, this question is ok as is.
51 H 3 M S B, CR, 41.11, NRC 2003 B, CR, 41.7 - This is a KA mismatch because it does not test on the electrical power H U 52 3 U B supply to the pump motor.
F S This question was revised.
U D, CR, 41.4 - This is a KA mismatch because it does not test on the IAS.
53 F 2 U N S This question was revised.
D, CR, 41.9 - This is a KA mismatch because it does not test on containment U
54 H 3 U B changes based on operating containment controls.
S This question was revised.
55 F 2 N S B, CR, 41.9 56 H 2 B S C, CR, 41.6 57 H 3 B S D, CR, 41.2, NRC 1998 58 F 3 B S D, CR, 41.9, NRC 2002 59 F 2 B S B, CR, 41.9 A, CR, 43.7, NRC 2000 - This is an SRO only question because it tests on refueling U
60 H 3 U B interlocks that are outside the scope of the control room operator.
S This question was revised.
61 H 3 M S C, CR, 41.7, NRC 1998 62 F 2 B S A, CR, 41.4
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 63 F 3 N S C, CR, 41.13 64 F 2 B S D, CR, 41.11, NRC 2002 E B, CR, 41.7 - This question is unnecessarily complex.
65 H 3 E M S This question is ok as written.
66 F 2 B S C C, CR, 41.5 - This question is U because all four distractors will raise Tave and U choosing the best method is not acceptable. Second it does not test on the function 67 H 3 U U B or purpose of a major component.
S This question was revised.
68 F 2 B S B, CR, 41.10 69 F 3 B S D, CR, 41.10 A, CR, 41. This question doesnt test knowledge of processes for determining U
70 F 2 U N reactivity effects per KA.
S After discussing with the facility, this question is acceptable as is.
E D, CR, 41.10; the second A distractor should be a D; 71 H 2 M S The typo was corrected.
A, CR, 41.10, NRC 2003; this question asks what action is NOT taken (backward -
U not operationally valid); doesnt match KA (question is about environmental releases 72 F 3 U U B vice excess rad levels and personnel exposures);
S This question was revised.
73 F 2 B S C, CR, 41.10, NRC 1998; U D, CR, 41.10, NRC 2002, B & C are too similar and not credible; 74 F 2 U B S This question was revised.
75 H 3 B S A, CR, 41.10, NRC 2005 76 H 3 M S B, CR, 43.5 77 H 3 M S A, CR, 43.5 78 F 3 B S B, CR, 43.3, NRC 2002 79 H 3 N S D, CR, 43.5; 80 F 4 M S C, CR, 43.5; NRC 2006; 81 H 3 U B U A, CR, 43.5; B is also correct;
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S This question was revised.
E 82 H 3 E M A, CR, 43.5; change stem to ...required RCS pressure criteria...;
S E
83 H 3 N C, CR, 43.5; in A, change six to 24; in B, change six to two; S
84 F 3 N S C, CR, 43.5; E B, CR, 43.2; This was submitted as a modified question, but its really still bank; C 85 H 3 X B is not credible as written (if C then D also - it needs to be distinguished from D); also, S change doors to door in C; 86 H 3 N S B, CR, 43.5 87 H 3 N S D, OR, 43.2 U C, CR, 43.5; doesnt test knowledge of the E-Plan in connection with MFW per KA; 88 H 2 U M S This question was revised.
E D, OR, 43.5; references provided did not support correct answer (no documentation 89 H 3 N S to support the 60 minute NRC notification requirement);
B, CR, 43.4, NRC 2002; C is also a correct answer; A is not credible in that it is not a U
90 F 3 U U B measure to prevent an accidental release per the stem; in is misspelled in D; S
Following a discussion with the facility, this question is ok as is.
U A, OR, 43.5; direct lookup; in the stem, Radiation Monitors is... is a grammatical 91 H 2 U U M E error; S The question will be revised to correct the grammatical error, otherwise it is ok.
C, OR, 43.5; challenge to containment criteria seem to be a judgment call which U would allow some candidates to argue that a GE is appropriate for this case (making 92 H 3 U N C and D correct);
S This question was revised.
D, CR, 43.5; it looks like the actions in either C or D are the same and candidate are U not generally required to have the attachments from a normal operating procedures 93 H 3 U M memorized.
S The question is ok as written.
D, CR, 41.10; this question only requires knowledge of the 55.53 condition, which is U RO knowledge; the stem and correct answer have not been significantly modified 94 F 2 U U B S from the original; B is also correct in that it meets the shift manning and exceeds (and therefore meets) the requirements of 55.53;
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only This question was revised.
A, CR, 43.5; question has not been significantly modified per ES-401 D.2.f; second half of answer is two narrow since no system temperature is given in the stem (there U may be no correct answer listed depending on RCS temperature), last, this is an RO 95 F 2 U U B question since it is testing on a system limit/caution. Suggest a question using tech S
spec basis.
This question was revised.
C, CR, 41.10; this appears to be RO knowledge since it is associated with an RO U lesson plan.
96 F 2 U B S The NOTES section of the question was revised to reflect why it is an SRO only question.
97 H 3 N S A, CR, 43.2; E
98 H 2 B D, CR, 43.5, NRC 2002, it should be an RWP in A and a specific RWP in B.
S C, CR, 43.4 - Based on the notes, it looks like terminating the release is the correct U
99 H 3 U N answer - which is B (step 4.17.5 of 2104.022);
S This question was revised to reflect the correct answer.
U A, CR, 41.10; this is RO LOK - this KA is not suitable for an SRO-only question; 100 H 3 U N S This question was revised.
B= 47 F= 26 E= 01 Additional Notes: 65% exceeds the 50 - 60% required band for higher cognitive RO TOTALS: M= 06 H= 49 (65%) U= 25 (33%) questions.
N= 22 (29%)
B= 8 F= 7 E= 5 SRO TOTALS: M= 7 H= 18 (64%) U= 11 (44%) Additional Notes:
N= 10 (40%)
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
- 2. Chief Examiner comments are indicated in blue in the explanation section.
- 3. Average difficulty is 2.68 on the RO exam and 2.8 on the SRO exam. The average difficulty on the RO exam is lower than the typical average (2.8 - 3.0) for other similar exams.
- 4. The 10CFR55.41/43 distribution is:
41.1 = 0 41.8 = 2 43.1 = 0 41.2 = 2 41.9 = 6 43.2 = 3 41.3 = 1 41.10 = 19 43.3 = 1 41.4 = 13 41.11 = 2 43.4 = 2 41.5 = 9 41.12 = 0 43.5 = 16 41.6 = 2 41.13 = 1 43.6 = 0 41.7 = 16 41.14 = 0 43.7 = 0
- 5. The answer distribution is RO / SRO:
a = 17 / 7 b = 19 / 5 c = 20 / 7 d = 19 / 6
- 6. There are 3 / 4 (RO/SRO) questions with attachments provided.
- 7. There are 30 questions (40%) on the RO exam that have been used in previous ANO-2 licensing exams. This is a magnitude greater than what has been seen on any other exam in this reviewers experience. Only 1 of the 30 was used on the previous 2 NRC exams. There are 5 repeat NRC exam questions on the SRO exam with one of those used on the previous 2 exams.
- 8. The 16 questions on the SRO exam from 10 CFR 55.43.5 (assessment of facility conditions and procedure selection) is higher than the typical 9-12 seen for this knowledge topic.