ML120310427
| ML120310427 | |
| Person / Time | |
|---|---|
| Site: | Grand Gulf  |
| Issue date: | 12/02/2011 |
| From: | NRC Region 4 |
| To: | Entergy Operations |
| References | |
| 50-416/11-301 | |
| Download: ML120310427 (7) | |
Text
ES-401 GG-2011-12 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only C[41.8 ] Answer B is also correct because it too connects electrical power to the E12-1 F 3 N E F008, only it is not direct. Reword the stem to The supply breaker that directly connects electrical power to Fixed 2 H 2 B S D[41.7]
3 H 4 B S A[41.7]
A [41.10] Recommend adding While in the Restricted Region, to the stem. Also, 4 F 2 B E increase 2% on distracter B to 8-9%, closer to set point of 10% for increased plausibility. Fixed 5 F 2 N S B[41.10]
6 H 2 N S A[41.3]
7 H 3 B S C[41.7]
8 H 4 N S B[41.4]
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process
ES-401 GG-2011-12 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 9 H 2 B S D[41.4]
10 H 2 N S HANDOUT C[41.10]
11 H 2 B S A[41.7]
- 1. B[41.5] Isnt going above 35% with BPVs open violating 12 H 3 B E procedure? Reference Q52. If so, change reactor power from 45% to something less than 35% but still above the LPSP. Fixed 13 F 2 N S C[41.7]
B[41.8] Recommend changing distracters to 6, 8, and 12. Reasoning being that 2 14 F 2 N E seems too short and 24 seems too long. Fixed 15 H 3 N S A[41.10]
16 H 2 X N U C[41.10] A and D not plausible. SLC injection rate not level dependent. Fixed A[41.7] Distracter D should just say Closed - This will balance out correct answer 17 H 2 B E A in length and enhance the distracters psychometrically. Fixed C[41.7] Why now in the question stem. Its already in the setup to the question -
18 H 2 N E now at -50 inches. Fixed D[41.7] Correct answer D can be answered by only knowing the second part dealing 19 H 2 B E with Alternate Feed. Its the only one with the ON first OFF second variation. Fixed 20 F 2 N S C[41.4]
21 F 3 N S B[41.10]
A[41.7] Based on the training material, the given power between 10^2 and 10^5 is below the threshold of exceeding 10^5, therefore, a rod block does not exist. The 22 H 2 N E stem states the Upscale Trip unit is failed; therefore, only the RPS function is affected, and this is bypassed by the shorting links. D seems to be the correct answer. Fixed 23 H 2 B S B[41.4]
24 H 3 B S C[41.7]
25 F 2 N S D[41.7]
26 H 2 N S B[41.8]
27 F 2 N S A[41.10] Page 2 of 7 OBDI 202 - IOLE Process
ES-401 GG-2011-12 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 28 H 2 N S A[41.2]
29 H 2 N S D[41.10]
30 H 3 N S D[41.3]
31 F 2 B S C[41.10]
32 F 2 B S B[41.10]
33 F 2 N S B[41.12]
34 F 3 N S C[41.10]
35 H 2 B S B[41.7]
36 H 2 B S HANDOUT C[41.10]
A[41.7] The question is a giveaway. It gives the applicant a controller and its setpoint, then defines what is a higher d/p and finally asks the applicant which of two 37 H 1 X X N U controller setpoints (one above and one below the current setpoint) would provide a higher output. Fixed D[41.8] The GGNS simulator is not a good technical reference. The simulator only 38 F 2 N E does what its told. Somehow the shutoff head of the pumps may have something to do with flow rates and pressure or there is some validation from the plant, etc. Fixed 39 F 2 N S B[41.7]
40 H 2 N S A[41.7]
41 F 3 N S C[41.8]
42 H 2 B S B[41.7]
43 F 2 N S D[41.7]
- 2. A[41.3] Can answer question by just knowing first part of ATwo valves that connect to the drywell equipment drain 44 F 2 N E sump Knowledge of one connection to the A main steam line is not necessary. Fixed 45 F 2 N S A[41.4]
- 3. A[41.11] Same issue as Q44. Only need to know one part of the 46 F 2 N E question to answer correctly . Recommend 2 x 2 format. Fixed 47 F 2 N S C[41.10] Page 3 of 7 OBDI 202 - IOLE Process
ES-401 GG-2011-12 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 48 H 2 B S B[41.5]
- 4. D[41.10] Editorial Comment: The technical reference is section 49 F 2 B E 3.4. Section 3.8 does not exist. Question revised and resubmitted. Fixed.
D[41.10] Backward logic. See examples of this in NUREG-1021, Appendix B, 50 F 2 X B U Attachment 2, Section G. New question submitted. Fixed.
51 F 3 N S B[41.10]
52 H 3 N S D[41.7]
C[41.14] Editorial Comment: Remove Then, from second sentence. Bulleting the 53 H 2 N E first two sentences as initial conditions would be a better format. Fixed 54 F 2 N S B[41.10]
55 F 2 N S B[41.10]
56 F 2 B S C[41.10]
57 H 4 N S HANDOUT A[41.10]
58 H 3 B S HANDOUT B[41.10]
59 F 2 B S 2009 PRIOR NRC B[41.6]
60 F 2 N S A[41.10]
61 H 2 N S D[41.7]
62 H 2 X N S C[41.10]
63 H 2 B S B[41.7]
64 H 2 N S D[41.7]
65 H 4 N S B[41.7]
66 H 2 N S A[41.7]
67 H 2 N S D[41.7]
68 H 2 B S C[41.7]
69 F 2 B S A[41.4]
70 H 2 B S C[41.7] Page 4 of 7 OBDI 202 - IOLE Process
ES-401 GG-2011-12 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 71 H 2 N S B[41.5]
72 H 3 B S B[41.8]
73 F 2 B S C[41.10]
74 H 2 N S D[41.10]
75 F 2 N S A[41.10]
D[43.2] Backward logic. See examples of this in NUREG-1021, Appendix B, 76 F 3 X N U Attachment 2, Section G. New question submitted. Fixed.
77 H 3 B S HANDOUT C[43.5]
78 H 2 N S HANDOUT C[43.5]
A[41.10] Asking for entry condition to EP-2 or EP-2A. Look at Q90. The stem shows 79 H 1 X X N U the crew is in EP-2A at 5% power. This question starts at 22%. Pretty obvious. The second part of the answers is RO knowledge. New question submitted. Fixed.
80 H 2 B S HANDOUT C[43.4]
81 H 2 N S HANDOUT B[43.5]
82 H 3 N S B[43.5]
83 H 2 N S C[43.5]
84 F 2 N S D[43.2]
85 H 3 N S D[43.5]
86 F 2 N S C[43.5]
87 H 3 N S HANDOUT A[43.2]
2009 LAST NRC EXAM B[43.5] This question requires knowledge of the same procedure step that is being used as the basis for question 4 on the RO exam. The 88 H 2 X X B U stem for Q4 gives away the answer or at least eliminates two distractors for this question. New question submitted. Fixed.
89 F 3 N S D[43.2]
90 H 3 N S C[43.5]
91 H 3 N S A[43.2] Page 5 of 7 OBDI 202 - IOLE Process
ES-401 GG-2011-12 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only HANDOUT B[43.5] We do not have a regulatory basis for making a licensing 92 H 2 X N U decision based upon beyond-design-basis procedures. This need to be changed to EOPs. New question submitted. Fixed.
93 H 3 N S A[43.2]
C[43.6] Editorial Comment: Add are to distracter D between there and no for 94 H 2 B S readability and to ensure no distraction. Fixed 95 F 2 N S D[43.5]
96 F 3 X N E A[43.5] C is not credible. Fixed 97 F 2 N S B[41.10]
HANDOUT B[41.10]This seems like a direct look-up. Additionally, it is not asking 98 H 2 X X N U about the process for determining risk. New question submitted. Fixed.
B[43.5] We do not have a regulatory basis for making a licensing decision based 99 H 2 X N U upon beyond-design-basis procedures. This need to be changed to EOPs. New question submitted. Fixed.
D[41.10] The first part of this question is RO knowledge because it is asking about a note in the procedure. The second part is RO too because it is a step in the 100 F 2 X N U attachment that is normally given to an operator to install. The emergency director could be one of the SRO crew members prior to TSC/EOF activation?? Fixed B= 25 F= 32 / 42% E= 13 RO TOTALS: M= 0 H= 43 / 57% U= 3 Additional Notes:
N= 50 B= 3 F= 10 / 40% E= 1 SRO TOTALS: M= 0 H= 15 / 60% U= 7/ 28% Additional Notes:
N= 22
GENERAL COMMENT
S: Review performed by Chris Steely
- 1. Bank questions are indicated by B
- 2. Chief Examiner comments are indicated in blue.
- 3. Average difficulty is __ 2.24___ on the RO exam and 2.36___ on the SRO exam. Page 6 of 7 OBDI 202 - IOLE Process
ES-401 GG-2011-12 7 Form ES-401-9
41.2 = 1 43.2 = 5 41.3 = 3 43.3 =
41.4 = 6 43.4 = 1 41.5 = 2 43.5 = 11 41.6 = 1 43.6 = 1 41.7 = 25 43.7 =
41.8 = 6 41.9 =
41.10 = 25 41.11 = 1 41.12 = 1 41.13 =
41.14 = 1
- 6. There are 11 questions with attachments provided. Page 7 of 7 OBDI 202 - IOLE Process