ML120310427: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
 
Line 16: Line 16:


=Text=
=Text=
{{#Wiki_filter:ES-401 GG-2011-12 Written Examination Review Worksheet Form ES-401-9 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:
{{#Wiki_filter:ES-401               GG-2011-12                                             Written Examination Review Worksheet                                                                           Form ES-401-9
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
: 1.      2.           3. Psychometric Flaws              4. Job Content Flaws        5. Other      6.      7.                                              8.
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
Q#    LOK     LOD (F/H)   (1-5)   Stem Cues T/F        Cred. Partial Job- Minutia #/ Back-            Q= SRO B/M/N U/E/S                                                Explanation Focus                Dist.           Link            units ward    K/A Only C[41.8 ] Answer B is also correct because it too connects electrical power to the E12-1      F      3                                                                                         N      E    F008, only it is not direct. Reword the stem to The supply breaker that directly connects electrical power to     Fixed 2      H      2                                                                                          B      S    D[41.7]
$ The answer choices are a collection of unrelated true/false statements.  
3      H      4                                                                                          B      S    A[41.7]
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.  
A [41.10] Recommend adding While in the Restricted Region, to the stem. Also, 4      F      2                                                                                          B      E    increase 2% on distracter B to 8-9%, closer to set point of 10% for increased plausibility. Fixed 5      F      2                                                                                          N      S    B[41.10]
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:
6      H      2                                                                                          N      S    A[41.3]
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).  
7      H      3                                                                                          B      S    C[41.7]
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
8      H      4                                                                                          N      S    B[41.4]
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).  
Instructions
$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
Attachment 8 Page 1 of 7 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
: 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
: 3. Check the appropriate box if a psychometric flaw is identified:
Dist. Partial Job-LinkMinutia#/
      $   The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
units Back-wardQ= K/A SROOnly1 F 3            N E C[41.8 ] Answer B is also correct because it too connects electrical power to the E12-F008, only it is not direct. Reword the stem to "The supply breaker that directly connects electrical power to-"      Fixed 2 H 2            B S D[41.7] 3 H 4            B S A[41.7] 4 F 2 B E A [41.10] Recommend adding "While in the Restricted Region, -" to the stem. Also, increase 2% on distracter 'B' to 8-9%, closer to set point of 10% for increased plausibility. Fixed 5 F 2            N S B[41.10] 6 H 2            N S A[41.3] 7 H 3            B S C[41.7] 8 H 4            N S B[41.4]
      $    The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
ES-401 GG-2011-12 2 Form ES-401-9 Attachment 8 Page 2 of 7 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
      $    The answer choices are a collection of unrelated true/false statements.
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
      $    The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
Dist. Partial Job-LinkMinutia#/
      $    One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
units Back-wardQ= K/A SROOnly9 H 2           B S D[41.4] 10 H 2           N S HANDOUT C[41.10] 11 H 2           B S A[41.7]   12 H 3 B E 1. B[41.5] Isn't going above 35% with BPVs open violating procedure? Reference Q52. If so, change reactor power from 45% to something less than 35% but still above the LPSP. Fixed 13 F 2           N S C[41.7] 14 F 2            N E B[41.8] Recommend changing distracters to 6, 8, and 12. Reasoning being that 2 seems too short and 24 seems too long. Fixed 15 H 3           N S A[41.10] 16 H 2   X       N U C[41.10] A and D not plausible. SLC injection rate not level dependent. Fixed 17 H 2 B E A[41.7] Distracter 'D' should just say "Closed" - This will balance out correct answer 'A' in length and enhance the distracters psychometrically. Fixed 18 H 2            N E C[41.7] Why now in the question stem. It's already in the setup to the question - now at -50 inches. Fixed 19 H 2 B E D[41.7] Correct answer 'D' can be answered by only knowing the second part dealing with Alternate Feed. It's the only one with the ON first OFF second variation. Fixed 20 F 2           N S C[41.4]
: 4. Check the appropriate box if a job content error is identified:
21 F 3           N S B[41.10] 22 H 2            N E A[41.7] Based on the training material, the given power between 10^2 and 10^5 is below the threshold of exceeding 10^5, therefore, a rod block does not exist. The stem states the Upscale Trip unit is failed; therefore, only the RPS function is affected, and this is bypassed by the shorting links. D seems to be the correct answer. Fixed 23 H 2           B S B[41.4] 24 H 3           B S C[41.7] 25 F 2           N S D[41.7] 26 H 2           N S B[41.8] 27 F 2           N S A[41.10]
      $ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
ES-401 GG-2011-12 3 Form ES-401-9 Attachment 8 Page 3 of 7 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
      $ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
      $ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
Dist. Partial Job-LinkMinutia#/
      $ The question requires reverse logic or application compared to the job requirements.
units Back-wardQ= K/A SROOnly28 H 2           N S A[41.2] 29 H 2           N S D[41.10] 30 H 3           N S D[41.3] 31 F 2           B S C[41.10] 32 F 2           B S B[41.10] 33 F 2           N S B[41.12]   34 F 3           N S C[41.10] 35 H 2           B S B[41.7] 36 H 2 B S HANDOUT C[41.10] 37 H 1 X X          N U A[41.7] The question is a giveaway. It gives the applicant a controller and its setpoint, then defines what is a higher d/p and finally asks the applicant which of two controller setpoints (one above and one below the current setpoint) would provide a higher output. Fixed 38 F 2            N E D[41.8] The GGNS simulator is not a good technical reference. The simulator only does what it's told. Somehow the shutoff head of the pumps may have something to do with flow rates and pressure or there is some validation from the plant, etc. Fixed 39 F 2           N S B[41.7] 40 H 2           N S A[41.7] 41 F 3           N S C[41.8] 42 H 2           B S B[41.7] 43 F 2           N S D[41.7] 44 F 2            N E
: 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
: 2. A[41.3] Can answer question by just knowing first part of 'A'-"Two valves that connect to the drywell equipment drain sump-"  Knowledge of one connection to the 'A' main steam line is not necessary. Fixed 45 F 2           N S A[41.4] 46 F 2            N E
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
: 3. A[41.11] Same issue as Q44. Only need to know one part of the question to answer correctly . Recommend 2 x 2 format. Fixed 47 F 2           N S C[41.10]
: 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
ES-401 GG-2011-12 4 Form ES-401-9 Attachment 8 Page 4 of 7 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
: 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).                                                                                       Page 1 of 7                                                         OBDI 202 - IOLE Process
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
 
Dist. Partial Job-LinkMinutia#/
ES-401               GG-2011-12                                                     2                                                                             Form ES-401-9
units Back-wardQ= K/A SROOnly48 H 2           B S B[41.5] 49 F 2 B E 4. D[41.10] Editorial Comment: The technical reference is section 3.4. Section 3.8 does not exist.
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                             8.
Question revised and resubmitted. Fixed. 50 F 2        X B U D[41.10] Backward logic. See examples of this in NUREG-1021, Appendix B, Attachment 2, Section G. New question submitted. Fixed. 51 F 3           N S B[41.10] 52 H 3           N S D[41.7] 53 H 2            N E C[41.14] Editorial Comment: Remove "Then, " from second sentence. Bulleting the first two sentences as initial conditions would be a better format. Fixed 54 F 2           N S B[41.10] 55 F 2           N S B[41.10] 56 F 2           B S C[41.10] 57 H 4           N S HANDOUT A[41.10] 58 H 3           B S HANDOUT B[41.10] 59 F 2 B S 2009 PRIOR NRC B[41.6] 60 F 2           N S A[41.10] 61 H 2           N S D[41.7] 62 H 2     X       N S C[41.10]   63 H 2           B S B[41.7] 64 H 2           N S D[41.7] 65 H 4           N S B[41.7] 66 H 2           N S A[41.7] 67 H 2           N S D[41.7] 68 H 2           B  S C[41.7] 69 F 2           B S A[41.4] 70 H 2           B S C[41.7]
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                          Explanation Focus              Dist.        Link          units ward K/A Only 9    H     2                                                                         B     S D[41.4]
ES-401 GG-2011-12 5 Form ES-401-9 Attachment 8 Page 5 of 7 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
10   H     2                                                                         N     S HANDOUT C[41.10]
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
11   H     2                                                                         B     S A[41.7]
Dist. Partial Job-LinkMinutia#/
: 1. B[41.5] Isnt going above 35% with BPVs open violating 12    H    3                                                                        B    E                        procedure? Reference Q52. If so, change reactor power from 45% to something less than 35% but still above the LPSP. Fixed 13   F     2                                                                         N     S C[41.7]
units Back-wardQ= K/A SROOnly71 H 2           N S B[41.5] 72 H 3           B S B[41.8] 73 F 2           B S C[41.10] 74 H 2           N S D[41.10] 75 F 2           N S A[41.10] 76 F 3        X  N U D[43.2] Backward logic. See examples of this in NUREG-1021, Appendix B, Attachment 2, Section G. New question submitted. Fixed. 77 H 3 B S HANDOUT C[43.5] 78 H 2           N S HANDOUT C[43.5] 79 H 1  X        X N U A[41.10] Asking for entry condition to EP-2 or EP-2A. Look at Q90. The stem shows the crew is in EP-2A at 5% power. This question starts at 22%. Pretty obvious. The second part of the answers is RO knowledge. New question submitted. Fixed. 80 H 2 B S HANDOUT C[43.4] 81 H 2           N S HANDOUT B[43.5] 82 H 3           N S B[43.5] 83 H 2           N S C[43.5] 84 F 2           N S D[43.2] 85 H 3           N S D[43.5] 86 F 2           N S C[43.5] 87 H 3           N S HANDOUT A[43.2] 88 H 2 X X B U 2009 LAST NRC EXAM B[43.5] This question requires knowledge of the same procedure step that is being used as the basis for question 4 on the RO exam. The stem for Q4 gives away the answer or at least eliminates two distractors for this question. New question submitted. Fixed. 89 F 3           N S D[43.2]   90 H 3           N S C[43.5] 91 H 3           N S A[43.2]
B[41.8] Recommend changing distracters to 6, 8, and 12. Reasoning being that 2 14    F    2                                                                        N    E seems too short and 24 seems too long. Fixed 15   H     3                                                                         N     S A[41.10]
ES-401 GG-2011-12 6 Form ES-401-9 Attachment 8 Page 6 of 7 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
16   H     2                       X                                               N     U C[41.10] A and D not plausible. SLC injection rate not level dependent. Fixed A[41.7] Distracter D should just say Closed - This will balance out correct answer 17    H    2                                                                        B    E A in length and enhance the distracters psychometrically. Fixed C[41.7] Why now in the question stem. Its already in the setup to the question -
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
18    H    2                                                                        N    E now at -50 inches. Fixed D[41.7] Correct answer D can be answered by only knowing the second part dealing 19    H    2                                                                        B    E with Alternate Feed. Its the only one with the ON first OFF second variation. Fixed 20   F     2                                                                         N     S C[41.4]
Dist. Partial Job-LinkMinutia#/
21   F     3                                                                         N     S B[41.10]
units Back-wardQ= K/A SROOnly92 H 2      X      N U HANDOUT B[43.5] We do not have a regulatory basis for making a licensing decision based upon beyond-design-basis procedures. This need to be changed to EOPs. New question submitted. Fixed. 93 H 3           N S A[43.2] 94 H 2 B S C[43.6] Editorial Comment: Add "are" to distracter 'D' between "there" and "no" for readability and to ensure no distraction. Fixed 95 F 2           N S D[43.5] 96 F 3   X       N E A[43.5] C is not credible. Fixed 97 F 2           N S B[41.10] 98 H 2          X X N U HANDOUT B[41.10]This seems like a direct look-up. Additionally, it is not asking about the process for determining risk. New question submitted. Fixed. 99 H 2      X      N U B[43.5] We do not have a regulatory basis for making a licensing decision based upon beyond-design-basis procedures. This need to be changed to EOPs. New question submitted. Fixed. 100 F 2          X N U D[41.10] The first part of this question is RO knowledge because it is asking about a note in the procedure. The second part is RO too because it is a step in the attachment that is normally given to an operator to install. The emergency director could be one of the SRO crew members prior to TSC/EOF activation?? Fixed RO TOTALS: B= 25 F=32 / 42% E=13 Additional Notes:   M= 0 H=43 / 57% U=3 N= 50  SRO TOTALS: B= 3 F=10 / 40% E=1 Additional Notes: M= 0 H=15 / 60% U=7/ 28% N= 22  
A[41.7] Based on the training material, the given power between 10^2 and 10^5 is below the threshold of exceeding 10^5, therefore, a rod block does not exist. The 22    H    2                                                                        N    E  stem states the Upscale Trip unit is failed; therefore, only the RPS function is affected, and this is bypassed by the shorting links. D seems to be the correct answer. Fixed 23   H     2                                                                         B     S B[41.4]
24   H     3                                                                         B     S C[41.7]
25   F     2                                                                         N     S D[41.7]
26   H     2                                                                         N     S B[41.8]
27   F     2                                                                         N     S A[41.10]                                                                     Page 2 of 7                                                  OBDI 202 - IOLE Process
 
ES-401               GG-2011-12                                                     3                                                                           Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                             8.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                          Explanation Focus              Dist.        Link          units ward K/A Only 28    H     2                                                                         N     S A[41.2]
29   H     2                                                                         N     S D[41.10]
30   H     3                                                                         N     S D[41.3]
31   F     2                                                                         B     S C[41.10]
32   F     2                                                                         B     S B[41.10]
33   F     2                                                                         N     S B[41.12]
34   F     3                                                                         N     S C[41.10]
35   H     2                                                                         B     S B[41.7]
36   H     2                                                                         B     S HANDOUT C[41.10]
A[41.7] The question is a giveaway. It gives the applicant a controller and its setpoint, then defines what is a higher d/p and finally asks the applicant which of two 37    H    1    X      X                                                            N    U controller setpoints (one above and one below the current setpoint) would provide a higher output. Fixed D[41.8] The GGNS simulator is not a good technical reference. The simulator only 38    F    2                                                                        N    E  does what its told. Somehow the shutoff head of the pumps may have something to do with flow rates and pressure or there is some validation from the plant, etc. Fixed 39   F     2                                                                         N     S B[41.7]
40   H     2                                                                         N     S A[41.7]
41   F     3                                                                         N     S C[41.8]
42   H     2                                                                         B     S B[41.7]
43   F     2                                                                         N     S D[41.7]
: 2. A[41.3] Can answer question by just knowing first part of ATwo valves that connect to the drywell equipment drain 44    F    2                                                                        N    E sump Knowledge of one connection to the A main steam line is not necessary. Fixed 45   F     2                                                                         N     S A[41.4]
: 3. A[41.11] Same issue as Q44. Only need to know one part of the 46    F    2                                                                        N    E question to answer correctly . Recommend 2 x 2 format. Fixed 47   F     2                                                                         N     S C[41.10]                                                                     Page 3 of 7                                                  OBDI 202 - IOLE Process
 
ES-401               GG-2011-12                                                     4                                                                             Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                             8.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                          Explanation Focus              Dist.        Link          units ward K/A Only 48    H     2                                                                         B     S B[41.5]
: 4. D[41.10] Editorial Comment: The technical reference is section 49    F    2                                                                        B    E                      3.4. Section 3.8 does not exist. Question revised and resubmitted. Fixed.
D[41.10] Backward logic. See examples of this in NUREG-1021, Appendix B, 50    F    2                                                          X            B    U Attachment 2, Section G. New question submitted. Fixed.
51   F     3                                                                         N     S B[41.10]
52   H     3                                                                         N     S D[41.7]
C[41.14] Editorial Comment: Remove Then, from second sentence. Bulleting the 53    H    2                                                                        N    E first two sentences as initial conditions would be a better format. Fixed 54   F     2                                                                         N     S B[41.10]
55   F     2                                                                         N     S B[41.10]
56   F     2                                                                         B     S C[41.10]
57   H     4                                                                         N     S HANDOUT A[41.10]
58   H     3                                                                         B     S HANDOUT B[41.10]
59   F     2                                                                         B     S 2009 PRIOR NRC B[41.6]
60   F     2                                                                         N     S A[41.10]
61   H     2                                                                         N     S D[41.7]
62   H     2                             X                                         N     S C[41.10]
63   H     2                                                                         B     S B[41.7]
64   H     2                                                                         N     S D[41.7]
65   H     4                                                                         N     S B[41.7]
66   H     2                                                                         N     S A[41.7]
67   H     2                                                                         N     S D[41.7]
68   H     2                                                                         B     S C[41.7]
69   F     2                                                                         B     S A[41.4]
70   H     2                                                                         B     S C[41.7]                                                                     Page 4 of 7                                                  OBDI 202 - IOLE Process
 
ES-401               GG-2011-12                                                     5                                                                           Form ES-401-9
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                           8.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                        Explanation Focus              Dist.        Link          units ward K/A Only 71    H     2                                                                         N     S B[41.5]
72   H     3                                                                         B     S B[41.8]
73   F     2                                                                         B     S C[41.10]
74   H     2                                                                         N     S D[41.10]
75   F     2                                                                         N     S A[41.10]
D[43.2] Backward logic. See examples of this in NUREG-1021, Appendix B, 76    F    3                                                          X            N    U Attachment 2, Section G. New question submitted. Fixed.
77   H     3                                                                         B     S HANDOUT C[43.5]
78   H     2                                                                         N     S HANDOUT C[43.5]
A[41.10] Asking for entry condition to EP-2 or EP-2A. Look at Q90. The stem shows 79    H    1            X                                                      X    N    U  the crew is in EP-2A at 5% power. This question starts at 22%. Pretty obvious. The second part of the answers is RO knowledge. New question submitted. Fixed.
80   H     2                                                                         B     S HANDOUT C[43.4]
81   H     2                                                                         N     S HANDOUT B[43.5]
82   H     3                                                                         N     S B[43.5]
83   H     2                                                                         N     S C[43.5]
84   F     2                                                                         N     S D[43.2]
85   H     3                                                                         N     S D[43.5]
86   F     2                                                                         N     S C[43.5]
87   H     3                                                                         N     S HANDOUT A[43.2]
2009 LAST NRC EXAM B[43.5] This question requires knowledge of the same procedure step that is being used as the basis for question 4 on the RO exam. The 88    H    2    X      X                                                            B    U stem for Q4 gives away the answer or at least eliminates two distractors for this question. New question submitted. Fixed.
89   F     3                                                                         N     S D[43.2]
90   H     3                                                                         N     S C[43.5]
91   H     3                                                                         N     S A[43.2]                                                                     Page 5 of 7                                                OBDI 202 - IOLE Process
 
ES-401                     GG-2011-12                                                       6                                                                           Form ES-401-9
: 1. 2.         3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.                                             8.
Q#   LOK  LOD (F/H)  (1-5)  Stem Cues T/F      Cred. Partial Job- Minutia #/ Back-       Q= SRO B/M/N U/E/S                                          Explanation Focus                Dist.        Link          units ward    K/A Only HANDOUT B[43.5] We do not have a regulatory basis for making a licensing 92    H      2                                        X                                      N    U  decision based upon beyond-design-basis procedures. This need to be changed to EOPs. New question submitted. Fixed.
93   H     3                                                                               N     S A[43.2]
C[43.6] Editorial Comment: Add are to distracter D between there and no for 94    H      2                                                                              B    S readability and to ensure no distraction. Fixed 95   F     2                                                                               N     S D[43.5]
96   F     3                           X                                                   N     E A[43.5] C is not credible. Fixed 97   F     2                                                                               N     S B[41.10]
HANDOUT B[41.10]This seems like a direct look-up. Additionally, it is not asking 98    H      2                                                                    X    X    N    U about the process for determining risk. New question submitted. Fixed.
B[43.5] We do not have a regulatory basis for making a licensing decision based 99    H      2                                        X                                      N    U  upon beyond-design-basis procedures. This need to be changed to EOPs. New question submitted. Fixed.
D[41.10] The first part of this question is RO knowledge because it is asking about a note in the procedure. The second part is RO too because it is a step in the 100    F      2                                                                          X    N    U attachment that is normally given to an operator to install. The emergency director could be one of the SRO crew members prior to TSC/EOF activation?? Fixed B= 25               F= 32 / 42%           E= 13 RO TOTALS:                M= 0                H= 43 / 57%          U= 3                    Additional Notes:
N= 50 B= 3                F= 10 / 40%          E= 1 SRO TOTALS:                M= 0                 H= 15 / 60%           U= 7/ 28%                Additional Notes:
N= 22


==GENERAL COMMENT==
==GENERAL COMMENT==
S: Review performed by Chris Steely
S: Review performed by Chris Steely
: 1. Bank questions are indicated by B
: 1. Bank questions are indicated by B
: 2. Chief Examiner comments are indicated in blue. 3. Average difficulty is __   2.24___   on the RO exam and     2.36___
: 2. Chief Examiner comments are indicated in blue.
on the SRO exam.
: 3. Average difficulty is __ 2.24___ on the RO exam and           2.36___     on the SRO exam.                                                                             Page 6 of 7                                                  OBDI 202 - IOLE Process
ES-401 GG-2011-12 7 Form ES-401-9 Attachment 8 Page 7 of 7 OBDI 202 - IOLE Process
 
: 4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 =   43.1 =   41.2 = 1 43.2 = 5 41.3 = 3 43.3 =   41.4 = 6 43.4 = 1 41.5 = 2 43.5 = 11 41.6 = 1 43.6 = 1 41.7 = 25 43.7 =   41.8 = 6 41.9 =
ES-401                 GG-2011-12                                 7                     Form ES-401-9
41.10 = 25 41.11 = 1 41.12 = 1 41.13 =   41.14 = 1
: 4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 =                 43.1 =
: 5. The answer distribution is: RO / SRO A = 19 / 5 B = 22 / 7 C = 20 / 7 D = 14 / 6
41.2 = 1               43.2 = 5 41.3 = 3               43.3 =
: 6. There are   11    questions with attachments provided.}}
41.4 = 6               43.4 = 1 41.5 = 2               43.5 = 11 41.6 = 1               43.6 = 1 41.7 = 25             43.7 =
41.8 = 6 41.9 =
41.10 = 25 41.11 = 1 41.12 = 1 41.13 =
41.14 = 1
: 5.       The answer distribution is: RO / SRO A = 19 / 5 B = 22 / 7 C = 20 / 7 D = 14 / 6
: 6.       There are 11    questions with attachments provided.                                                   Page 7 of 7 OBDI 202 - IOLE Process}}

Latest revision as of 09:03, 12 November 2019

12 Written Exam Comments
ML120310427
Person / Time
Site: Grand Gulf Entergy icon.png
Issue date: 12/02/2011
From:
NRC Region 4
To:
Entergy Operations
References
50-416/11-301
Download: ML120310427 (7)
v • d • e


Text

ES-401 GG-2011-12 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only C[41.8 ] Answer B is also correct because it too connects electrical power to the E12-1 F 3 N E F008, only it is not direct. Reword the stem to The supply breaker that directly connects electrical power to Fixed 2 H 2 B S D[41.7]

3 H 4 B S A[41.7]

A [41.10] Recommend adding While in the Restricted Region, to the stem. Also, 4 F 2 B E increase 2% on distracter B to 8-9%, closer to set point of 10% for increased plausibility. Fixed 5 F 2 N S B[41.10]

6 H 2 N S A[41.3]

7 H 3 B S C[41.7]

8 H 4 N S B[41.4]

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process

ES-401 GG-2011-12 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 9 H 2 B S D[41.4]

10 H 2 N S HANDOUT C[41.10]

11 H 2 B S A[41.7]

1. B[41.5] Isnt going above 35% with BPVs open violating 12 H 3 B E procedure? Reference Q52. If so, change reactor power from 45% to something less than 35% but still above the LPSP. Fixed 13 F 2 N S C[41.7]

B[41.8] Recommend changing distracters to 6, 8, and 12. Reasoning being that 2 14 F 2 N E seems too short and 24 seems too long. Fixed 15 H 3 N S A[41.10]

16 H 2 X N U C[41.10] A and D not plausible. SLC injection rate not level dependent. Fixed A[41.7] Distracter D should just say Closed - This will balance out correct answer 17 H 2 B E A in length and enhance the distracters psychometrically. Fixed C[41.7] Why now in the question stem. Its already in the setup to the question -

18 H 2 N E now at -50 inches. Fixed D[41.7] Correct answer D can be answered by only knowing the second part dealing 19 H 2 B E with Alternate Feed. Its the only one with the ON first OFF second variation. Fixed 20 F 2 N S C[41.4]

21 F 3 N S B[41.10]

A[41.7] Based on the training material, the given power between 10^2 and 10^5 is below the threshold of exceeding 10^5, therefore, a rod block does not exist. The 22 H 2 N E stem states the Upscale Trip unit is failed; therefore, only the RPS function is affected, and this is bypassed by the shorting links. D seems to be the correct answer. Fixed 23 H 2 B S B[41.4]

24 H 3 B S C[41.7]

25 F 2 N S D[41.7]

26 H 2 N S B[41.8]

27 F 2 N S A[41.10] Page 2 of 7 OBDI 202 - IOLE Process

ES-401 GG-2011-12 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 28 H 2 N S A[41.2]

29 H 2 N S D[41.10]

30 H 3 N S D[41.3]

31 F 2 B S C[41.10]

32 F 2 B S B[41.10]

33 F 2 N S B[41.12]

34 F 3 N S C[41.10]

35 H 2 B S B[41.7]

36 H 2 B S HANDOUT C[41.10]

A[41.7] The question is a giveaway. It gives the applicant a controller and its setpoint, then defines what is a higher d/p and finally asks the applicant which of two 37 H 1 X X N U controller setpoints (one above and one below the current setpoint) would provide a higher output. Fixed D[41.8] The GGNS simulator is not a good technical reference. The simulator only 38 F 2 N E does what its told. Somehow the shutoff head of the pumps may have something to do with flow rates and pressure or there is some validation from the plant, etc. Fixed 39 F 2 N S B[41.7]

40 H 2 N S A[41.7]

41 F 3 N S C[41.8]

42 H 2 B S B[41.7]

43 F 2 N S D[41.7]

2. A[41.3] Can answer question by just knowing first part of ATwo valves that connect to the drywell equipment drain 44 F 2 N E sump Knowledge of one connection to the A main steam line is not necessary. Fixed 45 F 2 N S A[41.4]
3. A[41.11] Same issue as Q44. Only need to know one part of the 46 F 2 N E question to answer correctly . Recommend 2 x 2 format. Fixed 47 F 2 N S C[41.10] Page 3 of 7 OBDI 202 - IOLE Process

ES-401 GG-2011-12 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 48 H 2 B S B[41.5]

4. D[41.10] Editorial Comment: The technical reference is section 49 F 2 B E 3.4. Section 3.8 does not exist. Question revised and resubmitted. Fixed.

D[41.10] Backward logic. See examples of this in NUREG-1021, Appendix B, 50 F 2 X B U Attachment 2, Section G. New question submitted. Fixed.

51 F 3 N S B[41.10]

52 H 3 N S D[41.7]

C[41.14] Editorial Comment: Remove Then, from second sentence. Bulleting the 53 H 2 N E first two sentences as initial conditions would be a better format. Fixed 54 F 2 N S B[41.10]

55 F 2 N S B[41.10]

56 F 2 B S C[41.10]

57 H 4 N S HANDOUT A[41.10]

58 H 3 B S HANDOUT B[41.10]

59 F 2 B S 2009 PRIOR NRC B[41.6]

60 F 2 N S A[41.10]

61 H 2 N S D[41.7]

62 H 2 X N S C[41.10]

63 H 2 B S B[41.7]

64 H 2 N S D[41.7]

65 H 4 N S B[41.7]

66 H 2 N S A[41.7]

67 H 2 N S D[41.7]

68 H 2 B S C[41.7]

69 F 2 B S A[41.4]

70 H 2 B S C[41.7] Page 4 of 7 OBDI 202 - IOLE Process

ES-401 GG-2011-12 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 71 H 2 N S B[41.5]

72 H 3 B S B[41.8]

73 F 2 B S C[41.10]

74 H 2 N S D[41.10]

75 F 2 N S A[41.10]

D[43.2] Backward logic. See examples of this in NUREG-1021, Appendix B, 76 F 3 X N U Attachment 2, Section G. New question submitted. Fixed.

77 H 3 B S HANDOUT C[43.5]

78 H 2 N S HANDOUT C[43.5]

A[41.10] Asking for entry condition to EP-2 or EP-2A. Look at Q90. The stem shows 79 H 1 X X N U the crew is in EP-2A at 5% power. This question starts at 22%. Pretty obvious. The second part of the answers is RO knowledge. New question submitted. Fixed.

80 H 2 B S HANDOUT C[43.4]

81 H 2 N S HANDOUT B[43.5]

82 H 3 N S B[43.5]

83 H 2 N S C[43.5]

84 F 2 N S D[43.2]

85 H 3 N S D[43.5]

86 F 2 N S C[43.5]

87 H 3 N S HANDOUT A[43.2]

2009 LAST NRC EXAM B[43.5] This question requires knowledge of the same procedure step that is being used as the basis for question 4 on the RO exam. The 88 H 2 X X B U stem for Q4 gives away the answer or at least eliminates two distractors for this question. New question submitted. Fixed.

89 F 3 N S D[43.2]

90 H 3 N S C[43.5]

91 H 3 N S A[43.2] Page 5 of 7 OBDI 202 - IOLE Process

ES-401 GG-2011-12 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only HANDOUT B[43.5] We do not have a regulatory basis for making a licensing 92 H 2 X N U decision based upon beyond-design-basis procedures. This need to be changed to EOPs. New question submitted. Fixed.

93 H 3 N S A[43.2]

C[43.6] Editorial Comment: Add are to distracter D between there and no for 94 H 2 B S readability and to ensure no distraction. Fixed 95 F 2 N S D[43.5]

96 F 3 X N E A[43.5] C is not credible. Fixed 97 F 2 N S B[41.10]

HANDOUT B[41.10]This seems like a direct look-up. Additionally, it is not asking 98 H 2 X X N U about the process for determining risk. New question submitted. Fixed.

B[43.5] We do not have a regulatory basis for making a licensing decision based 99 H 2 X N U upon beyond-design-basis procedures. This need to be changed to EOPs. New question submitted. Fixed.

D[41.10] The first part of this question is RO knowledge because it is asking about a note in the procedure. The second part is RO too because it is a step in the 100 F 2 X N U attachment that is normally given to an operator to install. The emergency director could be one of the SRO crew members prior to TSC/EOF activation?? Fixed B= 25 F= 32 / 42% E= 13 RO TOTALS: M= 0 H= 43 / 57% U= 3 Additional Notes:

N= 50 B= 3 F= 10 / 40% E= 1 SRO TOTALS: M= 0 H= 15 / 60% U= 7/ 28% Additional Notes:

N= 22

GENERAL COMMENT

S: Review performed by Chris Steely

1. Bank questions are indicated by B
2. Chief Examiner comments are indicated in blue.
3. Average difficulty is __ 2.24___ on the RO exam and 2.36___ on the SRO exam. Page 6 of 7 OBDI 202 - IOLE Process

ES-401 GG-2011-12 7 Form ES-401-9

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 43.1 =

41.2 = 1 43.2 = 5 41.3 = 3 43.3 =

41.4 = 6 43.4 = 1 41.5 = 2 43.5 = 11 41.6 = 1 43.6 = 1 41.7 = 25 43.7 =

41.8 = 6 41.9 =

41.10 = 25 41.11 = 1 41.12 = 1 41.13 =

41.14 = 1

5. The answer distribution is: RO / SRO A = 19 / 5 B = 22 / 7 C = 20 / 7 D = 14 / 6
6. There are 11 questions with attachments provided. Page 7 of 7 OBDI 202 - IOLE Process
Retrieved from "https://kanterella.com/w/index.php?title=ML120310427&oldid=2106952"