ML102450367: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
 
(One intermediate revision by the same user not shown)
Line 3: Line 3:
| issue date = 04/12/2010
| issue date = 04/12/2010
| title = RB-2010-Draft Written Exam Review
| title = RB-2010-Draft Written Exam Review
| author name = Garchow S M
| author name = Garchow S
| author affiliation = NRC/RGN-IV/DRS/OB
| author affiliation = NRC/RGN-IV/DRS/OB
| addressee name =  
| addressee name =  
Line 16: Line 16:


=Text=
=Text=
{{#Wiki_filter:ES-401 RB-2010-04 Written Examination Review Worksheet Form ES-401-9 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:
{{#Wiki_filter:ES-401               RB-2010-04                                             Written Examination Review Worksheet                                                                       Form ES-401-9
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).  
: 1.      2.           3. Psychometric Flaws              4. Job Content Flaws        5. Other      6.      7.                                                8.
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).  
Q#    LOK     LOD (F/H)   (1-5)   Stem Cues T/F        Cred. Partial Job- Minutia #/ Back-            Q= SRO B/M/N U/E/S                                                Explanation Focus                Dist.           Link            units ward    K/A Only 1     H      2                                                                                         N      S    C, CR, 41.10 2      F      2                                                                                          N      S    A, CR, 41.7 3      F      3                                                                                         N      S    A, CR, 41.7 B, CR, 41.6 - Change trips to trip in distactors.
$ The answer choices are a collection of unrelated true/false statements.
E 4      H      2                                                                                          M            This change will be made.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.  
S The text editing was made as recommended.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:
5      F      3                                                                                          N      S    D, CR, 41.7 6      F      3                                                                                          N      S    C, CR, 41.7 D, CR, 41.10 - This is a KA mismatch due to being focused on TBCW rather than U
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).  
7      F      3                                                                            U            B            CCW.
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).  
S This question will be rewritten.
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).  
Instructions
$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
Attachment 8 Page 1 of 7 OBDI 202 - IOLE Process
: 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
: 3. Check the appropriate box if a psychometric flaw is identified:
Dist. Partial Job-LinkMinutia#/
      $    The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 1 H 2           N S C, CR, 41.10 2 F 2            N S A, CR, 41.7 3 F 3            N S A, CR, 41.7 4 H 2            M E S B, CR, 41.6 - Change "trips" to "trip" in distactors. This change will be made.
      $   The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The text editing was made as recommended. 5 F 3            N S D, CR, 41.7 6 F 3            N S C, CR, 41.7 7 F 3          U B U S D, CR, 41.10 - This is a KA mismatch due to being focused on TBCW rather than CCW. This question will be rewritten.
      $   The answer choices are a collection of unrelated true/false statements.
ES-401 RB-2010-04 2 Form ES-401-9 Attachment 8 Page 2 of 7 OBDI 202 - IOLE Process
      $    The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
      $    One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
Dist. Partial Job-LinkMinutia#/
: 4. Check the appropriate box if a job content error is identified:
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation This question was replaced with a new question. 8 H 3            N S B, CR, 41.4 9 F 2            N S A, CR, 41.8 10 F 3           N S C, CR, 41.4 11 H 3           N S A, CR, 41.3 12 H 4            N S B, CR, 41.5 13 F 3           M S A, CR, 41.7 14 F 3           N S D, CR, 41.9 15 H 3           N S C, CR, 41.10 16 H 3            N S B, CR, 41.2 17 H 3           N S D, CR, 41.10 18 F 3           N S B, CR, 41.13 19 F 3           N S C, CR, 41.10 20 H 4            N S A, CR, 41.5 21 H 3           N S B, CR, 41.5 22 H 3           N S A, CR, 41.9 23 H 3           N S B, CR, 41.7 24 H 3    U      N U S A, CR, 41.6 - This question is U because distractor B is also correct. Distractor B will be revised. Distractor B was rewritten. 25 F 3            N S B, CR, 41.7 26 F 3            N S A, CR, 41.10 27 H 3            N S C, CR, 41.11 28 H 3            N S C, CR, 41.7 29 H 3           N S D, CR, 41.7 ES-401 RB-2010-04 3 Form ES-401-9 Attachment 8 Page 3 of 7 OBDI 202 - IOLE Process
      $ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
      $ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
Dist. Partial Job-LinkMinutia#/
      $ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 30 H 3            N S D, CR, 41.8 31 H 3           N S D, CR, 41.7 32 F 3            N S C, CR, 41.10 33 H 3            N S B, CR, 41.6 34 F 3            N S C, CR, 41.10 35 H 3           N S B, CR, 41.2 36 F 3           N S C, CR, 41.6 37 H 3           N S D, CR, 41.6 38 H 3    E      N E S B, CR, 41.6 - Distractor A can also be argued as being correct since it is a subset of the correct answer. Distractor A will be revised. Distractor A was rewritten. 39 H 3           N S C, CR, 41.7 40 H 3            N S C, CR, 41.7 41 H 3            N S D, CR, 41.7 42 H 3            N S B, CR, 41.7 43 H 3           B S B, CR, 41.5 44 H 3           N S A, CR, 41.5 45 H 3           N E S D, CR, 41.7 - Need to discuss this question as it seems annulus pressure would normally be negative and would remain so under the described conditions in the stem. After discussing with the exam author, this question is correct as written. 46 H 3           N S B, CR, 41.7 47 H 3           M S A, CR, 41.7 48 F 3           N S A, CR, 41.7 49 H 3            N S D, CR, 41.7 50 H 3            B S C, CR, 41.7 ES-401 RB-2010-04 4 Form ES-401-9 Attachment 8 Page 4 of 7 OBDI 202 - IOLE Process
      $ The question requires reverse logic or application compared to the job requirements.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
: 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
Dist. Partial Job-LinkMinutia#/
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 51 H 3           N S D, CR, 41.7 52 H 3           N S D, CR, 41.7 53 H 2           B S B, CR, 41.10 54 F 2    E       N E S D, CR, 41.3 - Discuss this question with exam author. This question is ok as written. 55 F 3     E       N E S C, CR, 41.4 - Discuss this question with exam author. The stem will be revised to improve the readability.
: 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
Part of the question in the stem was highlighted to improve the focus of the question. 56 H 3           N S B, CR, 41.4 57 H 3           N S B, CR, 41.3 58 F 2           N S A, CR, 41.7 59 F 3          U  N U S C, CR, 41.13 - This is a KA mismatch because it does not involve fuel handling. Suggest this KA be rejected and a new one selected. This question will be rewritten. This question was replaced with a new question. 60 F 3           N S B, CR, 41.10 61 H 3    E        M E S A, CR, 41.5 - Revise distractor B so it reads 980 MW instead of 750 MW. This edit change will be made.
: 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).                                                                                       Page 1 of 7                                                          OBDI 202 - IOLE Process
This change was made as recommended. 62 H 3           N S C, CR, 41.4 63 H 3           N S A, CR, 41.5 64 F 3           N S D, CR, 41.4 65 H 2           N S D, CR, 41.5 66 F 3           N S D, CR, 41.10 67 F 2           N S C, CR, 41.10 68 F 3           N S D, CR, 41.10 69 F 3           B S C, CR, 41.10 ES-401 RB-2010-04 5 Form ES-401-9 Attachment 8 Page 5 of 7 OBDI 202 - IOLE Process
 
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
ES-401                RB-2010-04                                                    2                                                                          Form ES-401-9
Dist. Partial Job-LinkMinutia#/
: 1. 2.       3. Psychometric Flaws          4. Job Content Flaws  5. Other  6. 7.                                          8.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 70 F 3      E     N E S A, CR, 41.10 - Not sure this is a memory item; check with exam author. Based on feedback from the validating crew, this is a memory item. 71 F 3           N S B, CR, 41.10 72 F 3           N S A, CR, 41.9 73 H 4           N S A, CR, 41.11 74 F 3          U N U S C, CR, 43.5 - This is an SRO Only question in that it tests on the emergency plan notification requirements. This question will be revised. This question was replaced with a new question. 75 F 2           B S C, CR, 41.10 76 H 3           B S D, CR, 43.5 77 H 3           N S C, CR, 43.5 78 H 3           N S C, CR, 43.5 79 F 2           N S B, CR, 43.5 80 H 3           N S A, OR, 43.5 81 H 3 E          N E S A, CR, 43.5 - Add another EOP related distractor other than the correct answer. This change will be made. Distractor "D" was rewritten. 82 F 3           N S D, OR, 43.2 83 F 2           N S A, CR, 43.5 84 H 3           M S B, CR, 43.5 85 F 3           N S D, CR, 43.5 86 H 3           N S B, CR, 43.5 87 H 3           N S C, CR, 43.5 88 H 3           N S A, CR, 43.5 89 F 3           N S A, CR, 43.2 ES-401 RB-2010-04 6 Form ES-401-9 Attachment 8 Page 6 of 7 OBDI 202 - IOLE Process
Q# LOK  LOD (F/H) (1-5) Stem Cues T/F      Cred. Partial Job- Minutia #/ Back-    Q= SRO B/M/N U/E/S                                      Explanation Focus              Dist.        Link          units ward K/A Only This question was replaced with a new question.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
8   H    3                                                                        N    S  B, CR, 41.4 9    F     2                                                                         N    S  A, CR, 41.8 10    F     3                                                                        N    S  C, CR, 41.4 11    H    3                                                                        N    S  A, CR, 41.3 12    H    4                                                                        N    S  B, CR, 41.5 13    F     3                                                                        M    S A, CR, 41.7 14    F     3                                                                         N     S D, CR, 41.9 15    H     3                                                                         N     S C, CR, 41.10 16    H    3                                                                         N     S B, CR, 41.2 17    H    3                                                                         N    S D, CR, 41.10 18    F     3                                                                         N     S B, CR, 41.13 19    F    3                                                                         N     S C, CR, 41.10 20    H     4                                                                        N     S A, CR, 41.5 21    H     3                                                                         N     S B, CR, 41.5 22    H    3                                                                         N     S A, CR, 41.9 23    H    3                                                                         N     S B, CR, 41.7 A, CR, 41.6 - This question is U because distractor B is also correct.
Dist. Partial Job-LinkMinutia#/
U 24    H     3                             U                                          N       Distractor B will be revised.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 90 H 3           N S D, CR, 43.5 91 H 3           N S B, CR, 43.2 92 H 3           N S C, CR, 43.5 93 F 3      E    N E S A, CR, 43.2 - Candidates are not typically expected to have 12 hour TS action statements memorized. The exam author will check with Ops on this. This question was replaced with a new question. 94            E  N E S C, CR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not symptom/system specific. This question was replaced. 95 H 3          E  N E S D, OR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not symptom specific. Recommend deleting the RHR status from the stem and change so it reads "Subsequent to the above conditions, a loss of all shutdown cooling occurs." This change will be made.
S Distractor B was rewritten.
The question was revised as discussed above. 96 F 2           N S A, CR, 43.5 97 H 2    E      N E S D, CR, 43.4 - Discuss this question with exam author. This question is ok as written. 98 F 3           N S B, CR, 43.4 99 H 2          E  N E S B, CR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not symptom specific. This question was replaced with a new question. 100 H 3           N S D, CR, 43.5 B= 2 F=32 (43%) E=7 M= 9 H=43 (57%) U=4 RO TOTALS: N= 64   Additional Notes:   B= 1 F=8 (32%) E=6 SRO TOTALS: M= 1 H=17 (68%) U=0 Additional Notes:
25    F    3                                                                        N    S B, CR, 41.7 26    F    3                                                                         N     S A, CR, 41.10 27    H     3                                                                         N    S  C, CR, 41.11 28    H    3                                                                        N     S C, CR, 41.7 29    H     3                                                                         N     S D, CR, 41.7                                                                      Page 2 of 7                                                OBDI 202 - IOLE Process
ES-401 RB-2010-04 7 Form ES-401-9 Attachment 8 Page 7 of 7 OBDI 202 - IOLE Process N= 23  
 
ES-401                RB-2010-04                                                    3                                                                            Form ES-401-9
: 1. 2.      3. Psychometric Flaws          4. Job Content Flaws  5. Other  6. 7.                                           8.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F      Cred. Partial Job- Minutia #/ Back-    Q= SRO B/M/N U/E/S                                       Explanation Focus              Dist.         Link          units ward K/A Only 30    H     3                                                                         N     S D, CR, 41.8 31    H    3                                                                         N    S  D, CR, 41.7 32    F    3                                                                         N    S  C, CR, 41.10 33    H    3                                                                        N    S  B, CR, 41.6 34    F     3                                                                        N    S  C, CR, 41.10 35    H    3                                                                        N    S  B, CR, 41.2 36    F     3                                                                        N     S C, CR, 41.6 37    H     3                                                                         N     S D, CR, 41.6 B, CR, 41.6 - Distractor A can also be argued as being correct since it is a subset of E  the correct answer.
38    H     3                             E                                          N S Distractor A will be revised.
Distractor A was rewritten.
39    H    3                                                                         N     S C, CR, 41.7 40    H     3                                                                         N     S C, CR, 41.7 41    H     3                                                                         N     S  D, CR, 41.7 42    H    3                                                                        N     S B, CR, 41.7 43    H    3                                                                        B    S  B, CR, 41.5 44    H     3                                                                         N     S A, CR, 41.5 D, CR, 41.7 - Need to discuss this question as it seems annulus pressure would E  normally be negative and would remain so under the described conditions in the 45    H     3                                                                         N        stem.
S After discussing with the exam author, this question is correct as written.
46    H     3                                                                         N     S B, CR, 41.7 47    H     3                                                                         M    S A, CR, 41.7 48    F    3                                                                        N    S  A, CR, 41.7 49    H    3                                                                         N     S D, CR, 41.7 50    H     3                                                                         B    S C, CR, 41.7                                                                     Page 3 of 7                                                 OBDI 202 - IOLE Process
 
ES-401                RB-2010-04                                                     4                                                                           Form ES-401-9
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                            8.
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                       Explanation Focus              Dist.         Link          units ward K/A Only 51    H    3                                                                         N     S D, CR, 41.7 52   H     3                                                                         N     S D, CR, 41.7 53   H     2                                                                         B     S B, CR, 41.10 E D, CR, 41.3 - Discuss this question with exam author.
54    F    2                              E                                         N S This question is ok as written.
C, CR, 41.4 - Discuss this question with exam author.
E 55    F    3                              E                                          N        The stem will be revised to improve the readability.
S Part of the question in the stem was highlighted to improve the focus of the question.
56   H     3                                                                         N     S B, CR, 41.4 57   H     3                                                                         N     S B, CR, 41.3 58   F     2                                                                         N     S A, CR, 41.7 C, CR, 41.13 - This is a KA mismatch because it does not involve fuel handling.
U  Suggest this KA be rejected and a new one selected.
59    F    3                                                              U          N S  This question will be rewritten.
This question was replaced with a new question.
60   F     3                                                                         N     S B, CR, 41.10 A, CR, 41.5 - Revise distractor B so it reads 980 MW instead of 750 MW.
E 61    H    3                        E                                                M        This edit change will be made.
S This change was made as recommended.
62   H     3                                                                         N     S C, CR, 41.4 63   H     3                                                                         N     S A, CR, 41.5 64   F     3                                                                         N     S D, CR, 41.4 65   H     2                                                                         N     S D, CR, 41.5 66   F     3                                                                         N     S D, CR, 41.10 67   F     2                                                                         N     S C, CR, 41.10 68   F     3                                                                         N     S D, CR, 41.10 69   F     3                                                                         B     S C, CR, 41.10                                                                     Page 4 of 7                                                  OBDI 202 - IOLE Process
 
ES-401               RB-2010-04                                                     5                                                                           Form ES-401-9
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                          8.
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                       Explanation Focus              Dist.         Link          units ward K/A Only E A, CR, 41.10 - Not sure this is a memory item; check with exam author.
70    F    3                                            E                          N S  Based on feedback from the validating crew, this is a memory item.
71   F     3                                                                         N     S B, CR, 41.10 72   F     3                                                                         N     S A, CR, 41.9 73   H     4                                                                         N     S A, CR, 41.11 C, CR, 43.5 - This is an SRO Only question in that it tests on the emergency plan U  notification requirements.
74    F    3                                                                    U    N S  This question will be revised.
This question was replaced with a new question.
75   F     2                                                                         B     S C, CR, 41.10 76   H     3                                                                         B     S D, CR, 43.5 77   H     3                                                                         N     S C, CR, 43.5 78   H     3                                                                         N     S C, CR, 43.5 79   F     2                                                                         N     S B, CR, 43.5 80   H     3                                                                         N     S A, OR, 43.5 A, CR, 43.5 - Add another EOP related distractor other than the correct answer.
E 81    H    3    E                                                                  N        This change will be made.
S Distractor D was rewritten.
82   F     3                                                                         N     S D, OR, 43.2 83   F     2                                                                         N     S A, CR, 43.5 84   H     3                                                                         M     S B, CR, 43.5 85   F     3                                                                         N     S D, CR, 43.5 86   H     3                                                                         N     S B, CR, 43.5 87   H     3                                                                         N     S C, CR, 43.5 88   H     3                                                                         N     S A, CR, 43.5 89   F     3                                                                         N     S A, CR, 43.2                                                                     Page 5 of 7                                                OBDI 202 - IOLE Process
 
ES-401                RB-2010-04                                                       6                                                                           Form ES-401-9
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.                                            8.
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-       Q= SRO B/M/N U/E/S                                         Explanation Focus              Dist.         Link          units ward    K/A Only 90   H     3                                                                           N     S D, CR, 43.5 91   H     3                                                                           N     S B, CR, 43.2 92   H     3                                                                           N     S C, CR, 43.5 A, CR, 43.2 - Candidates are not typically expected to have 12 hour TS action E  statements memorized.
93  F      3                                            E                              N S  The exam author will check with Ops on this.
This question was replaced with a new question.
C, CR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not E
94                                                                            E          N        symptom/system specific.
S This question was replaced.
D, OR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not symptom specific. Recommend deleting the RHR status from the stem and change E  so it reads Subsequent to the above conditions, a loss of all shutdown cooling 95  H      3                                                                E          N        occurs.
S This change will be made.
The question was revised as discussed above.
96   F     2                                                                           N     S A, CR, 43.5 E  D, CR, 43.4 - Discuss this question with exam author.
97  H      2                              E                                            N S  This question is ok as written.
98   F     3                                                                           N     S B, CR, 43.4 B, CR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not E
99  H      2                                                                E          N        symptom specific.
S This question was replaced with a new question.
100   H     3                                                                           N     S D, CR, 43.5 B= 2                 F= 32 (43%)           E= 7 RO TOTALS:            M= 9                 H= 43 (57%)           U= 4                     Additional Notes:
N= 64 SRO TOTALS:            B= 1                F= 8 (32%)            E= 6                    Additional Notes:
M= 1                 H= 17 (68%)          U= 0                                                                          Page 6 of 7                                                  OBDI 202 - IOLE Process
 
ES-401                   RB-2010-04                                                 7                                                                 Form ES-401-9 N= 23


==GENERAL COMMENT==
==GENERAL COMMENT==
S:
S:
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
: 2. Chief Examiner follow-up comments are indicated in blue. 3. Average difficulty is _2.9 ___   on the RO exam and     2.8___     on the SRO exam. These values fall within the expected range for this reviewer
: 2. Chief Examiner follow-up comments are indicated in blue.
: 4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 0 41.2 = 2 43.2 = 4 41.3 = 2 43.3 = 0 41.4 = 6 43.4 = 2 41.5 = 9 43.5 = 19 41.6 = 6 43.6 = 0 41.7 = 23 43.7 = 0 41.8 = 2 41.9 = 3 41.10 = 18 41.11 = 2 41.12 = 0 41.13 = 2 41.14 =
: 3. Average difficulty is _2.9 ___ on the RO exam and       2.8___   on the SRO exam. These values fall within the expected range for this reviewer
: 5. The answer distribution is: RO / SRO A = 18 (25%) / 7 (28%) B = 19 (25%) / 6 (24%) C = 19 (25%) / 5 (20%)
: 4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0               43.1 = 0 41.2 = 2               43.2 = 4 41.3 = 2               43.3 = 0 41.4 = 6               43.4 = 2 41.5 = 9               43.5 = 19 41.6 = 6               43.6 = 0 41.7 = 23               43.7 = 0 41.8 = 2 41.9 = 3 41.10 = 18 41.11 = 2 41.12 = 0 41.13 = 2 41.14 =
D = 19 (25%) / 7 (28%)
: 5.       The answer distribution is: RO / SRO A = 18 (25%)       /         7 (28%)
: 6. There are 0 / 3    questions with attachments provided.
B = 19 (25%)       /         6 (24%)
: 7. This exam was reviewed by S. Garchow on March 9-10, 2010.}}
C = 19 (25%)       /         5 (20%)
D = 19 (25%)       /         7 (28%)
: 6.       There are 0 / 3    questions with attachments provided.
: 7.       This exam was reviewed by S. Garchow on March 9-10, 2010.                                                                     Page 7 of 7                                              OBDI 202 - IOLE Process}}

Latest revision as of 13:49, 13 November 2019

RB-2010-Draft Written Exam Review
ML102450367
Person / Time
Site: River Bend Entergy icon.png
Issue date: 04/12/2010
From: Garchow S
Operations Branch IV
To:
Entergy Operations
References
50-458/10-04
Download: ML102450367 (7)
v • d • e


Text

ES-401 RB-2010-04 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 2 N S C, CR, 41.10 2 F 2 N S A, CR, 41.7 3 F 3 N S A, CR, 41.7 B, CR, 41.6 - Change trips to trip in distactors.

E 4 H 2 M This change will be made.

S The text editing was made as recommended.

5 F 3 N S D, CR, 41.7 6 F 3 N S C, CR, 41.7 D, CR, 41.10 - This is a KA mismatch due to being focused on TBCW rather than U

7 F 3 U B CCW.

S This question will be rewritten.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only This question was replaced with a new question.

8 H 3 N S B, CR, 41.4 9 F 2 N S A, CR, 41.8 10 F 3 N S C, CR, 41.4 11 H 3 N S A, CR, 41.3 12 H 4 N S B, CR, 41.5 13 F 3 M S A, CR, 41.7 14 F 3 N S D, CR, 41.9 15 H 3 N S C, CR, 41.10 16 H 3 N S B, CR, 41.2 17 H 3 N S D, CR, 41.10 18 F 3 N S B, CR, 41.13 19 F 3 N S C, CR, 41.10 20 H 4 N S A, CR, 41.5 21 H 3 N S B, CR, 41.5 22 H 3 N S A, CR, 41.9 23 H 3 N S B, CR, 41.7 A, CR, 41.6 - This question is U because distractor B is also correct.

U 24 H 3 U N Distractor B will be revised.

S Distractor B was rewritten.

25 F 3 N S B, CR, 41.7 26 F 3 N S A, CR, 41.10 27 H 3 N S C, CR, 41.11 28 H 3 N S C, CR, 41.7 29 H 3 N S D, CR, 41.7 Page 2 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 30 H 3 N S D, CR, 41.8 31 H 3 N S D, CR, 41.7 32 F 3 N S C, CR, 41.10 33 H 3 N S B, CR, 41.6 34 F 3 N S C, CR, 41.10 35 H 3 N S B, CR, 41.2 36 F 3 N S C, CR, 41.6 37 H 3 N S D, CR, 41.6 B, CR, 41.6 - Distractor A can also be argued as being correct since it is a subset of E the correct answer.

38 H 3 E N S Distractor A will be revised.

Distractor A was rewritten.

39 H 3 N S C, CR, 41.7 40 H 3 N S C, CR, 41.7 41 H 3 N S D, CR, 41.7 42 H 3 N S B, CR, 41.7 43 H 3 B S B, CR, 41.5 44 H 3 N S A, CR, 41.5 D, CR, 41.7 - Need to discuss this question as it seems annulus pressure would E normally be negative and would remain so under the described conditions in the 45 H 3 N stem.

S After discussing with the exam author, this question is correct as written.

46 H 3 N S B, CR, 41.7 47 H 3 M S A, CR, 41.7 48 F 3 N S A, CR, 41.7 49 H 3 N S D, CR, 41.7 50 H 3 B S C, CR, 41.7 Page 3 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 51 H 3 N S D, CR, 41.7 52 H 3 N S D, CR, 41.7 53 H 2 B S B, CR, 41.10 E D, CR, 41.3 - Discuss this question with exam author.

54 F 2 E N S This question is ok as written.

C, CR, 41.4 - Discuss this question with exam author.

E 55 F 3 E N The stem will be revised to improve the readability.

S Part of the question in the stem was highlighted to improve the focus of the question.

56 H 3 N S B, CR, 41.4 57 H 3 N S B, CR, 41.3 58 F 2 N S A, CR, 41.7 C, CR, 41.13 - This is a KA mismatch because it does not involve fuel handling.

U Suggest this KA be rejected and a new one selected.

59 F 3 U N S This question will be rewritten.

This question was replaced with a new question.

60 F 3 N S B, CR, 41.10 A, CR, 41.5 - Revise distractor B so it reads 980 MW instead of 750 MW.

E 61 H 3 E M This edit change will be made.

S This change was made as recommended.

62 H 3 N S C, CR, 41.4 63 H 3 N S A, CR, 41.5 64 F 3 N S D, CR, 41.4 65 H 2 N S D, CR, 41.5 66 F 3 N S D, CR, 41.10 67 F 2 N S C, CR, 41.10 68 F 3 N S D, CR, 41.10 69 F 3 B S C, CR, 41.10 Page 4 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E A, CR, 41.10 - Not sure this is a memory item; check with exam author.

70 F 3 E N S Based on feedback from the validating crew, this is a memory item.

71 F 3 N S B, CR, 41.10 72 F 3 N S A, CR, 41.9 73 H 4 N S A, CR, 41.11 C, CR, 43.5 - This is an SRO Only question in that it tests on the emergency plan U notification requirements.

74 F 3 U N S This question will be revised.

This question was replaced with a new question.

75 F 2 B S C, CR, 41.10 76 H 3 B S D, CR, 43.5 77 H 3 N S C, CR, 43.5 78 H 3 N S C, CR, 43.5 79 F 2 N S B, CR, 43.5 80 H 3 N S A, OR, 43.5 A, CR, 43.5 - Add another EOP related distractor other than the correct answer.

E 81 H 3 E N This change will be made.

S Distractor D was rewritten.

82 F 3 N S D, OR, 43.2 83 F 2 N S A, CR, 43.5 84 H 3 M S B, CR, 43.5 85 F 3 N S D, CR, 43.5 86 H 3 N S B, CR, 43.5 87 H 3 N S C, CR, 43.5 88 H 3 N S A, CR, 43.5 89 F 3 N S A, CR, 43.2 Page 5 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 90 H 3 N S D, CR, 43.5 91 H 3 N S B, CR, 43.2 92 H 3 N S C, CR, 43.5 A, CR, 43.2 - Candidates are not typically expected to have 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> TS action E statements memorized.

93 F 3 E N S The exam author will check with Ops on this.

This question was replaced with a new question.

C, CR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not E

94 E N symptom/system specific.

S This question was replaced.

D, OR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not symptom specific. Recommend deleting the RHR status from the stem and change E so it reads Subsequent to the above conditions, a loss of all shutdown cooling 95 H 3 E N occurs.

S This change will be made.

The question was revised as discussed above.

96 F 2 N S A, CR, 43.5 E D, CR, 43.4 - Discuss this question with exam author.

97 H 2 E N S This question is ok as written.

98 F 3 N S B, CR, 43.4 B, CR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not E

99 H 2 E N symptom specific.

S This question was replaced with a new question.

100 H 3 N S D, CR, 43.5 B= 2 F= 32 (43%) E= 7 RO TOTALS: M= 9 H= 43 (57%) U= 4 Additional Notes:

N= 64 SRO TOTALS: B= 1 F= 8 (32%) E= 6 Additional Notes:

M= 1 H= 17 (68%) U= 0 Page 6 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 7 Form ES-401-9 N= 23

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
2. Chief Examiner follow-up comments are indicated in blue.
3. Average difficulty is _2.9 ___ on the RO exam and 2.8___ on the SRO exam. These values fall within the expected range for this reviewer
4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 0 41.2 = 2 43.2 = 4 41.3 = 2 43.3 = 0 41.4 = 6 43.4 = 2 41.5 = 9 43.5 = 19 41.6 = 6 43.6 = 0 41.7 = 23 43.7 = 0 41.8 = 2 41.9 = 3 41.10 = 18 41.11 = 2 41.12 = 0 41.13 = 2 41.14 =
5. The answer distribution is: RO / SRO A = 18 (25%) / 7 (28%)

B = 19 (25%) / 6 (24%)

C = 19 (25%) / 5 (20%)

D = 19 (25%) / 7 (28%)

6. There are 0 / 3 questions with attachments provided.
7. This exam was reviewed by S. Garchow on March 9-10, 2010. Page 7 of 7 OBDI 202 - IOLE Process
Retrieved from "https://kanterella.com/w/index.php?title=ML102450367&oldid=2128274"