ML101960329: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
 
(One intermediate revision by the same user not shown)
Line 3: Line 3:
| issue date = 11/02/2009
| issue date = 11/02/2009
| title = PV-2009-11-Draft Written Exam Comments
| title = PV-2009-11-Draft Written Exam Comments
| author name = Garchow S M
| author name = Garchow S
| author affiliation = NRC/RGN-IV/DRS/OB
| author affiliation = NRC/RGN-IV/DRS/OB
| addressee name =  
| addressee name =  
Line 15: Line 15:


=Text=
=Text=
{{#Wiki_filter:ES-401 PV-2009-11 Written Examination Review Worksheet Form ES-401-9 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:
{{#Wiki_filter:ES-401               PV-2009-11                                             Written Examination Review Worksheet                                                                         Form ES-401-9
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).  
: 1.      2.           3. Psychometric Flaws              4. Job Content Flaws        5. Other      6.      7.                                            8.
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).  
Q#    LOK     LOD (F/H)   (1-5)   Stem Cues T/F        Cred. Partial Job- Minutia #/ Back-            Q= SRO B/M/N U/E/S                                              Explanation Focus                Dist.           Link            units ward    K/A Only C41.10 1      H      3                                                                                         N      S Agree D41.4 Answer states that the switch needs to be placed in START; however, the E
$ The answer choices are a collection of unrelated true/false statements.  
2      H      2                                                                                          N            reference says place the switch in AUTO.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.  
S Agree - Stem was revised to correct wording.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:
E    D41.7 Reformat the answers without all the bullets.
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).  
3      F      2                                                                                          N S    Agree A41.10 May clue that question 1, distractors A and B are incorrect.
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
4      F      2                                                                                          M      S OK as written.
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).  
5      H      3                                                                                          N      S    D41.3 E    B41.7 See markup 6      H      3                                                                                          N S    Comment was corrected.
$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer
Instructions
=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
Attachment 8 Page 1 of 7 OBDI 202 - IOLE Process
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
: 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
Dist. Partial Job-LinkMinutia#/
: 3. Check the appropriate box if a psychometric flaw is identified:
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 1 H 3            N S C41.10 Agree 2 H 2            N E S D41.4 Answer states that the switch needs to be placed in START; however, the reference says place the switch in AUTO. Agree - Stem was revised to correct wording. 3 F 2            N E S D41.7 Reformat the answers without all the bullets.
      $   The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
Agree 4 F 2            M S A41.10 May clue that question 1, distractors A and B are incorrect. OK as written. 5 H 3            N S D41.3 6 H 3            N E S B41.7 See markup Comment was corrected.
      $    The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
ES-401 PV-2009-11 2 Form ES-401-9 Attachment 8 Page 2 of 7 OBDI 202 - IOLE Process
      $    The answer choices are a collection of unrelated true/false statements.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
      $    The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
Dist. Partial Job-LinkMinutia#/
      $    One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 7 H 3            N E S B41.4 See markup Markup was corrected. 8 H 2            B S B41.3 9 F 2            N S D41.1 10 F 3            B E S C41.10 See markup Markup was corrected. 11 F 2           N S B41.5 12 F 2           N S A41.5 13 H 3            N S D41.5 14 H 2            M S D41.10 15 F 2            B S B41.7 16 H 3            N S C41.5 17 F 3            N S B41.4 18 H 2            M S D41.5 19 F 2           U N U S C41.6, 43.2 See markups. Agree - TS bases makes it an SRO question.
: 4. Check the appropriate box if a job content error is identified:
The TS bases was removed from the question. 20 F 3           N S B41.5, 10 21 H 3           M S B41.7 22 H 3            M E S D41.5 See markup.
      $ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
Markup was revised. 23 F 2           B S C41.7 24 H 3           N S C41.10 25 F 3           B S A41.11 26 H 3           N S B41.10 27 H 3           N E D41.10 VERIFY that PBB-S04 cannot be energized based on given information.
      $ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
ES-401 PV-2009-11 3 Form ES-401-9 Attachment 8 Page 3 of 7 OBDI 202 - IOLE Process
      $ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
      $ The question requires reverse logic or application compared to the job requirements.
Dist. Partial Job-LinkMinutia#/
: 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation S It would seem distractor B could be argued as also correct. Distractor B was revised. 28 F 2            N E S A41.5 See markup. Markup was corrected. 29 H 2           M S A41.7 30 H 3           N S D41.7 31 F 3           B S D41.4 32 H 2           N S B41.7 33 H 3           N S B41.5 34 H 2            M E S D41.5 See markup. Markup was corrected. 35 F 2           N S A41.10 36 H 2           N S A41.7 37 F 3            B E S C41.5 See markup. Markup was corrected. 38 H 3           N S C41.7 39 F 3           N S D41.7 40 F 3           N S C41.7 41 F 3           N S A41.4 42 H 3           N S C41.7 NOT true/false because knowledge of mini-flow ops is required to answer. 43 H 3            N E S C41.7 See markup. Markup was corrected. 44 H 3           N S A41.7 45 H 2           M S C41.7 46 H 3           N S A, 41.7 47 F 2           N S A, 41.10 ES-401 PV-2009-11 4 Form ES-401-9 Attachment 8 Page 4 of 7 OBDI 202 - IOLE Process
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
: 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
Dist. Partial Job-LinkMinutia#/
: 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).                                                                                       Page 1 of 7                                                        OBDI 202 - IOLE Process
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 48 H 3           M S A, 41.7 49 H 3           B S A, 41.7 50 H 3           B S A, 41.7 51 F 2           B S B, 41.7 52 H 3           N S C, 41.11 53 H 3           N S B, 41.7 54 F 2           B S A, 41.7 55 H 3           N S C, 41.7 56 H 3           N S A, 41.5 57 F 2           M S B, 41.2 58 F 2           N S A, 41.7 59 F 2           N S D, 41.2 60 H 2           B S A, 41.7 61 F 3            N E S A, 41.10 - This question could be argued to have no correct answer since it does not address the conditions as presented in the LOCA EOP (step 66). FIX The question was revised to correct the stem. 62 H 3           N S A, 41.7 63 H 3           M S B, 41.5 64 F 2           N S B, 41.4 65 H 3           N S C, 41.7 66 F 2           N S B, 41.10 67 H 3          U  M U S D, 41.7 - This is a KA mismatch because Tier 3 questions are meant to have generic implications and not system specific. This question was revised. 68 F 3           B S C, 41.10 69 F 3            N E D, 41.3 - Not sure the question is correct. The TS limits may not be the same as the design criteria. CAF.
 
ES-401 PV-2009-11 5 Form ES-401-9 Attachment 8 Page 5 of 7 OBDI 202 - IOLE Process
ES-401                PV-2009-11                                                    2                                                                    Form ES-401-9
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
: 1. 2.       3. Psychometric Flaws          4. Job Content Flaws  5. Other  6. 7.                                       8.
Dist. Partial Job-LinkMinutia#/
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-    Q= SRO B/M/N U/E/S                                   Explanation Focus              Dist.        Link          units ward K/A Only E  B41.4 See markup 7    H    3                                                                        N S  Markup was corrected.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation S This question is ok as written. 70 H 3           N S A, 41.10 71 F 3           N S B, 41.10 72 H 3           N S B, 41.12 73 F 2           N S B, 41.12 74 H 3            N E S D, 41.10 - Clean up the stem. The stem was redone to fix the formatting. 75 F 2           N S C, 41.10 - Could not locate the referenced picture. 76 H 3           N S B, 43.5 77 H 3           N S C, 43.5 78 H 3           M S D, 43.5 79 H 3           M S D, 43.5 80 H 2           B S A, 43.5 81 H 3          U M U S D, 41.10 - This is an RO question because it tests on specific actions taken by the operator. This question was revised. 82 H 3           M S C, 43.5 83 H 3           N S A, 43.5 84 H 3           M S A, 43.5 85 F 3            M E S B, 43.2 - Not sure why this is a memory question. CAF This question is supported by a learning objective. 86 F 2           N S D, 41.10 87 H 3           N S D, 43.5 88 H 3           N S B, 41.10 89 H 4           N S A, 43.5 90 F 3           M S A, 43.5 ES-401 PV-2009-11 6 Form ES-401-9 Attachment 8 Page 6 of 7 OBDI 202 - IOLE Process
8    H    2                                                                         B    S B41.3 9    F     2                                                                         N     S D41.1 E  C41.10 See markup 10    F     3                                                                        B S Markup was corrected.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
11    F    2                                                                        N     S B41.5 12    F     2                                                                        N     S A41.5 13    H     3                                                                        N    S D41.5 14    H    2                                                                        M    S  D41.10 15    F     2                                                                         B    S  B41.7 16    H    3                                                                        N     S C41.5 17    F    3                                                                        N    S  B41.4 18    H    2                                                                        M    S  D41.5 C41.6, 43.2 See markups.
Dist. Partial Job-LinkMinutia#/
U 19    F    2                                                                    U    N        Agree - TS bases makes it an SRO question.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 91 H 3           N S A, 43.5 92 H 3           B S A, 43.5 93 H 3           N S B, 43.2 94 H 3          U N U S D, 41.10 - This is RO knowledge since it can be answered without doing a procedure selection. This question is ok as written. 95 F 3           N S A, 41.10 96 F 3            B E S B, 43.2 - Not sure SROs need to have all the 3.0.3 entry specs memorized. CAF OK as written. 97 H 3           N S C, 43.5 98 H 3          U N U S C, 41.10 - This is RO knowledge since it tests on specific operator actions. This question is ok as written. 99 F 2          U N U S B, 41.10 - Procedure usage is RO knowledge. This question was replaced. 100 F 3      E     N E S B, 41.10 - Not sure this is a valid memory question. This question is supported by a learning objective. B= F= E= M= H= U= RO TOTALS: N=   Additional Notes:  B=  F= E= M=  H= U= SRO TOTALS: N=    Additional Notes:
S The TS bases was removed from the question.
20   F     3                                                                         N     S B41.5, 10 21   H     3                                                                         M     S B41.7 E D41.5 See markup.
22    H    3                                                                        M S  Markup was revised.
23   F     2                                                                         B     S C41.7 24   H     3                                                                         N     S C41.10 25   F     3                                                                         B     S A41.11 26   H     3                                                                         N     S B41.10 27   H     3                                                                         N     E D41.10 VERIFY that PBB-S04 cannot be energized based on given information.                                                                     Page 2 of 7                                            OBDI 202 - IOLE Process
 
ES-401               PV-2009-11                                                     3                                                                       Form ES-401-9
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                          8.
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                       Explanation Focus              Dist.         Link          units ward K/A Only S It would seem distractor B could be argued as also correct.
Distractor B was revised.
E A41.5 See markup.
28    F    2                                                                        N S  Markup was corrected.
29   H     2                                                                         M     S A41.7 30   H     3                                                                         N     S D41.7 31   F     3                                                                         B     S D41.4 32   H     2                                                                         N     S B41.7 33   H     3                                                                         N     S B41.5 E D41.5 See markup.
34    H    2                                                                        M S  Markup was corrected.
35   F     2                                                                         N     S A41.10 36   H     2                                                                         N     S A41.7 E C41.5 See markup.
37    F    3                                                                        B S  Markup was corrected.
38   H     3                                                                         N     S C41.7 39   F     3                                                                         N     S D41.7 40   F     3                                                                         N     S C41.7 41   F     3                                                                         N     S A41.4 42   H     3                                                                         N     S C41.7 NOT true/false because knowledge of mini-flow ops is required to answer.
E C41.7 See markup.
43    H    3                                                                        N S  Markup was corrected.
44   H     3                                                                         N     S A41.7 45   H     2                                                                         M     S C41.7 46   H     3                                                                         N     S A, 41.7 47   F     2                                                                         N     S A, 41.10                                                                     Page 3 of 7                                                OBDI 202 - IOLE Process
 
ES-401               PV-2009-11                                                     4                                                                         Form ES-401-9
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                            8.
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                         Explanation Focus              Dist.         Link          units ward K/A Only 48   H     3                                                                         M     S A, 41.7 49   H     3                                                                         B     S A, 41.7 50   H     3                                                                         B     S A, 41.7 51   F     2                                                                         B     S B, 41.7 52   H     3                                                                         N     S C, 41.11 53   H     3                                                                         N     S B, 41.7 54   F     2                                                                         B     S A, 41.7 55   H     3                                                                         N     S C, 41.7 56   H     3                                                                         N     S A, 41.5 57   F     2                                                                         M     S B, 41.2 58   F     2                                                                         N     S A, 41.7 59   F     2                                                                         N     S D, 41.2 60   H     2                                                                         B     S A, 41.7 A, 41.10 - This question could be argued to have no correct answer since it does not E
61    F    3                                                                        N        address the conditions as presented in the LOCA EOP (step 66). FIX S
The question was revised to correct the stem.
62   H     3                                                                         N     S A, 41.7 63   H     3                                                                         M     S B, 41.5 64   F     2                                                                         N     S B, 41.4 65   H     3                                                                         N     S C, 41.7 66   F     2                                                                         N     S B, 41.10 D, 41.7 - This is a KA mismatch because Tier 3 questions are meant to have generic U
67    H    3                                                              U          M        implications and not system specific.
S This question was revised.
68   F     3                                                                         B     S C, 41.10 D, 41.3 - Not sure the question is correct. The TS limits may not be the same as the 69    F    3                                                                        N    E design criteria. CAF.                                                                     Page 4 of 7                                                  OBDI 202 - IOLE Process
 
ES-401               PV-2009-11                                                     5                                                                           Form ES-401-9
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                          8.
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                       Explanation Focus              Dist.         Link          units ward K/A Only S This question is ok as written.
70   H     3                                                                         N     S A, 41.10 71   F     3                                                                         N     S B, 41.10 72   H     3                                                                         N     S B, 41.12 73   F     2                                                                         N     S B, 41.12 E D, 41.10 - Clean up the stem.
74    H    3                                                                        N S  The stem was redone to fix the formatting.
75   F     2                                                                         N     S C, 41.10 - Could not locate the referenced picture.
76   H     3                                                                         N     S B, 43.5 77   H     3                                                                         N     S C, 43.5 78   H     3                                                                         M     S D, 43.5 79   H     3                                                                         M     S D, 43.5 80   H     2                                                                         B     S A, 43.5 D, 41.10 - This is an RO question because it tests on specific actions taken by the U
81    H    3                                                                    U    M        operator.
S This question was revised.
82   H     3                                                                         M     S C, 43.5 83   H     3                                                                         N     S A, 43.5 84   H     3                                                                         M     S A, 43.5 E B, 43.2 - Not sure why this is a memory question. CAF 85    F    3                                                                        M S  This question is supported by a learning objective.
86   F     2                                                                         N     S D, 41.10 87   H     3                                                                         N     S D, 43.5 88   H     3                                                                         N     S B, 41.10 89   H     4                                                                         N     S A, 43.5 90   F     3                                                                         M     S A, 43.5                                                                     Page 5 of 7                                                OBDI 202 - IOLE Process
 
ES-401                     PV-2009-11                                                       6                                                                         Form ES-401-9
: 1. 2.          3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.                                          8.
Q#   LOK   LOD (F/H) (1-5)   Stem Cues T/F       Cred. Partial Job- Minutia #/ Back-     Q= SRO B/M/N U/E/S                                         Explanation Focus                Dist.         Link          units ward  K/A Only 91   H     3                                                                             N     S A, 43.5 92   H     3                                                                             B     S A, 43.5 93   H     3                                                                             N     S B, 43.2 D, 41.10 - This is RO knowledge since it can be answered without doing a procedure U
94    H      3                                                                        U    N        selection.
S This question is ok as written.
95   F     3                                                                             N     S A, 41.10 E B, 43.2 - Not sure SROs need to have all the 3.0.3 entry specs memorized. CAF 96    F      3                                                                              B S  OK as written.
97   H     3                                                                             N     S C, 43.5 U C, 41.10 - This is RO knowledge since it tests on specific operator actions.
98    H      3                                                                        U    N S  This question is ok as written.
U B, 41.10 - Procedure usage is RO knowledge.
99    F      2                                                                        U    N S  This question was replaced.
E B, 41.10 - Not sure this is a valid memory question.
100    F      3                                                E                            N S  This question is supported by a learning objective.
B=                   F=                   E=
RO TOTALS:                M=                  H=                    U=                      Additional Notes:
N=
B=                  F=                    E=
SRO TOTALS:                M=                   H=                    U=                      Additional Notes:
N=


==GENERAL COMMENT==
==GENERAL COMMENT==
S:
S:
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
: 2. Chief Examiner comments are indicated in blue. 3. Average difficulty is __   ___   on the RO exam and    ___      on the SRO exam.
: 2. Chief Examiner comments are indicated in blue.
ES-401 PV-2009-11 7 Form ES-401-9 Attachment 8 Page 7 of 7 OBDI 202 - IOLE Process
: 3. Average difficulty is __   ___ on the RO exam and         ___     on the SRO exam.                                                                             Page 6 of 7                                                OBDI 202 - IOLE Process
: 4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 =   43.1 =   41.2 =   43.2 =   41.3 =   43.3 =   41.4 =   43.4 =
 
41.5 =   43.5 =   41.6 =   43.6 =   41.7 =   43.7 =
ES-401                 PV-2009-11                             7                     Form ES-401-9
41.8 =     41.9 =   41.10 =
: 4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 =                 43.1 =
41.11 =   41.12 =   41.13 =
41.2 =                 43.2 =
41.3 =                 43.3 =
41.4 =                 43.4 =
41.5 =                 43.5 =
41.6 =                 43.6 =
41.7 =                 43.7 =
41.8 =
41.9 =
41.10 =
41.11 =
41.12 =
41.13 =
41.14 =
41.14 =
: 5. The answer distribution is: RO / SRO A = 13 (17%) / 6 (24%) B = 16 (22%) / 7 (28%)
: 5.       The answer distribution is: RO / SRO A = 13 (17%)     /         6 (24%)
C = 21 (28%) / 5 (20%) D = 25 (33%) / 7 (28%)
B = 16 (22%)     /         7 (28%)
: 6. There are       questions with attachments provided.}}
C = 21 (28%)     /         5 (20%)
D = 25 (33%)     /         7 (28%)
: 6.       There are     questions with attachments provided.                                                 Page 7 of 7 OBDI 202 - IOLE Process}}

Latest revision as of 16:07, 13 November 2019

PV-2009-11-Draft Written Exam Comments
ML101960329
Person / Time
Site: Palo Verde  Arizona Public Service icon.png
Issue date: 11/02/2009
From: Garchow S
Operations Branch IV
To:
Arizona Public Service Co
References
Download: ML101960329 (7)
v • d • e


Text

ES-401 PV-2009-11 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only C41.10 1 H 3 N S Agree D41.4 Answer states that the switch needs to be placed in START; however, the E

2 H 2 N reference says place the switch in AUTO.

S Agree - Stem was revised to correct wording.

E D41.7 Reformat the answers without all the bullets.

3 F 2 N S Agree A41.10 May clue that question 1, distractors A and B are incorrect.

4 F 2 M S OK as written.

5 H 3 N S D41.3 E B41.7 See markup 6 H 3 N S Comment was corrected.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E B41.4 See markup 7 H 3 N S Markup was corrected.

8 H 2 B S B41.3 9 F 2 N S D41.1 E C41.10 See markup 10 F 3 B S Markup was corrected.

11 F 2 N S B41.5 12 F 2 N S A41.5 13 H 3 N S D41.5 14 H 2 M S D41.10 15 F 2 B S B41.7 16 H 3 N S C41.5 17 F 3 N S B41.4 18 H 2 M S D41.5 C41.6, 43.2 See markups.

U 19 F 2 U N Agree - TS bases makes it an SRO question.

S The TS bases was removed from the question.

20 F 3 N S B41.5, 10 21 H 3 M S B41.7 E D41.5 See markup.

22 H 3 M S Markup was revised.

23 F 2 B S C41.7 24 H 3 N S C41.10 25 F 3 B S A41.11 26 H 3 N S B41.10 27 H 3 N E D41.10 VERIFY that PBB-S04 cannot be energized based on given information. Page 2 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S It would seem distractor B could be argued as also correct.

Distractor B was revised.

E A41.5 See markup.

28 F 2 N S Markup was corrected.

29 H 2 M S A41.7 30 H 3 N S D41.7 31 F 3 B S D41.4 32 H 2 N S B41.7 33 H 3 N S B41.5 E D41.5 See markup.

34 H 2 M S Markup was corrected.

35 F 2 N S A41.10 36 H 2 N S A41.7 E C41.5 See markup.

37 F 3 B S Markup was corrected.

38 H 3 N S C41.7 39 F 3 N S D41.7 40 F 3 N S C41.7 41 F 3 N S A41.4 42 H 3 N S C41.7 NOT true/false because knowledge of mini-flow ops is required to answer.

E C41.7 See markup.

43 H 3 N S Markup was corrected.

44 H 3 N S A41.7 45 H 2 M S C41.7 46 H 3 N S A, 41.7 47 F 2 N S A, 41.10 Page 3 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 48 H 3 M S A, 41.7 49 H 3 B S A, 41.7 50 H 3 B S A, 41.7 51 F 2 B S B, 41.7 52 H 3 N S C, 41.11 53 H 3 N S B, 41.7 54 F 2 B S A, 41.7 55 H 3 N S C, 41.7 56 H 3 N S A, 41.5 57 F 2 M S B, 41.2 58 F 2 N S A, 41.7 59 F 2 N S D, 41.2 60 H 2 B S A, 41.7 A, 41.10 - This question could be argued to have no correct answer since it does not E

61 F 3 N address the conditions as presented in the LOCA EOP (step 66). FIX S

The question was revised to correct the stem.

62 H 3 N S A, 41.7 63 H 3 M S B, 41.5 64 F 2 N S B, 41.4 65 H 3 N S C, 41.7 66 F 2 N S B, 41.10 D, 41.7 - This is a KA mismatch because Tier 3 questions are meant to have generic U

67 H 3 U M implications and not system specific.

S This question was revised.

68 F 3 B S C, 41.10 D, 41.3 - Not sure the question is correct. The TS limits may not be the same as the 69 F 3 N E design criteria. CAF. Page 4 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S This question is ok as written.

70 H 3 N S A, 41.10 71 F 3 N S B, 41.10 72 H 3 N S B, 41.12 73 F 2 N S B, 41.12 E D, 41.10 - Clean up the stem.

74 H 3 N S The stem was redone to fix the formatting.

75 F 2 N S C, 41.10 - Could not locate the referenced picture.

76 H 3 N S B, 43.5 77 H 3 N S C, 43.5 78 H 3 M S D, 43.5 79 H 3 M S D, 43.5 80 H 2 B S A, 43.5 D, 41.10 - This is an RO question because it tests on specific actions taken by the U

81 H 3 U M operator.

S This question was revised.

82 H 3 M S C, 43.5 83 H 3 N S A, 43.5 84 H 3 M S A, 43.5 E B, 43.2 - Not sure why this is a memory question. CAF 85 F 3 M S This question is supported by a learning objective.

86 F 2 N S D, 41.10 87 H 3 N S D, 43.5 88 H 3 N S B, 41.10 89 H 4 N S A, 43.5 90 F 3 M S A, 43.5 Page 5 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 91 H 3 N S A, 43.5 92 H 3 B S A, 43.5 93 H 3 N S B, 43.2 D, 41.10 - This is RO knowledge since it can be answered without doing a procedure U

94 H 3 U N selection.

S This question is ok as written.

95 F 3 N S A, 41.10 E B, 43.2 - Not sure SROs need to have all the 3.0.3 entry specs memorized. CAF 96 F 3 B S OK as written.

97 H 3 N S C, 43.5 U C, 41.10 - This is RO knowledge since it tests on specific operator actions.

98 H 3 U N S This question is ok as written.

U B, 41.10 - Procedure usage is RO knowledge.

99 F 2 U N S This question was replaced.

E B, 41.10 - Not sure this is a valid memory question.

100 F 3 E N S This question is supported by a learning objective.

B= F= E=

RO TOTALS: M= H= U= Additional Notes:

N=

B= F= E=

SRO TOTALS: M= H= U= Additional Notes:

N=

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
2. Chief Examiner comments are indicated in blue.
3. Average difficulty is __ ___ on the RO exam and ___ on the SRO exam. Page 6 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 7 Form ES-401-9

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 43.1 =

41.2 = 43.2 =

41.3 = 43.3 =

41.4 = 43.4 =

41.5 = 43.5 =

41.6 = 43.6 =

41.7 = 43.7 =

41.8 =

41.9 =

41.10 =

41.11 =

41.12 =

41.13 =

41.14 =

5. The answer distribution is: RO / SRO A = 13 (17%) / 6 (24%)

B = 16 (22%) / 7 (28%)

C = 21 (28%) / 5 (20%)

D = 25 (33%) / 7 (28%)

6. There are questions with attachments provided. Page 7 of 7 OBDI 202 - IOLE Process
Retrieved from "https://kanterella.com/w/index.php?title=ML101960329&oldid=2130786"