ML090750879: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
 
(One intermediate revision by the same user not shown)
Line 15: Line 15:


=Text=
=Text=
{{#Wiki_filter:ES-401 RB-2008-12 RO Written Examination Review WorksheetRev 1 Form ES-401-9 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:  
{{#Wiki_filter:ES-401               RB-2008-12                                           RO Written Examination Review Worksheet Rev 1                                                               Form ES-401-9
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).  
: 1.      2.          3. Psychometric Flaws              4. Job Content Flaws        5. Other      6.      7.                                            8.
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).  
Q#    LOK    LOD (F/H)  (1-5)  Stem Cues T/F        Cred. Partial Job- Minutia #/ Back-            Q= SRO B/M/N U/E/S                                              Explanation Focus                Dist.            Link            units ward    K/A Only 1N      F      3                                                                                                  E    B, CR, 41.5 Need reference that is not TS bases to be RO level question. Fixed 2N      H      3                                                                                                  S    B, CR, 41.7 3M      F      3                                                                                                  S    B, CR, 41.7 Need original question Done 4N      H      2                                                                                                  S    A, CR, 41.6 5B      H      3                                                                                                  S    A, CR, 41.7, N97 D, CR, 43.5 - It is SRO only because it requires the candidate to evaluate a given set 6B      F      3                                                                                  X              U    of plant conditions and make a procedure selection based on this assessment.
$ The answer choices are a collection of unrelated true/false statements.  
Submitted new question 7N      H      3                                                                                                  S    C, CR, 41.4 8N      H      3                                                                                                  S    D, CR, 41.4 9N      F      2                                                                                                  S    D, CR, 41.10 Instructions
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.  
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:  
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).  
: 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).  
: 3. Check the appropriate box if a psychometric flaw is identified:
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).  
      $   The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer
      $   The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
      $   The answer choices are a collection of unrelated true/false statements.
Attachment 8 Page 1 of 5 OBDI 202 - IOLE Process  
      $   The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other  Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia#/ units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 1N F 3            E B, CR, 41.5  Need reference that is not TS bases to be RO level question. Fixed 2N H 3            S B, CR, 41.7 3M F 3            S B, CR, 41.7 Need original question Done 4N H 2            S A, CR, 41.6 5B H 3            S A, CR, 41.7, N97 6B F 3          X  U D, CR, 43.5 - It is SRO only because it requires the candidate to evaluate a given set of plant conditions and make a procedure selection based on this assessment. Submitted new question 7N H 3            S C, CR, 41.4 8N H 3            S D, CR, 41.4 9N F 2            S D, CR, 41.10 ES-401 RB-2008-12 2 Rev 1 Form ES-401-9  Attachment 8 Page 2 of 5 OBDI 202 - IOLE Process
      $   One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
: 4. Check the appropriate box if a job content error is identified:
Dist. Partial Job-LinkMinutia#/
      $ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 10N F 3            S C, CR, 41.10 11N F 3            S A, CR, 41.7 12N F 3            E B, CR, 41.4 - minor enhancements 13M H 3            S A, CR, 41.10, N03  need original question.
      $ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
done 14N F 4            S B, CR, 41.7 15N F 3  X          E A, CR, 41.8 - The experienced test taker would probably select distractor A if the answer were not known since it is the lowest of the four distractors. Suggest the first distractor be something less than 13 feet. Second, remove the "greater than" symbol from all four distractors since the minimum level is, in fact, 13 feet. revised 16B H 2            S A, CR, 41.7 17B H 3            S C, CR, 41.1, N99  - Need original question. Done - changed to Bank 18N H 3            S B, CR, 41.11 19N F 3            S B, CR, 41.4 20N H 3            S C, CR, 41.5 21N H 3            S D, CR, 41.4 22N H 3            S A, CR, 41.7 23N H 2            S D, CR, 41.4 24N F 3          X  U C, CR, 41.10  - Revise to make RO level knowledge fixed 25N F 2            S D, CR, 41.6 26B F 2            E C, CR, 41.11  N07  - Current reading of 100 mr/hr is not "at" the max normal op value
      $ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
      $ The question requires reverse logic or application compared to the job requirements.
: 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
: 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).                                                                                       Page 1 of 5                                                       OBDI 202 - IOLE Process


as stated in distractor "c."
ES-401               RB-2008-12                                                     2                              Rev 1                                         Form ES-401-9
fixed 27N F 2            S C, CR, 4.7 28B H 3            E B, CR, 4.7, N00 - Distractor "b" contains a cue.
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                            8.
fixed 29N H 3            E A, CR, 41.7  - Reword stem for clarification done 30N H 3            E D, CR, 41.5  -  Distractors don't answer the question.
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                         Explanation Focus              Dist.         Link          units ward K/A Only 10N  F    3                                                                               S C, CR, 41.10 11N  F    3                                                                              S A, CR, 41.7 12N  F     3                                                                              E  B, CR, 41.4 - minor enhancements 13M    H     3                                                                               S  A, CR, 41.10, N03 need original question. done 14N  F    4                                                                              S  B, CR, 41.7 A, CR, 41.8 - The experienced test taker would probably select distractor A if the answer were not known since it is the lowest of the four distractors. Suggest the first 15N  F    3           X                                                                 E distractor be something less than 13 feet. Second, remove the greater than symbol from all four distractors since the minimum level is, in fact, 13 feet. revised 16B   H    2                                                                              S  A, CR, 41.7 17B  H    3                                                                              S  C, CR, 41.1, N99 - Need original question. Done - changed to Bank 18N  H    3                                                                              S B, CR, 41.11 19N  F     3                                                                              S  B, CR, 41.4 20N  H    3                                                                              S  C, CR, 41.5 21N  H    3                                                                              S D, CR, 41.4 22N  H     3                                                                               S  A, CR, 41.7 23N  H    2                                                                              S  D, CR, 41.4 24N  F     3                                                                   X          U C, CR, 41.10 - Revise to make RO level knowledge fixed 25N  F    2                                                                              S  D, CR, 41.6 C, CR, 41.11 N07 - Current reading of 100 mr/hr is not at the max normal op value 26B  F    2                                                                              E as stated in distractor c. fixed 27N  F    2                                                                              S  C, CR, 4.7 28B  H    3                                                                              E B, CR, 4.7, N00 - Distractor b contains a cue. fixed 29N  H     3                                                                               E A, CR, 41.7 - Reword stem for clarification done 30N  H    3                                                                              E  D, CR, 41.5 - Distractors dont answer the question. Revised stem 31N  F    2                                                                              S  B, CR, 41.7 32B  H    3                                                                              S  B, CR, 41.7, N97                                                                      Page 2 of 5                                                  OBDI 202 - IOLE Process
Revised stem 31N F 2            S B, CR, 41.7 32B H 3            S B, CR, 41.7, N97 ES-401 RB-2008-12 3 Rev 1 Form ES-401-9 Attachment 8 Page 3 of 5 OBDI 202 - IOLE Process
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 33N H 3             S D, CR, 41.7 34N H 2            S A, CR, 41.7 35N F 2            S B, CR, 41.10 36M H 3             E B, CR, 41.6, N97  - need original question done - minor enhancements 37M H 3         X  U A, CR, 41.6, N00  -KA requires "procedures to correct/control/mitigate-" Also, need original question.
fixed 38N F 2            S D, CR, 41.6 39B H 3             E D, CR, 41.7 minor enhancements - removed cue from stem 40N F 3             U C, OR, 41.7  Direct lookup with reference - revised, removed ref 41N H 3             E C, CR, 41.7 - Both G33-F004 in the stem and G33-F054 in the correct distractor C have the same noun name. Revise noun names to include RWCU pump suction and


discharge valves.
ES-401                RB-2008-12                                                    3                            Rev 1                                      Form ES-401-9
fixed 42N H 3             S C, CR, 41.7 43N H 3             S B, CR, 41.3 44N H 2             S D, CR, 41.13 45B F 2             S D, CR, 41.7, N07 46B H 3             E C, CR, 41.4 N97 - Clarify distractors C & D. done 47N H 3             S C, CR, 41.4 48N F 3   X         E C, CR, 41.4 - Need to revise distractor wording 49N F 3             S A, CR, 41.4 50N F 3             S B, CR, 41.4 51N H 3 X            U C, CR, 41.7 - The stem asks about changes in reactive load or KVA. KVA is not reactive load; KVAR is reactive load.
: 1. 2.      3. Psychometric Flaws          4. Job Content Flaws  5. Other  6. 7.                                          8.
fixed 52N H 3             E A, CR, 41.4 - KA requires "procedures to correct/control/mitigate-"
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F      Cred. Partial Job- Minutia #/ Back-    Q= SRO B/M/N U/E/S                                        Explanation Focus              Dist.        Link          units ward K/A Only 33N  H    3                                                                              S  D, CR, 41.7 34N  H    2                                                                              S  A, CR, 41.7 35N  F    2                                                                              S  B, CR, 41.10 36M    H    3                                                                              E  B, CR, 41.6, N97 - need original question done - minor enhancements A, CR, 41.6, N00 - KA requires procedures to correct/control/mitigate Also, need 37M    H    3                                                              X                U original question. fixed 38N  F    2                                                                              S  D, CR, 41.6 39B  H    3                                                                              E  D, CR, 41.7 minor enhancements - removed cue from stem 40N  F    3                                                                              U  C, OR, 41.7 Direct lookup with reference - revised, removed ref C, CR, 41.7 - Both G33-F004 in the stem and G33-F054 in the correct distractor C 41N  H    3                                                                              E  have the same noun name. Revise noun names to include RWCU pump suction and discharge valves. fixed 42N   H     3                                                                               S C, CR, 41.7 43N   H     3                                                                               S B, CR, 41.3 44N   H     2                                                                               S D, CR, 41.13 45B   F     2                                                                               S D, CR, 41.7, N07 46B   H     3                                                                               E C, CR, 41.4 N97 - Clarify distractors C & D. done 47N   H     3                                                                               S C, CR, 41.4 48N   F     3                       X                                                     E C, CR, 41.4 - Need to revise distractor wording 49N   F     3                                                                               S A, CR, 41.4 50N   F     3                                                                               S B, CR, 41.4 C, CR, 41.7 - The stem asks about changes in reactive load or KVA. KVA is not 51N  H    3    X                                                                        U reactive load; KVAR is reactive load. fixed 52N   H     3                                                                               E A, CR, 41.4 - KA requires procedures to correct/control/mitigate fixed 53B   H     3                                                                               S B, CR, 41.4, N04 54N   H     3                                                                               S B, CR, 41.10 55B   H     3                                                                               E C, CR, 41.6, N99 - Delete the second part (i.e. following due to) of the distractors                                                                     Page 3 of 5                                                OBDI 202 - IOLE Process
fixed 53B H 3             S B, CR, 41.4, N04 54N H 3             S B, CR, 41.10 55B H 3             E C, CR, 41.6, N99 - Delete the second part (i.e. following "due to") of the distractors ES-401 RB-2008-12 4 Rev 1 Form ES-401-9 Attachment 8 Page 4 of 5 OBDI 202 - IOLE Process
 
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
ES-401               RB-2008-12                                                     4                               Rev 1                                       Form ES-401-9
Dist. Partial Job-LinkMinutia#/
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                              8.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation because it provides unnecessary cuing.
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                           Explanation Focus              Dist.         Link          units ward K/A Only because it provides unnecessary cuing. Done 56N   F     3                                                                               S C, CR, 41.7 57M   H     3                                                                               E D, CR, 41.7, N03 - need original question done - minor enhancements to distractors 58N   H     4                                                                               S A, CR, 41.5 A, CR, 43.7 - SRO only because it is testing on the refuel bridge and has no 59B  F    4                                                                    X          U  correlation to actions/activities in the control room. Selected new KA and submitted new question A, CR, 41.10 - It could be argued there is more than one correct answer. It is correct that it is preferred to send the RPV energy to the main condenser to conserve the margin to the HCTL. It is also true that the reason for maintaining the margin to 60N  F    3                              X                                    X          U the HCTL is to preclude exceeding the PCPL. Therefore, distractors a and d could be argued to both be correct. Also, is EOP bases RO level knowledge? Submitted new question 61N   H     2                                                                               S A, CR, 41.4 62B   F     3                                                                               S A, CR, 41.4 N97 63N   F     3                                                                               S C, CR, 41.4 64B   F     4                                                                               S D, CR, 41.4, N00 65N   F     3                                                                               S B, CR, 41.6 66B   F     2                                                             X               U D, CR, 41.5 - Doesnt test ability to use the plant computer. Submitted new question C, CR, 41.10, N99 - Revise the question in the stem to What is required following 67B  F    2    X                                                                        E this evolution? fixed 68B   F     4                                                                               S B, CR, 41.10 69N   F     2                                                                               S C, CR, 41.10 - Need reference. Done C, CR, 41.10 - Distractor d could be considered a subset of distractor c since both of 70N  F    3                              X                                                E  them allow entry into the switchyard. Change distractors a and d to directly enter the switchyard... fixed 71N   F     2                                                                               S B, CR, 41.10 72N   F     2                                                                               S B, CR, 41.12 73B   H     3                                                                               S D, CR, 41.7 N99                                                                     Page 4 of 5                                                    OBDI 202 - IOLE Process
Done 56N F 3             S C, CR, 41.7 57M H 3             E D, CR, 41.7, N03 - need original question done - minor enhancements to distractors 58N H 4             S A, CR, 41.5 59B F 4          X  U A, CR, 43.7 - SRO only because it is testing on the refuel bridge and has no correlation to actions/activities in the control room.
 
Selected new KA and submitted new question 60N F 3    X      X  U A, CR, 41.10 - It could be argued there is more than one correct answer. It is correct that it is preferred to send the RPV energy to the main condenser to conserve the margin to the HCTL. It is also true that the reason for maintaining the margin to the HCTL is to preclude exceeding the PCPL. Therefore, distractors a and d could be argued to both be correct. Also, is EOP bases RO level knowledge?
ES-401                     RB-2008-12                                                       5                           Rev 1                               Form ES-401-9
Submitted new question 61N H 2             S A, CR, 41.4 62B F 3             S A, CR, 41.4 N97 63N F 3             S C, CR, 41.4 64B F 4             S D, CR, 41.4, N00 65N F 3             S B, CR, 41.6 66B F 2         X   U D, CR, 41.5 - Doesn't test ability to use the plant computer. Submitted new question 67B F 2 X            E C, CR, 41.10, N99 - Revise the question in the stem to "What is required following this evolution?"
: 1.      2.        3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.                                          8.
fixed 68B F 4             S B, CR, 41.10 69N F 2             S C, CR, 41.10 - Need reference. Done 70N F 3    X        E C, CR, 41.10 - Distractor d could be considered a subset of distractor c since both of them allow entry into the switchyard. Change distractors a and d to "-directly enter the switchyard..."
Q#   LOK   LOD (F/H)   (1-5) Stem Cues T/F       Cred. Partial Job- Minutia #/ Back-     Q= SRO B/M/N U/E/S                                       Explanation Focus                Dist.         Link          units ward  K/A Only 74B   H       3                                                                                   S B, CR, 41.10 N93 75N   F       2                                                                                   E B, CR, 43.7 - need learning objective done B= 18               F= 37 / 49%           E= 18 RO TOTALS:                M=    6             H= 38 / 51%           U= 8                   Additional Notes:
fixed 71N F 2             S B, CR, 41.10 72N F 2             S B, CR, 41.12 73B H 3             S D, CR, 41.7 N99 ES-401 RB-2008-12 5 Rev 1 Form ES-401-9 Attachment 8 Page 5 of 5 OBDI 202 - IOLE Process
N= 51                                     S= 49 B=                   F=                   E=
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
SRO TOTALS:                M=                   H=                   U=                     Additional Notes:
Dist. Partial Job-LinkMinutia#/
N=                                         S=
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 74B H 3             S B, CR, 41.10 N93 75N F 2             E B, CR, 43.7 - need learning objective done                                   B=   18 F= 37 / 49%
E= 18 M=    6 H= 38 / 51%
U= 8 RO TOTALS:
N=   51 S= 49 Additional Notes:
B=   F= E=   M=   H= U=   SRO TOTALS:
N=     S=   Additional Notes: 


==GENERAL COMMENT==
==GENERAL COMMENT==
S:
S:
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.  
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
: 2. Chief Examiner comments are indicated in blue.
: 2. Chief Examiner comments are indicated in blue.
: 3. Average difficulty is 2.8.  
: 3. Average difficulty is 2.8.
: 4. The answer distribution is: RO / SRO A = 17 (23%)   B = 22 (29%)
: 4. The answer distribution is: RO / SRO A = 17 (23%)
C = 20 (27%) D = 16 (21%)
B = 22 (29%)
: 5. There are zero RO questions with proposed references.  
C = 20 (27%)
 
D = 16 (21%)
ES-401 RB-2008-12 SRO Written Examination Review WorksheetRev 1 Form ES-401-9  Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:
: 5.       There are zero RO questions with proposed references.                                                                             Page 5 of 5                                               OBDI 202 - IOLE Process
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).
$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer
=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Attachment 8 Page 1 of 3 OBDI 202 - IOLE Process  
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other  Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia#/ units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 76N H 3            S D, CR, 43.5 77N H 3          X  UA, CR, 41.10 - Not SRO-only. Entry conditions to major EOP's and RPS trip setpoints is RO knowledge.
Note:  add the word "to" between due and TSV in distracters b & d so they read correctly. submitted new question 78N H 3          X  UB, CR, 41.10 - This is an RO question since there is no procedure selection (all the distractors reference the same procedure) and tests on AOP entry conditions and


specific operator actions. submitted new question 79N H 3             S B, CR, 43.5 80N H 3             E A, CR, 43.5 - minor enhancements made 81N F 3          X  U B, CR, 41.5 - This is an RO question because it tests on operating limits, indications, locations of component controls. Resubmitted new question
ES-401              RB-2008-12                                        SRO Written Examination Review Worksheet Rev 1                                                                Form ES-401-9
: 1.      2.          3. Psychometric Flaws              4. Job Content Flaws        5. Other      6.      7.                                            8.
Q#    LOK    LOD (F/H)  (1-5)  Stem Cues T/F        Cred. Partial Job- Minutia #/ Back-            Q= SRO B/M/N U/E/S                                              Explanation Focus                Dist.            Link            units ward    K/A Only 76N    H      3                                                                                                  S    D, CR, 43.5 A, CR, 41.10 - Not SRO-only. Entry conditions to major EOPs and RPS trip setpoints is RO knowledge.
77N    H      3                                                                                  X              U Note: add the word to between due and TSV in distracters b & d so they read correctly. submitted new question B, CR, 41.10 - This is an RO question since there is no procedure selection (all the 78N    H      3                                                                                  X              U    distractors reference the same procedure) and tests on AOP entry conditions and specific operator actions. submitted new question 79N     H       3                                                                                                 S   B, CR, 43.5 80N     H       3                                                                                                 E   A, CR, 43.5 - minor enhancements made B, CR, 41.5 - This is an RO question because it tests on operating limits, indications, 81N    F      3                                                                                  X              U locations of component controls. Resubmitted new question Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
: 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
: 3. Check the appropriate box if a psychometric flaw is identified:
      $    The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
      $    The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
      $    The answer choices are a collection of unrelated true/false statements.
      $    The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
      $    One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
: 4. Check the appropriate box if a job content error is identified:
      $ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
      $ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
      $ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
      $ The question requires reverse logic or application compared to the job requirements.
: 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
: 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).                                                                                        Page 1 of 3                                                      OBDI 202 - IOLE Process


ES-401 RB-2008-12 2 Rev 1 Form ES-401-9 Attachment 8 Page 2 of 3 OBDI 202 - IOLE Process
ES-401               RB-2008-12                                                     2                               Rev 1                                         Form ES-401-9
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                              8.
Dist. Partial Job-LinkMinutia#/
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                           Explanation Focus              Dist.         Link          units ward K/A Only 82N   H     3                                                                               S D, CR, 43.5 83N   F     3                                                                               E B, CR, 43.2 minor enhancements made 84N   H     3                                                                               E B, CR, 43.5 minor enhancements made 85N   H     3                                                                               E A, OR, 43.5 enhancements made C, OR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. KA requires Predict impact of high supp 86N  H    2    X                                                        X    X          U pool temp and use procedures to correct/control/mitigate. Referenced procedures and eliminated handout 87N   H     3                                                                               E D, CR, 43.5 enhancements made 88N   H     3                                                                               E C, OR, 43.5     minor enhancements made 89N   H     3                                                                               E B, CR, 43.5 enhancements made A, CR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. It is a KA mismatch because it does not 90N  F    2                        X                                    X    X          U  address the predict the impact part of the KA. It is graded as difficulty level 1 because it is intuitive the first action would be to attempt to start the pump that failed to auto start, and distracters c and d dont make sense. submitted new question C, CR, 41.10 - This is an RO question because it tests on systems knowledge and 91N   H    3                                                                    X          U  specific operator actions to be taken in response to a set of conditions. revised question with procedure entry in all distractors 92N  H    4                                                                              S  C, CR, 43.2 93N  H     2                                                                              E  B, CR, 43.5 minor enhancements made D, OR, 41.10/43.1 - Direct lookup with handout - too hard without handout.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 82N H 3             S D, CR, 43.5 83N F 3             E B, CR, 43.2 minor enhancements made 84N H 3             E B, CR, 43.5 minor enhancements made 85N H 3             E A, OR, 43.5 enhancements made 86N H 2 X        X X  U C, OR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditi ons. KA requires "Predict impact of high supp pool temp and use procedures to correct/control/mitigate." Referenced procedures and eliminated handout 87N H 3             E D, CR, 43.5 enhancements made 88N H 3             E C, OR, 43.5   minor enhancements made 89N H 3             E B, CR, 43.5 enhancements made 90N F 2    X      X X  U A, CR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. It is a KA mismatch because it does not address the "predict the impact" part of the KA. It is graded as difficulty level 1 because it is intuitive the first action would be to attempt to start the pump that failed to auto start, and distracters c and d don't make sense. submitted new question 91N H 3           X  UC, CR, 41.10 - This is an RO question because it tests on systems knowledge and
94N  F    3     X                                                                         U submitted new question 95N  F    3                                                                              S C, CR, 43.7 96N  F    2                                                                              S  C, CR, 43.3 97N  F    3                                                                              E  C, CR, 43.4 minor enhancements made 98N  F    3                                                                              E  A, CR, 43.4/5 minor enhancements made 99N  F    3                                                                              S  C, CR, 43.5 100N  F    3                                                                    X          U  C, CR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. revised question with procedure entry in                                                                      Page 2 of 3                                                    OBDI 202 - IOLE Process


specific operator actions to be taken in response to a set of conditions. revised question with procedure entry in all distractors 92N H 4            S C, CR, 43.2 93N H 2            E B, CR, 43.5 minor enhancements made 94N F 3 X            U D, OR, 41.10/43.1 - Direct lookup with handout - too hard without handout.
ES-401                   RB-2008-12                                                       3                           Rev 1                   Form ES-401-9
submitted new question 95N F 3            S C, CR, 43.7 96N F 2            S C, CR, 43.3 97N F 3            E C, CR, 43.4 minor enhancements made 98N F 3            E A, CR, 43.4/5  minor enhancements made 99N F 3            S C, CR, 43.5 100N F 3          X  U C, CR, 41.10 -This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. revised question with procedure entry in ES-401 RB-2008-12 3 Rev 1 Form ES-401-9 Attachment 8 Page 3 of 3 OBDI 202 - IOLE Process
: 1. 2.          3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.                            8.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
Q#   LOK   LOD (F/H) (1-5)   Stem Cues T/F       Cred. Partial Job- Minutia #/ Back-     Q= SRO B/M/N U/E/S                           Explanation Focus                Dist.         Link          units ward  K/A Only all distractors B=                   F=                   E=
Dist. Partial Job-LinkMinutia#/
RO TOTALS:                M=                   H=                   U=                     Additional Notes:
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation all distractors B=   F=   E=   M=   H=   U=   RO TOTALS:
N=                                         S=
N=     S=   Additional Notes:
B= 00               F= 10                 E= 10 SRO TOTALS:                M= 00               H= 15                 U= 8                   Additional Notes:
B=   00 F= 10 E= 10 M=   00 H= 15 U= 8 SRO TOTALS:
N= 25                                     S= 7
N=   25 S= 7 Additional Notes: 


==GENERAL COMMENT==
==GENERAL COMMENT==
S:
S:
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.  
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
: 2. Chief Examiner comments are indicated in blue.
: 2. Chief Examiner comments are indicated in blue.
: 3. Average difficulty is 2.8.  
: 3. Average difficulty is 2.8.
: 4. The answer distribution is:   A = 5 (20%) B = 7 (28%) C = 9 (36%)
: 4. The answer distribution is:
D = 4 (16%)  
A=       5 (20%)
: 5. There one question with proposed references.  
B=       7 (28%)
: 6. CR = Closed Reference OR = Open Reference}}
C=       9 (36%)
D=       4 (16%)
: 5.     There one question with proposed references.
: 6.     CR = Closed Reference       OR = Open Reference                                                                           Page 3 of 3                                  OBDI 202 - IOLE Process}}

Latest revision as of 09:05, 14 November 2019

2008-12-Draft Written Exam Comments
ML090750879
Person / Time
Site: River Bend Entergy icon.png
Issue date: 11/21/2008
From:
NRC Region 4
To:
References
Download: ML090750879 (8)
v • d • e


Text

ES-401 RB-2008-12 RO Written Examination Review Worksheet Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1N F 3 E B, CR, 41.5 Need reference that is not TS bases to be RO level question. Fixed 2N H 3 S B, CR, 41.7 3M F 3 S B, CR, 41.7 Need original question Done 4N H 2 S A, CR, 41.6 5B H 3 S A, CR, 41.7, N97 D, CR, 43.5 - It is SRO only because it requires the candidate to evaluate a given set 6B F 3 X U of plant conditions and make a procedure selection based on this assessment.

Submitted new question 7N H 3 S C, CR, 41.4 8N H 3 S D, CR, 41.4 9N F 2 S D, CR, 41.10 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 2 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 10N F 3 S C, CR, 41.10 11N F 3 S A, CR, 41.7 12N F 3 E B, CR, 41.4 - minor enhancements 13M H 3 S A, CR, 41.10, N03 need original question. done 14N F 4 S B, CR, 41.7 A, CR, 41.8 - The experienced test taker would probably select distractor A if the answer were not known since it is the lowest of the four distractors. Suggest the first 15N F 3 X E distractor be something less than 13 feet. Second, remove the greater than symbol from all four distractors since the minimum level is, in fact, 13 feet. revised 16B H 2 S A, CR, 41.7 17B H 3 S C, CR, 41.1, N99 - Need original question. Done - changed to Bank 18N H 3 S B, CR, 41.11 19N F 3 S B, CR, 41.4 20N H 3 S C, CR, 41.5 21N H 3 S D, CR, 41.4 22N H 3 S A, CR, 41.7 23N H 2 S D, CR, 41.4 24N F 3 X U C, CR, 41.10 - Revise to make RO level knowledge fixed 25N F 2 S D, CR, 41.6 C, CR, 41.11 N07 - Current reading of 100 mr/hr is not at the max normal op value 26B F 2 E as stated in distractor c. fixed 27N F 2 S C, CR, 4.7 28B H 3 E B, CR, 4.7, N00 - Distractor b contains a cue. fixed 29N H 3 E A, CR, 41.7 - Reword stem for clarification done 30N H 3 E D, CR, 41.5 - Distractors dont answer the question. Revised stem 31N F 2 S B, CR, 41.7 32B H 3 S B, CR, 41.7, N97 Page 2 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 3 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 33N H 3 S D, CR, 41.7 34N H 2 S A, CR, 41.7 35N F 2 S B, CR, 41.10 36M H 3 E B, CR, 41.6, N97 - need original question done - minor enhancements A, CR, 41.6, N00 - KA requires procedures to correct/control/mitigate Also, need 37M H 3 X U original question. fixed 38N F 2 S D, CR, 41.6 39B H 3 E D, CR, 41.7 minor enhancements - removed cue from stem 40N F 3 U C, OR, 41.7 Direct lookup with reference - revised, removed ref C, CR, 41.7 - Both G33-F004 in the stem and G33-F054 in the correct distractor C 41N H 3 E have the same noun name. Revise noun names to include RWCU pump suction and discharge valves. fixed 42N H 3 S C, CR, 41.7 43N H 3 S B, CR, 41.3 44N H 2 S D, CR, 41.13 45B F 2 S D, CR, 41.7, N07 46B H 3 E C, CR, 41.4 N97 - Clarify distractors C & D. done 47N H 3 S C, CR, 41.4 48N F 3 X E C, CR, 41.4 - Need to revise distractor wording 49N F 3 S A, CR, 41.4 50N F 3 S B, CR, 41.4 C, CR, 41.7 - The stem asks about changes in reactive load or KVA. KVA is not 51N H 3 X U reactive load; KVAR is reactive load. fixed 52N H 3 E A, CR, 41.4 - KA requires procedures to correct/control/mitigate fixed 53B H 3 S B, CR, 41.4, N04 54N H 3 S B, CR, 41.10 55B H 3 E C, CR, 41.6, N99 - Delete the second part (i.e. following due to) of the distractors Page 3 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 4 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only because it provides unnecessary cuing. Done 56N F 3 S C, CR, 41.7 57M H 3 E D, CR, 41.7, N03 - need original question done - minor enhancements to distractors 58N H 4 S A, CR, 41.5 A, CR, 43.7 - SRO only because it is testing on the refuel bridge and has no 59B F 4 X U correlation to actions/activities in the control room. Selected new KA and submitted new question A, CR, 41.10 - It could be argued there is more than one correct answer. It is correct that it is preferred to send the RPV energy to the main condenser to conserve the margin to the HCTL. It is also true that the reason for maintaining the margin to 60N F 3 X X U the HCTL is to preclude exceeding the PCPL. Therefore, distractors a and d could be argued to both be correct. Also, is EOP bases RO level knowledge? Submitted new question 61N H 2 S A, CR, 41.4 62B F 3 S A, CR, 41.4 N97 63N F 3 S C, CR, 41.4 64B F 4 S D, CR, 41.4, N00 65N F 3 S B, CR, 41.6 66B F 2 X U D, CR, 41.5 - Doesnt test ability to use the plant computer. Submitted new question C, CR, 41.10, N99 - Revise the question in the stem to What is required following 67B F 2 X E this evolution? fixed 68B F 4 S B, CR, 41.10 69N F 2 S C, CR, 41.10 - Need reference. Done C, CR, 41.10 - Distractor d could be considered a subset of distractor c since both of 70N F 3 X E them allow entry into the switchyard. Change distractors a and d to directly enter the switchyard... fixed 71N F 2 S B, CR, 41.10 72N F 2 S B, CR, 41.12 73B H 3 S D, CR, 41.7 N99 Page 4 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 5 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 74B H 3 S B, CR, 41.10 N93 75N F 2 E B, CR, 43.7 - need learning objective done B= 18 F= 37 / 49% E= 18 RO TOTALS: M= 6 H= 38 / 51% U= 8 Additional Notes:

N= 51 S= 49 B= F= E=

SRO TOTALS: M= H= U= Additional Notes:

N= S=

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
2. Chief Examiner comments are indicated in blue.
3. Average difficulty is 2.8.
4. The answer distribution is: RO / SRO A = 17 (23%)

B = 22 (29%)

C = 20 (27%)

D = 16 (21%)

5. There are zero RO questions with proposed references. Page 5 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 SRO Written Examination Review Worksheet Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 76N H 3 S D, CR, 43.5 A, CR, 41.10 - Not SRO-only. Entry conditions to major EOPs and RPS trip setpoints is RO knowledge.

77N H 3 X U Note: add the word to between due and TSV in distracters b & d so they read correctly. submitted new question B, CR, 41.10 - This is an RO question since there is no procedure selection (all the 78N H 3 X U distractors reference the same procedure) and tests on AOP entry conditions and specific operator actions. submitted new question 79N H 3 S B, CR, 43.5 80N H 3 E A, CR, 43.5 - minor enhancements made B, CR, 41.5 - This is an RO question because it tests on operating limits, indications, 81N F 3 X U locations of component controls. Resubmitted new question Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 3 OBDI 202 - IOLE Process

ES-401 RB-2008-12 2 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 82N H 3 S D, CR, 43.5 83N F 3 E B, CR, 43.2 minor enhancements made 84N H 3 E B, CR, 43.5 minor enhancements made 85N H 3 E A, OR, 43.5 enhancements made C, OR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. KA requires Predict impact of high supp 86N H 2 X X X U pool temp and use procedures to correct/control/mitigate. Referenced procedures and eliminated handout 87N H 3 E D, CR, 43.5 enhancements made 88N H 3 E C, OR, 43.5 minor enhancements made 89N H 3 E B, CR, 43.5 enhancements made A, CR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. It is a KA mismatch because it does not 90N F 2 X X X U address the predict the impact part of the KA. It is graded as difficulty level 1 because it is intuitive the first action would be to attempt to start the pump that failed to auto start, and distracters c and d dont make sense. submitted new question C, CR, 41.10 - This is an RO question because it tests on systems knowledge and 91N H 3 X U specific operator actions to be taken in response to a set of conditions. revised question with procedure entry in all distractors 92N H 4 S C, CR, 43.2 93N H 2 E B, CR, 43.5 minor enhancements made D, OR, 41.10/43.1 - Direct lookup with handout - too hard without handout.

94N F 3 X U submitted new question 95N F 3 S C, CR, 43.7 96N F 2 S C, CR, 43.3 97N F 3 E C, CR, 43.4 minor enhancements made 98N F 3 E A, CR, 43.4/5 minor enhancements made 99N F 3 S C, CR, 43.5 100N F 3 X U C, CR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. revised question with procedure entry in Page 2 of 3 OBDI 202 - IOLE Process

ES-401 RB-2008-12 3 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only all distractors B= F= E=

RO TOTALS: M= H= U= Additional Notes:

N= S=

B= 00 F= 10 E= 10 SRO TOTALS: M= 00 H= 15 U= 8 Additional Notes:

N= 25 S= 7

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
2. Chief Examiner comments are indicated in blue.
3. Average difficulty is 2.8.
4. The answer distribution is:

A= 5 (20%)

B= 7 (28%)

C= 9 (36%)

D= 4 (16%)

5. There one question with proposed references.
6. CR = Closed Reference OR = Open Reference Page 3 of 3 OBDI 202 - IOLE Process
Retrieved from "https://kanterella.com/w/index.php?title=ML090750879&oldid=2148065"