ML091490493: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
 
Line 17: Line 17:


=Text=
=Text=
{{#Wiki_filter:ES-401 RO Written Examination Review Worksheet Rev 1 Form ES-401-9 FC-2009-03 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:
{{#Wiki_filter:ES-401               FC-2009-03                                          RO Written Examination Review Worksheet                               Rev 1                                   Form ES-401-9
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).  
: 1.     2.           3. Psychometric Flaws              4. Job Content Flaws        5. Other      6.      7.                                              8.
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
Q#    LOK    LOD (F/H)   (1-5)  Stem Cues T/F        Cred. Partial Job- Minutia #/ Back-            Q= SRO B/M/N U/E/S                                                  Explanation Focus                Dist.           Link            units ward    K/A Only 1      H      2                                                                                          B      E    B/10 A/5 - EOP-00 (SPTA) step 11 has the operators close the block valve. This question assumes operation in EOP-03 (LOCA); however, the crew had to go through EOP-00 2      F      2      X                                                                                  N      U    first. There is no caution in EOP-00 about solid plant conditions and PORV operation. The applicant would correctly assume the PORV block is already closed, and there would be no correct answer. Revised question A/5 - Doesnt address KA - time available for action. Also, what is reference SHB:
$ The answer choices are a collection of unrelated true/false statements.  
3      H      2                                                                            X            B      U    07-15-12? Also, ref states thumb rule is 10%/deg F - is it 10% or 1%. Also, ref is for uncomplicated reactor trip but KA applies to SBLOCA. Replaced question 4      H      3                                                                                          N      E    D/5 - Handout C/7 - Reword to operational event. Plant is in Mode 4. SDC is in operation per procedure XXX, with LPSI pump XX in service. A loss of off-site power occurs.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.  
5      H      2                                                                                          B      E EDGs operate as designed. What action (if any)  (or something similar). Also, STM-15, para 3.19 on page 71 is a better reference. Fixed D/10 - What is the reactor trip criteria? What is the failure? A leak (surge tank level 6      H      3      X                                                                                  B      U 8 and lowering) and/or trashed pump (loud rumbling noise)? Revised question 7      H      3      X                                                                                  N      U    D/14 - Rewrite into bullet format and let applicant determine system response Done Instructions
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
: 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).
: 3. Check the appropriate box if a psychometric flaw is identified:
$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer
      $    The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
      $    The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
Attachment 8 Page 1 of 6 OBDI 202 - IOLE Process
      $    The answer choices are a collection of unrelated true/false statements.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other  Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia#/ units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 1 H 2            B E B/10  2 F 2 X          N U A/5 - EOP-00 (SPTA) step 11 has the operators close the block valve. This question assumes operation in EOP-03 (LOCA); however, the crew had to go through EOP-00 first. There is no caution in EOP-00 about solid plant conditions and PORV operation. The applicant would correctly assume the PORV block is already closed, and there would be no correct answer. Revised question 3 H 2          X  B U A/5 - Doesn't address KA - time available for action. Also, what is reference SHB: 07-15-12?  Also, ref states thumb rule is 10%/deg F - is it 10% or 1%. Also, ref is for uncomplicated reactor trip but KA applies to SBLOCA. Replaced question 4 H 3            N E D/5 - Handout 5 H 2            B E C/7 - Reword to operational event. Plant is in Mode 4. SDC is in operation per procedure XXX, with LPSI pump XX in service. A loss of off-site power occurs.
      $    The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
EDGs operate as designed. What action (if any) - (or something similar). Also, STM-15, para 3.19 on page 71 is a better reference. Fixed 6 H 3 X          B U D/10 - What is the reactor trip criteria?  W hat is the failure? A leak (surge tank level 8" and lowering) and/or trashed pump (loud rumbling noise)?
      $    One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
Revised question 7 H 3 X          N U D/14 - Rewrite into bullet format and let applicant determine system response Done ES-401 2 Rev 1 Form ES-401-9 FC-2009-03 Attachment 8 Page 2 of 6 OBDI 202 - IOLE Process
: 4. Check the appropriate box if a job content error is identified:
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
      $ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
Dist. Partial Job-LinkMinutia#/
      $ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 8 H 2 X           M E D/10 9 F 2            N U B/11 - The references do not support the correct answer or explain why the others are incorrect. Nothing in the lesson plan 7-15-33 or SO-G-105 supports the correct answer. The material states that RM-54 MAY alarm first for small leaks IF blowdown is in service.
      $ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
Revised 10 H 3           N E C/10   11 H 3            N E C/10 - Need ref for the why part of question. Also, need procedure reference in stem. Also, revise stem to remove conclusions. Done 12 H 3           M E B/7 13 H 2           B S D/5 14 F 3            N E B/10 - Rewrite in bullet format as it would be more operationally valid.
      $ The question requires reverse logic or application compared to the job requirements.
Done Also, are FCS RO applicants expected to know AOP bases?   Facility Rep accepts question as acceptable for RO applicants 15 H 3            N E C/10 - If Panel 1 is lost, will the low voltage annunciator also alarm? Yes added to stem  Also, according the AOP reference provided, this is one pushbutton on 1A1-1A3 Aux Power Compartment.
: 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
Revised   16 H 3           N S D/7 17 F 2            B E D/10 - The AOP states that MFPs are stopped to prevent the RCS cooldown following the trip, not overfilling the generators (although overfilling would have the effect of overcooling.) Prefer to follow wording of the AOP. Done 18 H 3           N E B/10 19 F 3           B E A/7 20 H 2           B E A/5 - Handout.
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
21 F 3        X X  N U B/6 - Doesn't address KA - startup termination on SR loss. Backwards logic -asks a question about what is needed prior to s/u yet a s/u is already in progress. New KA 22 H 2           N E B/6 23 F 2          X  N E D/13   24 F 2    X      X  B U B/13 - Doesn't address KA - same as above but for accidental gaseous radwaste release. Distractors C and D are not credible.
: 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
Revised 25 H 3           M E D/7 - Worksheet says Source: NEW - should this be MODIFIED BANK? Yes 26 F 2         X   N U B/5 - Backward logic. Give parameter(s), then choose a procedure.
: 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).                                                                                        Page 1 of 6                                                          OBDI 202 - IOLE Process
Revised 27 H 3           N E A/3 ES-401 3 Rev 1 Form ES-401-9 FC-2009-03 Attachment 8 Page 3 of 6
 
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
ES-401               FC-2009-03                                                    2                                 Rev 1                                   Form ES-401-9
Dist. Partial Job-LinkMinutia#/
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                              8.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 28 H 3           N E C/10 29 H 3           N E D/7 30 H 2           N S C/11 31 F 2           N S B/10 32 H 3 X         X N E B/5 - Stem doesn't state reactor is critical. Fixed 33 F 2            B E C/10 - Question has not been modified significantly. It is a BANK question. Updated Ref for venting NCG's to the PQT or VCT? OI-CH-3 34 H 3           B E A/5 35 F 3 U    U      N E B/10 -The way this question is worded, the operator could assume that the prerequisite checklist is being performed, and the discharge valve for AC-3A is being opened as required by OI-CC-1-CL-A (Page 29 of OI-CC-1). This would make C the correct answer. Add to stem - prerequisites for OI-CC-1 are complete. Fixed 36 H 3           N S B/7 - Ref not adequate to support answer provided 37 H 3           N E D/10 38 H 3            N E C/6 - Stem - provide CR indications of a loss of 125 VDC power and ask impact on DSS. Otherwise cognitive level is LOW. Fixed 39 H 3           N E B/10 - Cognitive level is HIGH. updated 40 H 3          X  B U D/7 - Doesn't address part b of KA - use procedures to correct, control, or mitigate.
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                           Explanation Focus              Dist.        Link          units ward K/A Only 8   H     2     X                                                                   M     E D/10 B/11 - The references do not support the correct answer or explain why the others are incorrect. Nothing in the lesson plan 7-15-33 or SO-G-105 supports the correct 9    F    2                                                                        N    U answer. The material states that RM-54 MAY alarm first for small leaks IF blowdown is in service. Revised 10   H     3                                                                         N     E C/10 C/10 - Need ref for the why part of question. Also, need procedure reference in 11    H    3                                                                        N    E stem. Also, revise stem to remove conclusions. Done 12   H     3                                                                         M     E B/7 13   H     2                                                                         B     S D/5 B/10 - Rewrite in bullet format as it would be more operationally valid. Done Also, 14    F    3                                                                        N    E are FCS RO applicants expected to know AOP bases? Facility Rep accepts question as acceptable for RO applicants C/10 - If Panel 1 is lost, will the low voltage annunciator also alarm? Yes added to 15    H    3                                                                        N    E stem Also, according the AOP reference provided, this is one pushbutton on 1A1-1A3 Aux Power Compartment. Revised 16   H     3                                                                         N     S D/7 D/10 - The AOP states that MFPs are stopped to prevent the RCS cooldown 17    F    2                                                                        B    E following the trip, not overfilling the generators (although overfilling would have the effect of overcooling.) Prefer to follow wording of the AOP. Done 18   H     3                                                                         N     E B/10 19   F     3                                                                         B     E A/7 20   H     2                                                                         B     E A/5 - Handout.
Also, refs do not detail impact of 10 sec drop in voltage.
B/6 - Doesnt address KA - startup termination on SR loss. Backwards logic - asks a 21    F    3                                                          X  X          N    U question about what is needed prior to s/u yet a s/u is already in progress. New KA 22   H     2                                                                         N     E B/6 23   F     2                                                             X         N     E D/13 B/13 - Doesnt address KA - same as above but for accidental gaseous radwaste 24    F    2                        X                                    X          B    U release. Distractors C and D are not credible. Revised 25   H     3                                                                         M     E D/7 - Worksheet says Source: NEW - should this be MODIFIED BANK? Yes 26   F     2                                                           X             N     U B/5 - Backward logic. Give parameter(s), then choose a procedure. Revised 27   H     3                                                                         N     E A/3                                                                     Page 2 of 6                                                    OBDI 202 - IOLE Process
fixed 41 F 2            M E B/4 - Reword stem to read "From where do the CEDM Cooling Fans, VA-2A and VA-2B, take suction? Then remove the phrase "The CEDM cooling fans take (cool) air.." from each of the answers.
 
done 42 H 3           N E C/10 43 F 3           N E C/5   44 H 2          X  N U A/10 - Does not address auto operation of MFW and level programming. This addresses AFW and safety function checks of the EOPs. Also, to meet the safety function, must be feeding the S/Gs - stem states BOP stopped feeding. Rewrote question 45 F 3            N E C/10 - The words "prior to:" in the stem would make the correct answer incorrect.
ES-401               FC-2009-03                                                    3                                 Rev 1                                   Form ES-401-9
The continuous action step (15) of the EOP states that once through core cooling is performed if there is a rise of >5F. Revise the stem.
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                            8.
Fixed 46 H 3 X X  X        N U D/10 - Assuming Step 8 of AOP-31 is performed sequentially, bus 1A4 is deenergized first, and 1A4 loads are started manually after EDG start. Then bus 1A3 is deeneergized and operators ensure loads started after EDG start. Answers C & D ES-401 4  Rev 1 Form ES-401-9 FC-2009-03 Attachment 8 Page 4 of 6 OBDI 202 - IOLE Process
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                         Explanation Focus              Dist.         Link          units ward K/A Only 28   H     3                                                                         N     E C/10 29   H     3                                                                         N     E D/7 30   H     2                                                                         N     S C/11 31   F     2                                                                         N     S B/10 32   H     3     X                                                             X   N     E B/5 - Stem doesnt state reactor is critical. Fixed C/10 - Question has not been modified significantly. It is a BANK question. Updated 33    F    2                                                                        B    E Ref for venting NCGs to the PQT or VCT? OI-CH-3 34   H     3                                                                         B     E A/5 B/10 -The way this question is worded, the operator could assume that the prerequisite checklist is being performed, and the discharge valve for AC-3A is being 35    F    3    U                        U                                          N    E opened as required by OI-CC-1-CL-A (Page 29 of OI-CC-1). This would make C the correct answer. Add to stem - prerequisites for OI-CC-1 are complete. Fixed 36   H     3                                                                         N     S B/7 - Ref not adequate to support answer provided 37   H     3                                                                         N     E D/10 C/6 - Stem - provide CR indications of a loss of 125 VDC power and ask impact on 38    H    3                                                                        N    E DSS. Otherwise cognitive level is LOW. Fixed 39   H     3                                                                         N     E B/10 - Cognitive level is HIGH. updated D/7 - Doesnt address part b of KA - use procedures to correct, control, or mitigate.
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
40    H    3                                                              X          B    U Also, refs do not detail impact of 10 sec drop in voltage. fixed B/4 - Reword stem to read From where do the CEDM Cooling Fans, VA-2A and VA-41    F    2                                                                        M    E 2B, take suction? Then remove the phrase The CEDM cooling fans take (cool) air..
Dist. Partial Job-LinkMinutia#/
from each of the answers. done 42   H     3                                                                         N     E C/10 43   F     3                                                                         N     E C/5 A/10 - Does not address auto operation of MFW and level programming. This addresses AFW and safety function checks of the EOPs. Also, to meet the safety 44    H    2                                                              X          N    U function, must be feeding the S/Gs - stem states BOP stopped feeding. Rewrote question C/10 - The words prior to: in the stem would make the correct answer incorrect.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation state that both buses are deenergized and then loads are started. Reword C & D. Also, eliminate values of 3750 and 430 volts as they could be used to assist in answering Q87 (state "less than minimum bus voltage").
45    F    3                                                                        N    E The continuous action step (15) of the EOP states that once through core cooling is performed if there is a rise of >5F. Revise the stem. Fixed D/10 - Assuming Step 8 of AOP-31 is performed sequentially, bus 1A4 is 46    H    3    X      X          X                                                N    U deenergized first, and 1A4 loads are started manually after EDG start. Then bus 1A3 is deeneergized and operators ensure loads started after EDG start. Answers C & D                                                                     Page 3 of 6                                                  OBDI 202 - IOLE Process
Fixed 47 F 2          X  B U B/10 - Does not address 2 nd part of the KA. Fixed 48 F 2           N E D/7 49 H 2           N E A/5 50 H 3        X  M U A/11 - Backward logic (see 1021 Appendix B). Give the alarm and have applicant determine system response. New KA 51 H 3           B E C/11 - 52 F 3         X   M U B/7 - Backward logic. Rewrote question 53 H 3           N E B/7 - Ref provided doesn't contain info on sequencer starting AC-10C. ref provided 54 H 3           N E D/7 55 F 2           N E B/9 56 H 3           M E C/10 57 F 2           B E A/9 58 H 3           N E C/10 59 H 3            M E D/5 - Does the reactor have to be > 60% for the valves to stroke full open? Yes If so, add initial power to stem. Done 60 F 2            N E B/7 - Ref provided doesn't contain info on 43/FW switch or trip signals to FW-8A.
 
Ref provided 61 H 3          X  N U D/7 - Doesn't address 2 nd part of KA - use procedures to correct, control -. Fixed 62 F 2          X  B U D/13 - Doesn't address the KA - loss or malfunction of Rad Monitor on Liquid Radwaste System (this is an admin type question). Also, this is not a significant modification. SOURCE is BANK. New KA and question 63 H 3           N E B/13 64 F 2           N E D/6 65 F 3    X        N U A/10 - Distractors B and C are not credible. Also, this is not a significant modification. SOURCE is BANK. New KA 66 F 2           N E A/10 67 F 3   X       N E C/10 68 F 2           B E D/10 ES-401 5 Rev 1 Form ES-401-9 FC-2009-03 Attachment 8 Page 5 of 6 OBDI 202 - IOLE Process
ES-401               FC-2009-03                                                     4                                 Rev 1                                  Form ES-401-9
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                            8.
Dist. Partial Job-LinkMinutia#/
Q#  LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                       Explanation Focus              Dist.         Link          units ward K/A Only state that both buses are deenergized and then loads are started. Reword C & D.
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 69 H 2            N E B/5 70 H 3   X       N E B/5 - Add power statement to stem.
Also, eliminate values of 3750 and 430 volts as they could be used to assist in answering Q87 (state less than minimum bus voltage). Fixed nd 47   F     2                                                             X         B     U B/10 - Does not address 2 part of the KA. Fixed 48   F     2                                                                         N     E D/7 49   H     2                                                                         N     E A/5 A/11 - Backward logic (see 1021 Appendix B). Give the alarm and have applicant 50    H    3                                                          X            M    U determine system response. New KA 51   H     3                                                                         B     E C/11 -
Done 71 F 2   X       N U D/12 - Distractor A is not credible.
52   F     3                                                           X             M     U B/7 - Backward logic. Rewrote question 53   H     3                                                                         N     E B/7 - Ref provided doesnt contain info on sequencer starting AC-10C. ref provided 54   H     3                                                                         N     E D/7 55   F     2                                                                         N     E B/9 56   H     3                                                                         M     E C/10 57   F     2                                                                         B     E A/9 58   H     3                                                                         N     E C/10 D/5 - Does the reactor have to be > 60% for the valves to stroke full open? Yes If 59    H    3                                                                        M    E so, add initial power to stem. Done B/7 - Ref provided doesnt contain info on 43/FW switch or trip signals to FW-8A.
Fixed 72 F 3           B E A/11 - This is not a significant modification. SOURCE is BANK.
60    F    2                                                                        N    E Ref provided nd 61   H     3                                                             X         N     U D/7 - Doesnt address 2 part of KA - use procedures to correct, control . Fixed D/13 - Doesnt address the KA - loss or malfunction of Rad Monitor on Liquid 62    F    2                                                              X          B    U Radwaste System (this is an admin type question). Also, this is not a significant modification. SOURCE is BANK. New KA and question 63   H     3                                                                         N     E B/13 64   F     2                                                                         N     E D/6 A/10 - Distractors B and C are not credible. Also, this is not a significant modification.
Fixed 73 F 2           N E A/13 74 F 2           N E B/10 75 H 3            N E B/6 - The training manual states that the zero power mode bypass automatically removed when power is above 10-4%. The answers B and D state the Rx would trip when power reaches 10-4%. This is contradictory. Change B and D to read "exceeds 1 x 10-4%."
65    F    3                        X                                                N    U SOURCE is BANK. New KA 66   F     2                                                                         N     E A/10 67   F     3                       X                                               N     E C/10 68   F     2                                                                         B     E D/10                                                                     Page 4 of 6                                                OBDI 202 - IOLE Process
Fixed B= 18 (24%) F=31 (42%) E=52 (69%) M= 10 (13%) H=44 (58%) U=18 (24%) N= 47 (63%) S=5 (7%) RO TOTALS:
 
Additional Notes: 1. Any performance data available on Bank questions? 2. DRAFT submittal should include explanation of why answers are correct and why distractors are incorrect but credible. 3. Average difficulty is lower than normal (2.57 vs 2.8).
ES-401                   FC-2009-03                                                        5                                 Rev 1                             Form ES-401-9
: 1.      2.      3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.                                              8.
Q#   LOK     LOD (F/H)   (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-       Q= SRO B/M/N U/E/S                                         Explanation Focus              Dist.         Link          units ward    K/A Only 69    H      2                                                                            N     E B/5 70     H       3                       X                                                   N     E B/5 - Add power statement to stem. Done 71     F       2                       X                                                   N     U D/12 - Distractor A is not credible. Fixed 72     F       3                                                                           B     E A/11 - This is not a significant modification. SOURCE is BANK. Fixed 73     F       2                                                                           N     E A/13 74     F       2                                                                           N     E B/10 B/6 - The training manual states that the zero power mode bypass automatically removed when power is above 10-4%. The answers B and D state the Rx would trip 75    H      3                                                                            N    E when power reaches 10-4%. This is contradictory. Change B and D to read exceeds 1 x 10-4%. Fixed B= 18 (24%)         F= 31 (42%)           E= 52 (69%)             Additional Notes:
M= 10 (13%)         H= 44 (58%)           U= 18 (24%)             1. Any performance data available on Bank questions?
RO TOTALS:                                                                                2. DRAFT submittal should include explanation of why answers are correct and why N= 47 (63%)                               S= 5 (7%)
distractors are incorrect but credible.
: 3. Average difficulty is lower than normal (2.57 vs 2.8).


==GENERAL COMMENT==
==GENERAL COMMENT==
Line 67: Line 73:
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
: 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
: 2. Chief Examiner comments are indicated in blue.
: 2. Chief Examiner comments are indicated in blue.
: 3. Average difficulty is 2.57 on the RO exam.
: 3. Average difficulty is 2.57 on the RO exam.
: 4. The 10CFR55.41 distribution is: 41.1 =   41.11 = 5 41.2 =   41.12 = 1 41.3 = 1 41.13 = 5 41.4 = 1 41.14 = 1 41.5 = 13 41.6 = 5 41.7 = 14 41.8 =
: 4. The 10CFR55.41 distribution is:
41.9 = 2 41.10 = 27
41.1 =                 41.11 = 5 41.2 =                 41.12 = 1 41.3 = 1               41.13 = 5 41.4 = 1               41.14 = 1 41.5 = 13 41.6 = 5 41.7 = 14 41.8 =
: 5. The answer distribution is: RO A = 14 (19%) B = 25 (33%)
41.9 = 2 41.10 = 27
ES-401 6 Rev 1 Form ES-401-9 FC-2009-03 Attachment 8 Page 6 of 6 OBDI 202 - IOLE Process
: 5.       The answer distribution is: RO A = 14 (19%)
: 6. TC = 15 (20%) D = 21 (28%)
B = 25 (33%)                                                                           Page 5 of 6                                                  OBDI 202 - IOLE Process
here are 2  questions with handouts (Q4, Q20).  
 
ES-401             FC-2009-03                          6       Rev 1           Form ES-401-9 C = 15 (20%)
D = 21 (28%)
: 6. There are 2 questions with handouts (Q4, Q20).                                          Page 6 of 6       OBDI 202 - IOLE Process
 
ES-401              FC-2009-03                                        SRO Written Examination Review Worksheet                                Rev 1                                  Form ES-401-9
: 1.      2.          3. Psychometric Flaws              4. Job Content Flaws        5. Other      6.     7.                                            8.
Q#    LOK    LOD (F/H)  (1-5)  Stem Cues T/F        Cred. Partial Job- Minutia #/ Back-            Q= SRO B/M/N U/E/S                                              Explanation Focus                Dist.            Link            units ward    K/A Only 76      F      3                                                                                          N      S    A/2 - Note: second question on PDIL (Q20)
C/5 - How credible are distractors A & B - are there any EOP or EOP Attachment 77      H      3                                                                                          N      E    transitions to the TDBs? Changed to EOP-AOP Attachment 2 How credible is distractor D - what is PRC? Plant Review Committee - acceptable C/5 - Doesnt address loss of RC flow as identified in the KA - cause of RCP failure.
78      H      3                                                                            X            N      U Modified stem to ensure operationally valid for Plant Operator response 79      H      3                                                                                          N      E    A/5 80      H      3      U                                                                                  N      E    D/5 C/5 - RO knowledge. Also, doesnt address KA for ability to determine and interpret.
81      F      2                                                                            X      X      N      U No assessment of plant conditions to determine course of action. Revised question D/5 - Doesnt address KA - using incore/excore instrumentation, incore or loop 82      H      3                                                                            X            N      U temperature measurements Revised question Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
: 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
: 3. Check the appropriate box if a psychometric flaw is identified:
      $    The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
      $    The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
      $    The answer choices are a collection of unrelated true/false statements.
      $    The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
      $    One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
: 4. Check the appropriate box if a job content error is identified:
      $ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
      $ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
      $ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
      $ The question requires reverse logic or application compared to the job requirements.
: 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
: 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).                                                                                       Page 1 of 3                                                        OBDI 202 - IOLE Process


ES-401 SRO Written Examination Review Worksheet Rev 1 Form ES-401-9 FC-2009-03 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:
ES-401               FC-2009-03                                                    2                                  Rev 1                                 Form ES-401-9
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).
: 1. 2.       3. Psychometric Flaws          4. Job Content Flaws  5. Other  6. 7.                                             8.
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
Q# LOK   LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO B/M/N U/E/S                                         Explanation Focus              Dist.        Link          units ward K/A Only 83    H    3                                                                        B    E C/2 C/7 - Having all answers contain within one hour adds no discriminatory value.
$ The answer choices are a collection of unrelated true/false statements.
84    H    3                                                                        N    E Moved to stem 85    H    3                                                                        N    E A/5 86    H    3                                                                         N     S A/5 A/5 - AOP-32 not credible distractor since loss of bus is required for entry. Changed 87    H    3                                                                        N    E to AOP-31 Also, would operator expect to receive "SHUTDOWN COOLING FLOW HI-LO" alarm (CB-1,2,3; A2)? Yes - add to stem 88    H     4                                                                        N     E C/5 B/5 - Distractor C - is the FC not already in 3 element control? Yes - change to 89    H    3                                                                        N    E single element Distractor D - Feed Reg Bypass Valve already in auto Changed to DCS controller A/5 - Any annuciator alarms expected for this condition? Yes - add to stem. How would the operators in the CR know of the condition? annuciators Any entry conditions for AOP-18? Yes Distractor C - credible given stem identifies reduced 90    H     3                       X                                               N     U flow in RW system only? Changed to AOP-18 Consider two Distractors with AOP-18 with different actions (section IV and section V). Done Add to stem which RW pumps are running? yes B/2 - Is the immovable CEA misaligned from its group (info needed in stem)? Yes 91    H    3                                                                        N    E Are applicants expected to memorize 6-8 hr actions (KA states entry-level conditions)? Determined by Facility Rep to be okay.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
B/5 - Add 100% relative humidity to stem since it is a pre-req to correcting the H2 92    F    3                                                                        N    E conc. Done 93    H     3                                                                         N     S D/5 94    H    2                                                                        N    S D/2 - Handout 95    F    2                                                                        N    E A/7 96    H     2                                                                        N     E D/2 - Add to stem that all SITs contain refueling boron concentration. Done D/2 - Add to stem  has just been declared inoperable by the Shift Manager.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:
97    H    3                                                                        N     E Done. Why is Distractor B credible? Changed to 20,000 lbm/hr to match procedure C/4 - RO system level knowledge. Also not operationally valid since the procedure 98    H    2    X                              X                              X    M    U directs alignment to preferred power supply if available. Marginal KA match. Revised question.                                                                      Page 2 of 3                                                 OBDI 202 - IOLE Process
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).
$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer
=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Attachment 8 Page 1 of 3 OBDI 202 - IOLE Process
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other  Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia#/ units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 76 F 3           N S A/2 - Note: second question on PDIL (Q20) 77 H 3            N E C/5 - How credible are distractors A & B - are there any EOP or EOP Attachment transitions to the TDBs? Changed to EOP-AOP Attachment 2  How credible is distractor D - what is PRC? Plant Review Committee - acceptable 78 H 3         X N U C/5 - Doesn't address loss of RC flow as identified in the KA - cause of RCP failure. Modified stem to ensure operationally valid for Plant Operator response 79 H 3           N E A/5 80 H 3 U          N E D/5 81 F 2         X X N U C/5 - RO knowledge. Also, doesn't address KA for ability to determine and interpret.
No assessment of plant conditions to determine course of action. Revised question 82 H 3         X  N U D/5 - Doesn't address KA - using incore/excore instrumentation, incore or loop temperature measurements Revised question


ES-401 2  Rev 1 Form ES-401-9 FC-2009-03 Attachment 8 Page 2 of 3 OBDI 202 - IOLE Process
ES-401                     FC-2009-03                                                        3                              Rev 1                                   Form ES-401-9
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
: 1.      2.        3. Psychometric Flaws         4. Job Content Flaws     5. Other   6. 7.                                          8.
Dist. Partial Job-LinkMinutia#/
Q#   LOK     LOD (F/H)   (1-5) Stem Cues T/F       Cred. Partial Job- Minutia #/ Back-       Q= SRO B/M/N U/E/S                                       Explanation Focus               Dist.         Link          units ward    K/A Only 99     F       2                                                                             B     S B/5 B/5 - Add procedure in effect to stem. Done. Is the correct terminology EOP floating 100    H      4                                                                            N    E step or EOP/AOP floating step in A, B, C, D? Fixed B= 2                 F= 5 (20%)           E= 15 (60%)
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 83 H 3            B E C/2 84 H 3            N E C/7 - Having all answers contain "within one hour" adds no discriminatory value.
Additional Notes: Higher number of cognitive questions than normal - supported by SRO TOTALS:              M= 1                 H= 20 (80%)           U= 5 (20%)             random KA selection.
Moved to stem 85 H 3            N E A/5 86 H 3            N S A/5  87 H 3            N E A/5 - AOP-32 not credible distractor since loss of bus is required for entry. Changed to AOP-31 Also, would operator expect to receive "SHUTDOWN COOLING FLOW HI-LO" alarm (CB-1,2,3; A2)? Yes - add to stem 88 H 4            N E C/5 89 H 3            N E B/5 - Distractor C - is the FC not already in 3 element control? Yes - change to single element  Distractor D - Feed Reg Bypass Valve already in auto  Changed to DCS controller 90 H 3    X        N U A/5 - Any annuciator alarms expected for this condition?  Yes - add to stem. How would the operators in the CR know of the condition?
N= 22                                     S= 5 (20%)
annuciators Any entry conditions for AOP-18?  Yes  Distractor C - credible given stem identifies reduced flow in RW system only? Changed to AOP-18  Consider two Distractors with AOP-18 with different actions (s ection IV and section V). Done Add to stem which RW pumps are running?
yes 91 H 3            N E B/2 -  Is the immovable CEA misaligned from its group (info needed in stem)? Yes Are applicants expected to memorize 6-8 hr actions (KA states entry-level conditions)?  Determined by Facility Rep to be okay.
92 F 3            N E B/5 - Add "100% relative humidity" to stem since it is a pre-req to correcting the H2 conc. Done 93 H 3            N S D/5  94 H 2            N S D/2 - Handout 95 F 2            N E A/7 96 H 2            N E D/2 - Add to stem that all SIT's contain refueling boron concentration.
Done 97 H 3            N E D/2 - Add to stem " - has just been declared inoperable by the Shift Manager."
Done. Why is Distractor B credible?  Changed to 20,000 lbm/hr to match procedure 98 H 2 X    X    X M U C/4 - RO system level knowledge. Also not operationally valid since the procedure directs alignment to preferred power supply if available. Marginal KA match.
Revised question.
ES-401 3  Rev 1 Form ES-401-9 FC-2009-03 Attachment 8 Page 3 of 3 OBDI 202 - IOLE Process
: 3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 99 F 2           B S B/5 100 H 4            N E B/5 - Add procedure in effect to stem.
Done. Is the correct terminology EOP floating step or EOP/AOP floating step in A, B, C, D? Fixed B= 2 F=5 (20%) E=15 (60%) M= 1 H=20 (80%)
U=5 (20%) SRO TOTALS:
N= 22 S=5 (20%) Additional Notes:  Higher number of cognitive questions than normal - supported by random KA selection.


==GENERAL COMMENT==
==GENERAL COMMENT==
Line 114: Line 127:
: 2. Chief Examiner comments are indicated in blue.
: 2. Chief Examiner comments are indicated in blue.
: 3. Average difficulty is 2.84    on the SRO exam.
: 3. Average difficulty is 2.84    on the SRO exam.
: 4. The 10CFR55.41/43 distribution is: RO / SRO 43.1 =   43.2 = 6 43.3 =   43.4 = 1 43.5 = 16 43.6 =   43.7 = 2
: 4. The 10CFR55.41/43 distribution is: RO / SRO 43.1 =
: 5. The answer distribution is: SRO A = 7 (28%) B = 5 (20%) C = 7 (28%) D = 6 (24%)
43.2 = 6 43.3 =
: 6. There is   1    question with HANDOUT provided (Q94).}}
43.4 = 1 43.5 = 16 43.6 =
43.7 = 2
: 5.       The answer distribution is: SRO A=   7 (28%)
B=   5 (20%)
C=   7 (28%)
D=   6 (24%)
: 6.       There is   1    question with HANDOUT provided (Q94).                                                                             Page 3 of 3                                                OBDI 202 - IOLE Process}}

Latest revision as of 06:16, 14 November 2019

2009-03 - Draft Written Exam Comments
ML091490493
Person / Time
Site: Fort Calhoun Omaha Public Power District icon.png
Issue date: 03/27/2009
From: Brian Larson
Operations Branch IV
To:
Omaha Public Power District
References
50-285/09-301, ES-401, ES-401-9 50-285/09-301
Download: ML091490493 (9)
v • d • e


Text

ES-401 FC-2009-03 RO Written Examination Review Worksheet Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 2 B E B/10 A/5 - EOP-00 (SPTA) step 11 has the operators close the block valve. This question assumes operation in EOP-03 (LOCA); however, the crew had to go through EOP-00 2 F 2 X N U first. There is no caution in EOP-00 about solid plant conditions and PORV operation. The applicant would correctly assume the PORV block is already closed, and there would be no correct answer. Revised question A/5 - Doesnt address KA - time available for action. Also, what is reference SHB:

3 H 2 X B U 07-15-12? Also, ref states thumb rule is 10%/deg F - is it 10% or 1%. Also, ref is for uncomplicated reactor trip but KA applies to SBLOCA. Replaced question 4 H 3 N E D/5 - Handout C/7 - Reword to operational event. Plant is in Mode 4. SDC is in operation per procedure XXX, with LPSI pump XX in service. A loss of off-site power occurs.

5 H 2 B E EDGs operate as designed. What action (if any) (or something similar). Also, STM-15, para 3.19 on page 71 is a better reference. Fixed D/10 - What is the reactor trip criteria? What is the failure? A leak (surge tank level 6 H 3 X B U 8 and lowering) and/or trashed pump (loud rumbling noise)? Revised question 7 H 3 X N U D/14 - Rewrite into bullet format and let applicant determine system response Done Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 6 OBDI 202 - IOLE Process

ES-401 FC-2009-03 2 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 8 H 2 X M E D/10 B/11 - The references do not support the correct answer or explain why the others are incorrect. Nothing in the lesson plan 7-15-33 or SO-G-105 supports the correct 9 F 2 N U answer. The material states that RM-54 MAY alarm first for small leaks IF blowdown is in service. Revised 10 H 3 N E C/10 C/10 - Need ref for the why part of question. Also, need procedure reference in 11 H 3 N E stem. Also, revise stem to remove conclusions. Done 12 H 3 M E B/7 13 H 2 B S D/5 B/10 - Rewrite in bullet format as it would be more operationally valid. Done Also, 14 F 3 N E are FCS RO applicants expected to know AOP bases? Facility Rep accepts question as acceptable for RO applicants C/10 - If Panel 1 is lost, will the low voltage annunciator also alarm? Yes added to 15 H 3 N E stem Also, according the AOP reference provided, this is one pushbutton on 1A1-1A3 Aux Power Compartment. Revised 16 H 3 N S D/7 D/10 - The AOP states that MFPs are stopped to prevent the RCS cooldown 17 F 2 B E following the trip, not overfilling the generators (although overfilling would have the effect of overcooling.) Prefer to follow wording of the AOP. Done 18 H 3 N E B/10 19 F 3 B E A/7 20 H 2 B E A/5 - Handout.

B/6 - Doesnt address KA - startup termination on SR loss. Backwards logic - asks a 21 F 3 X X N U question about what is needed prior to s/u yet a s/u is already in progress. New KA 22 H 2 N E B/6 23 F 2 X N E D/13 B/13 - Doesnt address KA - same as above but for accidental gaseous radwaste 24 F 2 X X B U release. Distractors C and D are not credible. Revised 25 H 3 M E D/7 - Worksheet says Source: NEW - should this be MODIFIED BANK? Yes 26 F 2 X N U B/5 - Backward logic. Give parameter(s), then choose a procedure. Revised 27 H 3 N E A/3 Page 2 of 6 OBDI 202 - IOLE Process

ES-401 FC-2009-03 3 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 28 H 3 N E C/10 29 H 3 N E D/7 30 H 2 N S C/11 31 F 2 N S B/10 32 H 3 X X N E B/5 - Stem doesnt state reactor is critical. Fixed C/10 - Question has not been modified significantly. It is a BANK question. Updated 33 F 2 B E Ref for venting NCGs to the PQT or VCT? OI-CH-3 34 H 3 B E A/5 B/10 -The way this question is worded, the operator could assume that the prerequisite checklist is being performed, and the discharge valve for AC-3A is being 35 F 3 U U N E opened as required by OI-CC-1-CL-A (Page 29 of OI-CC-1). This would make C the correct answer. Add to stem - prerequisites for OI-CC-1 are complete. Fixed 36 H 3 N S B/7 - Ref not adequate to support answer provided 37 H 3 N E D/10 C/6 - Stem - provide CR indications of a loss of 125 VDC power and ask impact on 38 H 3 N E DSS. Otherwise cognitive level is LOW. Fixed 39 H 3 N E B/10 - Cognitive level is HIGH. updated D/7 - Doesnt address part b of KA - use procedures to correct, control, or mitigate.

40 H 3 X B U Also, refs do not detail impact of 10 sec drop in voltage. fixed B/4 - Reword stem to read From where do the CEDM Cooling Fans, VA-2A and VA-41 F 2 M E 2B, take suction? Then remove the phrase The CEDM cooling fans take (cool) air..

from each of the answers. done 42 H 3 N E C/10 43 F 3 N E C/5 A/10 - Does not address auto operation of MFW and level programming. This addresses AFW and safety function checks of the EOPs. Also, to meet the safety 44 H 2 X N U function, must be feeding the S/Gs - stem states BOP stopped feeding. Rewrote question C/10 - The words prior to: in the stem would make the correct answer incorrect.

45 F 3 N E The continuous action step (15) of the EOP states that once through core cooling is performed if there is a rise of >5F. Revise the stem. Fixed D/10 - Assuming Step 8 of AOP-31 is performed sequentially, bus 1A4 is 46 H 3 X X X N U deenergized first, and 1A4 loads are started manually after EDG start. Then bus 1A3 is deeneergized and operators ensure loads started after EDG start. Answers C & D Page 3 of 6 OBDI 202 - IOLE Process

ES-401 FC-2009-03 4 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only state that both buses are deenergized and then loads are started. Reword C & D.

Also, eliminate values of 3750 and 430 volts as they could be used to assist in answering Q87 (state less than minimum bus voltage). Fixed nd 47 F 2 X B U B/10 - Does not address 2 part of the KA. Fixed 48 F 2 N E D/7 49 H 2 N E A/5 A/11 - Backward logic (see 1021 Appendix B). Give the alarm and have applicant 50 H 3 X M U determine system response. New KA 51 H 3 B E C/11 -

52 F 3 X M U B/7 - Backward logic. Rewrote question 53 H 3 N E B/7 - Ref provided doesnt contain info on sequencer starting AC-10C. ref provided 54 H 3 N E D/7 55 F 2 N E B/9 56 H 3 M E C/10 57 F 2 B E A/9 58 H 3 N E C/10 D/5 - Does the reactor have to be > 60% for the valves to stroke full open? Yes If 59 H 3 M E so, add initial power to stem. Done B/7 - Ref provided doesnt contain info on 43/FW switch or trip signals to FW-8A.

60 F 2 N E Ref provided nd 61 H 3 X N U D/7 - Doesnt address 2 part of KA - use procedures to correct, control . Fixed D/13 - Doesnt address the KA - loss or malfunction of Rad Monitor on Liquid 62 F 2 X B U Radwaste System (this is an admin type question). Also, this is not a significant modification. SOURCE is BANK. New KA and question 63 H 3 N E B/13 64 F 2 N E D/6 A/10 - Distractors B and C are not credible. Also, this is not a significant modification.

65 F 3 X N U SOURCE is BANK. New KA 66 F 2 N E A/10 67 F 3 X N E C/10 68 F 2 B E D/10 Page 4 of 6 OBDI 202 - IOLE Process

ES-401 FC-2009-03 5 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 69 H 2 N E B/5 70 H 3 X N E B/5 - Add power statement to stem. Done 71 F 2 X N U D/12 - Distractor A is not credible. Fixed 72 F 3 B E A/11 - This is not a significant modification. SOURCE is BANK. Fixed 73 F 2 N E A/13 74 F 2 N E B/10 B/6 - The training manual states that the zero power mode bypass automatically removed when power is above 10-4%. The answers B and D state the Rx would trip 75 H 3 N E when power reaches 10-4%. This is contradictory. Change B and D to read exceeds 1 x 10-4%. Fixed B= 18 (24%) F= 31 (42%) E= 52 (69%) Additional Notes:

M= 10 (13%) H= 44 (58%) U= 18 (24%) 1. Any performance data available on Bank questions?

RO TOTALS: 2. DRAFT submittal should include explanation of why answers are correct and why N= 47 (63%) S= 5 (7%)

distractors are incorrect but credible.

3. Average difficulty is lower than normal (2.57 vs 2.8).

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
2. Chief Examiner comments are indicated in blue.
3. Average difficulty is 2.57 on the RO exam.
4. The 10CFR55.41 distribution is:

41.1 = 41.11 = 5 41.2 = 41.12 = 1 41.3 = 1 41.13 = 5 41.4 = 1 41.14 = 1 41.5 = 13 41.6 = 5 41.7 = 14 41.8 =

41.9 = 2 41.10 = 27

5. The answer distribution is: RO A = 14 (19%)

B = 25 (33%) Page 5 of 6 OBDI 202 - IOLE Process

ES-401 FC-2009-03 6 Rev 1 Form ES-401-9 C = 15 (20%)

D = 21 (28%)

6. There are 2 questions with handouts (Q4, Q20). Page 6 of 6 OBDI 202 - IOLE Process

ES-401 FC-2009-03 SRO Written Examination Review Worksheet Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 76 F 3 N S A/2 - Note: second question on PDIL (Q20)

C/5 - How credible are distractors A & B - are there any EOP or EOP Attachment 77 H 3 N E transitions to the TDBs? Changed to EOP-AOP Attachment 2 How credible is distractor D - what is PRC? Plant Review Committee - acceptable C/5 - Doesnt address loss of RC flow as identified in the KA - cause of RCP failure.

78 H 3 X N U Modified stem to ensure operationally valid for Plant Operator response 79 H 3 N E A/5 80 H 3 U N E D/5 C/5 - RO knowledge. Also, doesnt address KA for ability to determine and interpret.

81 F 2 X X N U No assessment of plant conditions to determine course of action. Revised question D/5 - Doesnt address KA - using incore/excore instrumentation, incore or loop 82 H 3 X N U temperature measurements Revised question Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 3 OBDI 202 - IOLE Process

ES-401 FC-2009-03 2 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 83 H 3 B E C/2 C/7 - Having all answers contain within one hour adds no discriminatory value.

84 H 3 N E Moved to stem 85 H 3 N E A/5 86 H 3 N S A/5 A/5 - AOP-32 not credible distractor since loss of bus is required for entry. Changed 87 H 3 N E to AOP-31 Also, would operator expect to receive "SHUTDOWN COOLING FLOW HI-LO" alarm (CB-1,2,3; A2)? Yes - add to stem 88 H 4 N E C/5 B/5 - Distractor C - is the FC not already in 3 element control? Yes - change to 89 H 3 N E single element Distractor D - Feed Reg Bypass Valve already in auto Changed to DCS controller A/5 - Any annuciator alarms expected for this condition? Yes - add to stem. How would the operators in the CR know of the condition? annuciators Any entry conditions for AOP-18? Yes Distractor C - credible given stem identifies reduced 90 H 3 X N U flow in RW system only? Changed to AOP-18 Consider two Distractors with AOP-18 with different actions (section IV and section V). Done Add to stem which RW pumps are running? yes B/2 - Is the immovable CEA misaligned from its group (info needed in stem)? Yes 91 H 3 N E Are applicants expected to memorize 6-8 hr actions (KA states entry-level conditions)? Determined by Facility Rep to be okay.

B/5 - Add 100% relative humidity to stem since it is a pre-req to correcting the H2 92 F 3 N E conc. Done 93 H 3 N S D/5 94 H 2 N S D/2 - Handout 95 F 2 N E A/7 96 H 2 N E D/2 - Add to stem that all SITs contain refueling boron concentration. Done D/2 - Add to stem has just been declared inoperable by the Shift Manager.

97 H 3 N E Done. Why is Distractor B credible? Changed to 20,000 lbm/hr to match procedure C/4 - RO system level knowledge. Also not operationally valid since the procedure 98 H 2 X X X M U directs alignment to preferred power supply if available. Marginal KA match. Revised question. Page 2 of 3 OBDI 202 - IOLE Process

ES-401 FC-2009-03 3 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 99 F 2 B S B/5 B/5 - Add procedure in effect to stem. Done. Is the correct terminology EOP floating 100 H 4 N E step or EOP/AOP floating step in A, B, C, D? Fixed B= 2 F= 5 (20%) E= 15 (60%)

Additional Notes: Higher number of cognitive questions than normal - supported by SRO TOTALS: M= 1 H= 20 (80%) U= 5 (20%) random KA selection.

N= 22 S= 5 (20%)

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
2. Chief Examiner comments are indicated in blue.
3. Average difficulty is 2.84 on the SRO exam.
4. The 10CFR55.41/43 distribution is: RO / SRO 43.1 =

43.2 = 6 43.3 =

43.4 = 1 43.5 = 16 43.6 =

43.7 = 2

5. The answer distribution is: SRO A= 7 (28%)

B= 5 (20%)

C= 7 (28%)

D= 6 (24%)

6. There is 1 question with HANDOUT provided (Q94). Page 3 of 3 OBDI 202 - IOLE Process
Retrieved from "https://kanterella.com/w/index.php?title=ML091490493&oldid=2145096"