ML051050532
| ML051050532 | |
| Person / Time | |
|---|---|
| Site: | Callaway  |
| Issue date: | 03/14/2004 |
| From: | Gody A Operations Branch IV |
| To: | Randolph G Union Electric Co |
| References | |
| 50-483/04-301 50-483/04-301 | |
| Download: ML051050532 (9) | |
Text
ES-401 CW-03-2004 Written Examination Form ES-401-8 Review Worksheet
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 3 - S 2 H 3 - S 3 H 3 X - U both A and B seem to answer the question. Will accept OK.
4 H 3 X X U both A and C answer the question; doesnt test knowledge of REASONS for sequence of events... per K/A. Will accept, K/A meets OR aspect.
5 H 3 X U question doesnt test ability to locate and operate controls per K/A. K/A is 2.1.20 ,
event based O/K 6 H 3 - E changes from parent are not significant enough to consider this a modified question; 7 H 3 - S 8 H 3 - S 9 H 3 X U change stem from ...(SLIS), steam flow INCREASED on all four... to ...(SLIS),
indicated steam flow INCREASED on all four...; question doesnt test high-energy steam break considerations per K/A. Change made to stem, K/A is OK Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
@ The answer choices are a collection of unrelated true/false statements.
@ One or more distractors is not credible.
@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
@ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
32 of 33 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 10 H 3 - S 11 H 4 - E changes from parent are not significant enough to consider this a modified question; appears to be an SRO-only level question; Change was not significant, has RO learning obj, O/K.
12 H 3 - S 13 H 3 - S 14 H 3 - E changes from parent are not significant enough to consider this a modified question; Change was not significant 15 F 3 - E changes from parent are not significant enough to consider this a modified question; I disagree with CWs LOK rating. Is low per licensee Change was not significant, 16 H 3 X X U less probable distractors are still possible correct answers (dont ask for most probable in stem - must be only one possible correct answer); doesnt test ability to verify setpoints and operate controls... per K/A. Changes being made to stem, K/A of change will match.
17 F 2 X X U distractors B and C are not credible; doesnt test knowledge of heat removal systems as interrelated with LOCA outside cntmt per K/A. B will be changed, K/A meets OR statment in catalog.
18 F 2 X - U distractors A and D are not credible; I disagree with CWs LOK rating. Examinee has to make sure which sys avail to answer, and must know the most preferable. O/K 19 F 2 - E change the stem question to What is the required Immediate Action? since only one of the choices is actually listed as an Immediate Action in the referenced procedure; Change made.
20 H 3 - E changes from parent are not significant enough to consider this a modified question; Licensee agrees 21 H 3 - E changes from parent are not significant enough to consider this a modified question; Modifications made 22 F 3 X U doesnt test knowledge of the reasons for... per K/A. Licensee will rewrite 23 H 2 X U doesnt test biological effects on humans... per K/A. O/K 24 F 1 X U too easy; doesnt test use of response instructions per K/A. OK 25 H 3 - S 26 F 2 - S 33 of 33 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 27 H 3 - S 28 F 3 - E changes from parent are not significant enough to consider this a modified question; I disagree with CWs LOK rating. Licensee agrees, modified 29 F 3 - S 30 H 3 - S 31 F 3 X U doesnt test ability to operate controls/ind. for RHR pumps per K/A. K/A is OK 32 F 2 - S 33 H 1 - U answer seems too obvious to be a good license exam discriminator; Examinee will have to evaluate all distractors to eliminate, harder than appears on surface. OK 34 H 2 - S 35 H 3 X X - U changes from parent are not significant enough to consider this a modified question; stem question should be changed to: Which ONE of the following components could be a source of in-leakage to the CCW system under the current conditions?; A and C are not credible. Changes sufficient OK.
36 H 4 - S 37 H 4 X U doesnt test ability to monitor trip logic per K/A. Changes sufficient to emphasize panel monitoring, 38 F 3 - S 39 H 3 - S 40 F 3 X X U distractor A is not credible; doesnt test knowledge of connections with cooling water per K/A. Will accept OK 41 H 3 - S 42 F 3 - S 43 H 3 - E I disagree with CWs LOK rating Licesnee agrees 44 F 3 X U doesnt test knowledge of effect of Loss of MFW on AFW per K/A. Answer on our copy wrong, OK 45 H 3 - E changes from parent are not significant enough to consider this a modified question.
46 H 3 X - U Distractors C and D are not credible. Licensee will fix 33 of 33 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 47 H 3 - S 48 F 2 - S 49 F 3 X U doesnt test knowledge of power supplies to major DC loads per K/A. Substitute new question for K/A 50 H 3 X X U changes from parent are not significant enough to consider this a modified question; distractor A and B are not separately credible; doesnt test knowledge of sequencer:
blackout per K/A. Will accept as is OK 51 F 3 X U changes from parent are not significant enough to consider this a modified question; doesnt test ability to explain and apply system limits and precautions for PRM per K/A. Is a bank not modified question, the answer comes from a procedure precaution OK 52 H 3 - S 53 F 3 - S 54 F 4 - S 55 F 3 - S 56 F 3 - S 57 H 3 X U doesnt test knowledge of RPIS design/interlocks associated with rod misalignment per K/A. OK K/A appears to be best fit.
58 H 3 - S 59 H 3 - S 60 F 3 X U doesnt test ability to predict/monitor hydrogen concentration per K/A. 028A4.01 is best match, wont change outline OK 61 F 2 - S 62 F 3 - E I disagree with CWs LOK rating; this relates directly to 55.43(b)(7) and should probably be considered SRO-only; Facility LO for Ros indicates they must know OK 63 F 3 - S 64 H 2 - E I disagree with CWs LOK rating Licensee agrees 65 H 4 - S 33 of 33 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 66 H 3 - E this relates directly to 55.43(b)(2) and should probably be considered SRO-only Facility LO requires RO knowledge OK 67 F 2 - S 68 H 2 - S 69 H 2 - S 70 F 3 X - U distractor C appears to be a possible correct answer if certain assumptions are made; C modified to indicate RX is critical 71 F 3 - S 72 F 2 - S 73 F 2 - S 74 H 3 - S 75 H 3 - S 76 H 3 - X U changes from parent are not significant enough to consider this a modified question; doesnt test SRO-only knowledge; Replacemnt needed use same K/A 77 H 3 - - S 78 H 4 - - E I disagree with CWs LOK rating; change ...your initial... to ...the correct initial...
Change made 79 H 3 - X U doesnt test SRO-only knowledge; Will need replacement 80 H 3 X X U doesnt test ability to recognize EOP/AOP entry conditions per K/A; doesnt test SRO-only knowledge; Rewrite to reflect EOP/AOP SRO level 81 H 3 X X U doesnt test ability to interpret control room indications per K/A; doesnt test SRO-only knowledge; Rewrite to cover K/A 82 H 3 - - S 83 H 3 - - S 84 F 3 - - U not sure about the correct answer on this one; B modified to be clearly incorrect 85 H 3 - - S 33 of 33 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LO LOD K (1-5) U/E/S Explanation (F/ Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO H) Focus Dist. Link units ward K/A Only 86 H 3 - X U doesnt test SRO-only knowledge Rewrite to cover SRO K/A 87 H 3 - - S 88 H 3 X X U changes from parent are not significant enough to consider this a modified question; doesnt test ability to predict impact and correct conditions for abnormal shutdown of RCP in comparison with normal RCP shutdown conditions per K/A; doesnt test SRO-only knowledge K/A OK rewrite to cover SRO level 89 F 3 - X U doesnt test SRO-only knowledge OK requires SRO to apply limits and precautions 90 H 2 - - S 91 H 2 X - X U distractors B and D are not credible; doesnt test SRO-only knowledge OK SRO duties distractors OK after discussion with licensee 92 H 2 - X U doesnt test SRO-only knowledge OK plant procedures require SRO presence to monitor RCP seal backseating 93 H 3 - X U in the stem, change conditions: to indications:; doesnt test SRO-only knowledge Rewrite to cover SRO 94 F 2 - X U doesnt test SRO-only knowledge New question to replace 95 H 3 - - S 96 F 2 - - S 97 H 3 X - U doesnt test knowledge of SRO fuel handling responsibilies. Will modify to reflect SRO responsibilities 98 F 3 - - S 99 H 3 - - E changes from parent are not significant enough to consider this a modified question; Licesnee agrees 100 H 3 - X U changes from parent are not significant enough to consider this a modified question; doesnt test SRO-only knowledge Entry condition to EOPs OK, licensee made sufficient argument is modified
- 80% of the SRO questions are Higher Cognitive Level, which is contrary to the 50-70% required by the NUREG 1021.
- Use of Bank for RO and SRO exam meets NUREG 1021 requirements.
- There are only 3 questions that have appeared on one of the last two NRC exams; overlap does not appear inconsistent.
33 of 33 NUREG-1021, Draft Revision 9
E:\Filenet\ML051050532.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX
- 1. 2. 3. Attributes 4. Job 5. 6.
JPM# Dyn LOD Content U/E Explanation (D/ (1-5) Errors /S (See below for instructions)
S)
IC Cue Critic Scop Ove Jo Minuti Focu s al e r- b- a s Step (N/B lap Link s )
- a. D 3 S b D 4 S
- c. D 3 S
- d. D 3 S
- e. D 3 S
- f. D 4 S
- g. D 3 S
- h. S 3 S B/U S
- i. S 3 S
- j. S 4 S
- k. S 3 S
- Page 7 -
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
- 1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
- 2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
- 3. Check the appropriate box when an attribute weakness is identified:
- The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
- The JPM does not contain sufficient cues that are objective (not leading).
- All critical steps (elements) have not been properly identified.
- Scope of the task is either too narrow (N) or too broad (B).
- Excessive overlap with other part of operating test or written examination.
- 4. Check the appropriate box when a job content error is identified:
- Topics not linked to job content (e.g., disguised task, not required in real job).
- Task is trivial and without safety significance.
- 5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).
- Page 8 -
E:\Filenet\ML051050532.wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen 1 2 3 4 5 6 7 8 9 10 Explanation (See below for instructions)
Set ES TS Crit IC Pred TL L/C Eff U/E/S 1 S S
2 BU S Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
- 1. ES: ES-301 checklists 4, 5, & 6 satisfied.
- 3. Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
- 4. IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
- 5. Pred: Scenario sequence and other factors avoid predictability issues.
- 6. TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
- 7. L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
- 8. Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
- 9. Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 10. Provide a brief description of problem in the explanation column
- Page 9 -