ML042080289
| ML042080289 | |
| Person / Time | |
|---|---|
| Site: | Wolf Creek  |
| Issue date: | 05/13/2004 |
| From: | Gody A Operations Branch IV |
| To: | Muench R Wolf Creek |
| References | |
| 50-482/04-301, ES-401, ES-401-9 50-482/04-301 | |
| Download: ML042080289 (9) | |
Text
ES-401 WC-05-2004 - Written Examination Form ES-401-9 Review Worksheet
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 X S 2 H 3 X X E The correct answer is not listed among the choices; there are enthalpy issues that must be dealt with to get the correct answer - it is not as simple as finding the saturation temperature of the PRT.
3 3 X E This question LOK is F vice H.
4 H 4 X S 5 2 X X E This queston LOK is F vice H; distractor B is not related to indicated pump parameters and is not credible.
6 H 3 X NO U Distractor B should mention Aux Building vice Turbine Building in order to prevent only having to know the valves are in the Aux Bldg to answer the question; this question is inconsistent with the K/A which involves a LOSS of RX COOLANT MAKE-UP.
7 H 3 X S 8 H 3 X S Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
@ The answer choices are a collection of unrelated true/false statements.
@ One or more distractors is (are) not credible.
@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
@ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
33 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 9 H 4 X X X E Change the stem from ...over a 10-minute period to ...over a couple of minutes to clarify that the INITIAL response in question is the response after the setpoint has reached 2100, not the FIRST automatic action to occur during the drift.
10 H 3 X X X U Change stem to indicate whether the IR Channel N-35 failed high or low; it appears that, although inserting rods is not directed by the RNO of E-0, it IS a correct action that the RO will take, and therefore both A and B are correct actions.
11 H 2 X X S 12 H 2 X X E Change the stem question to Using the preferred method, calculate subcooling of the reactor.
13 3 X E This question LOK is H vice F; for distractor A, there be a for or on between channel and BB.
14 F 3 X S 15 F 3 NO U Does not test REASONS for the automatic actions of the ESW system as result of the actuation of ESFAS as specified in the K/A.
16 F 2 X X E It appears that Distractors A and C are fundamentally the same based on the reference document provided. Both are connecting the RHR System to the RCS 17 H 3 X X U This question LOK is H vice F; both B and D are correct answers.
18 F 3 X S this question LOK is F vice H 19 H 3 X NO U In the stem, add rods between rod control causes... and ...to step out...; the references provided do not prove that, at 100% power, the C-2 or C-4 interlocks wont be reached before 232 steps; what justification is there for operators knowing that the reactivity added by continuous rod withdrawal will not be enough to activate C-2 or C-4 before 232 steps is reached? The question doesnt test the ability to determine/interpret proper actions to be taken if automatic safety functions have not taken place per the K/A.
20 F 2 X X E This question directly addresses 10CFR55.43.b.5 and is therefore more appropriately an SRO-only question.
21 F 3 X S 22 H 3 X S 23 H 2 X S ES-401 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 24 3 X X E This question LOK is F vice H; change stem question to: ...would require off-site dose calculations to determine... to eliminate ambiguity regarding who directs action.
25 H 3 X S This question LOK is H vice F.
26 F 3 X X E This question directly addresses 10CFR55.43.b.2 and is therefore more appropriately an SRO-only question.
27 F 3 X E Eliminate the hyphens between nomenclature and parameter values in the stem conditions and use colons instead (hyphens can be misunderstood as negative signs);
28 H 3 X S 29 H 3 X S 30 F 3 X S 31 H 2 X S 32 H 3 X S 33 F 2 X S 34 H 4 X E Eliminate the hyphens between nomenclature and parameter values in the answer choices and use colons instead (hyphens can be misunderstood as negative signs);
35 F 3 X S 36 H 3 X S 37 F 2 X S 38 F 3 X S 39 F 3 X S 40 F 3 X E This question LOK is F vice H; neither the conditions of the stem nor the distractors are sufficiently changed to allow calling this a modified question - this is a BANK question.
41 H 3 X S 42 H 2 X S 43 H 2 X S 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 44 H 3 X S 45 H 3 X S 46 H 2 X S 47 H 3 X S 48 F 3 X S 49 F 3 X S 50 F 3 X X E This question directly addresses 10CFR55.43.b.2 and is therefore more appropriately an SRO-only question.
51 F 2 X S 52 F 3 X S 53 H 3 X X E Change stem question to What actions should be taken? - as currently worded, the question cues the applicant that B cant be correct.
54 F 3 X S 55 H 3 X S 56 H 3 X S 57 H 3 X S 58 H 3 X S 59 F 3 X S 60 F 3 X S 61 F 2 X S 62 H 2 X S 63 F 2 X S 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 64 F 2 X E Distractor A and proposed answer A are worded differently.
65 H 3 X X X U The stem cues the correct answer since the only abnormal indication given is the circ-water pump trip, and only A mentions low circ-water flow; delete most probable in the stem - and consider using What could have caused the valve closure given these conditions?
66 F 2 X S 67 F 3 X X U Since, based on the stem, youve transitioned out of E-1 and into FR-C1, then C and D are both correct.
68 H 3 X E Change stem to ...and steam dumps are closed vice ...closing to eliminate ambiguity as to whether step 6.42.2.4 is in effect or not.
69 F 3 X X E This question directly addresses 10CFR55.43.b.2 (and 43.b.7) and is therefore more appropriately an SRO-only question.
70 F 2 X S 71 F 3 X X E The action taken in Distractor C i.e., that the TDAFW pump is to remain isolated, does not appear to be consistent with the procedure.
72 H 3 X S 73 H 3 X X E This question directly addresses 10CFR55.43.b.5 and is therefore more appropriately an SRO-only question.
74 H 2 X S 75 F 3 X S 76 H 3 X NO U This question really doesnt evaluate knowledge at the SRO level since any RO would be expected to know when an SI is required 77 H 3 X X NO U Distractors A, B and C each correctly answer the stem question, which was what actions should be taken - the answer being leave all RCPs running; the fact that each answer also has a conditional qualifier does not sufficiently discriminate between the seemingly correct answers; this question is also confusing from the standpoint that the conditions in the stem lead the operators to trip all the RCPs before the running SI pump tripped; this is a 10CFR55.41.b.5 level question, and is therefore not a SRO-only question.
78 F 3 X NO U This is RO level knowledge, not SRO-only (not 55.43.b.5).
34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 79 F 2 X NO U This question LOK is F vice H; this is RO procedure knowledge and not SRO-only (not 55.43.b.5).
80 F 2 X NO U This is RO procedure knowledge and not SRO-only (not 55.43.b.2).
81 H 3 X X NO U Change stem question to Which power supply failure caused these conditions?; this is RO knowledge and not SRO-only (not 55.43.b.5) 82 H 2 X X S 83 H 3 X NO U This is RO knowledge and not SRO-only (not 55.43.b.5) 84 H 3 X NO U This is RO knowledge and not SRO-only (not 55.43.b.5) 85 H 3 X X X E Distractors A and B (excessive cooldown after turbine trip) are not credible; Change stem to say What is the TS basis...
86 H 3 X X S 87 F 3 X X NO U This is RO knowledge and not SRO-only (more like 55.41.b.12) 88 H 3 X X X E Distractors A and C are not distinguishable as potential answers; no distractor has been changed from the parent and therefore question does not meet criteria for a modified question.
89 H 3 X X NO U The word higher is repeated in stem - eliminate one of them; eliminate tech spec from distractor B; this is RO knowledge and not SRO-only (not 55.43.b.5) 90 H 3 X X S 91 F 2 X X S 92 F 3 X NO U This is RO knowledge and not SRO-only (not 55.43.b.4; more like 55.41.b.13) 93 F 3 X X S 94 H 3 X X S 95 F 3 X X S 96 F 3 X X S 97 F 3 X X S 98 F 2 X NO U This is RO knowledge vice SRO-only (not 55.43.b.4, more like 55.41.b.12) 99 F 3 X NO U This is RO knowledge vice SRO-only (not 55.43.b.4, more like 55.41.b.13) 100 H 3 X X S 34 of 34 NUREG-1021, Draft Revision 9
S:\DRS\OB\EXAMS\WC\WC-INIT-EXAM 05-2004\WC-05-2004-NRC DRAFT EXAM COMMENTS\WC-05-2004-APP-E - JPM-A-QualityRvw R1.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX
- 1. 2. 3. Attributes 4. Job Content 5. 6.
JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)
IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link RO-A1a S 2 E no key was provided RO-A1b S 3 E Step 1 is identified as critical, but it has several aspects for the applicant to note - are all aspects critical, or just the calculation of 1/M? No key was provided.
RO-A2 S 1 X E This JPM needs to be revised such that the applicant has to locate data in the procedure instead of from a table.
RO-A3 S 2 S SRO-A1a S 2 E No key was provided. This JPM is essentially similar to the RO JPM and should be considered to be a BANK question.
SRO-A1b S 3 X E Need to revise the JPM to require documentation of the actions.
SRO-A2 S 3 E Dates incorrect in task standard and Step 1. Step 1 bullet #2 is the same as bullet #1.
SRO-A3 S 3 S SRO-A4 S 3 S Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests.
Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
- 1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.
- 2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
- 3. Check the appropriate box when an attribute weakness is identified:
- The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
- The JPM does not contain sufficient cues that are objective (not leading).
- All critical steps (elements) have not been properly identified.
- Scope of the task is either too narrow (N) or too broad (B).
- Excessive overlap with other part of operating test or written examination.
- 4. Check the appropriate box when a job content error is identified:
- Topics not linked to job content (e.g., disguised task, not required in real job).
- Task is trivial and without safety significance.
- 5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).
- Page 1 -
S:\DRS\OB\EXAMS\WC\WC-INIT-EXAM 05-2004\WC-05-2004-NRC DRAFT EXAM COMMENTS\WC-05-2004-APP-E - JPM-B-QualityRvw R1.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX
- 1. 2. 3. Attributes 4. Job Content 5. 6.
JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)
IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link C-1 S 2 X U This is an Admin task, not a Control Room/In-Plant Systems JPM; the approved outline (ES-301-2) indicated this JPM would require the applicant to Align Power to Safeguards Bus using the OFN -
marking up the procedure does not match this outline; the steps and standards listed in the JPM seem to indicate that the applicant is expected to actually perform the procedure rather than just mark it up, but the initiating cue and the task completion standard suggest that the goal is simply to mark up steps that need to be performed.
P-1 S 1 X U JPM only requires the operator to check one valve open and open one other valve - a little too simplistic to provide discriminatory validity.
P-2 S 2 S P-3 S 2 X E Need a cue on Page 3.
S-1 D 3 X E Why is Step 2 critical when nothing is performed? The notes appear to reduce validity.
S-2 D 3 X E Step 7, Items 2 & 4, do not appear to be critical. The note in Step 6, states in the STANDARD that the action was a part of initial conditions, but the initial conditions do not identify the stated lineup.
S-3 D 1 X U JPM only requires the operator to check one indication and close one valve - a little too simplistic to provide discriminatory validity.
S-4 D 3 E Flow rates in Step 5 are incorrect.
S-5 D 3 S S-6 D 2 X U JPM only requires the operator to pull rods to increase power and then stabilize power when the instrument doesnt respond (only one step in the procedure) - a little too simplistic to provide discriminatory validity. Not a valid alternate path JPM.
S-7 S 2 U even though this JPM requires numerous switch manipulations, they are all the same and successfully completing this task provides little to no opportunity to evaluate knowledge and ability of the applicant -
a little too simplistic to provide discriminatory validity. Cues are missing.
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests.
Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
- 1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
- 2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
- 3. Check the appropriate box when an attribute weakness is identified:
- The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
- The JPM does not contain sufficient cues that are objective (not leading).
- All critical steps (elements) have not been properly identified.
- Scope of the task is either too narrow (N) or too broad (B).
- Excessive overlap with other part of operating test or written examination.
- 4. Check the appropriate box when a job content error is identified:
- Topics not linked to job content (e.g., disguised task, not required in real job).
- Task is trivial and without safety significance.
- 5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).
- Page 1 -
S:\DRS\OB\EXAMS\WC\WC-INIT-EXAM 05-2004\WC-05-2004-NRC DRAFT EXAM COMMENTS\WC-05-2004-APP-N - SCENARIO-QualityRvw R1.wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen 1 2 3 4 5 6 7 8 9 10 Explanation (See below for instructions)
Set ES TS Crit IC Pred TL L/C Eff U/E/S 1 - - - - - - - - S Appears to be lengthy, will check during validation.
2 - - - - - - - - S Appears to be lengthy, will check during validation.
3 - - - - - - - - S Appears to be lengthy, will check during validation.
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
- 1. ES: ES-301 checklists 4, 5, & 6 satisfied.
- 3. Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
- 4. IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
- 5. Pred: Scenario sequence and other factors avoid predictability issues.
- 6. TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
- 7. L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
- 8. Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
- 9. Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 10. Provide a brief description of problem in the explanation column
- Page 1 -