ML053320596
| ML053320596 | |
| Person / Time | |
|---|---|
| Site: | Cooper  |
| Issue date: | 06/27/2005 |
| From: | NRC Region 4 |
| To: | |
| References | |
| CNS-06-2005 | |
| Download: ML053320596 (9) | |
Text
ES-401 CNS-06-2005 Written Examination Form ES-401-9 Review Worksheet
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 2 - S 2 H 2 - S 3 H 3 - E(S) explanation claims D as answer but sounds like C should be correct instead. (Fixed) 4 H 3 - S 5 H* 3 X X U(S) LOK is H vice F; both A and B are correct; explanation of why B is incorrect in comments does not match actual B in Q; doesnt test ability to monitor or operate RPS per KA. (Replaced) 6 F 2 - S 7 F 2 X X X - U(S) as the stem is currently worded, B is a correct answer; also, dont use all of the above or none of the above as an answer or distractor (NUREG-1021 Appendix B-C.2.b). (Fixed) 8 H 3 - E(S) incorrect answers listed and explained in comments dont match Q. (Fixed) 9 H 3 X - E(S) Q asks for the FIRST plant response to lowering air pressure, but shouldnt the FIRST plant response be an alarm (which is not a choice); also, the correct answer is the ONLY one of the four that is a plant response to the conditions; recommend choosing distractors that are responses to dropping air pressure, and ask which of the four occurs first. (Fixed) 10 H 3 - S Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
@ The answer choices are a collection of unrelated true/false statements.
@ One or more distractors is (are) not credible.
@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
@ The question requires reverse logic or appli cation compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditori al enhancement, or (S)atisfactory?
- 7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
33 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 3 - S 12 F 3 X - U(S) Both B and C are correct as written; change stem to If pressure continues to increase with no operator action, the operator should... (Fixed) 13 H 3 - S 14 H 3 X X U(S) doesnt test knowledge of relationship between SPC and high temp. per KA; the use of the phrase in this configuration implies that no configuration change would be necessary to establish suppression pool cooling; however some change must be needed, and if action to change the configuration is assumed, then the configuration could be changed such that all answers are correct. (Replaced) 15 F 3 - S 16 F 2 - S 17 F* 2 - E(S) LOK is F vice H; (Fixed) 18 H* 2 - E(S) LOK is H vice F; (Fixed) 19 F 2 X U(S) KA on Q doesnt match outline; Q doesnt match either the KA on the Q or the KA in the outline; (Fixed) 20 F 3 - S 21 H 2 - S 22 F 3 - S 23 H 4 - S 24 H 3 - E(S) stem use of followup unclear; recommend changing to ...conditions could be causing the problem...; (Fixed) 25 H 2 - S 26 H 3 X U(S) Q does match the KA listed on the Q sheet, but that KA is not on the outline; (Fixed) 27 H 3 - E(S) first two bullets of stem have ...Pump are... - should either be Pumps are or Pump is; (Fixed) 28 H 3 X U(S) doesnt test knowledge of the effect RHR malfunction will have on RPV level per KA; (Replaced) 29 F 2 - S 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 30 F 3 - E(S) in the stem, recommend replacing provided with supplied; (Fixed) 31 F 3 - S 32 F* 2 - E(S) LOK is F vice H; (Fixed) 33 H(F) 3 - X U(S) this is SRO-only material (55.43); (replaced) 34 F 2 - E(S) recommend changing stem to The normal power supply... (Fixed) 35 F 3 - S 36 F 3 X - U(S) D is not credible; (Fixed) 37 H 3 - S 38 H 3 - S 39 H 3 X - U(S) A is not credible; (Fixed) 40 H 3 X U(S) KA at bottom of Q does not match KA at top of Q which does not match KA on outline; (Fixed) 41 F 3 - S 42 F 3 - S 43 F 3 X U(S) Q indicates original KA was replaced, but original KA is still listed as correct on outline - the Q does, however, match the new KA; (Fixed) 44 H 3 - S 45 F 3 - S 46 F 3 - S 47B H 3 - S 48 F* 3 X U(S) LOK is F vice H; Q indicates original KA was replaced, but this is not reflected on the outline - Q does, however match the new KA; (Fixed) 49B H 3 - E(S) stem is missing a word: With the ? operating... (Fixed) 50 H 3 - S 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 51 H 3 - E(S) recommend changing C and D to overheat and slip a pole respectively, otherwise knowing the first half of the answer makes the second half moot; (Fixed) 52 H 3 - S 53B H 3 - S 54 F 3 X X U(S) stem asks about symmetric power distribution but answers are focused on symmetric rod pattern - I dont think the two are technically the same; Q matches KA on Q, but doesnt match outline; (Fixed) 55 H 3 X U(S) KA on Q doesnt match outline, but Q does match KA on Q; (Fixed) 56 F 3 - S 57 H 3 - S 58 H 3 - S 59 H 3 X U(S) KA on Q doesnt match outline, but Q does match KA on Q; (Fixed) 60 H 3 X U doesnt test knowledge related to CONDENSATE system per KA; (it appears that the KA was changed to match the question rather than the question being changed to match the randomly selected KA) 61 H 3 X X U(S) explanation of why A, B, and D are incorrect dont actually eliminate any of the distractors as incorrect; it seems the intent was to focus on alarm and indications, but the stem and distractors are not clearly pointed at specific alarms, but rather conditions which would likely result in alarms; (Fixed) 62 H 3 - E(S) recommend changing C to better match wording in procedure: local area radiation monitors may not indicate elevated radiation levels following a scram-induced crud burst (Fixed) 63 H 3 - E is it appropriate to say supervisory air system pressure for that area is 14 oz ? Ive never heard of referring to an air PRESSURE in ounces; (no change was made and I cannot find the reference that supports this answer) 64 H 3 X - E(S) what is meant by the alarm can be mitigated by____? (Fixed) 65 F 3 X X - U(S) distractor A is not credible and it is not clear why D is incorrect; (Replaced) 66 H* 3 - E LOK is H vice F; 67 F 3 X - U(S) change stem to ...in the Level B position...; it appears that D is also correct; (Fixed) 68 F 3 X - E(S) LOK is F vice H (asking examinee to recall what adverse condition requires TS entry); distractor explanations do not match distractors; attachment is not needed and should not be provided; stem discussion of HPCI inop does not seem relevant to getting correct answer; (Fixed) 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 69 F 2 - E(S) distractor explanations do not match distractors in Q; indicated answer is B, but explanation and reference indicate C is correct instead; (Fixed) 70 F 2 X - U(S) distractors A and B are not credible; indicated answer is D, but explanation indicates C is correct instead; (Fixed) 71 H 3 X - U what proof is there that B is incorrect? (Modified, but still looks like B could be correct) 72 F 1 X - U why is D incorrect - what is meant by external sources? The answer is too obvious; (Modified; D is still also correct) 73 F 4 - S 74 H 2 - S 75 H 2 X X - U should the stem be Low Pressure Turbine Absolute Pressure Recorder? Distractors A and B are not credible; (Fixed; explanations of distractors do not match Q)
S1 F 3 - S S2 H 2 - X U(S) not SRO-only (Replaced)
S3 F* 3 - X U(S) not SRO-only (Replaced)
S4 H 3 - - S S5 H 3 X - - U(S) reference to HCTL and failure to Scram in the stem are unnecessary cues to the correct answer; (Fixed)
S6 F 2 - - S S7 F* 2 X X X - E(S) LOK is F vice H; too much info in the stem; (Fixed)
S8 H 3 - - S S9 F 2 - X U(S) not SRO-only (this is 55.41.b.10) (Replaced)
S10 F 2 - - S S11 F 2 X - X U(S) not SRO-only (this is 55.41.b.10); distractor C is a subset of A; (Fixed; SRO learning objectives capture this as SRO only) 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only S12 H 3 X X - U(S) Q doesnt match KA in outline or listed at top of Q - however, Q does match KA listed at bottom of Q; A is not credible; (Replaced)
S13 H 2 - - S S14 H 1 X X X X U add only to B and C; A is technically correct, since de-energizing it does minimize the loads on it; answer seems obvious; not SRO-only; doesnt test ability to predict the impact of opening a disconnect UNDER LOAD per KA; (modified, but still doesnt match KA)
S15 H 3 X - X U(S) D is not credible; not SRO-only; (Fixed)
S16 H 3 X - - U(S) B is not credible in that if I think B is true, that makes A also true and therefore I eliminate B by simple logic; (Fixed)
S17 H 2 X X - - U(S) providing the one procedure is a cue which eliminates C and D as credible; (modified)
S18 H 3 X X X - U the explanation does not support the indicated correct answer - if MCPR was exceeded, then
>.1% damage would be expected; rad alarms cue that there was fuel damage; doesnt deal with reactor vessel internals per KA (fuel is not considered vessel internals); (it appears that the DRAFT SRO outline has been modified by deleting the original KA for this question and adopting a new one; the DRAFT should have been kept in-tact as it was with a new version generated reflecting changes since the DRAFT; distractor D is not credible)
S19 H 3 - X U(S) not SRO-only (replaced)
S20 F 3 - - S S21 F 2 - - S S22 F 2 X X - - U how would anyone choose is not with the TS provided and since the second half of the question asks whether you are in TS limits - this makes C and D not credible; (replaced, but now it doesnt match the KA)
S23 F 3 X - - U(S) it is not clear why B is not true (explanation is confusing), and D is definitely correct; (modified)
S24 H 2 X - U(S) doesnt test knowledge of BASES for prioritizing per KA; (Fixed)
S25 H 2 - - S
GENERAL COMMENT
S:
- 1. Recommend consistency with use of units (use % vice percent and o F vice degrees for example); also be consistent with the way bullets are implemented (just use dashes or dots; recommend against check-marks).
- 2. The outline should indicate which question number it belongs to (ideally, in the # column on the right); there is currently no cross-reference from the outline to the exam other than hunting for matching KA numbers (which gets cumbersome when things are out of order).
34 of 34 NUREG-1021, Draft Revision 9
S:\DRS\OB\EXAMS\CNS\June 2005 Exam (PJE)\ADAMS\CNS-06-2005-NRC DRAFT EXAM COMMENTS\CNS-06-2005-GP-Operating Exam Review Worksheet.wpd APPENDIX O - REGION IV OPERATING TEST ADMIN JPM QUALITY REVIEW MATRIX
- 1. 2. 3. Attributes 4. Job Content 5. 6.
JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)
IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link ADM-RO1 S 1 N U NEW; Seems a little narrow and easy - not sure what administrative skills this is testing; ADM-RO2 S 2 X E step 6 (Notifies CRS) should not be CRITICAL; why is this titled Review of Daily Logs since it has nothing to do with reviewing logs - its simply filling out an Attachment and recognizing an LCO.
ADM-RO3 S 2 S ADM-RO4 S 4 X U MODIFIED; Is this really a task that is expected of ROs?
ADM-SRO1 S 2 X E Step 8 should not be CRITICAL; ADM-SRO2 S 2 X E NEW; no CRITICAL steps are identified in this JPM; ADM-SRO3 S 3 X X U NEW; change initiating cue to ...you have just been notified...; is drop dead date a technical term or slang? What exactly is the assessment supposed to determine (...to see if it can be run and when...; what is it - the RCIC system or the surveillance?)? Step 5 is indicated as CRITICAL, yet the applicant does not need to do this step in order to respond to the initiating cue; ADM-SRO4 S 3 S MODIFIED; ADM-SRO5 S 4 S MODIFIED; Time-Critical; Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
- 1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
- 2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
- 3. Check the appropriate box when an attribute weakness is identified:
- The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
- The JPM does not contain sufficient cues that are objective (not leading).
- All critical steps (elements) have not been properly identified.
- Scope of the task is either too narrow (N) or too broad (B).
- Excessive overlap with other part of operating test or written examination.
- 4. Check the appropriate box when a job content error is identified:
- Topics not linked to job content (e.g., disguised task, not required in real job).
- Task is trivial and without safety significance.
- 5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 6. Provide a brief description of any U or E rating in the explanation column.
- 7. Save initial review comments as normal black text; indicate how comments were resolved using blue text so that each JPM used on the exam is reflected by a (S)atisfactory resolution on this form.
- Page 1 -
S:\DRS\OB\EXAMS\CNS\June 2005 Exam (PJE)\ADAMS\CNS-06-2005-NRC DRAFT EXAM COMMENTS\CNS-06-2005-GP-Operating Exam Review Worksheet.wpd APPENDIX O - REGION IV OPERATING TEST SYSTEM JPM QUALITY REVIEW MATRIX
- 1. 2. 3. Attributes 4. Job Content 5. 6.
JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)
IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link SYS-A D 3 X E Step 2 (recognize and report) should not be CRITICAL; SYS-B D 3 X E all the CRITICAL steps are operator checks/verifies - these type actions cannot be CRITICAL; JPM title on outline doesnt match actual JPM title/actions; SYS-C D 3 S MODIFIED; SYS-D D 2 X E NEW; step 2: is ...late hour... supposed to be ...last hour...? AND step 2 should be CRITICAL since it is one of the two things the cue said to do; SYS-E D 2 X E MODIFIED; step 11 should be CRITICAL since it is directed by the cue; SYS-F D 3 X E outline indicates this is ALTERNATE PATH, but it isnt; step 11 should be CRITICAL; steps 9, 10, and 12 should not be CRITICAL (ensure is not a CRITICAL action);
SYS-G D 3 X E step 1 should not be CRITICAL; why are steps 4 and 5 CRITICAL but steps 6 and 7 are not? Step 16 says to leave APRM D in bypass, but there is no step which previously put it into bypass; SYS-H D 2 S INPLANT-I D 3 X E steps 2 and 3 should not be CRITICAL; INPLANT-J D 3 X E NEW; RCA entry is required but not noted on the outline; no steps are indicated as CRITICAL; INPLANT-K D 3 X E NEW; RCA entry; steps 15-23 are not CRITICAL if all it involves is ensuring a valve position; Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
- 1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
- 2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
- 3. Check the appropriate box when an attribute weakness is identified:
- The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
- The JPM does not contain sufficient cues that are objective (not leading).
- All critical steps (elements) have not been properly identified.
- Scope of the task is either too narrow (N) or too broad (B).
- Excessive overlap with other part of operating test or written examination.
- 4. Check the appropriate box when a job content error is identified:
- Topics not linked to job content (e.g., disguised task, not required in real job).
- Task is trivial and without safety significance.
- 5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 6. Provide a brief description of any U or E rating in the explanation column.
- 7. Save initial review comments as normal black text; indicate how comments were resolved using blue text so that each JPM used on the exam is reflected by a (S)atisfactory resolution on this form.
- Page 2 -
S:\DRS\OB\EXAMS\CNS\June 2005 Exam (PJE)\ADAM S\CNS-06-2005-NRC DRAFT EXAM CO MMEN TS\CNS-06-2005-GP-Operating Exam Review W orksheet.wpd APPENDIX O - REGION IV OPERATING TEST SCENARIO QUALITY REVIEW MATRIX Scen 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Explanation (See below for instructions)
Set ES TS Crit IC Pred TL L/C Eff U/E/S 1 ES-301-4,5,6 not provided; 2 ES-301-4,5,6 not provided; 3 ES-301-4,5,6 not provided; 4 ES-301-4,5,6 not provided; Ins tructio ns for Co m pleting Matrix This form is not contained in or required by NURE G-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comm ent and explain the issue in the space provided.
- 1. ES: ES-301 checklists 4, 5, & 6 satisfied.
- 3. Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
- 4. IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
- 5. Pred: Scenario sequence and other factors avoid predictability issues.
- 6. TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
- 7. L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
- 8. Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
- 9. Based on the reviewers judgment, rate the scenario set as (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory.
- 10. Provide a brief description of problem in the explanation column .
- 11. Save initial review comm ents as normal black text; indicate how comm ents were resolved using blue text so that each JPM used on the exam is reflected by a (S)atisfactory resolution on this form.
- Page 3 -