ML030350659
| ML030350659 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  (NPF-041, NPF-051, NPF-074) |
| Issue date: | 12/06/2002 |
| From: | NRC Region 4 |
| To: | Arizona Public Service Co |
| References | |
| 50-528/02-301, 50-529/02-301, 50-530/02-301 | |
| Download: ML030350659 (34) | |
Text
V = Changed as result of validation ER = Changed From NRC Chief Examiners review of examination submittal OR = Changed From NRC Chief Examiners review of outline submittal EV = Changed From NRC validation of examination materials PV = Palo Verde initiated change 1
General Item Modification Source Date
V = Changed as result of validation ER = Changed From NRC Chief Examiners review of examination submittal OR = Changed From NRC Chief Examiners review of outline submittal EV = Changed From NRC validation of examination materials PV = Palo Verde initiated change 2
Written Examination Question #
Modification Source Date 3
Reworded question to better fit the designated KA ER 10/3/02 4
Reworded to specify the procedure the guidance refers to ER 10/3/02 6
Added "This will provided protection against local criticality event" to better show tie to KA ER 10/3/02 9
Rewritten to identify the effect of a loss of the core support barrel ER 10/3/02 14 Minor editorial change. Changed "____to____" to
"_______or______"
ER 10/3/02 17 Removed "should" from "D" response ER 10/3/02 24 Minor editorial change. Deleted "one" from "C" response ER 10/3/02 28 Reworded stem of question to better match KA statement ER 10/3/02 32 New question selected from INPO Bank that matches KA ER 10/3/02 36 Question rewritten to exactly match KA criteria ER 10/3/02 41 Changed trolley to bridge/trolley for consistency in question ER 10/3/02 44 Added "initially" to "Who is initially responsible" to meet exact wording of procedure and to clarify.
ER 10/3/02
V = Changed as result of validation ER = Changed From NRC Chief Examiners review of examination submittal OR = Changed From NRC Chief Examiners review of outline submittal EV = Changed From NRC validation of examination materials PV = Palo Verde initiated change 3
Modification Source Date A1-1 Changed the initiating cue to clarify the candidate is the LSRO in question to identify along with the Refuel machine operator if they have exceeded any administrative work limitations.
ER 10/2/02 A1-2 Changed the initiating cue to remove the procedure reference "Sensitive Issues Manual Appendix A & B" ER 10/2/02 A2 Changed the initiating cue to remove the procedure reference "Housekeeping & System Cleanliness procedure" ER 10/2/02 A3 Changed the JPM to challenge the candidates RP knowledge of the REP criteria for changing jobs from the original task signed in on.
ER 10/2/02 A4 Changed the initiating cue to remove the procedure reference "EPIP-01" ER 10/2/02 B1 Editorial: Applicants cue sheet was not the same as the examiners. Also changed step 17 & 18 to only give U2 values for examiner to be consistent with remaining steps of the JPM.
ER 10/3/02 B4 Question 1 marked as a closed reference question Question 2 marked as an open reference question and reworded to state that the AO has taken action to supply backup air to the cask loading seal but that it is still leaking with a pressure being applied by the AO that is outside of the minimum band and for the LSRO to provide guidance to correct this condition.
ER 10/3/02
V = Changed as result of validation ER = Changed From NRC Chief Examiners review of examination submittal OR = Changed From NRC Chief Examiners review of outline submittal EV = Changed From NRC validation of examination materials PV = Palo Verde initiated change 4
Operating (Scenario) Test Scenario #
Modification Source Date General Included timeline information ER 10/2/02 Scenario #1 Event 2, minor editorial corrections to add the word "in" to grammatically correct the sentence structure in the cue.
ER 10/2/02 Scenario #2 Event 2 changed cue to "Take all appropriate LSRO actions per the appropriate procedure(s) for this condition." This will make the event more challenging, as the LSRO will need to reference the correct procedure(s) and identify the LSRO applicable steps.
ER 10/2/02
Comments on the PV LSRO Exam Scenario 1:
Initial conditions/Cue - Give the LSRO the associated paperwork for tracking fuel movement - ie fuel bundle ID number, location from, location to, core map (include as part of the initial cue).
Make this as realistic as possible.
Event 1 - Have them record the storage location of the fuel bundle as part of the required response for Step 2 of the scenario.
The transition to Event 2 is unclear. Shouldnt there be some discussion on how they are getting to Event 2? Have the LSRO discuss the sequence of events that would lead up to Event 2 verses the examiner giving the cue. This would be more in line with performing in the plant. Start the cue when the LSRO describes that the containment refueling machine has lifted the fuel from the upender and is beginning to move the fuel to the designated core location.
Scenario 2:
Initial conditions/Cue - Give the LSRO the associated paperwork for tracking fuel movement - ie fuel bundle ID number, location from, location to, core map (include as part of the initial cue).
Make this as realistic as possible.
Cue - does the refueling machine have a fuel bundle suspended? Seems like the examiner cue indicates that the RTMZ is empty and ready to pickup the next bundle. The candidate cue needs to be clarified to specify that a bundle is suspended.
With the scenarios as written, how we will be able to evaluate Competencies 5 and 8 on Form ES 701-2?
JPMs A1.1 - I understand the examinee cue, but not sure I understand Step 4 of the JPM. This appears to be another separate question for the same K/A or task standard. Seems to me that you now meet the two question criteria even without A1.2. If you desire to test this as one question, then include as part of the examinee cue.
A1.2 - Spoke with Phil Caphart and he indicated he was concerned with removing the reference to the procedure from the cue. He indicated that the LSRO would most likely ask an SRO and the SRO would tell him what procedure to look at. If we decide this is ok, then we need to pulse some onshift SROs to see if they know what procedure to refer the LSROs to. If they dont, then, we need to.......?????
A.2 - seems a little easy, but, based on language in the procedure, the candidate could elect to continue on with the fuel movement. You could increase the difficulty by having a fuel movement plan and have the JPM written where the candidate would have to ask for the location of fuel assembly. Provide a core map with the fuel assembly marked and then have him look at the fuel movement plan to determine which fuel assembly is to be moved next verses telling him that information.
A.3 - looks ok
A.4 - looks ok B.1 - Where is this JPM going to be performed? If we are not going to be in the plant, then is the refueling mockup available for any of these types of JPMs (refueling machine)? Even if it is not operable, if all the parts are on the mockup, we could have them show us what equipment they would manipulate. If this is not possible, then the licensee should take numerous pictures of the refueling bridge and have the candidates show us what equipment on the bridge they would be operating. Put the pictures in a binder - be sure to include pictures of name plate data with the equipment.
Provide a current core map.
The initiating cue for the candidate and the examiner is different. The examiner cue contains -
The fuel spreader is being used. Based on this statement, I assume that location M7 is surrounded by other fuel assemblies. This may be important for the candidate to know.
Step 1 - How is the mast orientation verified - digital or a fixed indicator or both? Need to make sure we understand so we can verify candidate knowledge.
Step 2 - Where is the bridge/trolley position indicator located? I assume this is a digital indicator. Some bridges will also have manual indicators. Does this bridge?
Step 3 - How does the candidate reference core position Mike 7? If by core location, provide a core map and have him point out the location.
Step 16 and 17 - What are the designators for these input shafts? Based on my memory -
there should be 2 - one for east/west movement and one for north/south movement. What direction should the handwheel be turned? How many revolutions of the handwheel per inch of bridge movement? If moving the bridge east and south, what directions do I expect my bridge position to be moving. How far should I expect the candidate to try and move the assembly?
Based on the core load, how far could the candidate move the assembly? What distance would be acceptable for the assembly to clear the North side fuel assembly and not contact the other adjacent fuel assemblies?
Step 16 - Second cue needs to be changed. Again provide a core map and show the candidate the side that is contacting the fuel assembly. Let the candidate determine it is the North side and from there, which way to move the assembly.
Steps 20 and 21 - What are the designators for these input shafts? What direction should the handwheel be turned? How many revolutions of the handwheel per inch of bridge movement?
Step 28 - What is the proper pin orientation? Cue should be based on pin orientation, NOT, Assembly is properly seated. This determination has to be made by the candidate, not given.
B.2 - Is this going to be simulated on the spent fuel handling machine?
B.3 - Why does the cue say a new experimental fuel assembly? How often, if ever, does PV get an experimental fuel assembly? Change cue to new fuel assembly. New fuel elevator is in the spent fuel pool transfer canal, so an extremely hot particle stuck on the elevator could cause radiation levels to be excessive. Does not make sense that problem would be with a new
experimental fuel assembly. They have to do a dry receipt inspection, before they place it into storage, so any abnormal radiation levels would have been detected then.
Step 10 - Change cue to have examiner respond as the Shift Supervisor and provide repeat back to the candidate B.4 - looks ok.
B.5 -
Step 1 - How is the mast orientation verified - digital or a fixed indicator or both? Need to make sure we understand so we can verify candidate knowledge.
Step 2 - Where is the bridge/trolley position indicator located? I assume this is a digital indicator. Some bridges will also have manual indicators. Does this bridge?
Step 3 - give candidate a full core map and let him decide if A7 location is empty.
C:\\ORPCheckout\\FileNET\\ML030350659.wpd
- Page 8 -
PALO VERDE LSRO EXAM - 11/02 (COMMENT RESOLUTION)
APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX JPM#
1.
Dyn (D/S
)
2.
LOD (1-5)
- 3. Attributes
- 4. Job Content Errors 5.
U/E/
S 6.
Explanation (See below for instructions)
IC Focus Cues Critic al Steps Scope (N/B)
Over-lap Job-Link Minuti a
A1-1 N/A 2
S Changed to specify in the cue that applicant is the subject LSRO in the problem.
A1-2 N/A 2
S Deleted reference to Sensitive issues Manual.
A2 N/A 2
S Deleted procedure reference. increased the difficulty by having a fuel movement plan and wrote the JPM where the candidate has to ask for the location of fuel assembly. Provided a core map with the fuel assembly marked so that the candidate must look at the fuel movement plan to determine which fuel assembly is to be moved.
A3 N/A 2
S Made task more challenging and switched Steps 1 & 2.
A4 N/A 2
S Deleted procedure reference and statement that would direct applicant to procedure.
B1 D
3 S
Changed cue sheets and added steps.
B2 D
3 S
B3 D
3 S
Changed cue to new fuel assembly.
B5 D
3 S
Added new steps.
- Page 9 -
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
1.
Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.
2.
Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3.
Check the appropriate box when an attribute weakness is identified:
The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
The JPM does not contain sufficient cues that are objective (not leading).
All critical steps (elements) have not been properly identified.
Scope of the task is either too narrow (N) or too broad (B).
Excessive overlap with other part of operating test or written examination.
4.
Check the appropriate box when a job content error is identified:
Topics not linked to job content (e.g., disguised task, not required in real job).
Task is trivial and without safety significance.
5.
Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6.
Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).
C:\\ORPCheckout\\FileNET\\ML030350659.wpd
- Page 10 -
PALO VERDE LSRO EXAM - 11/02 APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX JPM#
1.
Dyn (D/S
)
2.
LOD (1-5)
- 3. Attributes
- 4. Job Content Errors 5.
U/E/
S 6.
Explanation (See below for instructions)
IC Focu s
Cue s
Critic al Steps Scop e
(N/B
)
Ove r-lap Job-Link Minuti a
A1-1 N/A 2
E Specify in the cue that applicant is the subject LSRO in the problem.
A1-2 N/A 2
E Delete reference to Sensitive issues Manual.
A2 N/A 1
E This task is made unnecessarily easy by directing applicant to procedure. Given the FME issue he should be able to decide which procedure to follow. increase the difficulty by having a fuel movement plan and have the JPM written where the candidate would have to ask for the location of fuel assembly.
Provide a core map with the fuel assembly marked and then have him look at the fuel movement plan to determine which fuel assembly is to be moved next verses telling him that information.
A3 N/A 1
U Too simple. Consider a RWP with errors. Need to switch Steps 1 & 2.
A4 N/A 1
E This task is made unnecessarily easy by directing applicant to procedure.
B1 D
3 E
Applicants cue sheet should refer to Unit 2 to match cue given by examiner. Why are other Unit trolley positions shown?
B2 D
3 S
Need to check (during prep week) if spent fuel handling machine mock-up can be used.
B3 D
3 E
Change cue to new fuel assembly. New fuel elevator is in the spent fuel pool transfer canal, so an extremely hot particle stuck on the elevator could cause radiation levels to be excessive. Does not make sense that problem would be with a new experimental fuel assembly.
B5 D
3 E
Need to add the steps and information discussed by telephone on 10/22.
- Page 11 -
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
1.
Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.
2.
Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3.
Check the appropriate box when an attribute weakness is identified:
The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
The JPM does not contain sufficient cues that are objective (not leading).
All critical steps (elements) have not been properly identified.
Scope of the task is either too narrow (N) or too broad (B).
Excessive overlap with other part of operating test or written examination.
4.
Check the appropriate box when a job content error is identified:
Topics not linked to job content (e.g., disguised task, not required in real job).
Task is trivial and without safety significance.
5.
Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6.
Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).
C:\\ORPCheckout\\FileNET\\ML030350659.wpd
- Page 12 -
PALO VERDE LSRO - 11/02 (COMMENT RESOLUTION)
APPENDIX F - REGION IV OPERATING TEST QUESTION QUALITY REVIEW MATRIX Q#
1.
LOK (F/H
)
2.
LOD (1-5)
- 3. Psychometric Errors
- 4. Job Content Errors 5.
U/E/
S 6.
Explanation (See below for instructions)
Stem Focu s
Dire ct L/U One Ans Min Resp Scop e
(TS?
)
Job-Link Min u-tia SRO Only Bac k-ward s
B4.1 H
3 S
Made a closed reference Q.
B.4.
2 F
2 S
Question now discriminates by providing information that makes the AO respond to the seal leak and asking the LSRO what directions he would give to the AO to correct the problem.
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
1.
Classify level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.
2.
Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3.
Check the appropriate box when a psychometric error is identified:
Stem lacks sufficient focus to solicit only the answers listed (e.g., unclear on intent, answer needed, or unnecessarily negatively phrased)
Direct lookup (e.g., desired answer contained in obvious reference),
Question does NOT solicit single demonstrably correct answer,
Minimum response for passing credit NOT described in key,
Scope of question outside guidance of NUREG (e.g., why not how, TS bases not system, emergency not emergency plan),
4.
Check the appropriate box when a job content error is identified:
Topics not linked to job content (e.g., disguised task, not required in real job, invalid K/A).
Recall of too specific knowledge,
RO test items test at the SRO job level or vice versa,
Reverse logic or application compared to job.
5.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6.
Provide a brief description of problem in the explanation column.
C:\\ORPCheckout\\FileNET\\ML030350659.wpd
- Page 13 -
PALO VERDE LSRO - 11/02 APPENDIX F - REGION IV OPERATING TEST QUESTION QUALITY REVIEW MATRIX Q#
1.
LOK (F/H
)
2.
LOD (1-5)
- 3. Psychometric Errors
- 4. Job Content Errors 5.
U/E/
S 6.
Explanation (See below for instructions)
Stem Focu s
Dire ct L/U One Ans Min Resp Scop e
(TS?
)
Job-Link Min u-tia SRO Only Bac k-ward s
B4.1 H
3
E Only acceptable as closed reference. Will make a closed reference Q.
B.4.
2 F
1
U Question does not discriminate because it causes applicant to scan procedures until he finds numbers. Consider stating that the AO has determined that the Decon Area Gate Seal is leaking and there is 30 psig pressure on the seal. Then ask what directions he would give to the AO to correct the problem.
Will review and consider change.
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
1.
Classify level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.
2.
Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3.
Check the appropriate box when a psychometric error is identified:
Stem lacks sufficient focus to solicit only the answers listed (e.g., unclear on intent, answer needed, or unnecessarily negatively phrased)
Direct lookup (e.g., desired answer contained in obvious reference),
Question does NOT solicit single demonstrably correct answer,
Minimum response for passing credit NOT described in key,
Scope of question outside guidance of NUREG (e.g., why not how, TS bases not system, emergency not emergency plan),
4.
Check the appropriate box when a job content error is identified:
Topics not linked to job content (e.g., disguised task, not required in real job, invalid K/A).
Recall of too specific knowledge,
RO test items test at the SRO job level or vice versa,
Reverse logic or application compared to job.
5.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6.
Provide a brief description of problem in the explanation column.
- Page 14 -
PALO VERDE LSRO - NOVEMBER 2002 APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX (COMMENT RESOLUTION)
Scen 1
ES 2
TS 3
Crit 4
IC 5
Pred 6
TL 7
L/C 8
Eff 9
U/E/S 10 Explanation (See below for instructions) 1 S
The LSRO now has the associated paperwork for tracking fuel movement - ie fuel bundle ID number, location from, location to, core map (included as part of the initial cue) and events changed to make scenario more realistic.
2 S
The LSRO now has the associated paperwork for tracking fuel movement - ie fuel bundle ID number, location from, location to, core map (include as part of the initial cue) and events changed to make scenario more realistic.
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
1.
ES: ES-301 checklists 4, 5, & 6 satisfied.
3.
TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.
4.
Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
5.
IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
6.
Pred: Scenario sequence and other factors avoid predictability issues.
7.
TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
8.
L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
9.
Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
10.
Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
11.
Provide a brief description of problem in the explanation column
C:\\ORPCheckout\\FileNET\\ML030350659.wpd
- Page 15 -
PALO VERDE LSRO - NOVEMBER 2002 APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen 1
ES 2
TS 3
Crit 4
IC 5
Pred 6
TL 7
L/C 8
Eff 9
U/E/S 10 Explanation (See below for instructions) 1 U
U E
Initial conditions/Cue - Give the LSRO the associated paperwork for tracking fuel movement - ie fuel bundle ID number, location from, location to, core map (include as part of the initial cue). Make this as realistic as possible. Event 1 - Have them record the storage location of the fuel bundle as part of the required response for Step 2 of the scenario. Have the LSRO discuss resuming the fuel movement sequence that would lead up to Event 2 verses the examiner giving the cue. This would be more in line with performing in the plant. Start the cue when the LSRO describes that the containment refueling machine has lifted the fuel from the upender and is beginning to move the fuel to the designated core location. Need to cover Competency 5.
2 U
E Initial conditions/Cue - Give the LSRO the associated paperwork for tracking fuel movement - ie fuel bundle ID number, location from, location to, core map (include as part of the initial cue). Make this as realistic as possible. Cue - does the refueling machine have a fuel bundle suspended? Seems like the examiner cue indicates that the RTMZ is empty and ready to pickup the next bundle. The candidate cue needs to be clarified to specify that a bundle is suspended. Need to cover Competency 5.
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
1.
ES: ES-301 checklists 4, 5, & 6 satisfied.
12.
TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.
13.
Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
14.
IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
15.
Pred: Scenario sequence and other factors avoid predictability issues.
16.
TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
17.
L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
18.
Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
19.
Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
20.
Provide a brief description of problem in the explanation column
- Page 16 -
PALO VERDE LSRO - NOVEMBER 2002 ES-401 Written Examination (COMMENT RESOLUTION)
Form ES-401-9 (R8, S1)
Review Worksheet Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
H 3
U S
2 F
2 U
S Will provide complete TS and TRM to candidates.
3 F
3 U
S Revised Q.
4 H
3 U
S Enhanced the KA match.
5 H
3 U
S 6
H 3
U S
Clarified criticality in the comments section.
7 F
2 U
S 8
F 3
U S
N 9
F 3
U S
N. Completely revised Q.
10 H
3 U
S 11 H
3 U
S M.
12 F
2 U
S 13 H
2 U
S M
14 F
3 U
S N. Changed the to to or.
15 H
2 U
S 16 F
2 U
S 17 H
3 U
S N. Changed Distractor D to read No actuations occur.
18 H
3 U
S 19 F
2 U
S 20 F
3 U
S 21 H
2 U
S N.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- Page 17 -
22 H
3 U
S N
23 F
2 U
S N
24 H
3
S Changed Distractor C.
25 H
3 U
S N
26 H
3 U
S N
27 H
4 U
S N
28 H
3 U
S N. Changed to make a loss or malfunction.
29 H
3 U
S 30 F
3 U
S N
31 H
2 U
S 32 H
3 U
S New Q written and reviewed.
33 H
3 U
S 34 H
3 U
S N
35 H
3 U
S 36 H
3 U
S Wrote new Q.
37 H
3 U
S 38 F
2 U
S N
39 F
2 U
S 40 F
2 U
S 41 F
4 U
S Changed stem to state bridge/trolley.
42 F
2 U
S N
43 F
3 U
S N
44 F
3 U
S N. Added the word initially to stem.
45 H
3 U
S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- Page 18 -
46 F
2 U
S 47 H
3 U
S N
48 F
2 U
S 49 H
3 U
S 50 H
3 U
S N
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7.
At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
Legend: N = new question; M = Modified question; U in Q=K/A column indicates question reviewed for conformance to K/A
- Page 19 -
PALO VERDE LSRO - NOVEMBER 2002 ES-401 Written Examination Form ES-401-9 (R8, S1)
Review Worksheet Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
H 3
U S
2 F
2 U
S Will provide complete TS and TRM to candidates.
3 F
1 U
NO U
In essence the stem states failure of which one of the following would cause a power loss to the refueling machine. Appears to be a direct lookup. Will review; no reference is used.
4 H
3 NO E
Need to enhance the KA match.
5 H
3 U
S 6
H 3
U E
How ids the Q an indication of an approaching criticality? Will clarify in the comments section.
7 F
2 U
S 8
F 3
U S
N 9
F 3
NO U
N. Distractor D appears to be correct. Justify why Distractor A is not correct.. Will revise distractor.
10 H
3 U
S 11 H
3 U
S M. Provide original bank question.
12 F
2 U
S 13 H
2 U
S M
14 F
3 U
U E
N. Answer should be East or West. Distractors should be modified similarly. Will change the to to or.
15 H
2 U
S 16 F
2 U
S 17 H
3 U
E N. Change Distractor D to read No actuations occur. OK, will change.
18 H
3 U
S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- Page 20 -
19 F
2 U
S 20 F
3 U
S 21 H
2 U
S N
22 H
3 U
S N
23 F
2 U
S N
24 H
3 U
U E
Delete the word one in Distractor C. OK will change.
25 H
3 U
S N
26 H
3 U
S N
27 H
4 U
S N
28 H
3 NO E
N. Question does not appear to relate to a loss or malfunction of ECCS as stated in K/A. OK will change to make a loss of malfunction.
29 H
3 U
S 30 F
3 U
S N
31 H
2 U
S 32 H
3 NO E
How is knowledge of the refueling process covered by this question?
OK will rewrite.
33 H
3 U
S 34 H
3 U
S N
35 H
3 U
S 36 H
3 NO E
What does the Q have to do with containment isolation? OK will change.
37 H
3 U
S 38 F
2 U
S N
39 F
2 U
S 40 F
2 U
S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- Page 21 -
41 F
4 U
U E
The question is not clear. If it is meant that both the bridge and trolley are allowed to operate at fast speed, it should be clearly stated. OK will clarify.
42 F
2 U
S N
43 F
3 U
S N
44 F
3 U
S N. Should the Q state initially classifying? Yes, will revise.
45 H
3 U
S 46 F
2 U
S 47 H
3 U
S N
48 F
2 U
S 49 H
3 U
S 50 H
3 U
S N
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- Page 22 -
7.
At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
Legend: N = new question; M = Modified question; U in Q=K/A column indicates question reviewed for conformance to K/A