ML061560086

From kanterella
Jump to navigation Jump to search
CW-08-2005-Draft Exam Comments
ML061560086
Person / Time
Site: Callaway Ameren icon.png
Issue date: 03/10/2005
From:
NRC Region 4
To:
References
Download: ML061560086 (10)
v • d • e


Text

E:\Filenet\ML061560086.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link P1 S 3 S Combines elements of Admin and field actions. OK.

P2 S 3 S P3 S 3 S Involves acquiring material to perform. Validation should include close scrutiny of actions.

S1 D 3 S S2 D 3 S S3 D 3 S S4 D 3 S S5 D 3 S S6 D 3 S May require some time compression.

S7 D 3 S S8 D 3 S Has a lot of critical steps.

- Page 1 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 2 -

E:\Filenet\ML061560086.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link RO-A2 N/A 3 S S/RO- N/A 3 S A1/A2 RO-A3 N/A 3 S S/RO-A1-1 N/A 3 S

- Page 3 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 4 -

E:\Filenet\ML061560086.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link ROSRO 3 S A1-2 N/A RO A-3 N/A 3 S SRO A-2 N/A 3 S SRO A-4 N/A 3 S ROSRO N/A 3 S A1-1

- Page 5 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 6 -

ES-401 CW-08-2005 Written Examination Form ES-401-9 Review Worksheet (password: redsox -or- memrbm; TAC#X02286)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 2 - S 2 F 3 - S 3 H 2 - S 4 H 3 - S 5 F 3 - S 6 H 3 X - U all answers seem correct; 7 H 3 X - U D not credible since B is subset of D; 8 H 3 - S 9 H 3 - S 10 H 3 - S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ One or more distractors is (are) not credible.

@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

33 of 34 NUREG-1021, Draft Revision 9

- Page 7 -

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 11 F 3 X - E change stem to Which ONE (1) of the following is true regarding power to SSPS B input relays? Power is supplied by:

12 H 3 X - E references do not support answer; 13 F 3 - S 14 F 2 - S 15 H 3 - S 16 H 3 X E explanation for B doesnt make sense; KA on Q doesnt match outline; 17 F 3 X X U stem doesnt say whether the CSAS actuation is spurious or not; doesnt test knowledge of operator responsibilities regarding CCS per KA; 18 H 3 X - U is there a correct answer? Wont the control system close the other steam dumps and maintain Tave and Power about constant?

19 H 3 - E change will to to in stem; 20 H 3 - S 21 H 3 - E change stem to synchronized and loaded...;

22 F 3 - S 23

  • H 3 - E doesnt qualify as modified - must be counted as bank; typo in KA block of Q; 24 H 3 X X X U how does the reader know HS-A isnt in ISO/RUN? D is not credible; doesnt test knowledge of the effect of loss/ malfunction of the FUEL OIL STORAGE TANKS will have on the EDG per KA; 25 H 3 X - E A is not credible since D is a subset of it; 26 H 2 X X U A, B, and D are not credible (subsets); doesnt test knowledge of connection/relationship btwn SWS and its EMERGENCY HEAT LOADS per KA; explanation doesnt match question; 27 H 3 X U doesnt test knowledge of IAS features/interlocks providing for MANUAL/

AUTOMATIC TRANSFERS OF CONTROL per KA; 28 F 3 - S 34 of 34 NUREG-1021, Draft Revision 9

- Page 8 -

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 29 H 3 X - E do the falling rods result in a trip?

30 H 3 - E stem says no action by the crew - so how is plant load reduced?;

indicate that 17% in C is PZR Level; 31 F 3 - S 32 H 2 - E typo in KA block of Q; 33 F 3 - S 34 H 2 X - U A, C, and D are not credible; explanation of A doesnt match A; 35 F 3 - S 36 F 3 - S 37 F 3 X U doesnt test knowledge of effect of loss/malfunction of WGDS will have on ARM and PRM per KA; 38 F 3 - S 39 H 3 - S 40 H 2 - S 41 H 3 - S 42 H 3 - S 43 H 2 X - U B and C are not credible; 44 H 3 - S 45 H 2 X X U A and B are not credible; doesnt test ability to determine/interpret TS limits for RCS pressure per KA; 46 F 3 - S 47 H 3 X - E add only to A; 48 H 3 X - E add only to B; 34 of 34 NUREG-1021, Draft Revision 9

- Page 9 -

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 49 H 3 - S 50 H 3 - S 51 ? H 3 - E Q called new, but appears to have been modified from attached parent; 52 H 3 - S 53 H 3 - S 54 F 2 - S 55 H 3 X X - U change B to ...to prevent pump cavitation.; handout cues answer; 56 F 3 X U doesnt test ability to OPERATE/MONITOR desired operating results during steam line rupture per KA; 57 F 3 - E change D to ...acknowledged only if...;

58 F 2 - S 59 F 2 X U doesnt test knowledge of biological effects and units for exposure per KA; 60 H 3 X X U uses backward logic; doesnt test knowledge of reasons for coolant chemistry effects during rediagnosis per KA; 61

  • F 3 - E doesnt count as modified - must count as bank; change stem to

...blocking the low steamline pressure SI; 62 F 3 - S 63 F 3 - S 64 F 3 - S 65 H 3 - S 66 F 2 - S 34 of 34 NUREG-1021, Draft Revision 9

- Page 10 -

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 67 F 2 - E explanation of B doesnt match B; 68 F 3 - S 69 H 3 - S 70 H 2 - E doesnt count as modified - must count as bank; 71 H 3 - S 72 H 3 - S 73 F 2 - S 74 H 3 - S 75 F 2 X - U C and D not credible; 76 H 3 X X X - U KA on Q doesnt match KA on outline; handout makes B and D not credible; 77 F 2 - - S 78 H 2 - - S 79 H 3 X X - X U handout makes B and D not credible; this is RO-level knowledge (55.41.b.10);

80 H 3 - - E typo in KA block; is the stem missing the word sink?;

81 H 3 - - S 82 H 2 X - E KA on Q doesnt match KA on outline; 83 H 3 - - S 84 H 3 X X U KA on Q doesnt match KA on outline; this is RO-level knowledge; 85 H 3 X X - - U cannot tell the difference between A and B; D is not credible (reference to RWST alarm not mentioned in stem);

34 of 34 NUREG-1021, Draft Revision 9

- Page 11 -

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 86 H 2 - - S 87 H 3 X - - E edit B and D to make them identical except for initiating SI; 88 H 4 X - - E is this required to be known by memory?

89 H 3 - X U this is RO-level; 90 F 2 X - X U C and D are not credible; this is RO-level; 91 H 2 X - X U D not credible (recommend replacing with actions for 2 rods misaligned within a group);

explanation of A is incorrect; this is RO-level; 92 F 2 - - E add only to A and D; 93 H 3 X - - U A is not credible; 94 H 3 - - S 95 F 3 - - S 96 H 3 X - - U ODP handout not provided; TS handout provides direct lookup; 97 F 2 - X U this is RO-level (55.41.b.13) 98 H 2 - X U this is RO-level (55.41.b.12) 99 H 3 - - S 100 F 2 - - S 27 45H 3X 1X 11X 4X 1X 10X 16U RO TOTALS 6 30F 18E 10 19H 3X 6X 2X 1X 3X 7X 11U SRO TOTALS 1 6F 5E

GENERAL COMMENT

S:

1.

NOTE: Bank questions are indicated by ; Modified are indicated by  ;

comment resolution is indicated in blue; 34 of 34 NUREG-1021, Draft Revision 9

- Page 12 -

Retrieved from "https://kanterella.com/w/index.php?title=ML061560086&oldid=2190866"