ML041060064

From kanterella
Jump to navigation Jump to search
07/2004 Draft NRC Comments
ML041060064
Person / Time
Site: Palo Verde  Arizona Public Service icon.png
Issue date: 07/21/2003
From: Gody A
Operations Branch IV
To: Overbeck G
Arizona Public Service Co
References
50-528/03-301, 50-529/03-301, 50-530/03-301, ES-401, ES-401-9 50-528/03-301, 50-529/03-301, 50-530/03-301
Download: ML041060064 (23)
v • d • e


Text

ES-401 Written Examination Form ES-401-9 Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 3 X X E Change stem to read:

"These operator actions will result in..."

2 H .. 3 X S (Modified) 3 H 3 X S 4 F 3 X S 5 F 2 X S 6 F 2 X X U Q-41 cues the correct answer.

7 H 3 X S 8 H 4 X X U (NEW) Distractor D is also correct 9 H 2 X S (NEW) 10 H 2 X S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ More than one distractor is not credible.

@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 3 X S 12 H 2 X S (Modified) 13 H 3 X S (Modified) 14 F 2 X S 15 F 3 X S 16 H 2 X S 17 F 3 X S (Modified) 18 F 2 X S 19 F 2 X S 20 F X X U With condition given in stem, all distractors are correct.

21 F 3 X S 22 F 2 X S 23 F 2 X S 24 H 3 X S (NEW) 25 H 2 X S 26 H 2 X S (NEW) 27 H 3 X S (NEW) 28 H 3 X S 29 H 2 X S 30 H 3 X S

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 31 F 3 X X U Rewrite or replace.

32 H 3 X S 33 F 3 X S 34 F 2 X S 35 F 2 X S 36 H 3 X S 37 H 3 X S (NEW) 38 H 2 X S 39 H 2 X S (Modified) 40 H 3 X S 41 F 2 X X U Q-6 cues the answer to this question.

42 H 3 X S 43 F 2 X S 44 F 2 X S 45 H 3 X S 46 H 3 X S 47 H 3 X S 48 F 2 X S 49 H 3 X S (NEW) 50 H 3 X S 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 2 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 51 F 2 X S 52 F 2 X S 53 F 2 X X U Collection of true/false statements.

Replace.

54 F 2 X S (NEW) 55 F 3 X S 56 H 2 X S 57 H 3 X S 58 H 3 X S 59 H 3 X S (Modified) 60 F 2 X S (NEW) 61 H 2 X S 62 H 3 X S (NEW) 63 H 3 x S 64 F 2 X S 65 H 2 X S (NEW) 66 F 2 X X U Distractors A and C are not credible, the name infers both valves be open.

67 H 3 X S (Modified) 68 H 3 X S 69 F 2 X S (NEW) 70 F 2 X S

ES-401 2 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 71 H 3 X S (NEW) 72 H 3 X S 73 F 3 X S (NEW) 74 F 2 X S 75 H 3 X S (NEW)

ES-401 Written Examination Form ES-401-9 Review Worksheet (Comment Resolution)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 3 X S Changed stem as suggested.

2 H .. 3 X S (Modified) 3 H 3 X S 4 F 3 X S 5 F 2 X S 6 F 2 X S Wrote new Q-6.

7 H 3 X S 8 H 4 X S (NEW) Changed stem to resolve problem.

9 H 2 X S (NEW) 10 H 2 X S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ More than one distractor is not credible.

@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 3 X S 12 H 2 X S (Modified) 13 H 3 X S (Modified) 14 F 2 X S 15 F 3 X S 16 H 2 X S 17 F 3 X S (Modified) 18 F 2 X S 19 F 2 X S 20 F X S Comment was incorrect- changed comment.

21 F 3 X S 22 F 2 X S 23 F 2 X S 24 H 3 X S (NEW) 25 H 2 X S 26 H 2 X S (NEW) 27 H 3 X S (NEW) 28 H 3 X S 29 H 2 X S 30 H 3 X S

ES-401 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 31 F 3 X S Rewrote Q.

32 H 3 X S 33 F 3 X S 34 F 2 X S 35 F 2 X S 36 H 3 X S 37 H 3 X S (NEW) 38 H 2 X S 39 H 2 X S (Modified) 40 H 3 X S 41 F 2 X S New Q-6 written.

42 H 3 X S 43 F 2 X S 44 F 2 X S 45 H 3 X S 46 H 3 X S 47 H 3 X S 48 F 2 X S 49 H 3 X S (NEW) 50 H 3 X S 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 51 F 2 X S 52 F 2 X S 53 F 2 X S Wrote new Q.

54 F 2 X S (NEW) 55 F 3 X S 56 H 2 X S 57 H 3 X S 58 H 3 X S 59 H 3 X S (Modified) 60 F 2 X S (NEW) 61 H 2 X S 62 H 3 X S (NEW) 63 H 3 x S 64 F 2 X S 65 H 2 X S (NEW) 66 F 2 X S Wrote new Q.

67 H 3 X S (Modified) 68 H 3 X S 69 F 2 X S (NEW) 70 F 2 X S

ES-401 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 71 H 3 X S (NEW) 72 H 3 X S 73 F 3 X S (NEW) 74 F 2 X S 75 H 3 X S (NEW)

ES-401 Written Examination Form ES-401-9 (R8, S1)

Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 X X S 2 H 3 X X S (NEW) 3 H 3 X X S 4 H 3 X X S 5 H 3 X x S (NEW) 6 H 3 X X S (NEW) 7 F 3 X X S (NEW) 8 F 3 X X S 9 H 3 X X S 10 H 2 X X S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ More than one distractor is not credible.

@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 3 X X S (NEW) 12 H 3 X X S (NEW) 13 H 3 X X S (Modified) 14 F 3 X X S 15 F 3 X X S (Modified) 16 H 3 X X S (NEW) 17 F 3 X X S 18 H 3 X X S 19 F 3 X X S 20 F 3 X X S 21 F 3 X X S 22 H 2 X X S 23 F 2 X X S 24 F 3 X X S 25 F 3 X X S

S:\DRS\OB\EXAMS\PV\PVNGS-INIT-EXAM 07-2003\PVNGS 07-2003 NRC DRAFT EXAM COMMENTS\APPE.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link JS1 D 3 S JS2 S 2 S JS3 S 2 S JS4 D 3 S JS5 D 3 S JS6 S 2 X E Why is Step 1 critical?

JS7 D 3 S Not marked as an alternate path JPM.

JS8 D 3 E Add for evaluator use that no actions are expected to be taken in Step 1.

JP01 S 3 X X E Not marked as an alternate path JPM. Why arent Steps 2 and 11 critical?

RO written exam questions 6 and 41 address the same subjects.

JP2 S 2 X E Steps 2 and 4 have the same cue.

- Page 1 -

S:\DRS\OB\EXAMS\PV\PVNGS-INIT-EXAM 07-2003\PVNGS 07-2003 NRC DRAFT EXAM COMMENTS\APPE.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link JP03 D 2 S RA1-1 S 2 S S/RA1-2 S 3 S RA2 S 2 E Piping diagram lists Tag 5 as OPEN. The correct position is CLOSED.

RA3 S 2 X E Does candidate determine what REP to use? If so, how is this done?

SA1-1 S 3 S SA2 S 2 S SA3 S 2 S SA4 S 3 E Steps 4 and 8 have time blanks, do both have to be completed within 15 minutes of Step 1 start time?

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests.

Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 2 -

S:\DRS\OB\EXAMS\PV\PVNGS-INIT-EXAM 07-2003\PVNGS 07-2003 NRC DRAFT EXAM COMMENTS\APPE (ComRev).wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX (COMMENT RESOLUTION)

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link JS1 D 3 S JS2 S 2 S JS3 S 2 S JS4 D 3 S JS5 D 3 S JS6 S 2 S Removed Step 1 as critical and combined Steps 7& 8 into one critical step.

JS7 D 3 S Marked as an alternate path JPM.

JS8 D 3 S Changed Step 1 to state that no action will be taken in this step.

JP01 S 3 S Marked as an alternate path JPM. Steps 2 and 11 made critical. Rewrote Q-6 in written exam.

JP2 S 2 S Changed cue in Step 4.

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 1 -

S:\DRS\OB\EXAMS\PV\PVNGS-INIT-EXAM 07-2003\PVNGS 07-2003 NRC DRAFT EXAM COMMENTS\APPE (ComRev).wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX (Comment Review)

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link JP03 D 2 S RA1-1 S 2 S S/RA1-2 S 3 S RA2 S 2 S Piping diagram changed to list Tag 5 as CLOSED.

RA3 S 2 S Added a statement so that evaluator knows what the correct actions are.

SA1-1 S 3 S SA2 S 2 S SA3 S 2 S SA4 S 3 S Changed Step 8 to clarify when 15 minute time will start.

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 2 -

S:\DRS\OB\EXAMS\PV\PVNG S-INIT-EXAM 07-2003\PVNGS 07-2003 NRC DR AFT EXAM C OMME NTS\APP N.wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen 1 2 3 4 5 6 7 8 9 10 Explanation (See below for instructions)

Set ES TS Crit IC Pred TL L/C Eff U/E/S 1 y y y y y y y y S 2 Y Y Y Y Y Y Y Y S 3 Y Y Y Y Y Y Y Y E Appears to be only one critical task.

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. ES: ES-301 checklists 4, 5, & 6 satisfied.
2. TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.
3. Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
4. IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
5. Pred: Scenario sequence and other factors avoid predictability issues.
6. TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
7. L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
8. Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
9. Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
10. Provide a brief description of problem in the explanation column

- Page 1 -

S:\DRS\OB\EXAMS\PV\PVNG S-INIT-EXAM 07-2003\PVNGS 07-2003 NRC DR AFT EXAM C OMME NTS\APP N (ComR ev).wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX (Comment Resolution)

Scen 1 2 3 4 5 6 7 8 9 10 Explanation (See below for instructions)

Set ES TS Crit IC Pred TL L/C Eff U/E/S 1 y y y y y y y y S 2 Y Y Y Y Y Y Y Y S 3 Y Y Y Y Y Y Y Y S Modified scenario to include additional Critical Tasks.

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. ES: ES-301 checklists 4, 5, & 6 satisfied.
2. TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.
3. Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
4. IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
5. Pred: Scenario sequence and other factors avoid predictability issues.
6. TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
7. L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
8. Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
9. Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
10. Provide a brief description of problem in the explanation column

- Page 1 -

PVNGS 2003 NRC License Examination Operating Examination Description of Modifications General Item Modification Source Date SRO Exam Question #44 and #48 were the same question. There were PV 7/11/03 as Given 101 questions on the exam. Question # 49 was used twice.

Question # 48 was deleted and the remaining numbers on the exam were resequenced in order. The exam now correctly shows 100 questions in their proper sequence. (75 questions from the RO test, 25 SRO questions distributed throughout to make the 100 question SRO Exam)

V = Changed as result of validation ER = Changed From NRC Chief Examiners review of examination submittal OR = Changed From NRC Chief Examiners review of outline submittal EV = Changed From NRC validation of examination materials PV = Palo Verde initiated change 1

PVNGS 2003 NRC License Examination Operating Examination Description of Modifications Written Examination (ES401-5) Modification Source Date Question #

RO 66 C The question has two invalid distractors. The stem EV 7/9/03 asked for valves or hatches. Two of the distractors are flanges.

R A new question was written to replace this one.

SRO 2 C The comment statement on the 401-5 form states that PV 7/10/03 at least 3 hrs has passed since the LOCA occurred while the question stem states that 2 and a half hours has passed since the LOCA started.

R The comment statement should read that at least 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> has passed.

RO 15 C The correct answer needs to be changed to state: must PV 7/10/03 immediately trip the reactor instead of immediately trip the turbine to match the AOP requirement.

R The answer has been revised as noted above.

RO 47 C Need to clarify why PCN-V118 can not be open unless PV 7/10/03 the Refuel Pool level and the Fuel Canal level are both

>131 ft. 10 inches actual level.

R The ES-401-5 comment area has been revised as noted above.

RO 52 C Clarified question stem to state: Which ONE of the PV 7/10/03 following is true regarding Spent Fuel Pool Cooling related Control Room Board or Recorder indications?

To say: Which ONE of the following is true regarding Spent Fuel Pool Cooling as related to Control Room Board or Recorder indications?

R The ES-401-5 comment area has been revised as noted above.

V = Changed as result of validation ER = Changed From NRC Chief Examiners review of examination submittal OR = Changed From NRC Chief Examiners review of outline submittal EV = Changed From NRC validation of examination materials PV = Palo Verde initiated change 2

PVNGS 2003 NRC License Examination Operating Examination Description of Modifications Administrative Walkthrough Admin # Modification Source Date SA3 Changed JPM reference to KA catalog # 2.3.6 and associated PV 5/1/03 SRO Importance level to 3.1. This KA number better reflects the JPM task standard.

V = Changed as result of validation ER = Changed From NRC Chief Examiners review of examination submittal OR = Changed From NRC Chief Examiners review of outline submittal EV = Changed From NRC validation of examination materials PV = Palo Verde initiated change 3

PVNGS 2003 NRC License Examination Operating Examination Description of Modifications Job Performance Measure JPM # Modification Source Date V = Changed as result of validation ER = Changed From NRC Chief Examiners review of examination submittal OR = Changed From NRC Chief Examiners review of outline submittal EV = Changed From NRC validation of examination materials PV = Palo Verde initiated change 4

PVNGS 2003 NRC License Examination Operating Examination Description of Modifications Operating (Scenario) Test Scenario # Modification Source Date 2 Combined event 1 & 2 into a single event. Eliminated credit EV 7/3/03 for a Component Malfunction. No operator action was required for this previous 2nd event (DG Trip).

V = Changed as result of validation ER = Changed From NRC Chief Examiners review of examination submittal OR = Changed From NRC Chief Examiners review of outline submittal EV = Changed From NRC validation of examination materials PV = Palo Verde initiated change 5

Retrieved from "https://kanterella.com/w/index.php?title=ML041060064&oldid=2622967"