ML030640302

August 2002 - Draft Operating Exam Comments

ML030640302
Person / Time
Site:	Arkansas Nuclear
Issue date:	09/18/2002
From:	Gody A Operations Branch IV
To:	Anderson C Entergy Operations
References
50-313/02-302 50-313/02-302
Download: ML030640302 (10)

Text

C:\\MYFILES\\Copies\\AN 8-2002 draft operating exam comments.wpd

- Page 1 -

APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX-COMMENT RESOLUTION JPM#

1.

Dyn (D/S) 2.

LOD (1-5)

3. Attributes

4. Job Content Errors 5.

U/E/S 6.

Explanation (See below for instructions)

IC Focus Cues Critical Steps Scope (N/B)

Over-lap Job-Link Minutia RO-RCS2 S

3 S

RO-SURV2 S

3 S

RO-RAD1 S

2 S

RO-COMM2 S

2 S

SRO-QUAL1 S

2 S

SRO-PROC1 S

2 S

SRO-MNTC1 S

2 S

CRD03 D

3 S

STEPS 2 -11 ARE CRITICAL EOP-10 S

3 S

MFW04 D

3 S

STEP 7 IS NOT CRITICAL - STEP REVISED AOP-29 D

2 S

STEP 4 IS CRITICAL QT-001 D

2 S

EOP-09 S

2 S

STEP 1 IS CRITICAL ICW-01 D

3 S

STEP 4 WAS REVISED TO BE CONSISTENT WITH PROCEDURE.

CRD-04 S

2 S

STEPS 4 & 5 ARE NOT CRITICAL - STEP REVISED AOP-07 S

2 S

STEPS 1 & 6 ARE NOT CRITICAL - STEP REVISED EDO-19 S

3 S

STEP 1 IS CRITICAL

- Page 2 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1.

Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.

2.

Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.

3.

Check the appropriate box when an attribute weakness is identified:

The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.

The JPM does not contain sufficient cues that are objective (not leading).

All critical steps (elements) have not been properly identified.

Scope of the task is either too narrow (N) or too broad (B).

Excessive overlap with other part of operating test or written examination.

4.

Check the appropriate box when a job content error is identified:

Topics not linked to job content (e.g., disguised task, not required in real job).

Task is trivial and without safety significance.

5.

Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

6.

Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

C:\\MYFILES\\Copies\\AN 8-2002 draft operating exam comments.wpd

- Page 3 -

APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX JPM#

1.

Dyn (D/S) 2.

LOD (1-5)

3. Attributes

4. Job Content Errors 5.

U/E/S 6.

Explanation (See below for instructions)

IC Focus Cues Critical Steps Scope (N/B)

Over-lap Job-Link Minutia RO-RCS2 S

3 S

RO-SURV2 S

3 S

RO-RAD1 S

2 S

RO-COMM2 S

2 S

SRO-QUAL1 S

2 S

SRO-PROC1 S

2 S

SRO-MNTC1 S

2 S

CRD03 D

3 S

ARE STEPS 2 -11 ALL CRITICAL?

EOP-10 S

3 S

MFW04 D

3 S

WHY IS STEP 7 CRITICAL?

AOP-29 D

2 S

IS STEP 4 CRITICAL?

QT-001 D

2 S

EOP-09 S

2 S

IS STEP 1 CRITICAL?

ICW-01 D

3 S

STEP 4 APPEARS TO BE INCONSISTENT WITH PROCEDURE.

CRD-04 S

2 S

ARE STEPS 4 & 5 CRITICAL?

AOP-07 S

2 S

ARE STEPS 1 & 6 CRITICAL?

EDO-19 S

3 S

IS STEP 1 A CRITICAL STEP?

- Page 4 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1.

Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.

2.

Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.

3.

Check the appropriate box when an attribute weakness is identified:

The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.

The JPM does not contain sufficient cues that are objective (not leading).

All critical steps (elements) have not been properly identified.

Scope of the task is either too narrow (N) or too broad (B).

Excessive overlap with other part of operating test or written examination.

4.

Check the appropriate box when a job content error is identified:

Topics not linked to job content (e.g., disguised task, not required in real job).

Task is trivial and without safety significance.

5.

Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

6.

Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

C:\\MYFILES\\Copies\\AN 8-2002 draft operating exam comments.wpd

- Page 5 -

APPENDIX F - REGION IV OPERATING TEST QUESTION QUALITY REVIEW MATRIX-COMMENT RESOLUTION Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Errors

4. Job Content Errors 5.

U/E/S 6.

Explanation (See below for instructions)

Stem Focus Direct L/U One Ans Min Resp Scope (TS?)

Job-Link Minu-tia SRO Only Back-wards RO-A1Q1 F

2 S

RO-A1-Q2 F

2 S

ANSWER AND REFERENCE PROVIDED IN THE JPM.

SRO-A3-Q1 F

3 S

SRO-EAL7 H

3 S

INCLUDE APPLICABLE SECTIONS FROM ATT 1 & 3 OF PROC 1903.010.

APPENDIX F - REGION IV OPERATING TEST QUESTION QUALITY REVIEW MATRIX-COMMENT RESOLUTION Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Errors

4. Job Content Errors 5.

U/E/S 6.

Explanation (See below for instructions)

Stem Focus Direct L/U One Ans Min Resp Scope (TS?)

Job-Link Minu-tia SRO Only Back-wards

- Page 6 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it.

The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1.

Classify level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.

2.

Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.

3.

Check the appropriate box when a psychometric error is identified:

Stem lacks sufficient focus to solicit only the answers listed (e.g., unclear on intent, answer needed, or unnecessarily negatively phrased)

Direct lookup (e.g., desired answer contained in obvious reference),

Question does NOT solicit single demonstrably correct answer, Minimum response for passing credit NOT described in key, Scope of question outside guidance of NUREG (e.g., why not how, TS bases not system, emergency not emergency plan),

4.

Check the appropriate box when a job content error is identified:

Topics not linked to job content (e.g., disguised task, not required in real job, invalid K/A).

Recall of too specific knowledge, RO test items test at the SRO job level or vice versa, Reverse logic or application compared to job.

5.

Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

6.

Provide a brief description of problem in the explanation column.

C:\\MYFILES\\Copies\\AN 8-2002 draft operating exam comments.wpd

- Page 7 -

APPENDIX F - REGION IV OPERATING TEST QUESTION QUALITY REVIEW MATRIX Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Errors

4. Job Content Errors 5.

U/E/S 6.

Explanation (See below for instructions)

Stem Focus Direct L/U One Ans Min Resp Scope (TS?)

Job-Link Minu-tia SRO Only Back-wards RO-A1Q1 F

2 S

RO-A1-Q2 F

2 U

NEED ANSWER IN THE JPM.

REFERENCE NOT PROVIDED.

SRO-A3-Q1 F

3 S

SRO-EAL7 H

3 S

INCLUDE APPLICABLE SECTIONS FROM ATT 1 & 3 OF PROC 1903.010.

APPENDIX F - REGION IV OPERATING TEST QUESTION QUALITY REVIEW MATRIX Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Errors

4. Job Content Errors 5.

U/E/S 6.

Explanation (See below for instructions)

Stem Focus Direct L/U One Ans Min Resp Scope (TS?)

Job-Link Minu-tia SRO Only Back-wards

- Page 8 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it.

The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1.

Classify level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.

2.

Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.

3.

Check the appropriate box when a psychometric error is identified:

Stem lacks sufficient focus to solicit only the answers listed (e.g., unclear on intent, answer needed, or unnecessarily negatively phrased)

Direct lookup (e.g., desired answer contained in obvious reference),

Question does NOT solicit single demonstrably correct answer, Minimum response for passing credit NOT described in key, Scope of question outside guidance of NUREG (e.g., why not how, TS bases not system, emergency not emergency plan),

4.

Check the appropriate box when a job content error is identified:

Topics not linked to job content (e.g., disguised task, not required in real job, invalid K/A).

Recall of too specific knowledge, RO test items test at the SRO job level or vice versa, Reverse logic or application compared to job.

5.

Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

6.

Provide a brief description of problem in the explanation column.

C:\\MYFILES\\Copies\\AN 8-2002 draft operating exam comments.wpd

- Page 9 -

APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX -COMMENT RESOLUTION Scen Set 1

ES 2

TS 3

Crit 4

IC 5

Pred 6

TL 7

L/C 8

Eff 9

U/E/S 10 Explanation (See below for instructions) 1 S

TS ACTIONS ADDED, FORM FORMAT CORRECTED, TURNOVER INFO MADE CONSISTENT.

2 S

TS ACTIONS ADDED AND TURNOVER INFO MADE CONSISTENT.

3 S

TS ACTIONS ADDED, OUTLINE EVENTS 6 & 7 MADE CONSISTENT WITH SCENARIO, AND TURNOVER INFO REVISED.

4 S

TS ACTIONS ADDED, TURNOVER INFO REVISED. ONLY A BACK-UP IF SCENARIO 3 NOT RUN.

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1.

ES: ES-301 checklists 4, 5, & 6 satisfied.

2.

TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.

3.

Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.

4.

IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.

5.

Pred: Scenario sequence and other factors avoid predictability issues.

6.

TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.

7.

L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.

8.

Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.

9.

Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

10.

Provide a brief description of problem in the explanation column

C:\\MYFILES\\Copies\\AN 8-2002 draft operating exam comments.wpd

- Page 10 -

APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen Set 1

ES 2

TS 3

Crit 4

IC 5

Pred 6

TL 7

L/C 8

Eff 9

U/E/S 10 Explanation (See below for instructions) 1 NO E

NO TS ACTIONS INCLUDED FOR CRS. FORM FORMAT NEEDS CORRECTION.

TURNOVER INFO NOT CONSISTENT.

2 NO E

NO TS ACTIONS INCLUDED FOR CRS. INITIAL CONDITIONS/TURNOVER INCONSISTENT.

3 NO E

NO TS ACTIONS INCLUDED FOR CRS. OUTLINE EVENTS 6 & 7 INCONSISTENT WITH SCENARIO. TURNOVER ON OUTLINE INCOMPLETE.

4 NO E

NO TS ACTIONS INCLUDED FOR CRS. TURNOVER ON OUTLINE INCOMPLETE.

TOO SIMILAR TO SCENARIO 3.

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1.

ES: ES-301 checklists 4, 5, & 6 satisfied.

2.

TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.

3.

Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.

4.

IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.

5.

Pred: Scenario sequence and other factors avoid predictability issues.

6.

TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.

7.

L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.

8.

Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.

9.

Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

10.

Provide a brief description of problem in the explanation column