ML040710225

Init Exam - 08/2003 - Appendix F, Draft Operating Exam Comments

ML040710225
Person / Time
Site:	Waterford
Issue date:	02/14/2005
From:	NRC Region 4
To:
References
50-382/03-301 50-382/03-301
Download: ML040710225 (1)

Text

E:\Filenet\ML040710225.wpd APPENDIX F - REGION IV OPERATING TEST QUESTION QUALITY REVIEW MATRIX

1. 2. 3. Psychometric Errors 4. Job Content Errors 5. 6.

Q# LOK LOD Explanation (F/H) (1-5) Stem Direct One Min Scope Job- Minu- SRO Back- U/E/S (See below for instructions)

Focus L/U Ans Resp (TS?) Link tia Only wards Admin H 3 s A1.1 A1.2 h 3 s A2 h 3 s A3 h 2 s A4 h 3 s Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it.

The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Classify level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.

2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.

3. Check the appropriate box when a psychometric error is identified:

• Stem lacks sufficient focus to solicit only the answers listed (e.g., unclear on intent, answer needed, or unnecessarily negatively phrased)

• Direct lookup (e.g., desired answer contained in obvious reference),

• Question does NOT solicit single demonstrably correct answer,

• Minimum response for passing credit NOT described in key,

• Scope of question outside guidance of NUREG (e.g., why not how, TS bases not system, emergency not emergency plan),

4. Check the appropriate box when a job content error is identified:

• Topics not linked to job content (e.g., disguised task, not required in real job, invalid K/A).

• Recall of too specific knowledge,

• RO test items test at the SRO job level or vice versa,

• Reverse logic or application compared to job.

5. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

6. Provide a brief description of problem in the explanation column.

- Page 1 -