ML023100083

Draft - NRC Comments

ML023100083
Person / Time
Site:	Limerick
Issue date:	10/04/2002
From:	Bissett P NRC/RGN-I/DRS/OSB
To:	Wasong A Exelon Nuclear
Conte R
References
50-352/02-302, 50-353/02-302
Download: ML023100083 (8)

Text

LIMERICK INITIAL EXAM - OCTOBER 2002 ES-401 Written Examination Form ES-401-9 Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD ............

(FIH) (1-5) Stem ~Cues T/F Cred. ~Partial Job- Minutia, #/ Back- S~RO UIE/S Explanation

-Focusj __ _Dist. j Link _ _junits ward W/A JOnly =S __________________

1 F 2 E Include applicable procedure in stem. OK 2 F 2 S 3 F 2 X E Shouldn't upscale be included in all distractors? OK 4 H 3 S 5 F 3 E Is "as indicated" in stem really needed? YES, BUT ENHANCEMENT DONE 6 F 2 S 7 F 2 S 8 F 3 S 9 H 2 S 10 H 2 S

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem ICues IT/F Cred. IPartiall Job iui 1Bc-Q R // Explanation Focus Dist. Link units I ward K/A Only Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level,

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements, More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check the appropriate box if the sampled question does not match the approved K/A or an SRO-only question is not at the SRO level.

6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. For any "U"ratings, at a minimum, explain how the Appendix B psychometric attributes are not being met.

NUREG-1021, Revision 8, Supplement 1 44 of 46 I

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cue T/F Cred. Pataro-Mnta# ak- Q, 1SROj U/E/SExlnto IFocus Dist. Link units ward KWA Only 11 F 2 E Stem 12 H 3 S 13 F 2 e Nothing said about procedures? ADDED T-200 PROCEDURES 14 F 2 S 15 H 3 S 16 F 3 E Stem OK 17 H 3 S 18 F 2 S 19 F 3 S 20 F 2 S 21 F 3 S 22 F 2 S 23 H 3 S 24 H 3 S 25 F 2 S 26 F 2 S 27 F 3 E Stem OK 28 F 3 S 29 H 2 S 30 F 2 S 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem ICues T/F ICred. Partial Job- jMinutia #/ Back- Q~ SRO UIEIS Explanation FOcus I I I Dist. Link - J Iunits ward K./A Only 31 H 3 S 32 H 3 S 33 F 2 E 34 H 2 E Stem OK 35 F 2 S 36 H 2 S 37 F 3 S 38 F 3 X X U Distractors a and b easily eliminated. REVISED 39 F 2 S 40 F 3 S 41 F 3 S 42 F 2 S 43 F 3 S 44 H 2 S 45 H 2 E 46 H 2 E 47 H 3 S 48 H 3 S 49 F 2 S 50 F 2 S 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Ste ICues ITI/FqCred. Partial Job- Minutia #/ Back- Q, ISRO I UIEIS Explanation Foui Dst[ Link ~ unitsjward tK/A jOnlyJ _____________________

51 F 2 S 52 H 2 E Justification sheet.

53 F 2 S 54 F 2 S 55 H 3 S 56 H 2 S 57 H 2 S 58 F 2 S 59 H 2 S 60 H 3 E 61 F 2 S 62 H 2 S 63 H 3 S 64 H 3 E 65 H 3 S 66 F 2 E 67 F 2 S 68 H 2 s 69 F 2 S 70 H 2 E 45 of 46 NUREG-1021, Revision 8, Supplement 1

2 Form ES-401-9 ES-401 Q# LOK 1.

(F/H)

LOD 2.

(1-5)

3. Psychometric Flaws Stem Cues T/F I

4. Job Content Flaws

1. / Back-Cred. Partial Job- Minutia units

5. Other Q- SRO 6.

U/E/S 7.

Explanation

- IDist. - __Focus Link ward K/A Only 71 F 2 S 72 H 2 S 73 F 3 E IAW OK 74 H 3 E 75 H 2 S 76 H 3 E 77 F 2 S 78 F 3 E 79 H 2 E 80 F 1 E 81 H 2 S 82 F 2 S 83 H 3 S 84 H 2 S 85 F 2 E 86 H 2 E 87 H 3 S 88 H 2 S 89 H 2 X U Good question but Attachment dead give-away to the answer.

QUESTION TO BE REPLACED.

90 F 3 S 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/Fl Cred. lPartial Job- Minutia #/ Back- SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 91 F 2 S 92 F 2 S 93 F 2 S 94 F 3 S 95 H 2 E 96 F 2 S 97 F 2 S 98 H 2 S 99 F 2 S 100 F 2 S S R 0 0 N L y Generic: Attachments(T-procedures) need not be listed at beginning of question.

"SELECTEDTRIP PROCEDURES ARE PROVIDED" IS NEW VERBAGE.

51 H 2 E Answer Key page 52 H 2 E 53 F 2 S 54 H 3 S 55 F 3 S 56 H 3 E Remove T-1 16 statement, also editorial. OK 57 F 2 S 58 F 2 E 59 F 2_____ I E 45 of 46 NUREG-1021, Revision 8, Supplement 1

Form ES-401-9 ES-401

4. Job Content Flaws 5FOther 6. 7.

1. 2. 3. Psychometric Flaws Explanation Q# H LO)

(OK SelusTF 'Cred. Partial Job- MinutiaI #/ IBack- Q, ISRO U/E/S 1 I Dist. Link Iunits ward K/A Only

) )Focus 3 X S 60 H 3 E 61 F E Remove T-103.104 OK 62 H 3 3 S 63 F 3 S 64 F 2 S 65 H 3 S 66 H 2 E Remove T-1 11 OK 67 H E Remove T-102 OK 68 F 3 E Remove T-117 OK 69 H 2 2 S 70 H 2 E 71 F 2 S 72 H 2 E Remove T-1 02 OK 73 H 2 S 74 H 3 E 75 F 45 of 46 NUREG-1021, Revision 8, Supplement I