ML021610211

NRC Comments

ML021610211
Person / Time
Site:	Oyster Creek
Issue date:	05/31/2002
From:	Bissett P NRC/RGN-I/DRS/OSB
To:	Hackenberg J AmerGen Energy Co
References
50-219/02301
Download: ML021610211 (6)

Text

ES-401 Written Examination Form ES-401-9 Review Worksheet

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD jjI-T-(FIH)

(1-5)

Stem CuesI T/F Cred. Partial Job-Minutia

1. /

Back-Q,, SRO U/EJS Explanation Focus Dist.

Link units ward K/A Only 1

H 3

S 2

H 2

E 3

H 3

S 4

F 2

E 5

F 2

X U/E A.) Is a possible correct answer. Editorial corrections stem and distractors. OK 6

H 2

E Stem-based upon above conditions (BUAC) Unnecessary stem info.OK 7

F 2

E 8

F 2

E "lAW the EOP users guide" is needed. Stem focus?OK 9

H 2

S editorial on answer key OK 10 F

3 S

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 -4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check the appropriate box if the sampled question does not match the approved K/A or an SRO-only question is not at the SRO level.

6.

Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7.

For any "U" ratings, at a minimum, explain how the Appendix B psychometric attributes are not being met.

NUREG-1021, Revision 8, Supplement 1 44 of 46

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues TIF T Cred.'Partial Job-Minutia

1. /

Back-Q*

SRO U/E/S Explanation J j Focus JDist.

Lnk

____junits ward K/A Only ___

11 F

2 S

12 F

2 S

13 F

3 S

14 H

3 S

15 H

2 S

16 F

3 E

Stem focus? OK 17 F

2 S

18 H

2 S

19 F

3 S

20 H

3 X

U OK AFTER FOLLOUUP, NOT UNSAT 21 H

2 E

BUAC 22 H

3 S

23 F

3 E

24 H

2 E

BUAC; Answer key comment on justification.

25 F

2 S

Just a comment - I don't like this type of question - trick like question 26 F

2 S

27 F

2 E

28 H

3 E

29 F

3 E

BUAC 30 H

2 S

NUREG-1021, Revision 8, Supplement 1 45 of 46

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD I(F/H) 1(1-5)

Stem Cues TIF 1Cred. Partial Job-IMinutia

1. / Back-Q*SOUIEIS Explanation Focus Dist.

Link units ward K/A.Only 31 H

3 S

BUAC 32 F

2 S

33 H

3 S

BUAC 34 H

3 S

BUAC 35 F

2 S

36 H

3 E

BUAC, etc 37 H

3 E

BUAC 38 H

2 S

39 H

3 E

BUAC 40 F

2 E

41 F

2 E

42 H

3 X

E BUAC, Do not understand justification explanation OK 43 H

2 S

44 F

2 S

45 F

1 U

Essentially T/F between a) and b) OK 46 H

2 E

BUAC 47 H

2 S

48 H

3 E

BUAC 49 H

3 E

BUAC, question on time for C and D justifications 50 F

2 S

NUREG-1021, Revision 8, Supplement 1 45 of 46

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues TIF Cred. Partial Job-IMinutial # /

Back-QI SRO UEFS Explanation FocUS jDist.

Link units ward K/A.Only 51 F

3 S

52 H

3 E

BUAC 53 H

3 S

54 H

3 E

BUAC 55 F

3 X

E Distractors have more information than is requested In stem. OK 56 H

3 E

BUAC 57 F

3 E

BUAC 58 H

2 S

59 H

3 E

BUAC 60 F

2 E

BUAC, ETC 61 H

3 E

BUAC, Some question on Rx press and reseat of EMRV and SV?

62 F

2 E

63 H

2 E

BUAC 64 F

3 S

65 F

3 E

BUAC 66 F

2 E

67 H

3 E

BUAC 68 F

3 E

BUAC 69 F

3 X

U No correct answer unless assumption is made for C.)CORRECTIONS MADE 70 F

3 E

BUAC NUREG-1021, Revision 8, Supplement 1 45 of 46

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues TlF Cred. Partial Job-Minutia

1. / IBack-Q-

SRO UIEIS Explanation Focus Dist.

Link units ward K/A Only 71 H

2 E

BUAC, ETC.

72 H

2 E

73 F

3 E

BUAC 74 H

3 E

75 F

2 S

76 F

2 S

77 H

3 E

BUAC 78 H

3 E

BUAC 79 F

2 X

U BUAC OK - ADDED DC TO ANOTHER DISTRACTOR 80 F

3 S

81 H

3 E

BUAC 82 F

2 S

83 H

3 E

BUAC 84 H

3 E

BUAC 85 H

3 E

BUAC 86 H

2 E

BUAC 87 H

2 S

88 H

3 E

BUAC 89 H

2 S

90 H

2 S

NUREG-1021, Revision 8, Supplement 1 45 of 46

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues TIF Cred. Partial Job-Minutia

1. /

Back-Q-SRO UIEIS Explanation Focus Dist.

Link units ward K/A Only 91 F

3 S

92 F

3 S

93 H

3 S

94 F

2 S

85 H

3 S

96 H

3 S

97 H

3 S

98 H

3 E

BUAC, ETC.

99 H

3 E

BUAC 100 F

3 S

NUREG-1021, Revision 8, Supplement 1 45 of 46