ML120880625

From kanterella
Jump to navigation Jump to search
2012-03-DRAFT-Written Exam Comments
ML120880625
Person / Time
Site: Diablo Canyon  Pacific Gas & Electric icon.png
Issue date: 03/09/2012
From:
Division of Reactor Safety IV
To:
Pacific Gas & Electric Co
References
DC-2012-01, DC-2012-03, ES-401, ES-401-9
Download: ML120880625 (5)


Text

ES-401 DC-2012-01 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 5 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws
4. Job Content Flaws
5. Other
6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

  1. /

units Back-ward Q=

K/A SRO Only 1

H 3

N S

2 H

3 X

X N

ES Remove 25% from stem Removed 3

F 2

N S

4 H

3 X

N ES Add valve name; A&B explanations backward Valve name added, A&B revised 5

H 3

X X

N ES Add stem focus; make B&C more credible Stem and B&C modified 6

H 3

X X

B ES Add stem and distracter focus Added 7

H 2

X N

ES A not credible Replaced A 8

H 2

X N

US C only answer plausible Modified Q to make distracters more plausible 9

H 2

X N

ES Add stem focus Moved to Q 34 - Replaced with new Q 10 H

2 X

X N

US Got correct answer w/o reading Q Modified distracters to remove problem

ES-401 DC-2012-01 2

Form ES-401-9 Page 2 of 5 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws
4. Job Content Flaws
5. Other
6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

  1. /

units Back-ward Q=

K/A SRO Only 11 H

3 X

N US Two correct answers Modified Q to ensure only one correct answer 12 H

2 X

X M

ES Modify all distracters to make Q plausible Distracters modified 13 F

2 X

N ES Modify D&C to be consistent with A&B Modified 14 H

2 X

N ES Modify stem Modified 15 H

2 X

X N

ES Modify stem and distracters Modified 16 H

2 X

M ES Get rid of unnecessary info in stem Modified stem 17 H

2 X

B ES Add the in stem Added 18 F

3 N

S 19 H

3 X

N ES Modify stem Modified 20 F

3 X

M ES D not an action (replace D) Replaced D 21 F

3 N

S 22 F

3 X

X N

ES Modify stem; add second half to A Modified Stem and distracter A 23 F

2 X

N ES Modify stem Modified 24 F

2 X

B S

25 F

2 B

S 26 F

2 X

B ES Modify stem to include info in all four distracters Modified 27 F

2 X

B ES Change stem to make B correct Changed stem 28 F

2 B

S 29 H

3 X

N ES Modify stem Modified 30 H

2 N

S 31 H

3 B

S 32 F

3 X

N US A &C not credible Replaced 33 F

3 X

N ES Change D to average Changed 34 F

3 X

N US Three of four distracters can be eliminated with partial knowledge Replaced Q 35 F

2 X

N ES Modify stem Modified

ES-401 DC-2012-01 3

Form ES-401-9 Page 3 of 5 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws
4. Job Content Flaws
5. Other
6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

  1. /

units Back-ward Q=

K/A SRO Only 36 F

3 X

X N

ES Modify stem; add info to C&D Modified stem; C&D info added 37 F

3 X

N ES Add caps to only and and in all distracters Added 38 H

3 X

N ES Modify stem; change will to should Modified stem, change made 39 F

3 X

B ES Add actual temp in stem Added 40 H

2 X

N ES Get rid of steam table reference Removed 41 H

2 X

N ES Modify stem Modified 42 H

2 N

S 43 H

3 N

S 44 F

3 X

B ES Modify stem Modified 45 F

3 X

X N

ES Modify stem; C contradicts info in stem (change) Modified stem, Changed C 46 H

2 X

N ES Add clarification to all distracters Added 47 H

2 X

N US Two non credible distracters Replaced distracter A 48 F

3 X

X N

ES Change EDGs to 11 and 12 (change correct answer) Changed 49 H

3 X

N ES A none of the above (replace) Replaced A 50 H

3 X

X N

ES Modify stem; fix A to ensure not cueing correct answer Modified stem and distracter A 51 H

2 X

X N

ES Modify stem; change temps in distracters to off exact setpoint Modified stem & temps 52 H

2 X

M US Too similar to Q 27 (replace) Replaced Q 53 H

3 X

B ES Modify stem Modified 54 F

2 B

S 55 H

3 N

S 56 H

2 X

B US A&C not credible Replaced Q 57 H

2 N

S 58 H

2 X

M ES Remove turbine load info from stem Replaced Q 59 H

2 X

N ES Remove no effect from A Removed 60 H

3 X

B ES Capitalize only in stem Capitalized only

ES-401 DC-2012-01 4

Form ES-401-9 Page 4 of 5 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws
4. Job Content Flaws
5. Other
6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

  1. /

units Back-ward Q=

K/A SRO Only 61 H

2 X

N ES Capitalize maximum in stem Maximum capitalized 62 F

2 X

X N

ES Modify stem; revise A,B,&C Modified stem, revised A,B,&C 63 H

3 X

X N

ES Modify stem; verify A incorrect Modified stem and distracter A 64 H

3 X

B ES Modify stem Modified 65 H

3 X

N ES Modify stem Modified 66 F

2 B

S 67 F

2 N

S 68 H

3 N

S 69 F

2 X

B ES Modify stem to ensure only one correct answer Modified 70 F

2 N

S 71 F

3 X

N ES Modify stem Modified 72 F

3 X

B ES Modify stem Modified 73 F

2 X

N S

74 F

2 X

X N

ES Modify stem to ensure only one correct answer Modified 75 H

3 B

S 76 F

3 X

X N

ES Replace directs with considers in stem Revised stem 77 H

3 X

X N

ES Modify stem Modified 78 H

3 X

X N

ES Revise B&C Revised 79 H

3 X

X N

ES Ensure D incorrect Replaced D 80 H

3 X

X M

ES Modify stem, ensure B incorrect Modified stem and ensured B incorrect 81 F

2 X

X N

ES Modify A Replaced A 82 H

3 X

N S

83 H

3 X

B S

84 F

2 X

N S

85 F

3 X

X N

ES Modify stem Modified

ES-401 DC-2012-01 5

Form ES-401-9 Page 5 of 5 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws
4. Job Content Flaws
5. Other
6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

  1. /

units Back-ward Q=

K/A SRO Only 86 H

3 X

X N

ES Modify stem Modified 87 H

2 X

N S

88 H

2 X

X N

US A&B not credible - modify stem to have PZR level at 12% Modified A,B, & stem 89 H

2 X

X N

ES Add power level to stem - get rid of unnecessary info Added and revised 90 F

2 X

X X

N S

91 H

3 X

X N

ES Supplement D Supplemented 92 H

3 X

X N

ES Modify A Modified A 93 H

3 X

X N

ES Add additional words to C to make it consistent with other distracters Modified C 94 F

2 X

X B

ES Modify C Modified 95 F

2 X

X B

ES Modify stem Modified 96 F

2 X

X N

ES Put required in caps in stem Modified stem 97 F

3 X

X N

ES Modify stem Modified 98 F

3 X

X X

N ES Replace D; remove if at all from stem Modified stem, Replaced D 99 H

3 X

X B

ES Add where explosion occurred in stem Added 100 H

2 X

X N

ES Modify stem Modified