ML100910278

CG-2009-10-Draft Written Exam Review

ML100910278
Person / Time
Site:	Columbia
Issue date:	10/14/2009
From:	Garchow S Operations Branch IV
To:	Energy Northwest
References
Download: ML100910278 (7)

Text

ES-401 CG-2009-01 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 1

H 3

N S

B, CR, 41.5 2

H 3

N S

C, CR, 41.10 3

F 3

N S

C, CR, 41.7 4

H 3

M S

D, CR, 41.5 5

F 3

M S

A, CR, 41.6 6

F 3

E N

E S

C, CR, 41.10 - This question is testing on an operator action that is 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> into an event. Not sure this is a valid memory question.

Ops validation indicated this is a fair question. OK as is.

7 H

3 N

S D, CR, 41.4 8

H 3

N S

B, CR, 41.4 9

H 2

B E

A, CR, 41.5 - Recommend adding the reactor head is still installed.

ES-401 CG-2009-10 2

Form ES-401-9 Page 2 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation S

This was corrected.

10 F

3 U

N U

S C, CR, 41.10 - This question is considered an SRO Only question because the second part of the distractors is testing on refueling actions taken on the refueling bridge.

Will follow up on the LO.

This question is supported by an RO LO.

11 F

3 N

S B, CR, 41.7 12 F

3 B

S A, CR, 41.6 13 H

2 U

M U

S C, OR, 41.10 - This question is considered unsat because the correct distractor is 225 degrees and not 210 degrees as indicated by the answer key.

The answer key is correct.

14 H

3 U

N U

S D, OR, 41.10 - This question is considered unsat because with drywell temperature at 332 degrees, dw sprays should have already been initiated. Assuming dw sprays were initiated per the EOP and dw temperature continues to increase, then PPM 5.1.1 and an ED are required as stated in distractor D.

Will followup after researching.

This question was revised.

15 F

3 N

S B, CR, 41.9 16 F

3 N

S C, CR, 41.2 17 F

3 B

S A, CR, 41.10 18 F

3 M

S D, CR, 41.10 19 F

3 N

S C, CR, 41.7 20 H

3 N

S A, CR, 41.10 21 H

3 N

S B, CR, 41.6 22 F

3 N

S A, CR, 41.7 23 F

3 N

S D, CR, 41.10 24 F

2 N

S D, CR, 41.10 25 F

3 B

S B, CR, 41.9 26 H

3 N

S A, CR, 41.7

ES-401 CG-2009-10 3

Form ES-401-9 Page 3 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 27 H

3 N

S B, CR, 41.6 28 H

3 N

S D, CR, 41.7 29 F

3 E

N E

S A, CR, 41.7 - Recommend changing distractor B to RHR-P-2B trips after either Revised the question.

30 H

3 N

S B, CR, 41.7 31 H

2 N

S D, CR, 41.7 32 H

3 B

S B, CR, 41.7 33 H

3 N

S A, CR, 41.6 34 H

3 N

S A, CR, 41.2 35 H

3 M

S C, CR, 41.2 36 F

2 N

S B, CR, 41.6 37 F

2 N

S D, CR, 41.7 38 F

3 N

S B, CR, 41.7 39 H

3 N

S C, CR, 41.9 40 H

3 E

N E

S C, CR, 41.5 - Recommend changing the stem to read describes the immediate effect.

Made the recommended changes.

41 H

3 E

N E

S A, CR, 41.4 - Recommend adding the noun name for RFW-DPT-4B to the stem.

Made the recommended changes.

42 H

3 B

S D, CR, 41.7 43 F

4 U

N U

S A, CR, 41.4 - This is considered unsat because the second part of the distractors is E-plan / SRO knowledge.

Will make the distractors RO knowledge.

The distractors were revised.

44 H

3 N

S B, CR, 41.9 45 H

3 E

N E

S A, CR, 41.4 - Recommend making the stem read which indicates 5 K[ohms].

Recommended change was made.

ES-401 CG-2009-10 4

Form ES-401-9 Page 4 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 46 H

3 U

N U

S B, CR, 41.7 - This question is considered unsat because distractor A could also be considered correct.

Will reword distractor A.

Distractor A was revised.

47 H

3 U

N U

S C, CR, 41.4 - This question is considered unsat because distractor A is a subset of the correct distractor making it also correct.

Revised distractor A.

Distractor A was revised.

48 F

3 N

S B, CR, 41.4 49 F

3 N

S D, CR, 41.4 50 F

3 N

S A, CR, 41.7 51 H

3 N

S D, CR, 41.5 52 H

3 B

S B, CR, 41.2 53 H

3 N

S C, CR, 41.4 54 H

3 N

S A, CR, 41.4 55 H

3 N

S B, CR, 41.6 56 H

3 N

S D, CR, 41.3 57 H

3 N

S D, CR, 41.10 58 F

3 N

S C, CR, 41.3 59 F

3 N

S B, CR, 41.2 60 F

3 N

S A, CR, 41.4 61 H

3 U

N U

S A, OR, 43.2 - This is an SRO Only question because it tests on the application of a TS (RO knowledge is above the LCO line in TS).

This question is ok as written.

62 H

3 M

S C, CR, 41.4 63 F

3 N

S D, CR, 41.9 64 H

3 N

S B, CR, 41.7

ES-401 CG-2009-10 5

Form ES-401-9 Page 5 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 65 F

3 N

S D, CR, 41.6 66 F

2 N

S C, CR, 41.10 67 F

4 N

S B, CR, 41.10 68 F

2 N

S A, CR, 41.10 69 F

2 N

S C, CR, 41.10 70 H

2 N

S D, CR, 41.10 71 F

2 N

S A, CR, 41.10 72 F

2 N

S C, CR, 43.2 73 F

2 N

S B, OR, 41.12 74 H

3 N

S C, CR, 41.10 75 F

3 N

S D, CR, 41.10 76 H

3 N

S D, CR, 43.5 77 H

3 N

S C, OR, 43.5 78 H

3 N

S B, CR, 43.5 79 H

3 N

S C, CR, 43.2 80 H

1 U

N U

S A, OR, 43.5 - This is a direct lookup since the stem states H2 concentration cannot be determined to be below 0.6% verbatim from the provided procedure reference and the corresponding action is provided in the same reference.

Will revise following research.

The stem was revised.

81 H

3 N

S D, CR, 43.5 82 H

3 N

S C, OR, 43.5 83 H

2 N

S A, OR, 43.2 84 F

2 N

S B, CR, 43.3 85 H

3 N

S B, CR, 43.2 86 H

3 N

S C, CR, 43.5

ES-401 CG-2009-10 6

Form ES-401-9 Page 6 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 87 H

3 N

S A, OR, 43.5 88 H

2 N

S B, OR, 43.2 89 H

3 N

S D, CR, 43.5 90 H

3 N

S D, OR, 43.2 91 H

3 N

S A, OR, 43.2 92 H

3 N

S A, OR, 43.5 93 H

3 M

S A, OR, 43.2 94 H

3 N

S D, OR, 43.2 95 H

3 U

N U

S D, OR, 43.5 - Considered unsat because it is testing the same material as Q14.

Question is ok as is.

96 H

3 N

S A, CR, 43.5 97 F

3 N

S B, OR, 43.5 98 F

2 E

N E

S B, CR, 43.5 - The justification for the correct answer could not be found in the references provided.

This question is ok as written.

99 H

3 N

S C, CR, 43.5 100 F

2 N

S C, OR, 43.5 B= 6 F= 36 (48%)

E= 6 M= 8 H= 39 (52%)

U= 7 RO TOTALS:

N= 61 S= 62 Additional Notes:

B= 0 F= 3 (12%)

E= 1 M= 2 H= 22 (88%)

U= 2 SRO TOTALS:

N= 23 S= 23 Additional Notes:

GENERAL COMMENT

S:

1.

Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

ES-401 CG-2009-10 7

Form ES-401-9 Page 7 of 7 OBDI 202 - IOLE Process

2.

Chief Examiner comments are indicated in blue.

3.

Average difficulty is __2.85_ on the RO exam and 2.72 on the SRO exam. Both these values are within the normal expected difficulty range for this reviewer.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 0 41.2 = 7 43.2 = 8 41.3 = 2 43.3 = 1 41.4 = 12 43.4 = 0 41.5 = 5 43.5 = 16 41.6 = 8 43.6 = 0 41.7 = 16 43.7 = 0 41.8 = 0 41.9 = 5 41.10 = 19 41.11 = 0 41.12 = 1 41.13 =

41.14 =

5.

The answer distribution is: RO / SRO A = 19

/ 7 B = 20

/ 6 C = 18

/ 6 D = 18

/ 6

6.

There are 4 / 14 (RO/SRO) questions with attachments provided.

7.

This review was performed by S. Garchow the week of 8/10/2009.

8.

The SRO exam contains 88% higher cognitive questions. This is significantly higher than the recommended limit of 60%.