ML090790342

2009-02-Draft Written Exam Comments

ML090790342
Person / Time
Site:	Diablo Canyon
Issue date:	02/06/2009
From:	Brian Larson Operations Branch IV
To:	Pacific Gas & Electric Co
References
50-275/09-301, 50-323/09-301
Download: ML090790342 (8)

Text

ES-401 DC-2009-02 RO Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 5 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 1

H 3

B S

C, CR, 41.7 2

F 2

N S

A, CR, 41.5 3

F 2

N S

A, CR, 41.7 4

F 2

N S

B, CR, 41.5 5

F 3

N S

C, CR, 41.8 6

H 3

B S

C, CR, 41.7 7

F 3

N S

C, CR, 41.7 8

H 3

X N

U D, CR, 41.10 - KA requires testing on the impact and use of procedures and the question tests on operator action. Reword stem and distractors to read as operator actions and include part (a) of the KA. Revised 9

H 3

N S

A, CR, 41.7

ES-401 DC-2009-02 2

Form ES-401-9 Page 2 of 5 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 10 H

3 N

S A, CR, 41.7 11 H

3 N

S A, CR, 41.2 12 H

3 B

S A, CR, 41.7 13 F

3 B

S D, CR, 41.7 14 H

3 M

S B, CR, 41.10 15 H

3 N

S C, CR, 41.10 16 F

3 B

S A, CR, 41.8 17 H

3 N

S C, CR, 41.7 18 H

3 X

B U

B, CR, 41.5 - KA states power change, not steady state. Revised question 19 H

3 M

E C, CR, 41.5 - Reword stem to clarify location of break Fixed 20 F

3 X

N E

A, CR, 41.7 - Add the TDAFW pump is running to the stem. Fixed 21 H

3 X

B E

B, CR, 41.7 - The correct distractor is unique and could be a cue; add a different battery as a distractor. Fixed 22 F

3 X

B E

B, CR, 41.7 - Distractor A could be argued as being correct since 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> is a subset of 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> (i.e. if it is designed to last 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />, then it is by definition designed to last 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />). Chang stem to read: During a loss of all AC, what is the design lifetime of a vital 125 VDC battery during a design basis accident coincident with a loss of battery charger? Fixed 23 H

3 X

B U

A, CR, 41.7 - This question does not match the KA because it does not address the automatic loader as the KA requires. Replaced question 24 H

2 X

N E

A, CR, 41.11 - Add fail high to distractors B and D for clarification Fixed 25 F

3 M

E B. CR, 41.10 - Having b in all distractors eliminates it from being discriminatory.

Revised 26 H

3 N

S A, CR, 41.7 27 H

2 N

S A, CR, 41.4 28 F

3 N

S B, CR, 41.7 29 F

3 B

S D, CR, 41.6 30 F

2 N

S B, CR, 41.7

ES-401 DC-2009-02 3

Form ES-401-9 Page 3 of 5 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 31 F

3 N

S D, CR, 41.6 32 F

3 N

S B, CR, 41.5 33 H

3 X

M E

A, CR, 41.7 - Consider replacing KA to enable a better question/KA match.

Replaced KA 34 F

3 M

E C, CR, 41.11 - Reword A and add BOTH to B Done 35 H

3 B

S B, CR, 41.4 Bank, not Modified Bank Updated 36 H

3 X

N U

C, CR, 41.7 - Does not focus on the operating controls for the ventilation system. It may be necessary to select a new KA. Replaced KA 37 F

3 B

S B, CR, 41.4 38 F

3 B

S A, CR, 41.4 39 H

3 N

S D, CR, 41.5 40 H

3 N

S B, CR, 41.5 41 H

3 N

S C, CR, 41.7 42 H

3 X

B U

D, CR, 41.10 - Does not address the locate and determine the indications are correct as required by the KA. Revised question 43 H

3 N

S C, CR, 41.7 44 F

2 N

S D, CR, 41.7 45 H

3 N

S C, CR, 41.7 46 H

3 N

S B, CR, 41.6 47 H

3 N

S D, OR, 41.5 48 F

3 M

E C, CR, 41.7 - Capitalize the word MINIMUM Done 49 F

3 N

S A, CR, 41.4 - Confirm there is no conflict with In-Plant JPM Minor rev to JPM 50 H

3 N

S D, CR, 41.10 51 H

3 N

E D, CR, 41.7 - Add AOP to Tech Ref to show link from AOP to OP. Done 52 H

3 M

S C, CR, 41.7 - Need a complete copy of the BANK question Provided 53 F

3 N

S D, CR, 41.7

ES-401 DC-2009-02 4

Form ES-401-9 Page 4 of 5 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 54 H

3 N

S D, CR, 41.4 - Enhance explanations Done 55 H

3 M

U C, CR, 41.10 - This is JPM S3. Replace question. Done 56 F

3 B

S A, CR, 41.10 57 F

3 B

S B, CR, 41.10 - BANK, not MOD BANK Changed 58 H

3 N

S B, CR, 41.6 59 H

3 N

S B, CR, 41.2 60 H

3 X

B E

C, CR, 41.10 - Remove greater than statements. Fixed 61 H

4 B

E D, CR, 43.2 - Add OP as Tech Ref. (DCPP 2007 RO Exam) Done 62 H

4 X

X B

E D, CR, 41.10 - Is this a valid from memory question? SRO only? Licensee confirms this is RO level knowledge.

63 F

3 X

M E

C, CR, 41.10 - Clarify difference between second and third bullets. Revised 64 F

3 B

S B, CR, 41.10 65 H

3 X

B E

C, CR, 41.10 - Answer A may be slow, but is it available? May need to add to stem Per FR-H.2, which is ? Fixed 66 F

3 X

B E

C, CR, 41.10 - Change Fire Fighters to Fire Department. Done 67 F

3 B

S B, CR, 41.10 68 F

2 N

S D, CR, 41.10 69 F

2 N

S A, CR, 41.10 70 F

2 N

E B, CR, 41.10 - Clarify in stem that tags are being hung by Operations personnel.

Done 71 F

2 B

S D, CR, 41.10 72 F

2 M

S B, CR, 41.10 73 H

3 N

S D, CR, 41.5 74 F

3 N

S A, CR, 41.10 75 H

2 N

S B, CR, 41.10

ES-401 DC-2009-02 5

Form ES-401-9 Page 5 of 5 OBDI 202 - IOLE Process B= 24 (32%)

F= 34 (45%)

E= 16 (21%)

M= 10 (13%)

H= 41 (55%)

U= 6 (8%)

RO TOTALS:

N= 40 (55%)

S= 53 (71%)

SRO TOTALS:

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2.

Chief Examiner comments are indicated in blue.

3.

Average difficulty is _2.85 on the RO exam.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 41.2 = 3 41.3 = 0 41.4 = 5 41.5 = 13 41.6 = 4 41.7 = 24 41.8 = 2 41.9 = 0 41.10 = 22 41.11 = 2 41.12 =

41.13 =

41.14 =

5.

The answer distribution is: RO A = 18 (24%)

/

B = 21 (28%)

/

C = 19 (25%)

/

D = 17 (23%)

/

6.

There is (are) 1 / question(s) with attachments provided (steam tables).

ES-401 DC-2009-02 SRO Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 3 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 1

H 3

M S

A, OR, 43.5 Hand out 2

H 3

X N

E B, OR, 43.2 - Do not provide TS 3.0.3 because the applicants should know from memory the TS motherhood statements. Done Hand out 3

H 3

M S

D, CR, 43.5 - Add a reference that defines Go To and Refer To. Also add reference that defines applicability of OP SD-1 in Modes 5 & 6. Done 4

H 3

N S

B, CR, 43.5 5

H 3

B S

D, CR, 43.5 6

H 3

N S

C, CR, 43.5 7

F 2

B S

C, CR, 43.2 - Revise Answer A explanation. Done 8

H 3

N S

D, CR, 43.5 - looks like a new question, not a modified bank. Identified as NEW 9

H 3

N S

D, CR, 43.5 - Revise Answer C explanation. Done

ES-401 DC-2009-02 2

Form ES-401-9 Page 2 of 3 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 10 H

3 B

S A, CR, 43.5 11 H

3 N

E C, CR, 43.5 - Add procedure refs to Answers A and D. Done 12 H

3 X

X N

U A, CR, 43.5 - Does not address the predict the impact part of the KA. Answer D may not be plausible. Enhanced answers; modified D 13 H

3 X

B U

D, OR, 43.2 - Change stem to read taken to allow entering MODE 2? instead of taken to allow the startup to continue? then make necessary changes to all answers. Also, it could be argued that completing the actions contained in B would allow the startup to continue and therefore there are two correct answers. Revised Hand out.

14 H

4 N

E D, OR, 43.2 - Dont include P&IDs as a handout. Done Explanations for B and D are not clear. Revised Hand out.

15 H

3 X

X N

U A, CR, 43.5 - Answer B not plausible. Explanations not clear. Does not address the predict the impact part of the KA. Revised 16 F

3 N

E D, CR, 43.7 - Reword stem to ask what indications the Refuel SRO would use.

Revised 17 H

3 X

N U

A, CR, 43.5 - This is considered an RO question in that it is testing on the entry conditions for an AP. Revised 18 H

3 N

S D, CR, 43.5 19 F

2 X

N U

C, CR, 43.2 - This is an RO question in that it is testing on basic rad worker knowledge. Replaced KA 20 H

2 M

E A, CR, 43.5 - Similar to JPM S1. Select new KA if necessary. Revised with same KA 21 F

2 X

N U

D, CR, 43.3 - Answer A is not plausible. Replaced. Explanation for Answer B is not clear. Revised 22 F

2 N

E A, CR, 43.4 - revise explanations to address both parts of answers Done 23 H

3 N

S A, CR, 43.4 24 H

3 N

E D, CR, 43.5 - Revise stem to four TECs and revise correct answer. Done 25 F

3 N

E A, CR, 43.3 - Consider new KA. Replaced B= 4 (16%)

F= 6 (24%)

E= 8 (32%)

M= 3 (12%)

H= 19 (76%)

U= 6 (24%)

SRO TOTALS:

N= 18 (72%)

S= 11 (44%)

Additional Notes: The exam review indicates greater than 60% comprehension level questions. Chief Examiner has discussed with licensee and both determined this as acceptable given randomly selected KAs.

ES-401 DC-2009-02 3

Form ES-401-9 Page 3 of 3 OBDI 202 - IOLE Process

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2.

Chief Examiner comments are indicated in blue.

3.

Average difficulty is 2.84 on the SRO exam.

4.

The 10CFR55.41/43 distribution is:

43.1 = 0 43.2 = 5 43.3 = 2 43.4 = 2 43.5 = 15 43.6 = 0 43.7 = 1

5.

The answer distribution is:

A = 7 (28%)

B = 6 (24%)

C = 4 (16%)

D = 8 (32%)

6. There are 4 questions with attachments provided. Make sure hand outs dont provide cues to any other question.