ML113480099

2011-09-Draft Written Exam Comments

ML113480099
Person / Time
Site:	South Texas
Issue date:	03/02/2011
From:	NRC Region 4
To:	South Texas
References
50-498/11-301, 50-499/11-301, ST-2011-09
Download: ML113480099 (7)

Text

ES-401 ST-2011-09 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 1

F 2

B S

CR, 41.10 2

H 3

N S

CR, 41.10 3

F 2

N S

CR, 41.10 4

F 2

B S

CR, 41.7 5

H 3

B S

CR, 41.7 6

H 3

B S

CR, 41.5 7

H 3

N S

CR, 41.5 8

H 3

B S

CR, 41.7 9

F 3

N S

CR, 41.7 10 H

3 B

S CR, 41.4

ES-401 ST-2011-09 2

Form ES-401-9 Page 2 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 11 F

3 N

S CR, 41.4 12 H

3 B

S CR, 41.7 13 F

3 N

S CR, 41.7 14 H

2 E

N E

S OR, 41.5 - This question needs revised as it is close to a direct look up with the values given and the reference provided. Recommend revising to that the differential pressure is provided and one of the dps is out of spec. The distractors would remain the same.

This question was revised.

15 H

3 E

B E

S CR, 41.5 - Recommend changing effect to initial effect in the stem for clarification.

This change was made.

16 F

3 N

S CR, 41.7 - CAF on the terminology used in the distractors (depress).

The terminology is consistent with that used at STP.

17 F

3 N

S CR, 41.7 18 F

3 B

S CR, 41.7 19 H

3 B

S CR, 41.7 20 H

3 N

S CR, 41.7 21 F

3 N

S CR, 41.7 22 F

3 N

S CR, 41.13 23 F

3 B

S CR, 41.13 24 F

3 B

S CR, 41.4, NRC 05 25 F

3 N

S CR, 41.10 26 H

3 B

S CR, 41.5, NRC 01 27 F

3 B

S CR, 41.9 28 F

3 N

S CR, 41.8 29 F

3 N

S CR, 41.7 30 H

3 N

S CR, 41.4 31 H

3 B

S CR, 41.7

ES-401 ST-2011-09 3

Form ES-401-9 Page 3 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 32 H

3 B

S CR, 41.7 33 H

3 N

S CR, 41.5 34 H

3 N

S CR, 41.7 35 H

3 N

S CR, 41.7 36 F

3 N

S CR, 41. 4 37 H

3 N

S CR, 41.7 38 H

3 B

S CR, 41.5 39 F

2 N

S CR, 41.7 40 H

3 N

S CR, 41.7 41 F

3 N

S CR, 41.4 42 H

3 B

S CR, 41.4 - EDIT - The question indicates it was used on a previous NRC exam and the pedigree sheet indicates it is a new question. Both cannot be correct.

This is a bank question.

43 H

3 N

S CR, 41.5 44 F

3 B

S CR, 41.7, NRC 99 45 F

3 U

B U

S CR, 43.4 - This is an SRO question because it tests on the radiological hazard (see CFR 43.4. Recommend selecting another KA.

After discussing with the facility, this question is ok as written.

46 F

3 N

S CR, 41.12 47 H

2 B

S CR, 41.12, NRC 95 48 F

3 N

S CR, 41.11 49 H

3 N

S CR, 41.11 50 F

3 N

S CR, 41.10 51 H

3 B

S CR, 41.7 52 H

3 N

S CR, 41.5 53 F

2 N

S CR, 41.10

ES-401 ST-2011-09 4

Form ES-401-9 Page 4 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 54 H

3 N

S CR, 41.5 - This question will be rewritten (Q65).

This question was revised due to overlap with Q65.

55 H

3 N

S CR, 41.7 56 H

2 N

S CR, 41.7 57 F

2 N

S CR, 41.6 58 H

3 N

S CR, 41.7 59 H

4 N

S CR, 41.7 60 H

3 B

S CR, 41.10, NRC 07 61 F

2 N

S CR, 41.9 62 H

3 B

S CR, 41.7, NRC 07 63 F

3 U

B U

S CR, 41.10 - This question is a KA mismatch because it does not address the loss of containment integrity as required by the KA.

This question will be rewritten.

This question was revised.

64 H

2 B

S CR, 41.10 65 H

3 E

N E

S CR, 41.5 - This question tests the same knowledge as Q 54.

Q54 was rewritten.

66 H

3 N

S CR, 41.5 67 F

3 N

S CR, 41.10 68 F

3 N

S CR, 41.10 69 F

3 N

S CR, 41.10 70 F

3 N

S CR, 41.7 71 H

3 N

S CR, 41.7 72 F

2 N

S CR, 41.10 73 F

3 E

N E

S CR, 41.9 - It could be argued distractor B is also correct even though it is outside the control room.

B has the wrong location so it is incorrect.

ES-401 ST-2011-09 5

Form ES-401-9 Page 5 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 74 F

2 N

S CR, 41.10 75 F

2 N

S CR, 41.10 76 F

2 N

S CR, 43.5 77 F

3 B

S CR, 43.5 78 H

3 B

S CR, 43.5, NRC 07 79 H

4 N

S CR, 43.5 80 H

3 N

S CR, 43.5 81 H

3 B

S CR, 43.5 82 H

3 N

S CR, 43.5 83 H

3 N

S CR, 43.2 84 H

3 N

S CR, 43.5 85 H

2 N

S CR, 43.5 86 H

2 U

N U

S OR, 41.10 - This is an RO question because it can be answered just by determining the shutdown boron concentration. This activity is an RO ability.

This question is ok as written. The correct procedure has more than one boron requirement to chose from based on the given plant conditions.

87 H

3 U

N U

S CR, 41.10 - This can be answered using RO knowledge. I.e. understanding the turbine should be verified tripped and verifying the trip breakers are open will obtain the correct answer.

This question is ok as written because there is also a procedure selection required.

88 H

3 N

S CR, 43.5 89 H

3 U

N U

S CR, 41.5 - This can be answered using RO knowledge. I.e. understanding the effects of the first part of the 4 distractors will lead to the correct answer.

This question will be revised.

This question was rewritten.

90 H

3 N

S CR, 43.5 91 F

3 N

S CR, 43.5 92 H

3 U

N U

CR, 41.5 - This can be answered using RO knowledge. I.e. understanding the

ES-401 ST-2011-09 6

Form ES-401-9 Page 6 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

effects of the first part of the 4 distractors will lead to the correct answer.

This question will be revised.

This question was rewritten.

93 F

3 N

S CR, 43.7 94 H

3 N

S CR, 43.1 95 H

2 N

S CR, 43.1 96 H

3 E

N E

OR, 43.2 - Delete the third column of the distractor table.

Third column will be deleted.

97 H

3 N

S OR, 43.2 98 H

2 N

S OR, 43.2 99 F

3 N

S CR, 43.1 100 H

3 E

N S

OR, 43.2 RO TOTALS:

B= 26 F= 37 E= 4 Additional Notes:

M= 0 H= 38 U= 2 N= 49 SRO TOTALS:

B= 4 F= 5 E= 2 Additional Notes:

M= 0 H= 20 U= 4 N= 21

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2.

This exam was reviewed by S. Garchow on August 8-9, 2011. Chief Examiner resolution comments are indicated in blue.

3.

Average difficulty is __ _2.82 on the RO exam and _2.84 on the SRO exam. These are both within the typical values for this exam reviewer.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 3 41.2 = 0 43.2 = 5

ES-401 ST-2011-09 7

Form ES-401-9 Page 7 of 7 OBDI 202 - IOLE Process 41.3 = 0 43.3 = 0 41.4 = 8 43.4 = 0 41.5 = 12 43.5 = 14 41.6 = 1 43.6 = 0 41.7 = 29 43.7 = 1 41.8 = 1 41.9 = 3 41.10 = 15 41.11 = 2 41.12 = 2 41.13 = 2 41.14 = 0

5.

The answer distribution is: RO / SRO A = 18 (24%)

/

7 (28%)

B = 16 (21%)

/

5 (20%)

C = 21 (28%)

/

6 (24%)

D = 20 (27%)

/

7 (28%)

6.

There are 1 / 4 questions with attachments provided.