ML102450367

RB-2010-Draft Written Exam Review

ML102450367
Person / Time
Site:	River Bend
Issue date:	04/12/2010
From:	Garchow S Operations Branch IV
To:	Entergy Operations
References
50-458/10-04
Download: ML102450367 (7)

Text

ES-401 RB-2010-04 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 2 N S C, CR, 41.10 2 F 2 N S A, CR, 41.7 3 F 3 N S A, CR, 41.7 B, CR, 41.6 - Change trips to trip in distactors.

E 4 H 2 M This change will be made.

S The text editing was made as recommended.

5 F 3 N S D, CR, 41.7 6 F 3 N S C, CR, 41.7 D, CR, 41.10 - This is a KA mismatch due to being focused on TBCW rather than U

7 F 3 U B CCW.

S This question will be rewritten.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only This question was replaced with a new question.

8 H 3 N S B, CR, 41.4 9 F 2 N S A, CR, 41.8 10 F 3 N S C, CR, 41.4 11 H 3 N S A, CR, 41.3 12 H 4 N S B, CR, 41.5 13 F 3 M S A, CR, 41.7 14 F 3 N S D, CR, 41.9 15 H 3 N S C, CR, 41.10 16 H 3 N S B, CR, 41.2 17 H 3 N S D, CR, 41.10 18 F 3 N S B, CR, 41.13 19 F 3 N S C, CR, 41.10 20 H 4 N S A, CR, 41.5 21 H 3 N S B, CR, 41.5 22 H 3 N S A, CR, 41.9 23 H 3 N S B, CR, 41.7 A, CR, 41.6 - This question is U because distractor B is also correct.

U 24 H 3 U N Distractor B will be revised.

S Distractor B was rewritten.

25 F 3 N S B, CR, 41.7 26 F 3 N S A, CR, 41.10 27 H 3 N S C, CR, 41.11 28 H 3 N S C, CR, 41.7 29 H 3 N S D, CR, 41.7 Page 2 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 30 H 3 N S D, CR, 41.8 31 H 3 N S D, CR, 41.7 32 F 3 N S C, CR, 41.10 33 H 3 N S B, CR, 41.6 34 F 3 N S C, CR, 41.10 35 H 3 N S B, CR, 41.2 36 F 3 N S C, CR, 41.6 37 H 3 N S D, CR, 41.6 B, CR, 41.6 - Distractor A can also be argued as being correct since it is a subset of E the correct answer.

38 H 3 E N S Distractor A will be revised.

Distractor A was rewritten.

39 H 3 N S C, CR, 41.7 40 H 3 N S C, CR, 41.7 41 H 3 N S D, CR, 41.7 42 H 3 N S B, CR, 41.7 43 H 3 B S B, CR, 41.5 44 H 3 N S A, CR, 41.5 D, CR, 41.7 - Need to discuss this question as it seems annulus pressure would E normally be negative and would remain so under the described conditions in the 45 H 3 N stem.

S After discussing with the exam author, this question is correct as written.

46 H 3 N S B, CR, 41.7 47 H 3 M S A, CR, 41.7 48 F 3 N S A, CR, 41.7 49 H 3 N S D, CR, 41.7 50 H 3 B S C, CR, 41.7 Page 3 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 51 H 3 N S D, CR, 41.7 52 H 3 N S D, CR, 41.7 53 H 2 B S B, CR, 41.10 E D, CR, 41.3 - Discuss this question with exam author.

54 F 2 E N S This question is ok as written.

C, CR, 41.4 - Discuss this question with exam author.

E 55 F 3 E N The stem will be revised to improve the readability.

S Part of the question in the stem was highlighted to improve the focus of the question.

56 H 3 N S B, CR, 41.4 57 H 3 N S B, CR, 41.3 58 F 2 N S A, CR, 41.7 C, CR, 41.13 - This is a KA mismatch because it does not involve fuel handling.

U Suggest this KA be rejected and a new one selected.

59 F 3 U N S This question will be rewritten.

This question was replaced with a new question.

60 F 3 N S B, CR, 41.10 A, CR, 41.5 - Revise distractor B so it reads 980 MW instead of 750 MW.

E 61 H 3 E M This edit change will be made.

S This change was made as recommended.

62 H 3 N S C, CR, 41.4 63 H 3 N S A, CR, 41.5 64 F 3 N S D, CR, 41.4 65 H 2 N S D, CR, 41.5 66 F 3 N S D, CR, 41.10 67 F 2 N S C, CR, 41.10 68 F 3 N S D, CR, 41.10 69 F 3 B S C, CR, 41.10 Page 4 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E A, CR, 41.10 - Not sure this is a memory item; check with exam author.

70 F 3 E N S Based on feedback from the validating crew, this is a memory item.

71 F 3 N S B, CR, 41.10 72 F 3 N S A, CR, 41.9 73 H 4 N S A, CR, 41.11 C, CR, 43.5 - This is an SRO Only question in that it tests on the emergency plan U notification requirements.

74 F 3 U N S This question will be revised.

This question was replaced with a new question.

75 F 2 B S C, CR, 41.10 76 H 3 B S D, CR, 43.5 77 H 3 N S C, CR, 43.5 78 H 3 N S C, CR, 43.5 79 F 2 N S B, CR, 43.5 80 H 3 N S A, OR, 43.5 A, CR, 43.5 - Add another EOP related distractor other than the correct answer.

E 81 H 3 E N This change will be made.

S Distractor D was rewritten.

82 F 3 N S D, OR, 43.2 83 F 2 N S A, CR, 43.5 84 H 3 M S B, CR, 43.5 85 F 3 N S D, CR, 43.5 86 H 3 N S B, CR, 43.5 87 H 3 N S C, CR, 43.5 88 H 3 N S A, CR, 43.5 89 F 3 N S A, CR, 43.2 Page 5 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 90 H 3 N S D, CR, 43.5 91 H 3 N S B, CR, 43.2 92 H 3 N S C, CR, 43.5 A, CR, 43.2 - Candidates are not typically expected to have 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> TS action E statements memorized.

93 F 3 E N S The exam author will check with Ops on this.

This question was replaced with a new question.

C, CR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not E

94 E N symptom/system specific.

S This question was replaced.

D, OR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not symptom specific. Recommend deleting the RHR status from the stem and change E so it reads Subsequent to the above conditions, a loss of all shutdown cooling 95 H 3 E N occurs.

S This change will be made.

The question was revised as discussed above.

96 F 2 N S A, CR, 43.5 E D, CR, 43.4 - Discuss this question with exam author.

97 H 2 E N S This question is ok as written.

98 F 3 N S B, CR, 43.4 B, CR, 43.5 - This is a Tier 3 question and therefore needs to be generic and not E

99 H 2 E N symptom specific.

S This question was replaced with a new question.

100 H 3 N S D, CR, 43.5 B= 2 F= 32 (43%) E= 7 RO TOTALS: M= 9 H= 43 (57%) U= 4 Additional Notes:

N= 64 SRO TOTALS: B= 1 F= 8 (32%) E= 6 Additional Notes:

M= 1 H= 17 (68%) U= 0 Page 6 of 7 OBDI 202 - IOLE Process

ES-401 RB-2010-04 7 Form ES-401-9 N= 23

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner follow-up comments are indicated in blue.

3. Average difficulty is _2.9 ___ on the RO exam and 2.8___ on the SRO exam. These values fall within the expected range for this reviewer

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 0 41.2 = 2 43.2 = 4 41.3 = 2 43.3 = 0 41.4 = 6 43.4 = 2 41.5 = 9 43.5 = 19 41.6 = 6 43.6 = 0 41.7 = 23 43.7 = 0 41.8 = 2 41.9 = 3 41.10 = 18 41.11 = 2 41.12 = 0 41.13 = 2 41.14 =

5. The answer distribution is: RO / SRO A = 18 (25%) / 7 (28%)

B = 19 (25%) / 6 (24%)

C = 19 (25%) / 5 (20%)

D = 19 (25%) / 7 (28%)

6. There are 0 / 3 questions with attachments provided.

7. This exam was reviewed by S. Garchow on March 9-10, 2010. Page 7 of 7 OBDI 202 - IOLE Process