ML101960329

PV-2009-11-Draft Written Exam Comments

ML101960329
Person / Time
Site:	Palo Verde
Issue date:	11/02/2009
From:	Garchow S Operations Branch IV
To:	Arizona Public Service Co
References
Download: ML101960329 (7)

Text

ES-401 PV-2009-11 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only C41.10 1 H 3 N S Agree D41.4 Answer states that the switch needs to be placed in START; however, the E

2 H 2 N reference says place the switch in AUTO.

S Agree - Stem was revised to correct wording.

E D41.7 Reformat the answers without all the bullets.

3 F 2 N S Agree A41.10 May clue that question 1, distractors A and B are incorrect.

4 F 2 M S OK as written.

5 H 3 N S D41.3 E B41.7 See markup 6 H 3 N S Comment was corrected.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E B41.4 See markup 7 H 3 N S Markup was corrected.

8 H 2 B S B41.3 9 F 2 N S D41.1 E C41.10 See markup 10 F 3 B S Markup was corrected.

11 F 2 N S B41.5 12 F 2 N S A41.5 13 H 3 N S D41.5 14 H 2 M S D41.10 15 F 2 B S B41.7 16 H 3 N S C41.5 17 F 3 N S B41.4 18 H 2 M S D41.5 C41.6, 43.2 See markups.

U 19 F 2 U N Agree - TS bases makes it an SRO question.

S The TS bases was removed from the question.

20 F 3 N S B41.5, 10 21 H 3 M S B41.7 E D41.5 See markup.

22 H 3 M S Markup was revised.

23 F 2 B S C41.7 24 H 3 N S C41.10 25 F 3 B S A41.11 26 H 3 N S B41.10 27 H 3 N E D41.10 VERIFY that PBB-S04 cannot be energized based on given information. Page 2 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S It would seem distractor B could be argued as also correct.

Distractor B was revised.

E A41.5 See markup.

28 F 2 N S Markup was corrected.

29 H 2 M S A41.7 30 H 3 N S D41.7 31 F 3 B S D41.4 32 H 2 N S B41.7 33 H 3 N S B41.5 E D41.5 See markup.

34 H 2 M S Markup was corrected.

35 F 2 N S A41.10 36 H 2 N S A41.7 E C41.5 See markup.

37 F 3 B S Markup was corrected.

38 H 3 N S C41.7 39 F 3 N S D41.7 40 F 3 N S C41.7 41 F 3 N S A41.4 42 H 3 N S C41.7 NOT true/false because knowledge of mini-flow ops is required to answer.

E C41.7 See markup.

43 H 3 N S Markup was corrected.

44 H 3 N S A41.7 45 H 2 M S C41.7 46 H 3 N S A, 41.7 47 F 2 N S A, 41.10 Page 3 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 48 H 3 M S A, 41.7 49 H 3 B S A, 41.7 50 H 3 B S A, 41.7 51 F 2 B S B, 41.7 52 H 3 N S C, 41.11 53 H 3 N S B, 41.7 54 F 2 B S A, 41.7 55 H 3 N S C, 41.7 56 H 3 N S A, 41.5 57 F 2 M S B, 41.2 58 F 2 N S A, 41.7 59 F 2 N S D, 41.2 60 H 2 B S A, 41.7 A, 41.10 - This question could be argued to have no correct answer since it does not E

61 F 3 N address the conditions as presented in the LOCA EOP (step 66). FIX S

The question was revised to correct the stem.

62 H 3 N S A, 41.7 63 H 3 M S B, 41.5 64 F 2 N S B, 41.4 65 H 3 N S C, 41.7 66 F 2 N S B, 41.10 D, 41.7 - This is a KA mismatch because Tier 3 questions are meant to have generic U

67 H 3 U M implications and not system specific.

S This question was revised.

68 F 3 B S C, 41.10 D, 41.3 - Not sure the question is correct. The TS limits may not be the same as the 69 F 3 N E design criteria. CAF. Page 4 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S This question is ok as written.

70 H 3 N S A, 41.10 71 F 3 N S B, 41.10 72 H 3 N S B, 41.12 73 F 2 N S B, 41.12 E D, 41.10 - Clean up the stem.

74 H 3 N S The stem was redone to fix the formatting.

75 F 2 N S C, 41.10 - Could not locate the referenced picture.

76 H 3 N S B, 43.5 77 H 3 N S C, 43.5 78 H 3 M S D, 43.5 79 H 3 M S D, 43.5 80 H 2 B S A, 43.5 D, 41.10 - This is an RO question because it tests on specific actions taken by the U

81 H 3 U M operator.

S This question was revised.

82 H 3 M S C, 43.5 83 H 3 N S A, 43.5 84 H 3 M S A, 43.5 E B, 43.2 - Not sure why this is a memory question. CAF 85 F 3 M S This question is supported by a learning objective.

86 F 2 N S D, 41.10 87 H 3 N S D, 43.5 88 H 3 N S B, 41.10 89 H 4 N S A, 43.5 90 F 3 M S A, 43.5 Page 5 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 91 H 3 N S A, 43.5 92 H 3 B S A, 43.5 93 H 3 N S B, 43.2 D, 41.10 - This is RO knowledge since it can be answered without doing a procedure U

94 H 3 U N selection.

S This question is ok as written.

95 F 3 N S A, 41.10 E B, 43.2 - Not sure SROs need to have all the 3.0.3 entry specs memorized. CAF 96 F 3 B S OK as written.

97 H 3 N S C, 43.5 U C, 41.10 - This is RO knowledge since it tests on specific operator actions.

98 H 3 U N S This question is ok as written.

U B, 41.10 - Procedure usage is RO knowledge.

99 F 2 U N S This question was replaced.

E B, 41.10 - Not sure this is a valid memory question.

100 F 3 E N S This question is supported by a learning objective.

B= F= E=

RO TOTALS: M= H= U= Additional Notes:

N=

B= F= E=

SRO TOTALS: M= H= U= Additional Notes:

N=

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is __ ___ on the RO exam and ___ on the SRO exam. Page 6 of 7 OBDI 202 - IOLE Process

ES-401 PV-2009-11 7 Form ES-401-9

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 43.1 =

41.2 = 43.2 =

41.3 = 43.3 =

41.4 = 43.4 =

41.5 = 43.5 =

41.6 = 43.6 =

41.7 = 43.7 =

41.8 =

41.9 =

41.10 =

41.11 =

41.12 =

41.13 =

41.14 =

5. The answer distribution is: RO / SRO A = 13 (17%) / 6 (24%)

B = 16 (22%) / 7 (28%)

C = 21 (28%) / 5 (20%)

D = 25 (33%) / 7 (28%)

6. There are questions with attachments provided. Page 7 of 7 OBDI 202 - IOLE Process