ML101960329

PV-2009-11-Draft Written Exam Comments

ML101960329
Person / Time
Site:	Palo Verde
Issue date:	11/02/2009
From:	Garchow S M Operations Branch IV
To:	Arizona Public Service Co
References
Download: ML101960329 (7)

Text

ES-401 PV-2009-11 Written Examination Review Worksheet Form ES-401-9 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).

$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer

=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Attachment 8 Page 1 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 1 H 3 N S C41.10 Agree 2 H 2 N E S D41.4 Answer states that the switch needs to be placed in START; however, the reference says place the switch in AUTO. Agree - Stem was revised to correct wording. 3 F 2 N E S D41.7 Reformat the answers without all the bullets.

Agree 4 F 2 M S A41.10 May clue that question 1, distractors A and B are incorrect. OK as written. 5 H 3 N S D41.3 6 H 3 N E S B41.7 See markup Comment was corrected.

ES-401 PV-2009-11 2 Form ES-401-9 Attachment 8 Page 2 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 7 H 3 N E S B41.4 See markup Markup was corrected. 8 H 2 B S B41.3 9 F 2 N S D41.1 10 F 3 B E S C41.10 See markup Markup was corrected. 11 F 2 N S B41.5 12 F 2 N S A41.5 13 H 3 N S D41.5 14 H 2 M S D41.10 15 F 2 B S B41.7 16 H 3 N S C41.5 17 F 3 N S B41.4 18 H 2 M S D41.5 19 F 2 U N U S C41.6, 43.2 See markups. Agree - TS bases makes it an SRO question.

The TS bases was removed from the question. 20 F 3 N S B41.5, 10 21 H 3 M S B41.7 22 H 3 M E S D41.5 See markup.

Markup was revised. 23 F 2 B S C41.7 24 H 3 N S C41.10 25 F 3 B S A41.11 26 H 3 N S B41.10 27 H 3 N E D41.10 VERIFY that PBB-S04 cannot be energized based on given information.

ES-401 PV-2009-11 3 Form ES-401-9 Attachment 8 Page 3 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation S It would seem distractor B could be argued as also correct. Distractor B was revised. 28 F 2 N E S A41.5 See markup. Markup was corrected. 29 H 2 M S A41.7 30 H 3 N S D41.7 31 F 3 B S D41.4 32 H 2 N S B41.7 33 H 3 N S B41.5 34 H 2 M E S D41.5 See markup. Markup was corrected. 35 F 2 N S A41.10 36 H 2 N S A41.7 37 F 3 B E S C41.5 See markup. Markup was corrected. 38 H 3 N S C41.7 39 F 3 N S D41.7 40 F 3 N S C41.7 41 F 3 N S A41.4 42 H 3 N S C41.7 NOT true/false because knowledge of mini-flow ops is required to answer. 43 H 3 N E S C41.7 See markup. Markup was corrected. 44 H 3 N S A41.7 45 H 2 M S C41.7 46 H 3 N S A, 41.7 47 F 2 N S A, 41.10 ES-401 PV-2009-11 4 Form ES-401-9 Attachment 8 Page 4 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 48 H 3 M S A, 41.7 49 H 3 B S A, 41.7 50 H 3 B S A, 41.7 51 F 2 B S B, 41.7 52 H 3 N S C, 41.11 53 H 3 N S B, 41.7 54 F 2 B S A, 41.7 55 H 3 N S C, 41.7 56 H 3 N S A, 41.5 57 F 2 M S B, 41.2 58 F 2 N S A, 41.7 59 F 2 N S D, 41.2 60 H 2 B S A, 41.7 61 F 3 N E S A, 41.10 - This question could be argued to have no correct answer since it does not address the conditions as presented in the LOCA EOP (step 66). FIX The question was revised to correct the stem. 62 H 3 N S A, 41.7 63 H 3 M S B, 41.5 64 F 2 N S B, 41.4 65 H 3 N S C, 41.7 66 F 2 N S B, 41.10 67 H 3 U M U S D, 41.7 - This is a KA mismatch because Tier 3 questions are meant to have generic implications and not system specific. This question was revised. 68 F 3 B S C, 41.10 69 F 3 N E D, 41.3 - Not sure the question is correct. The TS limits may not be the same as the design criteria. CAF.

ES-401 PV-2009-11 5 Form ES-401-9 Attachment 8 Page 5 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation S This question is ok as written. 70 H 3 N S A, 41.10 71 F 3 N S B, 41.10 72 H 3 N S B, 41.12 73 F 2 N S B, 41.12 74 H 3 N E S D, 41.10 - Clean up the stem. The stem was redone to fix the formatting. 75 F 2 N S C, 41.10 - Could not locate the referenced picture. 76 H 3 N S B, 43.5 77 H 3 N S C, 43.5 78 H 3 M S D, 43.5 79 H 3 M S D, 43.5 80 H 2 B S A, 43.5 81 H 3 U M U S D, 41.10 - This is an RO question because it tests on specific actions taken by the operator. This question was revised. 82 H 3 M S C, 43.5 83 H 3 N S A, 43.5 84 H 3 M S A, 43.5 85 F 3 M E S B, 43.2 - Not sure why this is a memory question. CAF This question is supported by a learning objective. 86 F 2 N S D, 41.10 87 H 3 N S D, 43.5 88 H 3 N S B, 41.10 89 H 4 N S A, 43.5 90 F 3 M S A, 43.5 ES-401 PV-2009-11 6 Form ES-401-9 Attachment 8 Page 6 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 91 H 3 N S A, 43.5 92 H 3 B S A, 43.5 93 H 3 N S B, 43.2 94 H 3 U N U S D, 41.10 - This is RO knowledge since it can be answered without doing a procedure selection. This question is ok as written. 95 F 3 N S A, 41.10 96 F 3 B E S B, 43.2 - Not sure SROs need to have all the 3.0.3 entry specs memorized. CAF OK as written. 97 H 3 N S C, 43.5 98 H 3 U N U S C, 41.10 - This is RO knowledge since it tests on specific operator actions. This question is ok as written. 99 F 2 U N U S B, 41.10 - Procedure usage is RO knowledge. This question was replaced. 100 F 3 E N E S B, 41.10 - Not sure this is a valid memory question. This question is supported by a learning objective. B= F= E= M= H= U= RO TOTALS: N= Additional Notes: B= F= E= M= H= U= SRO TOTALS: N= Additional Notes:

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue. 3. Average difficulty is __ ___ on the RO exam and ___ on the SRO exam.

ES-401 PV-2009-11 7 Form ES-401-9 Attachment 8 Page 7 of 7 OBDI 202 - IOLE Process

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 43.1 = 41.2 = 43.2 = 41.3 = 43.3 = 41.4 = 43.4 =

41.5 = 43.5 = 41.6 = 43.6 = 41.7 = 43.7 =

41.8 = 41.9 = 41.10 =

41.11 = 41.12 = 41.13 =

41.14 =

5. The answer distribution is: RO / SRO A = 13 (17%) / 6 (24%) B = 16 (22%) / 7 (28%)

C = 21 (28%) / 5 (20%) D = 25 (33%) / 7 (28%)

6. There are questions with attachments provided.