ML13123A324

SO-2012-10-Draft Written Exam Comments

ML13123A324
Person / Time
Site:	San Onofre
Issue date:	11/01/2012
From:	Apger G Operations Branch IV
To:	Southern California Edison Co
laura hurley
References
Download: ML13123A324 (6)

Text

ES-401 SO-2012-10 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 2 N S B 2 F 2 M S B 3 F 2 N S B 4 H 3 B S C 5 F 2 B S C2009 6 H 3 B S C 7 H 3 N S A 8 H 3 M S C 9 H 3 B S C Is SBCS RPS? If so OK. Else unsat.

10 H 3 N S B Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 6 OBDI 202 - IOLE Process

ES-401 SO-2012-10 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 11 F 3 N S D 12 H 3 B S D 13 F 2 N S B 14 H 3 N S A 15 H 3 N S C 16 H 3 X N E A Need to reference procedure in stem.

17 H 4 N S B 18 H 3 N S C 19 H 3 N S D 20 H 3 N S B U

21 H 2 X B C RESOLVED S

22 H 3 N S A 23 F 2 B S B 2009 24 F 2 N S D 25 F 2 N S A 26 H 3 B S C 27 F 3 N S B 28 H 3 B S B2011 29 F 3 N S B 30 H 3 N S A 31 H 3 N S D 32 F 2 B S A 33 F 2 X N E C Look at other failure modes not causing 2 correct answers.

34 F 3 N S B Page 2 of 6 OBDI 202 - IOLE Process

ES-401 SO-2012-10 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 35 H 3 B S A 36 H 3 N S B 37 H 2 B S B 38 F 2 N S A 39 H 3 N S B 40 F 2 B S B 41 F 3 B S B2011 42 H 3 N S D 43 H 3 B S A 44 H 3 B E A Change entered to affected.

45 F 2 N E A Modify D distractor.

46 H 3 B S B2011 47 H 3 N S C 48 H 3 N S A 49 F 3 N S B 50 F 3 N S C 51 F 3 N S C 52 F 3 N S C 53 H 3 N S A 54 H 3 N S B 55 F 2 N S D 56 H 3 N S B 57 F 2 B S A 58 H 2 N S A Page 3 of 6 OBDI 202 - IOLE Process

ES-401 SO-2012-10 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 59 H 3 N E A Look at. Conditions may require trip.

60 F 2 B S A Changed cognitive to fundamental.

61 F 2 N S A 62 F 3 B S D 63 H 3 B S B 2006 64 F 2 B S D 65 H 4 N S A 66 F 3 N S C 67 H 3 N S B U

68 H 3 B S C Good question but change to more generic. RESOLVED 69 F 2 B S B 70 F 3 B S D 71 H 3 B S D 72 H 3 M S A 73 F 2 M S C 74 F 2 N S C 75 F 2 B S A 76 H 3 N S B 77 H 3 N S D 78 F 2 N S C 79 H 3 N S B 80 H 3 N E A Adding return to outfall 81 H 3 N S C 82 H 3 N S D Page 4 of 6 OBDI 202 - IOLE Process

ES-401 SO-2012-10 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 83 H 4 N S D Need explanations 84 H 2 B S D 85 H 3 B S C 86 H 3 B S C 87 H 2 N S C 88 H 4 N S A 89 H 3 B S B 90 H 3 N S A Added power 91 H 4 N S D 92 F 2 N S B U

93 H 3 X B B Changing KA. RO question. RESOLVED S

94 F 2 N S B 95 H 3 N S A 96 H 4 N S B Changed screening to evaluation.

97 F 3 N S B 98 H 2 B E A correct D Looked at generic vs system specific. Meets KA, asking about EOP basis, so OK 99 H 2 B S as is.

100 F 2 N S D B= 27 F= 34 E= 5 RO TOTALS: M= 2 H= 41 U= 2 Additional Notes:

N= 46 B= 7 F= 5 E= 2 SRO TOTALS: Additional Notes:

M= 0 H= 20 U= 1 Page 5 of 6 OBDI 202 - IOLE Process

ES-401 SO-2012-10 6 Form ES-401-9 N= 18 Page 6 of 6 OBDI 202 - IOLE Process