ML103210263

SONGS-2010-10-Draft Written Examination Comments

ML103210263
Person / Time
Site:	San Onofre
Issue date:	10/15/2010
From:	Clyde Osterholtz Operations Branch IV
To:	Southern California Edison Co
References
Download: ML103210263 (5)

Text

ES-401 SONGS 2010-10 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 X B US Two correct answers MODIFIED DISTRACTERS 2 F 2 N ES ADDED AUTOMATICALLY 3 F 2 B S 4 H 3 X B ES B and C same answer REVISED B 5 H 3 B ES Define Stem DEFINED STEM 6 H 3 X N ES B not credible REVISED B 7 F 2 N ES Add level ADDED 8 F 2 N S 9 F 2 B S 10 F 2 B ES Change A to one in A and B - fix margin REVISED Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 5 OBDI 202 - IOLE Process

ES-401 SONGS-2010-10 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 2 N S 12 H 3 X N ES Change D to be consistent with C REVISED 13 F 2 B S 14 H 2 N ES D- an to a REVISED 15 H 4 B ES Modify stem from <19% to actual level MODIFIED 16 H 3 N ES Fix typo FIXED 17 H 3 N ES Fix typo FIXED 18 H 3 B S 19 F 2 X B ES Change to 2 140s and 2 504s for consistency REVISED 20 H 4 X X N US Two correct answers REPLACED 21 H 3 X X B US Answer in stem REVISED 22 H 2 X N ES Change correct answer to be less obvious CHANGED 23 F 2 B S 24 F 2 B S 25 H 2 N S 26 H 3 N S 27 H 3 N S 28 F 2 N S Unnecessary info in stem makes question 50/50 - REMOVED UNNECESSARY 29 F 4 X M US INFO 30 H 3 X B US Correct answer obvious REVISED 31 F 3 X N ES D not plausible REVISED 32 H 3 X B ES Modify A for consistency MODIFIED 33 F 2 N ES Fix typo FIXED 34 F 2 N S Page 2 of 5 OBDI 202 - IOLE Process

ES-401 SONGS-2010-10 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 35 H 2 B S 36 H 3 B S 37 H 3 N ES Add EOI in stem ADDED 38 H 2 B ES Get rid of modulate in C REVISED 39 H 3 B S 40 H 2 B S 41 H 3 B S 42 H 2 B ES Get rid of underline in stem REVISED 43 H 2 X B US B and D not plausible MODIFIED 44 H 2 N ES D not plausible MODIFIED 45 H 2 X N ES Change C to be consistent with others MODIFIED 46 H 2 B S 47 H 2 B ES Clarify ESDE CLARIFIED 48 H 4 B S 49 H 3 B S 50 H 3 B S 51 H 3 N S 52 H 3 N ES Fix typo FIXED 53 H 3 N S 54 F 3 X M ES Modify distracters to use actual values MODIFIED 55 F 2 X B ES Modify distracters to remove unnecessary info MODIFIED 56 H 3 N S 57 F 2 B S 58 H 3 X N ES D mentions a deviation not described in stem MODIFIED 59 F 2 X N S Page 3 of 5 OBDI 202 - IOLE Process

ES-401 SONGS-2010-10 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 60 H 2 X N ES Modify D to make credible MODIFIED 61 F 3 X X M US No correct answer MODIFIED 62 H 2 N ES Add fully to dropped ADDED 63 H 2 M S 64 H 3 X N US A and C do not address question REPLACED 65 H 2 X B US Only one credible distracter REPLACED 66 F 2 X N US No correct answer REPLACED 67 F 3 B S 68 F 2 X B ES D not credible MODIFIED 69 F 2 B ES Add in service to temporary power ADDED 70 F 3 X B US A and D correct MODIFIED A 71 F 2 X B ES Add MINIMUM to stem ADDED 72 F 2 N S 73 F 2 X B ES Modify C and D for consistency MODIFIED 74 F 2 X X B ES Modify stem and distracters for consistency MODIFIED 75 F 2 B S 76 H 3 X N ES Add clarifying info ADDED 77 H 3 X X N US Question/K/A mismatch REPLACED 78 H 3 X N S 79 H 3 X X B ES D not plausible REVISED 80 F 3 X N S 81 F 4 X B S 82 F 3 X B S 83 H 4 X N S 84 H 2 X B S Page 4 of 5 OBDI 202 - IOLE Process

ES-401 SONGS-2010-10 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 85 H 2 X X B ES B not plausible REVISED Modify stem - add power level, CDM temperature, remove delay and rising 86 H 3 X N ES MODIFIED 87 H 2 X N ES Change psia to psig CHANGED 88 H 4 X B S 89 H 3 X X N ES Change Mode 5 to temperature in D MODIFIED 90 H 2 X X N ES Change 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> to 6 and 10 in A and D respectively CHANGED 91 H 3 X X N ES Stem needs a rad level to properly evaluate ADDED 92 H 3 X X N US Question/ K/A mismatch REPLACED 93 H 3 X X X N US Three correct answers REPLACED 94 H 2 X X B ES B not credible REVISED 95 H 3 X X M ES Modify stem to ensure only one correct answer MODIFIED 96 F 2 X B S 97 H 2 X X B ES A not credible MODIFIED 98 F 2 X X B US No correct answer ALL DISTRACTERS MODIFIED EXCEPT B 99 F 2 X B S 100 F 2 X B S Page 5 of 5 OBDI 202 - IOLE Process