ML120880625

2012-03-DRAFT-Written Exam Comments

ML120880625
Person / Time
Site:	Diablo Canyon
Issue date:	03/09/2012
From:	Division of Reactor Safety IV
To:	Pacific Gas & Electric Co
References
DC-2012-01, DC-2012-03, ES-401, ES-401-9
Download: ML120880625 (5)

Text

ES-401 DC-2012-01 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 N S 2 H 3 X X N ES Remove 25% from stem Removed 3 F 2 N S 4 H 3 X N ES Add valve name; A&B explanations backward Valve name added, A&B revised 5 H 3 X X N ES Add stem focus; make B&C more credible Stem and B&C modified 6 H 3 X X B ES Add stem and distracter focus Added 7 H 2 X N ES A not credible Replaced A 8 H 2 X N US C only answer plausible Modified Q to make distracters more plausible 9 H 2 X N ES Add stem focus Moved to Q 34 - Replaced with new Q 10 H 2 X X N US Got correct answer w/o reading Q Modified distracters to remove problem Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 5 OBDI 202 - IOLE Process

ES-401 DC-2012-01 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 3 X N US Two correct answers Modified Q to ensure only one correct answer 12 H 2 X X M ES Modify all distracters to make Q plausible Distracters modified 13 F 2 X N ES Modify D&C to be consistent with A&B Modified 14 H 2 X N ES Modify stem Modified 15 H 2 X X N ES Modify stem and distracters Modified 16 H 2 X M ES Get rid of unnecessary info in stem Modified stem 17 H 2 X B ES Add the in stem Added 18 F 3 N S 19 H 3 X N ES Modify stem Modified 20 F 3 X M ES D not an action (replace D) Replaced D 21 F 3 N S 22 F 3 X X N ES Modify stem; add second half to A Modified Stem and distracter A 23 F 2 X N ES Modify stem Modified 24 F 2 X B S 25 F 2 B S 26 F 2 X B ES Modify stem to include info in all four distracters Modified 27 F 2 X B ES Change stem to make B correct Changed stem 28 F 2 B S 29 H 3 X N ES Modify stem Modified 30 H 2 N S 31 H 3 B S 32 F 3 X N US A &C not credible Replaced 33 F 3 X N ES Change D to average Changed 34 F 3 X N US Three of four distracters can be eliminated with partial knowledge Replaced Q 35 F 2 X N ES Modify stem Modified Page 2 of 5 OBDI 202 - IOLE Process

ES-401 DC-2012-01 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 36 F 3 X X N ES Modify stem; add info to C&D Modified stem; C&D info added 37 F 3 X N ES Add caps to only and and in all distracters Added 38 H 3 X N ES Modify stem; change will to should Modified stem, change made 39 F 3 X B ES Add actual temp in stem Added 40 H 2 X N ES Get rid of steam table reference Removed 41 H 2 X N ES Modify stem Modified 42 H 2 N S 43 H 3 N S 44 F 3 X B ES Modify stem Modified 45 F 3 X X N ES Modify stem; C contradicts info in stem (change) Modified stem, Changed C 46 H 2 X N ES Add clarification to all distracters Added 47 H 2 X N US Two non credible distracters Replaced distracter A 48 F 3 X X N ES Change EDGs to 11 and 12 (change correct answer) Changed 49 H 3 X N ES A none of the above (replace) Replaced A 50 H 3 X X N ES Modify stem; fix A to ensure not cueing correct answer Modified stem and distracter A 51 H 2 X X N ES Modify stem; change temps in distracters to off exact setpoint Modified stem & temps 52 H 2 X M US Too similar to Q 27 (replace) Replaced Q 53 H 3 X B ES Modify stem Modified 54 F 2 B S 55 H 3 N S 56 H 2 X B US A&C not credible Replaced Q 57 H 2 N S 58 H 2 X M ES Remove turbine load info from stem Replaced Q 59 H 2 X N ES Remove no effect from A Removed 60 H 3 X B ES Capitalize only in stem Capitalized only Page 3 of 5 OBDI 202 - IOLE Process

ES-401 DC-2012-01 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 61 H 2 X N ES Capitalize maximum in stem Maximum capitalized 62 F 2 X X N ES Modify stem; revise A,B,&C Modified stem, revised A,B,&C 63 H 3 X X N ES Modify stem; verify A incorrect Modified stem and distracter A 64 H 3 X B ES Modify stem Modified 65 H 3 X N ES Modify stem Modified 66 F 2 B S 67 F 2 N S 68 H 3 N S 69 F 2 X B ES Modify stem to ensure only one correct answer Modified 70 F 2 N S 71 F 3 X N ES Modify stem Modified 72 F 3 X B ES Modify stem Modified 73 F 2 X N S 74 F 2 X X N ES Modify stem to ensure only one correct answer Modified 75 H 3 B S 76 F 3 X X N ES Replace directs with considers in stem Revised stem 77 H 3 X X N ES Modify stem Modified 78 H 3 X X N ES Revise B&C Revised 79 H 3 X X N ES Ensure D incorrect Replaced D 80 H 3 X X M ES Modify stem, ensure B incorrect Modified stem and ensured B incorrect 81 F 2 X X N ES Modify A Replaced A 82 H 3 X N S 83 H 3 X B S 84 F 2 X N S 85 F 3 X X N ES Modify stem Modified Page 4 of 5 OBDI 202 - IOLE Process

ES-401 DC-2012-01 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 86 H 3 X X N ES Modify stem Modified 87 H 2 X N S 88 H 2 X X N US A&B not credible - modify stem to have PZR level at 12% Modified A,B, & stem 89 H 2 X X N ES Add power level to stem - get rid of unnecessary info Added and revised 90 F 2 X X X N S 91 H 3 X X N ES Supplement D Supplemented 92 H 3 X X N ES Modify A Modified A 93 H 3 X X N ES Add additional words to C to make it consistent with other distracters Modified C 94 F 2 X X B ES Modify C Modified 95 F 2 X X B ES Modify stem Modified 96 F 2 X X N ES Put required in caps in stem Modified stem 97 F 3 X X N ES Modify stem Modified 98 F 3 X X X N ES Replace D; remove if at all from stem Modified stem, Replaced D 99 H 3 X X B ES Add where explosion occurred in stem Added 100 H 2 X X N ES Modify stem Modified Page 5 of 5 OBDI 202 - IOLE Process