ML101240982

2010-04 Draft Written Exam Comments

ML101240982
Person / Time
Site:	Comanche Peak
Issue date:	03/29/2010
From:	Clyde Osterholtz Operations Branch IV
To:	Luminant Generation Co
References
50-445/10-301, 50-446/10-301
Download: ML101240982 (6)

Text

ES-401 CP-2010-03 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 X N US Two possible correct answers (a & b) REPLACED 2 F 2 B S 3 H 2 X N S 4 H 3 N S 5 H 2 B ES 6 H 3 N ES Whats an escutcheon? CHANGE TO HAND SWITCH 7 F 2 B S 8 H 3 X N US Two possible correct answers MODIFIED B 9 H 3 N S 10 F 2 N S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 1 OBDI 202 - IOLE Process

ES-401 CP-2010-03 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 2 (or more) correct answers if A, B, or C is correct MODIFIED STEM AND 11 H 2 X B US DISTRACTERS 12 H 3 B S 13 F 2 X B S 14 H 3 N US 2 correct answers + wrong K/A REVISED K/A/ - REVISED D 15 H 2 B ES MODIFIED A , C & D for BALANCE 16 H 2 N ES PUT FWIV IN STEM 17 H 2 X N US 2.) not necessary - REPLACED; TOPIC COVERED IN JPM 18 H 3 M ES Superfluous info in stem - MODIFIED 19 H 3 B ES 20 F 2 B ES Get rid of unnecessary info STEM AND DISTRACTERS MODIFIED 21 H 3 N ES 22 H 3 B ES K/A mismatch? CHANGED K/A AND ADDED ADDITIONAL REFERENCE 23 H 4 N ES B and C need enhanced DISTRACTERS ENHANCED 24 H 3 N S 25 H 3 X N ES B not credible REPLACED 26 H 3 X N ES Added MSODE 1 in stem 27 H 3 N ES 28 H 3 X B ES C nSot credible - has nothing to do with mid loop REVISED C 29 H 2 X MB ES A not credible - closer to point source yet 1/4 dose rate (?) - REPLACED QUESTION 30 F 2 N S 31 H 4 N ES 32 H 3 N ES 33 H 2 X B US B is also correct MODIFIED STEM TO MAKE B INCORRECT 34 H 3 B S Page 2 of 2 OBDI 202 - IOLE Process

ES-401 CP-2010-03 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 35 H 3 N S C could also be correct (primary concern) - REVISED STEM TO MAKE C 36 H 2 N US INCORRECT 37 H 3 N S 38 H 2 N S 39 F 2 X B S 40 H 3 X N ES Get rid of 2.s - EOS-1.1A transition by procedure E-0 DISTRACTERS REVISED 41 H 3 X B ES Revise to make distracter C more consistent DISTRACTER C REVISED 42 F 2 X B ES 43 F 3 B ES Modify A for consistency - C should have no s A & C MODIFIED 44 F 3 B ES Modify distracters for consistency DISTRACTERS MODIFIED 45 F 3 X B ES Remove stem cue STEM MODIFIED 46 H 3 N ES Modify D unnecessary info D MODIFIED 47 H 3 N S A and D not credible - no reason to believe ECA condition exists MODIFIED TO 48 H 3 X N US GET RID OF ECA CHOICES 49 H 4 N ES is the last bullet in stem necessary? STEM MODIFIED 50 H 3 N S 51 H 3 X N ES A not credible - change D for consistency (?) A & D MODIFIED 52 H 3 X X N ES Stem not consistent with distracter D - Why is B wrong? STEM & B & D MODIFIED 53 H 3 X M ES C not consistent with stem C MODIFIED 54 H 3 MB US Nearly identical to original question - OK if change from M to B CHANGED TO B 55 H 3 N US Two correct answers MODIFIED DISTRACTERS 56 H 3 M S 57 F 2 B S Page 3 of 3 OBDI 202 - IOLE Process

ES-401 CP-2010-03 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 58 H 2 X MB US Identical to bank question + A and C not credible CHANGED TO BANK 59 F 2 B S 60 H 2 N ES 61 F 2 X N ES D credible (?) - MODIFIED D 62 H 3 N ES 63 H 3 N ES 64 H 3 N S 65 H 2 X N ES MODIFIED DISTRACTERS Q not credible - When would following procedures be considered non conservative?

66 F 1 X N US CHANGED TO A DIFFERENT CORRECT ANSWER 67 F 2 B S 68 F 2 B S 69 F 3 B S 70 F 2 N S 71 F 2 M ES Change 2009 to 2010 -CHANGE MADE 72 F 2 X M ES Add minimum -CHANGE MADE 73 F 2 X N ES Add FRP Bases to stem -CHANGE MADE 74 F 2 B S If A were correct, so would C (A not credible) - MODIFIED STEM TO MAKE C 75 F 2 X X N ES INCORRECT 76 H 3 X X N ES 77 H 3 X B S 78 H 3 X X N S 79 H 2 X N S 80 F 2 X N ES Page 4 of 4 OBDI 202 - IOLE Process

ES-401 CP-2010-03 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only If RV head not fully tensioned, isnt that Mode 6 (doesnt that limit temp?) CHANGE 81 F 2 X B ES TO MODE 5 82 F 2 X N ES Conditions in stem match question? MODIFIED STEM 83 H 3 X B ES Add parameters to stem -ADDED 84 H 3 X X X N US A looks possibly correct and B and D dont appear credible - STEM REWORDED Only correct answer has a time limit -REMOVED TIME LIMIT FROM CORRECT 85 H 2 X N ES ANSWER 86 H 2 X X N ES Why is B wrong -MODIFIED STEM AND DISTRACTERS 87 H 2 X X N US Answer in stem (last bullet) - REMOVED CUE FROM STEM 88 H 2 X N ES Modified stem What does the diesel (just the diesel - no designation as to which one) have to do 89 H 2 X X N US with Bus 1ECc1? - STEM AND DISTRACTERS MODIFIED B not credible - clarify stem info - why is ref material needed? - B REVISED; 90 H 3 X X N ES PROPOSED REFERENCE REMOVED Why is D wrong? - nothing mentioned about potential curie content in stem 91 H 2 X X N US D MODIFIED 92 F 2 X B ES Provide reference for C -REFERENCE ADDED FOR C 93 F 2 X N ES Simplify stem; explain FP terms and references - MODIFIED 94 F 3 X X B ES If C were correct, so would A - makes C not credible - DISTRACTERS MODIFIED 95 F 2 X B ES provide references - REFERENCES ADDED 96 F 2 X B S 97 F 2 X N ES C - change 14 to 7 based on TS(?) - MODIFIED C 98 F 3 X B S Change to higher EAL - MODIFIED QUESTION TO MAKE CORRECT ANSWER A 99 H 3 X N ES HIGHER EAL 100 H 2 X M ES RO TOTALS: B= F= E= Additional Notes: Page 5 of 5 OBDI 202 - IOLE Process

ES-401 CP-2010-03 6 Form ES-401-9 M= H= U=

N=

B= F= E=

SRO TOTALS: M= H= U= Additional Notes:

N=

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is __ ___ on the RO exam and ___ on the SRO exam.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 43.1 =

41.2 = 43.2 =

41.3 = 43.3 =

41.4 = 43.4 =

41.5 = 43.5 =

41.6 = 43.6 =

41.7 = 43.7 =

41.8 =

41.9 =

41.10 =

41.11 =

41.12 =

41.13 =

41.14 =

5. The answer distribution is: RO / SRO A = 13 (17%) / 6 (24%)

B = 16 (22%) / 7 (28%)

C = 21 (28%) / 5 (20%)

D = 25 (33%) / 7 (28%)

6. There are questions with attachments provided. Page 6 of 6 OBDI 202 - IOLE Process