ML100910278
| ML100910278 | |
| Person / Time | |
|---|---|
| Site: | Columbia  |
| Issue date: | 10/14/2009 |
| From: | Garchow S Operations Branch IV |
| To: | Energy Northwest |
| References | |
| Download: ML100910278 (7) | |
Text
ES-401 CG-2009-01 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 N S B, CR, 41.5 2 H 3 N S C, CR, 41.10 3 F 3 N S C, CR, 41.7 4 H 3 M S D, CR, 41.5 5 F 3 M S A, CR, 41.6 C, CR, 41.10 - This question is testing on an operator action that is 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> into an E
6 F 3 E N event. Not sure this is a valid memory question.
S Ops validation indicated this is a fair question. OK as is.
7 H 3 N S D, CR, 41.4 8 H 3 N S B, CR, 41.4 9 H 2 B E A, CR, 41.5 - Recommend adding the reactor head is still installed.
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process
ES-401 CG-2009-10 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S This was corrected.
C, CR, 41.10 - This question is considered an SRO Only question because the second part of the distractors is testing on refueling actions taken on the refueling U bridge.
10 F 3 U N S Will follow up on the LO.
This question is supported by an RO LO.
11 F 3 N S B, CR, 41.7 12 F 3 B S A, CR, 41.6 C, OR, 41.10 - This question is considered unsat because the correct distractor is U
13 H 2 U M 225 degrees and not 210 degrees as indicated by the answer key.
S The answer key is correct.
D, OR, 41.10 - This question is considered unsat because with drywell temperature at 332 degrees, dw sprays should have already been initiated. Assuming dw sprays U were initiated per the EOP and dw temperature continues to increase, then PPM 14 H 3 U N 5.1.1 and an ED are required as stated in distractor D.
S Will followup after researching.
This question was revised.
15 F 3 N S B, CR, 41.9 16 F 3 N S C, CR, 41.2 17 F 3 B S A, CR, 41.10 18 F 3 M S D, CR, 41.10 19 F 3 N S C, CR, 41.7 20 H 3 N S A, CR, 41.10 21 H 3 N S B, CR, 41.6 22 F 3 N S A, CR, 41.7 23 F 3 N S D, CR, 41.10 24 F 2 N S D, CR, 41.10 25 F 3 B S B, CR, 41.9 26 H 3 N S A, CR, 41.7 Page 2 of 7 OBDI 202 - IOLE Process
ES-401 CG-2009-10 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 27 H 3 N S B, CR, 41.6 28 H 3 N S D, CR, 41.7 E A, CR, 41.7 - Recommend changing distractor B to RHR-P-2B trips after either 29 F 3 E N S Revised the question.
30 H 3 N S B, CR, 41.7 31 H 2 N S D, CR, 41.7 32 H 3 B S B, CR, 41.7 33 H 3 N S A, CR, 41.6 34 H 3 N S A, CR, 41.2 35 H 3 M S C, CR, 41.2 36 F 2 N S B, CR, 41.6 37 F 2 N S D, CR, 41.7 38 F 3 N S B, CR, 41.7 39 H 3 N S C, CR, 41.9 C, CR, 41.5 - Recommend changing the stem to read describes the immediate E
40 H 3 E N effect.
S Made the recommended changes.
E A, CR, 41.4 - Recommend adding the noun name for RFW-DPT-4B to the stem.
41 H 3 E N S Made the recommended changes.
42 H 3 B S D, CR, 41.7 A, CR, 41.4 - This is considered unsat because the second part of the distractors is U E-plan / SRO knowledge.
43 F 4 U N S Will make the distractors RO knowledge.
The distractors were revised.
44 H 3 N S B, CR, 41.9 E A, CR, 41.4 - Recommend making the stem read which indicates 5 K[ohms].
45 H 3 E N S Recommended change was made. Page 3 of 7 OBDI 202 - IOLE Process
ES-401 CG-2009-10 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only B, CR, 41.7 - This question is considered unsat because distractor A could also be U considered correct.
46 H 3 U N S Will reword distractor A.
Distractor A was revised.
C, CR, 41.4 - This question is considered unsat because distractor A is a subset of U the correct distractor making it also correct.
47 H 3 U N S Revised distractor A.
Distractor A was revised.
48 F 3 N S B, CR, 41.4 49 F 3 N S D, CR, 41.4 50 F 3 N S A, CR, 41.7 51 H 3 N S D, CR, 41.5 52 H 3 B S B, CR, 41.2 53 H 3 N S C, CR, 41.4 54 H 3 N S A, CR, 41.4 55 H 3 N S B, CR, 41.6 56 H 3 N S D, CR, 41.3 57 H 3 N S D, CR, 41.10 58 F 3 N S C, CR, 41.3 59 F 3 N S B, CR, 41.2 60 F 3 N S A, CR, 41.4 A, OR, 43.2 - This is an SRO Only question because it tests on the application of a U
61 H 3 U N TS (RO knowledge is above the LCO line in TS).
S This question is ok as written.
62 H 3 M S C, CR, 41.4 63 F 3 N S D, CR, 41.9 64 H 3 N S B, CR, 41.7 Page 4 of 7 OBDI 202 - IOLE Process
ES-401 CG-2009-10 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 65 F 3 N S D, CR, 41.6 66 F 2 N S C, CR, 41.10 67 F 4 N S B, CR, 41.10 68 F 2 N S A, CR, 41.10 69 F 2 N S C, CR, 41.10 70 H 2 N S D, CR, 41.10 71 F 2 N S A, CR, 41.10 72 F 2 N S C, CR, 43.2 73 F 2 N S B, OR, 41.12 74 H 3 N S C, CR, 41.10 75 F 3 N S D, CR, 41.10 76 H 3 N S D, CR, 43.5 77 H 3 N S C, OR, 43.5 78 H 3 N S B, CR, 43.5 79 H 3 N S C, CR, 43.2 A, OR, 43.5 - This is a direct lookup since the stem states H2 concentration cannot be determined to be below 0.6% verbatim from the provided procedure reference and U the corresponding action is provided in the same reference.
80 H 1 U N S Will revise following research.
The stem was revised.
81 H 3 N S D, CR, 43.5 82 H 3 N S C, OR, 43.5 83 H 2 N S A, OR, 43.2 84 F 2 N S B, CR, 43.3 85 H 3 N S B, CR, 43.2 86 H 3 N S C, CR, 43.5 Page 5 of 7 OBDI 202 - IOLE Process
ES-401 CG-2009-10 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 87 H 3 N S A, OR, 43.5 88 H 2 N S B, OR, 43.2 89 H 3 N S D, CR, 43.5 90 H 3 N S D, OR, 43.2 91 H 3 N S A, OR, 43.2 92 H 3 N S A, OR, 43.5 93 H 3 M S A, OR, 43.2 94 H 3 N S D, OR, 43.2 U D, OR, 43.5 - Considered unsat because it is testing the same material as Q14.
95 H 3 U N S Question is ok as is.
96 H 3 N S A, CR, 43.5 97 F 3 N S B, OR, 43.5 B, CR, 43.5 - The justification for the correct answer could not be found in the E
98 F 2 E N references provided.
S This question is ok as written.
99 H 3 N S C, CR, 43.5 100 F 2 N S C, OR, 43.5 B= 6 F= 36 (48%) E= 6 RO TOTALS: M= 8 H= 39 (52%) U= 7 Additional Notes:
N= 61 S= 62 B= 0 F= 3 (12%) E= 1 SRO TOTALS: M= 2 H= 22 (88%) U= 2 Additional Notes:
N= 23 S= 23
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N. Page 6 of 7 OBDI 202 - IOLE Process
ES-401 CG-2009-10 7 Form ES-401-9
- 2. Chief Examiner comments are indicated in blue.
- 3. Average difficulty is __2.85_ on the RO exam and 2.72 on the SRO exam. Both these values are within the normal expected difficulty range for this reviewer.
- 4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 0 41.2 = 7 43.2 = 8 41.3 = 2 43.3 = 1 41.4 = 12 43.4 = 0 41.5 = 5 43.5 = 16 41.6 = 8 43.6 = 0 41.7 = 16 43.7 = 0 41.8 = 0 41.9 = 5 41.10 = 19 41.11 = 0 41.12 = 1 41.13 =
41.14 =
- 6. There are 4 / 14 (RO/SRO) questions with attachments provided.
- 7. This review was performed by S. Garchow the week of 8/10/2009.
- 8. The SRO exam contains 88% higher cognitive questions. This is significantly higher than the recommended limit of 60%. Page 7 of 7 OBDI 202 - IOLE Process