ML112901413
| ML112901413 | |
| Person / Time | |
|---|---|
| Site: | Comanche Peak  |
| Issue date: | 06/06/2011 |
| From: | NRC Region 4 |
| To: | |
| References | |
| 50-445/11-06 | |
| Download: ML112901413 (7) | |
Text
ES-401 CP-2011-06 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only
B/ Added SOP-108A, Step 5.1.G, Figure 1 as a reference to why seal differential 1 H 2 N S pressure rising in Distractors A and C is plausible.
2 H 2 N S D/
A/ Possible two correct answers. B is correct if water level is less than 23 feet and the stem information does not include water level. Recommend indicating water level 3 F 3 B E above the 23 foot mark in the stem to preclude B being correct. Added bullet for Refueling Cavity level at 858 feet.
4 F 3 B S A/
B/ Plausible to have normal pump suction as cooling for CCP bearings? Normal 5 F 2 B E cooling is provided by oil from a separate oil pump that is cooled by a heat exchanger. This affects two answers. Modified Distractors C &D (replaced the word Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process
ES-401 CP-2011-06 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only suction with recirculation via).
6 H 2 B S D/
7 H 3 B S B/
8 H 3 B S C/
9 H 3 B E A/ Capitalized NOT in question stem.
10 H 3 B S C/
11 F 2 N S A/
12 F 2 B S D/
13 F 2 B S C/
C/ Plausible that securing Containment Fan Coolers 3 and 4 would isolate the leak?
14 H 3 B S Added Reference Drawing and System Description info on why valves must be locally shut.
15 H 3 N S B/
16 H 3 B S C/
17 F 2 B E D/ Revised Distractor B.
18 F 2 B S C/
19 F 2 B S B/
20 H 2 N S A/
21 H 3 B S B/
22 F 2 B S A/
C/ Overlap with Q8. Replaced Question and substituted K/A to avoid duplicity with 23 H 3 X B U Q08.
24 H 2 M S A/
25 H 2 N E A/ Added SOP-706 procedure title to Stem.
26 H 3 N E B/ Removed reference to Train and added SOP-501A procedure title to Stem.
27 F 2 B E B/ Minor wording change and added ABN-301 procedure title to Stem. Page 2 of 7 OBDI 202 - IOLE Process
ES-401 CP-2011-06 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 28 F 2 B S D/
29 F 3 B S D/
30 F 2 B S C/
B/ Ability to eval plant performance AND make operational judgments Raised power to 28% and removed last two bullets from Stem. Added whether a Turbine or 31 H 2 X B U Reactor trip was required to each Distractor. Changed the words the affected loop to Loop 1. Added ALM-0071A reference for why the Reactor would be tripped when an MSIV closes.
32 H 3 B S D/
33 H 3 B E D/ Changed the words consistent with to equal to in distractors B, C and D.
A/ ROs do not make OPERABILITY calls or TS calls below the line. Change Stem from Technical Specifications to OPT-112A-1, Accident Monitoring Instrumentation 34 H 2 X N U Check. Removed reference to Technical Specifications, rewrote Explanation, and replaced Objective.
35 H 2 B S B/
36 F 3 B S B/
37 H 2 M S A/
38 F 3 B E D/ Changed RM-11 to PC-11 in Distractors A & C.
39 H 3 B S D/
40 H 2 M S B/
B/ Frame vibration exceeded 2 mils/hr at 0200, not 0230. Adjusted the 0200 value 41 F 3 X M U for Frame vibration to make 0230 the correct answer.
42 H 4 B S A/(OR) 43 F 3 B S B/
44 H 3 B E C/ Moved reference to Unit 2 and added ABN-502 procedure title to Stem.
45 H 2 N E B/ Renumbered distractors and removed heater status to enhance plausibility.
46 H 2 B S B/
47 F 3 B E C/ Editorial Comment - add period to end of sentence on Distracter C. Corrected. Page 3 of 7 OBDI 202 - IOLE Process
ES-401 CP-2011-06 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 48 F 3 B E A/ Removed "... under the worst-case environmental conditions ... from the Stem.
A/ Removed Reactor Coolant Pumps have been secured and added ...A Loss of 49 H 3 B E Offsite Power has occurred. in the Stem. This effectively explains why RCPs are secured.
50 F 3 M S B/
51 H 3 B S A/
52 H 3 B S C/
D/ Plausibility of Distracter C - tripping the RX only on loss of SWS? Added ... and 53 H 2 B E Reactor Coolant Pumps." to Distractor C for balance.
54 H 3 B E B/
55 H 3 B S B/
56 H 3 B E A/ Added ...Turbine Control Valves... to Stem.
57 H 3 B S A/
58 H 3 B S D/
59 H 3 N S B/(OR) 60 H 3 N E A/ Added ABN-102 procedure title to Stem.
61 F 3 B S C/
C/ Question asks applicant to direct operations through the appropriate procedures.
62 H 3 X N U SRO level question. Question does meet the current K/A. Replaced Question and K/A.
63 H 3 B E C/ Added EOS-1.1A procedure title to Stem.
64 F 2 N S B/
65 F 2 B S D/
A/ Corrected punctuation and changed "... Continuous Action step." to "...
66 F 2 N E Continuous Action." In Distractors B and D.
67 F 3 N S C/
68 H 2 X B U C/(OR) Multiple correct answers - If I pick 790# that still gives me the answer when rounded up to 800#. Same for picking 820# and rounding down to 800# on EOP- Page 4 of 7 OBDI 202 - IOLE Process
ES-401 CP-2011-06 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 3.0A, Step 6.b. Modified Steam Generator pressures in the Stem and deleted Bases information provided in the Written Exam References.
69 F 2 B E B/ Added OWI-110 procedure title to Stem.
70 F 2 N S D/
71 F 3 B S D/
72 H 2 B E C/ Added ABN-902 procedure title to Stem.
73 F 2 B S B/
74 F 2 B E A/ Added EOP-0.0A/B procedure title to Stem.
75 F 2 B S C/
D/ Possible two correct answers. Stem info gives Safeguards DC Bus voltage at 105 VDC. ECA-0.0 says that minimum system voltage required for equipment operation is APPROXIMATELY 105 volts. ECA-0.0 then states that IF DC bus voltage is Less 76 H 3 B E than 110 volts AND plant staff determine it necessary to conserve batteries THEN 1)
Reference ABN-603, Loss of Protection or Instrument Bus. This makes Distracters C and D arguably correct. Modified reason in the second part of Distractor C since ABN-603 entry could be made. Modified Plausibility statement.
77 F 3 N S C/
C/ Entry conditions for AOPs or plant parameters that require direct entry to major 78 H 3 X B U EOPs are RO level questions. Also, knowledge of second half of question not required to determine correct answer. Replaced Question and K/A.
79 H 4 B S B/ Added FRZ-0.1A Critical Safety Functions Status Tree to references.
80 H 3 M S D/
81 H 3 B S C/ Added FRH-0.1A, Step 4 to references.
82 H 2 M S A/
C/ Added punctuation and Technical Specification REQUIRED ACTION for 83 H 3 N E Condition F to references.
84 H 2 B E D/ Added FRC-0.2B procedure title to Stem.
D/ Enhanced Explanation for the correct answer to differentiate why the maximum 85 H 3 B S allowable channel disagreement as stated in the K/A was addressed in the provided reference (OPT-102A-6). Page 5 of 7 OBDI 202 - IOLE Process
ES-401 CP-2011-06 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only A/ Modified Stem to include ABN-501 procedure title and reworded Distractors so 86 H 3 B E that reason for the action was removed.
C/ Revise using procedures in distracters as required by KA. Reworded Distractors 87 H 3 X N U to include SOP and ABN procedure titles.
88 H 2 N S B/
89 F 2 N S B/ Modified Explanation for correct answer.
A/ K/A mismatch - K/A requires ability to locate and operate components, including 90 H 3 X B U local controls. Question as written asks knowledge of automatic functions. Replaced Question and K/A.
A/ Question can be answered by simple systems knowledge. Also not necessary to be knowledgeable of the applicable TS to answer this question. - RO level question.
91 H 2 X X N U Modified distractors by removing procedure references and expanding Technical Specification criteria.
92 H 3 N S C/
93 F 3 B E D/ Added RFO-302 procedure title to Stem.
94 F 2 N S A/
95 H 3 N S D/
96 H 3 N S B/
97 F 2 N S D/
98 H 3 B E A/ Revised distracters to move common wording to the stem.
99 H 2 N S C/
100 H 3 B S D/
B= 54 (72%) F= 32 (43%) E= 23 (31%)
RO TOTALS: M= 5 (7%) H= 43 (57) U= 6 (8%) Additional Notes:
N= 16 (21%) S= 46 (61%)
B= 11 (44%) F= 5 (20%) E= 6 (24%)
SRO TOTALS: M= 2 (8%) H= 20 (80%) U= 4 (16%) Additional Notes:
N= 12 (48%) S= 15 (60) Page 6 of 7 OBDI 202 - IOLE Process
ES-401 CP-2011-06 7 Form ES-401-9
GENERAL COMMENT
S:
B = 23 (31%) / 4 (16%)
C = 18 (24%) / 7 (28%)
D = 15 (20%) / 7 (28%)