ML13127A263
| ML13127A263 | |
| Person / Time | |
|---|---|
| Site: | Waterford  |
| Issue date: | 10/01/2012 |
| From: | Brian Larson Operations Branch IV |
| To: | Entergy Operations |
| laura hurley | |
| References | |
| Download: ML13127A263 (19) | |
Text
ES-401 WT-2012-10 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only B - CR: Why is page 6 of 902-001 used as ref? Where is ref for core heat removal safety function? Correct answer explanation should include statement of effect of 1 H 3 N S 86STB lockout actuated.
The correct answer explanation was revised.
C - OR: Fourth bullet - give indications where applicant can determine all levels E voided instead of making the decision for the applicant. Identify what 2 H 2 N Appendix/Attachment used in 902-009 (with Rev) in Tech Refs.
S Question was revised.
C - CR: Identified refs not provided. No ref for QSPDS answer. Ref provided is TS 3 H 2 N S Table 2.2-1? Need to explain correct answer.
References were provided and the explanation for the correct answer was revised.
B - CR: In stem 1) why response to a steam flow or feed flow change? Ref states 5 4 H 3 N U to 15 minutes after RCPs have stopped. 2) following the action is confusing. Also, highlighting applicable sections of ref material is helpful.
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only The question was revised.
B - OR: Third bullet gives OP entry while KA is testing knowledge of Entry U
5 H 3 X N Conditions.
S This question was revised.
B - CR: If LPSI pumps were secured due to leakage, why is it okay to start either E
6 H 3 B pump? Stem doesnt state leakage was stopped.
S This question is ok as written.
C - CR: Question doesnt test AUTO actions within the CCW sys - the correct U
7 H 2 X N answer states the valves remain open (ie, no action)
S This question was revised.
C - CR: second bullet should be the answer. Give the indications of failed controller U
8 H 2 X N and ask for either cause or correct actions.
S This question was revised.
B - CR: Given these are immediate operator actions, its not plausible for distracters U to be 31A and 31B. Consider question that asks status of Rx trip bkrs and M/G set 9 H 2 X B Power Supply bkrs for a Manual Rx Trip and DRTS trip.
S This question was revised.
U D - CR: Second part of question is not part of KA.
10 H 3 X N S This question was revised.
D - CR: 1. What is the impact of unisolable leak on SG1? 2) Third bullet states the crew has completed actions for stabilizing RCS temp (procedure step 16). Stem then U
11 X N asks what the ATC will do (which has already been done). Consider deleting third S bullet and reword stem for actions necessary to stabilize RCS temp.
This question was revised.
E C - CR: removed train of ACCW from each answer since its in the stem.
12 H 2 B S This edit was corrected.
A - CR: 1) consider replacing first 3 bullets with A Station Blackout has occurred.
E 2) consider using 550F instead of 520F in distracters since 550F is the limit for RCS 13 H 3 N temperature control in Step 8 of 902-005.
S The question was revised.
E A - CR: 1) For indications of EDG B, first bullet - dont tell applicant that diesel is 14 H 2 N running; give indications so applicant can make this determination (speed is xx rpm)
S
- 2) Add to stem In accordance with OP-902-000, SPTAs, 3) KA refers to bus tie Page 2 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only breakers - does W3 have bus tie breakers that reposition on LOOP?
This question was revised.
C - CR: 1) Verify there is indication of these breakers being open. 901-312 bullets state loss of ALL indications on Remote Operating Module (CP-7) and loss of ALL E
15 F 3 B red and green trip path indicating lights on local status panel (CP-10) 2) No ref S provided to validate second part of question (reactor will not trip).
This question is ok as written.
A - CR: 2010 SRO exam: 1) How is a bank SRO question used on an RO exam?
16 F 3 B Isolation - which is correct? 3) given all the valves that are affected, why list these 3 S valves?
This question was revised.
A - CR: Distractors C and D are not plausible given the immediately statement U
17 F 3 X N for aligning Essential Air System.
S This question was revised.
D - CR: 1) Operationally, the applicant would receive indications of high rotor field U winding temperature and then would have to determine the cause. Also, no ref was 18 F 2 X N provided to validate the second part of the question.
S This question is ok as written.
E C - CR: For D - change to Trip the Reactor and go to OP-902-000, SPTAs.
19 F 2 N S This question was revised.
D - CR: 1) Take in MODE 3 out of second bullet and make it the first bullet - Plant E is in MODE 3 2) Reorder answers from shortest to longest (C - D - A - B) Correct 20 H 3 N answer is now B. 3) No ref provided to validate power supply to BAM pumps.
S This question was revised.
B - CR: 1) Is it normal to have 2 charging pumps running at 100% pwr? 2) need to E confirm this instrument failure is NOT in any of the scenarios. 3) Reorder answers 21 H 3 N from shortest to longest (B - C - D - A).
S This question was revised.
A - CR: 1) KA requires indications of approaching criticality - answer is NOT critical U or approaching criticality. 2) Explanations use at this power level - not an accurate 22 H 3 X N description given MODE 6. 3) Answer A states startup channels and boron dilution S alarms are used to monitor for criticality - alarms arent used to monitor for criticality.
Suggest replacing startup rate for boron dilution alarm (see NOTE in 010-003, p.72) Page 3 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only This question was revised.
E C - CR: Reorder answers from shortest to longest (D - C - A - B) 23 F 2 B S This question was revised.
24 F 3 N S B - CR:
A - CR: KA requires testing knowledge of the reasons for Required sequence of U
25 F 3 X N actions for emergency evac of CR.
S This question was revised.
E B - CR: Need to submit original Bank question.
26 F 3 M S The original bank question was submitted.
D - OR (stm tables): 1) What procedure guidance is available to make the second part of question valid (with info provided in stem)? 2) Is PZR press 2250 valid for 20 27 H 3 M S minutes after LOOP? (EDGs running, PZR htrs energized?)
This question is ok as written.
D - CR: 1) A and D should be low RCS flow, not steam generator flow. It is based on diff press across SG but ref identifies it as RCS flow. 2) Third bullet not E
28 H 2 N required (see NUREG-1021, Appendix E). Good opportunity to ensure applicants S understand this concept.
This question was revised.
E D - CR: 1) Tech ref identified (010-005) not provided. 2) Explain SD-PLC ref.
29 H 2 B S This question was revised.
A - CR: 1) Tech ref TGOP not provided. 2) Explanation for A states procedure step 42 directs the LPSI pump to be tripped locally - not technically correct. Should the procedure say locally? - PEO? 3) No Tech ref provided to validate bus power 30 H 3 N S supply for LPSI pump A. 4) First part of question is not required by the KA. 5) What is the reference for the 5 minutes?
This question is ok as written.
C - CR: 1) Third bullet - what does Entry conditions are being met mean? 2)
Reorder answers from shortest to longest (D - C - B - A) 3) Is CP-10 required in E
31 H 4 N fourth bullet and all answers? 4) Add LO PZR PRESS after Channel D for A and S B; and after Channel C for C and D This question was revised.
32 H 3 N E D - OR (902-009, App 2, Att 2-E): 1) For A part 2 - Core uncovery and superheataed conditions are expected to prevent cueing 2) For C part 2 - Core Page 4 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S uncovery and superheated conditions are NOT expected 3) For D part 2 - the SI flow curves are NOT being met This question was revised.
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 Page 5 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 Page 6 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 7 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 B= F= E=
RO TOTALS: M= H= U= Additional Notes:
N=
B= F= E=
SRO TOTALS: Additional Notes:
M= H= U= Page 7 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 8 Form ES-401-9 N=
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
- 2. Chief Examiner comments are indicated in blue.
41.2 = 43.2 =
41.3 = 43.3 =
41.4 = 43.4 =
41.5 = 43.5 =
41.6 = 43.6 =
41.7 = 43.7 =
41.8 =
41.9 =
41.10 =
41.11 =
41.12 =
41.13 =
41.14 =
B= /
C= /
D= /
- 6. There are questions with attachments provided.
- 8. First 14 questions are 2 x 2 - how many more? Page 8 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only A - CR: Dont really need any of the bullets. Stem could read During a LOCA, E venting the vessel head using the RCS vent valves will (1) and should be stopped 33 F 2 N before reaching (2) to prevent rupturing the Quench Tank rupture disc.
S This question was revised.
D - CR, 2009 NRC Exam: Stem asks for sources of makeup the answers are 34 F 2 B S methods of makeup, not sources. Change answers to Hotwell and CST (to match KA)
This question was revised.
D - CR: First part A and B - change to remains open vs must be manually E
35 F 3 N closed S
This question was revised.
E A - CR, 2008 NRC Exam: 1. Make sure not covered by Op Test scenario 2) verify 36 F 3 B S the title of the trip for A and B Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 9 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 10 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only This question was revised.
C - CR: 1) Change stem to read Following a Reactor trip, which of the following E
37 F 3 N 2) For B - is this RCS Flow Low? 3) what is ref for distracters?
S This question was revised.
A - CR: Explanation for B not credible - break too large for other ECCS to provide E
38 X N core cooling?
S This question was revised.
B - CR: 1) lower case Pressure in all bullets 2) Delete At this time in stem and E
39 C 2 N change should to will S
This question was revised.
E A - CR: Could eliminate 2 x 2 (all answers unique) to raise LOD to 3.
40 F 2 N S This question is ok as written.
C - CR: 1) Delete second and last bullets 2) Stem asks alignment of CS system but E
41 C 2 B answers are more than just CS 3) what is SIS? Should this just be SI?
S This question was revised.
A - CR: 1) Change stem to read In accordance with OP-009-003, Emergency E Feedwater, how will MS-416, EFW Pump AB Turbine Stop Valve, be re-opened? 2) 42 F 3 M For C - delete motor 3) Should have submitted Bank question S
This question was revised.
A - CR: 1) more than one correct answer. 2) For A and C add only before the given answers 3) For B - add Both before the answer. 4) For D change to No U
43 C 3 N verifications required since the BOP wouldnt verify what didnt happen. 5) First S bullet - add event at the end.
This question was revised.
C - CR: 1) missing part a of KA statement 2) Add procedure ref to stem to clearly U
44 X N satisfy part b of KA 3) Question is essentially - What is Reactor Trip Override?
S This question was revised.
D - CR: This question has OE from 2011 exam (ask John). Green NCV was issued .
I need resolution of CR-WF3-2011-01619. What happens to answer if we change 45 N S question to read that EFW Pump AB tripped 15 minutes after trip?
This question was revised.
46 C 3 N E D - CR: 1) Add before bullets - Loss of MFW event has occurred. Following a manual Reactor trip, the following conditions are observed: 2) Change stem to read Page 10 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 11 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S Which of the following describes the proper EFW system response for SG 2? 3)
Re-order answers shortest to longest (B - C - D - A). 4) Delete SG 2 from all answers.
This question was revised.
E C - CR: add buses to end of stem and delete bus from all answers.
47 F 2 B S This question was revised.
A - CR: 1) What is the reference for first half of A? 2) delete all in stem before Train B to prevent cueing right answer. 3) For B and D - is the and then TEST 48 C 3 B S plausible?
This question was revised.
D - CR: 1) what is ref for first part of D? 2) need to explain question/answer 49 F 3 N S This question was revised.
A - CR: 1) SRO-only to determine operability 2) Direct lookup given handout 3)
U SD says engine will continue to crank unti all the air is exhausted during an 50 X N emergency start - is the answer dependent on EDG operating mode?
S This question was revised.
B - CR: 1) Bank question but no ID number 2) Delete all bullets, change stem to read With EDG A paralleled with the grid on the 3A bus and in Local Control mode, E
51 F 2 B the Speed Adjust Control Switch is taken to raise 2) Ref identified doesnt provide S correct answer. SD-EDG is better ref.
This question was revised.
C - CR: 1) stem not adequate. Procedure C.2 states and if DCT Sump 1 is aligned to CW or Storm Drain Hdr so is another bullet required to make C the correct U answer? 2) Change stem to read Which of the following AUTOMATIC actions are 52 F 2 X N expected for DCT 1 Sump Pumps? Then remove DCT 1 Sump Pumps from all S
answers. 3) Re-order from shortest to longest (C - A - D - B)
This question was revised.
A - CR: 1) part 2 of B and D not credible since this would cross-connect the two U
53 F 2 X N systems S
This question is ok as written.
A - CR: 1) Dont need any of the bullets to select correct answer. 2) what is ref for U
54 F 3 X N local indication at IA Dryers?
S This question was revised. Page 11 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 12 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only C - CR: 1) part 2 of A and B not credible - realign vacuum breakers when U
55 F 1 X B pressure delta increases?
S This question was revised.
U D - OR (OP-901-111, Attach 1): direct lookup with handout provided 56 C 1 X B S This question was revised.
B - CR: 1) What is reference for both parts of B? 2) Tech Ref states Revision 020 57 F 2 N S but attached copy is revision 11?
This question was revised.
E A - CR: What is ref for second part of A?
58 C 3 N S This question was revised.
E D - CR: 1) What is ref for first part of D? 2) verify not in Op Test scenario 59 C 3 N S This question was revised.
D - CR: 1) missing part b of KA 2) Ref for second part of D? 3) 1 of 3 CW pumps U
60 N trip causes condenser vacuum to lower to 9 in? Operationally valid?
S This question was revised.
B - CR: 1) Dont need any bullets to get right answer. 2) Reference states Train A U
61 X B 3) Why are A, C, D plausible?
S This question was revised.
A - CR: 1) TS states what must be OPERABLE and operating, not how SR monitors U
62 F 2 X B are verified operable.
S This question was revised.
B - CR: Question doesnt test monitoring operation of SDS including pressure, temp U
63 F 2 X B and reactor power. Putting them in stem doesnt count S
This question was revised.
U B - CR: 1) Part 1 of A and D not plausible 2) Part 2 of A and C not plausible.
64 X N S This question was revised.
B - CR: 1) D not plausible since stem states no fire water 2) Re-order first and E
65 C 3 B second bullets S
This question was revised.
66 F 2 N S C - CR: Page 12 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 13 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only B - CR: 1) Part 2 of A and D not plausible 2) If FSS is Field Support Supv, then U
67 F 1 X N Part 1 of A and C not plausible.
S This question is ok as written.
A - CR: 1) How will the operator verify A? 2) B and C ask for a verification if U
68 X N something else occurs - not operationally valid S
This question was revised.
69 F 3 N S A - CR:
E B - CR: Re-order shortest to longest (B - A - C - D) 70 F 2 B S This question was revised.
U D - CR: Not a Tier 3 question. See ES-401, D.2.a - last sentence of first paragraph.
71 X N S This question was revised.
U C - CR: Part 2 of A and B not credible (3 times the value for lifesaving activities).
72 X N S This question is ok as written.
C - CR: Can reduce to a 50-50 by only knowing 1 of 2. Keep part 1 of A and C, E
73 F 2 B part 2 of D and make a new B S
This question was revised.
U B - CR: Good KA, bad question (not discriminatory).
74 F 1 N S This question was revised.
U D - CR: Good KA, bad question (not discriminatory).
75 F 1 N S This question was revised.
76 77 78 79 80 81 82 83 Page 13 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 14 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 B= F= E=
RO TOTALS: M= H= U= Additional Notes:
N=
B= F= E=
SRO TOTALS: M= H= U= Additional Notes:
N=
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N Page 14 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 15 Form ES-401-9
- 2. Chief Examiner comments are indicated in blue.
41.2 = 43.2 =
41.3 = 43.3 =
41.4 = 43.4 =
41.5 = 43.5 =
41.6 = 43.6 =
41.7 = 43.7 =
41.8 =
41.9 =
41.10 =
41.11 =
41.12 =
41.13 =
41.14 =
B= /
C= /
D= /
- 6. There are questions with attachments provided.
- 8. First 14 questions are 2 x 2 - how many more? Page 15 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only B - CR: 1) need to add a bullet for SG levels 50-70% NR and maintained by MFW or E EFW as part of throttling criteria? 2) would cooldown have been initiated prior to 1 C 2 N reaching throttling criteria?
S This question was revised.
B - CR: 1) given primary to secondary leak, even without ATWS would lead to Functional? 2) Cue given by question of an ATWS - usually given indications them U
2 X N ask question about event 3) overlap with RO question regarding Aux Trip? 4) what S is Aux Trip setpoint reference?
This question is ok as written.
C - CR: 1) what is ref for part 2 of answers?
3 N S This question was revised.
C - CR: 1) no ref for part 1 of C 2) no explanation for part 2 of C 3) Explanation 4 N S for B make sense?
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 16 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 17 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only This question was revised.
E D - CR: How is this question Comprehensive?
5 F 2 N S This question was revised.
U D - OR (TS 3.8.1.1, TRM 3.8.1.1, excerpt from TRM bases) Direct lookup 6 B S This question was revised.
C - OR (EP-001-001) 1) Direct lookup 2) wouldnt need to handout all of procedure U
7 N since all answers are Abnormal Radiation Levels/Rad Effluents S
This question was revised.
U C - OR (TS 3.4.5.2): Direct lookup to reduce to 50-50 8 X N S This question was revised.
D - OR (TS 3.3.3.1, 3.9.4) 1) Direct lookup to reduce to 50-50 2) what is ref for part U
9 N 1 of answer?
S This question is ok as written.
D - OR (902-008 p. 1-23, 902-009 App 2) 1) handouts too excessive for possible U cueing for other questions 2) we have a similar exercise in an Op Test scenario 3) 10 B no bank number provided S
This question was revised.
U A - CR: not discriminatory for SRO level 11 N S This question is ok as written.
U D - OR (TS 2.2.1, 3.3.1, 3.3.2) Direct lookup 12 N S This question was revised.
C - CR: 1) CSAS setpoint? 2) Explanation for C - How would crew diagnose directly into 902-008 since stem states crew is already in 902-002? 3) stem not clear 13 C 2 B S as to when the CS pump tripped This question is ok as written.
B - CR: 1) RO level knowledge 2) weak use of procedures to correct, control, or U
14 X X M mitigate ... 3) original bank question not provided S
This question was revised.
E 15 C 3 N D - CR:
S Page 17 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 18 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only A - CR: 1) RO level knowledge to get to 50-50 choice using widely known info 2)
U 16 F 2 N weak use of procedure - would be better to ask what procedure is used S
This question was revised.
C - CR: 1) consider adding procedure 901-513 to A and B then state only for C and D 2) Ref for part 2 of C not provided 3) is fuel pool temp added to E
17 C 3 B support distracters? Doesnt seem likely it would alarm at same time of low pressure S and level This question was revised.
A - CR: 1) copy of RF procedures? 2) explain Explanation D 3) is SFHM same as E
18 F 3 N RFM and RM?
S This question was revised.
19 F 3 B S B - CR:
U D - CR: Not a Tier 3 type question 20 N S This question was revised.
B - CR: 1) add procedure to stem In accordance with EN-OP-116, Infrequently E Performed Test or Evolution (IPTE), an IPTE briefing 2) what is ref for RV flange 21 F 3 N being 20 feet? 3) Ref for part 2 of answer? (ENs not on disk)
S This question was revised.
A - CR: 1) step 5.2.1 states due to emergent activities is answer wrong? 2) add U
22 B procedure to stem IAW OI-037-000, S
This question was revised.
U B - CR: Not a Tier 3 type question 23 N S This question was revised.
U B - CR: Not a Tier 3 type question 24 B S This question was revised.
A - CR: 1) change C and D to Emergency Plan 2) add procedure to stem E
25 F 3 B IAW OP-901-523, Security Events, the NRC S
This question was revised.
B= F= E=
RO TOTALS: M= H= U= Additional Notes:
N= Page 18 of 19 OBDI 202 - IOLE Process
ES-401 WT-2012-10 19 Form ES-401-9 B= F= E=
SRO TOTALS: M= H= U= Additional Notes:
N=
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
- 2. Chief Examiner comments are indicated in blue.
41.2 = 43.2 =
41.3 = 43.3 =
41.4 = 43.4 =
41.5 = 43.5 =
41.6 = 43.6 =
41.7 = 43.7 =
41.8 =
41.9 =
41.10 =
41.11 =
41.12 =
41.13 =
41.14 =
B= /
C= /
D= /
- 6. There are questions with attachments provided.
- 8. First 14 questions are 2 x 2 - how many more? Page 19 of 19 OBDI 202 - IOLE Process