ML17198F907
| ML17198F907 | |
| Person / Time | |
|---|---|
| Site: | Fermi  |
| Issue date: | 04/03/2017 |
| From: | Bielby M Operations Branch III |
| To: | NRC/RGN-III/DRS/OLB |
| Shared Package | |
| ML17061A771 | List: |
| References | |
| Download: ML17198F907 (9) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 B S 1 of 2 twice 2 H 2 B S 1 of 2 twice 3 H 4 B S 3
4+ H 3 X B U 1 of 2 twice S NRC: 1) SRO ONLY, requires knowledge of TS, TSB and AOP.
- 2) Explain references.
LIC: References supplied were TS and TS bases, however, additional procedural references were supplied that contained same information that ROs are required to know, references such as AOP, ARP and OPs procedures and lesson plans.
3.5 NRC: LOD=2.
5 H X N E LIC: Agree LOD=2.
2 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
- T/F: The answer choices are a collection of unrelated true/false statements.
- Cred. Dist: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- Partial: One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- Backward: The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
A + in the Q# column indicates that question was reviewed as part of the representative sample of 30 questions.
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 3
6# F B S 2012 NRC Exam 2
7 H 3 N S 8+ H 3 N S Handout Air System Mimic 9 H 4 N S H 3 NRC: LOK=F, memory that only rad alarms for warning and 10 N E evacuation. LOD=2.
F 2 LIC: Agree 2013 NRC Exam 11 H 3 B E NRC: Revise last stem sentence to read: Based on these plant conditions what action(s) should be taken?
LIC: Incorporated.
12+ H 3 N S Handout 29.100.01, Sht. 6 13 H 3 M E 1 of 2/twice NRC: 1) Delete note at end of stem that identifies the curve reference. 2) Overpressure =5+3.5+(-70/30)=6.16 not 5.2. 3) Change distractor flows such that they are numbers based on incorrect number of pump(s) for RHR or CS.
LIC: Q15 based on vortex limits.
14 H 2 M S Handout Steam Tables; 29.100.01, Sht. 6 Handout 29.100.01, Sht. 6 15 H 3 B U NRC: Cue: same curves used on question 13 for determining CS and E RHR flows based on NPSH and Vortex Limits. Would be double jeopardy.
LIC: Q15 based on vortex limits.
16+ H 3 N S NRC: LOK=F, Q does not indicate a Handout, appears to be 17 H 3 N E memory. Are ROs expected to have this information memorized? 2) 2 LOD=2.
LIC: Analysis/comprehension required to determine answer, not memory. Expected RO knowledge.
18 H 3 X N U NRC: 1) LOK=F. Memorization of interlocks. 2) C and D not plausible, both N an S running in AUTO and each receive Hi-Hi rad F 2 signal, but only one gets isolated, believable (?). 3) LOD=2.
LIC: Rewrote C and D, LOD=2.
2 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 19 H 2 B S 20 H 3 N S 21+ F 2 M S NRC: 1) LOK=F. Memorization of LLS interlock for the first SRV to 22 H 2 B E open when RPS for reactor pressure scram setpoint of 1093 is F reached.
LIC: Agree.
1 of 2 twice 23+ H 2 N E NRC: 1) LOK=F, memorization of trip and bases.
F LIC: Agree.
24# H 2 B S 25 H 3 B S 26 H 3 B S 27 F 2 B S NRC: LOK=F, LOD=1, answer C. is only choice that has a pressure 28# H 1 X N U above the Minimum Steam Cooling Pressure (MSCP). Also, provide F justification of choices.
LIC: Will replace.
Early Review Replacement Q for 28:
28r H 3 N E NRC: 1) Incorrect reference supplied with question; however, correct reference located on reference disc. 2) Is Table 8, MSCP, an exam handout?
LIC: Per telecom: 1) Reference on References disc. 2) Applicants will not get Table 8, they are expected to know the table (memory).
29+ H 4 M S 1 of 2 twice 30# H 3 M S 1 of 2 twice 2010 NRC Exam 31 F 3 B S 1 of 2 twice NRC: LOD=2. What was miss rate on this question during 32+ F 3 N E validation?
2 LIC: Agree LOD=2.
33 F 2 M S 3 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 1 of 2 twice 34 H 3 N E NRC: 1) LOK=F, very little analysis, more memory knowledge of CS F 2 pump start and FO15A interlock with RWL. 2) LOD=2. What was miss rate on this question during validation?
LIC: Agree 2012 NRC Exam 35 H 2.5 B E NRC: 1) LOD=2. What was miss rate on this question during 2 validation?
LIC: Agree 2012 NRC Exam 36+ H 3 B E NRC: LOK=F. Memory knowledge, either know or dont know F information, no way to determine. (Identified as F on 2012 Exam).
LIC: Agree 1 of 2 twice 37 F 2 B S 38 F 2 B S 39 H 3 M S 1 of 2 twice 40+ H 3 B S NRC: LOD=3.
41 H 4 M E LIC: Agree 3
NRC: 1) As written, the applicant could say A, B, C and D contain 42 F 4 N E affected groups. Reword stem to ask the applicant to identify which of the following includes ONLY groups impacted by the error? 3)
Rewrite choices to only identify only 3 groups. Pick incorrect groups that include misconceptions of being affected by L1. For example: A.
Groups 1,3,5 (correct); B. Groups 1,2,4; C. Groups 2,3,6; D. Groups 4,5,6.
LIC: Agree, rewrote distractors.
43 H 3 N S NRC: Not significantly modified. Stem still has RFP B controller shift 44+ F 3 M E to EMERGENCY BYPASS mode and answer is still same.
B LIC: Agree 45 F 2 M S 46 H 3 M S 1 of 2 twice 47 F 3 N S 4 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N NRC: LOK=F, memory of power supplies.
48+ H 2 B E LIC: Agree F
NRC: LOD=3 49 H 4 M E LIC: Agree 3
NRC: LOD=3 50 H 2.5 B E LIC: Agree 3
51 H 3 B E 2013 NRC Exam F 2 NRC: 1) LOK=F, memory of setpoint; 2) LOD=2.
LIC: Agree F 3 N S 1 of 2 twice 52+
4 NRC: LOD=3 53 H B E 3 LIC: Agree NRC: 1) LOD=2; 2) LOK=F, memory of design basis reference info 54 H 3 X B E for the stabilizing valves. Stem term oscillating and valve name F 2 stabilizing are cues to answer.
LIC: Agree F NRC: LOK=H, using analysis/synthesis to arrive at answer.
55+ 3 M E H LIC: Agree 56 H 3 M S 57 H 2 M S 58 H 3 B S 59# F 3 B S 2013 NRC Exam NRC: LOK=F, using memory, not analysis/synthesis to arrive at 60+ H 2 N E answer.
F LIC: Agree 61 F 2 N S 1 of 2 twice 62 F 2 B S 63 F 2.5 B E 2012 NRC Exam 3 NRC: LOD=3, use whole numbers.
LIC: Agree 64+ H 3 B S 5 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 65 H 2 M S 66 F 2 B S 67 F 3 B E Handout Safety Handbook, Sec. 21 NRC: Delete reference note from question.
LIC: Agree 68+ F 2 B S NRC: LOK=L, no analysis/synthesis to determine correct answer.
69 H 2 B E Only memory.
F LIC: Agree F 3 N E NRC: LOD=2, low discrimination of competency.
70 2 LIC: Agree 71 H 3 B E 2006 NRC Exam F 2 M NRC: 1) LOK=L, simple math calculation of stay time and knowledge of Life Saving Measures WB TEDE limit; LOD=2, low discrimination of competency. 2) Modified Q71 on 2006 NRC exam.
LIC: Agree 72+ F 3 B S 73 H 4 N S 1 of 2 twice 74# H 3 X N U NRC: LOK=L, LOD=1, two implausible distractors, B and D. B implausible because there is no RWL problem indicated so RCIC is F 1 not required. D is implausible because of the major alarms given, which indicate something must have actuated or isolated. Also, stem wording is asking for isolations and/or isolations and choice D is saying no isolations and/or actuations.
LIC: Replaced question (74r).
74r H 3 N S Early Review Replacement Q for 74:
75# F 2 N U NRC: LOD=1, this is something done on each scenario. Not appropriate to have on both written and operating test, double 1
jeopardy.
LIC: Replaced (75r).
75r F 2 N S Early Review Replacement Q for 75:
R RO Exam: LOK: H(50-60%)=40 /75(53.3%); F=35/75(46.7%) N(8)=24/75(32%), B(56 max)=35/75=42.6%, M=16/75(21.3%): U=1/75(1.3%); E=29/75(38.7%); LOD=2.6 6 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 1 of 2 twice 76 H 3 X N S 2012 NRC Exam 77 H 3.5 X B S NRC: LOD=3, only use whole numbers.
3 LIC: Agree 78 H 3 X B S 2012 NRC Exam Handout Steam Tables 79 H 3 X M E 1 of 2 twice NRC: Q pedigree identifies LOK=L.
LIC: LOK=H, will change pedigree 1 of 2 twice 80 H 3 X B S 2012 NRC Exam 81 H 3 X B E NRC: This is Modified SRO Q5 from 2012 NRC Exam. Stem M changed temp from 130 to 170 deg F, replaced distractor C and rewrote distractor D.
LIC: Agree 82 H 3 X B S NRC: Modify last stem sentence to read, The CRS will direct which 83 H 3 X B E of the following procedural actions?.
LIC: Incorporated.
NRC: 1) Add copy of TR 3.3.6.4 AND TR 3.6.2.1 to Handout List 84 H 4 N E LIC: Agree NRC: 1) Change last stem sentence part (2) to read what procedural 85 H 2 X M E actions will the CRS direct? 2) Not significantly modified, still B.
B LIC: Agree NRC: 1) LOD=3, use whole numbers; 2) Pedigree incorrectly lists Q 86 H 3.5 X B E is B used on 2015 NRC exam.
3 LIC: Agree NRC: 1) KA for question (209001A2.04, DC failure) does not match 87# H 2.5 X X B U written exam outline KA (209001A2.08, valve opening). 2) Do SROs 3 get reference? 3) ) LOD=3, use whole numbers LIC: Rewrote (Q87r)
NRC: 1) LOD=3, use whole numbers; 87r H 2.5 X X B E LIC: Agree 3
1 / 2 twice; TS 3.3.1.1 88# H 3 X N E NRC: 1) Do SROs get reference? 2) LOD=2 2 LIC: 1) Get TS 3.3.1.1; 2) Agree 7 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 1 / 2 twice 89 H 3 X N S 90 H 3 X B S 91 H 2.5 X B S 3
NRC: 1) LOD=2, completing bases statement; 2) LOK=L, 92 H 3 X B E memorization of bases.
F 2 LIC: Agree 1/2 twice 93# H 4 X N E NRC: 1) LOD=3, simple math, memorization of EOP steps. 2) Modify 3 stem part 2) to reflect actions to be directed by the CRS.
LIC: Agree 94 F 2 X B S NRC: 1) LOD=1, the CRS is only licensed person in control room, 95 F 2 X B U also as SRO he also has RO license. Q is too simple, not good 1 discrimination. Would not expect applicant to not have this knowledge.
LIC: Replace Q95 with Q95r 95r F 2 X B S 96 H 2 X B E Handout MMR, App. H NRC: 1) Need to get rid of like terms in stem and reference. Modify stem, replace terms high sustained and gust with wind speeds 43 to 48 mph and wind bursts of 60 mph.
LIC: Agree 97 H 4 X B E Handout MMA26, Enc. C 3 NRC: 1) LOD=3; 2) Delete (See attached MMA26, Enc. C) from stem. Make reference part of handouts to applicants. 3) Delete first bullet in stem. Applicant should be able to identify the Risk Category as High Risk per the 1.1 list in Enclosure C, Section 1.0.
LIC: Agree 1 / 2 twice 98 F 4 X N E NRC: 1) LOD=2, common NGET knowledge for onsite rad workers; 2 LIC: Agree NRC: 1) Insert by the CRS between directed and to into the 99 H 3 X B E last stem sentence.
LIC: Agree 8 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 100 H 2 X N E Handout EAL Chart NRC: 1) Delete note about attached EAL chart in last stem sentence.
Chart will be part of the applicant handouts.
LIC: Agree For SRO portion of Exam: LOK: H(50%min)=21/25(85%); F=4/25(16%); N(2)=7/25(28%), B(19 max)=16/25(64%), M=2/25(8%): U=0/25(0%); E=15/25=60%; LOD=68/25=2.7 Were any Bank questions from previous NRC exams? Identified during validation.
Remove handout reference from questions. Make sure question numbers arent included on the Handout list to applicants during exam.
Submit original questions for Modified questions to verify they were modified IAW ES 1021. Questions submitted.
9 of 9