ML17198F907: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
 
Line 16: Line 16:


=Text=
=Text=
{{#Wiki_filter:ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10   2017 Fermi Exam 1 of 9  Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
{{#Wiki_filter:ES-401                                                             Written Examination Review Worksheet                                                       Form ES-401-9, Rev. 10 2017 Fermi Exam
: 4. Job Content Flaws
: 1.     2.           3. Psychometric Flaws               4. Job Content Flaws           5. Other     6. 7.                                     8.
: 5. Other 6. B/M/N  7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 1 H 3           B S 1 of 2 twice 2 H 2           B S 3 H 4 3            B S 1 of 2 twice 4+ H 3           X B U S 1 of 2 twice NRC: 1) SRO ONLY, requires knowledge of TS, TSB and AOP.
Q#    LOK      LOD (F/H)  (1-5)  Stem Cues         T/F   Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S                                                  Explanation Focus                    Dist.          Link            units ward K/A Only 1       H       3                                                                                             B     S   1 of 2 twice 2       H       2                                                                                             B     S 1 of 2 twice 3       H       4                                                                                             B     S 3
4+       H       3                                                                                     X     B     U   1 of 2 twice S    NRC: 1) SRO ONLY, requires knowledge of TS, TSB and AOP.
: 2) Explain references.
: 2) Explain references.
LIC: References supplied were TS and TS bases, however, additional procedural references were supplied that contained same information that ROs are required to know, references such as AOP, ARP and OPs procedures and lesson plans.
LIC: References supplied were TS and TS bases, however, additional procedural references were supplied that contained same information that ROs are required to know, references such as AOP, ARP and OPs procedures and lesson plans.
5 H 3.5 2    X        N E NRC: LOD=2. LIC: Agree LOD=2.
3.5                                                                                                       NRC: LOD=2.
Instructions
5      H                                      X                                                              N    E    LIC: Agree LOD=2.
2 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
: 1.         Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
: 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
: 2.         Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
: 3. Check the appropriate box if a psychometric flaw is identified:
: 3.         Check the appropriate box if a psychometric flaw is identified:
  "Stem Focus": The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
* Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  "Cues": The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc). "T/F": The answer choices are a collection of unrelated true/false statements.
* Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  "Cred. Dist": The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
* T/F: The answer choices are a collection of unrelated true/false statements.
  "Partial": One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
* Cred. Dist: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
: 4. Check the appropriate box if a job content error is identified:
* Partial: One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
  "Job Link": The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).     "Minutia": The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory). "#/Units": The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). "Backward": The question requires reverse logic or application compared to the job requirements.
: 4.         Check the appropriate box if a job content error is identified:
: 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO
* Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
-only (K/A and license level mismatches are unacceptable).
* Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
: 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
                *          #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
: 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
* Backward: The question requires reverse logic or application compared to the job requirements.
: 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
: 5.         Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
A "+" in the "Q#" column indicates that question was reviewed as part of the representative sample of 30 questions.
: 6.         Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
# 7 RO and 3 SRO Only questions submitted for NRC pre
: 7.         Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
-review ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10   2017 Fermi Exam 2 of 9    Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
: 8.         At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
: 4. Job Content Flaws
A + in the Q# column indicates that question was reviewed as part of the representative sample of 30 questions.
: 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 6# F 3 2            B S 2012 NRC Exam 7 H 3           N S 8+ H 3           N S Handout Air System Mimic 9 H 4           N S 10 H F 3 2            N E NRC: LOK=F, memory that only rad alarms for warning and evacuation. LOD=2.
# 7 RO and 3 SRO Only questions submitted for NRC pre-review 1 of 9
LIC: Agree 11 H 3           B E 201 3 NRC Exam NRC: Revise last stem sentence to read: "Based on these plant conditions what action(s) should be taken?"
 
ES-401                                               Written Examination Review Worksheet                                     Form ES-401-9, Rev. 10 2017 Fermi Exam
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other 6. 7.                                   8.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S                                Explanation Focus              Dist.        Link          units ward K/A Only N 3
6#   F                                                                             B S 2012 NRC Exam 2
7   H     3                                                                       N S 8+   H     3                                                                       N S Handout Air System Mimic 9   H     4                                                                       N S H     3                                                                             NRC: LOK=F, memory that only rad alarms for warning and 10                                                                                  N  E  evacuation. LOD=2.
F    2                                                                              LIC: Agree 2013 NRC Exam 11   H     3                                                                       B E NRC: Revise last stem sentence to read: Based on these plant conditions what action(s) should be taken?
LIC: Incorporated.
LIC: Incorporated.
12+ H 3           N S 13 H 3            M E Handout 29.100.01, Sht. 6 1 of 2/twice NRC: 1) Delete note at end of stem that identifies the curve reference. 2) Overpressure =5+3.5+(-70/30)=6.16 not 5.2. 3) Change distractor flows such that they are numbers based on incorrect number of pump(s) for RHR or CS.
12+   H     3                                                                       N S Handout 29.100.01, Sht. 6 13  H    3                                                                        M  E 1 of 2/twice NRC: 1) Delete note at end of stem that identifies the curve reference. 2) Overpressure =5+3.5+(-70/30)=6.16 not 5.2. 3) Change distractor flows such that they are numbers based on incorrect number of pump(s) for RHR or CS.
LIC: Q15 based on vortex limits.
14  H    2                                                                        M  S  Handout Steam Tables; 29.100.01, Sht. 6 Handout 29.100.01, Sht. 6 15  H    3                                                                        B  U NRC: Cue: same curves used on question 13 for determining CS and E  RHR flows based on NPSH and Vortex Limits. Would be double jeopardy.
LIC: Q15 based on vortex limits.
LIC: Q15 based on vortex limits.
14 H 2            M S Handout Steam Tables; 29.100.01, Sht. 6 15 H 3            B U E Handout 29.100.01, Sht. 6 NRC: Cue: same curves used on question 13 for determining CS and RHR flows based on NPSH and Vortex Limits. Would be double jeopardy. LIC: Q15 based on vortex limits.
16+   H     3                                                                       N S NRC: LOK=F, Q does not indicate a Handout, appears to be 17  H    3                                                                        N  E memory. Are ROs expected to have this information memorized? 2) 2                                                                              LOD=2.
16+ H 3           N S 17 H 3 2            N E NRC: LOK=F , Q does not indicate a Handout, appears to be memory. Are ROs expected to have this information memorized?
LIC: Analysis/comprehension required to determine answer, not memory. Expected RO knowledge.
: 2) LOD=2. LIC: Analysis/comprehension required to determine answer, not memory. Expected RO knowledge.
18   H     3                       X                                               N U NRC: 1) LOK=F. Memorization of interlocks. 2) C and D not plausible, both N an S running in AUTO and each receive Hi-Hi rad F    2                                                                              signal, but only one gets isolated, believable (?). 3) LOD=2.
18 H F 3 2    X       N U NRC: 1) LOK=F. Memorization of interlocks. 2) C and D not plausible, both N an S running in AUTO and each receive Hi
LIC: Rewrote C and D, LOD=2.
-Hi rad signal, but only one gets isolated, believable
2 of 9
(?). 3) LOD=2. LIC: Rewrote C and D, LOD=2.
 
ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10   2017 Fermi Exam 3 of 9  Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
ES-401                                               Written Examination Review Worksheet                                     Form ES-401-9, Rev. 10 2017 Fermi Exam
: 4. Job Content Flaws
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other 6. 7.                                   8.
: 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 19 H 2           B S 20 H 3           N S 21+ F 2           M S 22 H F 2            B E NRC: 1) LOK=F. Memorization of LLS interlock for the first SRV to open when RPS for reactor pressure scram setpoint of 1093 is reached. LIC: Agree. 23+ H F 2           N E 1 of 2 twice NRC: 1) LOK=F, memorization of trip and bases.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S                                  Explanation Focus              Dist.        Link          units ward K/A Only N 19   H     2                                                                       B S 20   H     3                                                                       N S 21+   F     2                                                                       M S NRC: 1) LOK=F. Memorization of LLS interlock for the first SRV to 22  H    2                                                                        B  E open when RPS for reactor pressure scram setpoint of 1093 is F                                                                                    reached.
LIC: Agree. 24# H 2           B S 25 H 3           B S 26 H 3           B S 27 F 2           B S 28# H F 1  X          N U NRC: LOK=F , LOD=1, answer C. is only choice that has a pressure above the Minimum Steam Cooling Pressure (MSCP). Also, provide justification of choices.
LIC: Agree.
1 of 2 twice 23+   H     2                                                                       N E NRC: 1) LOK=F, memorization of trip and bases.
F                                                                                    LIC: Agree.
24#   H     2                                                                       B S 25   H     3                                                                       B S 26   H     3                                                                       B S 27   F     2                                                                       B S NRC: LOK=F, LOD=1, answer C. is only choice that has a pressure 28#  H    1            X                                                          N  U above the Minimum Steam Cooling Pressure (MSCP). Also, provide F                                                                                    justification of choices.
LIC: Will replace.
LIC: Will replace.
28 r H 3            N E Early Review Replacement Q for 28:
Early Review Replacement Q for 28:
NRC: 1) Incorrect reference supplied with question; however, correct reference located on reference disc. 2) Is Table 8, MSCP, an exam handout? LIC: Per telecom: 1) Reference on References disc. 2) Applicants will not get Table 8, they are expected to know the table (memory).
28r  H    3                                                                        N  E NRC: 1) Incorrect reference supplied with question; however, correct reference located on reference disc. 2) Is Table 8, MSCP, an exam handout?
29+ H 4           M S 1 of 2 twice 30# H 3           M S 1 of 2 twice 31 F 3           B S 201 0 NRC Exam 1 of 2 twice 32+ F 3 2            N E NRC: LOD=2. What was miss rate on this question during validation?
LIC: Per telecom: 1) Reference on References disc. 2) Applicants will not get Table 8, they are expected to know the table (memory).
LIC: Agree LOD=2. 33 F 2           M S ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10   2017 Fermi Exam 4 of 9  Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
29+   H     4                                                                       M S 1 of 2 twice 30#   H     3                                                                       M S   1 of 2 twice 2010 NRC Exam 31   F     3                                                                       B S 1 of 2 twice NRC: LOD=2. What was miss rate on this question during 32+  F    3                                                                        N  E validation?
: 4. Job Content Flaws
2                                                                              LIC: Agree LOD=2.
: 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 34 H F 3 2            N E 1 of 2 twice NRC: 1) LOK=F, very little analysis, more memory knowledge of CS pump start and FO15A interlock with RWL. 2) LOD=2. What was miss rate on this question during validation?
33   F     2                                                                       M S 3 of 9
LIC: Agree 35 H 2.5 2            B E 201 2 NRC Exam NRC: 1) LOD=2. What was miss rate on this question during validation?
 
LIC: Agree 36+ H F 3             B E 201 2 NRC Exam NRC: LOK=F. Memory knowledge, either know or don't know information, no way to determine. (Identified as F on 2012 Exam).
ES-401                                               Written Examination Review Worksheet                                     Form ES-401-9, Rev. 10 2017 Fermi Exam
LIC: Agree 37 F 2           B S 1 of 2 twice 38 F 2           B S 39 H 3           M S 1 of 2 twice 40+ H 3           B S 41 H 4 3            M E NRC: LOD=3. LIC: Agree 42 F 4            N E NRC: 1) As written, the applicant could say A, B, C and D contain affected groups. Reword stem to ask the applicant to identify which of the following includes ONLY groups impacted by the error? 3) Rewrite choices to only identify only 3 groups. Pick incorrect groups that include misconceptions of being affected by L1. For example: A. Groups 1,3,5 (correct); B. Groups 1,2,4; C. Groups 2,3,6; D. Groups 4,5,6. LIC: Agree, rewrote distractors.
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other 6. 7.                                   8.
43 H 3           N S 44+ F 3            M B E NRC: Not significantly modified. Stem still has RFP B controller shift to EMERGENCY BYPASS mode and answer is still same.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S                                Explanation Focus              Dist.        Link          units ward K/A Only N 1 of 2 twice 34  H    3                                                                        N  E NRC: 1) LOK=F, very little analysis, more memory knowledge of CS F    2                                                                              pump start and FO15A interlock with RWL. 2) LOD=2. What was miss rate on this question during validation?
LIC: Agree 45 F 2           M S 46 H 3           M S 1 of 2 twice 47 F 3           N S ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10   2017 Fermi Exam 5 of 9  Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
LIC: Agree 2012 NRC Exam 35   H   2.5                                                                       B E NRC: 1) LOD=2. What was miss rate on this question during 2                                                                              validation?
: 4. Job Content Flaws
LIC: Agree 2012 NRC Exam 36+   H     3                                                                       B E NRC: LOK=F. Memory knowledge, either know or dont know F                                                                                    information, no way to determine. (Identified as F on 2012 Exam).
: 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 48+ H F 2            B E NRC: LOK=F, memory of power supplies.
LIC: Agree 1 of 2 twice 37   F     2                                                                       B S 38   F     2                                                                       B S 39   H     3                                                                       M S 1 of 2 twice 40+   H     3                                                                       B S NRC: LOD=3.
LIC: Agree 49 H 4 3            M E NRC: LOD=3  LIC: Agree 50 H 2.5 3            B E NRC: LOD=3  LIC: Agree 51 H F 3 2            B E 2013 NRC Exam NRC: 1) LOK=F, memory of setpoint; 2) LOD=2.
41  H    4                                                                       M E LIC: Agree 3
LIC: Agree 52+ F 3           N S 1 of 2 twice 53 H 4 3            B E NRC: LOD=3  LIC: Agree 54 H F 3 2  X          B E NRC: 1) LOD=2; 2) LOK=
NRC: 1) As written, the applicant could say A, B, C and D contain 42  F    4                                                                        N  E affected groups. Reword stem to ask the applicant to identify which of the following includes ONLY groups impacted by the error? 3)
F, memory of design basis reference info for the stabilizing valves. Stem term "oscillating" and valve name "stabilizing" are cues to answer.
Rewrite choices to only identify only 3 groups. Pick incorrect groups that include misconceptions of being affected by L1. For example: A.
LIC: Agree 55+ F H 3            M E NRC: LOK=H, using analysis/synthesis to arrive at answer.
Groups 1,3,5 (correct); B. Groups 1,2,4; C. Groups 2,3,6; D. Groups 4,5,6.
LIC: Agree 56 H 3           M S 57 H 2           M S 58 H 3           B S 59# F 3           B S 2013 NRC Exam 60+ H F 2            N E NRC: LOK=F, using memory, not analysis/synthesis to arrive at answer. LIC: Agree 61 F 2           N S 1 of 2 twice 62 F 2           B S 6 3 F 2.5 3            B E 201 2 NRC Exam NRC: LOD=3, use whole numbers.
LIC: Agree, rewrote distractors.
LIC: Agree 64+ H 3           B S ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10   2017 Fermi Exam 6 of 9  Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
43   H     3                                                                       N S NRC: Not significantly modified. Stem still has RFP B controller shift 44+  F    3                                                                        M  E to EMERGENCY BYPASS mode and answer is still same.
: 4. Job Content Flaws
B    LIC: Agree 45   F     2                                                                       M S 46   H     3                                                                       M S 1 of 2 twice 47   F     3                                                                       N S 4 of 9
: 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 65 H 2           M S 66 F 2           B S 67 F 3           B E Handout Safety Handbook, Sec. 21 NRC: Delete reference note from question.
 
LIC: Agree 68+ F 2           B S 69 H F 2            B E NRC: LOK=L, no analysis/synthesis to determine correct answer. Only memory.
ES-401                                               Written Examination Review Worksheet                                     Form ES-401-9, Rev. 10 2017 Fermi Exam
LIC: Agree 70 F 3 2            N E NRC: LOD=2, low discrimination of competency.
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other 6. 7.                                   8.
LIC: Agree 71 H F 3 2            B M E 20 06 NRC Exam NRC: 1) LOK=L, simple math calculation of stay time and knowledge of Life Saving Measures WB TEDE limit; LOD=2, low discrimination of competency.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S                                  Explanation Focus              Dist.        Link          units ward K/A Only N NRC: LOK=F, memory of power supplies.
: 2) Modified Q71 on 2006 NRC exam.
48+  H    2                                                                        B  E LIC: Agree F
LIC: Agree 72+ F 3           B S 73 H 4           N S 1 of 2 twice 74# H F 3 1    X       N U NRC: LOK=L, LOD=1, two implausible distractors, B and D. B implausible because there is no RWL problem indicated so RCIC is not required. D is implausible because of the major alarms given, which indicate something must have actuated or isolated. Also, stem wording is asking for isolations and/or isolations and choice D is saying "no isolations and/or actuations".
NRC: LOD=3 49   H     4                                                                       M E LIC: Agree 3
LIC: Replaced question (74r). 74r H 3           N S Early Review Replacement Q for 74:
NRC: LOD=3 50   H   2.5                                                                       B E LIC: Agree 3
75# F 2 1            N U NRC: LOD=1, this is something done on each scenario. Not appropriate to have on both written and operating test, double jeopardy. LIC: Replaced (75r)
51   H     3                                                                       B E 2013 NRC Exam F    2                                                                              NRC: 1) LOK=F, memory of setpoint; 2) LOD=2.
. 75r F 2           N S Early Review Replacement Q for 75:
LIC: Agree F     3                                                                       N S 1 of 2 twice 52+
R  RO Exam: LOK: H(50-60%)=40 /75(5 3.3%); F=3 5/75(46.74/75(32%), B(56 max)=
4                                                                             NRC: LOD=3 53   H                                                                              B  E 3                                                                              LIC: Agree NRC: 1) LOD=2; 2) LOK=F, memory of design basis reference info 54  H    3            X                                                          B  E for the stabilizing valves. Stem term oscillating and valve name F    2                                                                              stabilizing are cues to answer.
35/75=42.6%, M=16/75(21.3%):   U=1/75(1.3%); E=2 9/75(38.7%); LOD=2.
LIC: Agree F                                                                                   NRC: LOK=H, using analysis/synthesis to arrive at answer.
6 ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10   2017 Fermi Exam 7 of 9  Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
55+        3                                                                        M  E H                                                                                    LIC: Agree 56   H     3                                                                       M S 57   H     2                                                                       M S 58   H     3                                                                       B S 59#   F     3                                                                       B S 2013 NRC Exam NRC: LOK=F, using memory, not analysis/synthesis to arrive at 60+  H    2                                                                        N  E answer.
: 4. Job Content Flaws
F                                                                                    LIC: Agree 61   F     2                                                                       N S 1 of 2 twice 62   F     2                                                                       B S 63  F   2.5                                                                       B E 2012 NRC Exam 3                                                                              NRC: LOD=3, use whole numbers.
: 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 76 H 3           X N S 1 of 2 twice 77 H 3.5 3            X B S 201 2 NRC Exam NRC: LOD=3, only use whole numbers.
LIC: Agree 64+   H     3                                                                       B S 5 of 9
LIC: Agree 78 H 3           X B S 201 2 NRC Exam 79 H 3           X M E Handout Steam Tables 1 of 2 twice NRC: Q pedigree identifies LOK=L.
 
LIC: LOK=H, will change pedigree 80 H 3           X B S 1 of 2 twice 81 H 3           X B M E 201 2 NRC Exam NRC: This is Modified SRO Q5 from 2012 NRC Exam. Stem changed temp from 130 to 170 deg F, replaced distractor C and rewrote distractor D
ES-401                                               Written Examination Review Worksheet                                           Form ES-401-9, Rev. 10 2017 Fermi Exam
. LIC: Agree 82 H 3           X B S 83 H 3          X B E NRC: Modify last stem sentence to read, "The CRS will direct which of the following procedural actions?".
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other   6. 7.                                      8.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S                                        Explanation Focus              Dist.        Link          units ward K/A Only N 65   H     2                                                                         M     S 66   F     2                                                                         B     S 67   F     3                                                                         B     E   Handout Safety Handbook, Sec. 21 NRC: Delete reference note from question.
LIC: Agree 68+   F     2                                                                         B     S NRC: LOK=L, no analysis/synthesis to determine correct answer.
69  H    2                                                                          B    E Only memory.
F                                                                                          LIC: Agree F     3                                                                         N     E   NRC: LOD=2, low discrimination of competency.
70 2                                                                                    LIC: Agree 71   H     3                                                                         B     E   2006 NRC Exam F    2                                                                          M          NRC: 1) LOK=L, simple math calculation of stay time and knowledge of Life Saving Measures WB TEDE limit; LOD=2, low discrimination of competency. 2) Modified Q71 on 2006 NRC exam.
LIC: Agree 72+   F     3                                                                         B     S 73   H     4                                                                         N     S   1 of 2 twice 74#   H     3                       X                                                 N     U   NRC: LOK=L, LOD=1, two implausible distractors, B and D. B implausible because there is no RWL problem indicated so RCIC is F    1 not required. D is implausible because of the major alarms given, which indicate something must have actuated or isolated. Also, stem wording is asking for isolations and/or isolations and choice D is saying no isolations and/or actuations.
LIC: Replaced question (74r).
74r   H     3                                                                         N     S   Early Review Replacement Q for 74:
75#   F     2                                                                         N     U   NRC: LOD=1, this is something done on each scenario. Not appropriate to have on both written and operating test, double 1
jeopardy.
LIC: Replaced (75r).
75r   F     2                                                                         N     S   Early Review Replacement Q for 75:
R  RO Exam: LOK: H(50-60%)=40 /75(53.3%); F=35/75(46.7%) N(8)=24/75(32%), B(56 max)=35/75=42.6%, M=16/75(21.3%): U=1/75(1.3%); E=29/75(38.7%); LOD=2.6 6 of 9
 
ES-401                                               Written Examination Review Worksheet                                     Form ES-401-9, Rev. 10 2017 Fermi Exam
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other 6. 7.                                   8.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S                                Explanation Focus              Dist.        Link           units ward K/A Only N 1 of 2 twice 76  H     3                                                                     X N  S 2012 NRC Exam 77  H    3.5                                                                    X  B  S NRC: LOD=3, only use whole numbers.
3                                                                              LIC: Agree 78   H     3                                                                     X B S 2012 NRC Exam Handout Steam Tables 79   H     3                                                                     X M E 1 of 2 twice NRC: Q pedigree identifies LOK=L.
LIC: LOK=H, will change pedigree 1 of 2 twice 80   H     3                                                                     X B S 2012 NRC Exam 81   H     3                                                                     X B E NRC: This is Modified SRO Q5 from 2012 NRC Exam. Stem M    changed temp from 130 to 170 deg F, replaced distractor C and rewrote distractor D.
LIC: Agree 82   H     3                                                                     X B S NRC: Modify last stem sentence to read, The CRS will direct which 83  H    3                                                                    X  B  E of the following procedural actions?.
LIC: Incorporated.
LIC: Incorporated.
84 H 4            N E NRC: 1) Add copy of TR 3.3.6.4 AND TR 3.6.2.1 to Handout List LIC: Agree 85 H 2          X M B E NRC: 1) Change last stem sentence part (2) to read "what procedural actions will the CRS direct?"
NRC: 1) Add copy of TR 3.3.6.4 AND TR 3.6.2.1 to Handout List 84  H    4                                                                        N  E LIC: Agree NRC: 1) Change last stem sentence part (2) to read what procedural 85  H    2                                                                    X  M  E actions will the CRS direct? 2) Not significantly modified, still B.
: 2) Not significantly modified, still B.
B    LIC: Agree NRC: 1) LOD=3, use whole numbers; 2) Pedigree incorrectly lists Q 86  H    3.5                                                                    X  B  E is B used on 2015 NRC exam.
LIC: Agree 86 H 3.5 3          X B E NRC: 1) LOD=3, use whole numbers; 2) Pedigree incorrectly lists Q is B used on 2015 NRC exam.
3                                                                              LIC: Agree NRC: 1) KA for question (209001A2.04, DC failure) does not match 87#  H    2.5                                                              X    X  B  U written exam outline KA (209001A2.08, valve opening). 2) Do SROs 3                                                                              get reference? 3) ) LOD=3, use whole numbers LIC: Rewrote (Q87r)
LIC: Agree 87# H 2.5 3          X X B U NRC: 1) KA for question (209001A2.04, DC failure) does not match written exam outline KA (209001A2.08, valve opening).
NRC: 1) LOD=3, use whole numbers; 87r  H    2.5                                                              X    X  B  E LIC: Agree 3
: 2) Do SROs get reference?
1 / 2 twice; TS 3.3.1.1 88#  H    3                                                                    X  N  E NRC: 1) Do SROs get reference? 2) LOD=2 2                                                                              LIC: 1) Get TS 3.3.1.1; 2) Agree 7 of 9
: 3) ) LOD=3, use whole numbers LIC: Rewrote (Q87r) 87r H 2.5 3          X X B E NRC: 1) LOD=3, use whole numbers; LIC: Agree 88# H 3 2          X N E 1 / 2 twice; TS 3.3.1.1 NRC: 1) Do SROs get reference?
 
: 2) LOD=2 LIC: 1) Get TS 3.3.1.1; 2) Agree
ES-401                                              Written Examination Review Worksheet                                      Form ES-401-9, Rev. 10 2017 Fermi Exam
: 1. 2.      3. Psychometric Flaws          4. Job Content Flaws    5. Other 6. 7.                                    8.
Q#  LOK  LOD (F/H) (1-5) Stem Cues T/F      Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S                                  Explanation Focus              Dist.        Link          units ward K/A Only N 1 / 2 twice 89  H    3                                                                    X  N  S 90  H    3                                                                    X  B  S 91  H    2.5                                                                    X  B  S 3
NRC: 1) LOD=2, completing bases statement; 2) LOK=L, 92  H    3                                                                    X  B  E memorization of bases.
F    2                                                                              LIC: Agree 1/2 twice 93#  H    4                                                                    X  N  E NRC: 1) LOD=3, simple math, memorization of EOP steps. 2) Modify 3                                                                              stem part 2) to reflect actions to be directed by the CRS.
LIC: Agree 94  F    2                                                                    X  B  S NRC: 1) LOD=1, the CRS is only licensed person in control room, 95  F    2                                                                    X  B  U also as SRO he also has RO license. Q is too simple, not good 1                                                                              discrimination. Would not expect applicant to not have this knowledge.
LIC: Replace Q95 with Q95r 95r  F    2                                                                    X  B  S 96  H    2                                                                    X  B  E  Handout MMR, App. H NRC: 1) Need to get rid of like terms in stem and reference. Modify stem, replace terms high sustained and gust with wind speeds 43 to 48 mph and wind bursts of 60 mph.
LIC: Agree 97  H    4                                                                    X  B  E  Handout MMA26, Enc. C 3                                                                              NRC: 1) LOD=3; 2) Delete (See attached MMA26, Enc. C) from stem. Make reference part of handouts to applicants. 3) Delete first bullet in stem. Applicant should be able to identify the Risk Category as High Risk per the 1.1 list in Enclosure C, Section 1.0.
LIC: Agree 1 / 2 twice 98  F    4                                                                    X  N  E NRC: 1) LOD=2, common NGET knowledge for onsite rad workers; 2                                                                             LIC: Agree NRC: 1) Insert by the CRS between directed and to into the 99  H    3                                                                    X  B  E last stem sentence.
LIC: Agree 8 of 9


ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10   2017 Fermi Exam 8 of 9  Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
ES-401                                                     Written Examination Review Worksheet                                                 Form ES-401-9, Rev. 10 2017 Fermi Exam
: 4. Job Content Flaws
: 1. 2.         3. Psychometric Flaws             4. Job Content Flaws       5. Other   6.     7.                                       8.
: 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 89 H 3          X N S 1 / 2 twice 90 H 3          X B S  91 H 2.5 3          X B S  92 H F 3 2          X B E NRC: 1) LOD=2, completing bases statement; 2) LOK=L, memorization of bases.
Q#  LOK    LOD (F/H)  (1-5)  Stem Cues T/F       Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S                                               Explanation Focus                 Dist.         Link           units ward K/A Only N 100   H     2                                                                             X   N       E   Handout EAL Chart NRC: 1) Delete note about attached EAL chart in last stem sentence.
LIC: Ag r ee 93# H 4 3          X N E 1/2 twice NRC: 1) LOD=3, simple math, memorization of EOP steps.
Chart will be part of the applicant handouts.
: 2) Modify stem part 2) to reflect actions to be directed by the CRS.
LIC: Agree For SRO portion of Exam: LOK: H(50%min)=21/25(85%); F=4/25(16%); N(2)=7/25(28%), B(19 max)=16/25(64%), M=2/25(8%): U=0/25(0%); E=15/25=60%; LOD=68/25=2.7 Were any Bank questions from previous NRC exams? Identified during validation.
LIC: Agree 94 F 2          X B S  95 F 2 1          X B U NRC: 1) LOD=1, the CRS is only licensed person in control room, also as SRO he also has RO license. Q is too simple, not good discrimination. Would not expect applicant to not have this knowledge.
Remove handout reference from questions. Make sure question numbers arent included on the Handout list to applicants during exam.
LIC: Replace Q95 with Q95r 95r F 2          X B S  96 H 2          X B E Handout MMR, App. H NRC: 1) Need to get rid of like terms in stem and reference. Modify stem , replace terms "-high sustained-" and "-gust-" with "-wind speeds 43 to 48 mph-" and "-wind bursts of 60 mph-". LIC: Agree 97 H 4 3          X B E Handout MMA26, Enc. C NRC: 1) LOD=3; 2) Delete "(See attached MMA26, Enc. C)" from stem. Make reference part of handouts to applicants. 3) Delete first bullet in stem. Applicant should be able to identify the Risk Category as High Risk per the 1.1 list in Enclosure C, Section 1.0.
Submit original questions for Modified questions to verify they were modified IAW ES 1021. Questions submitted.
LIC: Agree 98 F 4 2          X N E 1 / 2 twice NRC: 1) LOD=2, common NGET knowledge for onsite rad workers; LIC: Agree 99 H 3          X B E NRC: 1) Insert "-by the CRS-" between directed and to into the last stem sentence.
9 of 9}}
LIC: Agree ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10  2017 Fermi Exam 9 of 9  Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
: 4. Job Content Flaws
: 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 100 H 2           X N E Handout EAL Chart NRC: 1) Delete note about attached EAL chart in last stem sentence. Chart will be part of the applicant handouts.
LIC: Agree   For SRO portion of Exam: LOK: H(50%min)=
21/25(85%); F=4/25(167/25(28%), B(19 max)=1 6/25(64%), M=2/25(8%):
U=0/25(0%); E=15/25=60%; LOD=6 8/25=2.7   Were any Bank questions from previous NRC exams? Identified during validation.
Remove handout reference from questions. Make sure question numbers aren't included on the Handout list to applicants during exam.
Submit original questions for Modified questions to verify they were modified IAW ES 1021. Questions submitted.}}

Latest revision as of 01:57, 30 October 2019

2017 Fer Ile Administrative Files ES-401-9
ML17198F907
Person / Time
Site: Fermi DTE Energy icon.png
Issue date: 04/03/2017
From: Bielby M
Operations Branch III
To:
NRC/RGN-III/DRS/OLB
Shared Package
ML17061A771 List:
References
Download: ML17198F907 (9)
v • d • e


Text

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 B S 1 of 2 twice 2 H 2 B S 1 of 2 twice 3 H 4 B S 3

4+ H 3 X B U 1 of 2 twice S NRC: 1) SRO ONLY, requires knowledge of TS, TSB and AOP.

2) Explain references.

LIC: References supplied were TS and TS bases, however, additional procedural references were supplied that contained same information that ROs are required to know, references such as AOP, ARP and OPs procedures and lesson plans.

3.5 NRC: LOD=2.

5 H X N E LIC: Agree LOD=2.

2 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
  • T/F: The answer choices are a collection of unrelated true/false statements.
  • Cred. Dist: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
  • Partial: One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4. Check the appropriate box if a job content error is identified:
  • Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
  • Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
  • #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
  • Backward: The question requires reverse logic or application compared to the job requirements.
5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

A + in the Q# column indicates that question was reviewed as part of the representative sample of 30 questions.

  1. 7 RO and 3 SRO Only questions submitted for NRC pre-review 1 of 9

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 3

6# F B S 2012 NRC Exam 2

7 H 3 N S 8+ H 3 N S Handout Air System Mimic 9 H 4 N S H 3 NRC: LOK=F, memory that only rad alarms for warning and 10 N E evacuation. LOD=2.

F 2 LIC: Agree 2013 NRC Exam 11 H 3 B E NRC: Revise last stem sentence to read: Based on these plant conditions what action(s) should be taken?

LIC: Incorporated.

12+ H 3 N S Handout 29.100.01, Sht. 6 13 H 3 M E 1 of 2/twice NRC: 1) Delete note at end of stem that identifies the curve reference. 2) Overpressure =5+3.5+(-70/30)=6.16 not 5.2. 3) Change distractor flows such that they are numbers based on incorrect number of pump(s) for RHR or CS.

LIC: Q15 based on vortex limits.

14 H 2 M S Handout Steam Tables; 29.100.01, Sht. 6 Handout 29.100.01, Sht. 6 15 H 3 B U NRC: Cue: same curves used on question 13 for determining CS and E RHR flows based on NPSH and Vortex Limits. Would be double jeopardy.

LIC: Q15 based on vortex limits.

16+ H 3 N S NRC: LOK=F, Q does not indicate a Handout, appears to be 17 H 3 N E memory. Are ROs expected to have this information memorized? 2) 2 LOD=2.

LIC: Analysis/comprehension required to determine answer, not memory. Expected RO knowledge.

18 H 3 X N U NRC: 1) LOK=F. Memorization of interlocks. 2) C and D not plausible, both N an S running in AUTO and each receive Hi-Hi rad F 2 signal, but only one gets isolated, believable (?). 3) LOD=2.

LIC: Rewrote C and D, LOD=2.

2 of 9

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 19 H 2 B S 20 H 3 N S 21+ F 2 M S NRC: 1) LOK=F. Memorization of LLS interlock for the first SRV to 22 H 2 B E open when RPS for reactor pressure scram setpoint of 1093 is F reached.

LIC: Agree.

1 of 2 twice 23+ H 2 N E NRC: 1) LOK=F, memorization of trip and bases.

F LIC: Agree.

24# H 2 B S 25 H 3 B S 26 H 3 B S 27 F 2 B S NRC: LOK=F, LOD=1, answer C. is only choice that has a pressure 28# H 1 X N U above the Minimum Steam Cooling Pressure (MSCP). Also, provide F justification of choices.

LIC: Will replace.

Early Review Replacement Q for 28:

28r H 3 N E NRC: 1) Incorrect reference supplied with question; however, correct reference located on reference disc. 2) Is Table 8, MSCP, an exam handout?

LIC: Per telecom: 1) Reference on References disc. 2) Applicants will not get Table 8, they are expected to know the table (memory).

29+ H 4 M S 1 of 2 twice 30# H 3 M S 1 of 2 twice 2010 NRC Exam 31 F 3 B S 1 of 2 twice NRC: LOD=2. What was miss rate on this question during 32+ F 3 N E validation?

2 LIC: Agree LOD=2.

33 F 2 M S 3 of 9

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 1 of 2 twice 34 H 3 N E NRC: 1) LOK=F, very little analysis, more memory knowledge of CS F 2 pump start and FO15A interlock with RWL. 2) LOD=2. What was miss rate on this question during validation?

LIC: Agree 2012 NRC Exam 35 H 2.5 B E NRC: 1) LOD=2. What was miss rate on this question during 2 validation?

LIC: Agree 2012 NRC Exam 36+ H 3 B E NRC: LOK=F. Memory knowledge, either know or dont know F information, no way to determine. (Identified as F on 2012 Exam).

LIC: Agree 1 of 2 twice 37 F 2 B S 38 F 2 B S 39 H 3 M S 1 of 2 twice 40+ H 3 B S NRC: LOD=3.

41 H 4 M E LIC: Agree 3

NRC: 1) As written, the applicant could say A, B, C and D contain 42 F 4 N E affected groups. Reword stem to ask the applicant to identify which of the following includes ONLY groups impacted by the error? 3)

Rewrite choices to only identify only 3 groups. Pick incorrect groups that include misconceptions of being affected by L1. For example: A.

Groups 1,3,5 (correct); B. Groups 1,2,4; C. Groups 2,3,6; D. Groups 4,5,6.

LIC: Agree, rewrote distractors.

43 H 3 N S NRC: Not significantly modified. Stem still has RFP B controller shift 44+ F 3 M E to EMERGENCY BYPASS mode and answer is still same.

B LIC: Agree 45 F 2 M S 46 H 3 M S 1 of 2 twice 47 F 3 N S 4 of 9

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N NRC: LOK=F, memory of power supplies.

48+ H 2 B E LIC: Agree F

NRC: LOD=3 49 H 4 M E LIC: Agree 3

NRC: LOD=3 50 H 2.5 B E LIC: Agree 3

51 H 3 B E 2013 NRC Exam F 2 NRC: 1) LOK=F, memory of setpoint; 2) LOD=2.

LIC: Agree F 3 N S 1 of 2 twice 52+

4 NRC: LOD=3 53 H B E 3 LIC: Agree NRC: 1) LOD=2; 2) LOK=F, memory of design basis reference info 54 H 3 X B E for the stabilizing valves. Stem term oscillating and valve name F 2 stabilizing are cues to answer.

LIC: Agree F NRC: LOK=H, using analysis/synthesis to arrive at answer.

55+ 3 M E H LIC: Agree 56 H 3 M S 57 H 2 M S 58 H 3 B S 59# F 3 B S 2013 NRC Exam NRC: LOK=F, using memory, not analysis/synthesis to arrive at 60+ H 2 N E answer.

F LIC: Agree 61 F 2 N S 1 of 2 twice 62 F 2 B S 63 F 2.5 B E 2012 NRC Exam 3 NRC: LOD=3, use whole numbers.

LIC: Agree 64+ H 3 B S 5 of 9

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 65 H 2 M S 66 F 2 B S 67 F 3 B E Handout Safety Handbook, Sec. 21 NRC: Delete reference note from question.

LIC: Agree 68+ F 2 B S NRC: LOK=L, no analysis/synthesis to determine correct answer.

69 H 2 B E Only memory.

F LIC: Agree F 3 N E NRC: LOD=2, low discrimination of competency.

70 2 LIC: Agree 71 H 3 B E 2006 NRC Exam F 2 M NRC: 1) LOK=L, simple math calculation of stay time and knowledge of Life Saving Measures WB TEDE limit; LOD=2, low discrimination of competency. 2) Modified Q71 on 2006 NRC exam.

LIC: Agree 72+ F 3 B S 73 H 4 N S 1 of 2 twice 74# H 3 X N U NRC: LOK=L, LOD=1, two implausible distractors, B and D. B implausible because there is no RWL problem indicated so RCIC is F 1 not required. D is implausible because of the major alarms given, which indicate something must have actuated or isolated. Also, stem wording is asking for isolations and/or isolations and choice D is saying no isolations and/or actuations.

LIC: Replaced question (74r).

74r H 3 N S Early Review Replacement Q for 74:

75# F 2 N U NRC: LOD=1, this is something done on each scenario. Not appropriate to have on both written and operating test, double 1

jeopardy.

LIC: Replaced (75r).

75r F 2 N S Early Review Replacement Q for 75:

R RO Exam: LOK: H(50-60%)=40 /75(53.3%); F=35/75(46.7%) N(8)=24/75(32%), B(56 max)=35/75=42.6%, M=16/75(21.3%): U=1/75(1.3%); E=29/75(38.7%); LOD=2.6 6 of 9

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 1 of 2 twice 76 H 3 X N S 2012 NRC Exam 77 H 3.5 X B S NRC: LOD=3, only use whole numbers.

3 LIC: Agree 78 H 3 X B S 2012 NRC Exam Handout Steam Tables 79 H 3 X M E 1 of 2 twice NRC: Q pedigree identifies LOK=L.

LIC: LOK=H, will change pedigree 1 of 2 twice 80 H 3 X B S 2012 NRC Exam 81 H 3 X B E NRC: This is Modified SRO Q5 from 2012 NRC Exam. Stem M changed temp from 130 to 170 deg F, replaced distractor C and rewrote distractor D.

LIC: Agree 82 H 3 X B S NRC: Modify last stem sentence to read, The CRS will direct which 83 H 3 X B E of the following procedural actions?.

LIC: Incorporated.

NRC: 1) Add copy of TR 3.3.6.4 AND TR 3.6.2.1 to Handout List 84 H 4 N E LIC: Agree NRC: 1) Change last stem sentence part (2) to read what procedural 85 H 2 X M E actions will the CRS direct? 2) Not significantly modified, still B.

B LIC: Agree NRC: 1) LOD=3, use whole numbers; 2) Pedigree incorrectly lists Q 86 H 3.5 X B E is B used on 2015 NRC exam.

3 LIC: Agree NRC: 1) KA for question (209001A2.04, DC failure) does not match 87# H 2.5 X X B U written exam outline KA (209001A2.08, valve opening). 2) Do SROs 3 get reference? 3) ) LOD=3, use whole numbers LIC: Rewrote (Q87r)

NRC: 1) LOD=3, use whole numbers; 87r H 2.5 X X B E LIC: Agree 3

1 / 2 twice; TS 3.3.1.1 88# H 3 X N E NRC: 1) Do SROs get reference? 2) LOD=2 2 LIC: 1) Get TS 3.3.1.1; 2) Agree 7 of 9

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 1 / 2 twice 89 H 3 X N S 90 H 3 X B S 91 H 2.5 X B S 3

NRC: 1) LOD=2, completing bases statement; 2) LOK=L, 92 H 3 X B E memorization of bases.

F 2 LIC: Agree 1/2 twice 93# H 4 X N E NRC: 1) LOD=3, simple math, memorization of EOP steps. 2) Modify 3 stem part 2) to reflect actions to be directed by the CRS.

LIC: Agree 94 F 2 X B S NRC: 1) LOD=1, the CRS is only licensed person in control room, 95 F 2 X B U also as SRO he also has RO license. Q is too simple, not good 1 discrimination. Would not expect applicant to not have this knowledge.

LIC: Replace Q95 with Q95r 95r F 2 X B S 96 H 2 X B E Handout MMR, App. H NRC: 1) Need to get rid of like terms in stem and reference. Modify stem, replace terms high sustained and gust with wind speeds 43 to 48 mph and wind bursts of 60 mph.

LIC: Agree 97 H 4 X B E Handout MMA26, Enc. C 3 NRC: 1) LOD=3; 2) Delete (See attached MMA26, Enc. C) from stem. Make reference part of handouts to applicants. 3) Delete first bullet in stem. Applicant should be able to identify the Risk Category as High Risk per the 1.1 list in Enclosure C, Section 1.0.

LIC: Agree 1 / 2 twice 98 F 4 X N E NRC: 1) LOD=2, common NGET knowledge for onsite rad workers; 2 LIC: Agree NRC: 1) Insert by the CRS between directed and to into the 99 H 3 X B E last stem sentence.

LIC: Agree 8 of 9

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/ U/E/S Explanation Focus Dist. Link units ward K/A Only N 100 H 2 X N E Handout EAL Chart NRC: 1) Delete note about attached EAL chart in last stem sentence.

Chart will be part of the applicant handouts.

LIC: Agree For SRO portion of Exam: LOK: H(50%min)=21/25(85%); F=4/25(16%); N(2)=7/25(28%), B(19 max)=16/25(64%), M=2/25(8%): U=0/25(0%); E=15/25=60%; LOD=68/25=2.7 Were any Bank questions from previous NRC exams? Identified during validation.

Remove handout reference from questions. Make sure question numbers arent included on the Handout list to applicants during exam.

Submit original questions for Modified questions to verify they were modified IAW ES 1021. Questions submitted.

9 of 9

Retrieved from "https://kanterella.com/w/index.php?title=ML17198F907&oldid=2006195"