ML17198F907

From kanterella
Revision as of 11:28, 19 June 2019 by StriderTol (talk | contribs) (Created page by program invented by StriderTol)
Jump to navigation Jump to search
2017 Fer Ile Administrative Files ES-401-9
ML17198F907
Person / Time
Site: Fermi DTE Energy icon.png
Issue date: 04/03/2017
From: Bielby M
Operations Branch III
To:
NRC/RGN-III/DRS/OLB
Shared Package
ML17061A771 List:
References
Download: ML17198F907 (9)
v • d • e


Text

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam 1 of 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws
5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 1 H 3 B S 1 of 2 twice 2 H 2 B S 3 H 4 3 B S 1 of 2 twice 4+ H 3 X B U S 1 of 2 twice NRC: 1) SRO ONLY, requires knowledge of TS, TSB and AOP.
2) Explain references.

LIC: References supplied were TS and TS bases, however, additional procedural references were supplied that contained same information that ROs are required to know, references such as AOP, ARP and OPs procedures and lesson plans.

5 H 3.5 2 X N E NRC: LOD=2. LIC: Agree LOD=2.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

"Stem Focus": The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

"Cues": The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc). "T/F": The answer choices are a collection of unrelated true/false statements.

"Cred. Dist": The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

"Partial": One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

"Job Link": The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content). "Minutia": The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory). "#/Units": The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). "Backward": The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO

-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

A "+" in the "Q#" column indicates that question was reviewed as part of the representative sample of 30 questions.

  1. 7 RO and 3 SRO Only questions submitted for NRC pre

-review ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam 2 of 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws
5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 6# F 3 2 B S 2012 NRC Exam 7 H 3 N S 8+ H 3 N S Handout Air System Mimic 9 H 4 N S 10 H F 3 2 N E NRC: LOK=F, memory that only rad alarms for warning and evacuation. LOD=2.

LIC: Agree 11 H 3 B E 201 3 NRC Exam NRC: Revise last stem sentence to read: "Based on these plant conditions what action(s) should be taken?"

LIC: Incorporated.

12+ H 3 N S 13 H 3 M E Handout 29.100.01, Sht. 6 1 of 2/twice NRC: 1) Delete note at end of stem that identifies the curve reference. 2) Overpressure =5+3.5+(-70/30)=6.16 not 5.2. 3) Change distractor flows such that they are numbers based on incorrect number of pump(s) for RHR or CS.

LIC: Q15 based on vortex limits.

14 H 2 M S Handout Steam Tables; 29.100.01, Sht. 6 15 H 3 B U E Handout 29.100.01, Sht. 6 NRC: Cue: same curves used on question 13 for determining CS and RHR flows based on NPSH and Vortex Limits. Would be double jeopardy. LIC: Q15 based on vortex limits.

16+ H 3 N S 17 H 3 2 N E NRC: LOK=F , Q does not indicate a Handout, appears to be memory. Are ROs expected to have this information memorized?

2) LOD=2. LIC: Analysis/comprehension required to determine answer, not memory. Expected RO knowledge.

18 H F 3 2 X N U NRC: 1) LOK=F. Memorization of interlocks. 2) C and D not plausible, both N an S running in AUTO and each receive Hi

-Hi rad signal, but only one gets isolated, believable

(?). 3) LOD=2. LIC: Rewrote C and D, LOD=2.

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam 3 of 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws
5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 19 H 2 B S 20 H 3 N S 21+ F 2 M S 22 H F 2 B E NRC: 1) LOK=F. Memorization of LLS interlock for the first SRV to open when RPS for reactor pressure scram setpoint of 1093 is reached. LIC: Agree. 23+ H F 2 N E 1 of 2 twice NRC: 1) LOK=F, memorization of trip and bases.

LIC: Agree. 24# H 2 B S 25 H 3 B S 26 H 3 B S 27 F 2 B S 28# H F 1 X N U NRC: LOK=F , LOD=1, answer C. is only choice that has a pressure above the Minimum Steam Cooling Pressure (MSCP). Also, provide justification of choices.

LIC: Will replace.

28 r H 3 N E Early Review Replacement Q for 28:

NRC: 1) Incorrect reference supplied with question; however, correct reference located on reference disc. 2) Is Table 8, MSCP, an exam handout? LIC: Per telecom: 1) Reference on References disc. 2) Applicants will not get Table 8, they are expected to know the table (memory).

29+ H 4 M S 1 of 2 twice 30# H 3 M S 1 of 2 twice 31 F 3 B S 201 0 NRC Exam 1 of 2 twice 32+ F 3 2 N E NRC: LOD=2. What was miss rate on this question during validation?

LIC: Agree LOD=2. 33 F 2 M S ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam 4 of 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws
5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 34 H F 3 2 N E 1 of 2 twice NRC: 1) LOK=F, very little analysis, more memory knowledge of CS pump start and FO15A interlock with RWL. 2) LOD=2. What was miss rate on this question during validation?

LIC: Agree 35 H 2.5 2 B E 201 2 NRC Exam NRC: 1) LOD=2. What was miss rate on this question during validation?

LIC: Agree 36+ H F 3 B E 201 2 NRC Exam NRC: LOK=F. Memory knowledge, either know or don't know information, no way to determine. (Identified as F on 2012 Exam).

LIC: Agree 37 F 2 B S 1 of 2 twice 38 F 2 B S 39 H 3 M S 1 of 2 twice 40+ H 3 B S 41 H 4 3 M E NRC: LOD=3. LIC: Agree 42 F 4 N E NRC: 1) As written, the applicant could say A, B, C and D contain affected groups. Reword stem to ask the applicant to identify which of the following includes ONLY groups impacted by the error? 3) Rewrite choices to only identify only 3 groups. Pick incorrect groups that include misconceptions of being affected by L1. For example: A. Groups 1,3,5 (correct); B. Groups 1,2,4; C. Groups 2,3,6; D. Groups 4,5,6. LIC: Agree, rewrote distractors.

43 H 3 N S 44+ F 3 M B E NRC: Not significantly modified. Stem still has RFP B controller shift to EMERGENCY BYPASS mode and answer is still same.

LIC: Agree 45 F 2 M S 46 H 3 M S 1 of 2 twice 47 F 3 N S ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam 5 of 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws
5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 48+ H F 2 B E NRC: LOK=F, memory of power supplies.

LIC: Agree 49 H 4 3 M E NRC: LOD=3 LIC: Agree 50 H 2.5 3 B E NRC: LOD=3 LIC: Agree 51 H F 3 2 B E 2013 NRC Exam NRC: 1) LOK=F, memory of setpoint; 2) LOD=2.

LIC: Agree 52+ F 3 N S 1 of 2 twice 53 H 4 3 B E NRC: LOD=3 LIC: Agree 54 H F 3 2 X B E NRC: 1) LOD=2; 2) LOK=

F, memory of design basis reference info for the stabilizing valves. Stem term "oscillating" and valve name "stabilizing" are cues to answer.

LIC: Agree 55+ F H 3 M E NRC: LOK=H, using analysis/synthesis to arrive at answer.

LIC: Agree 56 H 3 M S 57 H 2 M S 58 H 3 B S 59# F 3 B S 2013 NRC Exam 60+ H F 2 N E NRC: LOK=F, using memory, not analysis/synthesis to arrive at answer. LIC: Agree 61 F 2 N S 1 of 2 twice 62 F 2 B S 6 3 F 2.5 3 B E 201 2 NRC Exam NRC: LOD=3, use whole numbers.

LIC: Agree 64+ H 3 B S ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam 6 of 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws
5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 65 H 2 M S 66 F 2 B S 67 F 3 B E Handout Safety Handbook, Sec. 21 NRC: Delete reference note from question.

LIC: Agree 68+ F 2 B S 69 H F 2 B E NRC: LOK=L, no analysis/synthesis to determine correct answer. Only memory.

LIC: Agree 70 F 3 2 N E NRC: LOD=2, low discrimination of competency.

LIC: Agree 71 H F 3 2 B M E 20 06 NRC Exam NRC: 1) LOK=L, simple math calculation of stay time and knowledge of Life Saving Measures WB TEDE limit; LOD=2, low discrimination of competency.

2) Modified Q71 on 2006 NRC exam.

LIC: Agree 72+ F 3 B S 73 H 4 N S 1 of 2 twice 74# H F 3 1 X N U NRC: LOK=L, LOD=1, two implausible distractors, B and D. B implausible because there is no RWL problem indicated so RCIC is not required. D is implausible because of the major alarms given, which indicate something must have actuated or isolated. Also, stem wording is asking for isolations and/or isolations and choice D is saying "no isolations and/or actuations".

LIC: Replaced question (74r). 74r H 3 N S Early Review Replacement Q for 74:

75# F 2 1 N U NRC: LOD=1, this is something done on each scenario. Not appropriate to have on both written and operating test, double jeopardy. LIC: Replaced (75r)

. 75r F 2 N S Early Review Replacement Q for 75:

R RO Exam: LOK: H(50-60%)=40 /75(5 3.3%); F=3 5/75(46.74/75(32%), B(56 max)=

35/75=42.6%, M=16/75(21.3%): U=1/75(1.3%); E=2 9/75(38.7%); LOD=2.

6 ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam 7 of 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws
5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 76 H 3 X N S 1 of 2 twice 77 H 3.5 3 X B S 201 2 NRC Exam NRC: LOD=3, only use whole numbers.

LIC: Agree 78 H 3 X B S 201 2 NRC Exam 79 H 3 X M E Handout Steam Tables 1 of 2 twice NRC: Q pedigree identifies LOK=L.

LIC: LOK=H, will change pedigree 80 H 3 X B S 1 of 2 twice 81 H 3 X B M E 201 2 NRC Exam NRC: This is Modified SRO Q5 from 2012 NRC Exam. Stem changed temp from 130 to 170 deg F, replaced distractor C and rewrote distractor D

. LIC: Agree 82 H 3 X B S 83 H 3 X B E NRC: Modify last stem sentence to read, "The CRS will direct which of the following procedural actions?".

LIC: Incorporated.

84 H 4 N E NRC: 1) Add copy of TR 3.3.6.4 AND TR 3.6.2.1 to Handout List LIC: Agree 85 H 2 X M B E NRC: 1) Change last stem sentence part (2) to read "what procedural actions will the CRS direct?"

2) Not significantly modified, still B.

LIC: Agree 86 H 3.5 3 X B E NRC: 1) LOD=3, use whole numbers; 2) Pedigree incorrectly lists Q is B used on 2015 NRC exam.

LIC: Agree 87# H 2.5 3 X X B U NRC: 1) KA for question (209001A2.04, DC failure) does not match written exam outline KA (209001A2.08, valve opening).

2) Do SROs get reference?
3) ) LOD=3, use whole numbers LIC: Rewrote (Q87r) 87r H 2.5 3 X X B E NRC: 1) LOD=3, use whole numbers; LIC: Agree 88# H 3 2 X N E 1 / 2 twice; TS 3.3.1.1 NRC: 1) Do SROs get reference?
2) LOD=2 LIC: 1) Get TS 3.3.1.1; 2) Agree

ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam 8 of 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws
5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 89 H 3 X N S 1 / 2 twice 90 H 3 X B S 91 H 2.5 3 X B S 92 H F 3 2 X B E NRC: 1) LOD=2, completing bases statement; 2) LOK=L, memorization of bases.

LIC: Ag r ee 93# H 4 3 X N E 1/2 twice NRC: 1) LOD=3, simple math, memorization of EOP steps.

2) Modify stem part 2) to reflect actions to be directed by the CRS.

LIC: Agree 94 F 2 X B S 95 F 2 1 X B U NRC: 1) LOD=1, the CRS is only licensed person in control room, also as SRO he also has RO license. Q is too simple, not good discrimination. Would not expect applicant to not have this knowledge.

LIC: Replace Q95 with Q95r 95r F 2 X B S 96 H 2 X B E Handout MMR, App. H NRC: 1) Need to get rid of like terms in stem and reference. Modify stem , replace terms "-high sustained-" and "-gust-" with "-wind speeds 43 to 48 mph-" and "-wind bursts of 60 mph-". LIC: Agree 97 H 4 3 X B E Handout MMA26, Enc. C NRC: 1) LOD=3; 2) Delete "(See attached MMA26, Enc. C)" from stem. Make reference part of handouts to applicants. 3) Delete first bullet in stem. Applicant should be able to identify the Risk Category as High Risk per the 1.1 list in Enclosure C, Section 1.0.

LIC: Agree 98 F 4 2 X N E 1 / 2 twice NRC: 1) LOD=2, common NGET knowledge for onsite rad workers; LIC: Agree 99 H 3 X B E NRC: 1) Insert "-by the CRS-" between directed and to into the last stem sentence.

LIC: Agree ES-401 Written Examination Review Worksheet Form ES-401-9, Rev. 10 2017 Fermi Exam 9 of 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws
5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 100 H 2 X N E Handout EAL Chart NRC: 1) Delete note about attached EAL chart in last stem sentence. Chart will be part of the applicant handouts.

LIC: Agree For SRO portion of Exam: LOK: H(50%min)=

21/25(85%); F=4/25(167/25(28%), B(19 max)=1 6/25(64%), M=2/25(8%):

U=0/25(0%); E=15/25=60%; LOD=6 8/25=2.7 Were any Bank questions from previous NRC exams? Identified during validation.

Remove handout reference from questions. Make sure question numbers aren't included on the Handout list to applicants during exam.

Submit original questions for Modified questions to verify they were modified IAW ES 1021. Questions submitted.

Retrieved from "https://kanterella.com/w/index.php?title=ML17198F907&oldid=1701176"