ML18109A332
| ML18109A332 | |
| Person / Time | |
|---|---|
| Site: | Susquehanna  |
| Issue date: | 02/23/2018 |
| From: | Peter Presby Operations Branch I |
| To: | Talen Energy |
| Shared Package | |
| ML17138A109 | List: |
| References | |
| CAC U01950 | |
| Download: ML18109A332 (8) | |
Text
ES-401 Written Examination Quality Checklist Form ES-401-6 Facility:
SSES Date of Exam: March 2018 Exam Level:
RO/SRO Initial Item Description a
b*
c*#
- 1.
Questions and answers are technically accurate and applicable to the facility.
1...,-)
Cl
/2JI
- 2.
- a.
NRC K/As are referenced for all questions.
- b.
Facility learning objectives are referenced as available.
0--...)
- c.
Correct answer explanation and distractor analysis provided (ES-401, D.2.g)
~
~
- 3.
SRO questions are appropriate in accordance with Section D.2.d of ES-401 G'-'
c-'.>
/b 4
The sampling process was random and systematic. (If more than four RO or two SRO questions were repeated from the last two NRC licensing exams, consult the NRR/NRO OL 0
0'}
~
program office).
- 5.
Question duplication from the licensee screening/audit exam was controlled as indicated below (check the item that applies) and appears appropriate.
.JL The audit exam was systematically and randomly developed, or
_ the audit exam was completed before the license exam was started, o~
I..,.__)
(::.o
/b
_ the examinations were developed independently, or 4 7,..J ?oJ%,
--'-- thelice"""' rertm~ thatthe,e ;,oo dopUretioo, "
~ )
_ other (explain).
/1,(,
cl 1>0 )
,# "/2.1?,/z'/?"
Bank use meets limits (no more than 75% from the bank, \\
Mo~ ;'w N/A-Bank rJ/ A N(-,,.,
- 6.
at least 10% new, and the rest new or modified); enter the
,.. 32 /
r28 /
actual RO/SRO-only question distribution(s) at right.
15 / 9 10 I.A._)
c~
~
6
- 7.
Between 38 and 45 questions of the questions on the RO Memory CIA i--J/ A
)'/ /A 4~
exam and at least 13 questions of the questions on the SRO-only portion of the exam are written at the comprehension/analysis level (see ES-401, D.2.c); enter 35 / 6 40 / 19
\\,,._..)
0
~
the actual RO/SRO-only question distribution(s) at right.
- 8.
References/handouts provided do not give away answers or aid in the elimination of
~
distractors.
w
- 9.
Question content conforms to specific KIA statements in the previously approved examination outline and is appropriate for the tier to which they are assigned; deviations are justified.
l,0 C8
~
- 10.
Question psychometric quality and format meet the guidelines in Appendix B.
,...,,J VJ
~
- 11.
The exam contains the required number of one-point, multiple-choice items; the total is
'--".)
correct and agrees with the value on the cover sheet.
(.P
~
Printed Name/Signature Date
- a.
Author L11"lH:\\~L ~J;L.c..c..::t: /~ ! C:,.~-J_ e
'2.-
I 1
- If;
- b.
Facility Reviewer(*)
~-~\\*,," ~~' ~f\\C\\J\\/~~-~
ZL1ci.(J B
- c.
NRC Chief Examiner (#)
fl_-1,,Ap,tl_!'f.l,.U / ~_,e-£ ~
za-a/;r n..11... lol _\\A~k~ jJ) (1f ~1-
./
- d.
NRC Regional Supervisor 1./'ll[/B l /
Note:
The facility reviewer's initials or signature are not applicable for NRC-developed examinations.
Independent NRC reviewer initials items in Column "c"; chief examiner concurrence is required0
ES-401 Written Examination Review Worksheet Form ES-401-9
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q
- 1. LOK LOO
- 6. Source
- 7. Status
- 8. Explanation (F/H)
(1-5)
Stem Cred.
Job-Backward Q-KIA SRO Only (B/M/N)
(U/E/S)
Focus Cues T/F Dist Partial Link Minutia
- /Units 1
H 2.8 N
s 2
H 3
M s
Removed reference to breaker control power in Choice d to 3
F 2.8 B
E improve discriminatory value.
Minor edits to choice to improve symmetry.
4 H
3 N
s 5
F 2.8 N
u Overlapped with operating test.
Reselected KIA, wrote new Q.
6 H
3 B
E Modified part 1 of the choices to improve discriminatory value.
7 F
3 B
s 8
F 3
B s
Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOO) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.
- 3.
Check the appropriate box if a psychometric flaw is identified:
"Stem Focus": The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
"Cues": The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).
"T/F": The answer choices are a collection of unrelated true/false statements.
"Cred. Dist.": The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.
"Partial": One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).
- 4.
Check the appropriate box if a job content flaw is identified:
"Job Link": The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
"Minutia": The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).
#/Units": The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
"Backward": The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those KIAs that are designated "SRO-only." (KIA and license-level mismatches are unacceptable.)
- 6.
Enter question's source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.
- 7.
Based on the reviewer's judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any "U" status ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q
LOK LOO
- 6. Source
- 7. Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Link Minutia
- / Units Backward Q-K/A SRO Only (B/M/N)
(U/E/S)
Focus Dist 9
H 3
B E
Minor stem rewording for clarity.
10 H
3 B
s 11 H
3 B
s Used on 2017 NRG exam.
12 H
3 B
s 13 F
3 M
s Modified stem conditions slightly 14 H
3 M
E to ensure Q cannot be answered without application of KA knowledge.
15 H
3 B
s 16 H
3 B
s 17 H
3 N
s Removed stem reference to "pressurization mode" and Part 2 18 F
3 M
E column heading reference to "pressure" to improve discriminatory value.
19 F
3 B
s 20 F
3 B
s 21 H
3 M
s 22 H
3 M
s 23 H
4 M
s 24 F
3 B
s
ES-401 3
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q
LOK LOO
- 6. Source
- 7. Status
- 8. Explanation (F/H)
(1-5)
Stem Cred.
- I Units Backward Q-K/A SRO Only (B/M/N)
(U/E/S)
Focus Cues T/F Dist Partial Job-Link Minutia 25 F
3 N
s 26 H
2 N
s 27 F
3 N
s Modified stem and reworded 28 H
3 N
E some choices to improve olausibilitv of distractors.
29 H
3 M
s 30 H
3 B
s Removed specific temperatures 31 H
3 B
E in Choices C and D to improve distractor olausibilitv.
32 F
3 B
s 33 H
3 M
E Minor stem edits for clarity.
34 F
3 B
s 35 F
2 N
s Removed unnecessary information in Choice D about 36 H
3 B
E correct method for resetting scram to improve Q discriminatorv value.
37 F
3 B
s 38 F
3 N
s Used on 2015 NRC exam.
39 F
3 B
E Modified distractor choices to 1 and 4 vice 2 and 4 to improve olausibilitv.
ES-401 4
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other Q
LOK LOD
- 6. Source
- 7. Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Link Minutia
- I Units Backward Q-K/A SRO Only (B/M/N)
(U/E/S)
Focus Dist 40 H
3 N
s 41 H
4 N
E Minor stem modifications to enhance distractor plausibility.
42 H
3 M
E Minor choice wording edits to improve clarity.
43 F
3 B
s 44 H
3 N
s 45 H
3 N
E Modified Choices C and D to improve discriminatory value.
46 F
3 M
s Minor edits to stem and choices 47 H
3 N
E to eliminate unnecessary information and improve svmmetry of choices.
48 F
3 B
s 49 H
2 N
s 50 F
3 N
s 51 H
3 N
s 52 F
3 N
s 53 F
3 N
s Added specific component to 54 H
3 N
E stem and choices for improved svmmetrv between choices.
Removed terms from stem and 55 F
3 N
E distractors that would diagnose the condition for the applicant.
ES-401 5
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws 5.0ther Q
LOK LOO
- 6. Source
- 7. Status
- 8. Explanation (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Link Minutia
- / Units Backward Q-K/A SRO Only (B/M/N)
(U/E/S)
Focus Dist 56 H
3 B
s 57 H
3 B
s 58 F
3 B
E Minor editing of choices for balance.
59 F
3 N
s 60 F
4 N
s Modified choices and added 61 H
3 B
E information to stem to improve distractor plausibility.
Removed references to the 62 H
3 M
E backup component as unnecessary and potentially leadinq.
63 H
3 N
s Replaced one component with 64 F
3 B
E another in two of the choices to improve discriminatory value.
65 H
3 B
s 66 F
3 B
s Used on 2017 NRG exam.
67 F
3 M
s Removed extra information from 68 H
2 B
E choices to improve plausibility of distractors.
69 F
3 N
s Redacted power supply 70 H
3 B
E information from drawing to ensure this Q doesn't help to answer Q43.
ES-401 6
Form ES-401-9 Q was specific to a particular system and therefore not testing 71 F
3 X
N u
generic knowledge per expectations for Tier 3 questions.
Replaced with a new Q.
72 F
3 B
s Modified a stem bullet to ensure 73 H
3 M
E this Q doesn't help with answering Q#28.
74 F
3 B
s 75 F
3 N
s 76 H
3 N
s Redacted information in handout that helps with answering Q87.
77 H
3 N
s 78 H
3 N
E Changed value in Choice D to improve plausibility.
79 H
3 M
s 80 H
3 B
s 81 H
3 N
s 82 H
3 N
E Modiified stem and choices to improve plausibility of distractors.
83 H
3 M
s Redacted information in handout that helps with answering Q87.
Q too low LOO. Reselected KIA 84 H
1 N
u because could not write adequate SRO Q to original KIA 85 F
3 M
E Minor changes to choices to improve distractor plausibility.
Changes to improve 86 H
3 M
E discriminatory value of the question. Removed provided references. Changed Choice D.
87 H
3 M
s Used on 2015 NRG exam.
ES-401 7
Form ES-401-9 88 H
3 N
E Added a time reference to the stem for improved clarity.
89 H
3 B
s Too low LOO with provided reference. Modified stem conditions to require evaluation 90 H
1 N
u of operability. Redacted above the line information from TS and removed surveillance information from TS.
91 H
3 B
s Modified Q and reference 92 H
3 M
E provided to improve discriminatorv value of Q.
Modified format of some stem information, consistent with how 93 H
3 M
E information is displayed in the plant. Deleted reference to alarm state to ensure information is not leading.
94 H
3 M
s 95 H
3 B
s 96 H
3 N
s 97 F
3 M
s Added note to ensure references 98 F
3 B
s provided for other Qs do not contain key information that would help answering Q98.
99 F
3 N
s Used on 2016 NRG exam.
100 F
3 B
s