ML14010A035

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML14010A035
Person / Time
Site:	Millstone
Issue date:	10/18/2013
From:	Peter Presby Operations Branch I
To:	Dominion Nuclear Connecticut
Jackson D
Shared Package
ML13164A263	List: IR 05000423/2013301 ML13213A321 ML13331A064 ML14010A034 ML14010A035 ML14010A037 ML14010A038 ML14010A041 ML14010A042 ML14010A043 ML14010A045 ML14017A248
References
TAC U01874
Download: ML14010A035 (6)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only Multiple implausible distractors. Changed stem conditions and reworded 1 H 3 X B u distractors.

2 F 2 B s Re-categorized as memory level question.

3 F 3 N E Single implausible distractor. Wrote new question.

F 2 N E Re-categorized as memory level question. Minor wording changes in 4

choices.

5 F 2.5 N E Reworded to move text from choices into question stem.

6 H 2.5 X N u Multiple implausible distractors. Modified question.

7 H 2.8 X X N u Multiple potentially correct choices. KIA mismatch. Wrote new question.

H 3 B E Modified stem data for the actual TS limit. Adjusted choice wording to 8

match TS wording.

H 3 B E Added explanation of the KIA match. Added component alpha-numeric 9

designator.

Added to justification to explain this is RO level knowledge. Minor 10 H 3 M s wording change.

11 H 3 N E Re-arranged stem information to align with sequence of choices.

12 H 2.4 B E Low LOD, weak distractors. Replaced w/another bank Q.

13 H 2.8 N E Re-categorized as memory level question. Modified stem wording.

14 F 3 B E Re-categorized as memory level question.

F 2.5 X B E Single implausible distractor. Re-ordered choices and modified question 15 wording to make distractor plausible.

16 H 3 B E Reworded to move text from choices into question stem.

Good question but poor match. Per Chief Examiner, tied this Tier 1 EPE 17 H 3 M E KA with a more appropriate Section 4 generic KA. Minor wording changes to stem and choices.

Page 1

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

LOK LOD Q# (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation i Focus Dist. Link units ward KIA Only 18 H 3 N s 19 F 4 X X N u Low operational validity, minutia. Wrote new Q.

20 F 3 B E Re-categorized as memory level question. Reformatted.

H 3 N E Added to stem as necessary to eliminate possibility of multiple 21 correct answers based on assumptions on conditions not given.

22 H 3 B s 23 H 3 N E Modified one distractor to make more clearly wrong.

24 F 3 B s Added information to justification section.

25 H 3 B s F 3 X B E One distractor helps to answer another Q. Replaced the distractor 26 and made other minor wording changes.

27 F 3 N E Low plausibility distractors. Modified stem conditions and distractors.

28 F 3 B E Moved repetitive information from each of the choices to the stem.

29 H 3 B s Added information to justification section.

30 H 3 B E Reworded to move information from choices into question stem.

31 H 3 B s Minor distractor wording changes.

F 3 X N u Multiple implausible distractors. Reworded to move information from 32 choices into question stem. Modified question and distractors.

33 H 3 B s I H 3 N E Question and choices did not align. Modified Q to include control 34 system response. Revised to reduce test item word count. I H 3 X N u KA mismatch. Rejected KIA based on oversampling. Wrote new 35 question to match new KIA.

I 36 H 3 B s I

-- - - * -L _ - - - - - - - - --

Page 2

ES-401 2 Form ES-401*9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

LOK LOD Q# (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N UIE/S Explanation Focus Dist. Link units ward KIA Only H 3 B E Reworded to eliminate ambiguity introduced by use of "only and to 37 improve operational validity."

H Reword to move text from choices into question stem and to clarify 38 3 N E Q.

39 H 3 B s Modified Q to increase LOD and provided additional stem conditions 40 H 2 B E to ensure key answer correct regardless of RCS leakage assumptions.

41 H 3 N s 42 F 3 B E Reworded to move text from choices into question stem.

43 H 2 N E Minor wording changes.

44 H 3 B s F X B E Potential for multiple correct answers. Added to stem or choices to 45 3 ensure only one correct answer. Changed one distractor 46 H 3 N s 47 H 3 B s Minutia. Lacks operational validity. Does not provide adequate basis 48 F 2 X X B u for licensing decision. Replaced with a bank Q.

49 F 3 B E Added information to justification section. Minor formatting changes.

50 F 2.5 N s Corrected typo.

51 F 2 N s 52 F 1.5 N u Low LOD. Wrote a new Q.

53 F 3 N s 54 H L__- L___

3 L____

B s !dded information to justification section.

Page 3

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

LOK LOD Q# (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 55 F 2 N s F 2 N E KIA match but low operational validity. Re-select ed the KA. Wrote a 56 newQ.

57 F 2 N E Minor wording changes.

58 F 3 B E Minor wording changes.

59 F 2 B E Minor wording changes.

60 F 2 N E Reworded question to improve operational validity.

61 F 2 N E Clarified Q. Added additional condition to the stem.

62 F 2 B s 63 H 3 B E Minor wording change in a distractor choice.

64 H 3 B E Added information to justification section.

65 F 3 B E Replaced two distractors to improve plausibility.

66 F 2 N s Added information to justification section.

I 67 H 3 B E Modified stem to align with all choices.

F 3 X N E Single implausible distractor. Replaced the distractor. Added 68  !

information to stem. Minor wording changes.

F 2 B E At Chief Examiner's direction, replaced Q with another Q on the I 69 same topic that tests a more operationally significant knowledge.

70 F 2 M s I F 1 X B u This tests general rad worker knowledge. No requirement to possess 71 1

a license to have this knowledge. Replaced with a bank Q. ,

72 H 3 N s I 73 F 3 B s i 74 F 2 B s

- - - - -- ***---*-- J Page4

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

LOK LOD Q# (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO 8/M/N U/E/S Explanation Focus Dis!. Link units ward KIA Only 75 H 3 B s 76 H 3 B s 77 H 4 B s 78 H 3 X N u Not SRO-only. Modified Q to ask forTS basis information.

Low LOD because question is direct lookup with the provided 79 H 1 X N u references. Added additional conditions to stem to ensure two separate actions apply.

80 H 3 B s Minor wording changes.

Not SRO-only. Can be answered with RO knowledge. Modified 81 H 3 X N u choices to make SRO-only.

82 F 3 N E Simplistic, leading and memory level. Modified Q and choices.

83 F 3 X B E Added to stem to ensure distractor is not partially correct.

Not SRO-only. This Question #84 helps to answer Question #21.

84 H 3 X N u Wrote a new question.

E Added noun names for the instruments referenced in the question 85 H 4 N stem.

H N E Question simplistic. At Chief Examiner's direction, reselected KIA 86 2 and wrote new Q.

87 H 4 N E Too wordy and choices potentially leading. Reworded the Q.

88 H 3 B E Reworded stem and choices to improve Q.

H 2.2 N E Weak distractors. Modified Q and choices to improve discriminatory 89 value.

F E Weak distractors. Modified Q and choices to improve discriminatory 90 3 N value.

F 3 X N u Not SRO-only. Can be answered with RO knowledge. Wrote a new 91 Q.

Page 5

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

LOK LOD Q# (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO 8/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only H 3 N E Modified stem and choices to improve alignment between question 92 and choices.

93 H 3 B E Reworded to ensure no cueing in stem.

94 H 2 B s H 3 B E Minor rewording to improve distractor balance and align answers with 95 the question.

H 3 N s Added information to justification section to explain the SRO-only 96 nature of the question.

97 F 2 N s H 3 N E Modified question to ensure more clearly testing SRO-only 98 knowledge.

99 H 4 B s 100 H 3 N E Minor wording changes.

1...___~- - '---

Page 6