ML18074A360

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML18074A360
Person / Time
Site:	Millstone
Issue date:	10/30/2017
From:	Peter Presby Operations Branch I
To:	Dominion Energy Nuclear Connecticut
Shared Package
ML17094A469	List: ML18029A643 ML18029A662 ML18074A356 ML18074A358 ML18074A360 ML18074A361 ML18074A362 ML18074A363 ML18074A364 ML18074A365 ML18074A366 ML18074A368
References
CAC U01946, ES-401, ES-401-9
Download: ML18074A360 (7)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. LOK 6. Source 7. Status Q LOD 8. Explanation (F/H) Stem Cred. Job- (B/M/N) (U/E/S)

(1-5) Cues T/F Partial Minutia #/Units Backward Q-KIA SRO Only Focus Dist Link Modified distractor "d" to 1 H 3 N E improve plausibility.

Added detail to the sequence of 2 H 4 B E events (#4) and corrected typos Modified stem and distractor "d" 3 F 3 B E to improve distractor plausibility.

Re-categorized Q as Fundamental. Modified wording 4 F 3 N E of Q to eliminate need for the term "only" in the choices.

5 H 3 B s No comments Reworded distractors to 6 H 3 N E improve plausibility.

Changed from 2x2 format, distractors modified to raise 7 F 2 N E LOD and improve operational relevance. KIA match was explained better.

Reworded to ensure distractor 8 F 3 N E is wrong and to raise LOD.

Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3.

Check the appropriate box if a psychometric flaw is identified:

"Stem Focus": The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

"Cues": The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length) .

'T/F": The answer choices are a collection of unrelated true/false statements .

. "Cred. Dist.": The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable .

"Partial": One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).

4.

Check the appropriate box if a job content flaw is identified:

"Job Link": The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

. "Minutia": The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory) .

"#/Units": The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

• "Backward": The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those KIAs that are designated "SRO-only." (KIA and license-level mismatches are unacceptable.)

6. Enter question's source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.

7. Based on the reviewer's judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5.0ther

6. Source 7. Status Q LOK LOO 8. Explanation Stem Cred. (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Job-Link Minutia # I Units Backward Q-K/A SRO Only Focus Dist 9 H 4 B s Reworded choices to improve 10 H 3 B E discriminatory value.

Reworded choices to improve 11 H 3 B E discriminatory value.

The question was changed to 12 H 3 N E 2x2 format and distractors reworded for better plausibility.

Modified Q and choice wording 13 F 3 N E to reduce number of repeated words in choices.

Replaced with bank Q to match 14 H 4 X N u the KIA.

Removed window-dressing from 15 F 3 B E Q stem.

Reworded Q to address 16 F 2 N E psychometric and plausibility weaknesses.

17 F 3 N s No comments Reworded distractors to improve 18 F 3 N E plausibility Reworded distractors to improve 19 H 3 B E plausibility 20 F 3 B s No comments 21 H 3 B s No comments Distractors A and B were not plausible given the stem 22 H 3 X N u information. Reworded all 4 distractors to improve olausibility.

23 H 4 B E Minor wording change in stem Reduced word count and 24 H 3 B E simplified stem to use distractors that use times only

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5.0ther

6. Source 7. Status Q LOK LOD 8. Explanation Stem Cred. (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Job-Link Minutia #/ Units Backward Q-K/A SRO Only Focus Dist 25 F 3 B s No comments 26 F 2 B s No comments 27 F 3 B s No comments 28 H 3 B E Added detail to distractor "C" Added wording to distractors "C" 29 H 3 N E and "D" 30 F 3 N E Stem wording added Question was too verbose and 31 F 3 B E was simplified to 2x2 fonnat LOD1 - question too simple.

32 H 1 X N u New question written Poor KIA match. New bank 33 F 3 X B u question selected 34 H 3 B s No comments Distractor plausibililty needed 35 F 2 X B u improvement. New bank auestion selected 36 F 3 N s No comments Question was not operationally 37 H 3 X M E valid. New question written.

Too easy to eliminate distractors.

Distractor plausibililty needed 38 H 3 X B u improvement. New bank auestion selected Stem reworded to improve 39 F 3 N E clarity.

Choice C implausible. New bank 40 H 3 X B u question selected

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

6. Source 7. Status Q LOK LOO 8. Explanation Stem Cred. (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Job-Link Minutia # I Units Backward Q-KIA SRO Only Focus Dist Revised stem, changed to 2x2 41 H 4 B E format to clarify and improve KIA match.

Too many repeated words in the 42 H 3 N E stem. Simplified stem by removing repeated words.

43 H 1 X N u LOO 1. Rewrote new question 44 H 3 B E Minor stem rewording Modified distractors to improve 45 H 3 B E plausibility.

46 H 3 N s No comments Choice C was not plausible.

47 F 3 X B u Selected new bank question.

48 H 3 B s No comments Oistractor A was implausible and 49 F 2 X B u B and C were not logical.

Reselected a new bank auestion.

50 H 3 B s No comments Reworded stem to remove 51 H 4 B E repeated words Distractor A is not believable.

52 F 3 X B u Reselected new bank question 3 part question structure was too 53 H 3 B E confusing. Rewrote to 2 part.

Oistractors were reworded to 54 F 3 N E improve plausibility.

KIA mismatch. Question 55 H 3 X N u rewritten.

56 H 3 B E Minor rewording of stem Minor rewording of distractors in 57 F 2 N E part 2

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5.0ther

6. Source 7. Status Q LOK LOO 8. Explanation Stem Cred. (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Job-Link Minutia # I Units Backward Q-K/A SRO Only Focus Dist Question distractors revised to 58 F 1 B E improve plausibility.

59 H 4 B E Minor rewording in stem 60 H 4 N s No comments 61 H 3 N s No comments 62 F 3 N s No comments 63 F 3 B s No comments Minor format edits made in 2x2 64 H 3 B E structure 65 H/F 3 B E Changed to Fundamental level Modified distractors to improve 66 F 3 N E discriminatory value.

67 F 3 B s No comments 68 F 3 B s No comments 69 H 3 M s No comments 70 H 4 B s No comments Question was not discriminating.

71 F 3 X N u Reselected a bank question.

72 F 3 B s No comments Added to stem to bound time 73 H 3 N E frame Small distractor wording 74 F 3 N E modifications made

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

6. Source 7. Status Q LOK LOO 8. Explanation Stem Cred. (8/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Job-Link Minutia # I Units Backward Q-K/A SRO Only Focus Dist 75 H 3 B E Reworded distractors Changed initial conditions from 50% to 100% and reworded 76 H 4 N E distractors to improve discriminatory value.

More than one correct answer (A 77 H 4 X N u and C are both correct).

Rewrote auestion.

Minor changes to improve 78 H 3 N E discriminatory value.

79 H 3 N s No comments 80 H 3 B s No comments 81 H 4 B s No comments 82 H 3 N E Typo - changed ANSI to ANS 83 H 3 B s No comments 84 H 3 B s No comments 85 H 3 B s No comments Changed to 2x2 format and 86 H 3 B E modified stem 87 H 3 N s No comments Reworded distractor D for 88 H 3 B E plausibility Selected bank question as new 89 H 3 X N u question as-written was not discriminatina.

90 F 3 N s No comments

ES-401 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

6. Source 7. Status Q LOK LOD 8. Explanation Stem Cred. (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Job-Link Minutia # I Units Backward Q-K/A SRO Only Focus Dist Distractors reworded to improve 91 F 3 N E LOD.

KIA mismatch. Bank question 92 H 4 X B u selected.

Not SRO only level/ DLU.

93 H 3 X N u Question rewritten as new.

94 H 3 B s No comments 95 F 3 B E Minor stem rewording 96 H 3 B s No comments Reworded distractors to improve 97 H 3 B E discriminatory value.

98 F 3 B s No comments 99 F 3 B s No comments Not discriminating. Rewrote 100 F 3 X B u distractors to improve plausibility.