ML18074A360

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML18074A360
Person / Time
Site:	Millstone
Issue date:	10/30/2017
From:	Peter Presby Operations Branch I
To:	Dominion Energy Nuclear Connecticut
Shared Package
ML17094A469	List: ML18029A643 ML18029A662 ML18074A356 ML18074A358 ML18074A360 ML18074A361 ML18074A362 ML18074A363 ML18074A364 ML18074A365 ML18074A366 ML18074A368
References
CAC U01946, ES-401, ES-401-9
Download: ML18074A360 (7)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q

1. LOK LOD

6. Source

7. Status

8. Explanation (F/H)

(1-5)

Stem Cues T/F Cred.

Partial Job-Minutia

1. /Units Q-KIA SRO Only (B/M/N)

(U/E/S)

Focus Dist Link Backward 1

H 3

N E

Modified distractor "d" to improve plausibility.

2 H

4 B

E Added detail to the sequence of events (#4) and corrected typos 3

F 3

B E

Modified stem and distractor "d" to improve distractor plausibility.

Re-categorized Q as 4

F 3

N E

Fundamental. Modified wording of Q to eliminate need for the term "only" in the choices.

5 H

3 B

s No comments 6

H 3

N E

Reworded distractors to improve plausibility.

Changed from 2x2 format, distractors modified to raise 7

F 2

N E

LOD and improve operational relevance. KIA match was explained better.

8 F

3 N

E Reworded to ensure distractor is wrong and to raise LOD.

Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3.

Check the appropriate box if a psychometric flaw is identified:

"Stem Focus": The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

"Cues": The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).

'T/F": The answer choices are a collection of unrelated true/false statements.

"Cred. Dist.": The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.

"Partial": One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).

4.

Check the appropriate box if a job content flaw is identified:

"Job Link": The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

"Minutia": The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).

"#/Units": The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

"Backward": The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved KIA and those KIAs that are designated "SRO-only." (KIA and license-level mismatches are unacceptable.)

6.

Enter question's source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.

7.

Based on the reviewer's judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any "U" status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws 5.0ther Q

LOK LOO

6. Source

7. Status

8. Explanation (F/H)

(1-5)

Stem T/F Cred.

Partial Backward Q-K/A SRO Only (B/M/N)

(U/E/S)

Focus Cues Dist Job-Link Minutia

1. I Units 9

H 4

B s

10 H

3 B

E Reworded choices to improve discriminatory value.

11 H

3 B

E Reworded choices to improve discriminatory value.

The question was changed to 12 H

3 N

E 2x2 format and distractors reworded for better plausibility.

Modified Q and choice wording 13 F

3 N

E to reduce number of repeated words in choices.

14 H

4 X

N u

Replaced with bank Q to match the KIA.

15 F

3 B

E Removed window-dressing from Q stem.

Reworded Q to address 16 F

2 N

E psychometric and plausibility weaknesses.

17 F

3 N

s No comments 18 F

3 N

E Reworded distractors to improve plausibility 19 H

3 B

E Reworded distractors to improve plausibility 20 F

3 B

s No comments 21 H

3 B

s No comments Distractors A and B were not 22 H

3 X

N u

plausible given the stem information. Reworded all 4 distractors to improve olausibility.

23 H

4 B

E Minor wording change in stem Reduced word count and 24 H

3 B

E simplified stem to use distractors that use times only

ES-401 3

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws 5.0ther Q

LOK LOD

6. Source

7. Status

8. Explanation (F/H)

(1-5)

Stem Cues T/F Cred.

Partial Job-Link Minutia

1. / Units Backward Q-K/A SRO Only (B/M/N)

(U/E/S)

Focus Dist 25 F

3 B

s No comments 26 F

2 B

s No comments 27 F

3 B

s No comments 28 H

3 B

E Added detail to distractor "C" 29 H

3 N

E Added wording to distractors "C" and "D" 30 F

3 N

E Stem wording added 31 F

3 B

E Question was too verbose and was simplified to 2x2 fonnat 32 H

1 X

N u

LOD1 - question too simple.

New question written 33 F

3 X

B u

Poor KIA match. New bank question selected 34 H

3 B

s No comments Distractor plausibililty needed 35 F

2 X

B u

improvement. New bank auestion selected 36 F

3 N

s No comments 37 H

3 X

M E

Question was not operationally valid. New question written.

Too easy to eliminate distractors.

38 H

3 X

B u

Distractor plausibililty needed improvement. New bank auestion selected 39 F

3 N

E Stem reworded to improve clarity.

40 H

3 X

B u

Choice C implausible. New bank question selected

ES-401 4

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q

LOK LOO

6. Source

7. Status

8. Explanation (F/H)

(1-5)

Stem Cred.

SRO Only (B/M/N)

(U/E/S)

Focus Cues T/F Dist Partial Job-Link Minutia

1. I Units Backward Q-KIA Revised stem, changed to 2x2 41 H

4 B

E format to clarify and improve KIA match.

Too many repeated words in the 42 H

3 N

E stem. Simplified stem by removing repeated words.

43 H

1 X

N u

LOO 1. Rewrote new question 44 H

3 B

E Minor stem rewording 45 H

3 B

E Modified distractors to improve plausibility.

46 H

3 N

s No comments 47 F

3 X

B u

Choice C was not plausible.

Selected new bank question.

48 H

3 B

s No comments Oistractor A was implausible and 49 F

2 X

B u

B and C were not logical.

Reselected a new bank auestion.

50 H

3 B

s No comments 51 H

4 B

E Reworded stem to remove repeated words 52 F

3 X

B u

Distractor A is not believable.

Reselected new bank question 53 H

3 B

E 3 part question structure was too confusing. Rewrote to 2 part.

54 F

3 N

E Oistractors were reworded to improve plausibility.

55 H

3 X

N u

KIA mismatch. Question rewritten.

56 H

3 B

E Minor rewording of stem 57 F

2 N

E Minor rewording of distractors in part 2

ES-401 5

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws 5.0ther Q

LOK LOO

6. Source

7. Status

8. Explanation (F/H)

(1-5)

Stem T/F Cred.

Partial Minutia

1. I Units Backward Q-K/A SRO Only (B/M/N)

(U/E/S)

Focus Cues Dist Job-Link 58 F

1 B

E Question distractors revised to improve plausibility.

59 H

4 B

E Minor rewording in stem 60 H

4 N

s No comments 61 H

3 N

s No comments 62 F

3 N

s No comments 63 F

3 B

s No comments 64 H

3 B

E Minor format edits made in 2x2 structure 65 H/F 3

B E

Changed to Fundamental level 66 F

3 N

E Modified distractors to improve discriminatory value.

67 F

3 B

s No comments 68 F

3 B

s No comments 69 H

3 M

s No comments 70 H

4 B

s No comments 71 F

3 X

N u

Question was not discriminating.

Reselected a bank question.

72 F

3 B

s No comments 73 H

3 N

E Added to stem to bound time frame 74 F

3 N

E Small distractor wording modifications made

ES-401 6

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q

LOK LOO

6. Source

7. Status

8. Explanation (F/H)

(1-5)

Stem Cues T/F Cred.

Partial Job-Link Minutia

1. I Units Backward Q-K/A SRO Only (8/M/N)

(U/E/S)

Focus Dist 75 H

3 B

E Reworded distractors Changed initial conditions from 76 H

4 N

E 50% to 100% and reworded distractors to improve discriminatory value.

More than one correct answer (A 77 H

4 X

N u

and C are both correct).

Rewrote auestion.

78 H

3 N

E Minor changes to improve discriminatory value.

79 H

3 N

s No comments 80 H

3 B

s No comments 81 H

4 B

s No comments 82 H

3 N

E Typo - changed ANSI to ANS 83 H

3 B

s No comments 84 H

3 B

s No comments 85 H

3 B

s No comments 86 H

3 B

E Changed to 2x2 format and modified stem 87 H

3 N

s No comments 88 H

3 B

E Reworded distractor D for plausibility Selected bank question as new 89 H

3 X

N u

question as-written was not discriminatina.

90 F

3 N

s No comments

ES-401 7

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q

LOK LOD

6. Source

7. Status

8. Explanation (F/H)

(1-5)

Stem Cred.

1. I Units Backward Q-K/A SRO Only (B/M/N)

(U/E/S)

Focus Cues T/F Dist Partial Job-Link Minutia 91 F

3 N

E Distractors reworded to improve LOD.

92 H

4 X

B u

KIA mismatch. Bank question selected.

93 H

3 X

N u

Not SRO only level/ DLU.

Question rewritten as new.

94 H

3 B

s No comments 95 F

3 B

E Minor stem rewording 96 H

3 B

s No comments 97 H

3 B

E Reworded distractors to improve discriminatory value.

98 F

3 B

s No comments 99 F

3 B

s No comments 100 F

3 X

B u

Not discriminating. Rewrote distractors to improve plausibility.