ML19036A824

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML19036A824
Person / Time
Site:	Salem
Issue date:	01/31/2019
From:	David Silk Operations Branch I
To:
Shared Package
ML18081A982	List: IR 05000311/2019301 ML19008A330 ML19036A820 ML19036A821 ML19036A822 ML19036A823 ML19036A824 ML19036A825 ML19036A826 ML19036A827 ML19036A828 ML19046A230 ML19046A231
References
CAC 000956
Download: ML19036A824 (6)

Text

ES-401 Salem Written Examination Review Worksheet Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H) (1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 1

F 3

N s

2 F

2 M

s 3

H 2

M s

4 H

2 N

s 5

F 1

N u

Low LOO replace Q with higher cognitive Q.

6 F

1 N

u Low LOO replace with higher cognitive bank question.

7 F

2 B,D N

u Replace B2 & D2 with a lower pressure (25#).

8 F

2 BN s

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy-difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6.

Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M}odified questions meet criteria of ES-401 Section D.2.f.

7.

Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO 8/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 9

H 2

M s

10 H

2 X

B E

Delete line about steam table.

11 H

3 X

M E

Remove "properly".

12 H

2 M

s 13 H

2 N

s 14 H

2 N

s 15 H

2 N

s 16 H

1 X

N u

"Local Manual" is not plausible. Replace or rework Q.

17 F

2 M

s 18 F

2 N

s 19 F

1 N

u Rework Q with LOOP, EDG failure & faulted bus for higher cog Q.

20 H

2 C,D N

u 21-22AF923 is manually operated. Replace C & Dor replace Q.

21 F

2 X

BN E

Incorporate loss of transformer and its impact for higher level Q.

22 F

2 B

s 23 H

3 N

s 24 F

2 N

s 25 F

2 N

s 26 F

2 X

N E

Remove "2" from choice D.

27 F

2 N

s 28 F

2 C,D N

u Equipment hatch not credible. Rework part 1 of Q to include LCO.

ES-401 3

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 29 H

2 N

s 30 H

2 N

s 31 H

2 X

B E

Clarify Q to ask about AMSAC signal to AFW pumps.

32 F

1 X

B u

Replace Q with ~

setting on H2 recombiners for higher cog Q.

33 F

2 N

s 34 F

2 N

s 35 F

2 N

s 36 H

2 B

s 37 H

2 N

s 38 H

2 N

s 39 F

2 X

N E

Replace SOS with SM.

40 H

2 B

s 41 F

3 N

s 42 F

2 N

s 43 F

2 N

s 44 F

2 BN s

45 F

1 B

u Replace with stay time calculation for higher cognitive Q.

46 H

1 M

u Use parameter values instead of colored graphs.

47 F

4 N

s 1Aj8 F

2 M

s

ES-401 4

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 49 F

2 N

s 50 H

1 X

X B

u Low LOD based on picture. Replace Q to address KA.

51 H

2 N

s 52 H

2 B

s 53 H

2 N

s 54 H

3 N

s 55 H

2 N

s 56 F

2 N

s 57 H

2 N

s 58 F

3 N

s 59 F

2 X

N E

Add "at" to 151 line.

60 H

3 X

N E

Typo - remove 2nd "is".

61 F

2 N

s 62 H

4 N

s 63 F

2 X

M E

Delete 2nd part of first Q.

64 H

2 X

M E

Q is disjointed. Add "If you can proceed" to 2nd part.

65 H

2 BN s

66 H

2 M

s 67 F

2 N

s h£'8 F

2 N

s

ES-401 5

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 69 H

2 N

s 70 F

2 N

s 71 F

2 X

N u

Rework Q applies to match KA. Ask for reason for limiting AFW flow.

72 H

2 X

N E

Remove 2nd "in".

73 F

3 X

N u

There are multiple correct answers. Rework Q to avoid subset.

74 F

2 N

s 75 H

2 N

s N=51; M=11; 8=13 NRC: F=40, H=35 (46.6%)

76 H

2 X

N E

Describe LOCA conditions and add procedure/step to Band D.

77 H

2 X

N E

Rework Q for better SRO Q format.

78 H

2 X

M E

Add procedure name to choice D.

79 H

2 X

N E

Do not provide TS reference. This is basic TS 3.04 knowledge.

80 F

2 X

N E

Describe loss of IA conditions.

81 F

2 N

s 82 H

2 X

X N

E Rework Q for SRO format. Avoid Part 1 leading to 2nd answer.

83 H

2 X

N E

Do not provide TS reference. This is common TS knowledge.

84 H

2 B

s 85 H

2 X

N E

Add loss of second bus to make all choices more plausible.

86 H

3 N

s

ES-401 6

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 87 H

2 X

M u

Reselect KA for better SRO topic.

88 H

2 X

N E

Provide LOCA conditions in stem. Reformat for SRO Q.

89 H

2 B

s 90 H

2 N

s 91 F

2 X

M E

Reformat for procedure selection.

92 H

2 N

s 93 F

2 X

B E

Reformat for procedure selection.

94 F

2 N

s 95 H

2 X

N E

Provide a condition instead of using TS wording. Bold "initially".

96 H

2 X

N E

Do not provide TS. Basic application of TS 3.0.3. Revise Q.

97 F

2 N

s 98 F

2 N

s 99 H

2 X

N E

Reformat for procedure selection.

100 F

1 N

u Common EP knowledge. Replace with PAR upgrade Q.