ML15281A141

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML15281A141
Person / Time
Site:	Millstone
Issue date:	08/14/2015
From:	David Silk Operations Branch I
To:	Dominion Nuclear Connecticut
Shared Package
ML15072A211	List: IR 05000423/2015301 ML15078A430 ML15243A191 ML15281A139 ML15281A140 ML15281A141 ML15281A142 ML15281A143 ML15281A144 ML15281A146 ML15281A147 ML15281A148 ML15357A489
References
ES-401-9, TAC U01903
Download: ML15281A141 (6)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 1

H 3

N s

2 H

3 B

s 3

H 2

N s

4 H

3 N

s 5

H 2

B s

6 F

2 N

s 7

H 3

N s

Editorial stem changes: Provide conditions to indicate SIG are not available. Delete plant mode reference.

8 F

3 B

s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6.

Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7.

Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 9

F 2

B E

Minor editorial stem change to delete redundant statements.

10 F

2 B

s 11 H

3 N

E Minor editorial change to change the tense in the stem and answer choices.

12 H

2 M

s 13 H

3 B

s 14 H

2 B

E Minor editorial change to stem and answer choices to reduce the question from two parts to one.

15 H

3 B

s 16 H

3 N

N u

KAMM; Revise Q to have the plant in ECA-1.1 17 F

2 B

s 18 H

2 N

E Minor editorial change to stem to add "minimum" time requirement for the bus transfer time. Also editorial stem change to delete redundant statements.

19 F

2 B

s 20 H

3 N

s 21 F

2 B

s 22 F

2 B

s 23 H

3 B

E Minor editorial change to stem and answer choices to reduce the question from two parts to one.

24 H

3 M

s 25 H

2 M

E Minor editorial change to delete redundant statements and change "would" to "will" in the stem.

26 H

2 B

E Minor editorial stem change to delete redundant statements.

27 H

3 M

s 28 F

2 B

s

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KJA Only 29 F

2 N

E Minor editorial change to stem to delete redundant statements and associate the question with appropriate plant reference procedure.

30 H

2 x

N u

'B' and 'D' second part of the answer choices are not plausible distractors; replace the plausible distractors with new ones.

31 H

3 N

s 32 H

2 N

s 33 H

2 N

N u

KJA MM; redraw based topic not part of normal plant specific evaluation.

34 F

2 B

E Editorial stem change to remove breaker specific reference.

35 H

2 B

s 36 H

2 B

E Editorial stem change to add time reference, and enhance answer choices c and d to tail pipe temperature scale.

37 F

2 B

E Make 'C' answer choice more plausible.

38 H

3 N

s 39 F

3 B

E Editorial stem change to add reference document.

40 N

s 41 F

2 B

s 42 F

2 B

E Editorial stem change to add initial plant conditions in stem, and remove valve ids in answer choices to increase LOO.

43 F

2 B

s 44 H

2 B

E Editorial stem change to remove "if at all".

45 H

2 N

E Editorial stem change to add timeframe reference.

ES-401 4

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 46 F

2 B

E Minor editorial stem change to delete redundant statements.

47 H

3 N

s 48 H

2 N

s 49 H

2 B

s 50 F

2 B

E Minor editorial stem change to better match KIA.

51 H

2 B

s 52 F

2 N

E Minor editorial stem change to delete unnecessary statements.

53 F

2 B

s 54 F

2 B

s 55 F

2 B

E Minor editorial stem change to delete redundant statements.

56 H

2 B

E Minor editorial stem change to lower case words.

57 H

2 B

s 58 H

2 N

E Minor editorial stem change to provide indication that annunciator "just" came in to distinguish hi vs hihi setting.

59 F

2 N

s 60 F

2 B

s 61 F

1 B

u LOO - 1; Develop New Q per KIA.

62 H

2 B

E Minor editorial stem change to remove "likely" from stem.

63 H

3 N

s 64 F

2 N

s 65 F

2 B

E Replace answer choice 'B' with more plausible distractor.

ES-401 5

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5}

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 66 F

2 B

s 67 F

2 B

s 68 F

2 N

N u

KIA MM; Replace Q with new.

69 F

2 N

N u

KIA MM; reselect new KIA based on topic not part of normal job function.

70 H

2 B

E Minor editorial stem change to add Pzr solid parameters.

71 F

2 B

s 72 F

2 N

B u

KIA MM; Develop New Q per KIA.

73 H

2 N

s 74 H

2 B

E Minor editorial stem change to add timeframe and replace 'B' plausible distractor.

75 F

2 B

s 76 H

3 y

B s

77 H

3 y

B E

Editorial: Make 'B' more plausible; 530 psig; delete RCP trip criteria 78 H

3 y

N s

Editorial: Change the stem question to make it SRO related 79 H

3 y

B E

Editorial: B answer choice - change grammar to "is not" 80 H

4 y

N E

Add output bkr closure to part C & D to make it more plausible 81 H

3 y

B E

May need to evaluate to remove 3rd bullet or come up with another way of saying manual start of Charging pump was unsuccessful 82 H

3 y

B s

83 H

3 y

B s

84 H

3 y

B s

85 H

3 N

y N

u KIA MM; the Q does not test inadvertent CIA. Replace Q.

ES-401 6

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 86 H

3 y

B s

Editorial - Add KIA match info related to CVCS to the explanation.

87 H

3 y

B s

88 H

3 y

M s

89 F

3 y

N E

Add word "only" to choice 'C' 90 F

2 y

B E

Add the noun name for 3LWS-RE70, if needed.

91 H

3 y

N E

Editorial; Match A & B wording to match description of "P-A" 92 H

3 y

N s

93 H

3 y

N E

Recommend not provide table 3.7.3 and change the answer choices C & D to match A & B with 25.5% power to make it more plausible 94 H

3 y

B E

Add AFW status in the stem. Change feed setpoint to 21 % (the procedure refers to 21%); delete the first word of ALL answer choices (Attempt & Immediately) 95 F

2 y

B E

LOW LOD; see if that can be enhanced or be replaced.

96 F

2 y

N s

97 H

3 y

M E

Delete in stem "The SM will notify RE" (2"d bullet) to make C the only correct answer 98 F

3 N

y B

u KIA MM; Replace Q.

99 H

3 y

B s

100 F

2 y

N E

Editorial - Change part 2 of 'C' to "manually discharge the C02 system" to balance out the answer choices.

ES-401 Written Examination Review Worksheet Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 1

H 3

N s

2 H

3 B

s 3

H 2

N s

4 H

3 N

s 5

H 2

B s

6 F

2 N

s 7

H 3

N s

Editorial stem changes: Provide conditions to indicate SIG are not available. Delete plant mode reference.

8 F

3 B

s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6.

Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7.

Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 9

F 2

B E

Minor editorial stem change to delete redundant statements.

10 F

2 B

s 11 H

3 N

E Minor editorial change to change the tense in the stem and answer choices.

12 H

2 M

s 13 H

3 B

s 14 H

2 B

E Minor editorial change to stem and answer choices to reduce the question from two parts to one.

15 H

3 B

s 16 H

3 N

N u

KAMM; Revise Q to have the plant in ECA-1.1 17 F

2 B

s 18 H

2 N

E Minor editorial change to stem to add "minimum" time requirement for the bus transfer time. Also editorial stem change to delete redundant statements.

19 F

2 B

s 20 H

3 N

s 21 F

2 B

s 22 F

2 B

s 23 H

3 B

E Minor editorial change to stem and answer choices to reduce the question from two parts to one.

24 H

3 M

s 25 H

2 M

E Minor editorial change to delete redundant statements and change "would" to "will" in the stem.

26 H

2 B

E Minor editorial stem change to delete redundant statements.

27 H

3 M

s 28 F

2 B

s

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KJA Only 29 F

2 N

E Minor editorial change to stem to delete redundant statements and associate the question with appropriate plant reference procedure.

30 H

2 x

N u

'B' and 'D' second part of the answer choices are not plausible distractors; replace the plausible distractors with new ones.

31 H

3 N

s 32 H

2 N

s 33 H

2 N

N u

KJA MM; redraw based topic not part of normal plant specific evaluation.

34 F

2 B

E Editorial stem change to remove breaker specific reference.

35 H

2 B

s 36 H

2 B

E Editorial stem change to add time reference, and enhance answer choices c and d to tail pipe temperature scale.

37 F

2 B

E Make 'C' answer choice more plausible.

38 H

3 N

s 39 F

3 B

E Editorial stem change to add reference document.

40 N

s 41 F

2 B

s 42 F

2 B

E Editorial stem change to add initial plant conditions in stem, and remove valve ids in answer choices to increase LOO.

43 F

2 B

s 44 H

2 B

E Editorial stem change to remove "if at all".

45 H

2 N

E Editorial stem change to add timeframe reference.

ES-401 4

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 46 F

2 B

E Minor editorial stem change to delete redundant statements.

47 H

3 N

s 48 H

2 N

s 49 H

2 B

s 50 F

2 B

E Minor editorial stem change to better match KIA.

51 H

2 B

s 52 F

2 N

E Minor editorial stem change to delete unnecessary statements.

53 F

2 B

s 54 F

2 B

s 55 F

2 B

E Minor editorial stem change to delete redundant statements.

56 H

2 B

E Minor editorial stem change to lower case words.

57 H

2 B

s 58 H

2 N

E Minor editorial stem change to provide indication that annunciator "just" came in to distinguish hi vs hihi setting.

59 F

2 N

s 60 F

2 B

s 61 F

1 B

u LOO - 1; Develop New Q per KIA.

62 H

2 B

E Minor editorial stem change to remove "likely" from stem.

63 H

3 N

s 64 F

2 N

s 65 F

2 B

E Replace answer choice 'B' with more plausible distractor.

ES-401 5

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5}

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 66 F

2 B

s 67 F

2 B

s 68 F

2 N

N u

KIA MM; Replace Q with new.

69 F

2 N

N u

KIA MM; reselect new KIA based on topic not part of normal job function.

70 H

2 B

E Minor editorial stem change to add Pzr solid parameters.

71 F

2 B

s 72 F

2 N

B u

KIA MM; Develop New Q per KIA.

73 H

2 N

s 74 H

2 B

E Minor editorial stem change to add timeframe and replace 'B' plausible distractor.

75 F

2 B

s 76 H

3 y

B s

77 H

3 y

B E

Editorial: Make 'B' more plausible; 530 psig; delete RCP trip criteria 78 H

3 y

N s

Editorial: Change the stem question to make it SRO related 79 H

3 y

B E

Editorial: B answer choice - change grammar to "is not" 80 H

4 y

N E

Add output bkr closure to part C & D to make it more plausible 81 H

3 y

B E

May need to evaluate to remove 3rd bullet or come up with another way of saying manual start of Charging pump was unsuccessful 82 H

3 y

B s

83 H

3 y

B s

84 H

3 y

B s

85 H

3 N

y N

u KIA MM; the Q does not test inadvertent CIA. Replace Q.

ES-401 6

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-Q=

SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 86 H

3 y

B s

Editorial - Add KIA match info related to CVCS to the explanation.

87 H

3 y

B s

88 H

3 y

M s

89 F

3 y

N E

Add word "only" to choice 'C' 90 F

2 y

B E

Add the noun name for 3LWS-RE70, if needed.

91 H

3 y

N E

Editorial; Match A & B wording to match description of "P-A" 92 H

3 y

N s

93 H

3 y

N E

Recommend not provide table 3.7.3 and change the answer choices C & D to match A & B with 25.5% power to make it more plausible 94 H

3 y

B E

Add AFW status in the stem. Change feed setpoint to 21 % (the procedure refers to 21%); delete the first word of ALL answer choices (Attempt & Immediately) 95 F

2 y

B E

LOW LOD; see if that can be enhanced or be replaced.

96 F

2 y

N s

97 H

3 y

M E

Delete in stem "The SM will notify RE" (2"d bullet) to make C the only correct answer 98 F

3 N

y B

u KIA MM; Replace Q.

99 H

3 y

B s

100 F

2 y

N E

Editorial - Change part 2 of 'C' to "manually discharge the C02 system" to balance out the answer choices.