ML16085A260

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML16085A260
Person / Time
Site:	Nine Mile Point
Issue date:	12/22/2015
From:	Brian Fuller Exelon Generation Co
To:	Isham P Exelon Generation Co
Shared Package
ML15135A302	List: IR 05000410/2015302 ML15166A568 ML15266A017 ML15328A482 ML16020A444 ML16020A446 ML16085A255 ML16085A259 ML16085A260 ML16085A261 ML16085A262
References
TAC U01920
Download: ML16085A260 (6)

Text

ES-401 Nine Mile Point Unit 2 - 2015 Written Examination Review Worksheet Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 1

F 2

y N

B s

2010 NRG 2

F 3

y N

N s

3 F

3 y

N B

s 2010 NRG 4

H 3

y N

N s

5 H

y N

N s

6 H

3 y

N N

s 7

F 2

y N

B E

2009 NRG ENHANCEMENT TO RAISE LOO 8

H 3

y N

N s

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6.

Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7.

Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 9

H 3

y N

N s

10 H

3 y

N N

s EDITORIAL ENHANCEMENT 11 H

2 y

N B

s 12 H

y N

N s

13 H

3 y

N B

s 2012 NRG 14 H

y N

B u

REPLACE KIA, OVERLAP WITH JPM #7 15 H

2 y

N N

s 16 H

2 y

N N

s 17 F

2 y

N N

s 18 F

2 x

y N

M E

ENHANCED DISTRACTER 'C' 19 H

3 y

N B

s 20 H

3 y

N N

s 21 H

2 y

N B

E CORRECTED IMPLAUSIBLE DISTRACTER 22 H

3 x

y N

N E

ENHANCED DISTRACTER 23 H

3 y

N N

E ENHANCED STEM TO PRECLUDE 2 CORRECT ANSWERS 24 F

3 y

N B

s 2013 NRG 25 F

3 y

N N

s 26 F

2 y

N M

s 27 F

3 y

N B

s 28 H

4 y

N N

s

ES-401 3

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 29 F

3 y

N B

s 2012 NRG 30 H

3 y

N B

s 2005 NRG 31 F

3 y

N B

s 32 F

2 x

y N

B E

ENHANCED DISTRACTER 33 F

3 y

N B

s 2009 NRG 34 H

3 y

N N

s 35 H

2 x

y N

B s

2010 NRG IMPROVED JUSTIFICATIONS 36 H

3 y

N B

E 2010 NRG 37 H

3 y

N N

s 38 H

3 y

N N

s 39 F

2 x

y N

N E

CHANGED DISTRACTER 'B' 40 F

4 y

N B

E CLEANED UP CAPABILITY CURVE 41 F

3 y

N B

s 42 F

3 x

y N

B E

ENHANCED DISTRACTER 'C' 43 H

y N

B E

ENHANCED STEM 44 H

3 y

N N

s 45 H

3 y

N N

E ENHANCED STEM 46 F

2 y

N N

E ENHANCED ANSWER MATRIX 47 F

2 y

N B

E ENHANCED DISTRACTERS AND JUSTIFICATIONS 48 H

3 y

N N

s

ES-401 4

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 49 H

3 y

N N

s 50 F

2 y

N N

E MODIFY STEM. Requires SRO review 51 F

2 y

N N

E ENHANCED STEM 52 F

3 y

N B

s 53 F

3 y

N N

s 54 H

3 y

N N

s KIA JUSTIFICATION ENHANCED 55 H

3 y

N N

s 56 H

2 y

N N

s 57 F

3 y

N B

s 2010 NRG 58 H

2 y

N N

s 59 H

3 y

N B

E ENHANCED STEM 60 H

3 y

N B

s 2005 NRG 61 H

3 y

N B

s 62 H

2 x

y N

N E

MULTIPLE IMPLAUSIBLE DISTRACTERS 63 H

3 y

N N

s 64 H

3 y

N N

s 65 H

4 y

N N

s 66 F

2 y

N B

E ENHANCED STEM 67 F

3 y

N B

s 2010 NRG 68 H

3 y

N B

s 2010 NRG

ES-401 5

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 69 H

3 y

N N

s 70 F

3 y

N M

s 2010 NRG 71 H

2 y

N B

s 2009 NRG 72 H

3 y

N N

s 73 F

2 y

N N

s SENSITIVE INFO WITHHOLD Q 74 F

3 y

N N

s 75 F

2 y

N B

s 2012 NRG 76 H

3 y

y B

s 2014 NRG 77 H

3 y

y N

s 78 F

2 y

y B

s 79 H

3 y

y N

s 80 H

2 y

y B

E REARRANGED DISTRACTERS 81 F

3 y

y B

s 2009 NRG 82 H

2 y

y N

s 83 H

4 y

y N

s 84 H

2 y

y N

E ENHANCED STEM 85 H

3 y

y N

s 86 H

3 y

y N

s 87 H

2 y

y N

s 88 F

2 y

y N

s

ES-401 6

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 89 H

3 y

y B

s 90 F

2 y

y N

s 91 H

I y

N u

KIAMM RESAMPLED WROTE NEW Q 92 H

3 y

y N

E ENHANCED DISTRACTERS 93 H

2 y

y N

s 94 H

3

?

y N

E RESOLVE QUESTIONS FOR KIA MATCH 95 H

2 y

y B

s 96 H

3 y

y B

s 97 H

2 y

y N

y 98 H

2 y

y B

s 99 F

2 y

y N

s 100 H

2 y

y B

s

ES-401 Nine Mile Point Unit 2 - 2015 Written Examination Review Worksheet Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 1

F 2

y N

B s

2010 NRG 2

F 3

y N

N s

3 F

3 y

N B

s 2010 NRG 4

H 3

y N

N s

5 H

y N

N s

6 H

3 y

N N

s 7

F 2

y N

B E

2009 NRG ENHANCEMENT TO RAISE LOO 8

H 3

y N

N s

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6.

Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7.

Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOD (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 9

H 3

y N

N s

10 H

3 y

N N

s EDITORIAL ENHANCEMENT 11 H

2 y

N B

s 12 H

y N

N s

13 H

3 y

N B

s 2012 NRG 14 H

y N

B u

REPLACE KIA, OVERLAP WITH JPM #7 15 H

2 y

N N

s 16 H

2 y

N N

s 17 F

2 y

N N

s 18 F

2 x

y N

M E

ENHANCED DISTRACTER 'C' 19 H

3 y

N B

s 20 H

3 y

N N

s 21 H

2 y

N B

E CORRECTED IMPLAUSIBLE DISTRACTER 22 H

3 x

y N

N E

ENHANCED DISTRACTER 23 H

3 y

N N

E ENHANCED STEM TO PRECLUDE 2 CORRECT ANSWERS 24 F

3 y

N B

s 2013 NRG 25 F

3 y

N N

s 26 F

2 y

N M

s 27 F

3 y

N B

s 28 H

4 y

N N

s

ES-401 3

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 29 F

3 y

N B

s 2012 NRG 30 H

3 y

N B

s 2005 NRG 31 F

3 y

N B

s 32 F

2 x

y N

B E

ENHANCED DISTRACTER 33 F

3 y

N B

s 2009 NRG 34 H

3 y

N N

s 35 H

2 x

y N

B s

2010 NRG IMPROVED JUSTIFICATIONS 36 H

3 y

N B

E 2010 NRG 37 H

3 y

N N

s 38 H

3 y

N N

s 39 F

2 x

y N

N E

CHANGED DISTRACTER 'B' 40 F

4 y

N B

E CLEANED UP CAPABILITY CURVE 41 F

3 y

N B

s 42 F

3 x

y N

B E

ENHANCED DISTRACTER 'C' 43 H

y N

B E

ENHANCED STEM 44 H

3 y

N N

s 45 H

3 y

N N

E ENHANCED STEM 46 F

2 y

N N

E ENHANCED ANSWER MATRIX 47 F

2 y

N B

E ENHANCED DISTRACTERS AND JUSTIFICATIONS 48 H

3 y

N N

s

ES-401 4

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 49 H

3 y

N N

s 50 F

2 y

N N

E MODIFY STEM. Requires SRO review 51 F

2 y

N N

E ENHANCED STEM 52 F

3 y

N B

s 53 F

3 y

N N

s 54 H

3 y

N N

s KIA JUSTIFICATION ENHANCED 55 H

3 y

N N

s 56 H

2 y

N N

s 57 F

3 y

N B

s 2010 NRG 58 H

2 y

N N

s 59 H

3 y

N B

E ENHANCED STEM 60 H

3 y

N B

s 2005 NRG 61 H

3 y

N B

s 62 H

2 x

y N

N E

MULTIPLE IMPLAUSIBLE DISTRACTERS 63 H

3 y

N N

s 64 H

3 y

N N

s 65 H

4 y

N N

s 66 F

2 y

N B

E ENHANCED STEM 67 F

3 y

N B

s 2010 NRG 68 H

3 y

N B

s 2010 NRG

ES-401 5

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. I Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 69 H

3 y

N N

s 70 F

3 y

N M

s 2010 NRG 71 H

2 y

N B

s 2009 NRG 72 H

3 y

N N

s 73 F

2 y

N N

s SENSITIVE INFO WITHHOLD Q 74 F

3 y

N N

s 75 F

2 y

N B

s 2012 NRG 76 H

3 y

y B

s 2014 NRG 77 H

3 y

y N

s 78 F

2 y

y B

s 79 H

3 y

y N

s 80 H

2 y

y B

E REARRANGED DISTRACTERS 81 F

3 y

y B

s 2009 NRG 82 H

2 y

y N

s 83 H

4 y

y N

s 84 H

2 y

y N

E ENHANCED STEM 85 H

3 y

y N

s 86 H

3 y

y N

s 87 H

2 y

y N

s 88 F

2 y

y N

s

ES-401 6

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

8.

Q#

LOK LOO (F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-0= SRO B/M/N U/E/S Explanation Focus Dist.

Link units ward KIA Only 89 H

3 y

y B

s 90 F

2 y

y N

s 91 H

I y

N u

KIAMM RESAMPLED WROTE NEW Q 92 H

3 y

y N

E ENHANCED DISTRACTERS 93 H

2 y

y N

s 94 H

3

?

y N

E RESOLVE QUESTIONS FOR KIA MATCH 95 H

2 y

y B

s 96 H

3 y

y B

s 97 H

2 y

y N

y 98 H

2 y

y B

s 99 F

2 y

y N

s 100 H

2 y

y B

s