ML17261B168

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML17261B168
Person / Time
Site:	Oyster Creek
Issue date:	02/10/2016
From:	Brian Fuller Operations Branch I
To:	Exelon Nuclear
Shared Package
ML15219A259	List: IR 05000219/2016301 ML15224B314 ML16026A581 ML16064A171 ML16064A177 ML17216A172 ML17216A174 ML17261B060 ML17261B165 ML17261B167 ML17261B168 ML17261B169
References
TAC U01929
Download: ML17261B168 (6)

Text

ES-401 Oyster Creek - 2016 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- 0= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 2 y N M s Enhanced distractors 2 H 3 y N B s 3 M 2 y N N s Editorial 4 H 3 y N M s 5 M 3 y N N s 6 H 3 y N M s 7 M 2 y N N s 8 M 3 y N N s Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

.. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- 0= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 H 3 y N N s 10 F 3 N N N u KIA MM, rewrote stem, changed distracters 11 H 2 y N N s Editorial 12 F 3 y N B s 13 H 3 x y N N u Reworded stem 14 F 3 y N N s 15 F 2 y N N s 16 F 2 y N B s 17 H 3 y N B s 18 H 2 y N N s 19 H 3 y N N s 20 H 3 y N B s 21 H 2 x y N N u Three implausible distracters, reworded stem, changed distracter D 22 H 3 y N N s Modified stem 23 F 3 y N B s 24 H 3 y N B s Editorial 25 H 3 y N N s 26 H 2 y N B s Editorial 27 F 5 y N M u High LOO, reworded stem 28 H 4 y N N s Editorial

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- 0= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 F 3 y N B s 30 H 3 y N B s 31 H 3 y N N s 32 H 2 N N N u KIA MM, resampled, wrote new Q 33 H 2 y N B s Graph reading, matches KIA 34 H 3 y N M s 35 H 2 y N N s 36 H 3 y N B s 37 F 3 N N N u KIA MM, sampled new KIA, wrote new Q 38 H 3 y N N u Editorial, no correct answer; wrote new Q 39 H 2 y N N s 40 H 4 y N N s 41 H 3 y N B E Enhanced stem 42 H 3 y N B s 43 H y N B s 44 H 3 y N B s 45 H 2 y N B s 46 F 2 y N B E Editorial, 47 F 2 y N B s 48 H 3 y N B s

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5.0ther 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- 0= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 H 3 y N 8 s 50 H 2 y N 8 s 51 F 2 y N M E Editorial 52 F 3 y N 8 s 53 H 3 y N 8 s 54 F 3 y N N s 55 F 3 y N 8 E Editorial 56 F 2 y N N s 57 F 3 y N 8 E One implausible distracter, wrote new distracter 58 F 2 y N 8 E Editorial 59 H 3 y N M s 60 F 3 y N 8 s 61 F 3 y N 8 s 62 H 2 x y N 8 u Two implausible distracters, rewrote distracters 63 H 3 y N N E Stem modifications to increase plausibility 64 H 3 y N 8 s 65 H 4 y N B E One implausible distracter, wrote new distracter 66 F 2 y N N s Editorial 67 F 3 y N N s Editorial 68 F 3 y N B s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- 0= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 69 H 1 y N N u Low LOO, replaced with new Q 70 H 3 y N B s 71 F 2 y N B E Editorial 72 F 3 y N B s 73 H 2 y N B s 74 F 3 y N N s 75 F 2 y N N E Overlap w/ op test, replaced q 76 H 3 y y B s 77 H 3 y y N s 78 H 2 y y N E One implausible distracter, editorial; rewrote Q 79 F 3 y y N s 80 F 2 y y N s 81 H 3 y y B s 82 H 2 y y B u NO CORRECT ANSWER OR TWO CORRECT ANSWERS, REWROTE DISTRACTERS 83 H 4 y y N s 84 H 2 y y N s 85 H 3 y y N s 86 H 3 y y M s 87 H 2 y y B s Editorial 88 H 2 y y B s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

0# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- 0= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 89 H 3 y y N s 90 F 2 y y B s 91 H 3 y y N s 92 H 3 x y y B E Editorial 93 H 2 y y N s 94 F 3 y y B s 95 F 2 y N N u Not SRO only; rewrote 0 96 H 3 y y N E Enhanced distracters 97 H 2 y y B u Overlap with 0#78; wrote new 0 98 H 2 y y B s 99 F 2 y y B s 100 H 2 y y M s