ML17114A375

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML17114A375
Person / Time
Site:	Ginna
Issue date:	03/09/2017
From:	Exelon Generation Co
To:	NRC Region 1
Shared Package
ML162010A441	List: ML17114A375
References
U01943
Download: ML17114A375 (6)

Text

ES-401 Ginna Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 2 N s 2 F 2 B s 3 F 2.5 B s 4 H 1 M u Low discriminatory value. Replace or rework.

5 H 3 x B E "would" vs "could" in part 1 6 H 3 N s 7 H 3 B s 8 F 2 B s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check theThe appropriate box if a job content error is identified:

question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 H 2.5 B s 10 F 2 x B s Add annunciators to meet KIA.

11 F 2 B s 12 H 3.5 D B E Replace/modify D.

13 H 3 B s Add KA justification.

14 H 2.5 B,D B u Why would actual RIL change? Modify Q.

15 F 2 x N E Modify first sentence to avoid providing information.

16 H 2.5 x -B E Modify to ensure only one correct answer.

17 H 3.5 B s 18 F 2.5 x B E Add a temperature value to another distractor.

19 H 3 B s 20 F 3 x B E Clarify status of Bus 13 21 H 3 N s 22 F 3 x N E Explain 6 minutes or provide another realistic value.

23 F 2 B s 24 F 2 x B E Clarify wording in distractors to ensure only one correct answer.

25 F 2 B s 26 H 2.5 x N E Change 4th bullet to "less than 2 psig" vs 5. Provide KA justification.

27 H 2 B s 28 H 2.5 x B E Move last bullet into stem.

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 M s 30 H 3.5 M s Provide justification for RO level Q.

31 H 3 x B E Specify normal system/channel lineup.

32 H 3 B s 33 F 3 B B E Modify/replace choice B.

34 F 3 x N E Add IAW background document.

35 H 2.5 B s 36 H 2.5 x N E Replace Unit Supervisor with crew in part 1.

37 H 3 x N E Replace "lower power" in A&B with "approximately 65%."

38 H 3.5 x N E Replace "lists" with "describes".

39 F 3 x B E Specify the controlling channel.

40 H 2.5 N s 41 F 1 B u LOD 1 - Fix/replace 42 H 2.5 B s 43 F 2 B s 44 H 3 x N E Change "lists" to "describes" 45 F 2.5 N s 46 F 2.5 B s 47 H 2.5 N s Aj8 H 2 - cB s Previous NRG exam Q.

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 H 2.5 x N u KAMM.

50 H 2.5 B s 51 F 2.5 B s 52 F 2 x N E Replace "cloning" with procedure wording.

53 F 2.5 B s 54 F 3 N s 55 H 3.5 N s 56 F 2 B s 57 H 2.5 B s 58 H 2.5 N s Clarify justifications for A & D.

59 F 2.5 x B E Stem format - combine wording.

60 H 2 B s 61 F 2.5 x B E Modify data or clarify plausibility statement for D.

62 H 2.5 B s Provide KIA justification.

63 H 3 N s 64 F 2 B s 65 F 3 B s Clarify justification for B regarding integrity.

66 H 3 B s 67 F 2.5 N s

,,,!38 H 3 B s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 69 H 3 x N E Replace "guidance" with "mitigative actions".

70 H 3 B s 71 H 2.5 x B E Add "is" to 3rd bullet.

72 H 3 N s 73 F 2 B s 74 F 2.5 B s 75 F 1 N u Replace/modify/clarify.

76 F 3 B u Not SRO only. Replace/re-work. Justify format to meet b(5).

77 H 2.5 D N E Distractor "D" not plausible for a GE. Replace.

78 H 3 N s 79 H 2.5 x B u Not SRO only. Change answer format. Re-work.

80 H 3 x N E Don't provide TS 3.7.7. Modify distractors to include "immediately."

81 H 3.5 N s 82 F 3 N E Add comma to 3rd bullet.

83 F 3.5 N s Provide KIA justification.

84 H 3.5 x B E Add panel locations to distractors or provide justification.

85 H 3.5 M s Add justification for SRO only.

86 F 3.5 B s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other - . 6. I~ 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 87 H 3 x B N E Distractor B not plausible. Add loss of Bus 11A. Change B.

88 H 3 N s 89 H 2.5 N s 90 H 3 x M E Delete 3rd bullet.

91 H 3 x N u Can answer with system knowledge. Replace/re-work/reselect.

92 F 2 x N E Insert "to a minimum of' to ensure clarity.

93 H 2 N s 94 H 3 N s 95 H 2 N s 96 H 2.5 N s 97 F 3 B s 98 H 3 N s 99 H 3 x N u A is possible 2"d answer. Re-work/replace.

100 H 4 N s