ML15156A635

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML15156A635
Person / Time
Site:	Salem
Issue date:	11/24/2014
From:	David Silk Operations Branch I
To:	Public Service Enterprise Group
Shared Package
ML14174B408	List: IR 05000272/2014301 ML14191A180 ML14329A180 ML14337A574 ML14356A583 ML14356A584 ML14356A585 ML14356A586 ML15009A252 ML15009A280 ML15009A286 ML15009A311 ML15009A340 ML15020A701 ML15156A443 ML15156A635
References
TAC U01894
Download: ML15156A635 (6)

Text

ES-401 SALEM Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 4 x N E Insert "minimum required action" to eliminate possible subset.

2 H 3 x N E Move "The first RX trip signal was" to the beginning of distractors.

3 H 2 x M s H vs F. State that safety valve fails full open.

4 F 2 A M E Replace distractor A 5 H 2 x B E Reword question for clarity.

6 F 3 x N E Add "IAW ... "to the stem.

7 H 2 M s 8 F 2 B s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

.. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 H 3 M s 10 H 3 B s Clarify justification for PORV response.

11 H 2 N s H vs F.

12 F 2 x N E Fix wording in B (delete "is"). "Remove IAW Tech Spec."

13 F 2 B M E Replace B.

14 F 2 B s 15 F 2 N s 16 F 3 B s 17 F 4 x N E Move "only" in all choices for clarity.

18 H 3 B s 19 H 3 B s 20 H 2 B s Modify justification to explain KA match.

21 H 3 N s 22 H 3 B s 23 H 3 B s 24 H 2 M s 25 F 3 N s 26 F 2 B s 27 H 3 B s 28 H 2 B s

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 2 B s 30 H 3 N s 31 H 2 B s 32 H 4 N s 33 F 2 B s 34 H 2 B s 35 H 2 x N E Capitalize "unexpected" 36 H 2 N s 37 H 2 N s 38 F 2 A N E In A, swap 2PS3 & 2PS1.

39 H 3 B M E Make B 162 F (T-sat for 5 psia) and re-order choices.

40 H 2 N s 41 H 3 x N u H vs F. KAMM. What in the Q has to do with RPS? Replace Q.

42 H 3 C,D M u H vs F. Find different bases for C & D.

43 F 2 x A,B M u Reword question/distractors to elicit desired response.

44 F 3 N s F vs H.

45 F 2 N s 46 F 2 N s 47 F 3 N s F vs H.

~,.,18 F 2 B s

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 H 2 M s 50 F 2 x B,D N u Reformat stem. Replace demin water w/ boration source in B&D.

51 H 2 c M E Replace C. Fix D - 052 can help determine that D is correct.

52 H 2 M s 53 H 2 N s 54 H 2 x N E Delete manual actions in A,B,&D because they are not needed.

55 H 2 B s 56 H 2 B s 57 H 3 M s 58 F 1 B u Low LOD. Rework.

59 H 2 x N E Delete unnecessary words from C & D.

60 F 2 N s 61 H 2 N s H vs F.

62 H 2 x B u Edit stem to ensure KA match.

63 H 3 B s 64 H 2 B s 65 F 1 N u Replace choices with accident conditions.

66 F 3 D N E Delete D. Replace choices with Modes 3,4,5, & 6 respectively.

67 F 2 c M E Replace C.

bnfl,8 F 2 B s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 69 F 2 M s 70 F 2 x M E Use 2440# instead of 2485# to be more plausible.

71 H 3 M s 72 F 2 N s 73 F 3 x N E Modify B for clarity.

74 F 2 B s 75 F 2 B s 76 H 2 D N E Change D to NI fails low.

77 H 2 N E Add rod bottom light lit to stem. C&D add procedure names.

78 F 2 N s 79 F 2 N s 80 F 2 M s 81 H 2 N s 82 F 3 M s 83 H 3 N s 84 H 2 x B E Add "actions that will be directed by that procedure" to stem.

85 H 3 x M E Correct/add procedure names in choices.

86 F 1 B,D N u Replace Q.

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 87 H 2 x B E Add valve names in C&D.

88 H 2 M s 89 H 2 N s 90 H 3 x M E Delete air binding from stem.

91 H 2 M s 92 H 2 x N u Q can be answered from system knowledge. Replace/edit Q.

93 H 2 N s 94 H 2 x B E Add "Both monitors" to each choice. Fix justification for C.

95 H 2 x B E Change 2"d bullet to indicate that <50 F/ hr C/D rate is in effect.

96 F 2 x N u KAMM. Replace.

97 F 1 N u Low LOO. Replace or reselect.

98 F 2 x M E Replace Plant Manager with Ops Manager in C&D.

99 F 2 M s 100 F 2 B s