ML12353A160

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML12353A160
Person / Time
Site:	Salem
Issue date:	11/20/2012
From:	Operations Branch I
To:	Public Service Enterprise Group
Jackson D
References
TAC U01859
Download: ML12353A160 (6)

Text

ES-401 SALEM Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. B.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #/ Back- Q= SRO B/M/N UtE/S Explanation Focus Dist. Link units ward KIA Only 1 H 2 N B U KlAMM 2 F 1 X Y N U LOW LOD AND TWO IMPLAUSIBLE DISTRACTERS 3 H 3 Y B S 4 H 3 Y B S 5 H 3 Y N S FIX TYPO inA 6 F 2 Y B S Add details to Stem; FIX TYPO in 0 7 H 2 Y M S B H 3 Y M E ADD TIME AFTER SI SIGNAL CLEARED Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F}undamental or (H}igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

• • The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

4.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

Check the appropriate box if a job content error is identified:

• • The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

• • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-on/y (KIA and license level mismatches are unacceptable).

6. Enter question source: (B}ank, (M}odified, or (N}ew. Check that (M}odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer'S judgment, is the question as written (U}nsatisfactory (requiring repair or replacement), in need of (E}ditorial enhancement, or (S}atisfactory?

B. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N UtE/S Explanation Focus Dist. Link units ward KIA Only 9 H 4 Y B S 10 H 3 Y B E Add which AOP to Stem 11 F 3 Y B S REMOVE "TOTAL" FROM STEM 12 H 3 X Y N E ONE IMPLAUSIBLE DISTRACTOR, REWORD DISTRACTOR "C" 13 H 2 Y B S Fix typo in stem 14 F 2 Y B S 15 F 3 Y N S EDITORIAL CORRECTIONS in C 16 H 1 Y B U LOW LOD, Change B&D 17 F 2 Y B S 18 F 2 X Y N E ONE IMPLAUSIBLE DISTRACTOR AND FIX TYPO 19 H 3 X Y B E ONE IMPLAUSIBLE DISTRACTOR D 20 H 2 Y N S 21 F 2 Y B S 22 F 2 X Y B E REWORD DISTRACTOR "B" 23 H 2 X Y B E REWRITE DISTRACTOR "D" 24 H 2 Y B S 25 H 2 X Y B E REWORD DISTRACTOR "A" 26 H 3 X Y B S RESOLVE DISCREPANCY DISTRACTOR "C" 27 F 2 Y B S 28 H 2 Y B S

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N UtE/S Explanation Focus Dist. Link units ward KIA Only 29 H 3 Y N S 30 F 2 Y B S 31 F 2 Y B E A&D MAKE 2X2 FORMAT 32 H 2 Y B S ADD" lAW PROCEDURE XXX" TO STEM; FIX TYPO 33 F 2 Y N S EDITORIAL FIX 34 F 2 X Y B U Two implausible distractors 35 H 2 X Y N E TWO IMPLAUSIBLE DISTRACTORS ; replace B 36 F 3 N S 37 F 1 B U LOW LOD, GFE KNOWLEDGE ONLY 38 H 2 M S 39 H 3 N E REWORD STEM, check value in C 40 H 2 B E DELETE "IS" FROM STEM. REPLACE "PREDICT" WITH 'WHAT IS" 41 F 2 N E Provide appropriate valve numbers vice actuations 42 F 2 X N E ONE IMPLAUSIBLE DISTRACTOR CORRECT TYPO IN STEM and A

43 H 2 X N N U KIA MM; TWO IMPLAUSIBLE DISTRACTORS 44 H 2 N E Change superheat to ICC 45 H 2 B S 46 H 4 N S 47 F 2 N S

~18 F 2 B S

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 H 2 Y B S 50 H 2 Y B S 51 H 2 X Y N E ONE IMPLAUSIBLE DISTRACTOR 'A' 52 F 2 N S 53 H 3 N S 54 F 2 N S 55 H 2 N S 56 H 2 B S 57 F 2 B S 58 F 1 N U LOD 1 59 F 3 N S 60 H 2 N S 61 H 2 B S Possible modify 'A' 62 F 2 B S 63 F 2 B S 64 H 2 N S 65 H 2 N B U KlAMM 66 F 2 N S 67 F 2 B S q?,8 H 3 B S

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N UlE/S Explanation Focus Dist. Link units ward KIA Only 69 F 2 B S 70 H 2 N S 71 F 2 M E REWORD DISTRACTOR C & D 72 F 2 X B E ONE IMPLAUSIBLE DISTRACTOR 'C' 73 F 3 M S 74 F 2 B S 75 F 4 B S Editorial 76 H 2 Y N B U AOP Entry condition is only knowledge needed.

77 H 2 Y Y N S Editorial fixes 78 H 2 Y Y N S Correct Typo 79 H 4 Y Y B E Resolve HUR question 80 F 3 Y Y N E Editorial fixes 81 H 1 Y N M U Low LOD. Knowledge of mitigation strategy required 82 H 1 N Y N U KIA MM; Low LOD 83 H 2 Y Y B S 84 F 2 X Y Y B E D implausible 85 H 3 Y Y N E Needed darifications 86 H 2 Y Y N S

ES-401 6 Form ES-401-9 II r I i

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

00 LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #I Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 87 H 2 X Y Y B E Modify Q to increase diSCriminatory value.

88 F 2 Y Y M S Fix typo 89 H 2 Y Y B S Fix typo 90 H 2 X Y Y N E Modify Q to increase discriminatory value.

91 F 2 X Y Y B U A&C implausible 92 H 2 X Y Y N E Fix typo; Modify distractor B&C 93 F 2 Y Y B S Fix typo 94 F 3 X Y Y B E Resolve issues with C 95 H 2 N Y M U KlAMM 96 F 2 X Y Y N E Low LOD as written; Change B to some other site manager 97 H 2 Y Y B E Modify stem and add TS to distractor A 98 H 2 X Y Y B E Replace distractor D 99 F 3 X Y Y B E Capitalize "Authorize" in stem 100 F 2 Y Y N E Replace retract with "Downgrade'