ML16081A159

From kanterella
Jump to navigation Jump to search

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)
ML16081A159
Person / Time
Site: FitzPatrick Constellation icon.png
Issue date: 03/03/2016
From: David Silk
Operations Branch I
To:
FitzPatrick, NRC
Shared Package
ML15195A060 List:
References
TAC U01932
Download: ML16081A159 (6)
v • d • e


Text

ES-401 FITZPATRICK Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 F 2 N s F vs H 2 H 2 N s 3 F 1 x N u Replace Q.

4 F 1 N u Re-work Q.

5 F 3 N s 6 H 2 x B E Need to have an ADS malfunction in stem.

7 F 2 B N E Replace B 8 F 2 x N E Specify that operator action has been taken Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult} rating scale (questions in the 2 - 4 range are acceptable}.
3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent; more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable}.
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 B s 10 F 2 N s 11 F 2 N s 12 H 2 x M u Remove SRM shorting links. Replace if not procedurally driven.

13 H 2 x N E Typo in first bullet.

14 F 2 N s 15 H 3 M s 16 H 2 B s 17 F 2 x N E Need to have SRV auto action in stem.

18 H 2 x N E Have EOG "C" ready for a quarterly test 19 H 2 B s 20 F 2 N s 21 F 2 N s 22 H 3 N s 23 H 2 B,C N u Replace B&C.

24 F 2 x D N E Reword for clarity (available). Replace D.

25 H 3 N s 26 F 2 B s 27 F 2 N s 28 F 2 x N E Reformat distractors.

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 1 N u Rework Q to increase LOD.

30 F 3 c B E Make 34 vs 33.

31 H 2 B s 32 H 2 B s 33 F 2 N s 34 H 3 x N E Re-format Q for clarity.

35 H 3 x N E Re-word I re-format 36 F 3 N s 37 H 2 M s 38 H 2 B s 39 H 2 B s 40 H 3 x B E Typo - reactor vs Reactor (Do for entire exam.)

41 H 2 B s 42 H 3 x B E 1065 vs 1045 43 F 2 N s 44 H 2 M s 45 H 2 N s 46 F 2 B s 47 H 2 N s Aj8 H 2 x N E Reformat distractors for clarity.

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 F 2 N s 50 H 2 B s 51 F 1 N u Re-work Q to be more discriminating.

52 H 2 N s 53 H 2 M s 54 F 2 M s 55 F 2 x N E Reformat to eliminate 2"d column.

56 H 2 B s 57 H 2 M s 58 H 2 N s 59 H 2 M s 60 H 2 B s 61 H 3 A M E Replace distractor A.

62 F 2 N s 63 H 2 B s 64 H 2 B s 65 F 2 N s 66 F 2 B s 67 H 2 - N s laP8 F 3 M s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 69 H 3 M s 70 H 2 c B E Replace MSIVs (C) with sample valves.

71 H 1 B u GET. Reselect.

72 F 2 x M E Add second symbol to Q.

73 F 1 x N u Provide parameters values and ask what EOPs would be entered.

74 H 2 x N E Delete phrase from Q.

75 H 2 B s H=43 76 H 3 x u Re-do distractors to contain actual procedural steps as choices.

77 H 2 x E Fix 2nd bullet - "slowly" 78 H 3 s 79 H 2 s Make it a 43(b)(2) Q vs (5) 80 H 3 s 81 H 2 s 82 F 2 s 83 H 3 s 84 H 2 x E Provide both Hot and cold EAL charts. Make it a 43(b)(7) Q vs (5) 85 H 2 s 86 H 2 s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 87 H 2 s 88 H 2 s 89 H 2 x u Re-work Q to make SRO-only 90 H 2 s 91 H 3 s 92 H 2 s 93 F 2 x E Re-word question to diagnose electrical status.

94 F 2 s 95 H 2 s 96 F 2 s 97 H 2 x E Re-work Q or reselect KA.

98 F 2 A E Replace A 99 H 2 s Provide Hot and Cold EALs 100 H 2 s

ES-401 FITZPATRICK Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 F 2 N s F vs H 2 H 2 N s 3 F 1 x N u Replace Q.

4 F 1 N u Re-work Q.

5 F 3 N s 6 H 2 x B E Need to have an ADS malfunction in stem.

7 F 2 B N E Replace B 8 F 2 x N E Specify that operator action has been taken Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult} rating scale (questions in the 2 - 4 range are acceptable}.
3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent; more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable}.
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 B s 10 F 2 N s 11 F 2 N s 12 H 2 x M u Remove SRM shorting links. Replace if not procedurally driven.

13 H 2 x N E Typo in first bullet.

14 F 2 N s 15 H 3 M s 16 H 2 B s 17 F 2 x N E Need to have SRV auto action in stem.

18 H 2 x N E Have EOG "C" ready for a quarterly test 19 H 2 B s 20 F 2 N s 21 F 2 N s 22 H 3 N s 23 H 2 B,C N u Replace B&C.

24 F 2 x D N E Reword for clarity (available). Replace D.

25 H 3 N s 26 F 2 B s 27 F 2 N s 28 F 2 x N E Reformat distractors.

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 1 N u Rework Q to increase LOD.

30 F 3 c B E Make 34 vs 33.

31 H 2 B s 32 H 2 B s 33 F 2 N s 34 H 3 x N E Re-format Q for clarity.

35 H 3 x N E Re-word I re-format 36 F 3 N s 37 H 2 M s 38 H 2 B s 39 H 2 B s 40 H 3 x B E Typo - reactor vs Reactor (Do for entire exam.)

41 H 2 B s 42 H 3 x B E 1065 vs 1045 43 F 2 N s 44 H 2 M s 45 H 2 N s 46 F 2 B s 47 H 2 N s Aj8 H 2 x N E Reformat distractors for clarity.

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 F 2 N s 50 H 2 B s 51 F 1 N u Re-work Q to be more discriminating.

52 H 2 N s 53 H 2 M s 54 F 2 M s 55 F 2 x N E Reformat to eliminate 2"d column.

56 H 2 B s 57 H 2 M s 58 H 2 N s 59 H 2 M s 60 H 2 B s 61 H 3 A M E Replace distractor A.

62 F 2 N s 63 H 2 B s 64 H 2 B s 65 F 2 N s 66 F 2 B s 67 H 2 - N s laP8 F 3 M s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 69 H 3 M s 70 H 2 c B E Replace MSIVs (C) with sample valves.

71 H 1 B u GET. Reselect.

72 F 2 x M E Add second symbol to Q.

73 F 1 x N u Provide parameters values and ask what EOPs would be entered.

74 H 2 x N E Delete phrase from Q.

75 H 2 B s H=43 76 H 3 x u Re-do distractors to contain actual procedural steps as choices.

77 H 2 x E Fix 2nd bullet - "slowly" 78 H 3 s 79 H 2 s Make it a 43(b)(2) Q vs (5) 80 H 3 s 81 H 2 s 82 F 2 s 83 H 3 s 84 H 2 x E Provide both Hot and cold EAL charts. Make it a 43(b)(7) Q vs (5) 85 H 2 s 86 H 2 s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 87 H 2 s 88 H 2 s 89 H 2 x u Re-work Q to make SRO-only 90 H 2 s 91 H 3 s 92 H 2 s 93 F 2 x E Re-word question to diagnose electrical status.

94 F 2 s 95 H 2 s 96 F 2 s 97 H 2 x E Re-work Q or reselect KA.

98 F 2 A E Replace A 99 H 2 s Provide Hot and Cold EALs 100 H 2 s

- -

Retrieved from "https://kanterella.com/w/index.php?title=ML16081A159&oldid=2444826"