ML101620115
| ML101620115 | |
| Person / Time | |
|---|---|
| Site: | Oyster Creek |
| Issue date: | 03/08/2010 |
| From: | Exelon Nuclear |
| To: | D'Antonio J Operations Branch I |
| Hansell, S L, RI/DRS/OB, 610-337-5046 | |
| Shared Package | |
| ML092470040 | List: |
| References | |
| TAC U01771 | |
| Download: ML101620115 (18) | |
Text
S'IL '0 ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LaD (F/H)
(1-S)
Stem Cues TIF Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/M/N UlE/S Explanation Focus Oist.
Link units ward KIA Only 1
h 3
s 2
h 3
s 3
h 3
x U
Q is response to cooling loss, kla is determine cause f'C?'.::*,1 4
h 3
s 5
h 2
s 6
h 3
s 7
h 2
s 8
h 2
x U
Quite a reach to include "stm/fd mm" in this Q. More straight forward would be OK fP~"
Instructions
[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LaD) of each question using a 1 - S (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent. more information is needed, or too much needless information).
The stem or distractors contain cues (Le., clues, speCific determiners. phrasing. length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements the question has a valid KIA but, as written, is not operational in content}.
The question requires the recall of knowledge that too specific for the closed reference test mode (Le.. it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g.* panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
S.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B}ank, (M)odified. or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
- 7.
Based on the reviewer'S jUdgment, is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E)ditorial enhancement. or (S)atisfactory?
- 8.
At a minimum, explain any "U" ratings (e.g.. how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- /
Back-Q=
SRO B/M/N utE/S Explanation Focus Dist.
Link units ward KIA Only 9
h 1
U For distractors, give actions incl TS, no reference. DLO as is 10 h
3 e
Streamline distractor wording 11 h
2 U
Reference unnecessary 12 h
2 s
13 h
2 s
14 h
2 s
15 h
2 s
16 h
2 s
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 4.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 5.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 6.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible: single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check guestions that are sampled for conformance with the approved KIA and those that are designated SRO-on/y (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- /
Back-Q=
SRO B/M/N U/E/S Explanation Focus Dist.
Link units ward KIA Only 17 h
2 U
Too close to DLO - remove reference 18 h
2 s
19 h
2 s
20 h
2 s
21 f
2 s
22 h
2 U?
Remove reference not provided for review 23 h
3 S
24 h
3 U
Remove reference* giveaway Instructions
[Refer to Section D of ES*401 and Appendix B for additional information regarding each of the following concepts.]
- 7.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 8.
Enter the level of difficulty (LOD) of each question using a 1 5 (easy - difficult) rating scale (questions in the 2 4 range are acceptable).
- 9.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
S.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/M/N UlEtS Explanation Focus Dist.
Link units ward KIA Only 25 f
2 s
co_ ---~
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 10.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 11.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 4 range are acceptable).
- 12.
Check the appropriate box if a psychometric flaw is identified:
The slem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible dislractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (Blank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer'S judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement. or (S)atisfactory?
S.
At a minimum, explain any "un ratings (e.g., how the Appendix B psychometric attributes are not being met).
II- :>
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- /
Back-Q=
SRO B/M/N utE/S Explanation Focus Dist.
Link units ward KIA Only 1
H 2
S 2
H 3
S 3
F 2
S 4
F 1
U 5
H 2
S 6
H 3
S 7
F 2
S 8
F 2
S Instructions
[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 13.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 14.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 15.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Credo Partial Focus Dist.
9 H
2 10 F
2 11 H
3 12 F
2 13 H
3 14 H
2 15 H
3 16 H
3
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Job-Minutia
- /
Back-Q=
SRO B/M/N U/E/S Explanation Link units ward KIA Only S
S S
E Fix answer key wxplanation E
Fix answer key explanation S
S s
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.)
- 16.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 17.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 18.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any ;'U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- /
Back-Q=
SRO B/M/N utE/S Explanation Focus Dist.
Link units ward KIA Only 17 H
3 S
18 H
3 S
19 H
2 E
Q refers to srm 21, pic is srm 22 20 F
2 S
21 H
3 S
22 H
3 S
23 F
2 S
24 H
3 S
Instructions
[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 19.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 20.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 21.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent. more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but. as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer'S jUdgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement. or (S)atisfactory?
- 8.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/M/N U/E/S Explanation Focus Dist.
Link units ward KIA Only 25 F
1 S
26 F
2 S
27 H
2 S
28 F
2 S
29 H
2 S
30 F
2 S
31 H
2 S
32 h
2 S
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.)
- 22.
Enter the level of knowiedge (LOK) of each question as either (F}undamental or (H)igher cognitive level.
- 23.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 24.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent. more information is needed. or too much needless information).
The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated truelfalse statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (Le., the question has a valid KIA but. as written. is not operational in content).
The question requires the recall of knowiedge that is too specific for the closed reference test mode (Le.* it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified. or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer'S judgment, is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E)ditorial enhancement. or (S)atisfactory?
- 8.
At a minimum. explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOO (FtH)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- /
Back-Q= SRO B/MiN UlEtS Explanation Focus Oist.
Link units ward KIA Only 33 H
3 S
34 H
2 S
35 F
2 S
36 F
2 S
37 F
2 S
3B H
3 S
39 H
3 S
40 F
2 s
Instructions
[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 25.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 26.
Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 27.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (I.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated truelfalse statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g.. if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too speCific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or Inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatcihes are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section O.2.f.
- 7.
Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)aUsfactory?
B.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LaD (F/H) (1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-Q= SRO B/M/N UtEtS Explanation Focus Dist Link units ward KIA Only 41 F
1 U
Fixable. modify distractors 42 H
3 S
43 H
3 S
44 H
3 E
45 F
2 S
46 H
2 E
47 F
2 S
48 H
2 S
Instructions
[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 28.
Enter the level of knowiedge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 29.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 30.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g.* unclear intent. more information is needed, or too much needless information).
The stem or distractors contain cues (Le.* clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank. (M)odified. or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer's judgment. is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E)ditorial enhancement. or (S)atisfactory?
- 8.
At a minimum, explain any *U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (F/H) (1-5)
Stem Cues T/F Credo Partial Job-Minutia
- /
Back-Q=
SRO B/M/N UtE/S Explanation Focus Dist.
Link units ward KIA Only 49 H
2 s
50 F
2 s
51 H
2 e
Take out valve #, give aleak location, add "if all SA components operate as designed... "
52 H
2 s
53 F
2 s
54 H
2 s
55 H
2 s
56 h
2 s
Instructions
[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 31.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 32.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 33.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/M/N UlE/S Explanation Focus Dist.
Link units ward KiA Only 57 h
2 s
58 f
2 s
59 h
3
?
s 60 h
2 s
61 h
2 x
e Your distractors don't include all your answer choices; add "for this event" to avoid multiple correct answers 62 f
2 s
63 h
2 e
Fix explanation setpoint 64 h
2 s
Instructions
- 34.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 35.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 36.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (Le., the question has a valid KiA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KiA and those that are designated SRO-only (KiA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LaD (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/M/N utE/S Explanation Focus Oist.
Link units ward KIA Only 65 h
2 s
66 f
2 s
67 f
2 s
68 f
2 e
"potential" is potential trouble. Reword for definite effect 69 h
2 s
70 h
3 s
71 h
3 s
72 f
2 s
Instructions
[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 37.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H )igher cognitive level.
- 38.
Enter the level of difficulty (LOO) of each question using a 1 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 39.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed. or too much needless information).
The stem or distraclors contain cues (i.e.* clues, specific determiners. phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired. more than one is unacceptable.
One or more distractors is (are) partially correct (e.g.* if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (I.e.* the question has a valid KIA but. as written, is not operational in content).
The question requires the recall of knowiedge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KIA and those that are deSignated SRO-only (KIA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
- 7.
Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
B.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Credo Partial Job-Minutia
- 1 Back-Q=
SRO B/MtN utEtS Explanation Focus Dist.
Link units ward KIA Only 73 h
3
?
s Explain KiA match - ok 74 h
2 s
75 h
1 U
Single system setpoints low LOD. Give list of indications, selections in distractors Instructions
[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 40.
Enter the level of knowiedge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 41.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 42.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent. more information is needed, or too much needless information).
The stem or distractors contain cues (I.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (I.e., the question has a valid KIA but. as written, is not operational in content).
The question requires the recall of knowiedge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved KiA and those that are designated SRO-only (KiA and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
- 7.
Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E)ditorial enhancement, or (S)atisfactory?
B.
At a minimum, explain any "un ratings (e.g., how the Appendix B psychometric attributes are not being met).
Oyster Creek ILT 09-1 NRC Written Exam Joe D'Antonio, the Lead Examiner, called yesterday (4/8/10) with his comments on the 5RO written exam and RO questions 51-75. These comments are listed below. Additional comments were added by Joe D'Antonio on 4/12110.
Require RO/5RO Comment/Resolution
~
Validation?
53 U
Y KA Mismatch.
Question revised to match KA.
58 U
Y KA mismatch.
Question revised to match KA.
59 E
Y The provided reference is a direct lookup.
The reference has been deleted.
511 E
Y The provided reference is a direct lookup.
The reference has been deleted.
517 E
Y The provided reference is a direct lookup.
The reference has been deleted.
522 E
Y The provided reference is a direct lookup.
The reference has been deleted.
524 E
Y The provided reference is a direct lookup.
The reference has been deleted.
R4 U
Y LOD-1.
Wrote new question.
R19 E
N The question refers to SRM 21 while a diagram refers to 5RM 22.
Modified question to refer to SRM 22 only.
R36 U
Y LOO-1 Wrote new question.
R41 E
Y Change distracter B to a valve or something other than the SOC Pump.
Page 1 of 2
Oyster Creek ILT 09-1 NRC Written Exam Modified distrac1ter B.
51 E
N Delete V-6S-2, and put a leak in a location wrlich is downstream of V-6S-2.
V-6S-2 has been deleted and a location downstream of V-6S-2 has been inserted.
61 E
Y Distracter #3 was not used in any of the answer choices.
Used #3 in at least one distracter (answer A) 63 E
N The explanation stated that the TBV close on a lowering vacuum of 20" and 10".
The 20" was corrected to 1 0".
68 E
N The question needs to be made more definitive.
The question was made more definitive.
73 U
Y KA mismatch New question made to match KA.
75 U
Y The form of the question was not as expected for the KA.
New question made to match KA.
Page 2 of 2
Oyster Creek ILT 09-1 NRC Operating Exam Oyster Creek IL T 09-1 NRC Exam Changes from I\\IRC Prep Week (April 19, 2010)
Exam Component RO Admin JPM 1 RO Admin JPM2 ROAdmin JPM3 ROAdmin JPM4 SROAdmin JPM1 SRO Admin JPM2 SROAdmin JPM3 SRO Admin JPM4 SROAdmin JPM5 Sim JPM1 SimJPM2 Sim..IPM3 Changes to Exam Component Added note to include a copy of the procedural attachments for the evaluator.
~
Corrected procedure step number.
Added note to include a copy of the procedure for the evaluator.
Added that the JPM may be conducted in a classroom; Clarified the Cue in Step 2; Added note to include the reference procedure for the evaluator; Added that the JPM is Time Critical on the Student Handout.
Deleted the LMS qualifications for 2 EOs; Enhanced the Initial Conditions.
Enhanced the Initial Conditions; Enhanced the Initiating Cue.
Corrected circle/slash in the procedure handout and changed procedure entries (initials, data) to cursive font; Changed some critical steps to non-critical steps; Corrected a Standard for the Tech Spec applicability.
Added a reference.
Modified this JPM from an emergency re-classification plus State/Local Notification Form initiation, to an emergency re classification plus State/Local Notification Form and PAR determination and PAR Update Form initiation. Modified the Initial Conditions and the Initiating Cue.
Modified the Initial Conditions; Inserted a Cue in Step 3.
Modified the Initial Conditions; Enhanced the Simulator Setup Section; Added Note in Step 9; Enhanced the Standard in Step
- 10.
Enhanced the Initial Conditions; Modified the Simulator Setup; Page 1 of 2
Oyster Creek ILT 09-1 NRC Operating Exam r.lle in Step 7.
SimJPM 4 Added Note relating to Steps 8 and 9; Added Note stating the requirement to locate and trip a Core Spray Booster Pump breaker in the plant, during performance of in-plants JPMs.
Sim...IPM5 Enhanced the Cue on Step 7 and 14; Modified the Critical Step in Step 9.
Sim...IPM6 Modified Simulator Setup; Deleted Note in Step 7; Deleted several non-critical steps.
Sim JPM7 Corrected typo.
Sim...IPM8 Corrected the Standard in Step 3; Corrected procedural step number; modified malfunctions to trip a fan regardless of which fans were started Scenario 1 Minor enhancements and corrections.
Scenario 2 Modified the Event #3 malfunction which resulted in an annunciator not alarming (HSe).
Scenario 3 Minor enhancements and corrections.
Scenario 4 Minor enhancements and corrections.
Page 2 of 2