ML071550345
| ML071550345 | |
| Person / Time | |
|---|---|
| Site: | Cooper  |
| Issue date: | 01/12/2006 |
| From: | NRC Region 4 |
| To: | Nebraska Public Power District (NPPD) |
| References | |
| 50-298/05-302, ES-401-9, NUREG-1021, Draft Rev 9 | |
| Download: ML071550345 (8) | |
Text
ES-401 Cooper-10-2005 Written Examination Form ES-401-9 Review Worksheet (password: ) (TAC#: )
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1M F 2 B-10, CR. The KA is focused on loss of forced flow and natural circulation while the question is focused on loss of reactor recirc immediate actions. LOD is 2 versus 3.
2? H 3 S A-7, CR.
3M H 3 S D-7, CR.
4B F 3 S C-10, CR.
5? F 3 S B-10, CR.
6N F 3 S C-10, CR.
7B H 3 S A-7, CR.
8B F 3 S A-10, CR.
9M H 3 A-10, CR. Distractors B and D are not credible. RESOLUTION TO COMMENT: Replaced distractor B, D is acceptable as is.
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
@ The answer choices are a collection of unrelated true/false statements.
@ One or more distractors is (are) not credible.
@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
@ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
33 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 10? F 3 S B-11, CR.
11? H 3 S A-7, CR.
12N H 3 S A-7, CR.
13B F 3 D-5, OR, EOP. This question is open reference and according the material provided gives all the EOPs to the candidate. Rewrite the question such that it is a closed reference question. RESOLUTION TO COMMENT: Modified question. Modified question is acceptable.
16? H 3 S D-7, CR.
17M F 2 C-10, CR. Distractor B is not credible. The KA is about location and operation of components and the question does not address this.
RESOLUTION TO COMMENT: Replace with new question. New question is acceptable.
18? H 3 S C-10, CR, EOP.
19N F 3 S B-13, CR, EOP.
20? F 3 S A-7, CR.
21N F 2 S D-10, OR.
22? F 3 S B-6, CR.
23? F 3 S D-5, CR.
24? H 3 S A-10, CR.
25M F 3 S C-10, CR.
27N H 3 S B-7, CR 28N H 3 S C-7, CR 29? H 3 S B-5, OR 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 30? H 3 S A-7, CR.
31N F 3 S A-7, CR.
32? H 3 S D-7, CR.
33N H 3 S D-7, CR.
34B F 2 S B-6, CR.
35? H 3 S B-6, CR.
36? F 3 S C-6, CR.
37N F 2 S C-6, CR.
38 H 3 S C-6, CR.
39? H 3 S D-7, CR.
40 H 2 S C-7, CR.
41? H 3 S D-7, CR.
42N F 2 S B-7, CR.
43? F 3 S A-3, CR.
44? H 3 S A-4, CR.
45? H 3 S A-7, CR.
46? H 3 S B-10, CR.
47? H 4 S D-7, CR.
48N H 4 S C-4, CR.
34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 49? H 3 S B-4, CR.
50? H 3 S D-7, CR.
51? H 3 S A-4, CR.
52? H 3 S B-7, CR.
53M H 3 S C-4, CR.
54N F 3 S A-6, CR.
55M H 3 S A-6, CR.
56? H 3 S A-7, CR.
57? H 4 S C-2, CR.
58? H 3 B-43.2, OR. This is a TS interpretation question and therefore is SRO only. RESOLUTION TO COMMENT: Replace with new question. New question is acceptable.
59? H 2 S A-7, CR.
60? F 2 S B-4, CR.
61? H 3 S A-4, CR.
62? F 3 S D-4, CR.
63? F 3 S C-7, CR.
64? F 3 S C-13, CR.
65? F 3 S A-4, CR.
66 F 3 S A-6, CR.
34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 67? F 3 S B-10, CR.
68N F 3 S D-10, CR.
69? F 3 S C-2, CR.
70? H 3 U U A-43.2- OR. This is a TS interpretation question and is therefore an SRO only question.
71? H 3 S B-12, CR.
72? H 2 S C-12, CR.
73? H 3 S D-10. CR.
74? F 3 S D-10, CR.
75B H 3 S A-10, CR 76B F 3 S B-5, CR.
77N H 3 S C-5, CR.
78N H 3 D-41.10, OR. This question is focused on the actions contained in an AOP and is therefore an RO question. RESOLUTION TO COMMENT:
Replace with new question. New question acceptable.
79M H 3 A-41.10, CR. This question is focused on the actions contained in an AOP and is therefore an RO question. RESOLUTION TO COMMENT:
Replace with a modified INPO question. Replacement question is acceptable 80? H 3 S B-5, OR.
81M H 3 C-41.10, CR. This question focuses on the administrative requirements for entering a potentially oxygen deficient atmosphere and is therefore an RO question. RESOLUTION TO COMMENT: Replace with new question. New question acceptable 82B H 3 B-41.10, CR. This question is focused on EOP actions and is therefore an RO question. RESOLUTION TO COMMENT: Replace with new question. New question acceptable.
83? H 3 S A-2, CR.
84B H 3 S B-5, CR.
85 H 3 S B-5, CR.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 86? H 3 S A-2, OR.
34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LO (F/H) D U/E/S Explanation (1- St Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO
- 5) e Dist. Link units ward K/A Only m
Fo cu s
87? F 3 B-2, OR. The KA is focused on TS bases while the question is on a TS interpretation. RESOLUTION TO COMMENT: Replace with new question. New question acceptable.
88? H 3 S B-2, OR.
89N H 3 C-43.10, CR. This question tests on EOP actions and system response and is therefore an RO question. RESOLUTION TO COMMENT:
Replace with new question. New question acceptable.
90N F 3 S D-5, CR.
91? H 3 S A-2, OR.
92N H 2 S C-5, CR.
93N F 3 S A-2, CR.
94? H 3 S B-1, CR.
95? H 2 S B-2, OR.
96? H 2 S D-3, CR.
97? F 2 S C-3, CR.
98? F 3 S A-4, CR.
99? H 3 C-41.10, CR. This is a question focused on the actions contained within the EOPs and is therefore an RO question. RESOLUTION TO COMMENT: Replace with new question. New question acceptable.
100B F 2 D-5, CR. This is a non-discriminatory question. RESOLUTION TO COMMENT: Replace with new question. New question acceptable.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LO (F/H) D U/E/S Explanation (1- St Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO
- 5) e Dist. Link units ward K/A Only m
Fo cu s
RO N = 13 H = 43 U RO TOTALS B (56%) E M F = 32 Answers 10CFR Distribution A = 23 41.1 = 0 41.8 = 0 B = 18 41.2 = 3 41.9 = 0 C = 18 41.3 = 1 41.10 = 18 D = 16 41.4 = 9 41.11 = 1 41.5 = 4 41.12 = 2 41.6 = 9 41.13 = 2 41.7 = 24 41.14 = 0 SRO N=6 H = 18 U SRO TOTALS B (72%) E M F=7 Answers 10CFR Distrib ution (28%) A=6 43.1 = 1 43.5 = 8 B=9 43.2 = 7 43.6 = 0 C=6 43.3 = 2 43.7 = 0 D=4 43.4 = 0
GENERAL COMMENT
S:
- 1. The overall exam is well written in comparison to other NUREG 1021, Revision 9 exam submitted by licensees over the last year.
- 2. Many questions were not classified as new, bank, or modified.
Further, the original question was not submitted for those questions that were identified as a modified question. Because of this the modified/bank limitations could not be ascertained.
- 3. The number of higher cognitive questions (72%) exceeds the recommendation (60%) contained in NUREG 1021.
- 4. All metrics except those indicated above are within NUREG 1021 requirements.
NOTE: Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N; comment resolution is indicated in blue; 34 of 34 NUREG-1021, Draft Revision 9