ML11130A147
| ML11130A147 | |
| Person / Time | |
|---|---|
| Site: | Columbia  |
| Issue date: | 04/22/2011 |
| From: | NRC Region 4 |
| To: | Energy Northwest |
| References | |
| 50-397/11-301, ES-401, ES-401-9 50-397/11-301 | |
| Download: ML11130A147 (8) | |
Text
ES-401 CG-2011-04 Written Examination Review Worksheet Form ES-401-9 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 8 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
H 3
N S
A, OR, 41.10 2
H 2
N S
C, CR, 41.5 3
H 3
N S
B, CR, 41.4 4
H 2
B S
A, CR, 41.4 5
F 3
N S
D, CR, 41.4 6
F 3
N S
C, CR, 41.10 7
H 3
U N
U S
A, CR, 41.4 - This appears to be a KA mismatch because it is focused on the turbine building service water system rather than the component cooling water (RCC) system. CAF Question was rewritten.
8 F
2 N
S B, CR, 41.6
ES-401 CN-2011-04 2
Form ES-401-9 Page 2 of 8 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 9
F 2
N S
A, CR, 41.2 10 F
3 U
N U
S C, CR, 43.2 - This is an SRO only question because it tests on the technical specification bases.
In addition, it doesnt test the ability to determine/interpret fuel pool level during refueling accidents. It is a K/A mismatch.\\
Question was rewritten.
11 H
3 N
S D, CR, 41.7 12 F
2 N
S D, CR, 41.6 13 F
3 N
S A, CR, 41.10 14 F
4 N
S B, CR, 41.7 EDITORIAL: The K/A statement says it is written to 295028 EA1.02.
The question is written to EK1.02, which is consistent with the written exam outline.
Fixed.
15 F
3 N
S C, CR, 41.4 16 H
3 N
S D, CR, 41.7 17 H
3 N
S C, CR, 41.10 18 F
3 N
S B, CR, 41.10 19 F
3 B
S D, CR, 41.4 20 F
3 N
S C, CR, 41.10 - NEED a copy of ABN-ELEC-GRID to verify that there is plausibility on its related distracters. In the ABN-GENERATOR procedure there are steps for 100 MW and 200 MW swings. Writing the question to these two may add to the plausibility.
21 H
3 B
S C, CR, 41.4 22 H
3 N
S B, CR, 41.4 23 H
3 N
S B, CR, 41.7 24 F
3 N
S A, CR, 41.2 25 F
2 N
S D, CR, 41.10 26 H
2 N
S C, CR, 41.10 27 F
3 N
S A, CR, 41.10
ES-401 CN-2011-04 3
Form ES-401-9 Page 3 of 8 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 28 H
3 N
S D, CR, 41.10 29 F
3 N
E S
A, CR, 41.7 - Add noun names for valves in distractors.
This was done.
30 F
3 N
S C, CR, 41.7 31 H
3 N
S B, CR, 41.5 32 H
3 N
S D, CR, 41.6 - Fix the typo in distractor B (trips) 33 F
3 U
N U
S B, CR, 41.6 - The exam outline (ES-401-1) says that this question is to be written to K/A 212000 A4.12. The K/A statement provided in the question, as well as the content of the question, are written to K4.12.
Question was rewritten.
34 H
3 N
S D, CR, 41.2 35 F
3 N
S D, CR, 41.10 36 H
3 N
S C, CR, 41.7 37 H
3 N
S A, CR, 41.7 38 H
3 B
S D, CR, 41.7 39 F
3 N
S A, CR, 41.7 40 H
3 N
S A, CR, 41.5 41 F
3 N
S B, CR, 41.5 42 H
3 N
S B, CR, 41.7 43 H
4 N
S D, CR, 41.5 44 F
3 N
S C, CR, 41.77 - In review of the Control Instrument Air (CIA) training materials in office (SD000156, August 2003), the CIA can be cross-tied with the Control Air System (CAS) via CN. An applicant could argue that distracters B and D are therefore plausible correct answers. Both of the pressures indicated look like they could cause the MSIV to go closed (less than 80 psig). Recommend adjusting the pressures so that this conflict is resolved. CAF.
OK as written.
45 F
2 E
N E
S B, CR, 41.4 - Include the noun name for the FN in the stem. Also, there is no TS RPS Room temperature requirement that could be found. Justify that distracter C is
ES-401 CN-2011-04 4
Form ES-401-9 Page 4 of 8 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only credible.
These comments were corrected.
46 H
3 N
S D, CR, 41.7 47 H
3 N
S C, CR, 41.4 48 H
3 N
S A, CR, 41.44 - Distracter B refers to RCC-P-1C not automatically starting, but the second half of the sentence says that it auto starts. Clarify.
Cprrected.
49 F
3 E
N E
S B, CR, 41.7 - Distractor C could be argued as also being correct as it is a subset of the correct disctractor.
The distractor was rewritten.
50 N
S D, CR, 41.5 - The justification for two of the distracters has to do with locations where procedural steps USED to be, not where they are now in procedures they are training on. If these changes took place during the course of the classs training period, and they were trained on the differences, it may be considered. If not, asking them a question about how procedures used to be arranged isnt valid. CAF.
Training on the new procedure was provided.
51 H
3 N
S C, CR, 41.5 52 H
3 N
S A, CR, 41.5 53 H
3 N
S D, CR, 41.7 - Based on reading the justification for the question, it appears there is a question as to whether the situation could result in tripping the output breaker of the charger. That needs to be resolved. Otherwise, distracter A may be correct as well.
CAF.
Question was revised.
54 H
3 N
S C, CR, 41.7 Editorial: The K/A number stated is 212001 on the question. It is 202001.
Corrected.
55 F
2 M
S B, CR, 41.7 56 H
3 E
N E
S D, CR, 41.7 - Based on the reference, the extent of the response from the other components in DEH control (governor and bypass) depend on power level and valve position at the time of the failure. Is there a case where this failure could occur, and reactor pressure could lower and stabilize based on the response of the governor and bypass valves? If there is a situation that this would occur, then distracter B could be
ES-401 CN-2011-04 5
Form ES-401-9 Page 5 of 8 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only argued as correct as well. Clarify the stem.
This question was revised to address the comments.
57 H
3 N
S C, CR, 41.7 58 H
2 N
S D, CR, 41.7 59 F
3 N
S B, CR, 41.7 60 F
3 N
S A, CR, 41.4 61 H
3 N
S A, CR, 41.7 62 H
3 N
S C, CR, 41.5 - Will the Hi-Hi-Hi signal go through when doing a surveillance? CAF Question was revised.
63 F
3 N
S C, CR, 41.5 64 F
3 N
S B, CR, 41.7 65 F
3 N
S D, CR, 41.7 66 H
3 M
S B, CR, 41.5 67 F
3 E
N E
S A, CR, 41.10 - It would seem distractor C could also be considered correct. CAF This question was revised.
68 H
4 N
S B, CR, 41.5 69 F
3 U
N U
S C, CR, 41.12 - All the distractors are T/F statements.
Question was revised.
70 H
3 N
S D, CR, 41.12 71 F
3 N
S D, CR, 41.10 72 F
3 E
N E
S D, CR, 41.10 - The question says that the SCBAs are located on the Control Room south wall, and the justification says they are located on the 501 Turbine Deck west end/south wall. There is no procedure to reference for the correct answer. What is the correct location?
Justification was incorrect and was revised.
73 F
3 E
N E
S D, CR, 41.10 - Procedure PPM 1.3.1, step 4.14.1, fourth note says that a CSCO OR a Caution tag could be hung in this situation. Recommend changing the wording in the question stem from the means of identifying the component to a means of identifying the component. Caution tags are not included in the possible answers, so
ES-401 CN-2011-04 6
Form ES-401-9 Page 6 of 8 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only there would be one correct answer.
Revised the question.
74 F
3 M
S C, CR, 41.10 75 H
3 N
S B, CR, 41.10 76 H
3 N
S A, CR, 43.5 77 F
3 N
S B, CR, 43.5 78 H
3 N
S B, CR, 43.5 79 F
3 N
S C, CR, 43.5 80 F
3 U
N U
S D, CR, 43.6 - This is a T/F question.
This question was be revised.
81 H
3 U?
N S
B, OR, 43.5 - It would seem that the completion time would be 20 minutes from the discovery of the second inop accumulator, i.e. 1944.
Question is correct as written.
82 H
3 N
S B, OR, 43.5 83 H
3 N
S B, CR, 43.5 84 H
3 N
S C, CR, 43.5 85 H
3 N
S B, OR, 43.2 86 H
4 N
S D, CR, 43.5 87 H
3 N
S A, CR, 43.5 88 H
4 N
S A, OR, 43.1 - Do not include the bases document or section 3.0 of T.S.
This has been verified.
89 H
3 N
S B, CR, 43.5 90 F
3 N
S D, CR, 43.2 91 F
3 N
S C, CR, 43.5 92 F
3 N
S A, CR, 43.5 93 H
3 N
S C, OR, 43.4 - Annunciator Panel Alarm document 4.602.A5 6-6 (Rev. 24) contains the steps to complete based on the given alarm. A river discharge is in progress, per
ES-401 CN-2011-04 7
Form ES-401-9 Page 7 of 8 OBDI 202 - IOLE Process Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only E
S the question stem. If this is the case, Step 3 of the alarm document says that IF a river discharge is in progress, THEN secure the discharge to the river UNTIL THE MONITOR RETURNED TO OPERABLE STATUS. Step 7 also directs review of the EAL threshold limits. Based on this, distracter A is also correct.
This question has been reivised.
94 H
3 N
S D, CR, 43.1 95 H
3 N
S C, CR, 43.5 - In the justification for distracter D, it says that it is incorrect because E-CB-31/21 will only auto close if a lockout exists on breaker 2-21. Per the reference provided (SSD000182, pages 71 and 72), it will only auto close if a lockout exists on breaker 3-31. CAF This has been corrected.
96 H
2 N
S D, CR, 43.4 97 F
3 E
N E
S A, CR, 43.5 - It would seem a reasonable argument could be made that distractor B is also correct; at least until the third RCC pump is started.
This question has been revised.
98 H
3 E
U N
U S
C, CR, 43.5 - The proposed question can be answered correctly solely with a systems knowledge of the EDG and its associated electrical distribution breakers. An applicant could not have knowledge of specific procedure steps in the mitigation of this situation, and still arrive at the correct answer (RO level of knowledge).
Second, it is not clear that the EDG would trip because it is in the UNIT mode of operation. Typically, this switch changes the droop characteristic from 5% to 1-2%.
The droop cannot be totally eliminated as this would result in a lack of EDG stability.
This question was rewritten.
99 F
2 N
S A, CR, 43.5 100 H
3 N
S A, CR, 43.3 RO TOTALS:
B= 1 F= 34 (45%)
E= 7 Additional Notes:
M= 7 H= 41 (55%)
U= 4 N= 67 S= 64 SRO TOTALS:
B= 0 F= 8 (32%)
E= 2 Additional Notes:
M= 0 H= 17 (68%)
U= 2
ES-401 CN-2011-04 8
Form ES-401-9 Page 8 of 8 OBDI 202 - IOLE Process N= 25 S= 21
GENERAL COMMENT
S:
- 1.
Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
- 2.
Chief Examiner comments are indicated in blue.
- 3.
Average difficulty is _2.91 on the RO exam and 3.00 on the SRO exam. Both these values are within the normal expected difficulty range for this reviewer.
- 4.
The 10CFR55.41/43 distribution is: RO / SRO 41.1 =
43.1 =
41.2 =
43.2 =
41.3 = 43.3 =
41.4 =
43.4 =
41.5 =
43.5 =
41.6 =
43.6 =
41.7 =
43.7 =
41.8 =
41.9 =
41.10 =
41.11 =
41.12 =
41.13 =
41.14 =
- 5.
The answer distribution is: RO / SRO A =
16 / 7 B =
18 / 7 C =
19 / 6 D =
22/ 5
- 6.
There are 1 / 5 (RO/SRO) questions with attachments provided.
- 7.
This review was performed by Steve Garchow the week of March 14, 2011.