ML053330587
| ML053330587 | |
| Person / Time | |
|---|---|
| Site: | Grand Gulf  |
| Issue date: | 10/03/2005 |
| From: | Ryan Lantz Operations Branch IV |
| To: | Gerald Williams Entergy Operations |
| References | |
| 50-416/05-301, ES-401-9 50-416/05-301 | |
| Download: ML053330587 (14) | |
Text
ES-401, Rev. 9 Written Examination Review Worksheet Form ES-401-9 Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S
- 7. Explanation Green Comments are MSH Black Comments are SMG and KC Red Comments are Resolution Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
H F
2 2
X X
U S
Handout provides direct lookup cue (question OK if no handout provided); doesnt test ability to RECOGNIZE abnormal indications for system operating parameters which are entry-level conditions for EOP/AOP per KA (KA statement in outline doesnt match catalogue).
Entire ONEP provided as attachment. Open ref Q.
Entire ONEP provided as reference. Open reference question.
This question was rewritten 2
F 2
E S
Answer is not justified in question comments.
This question was rewritten.
3 H
2 X
U S
series of T/F statements; C is not credible; This question was rewritten.
4 H
3 E
S change effect to affect in stem; the question was rewritten - word effect was eliminated.
5 H
3 E
S change stem to...Margin is assured...;
Recommended change made.
6 H
2 U
S Question is not discriminatory.
This question and distractors were modified.
7 F
F 2
2 U
U U
S Question does not match KA in KA Catalogue, Reference should not be provided. Open ref Q.
The KA is what is the effect a loss of CCW would have on a component or system. The question asks what the operator actions are in response to a loss of CCW.
Question was rewritten to match KA.
8 F
2 U
U U
S Question does not match KA in KA Catalogue.
No changes made to the question - Recommend Mark review this one.
This question was rewritten.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S
- 7. Explanation Green Comments are MSH Black Comments are SMG and KC Red Comments are Resolution Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6.
Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7.
At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401, Rev. 9 2
Form ES-401-9 Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 9
F 2
X U
S doesnt qualify as modified - must count as bank; A is not credible; This question was rewritten.
10 F
F 2
2 X
U S
KA mismatch.
Question was rewritten.
11 H
3 S
S Open ref Q.
Question was rewritten and is now closed reference.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 12 H
F 3
2 S
S This question was rewritten.
13 H
H 3
3 U
S New or not? Handout required or not (is EP-2 provided)? A and B are not credible; EP-3 seems to require ED, but C seems to suggest that it is not required - is C really correct? Regardless, answer is cued by EP-3; This question was rewritten.
14 H
H 3
3 X
U U
U U
S Question does not match KA statement in KA Catalogue. Open ref Q.
It can be argued the question matches the KA but it is a bad question.
The stem is unnecessarily complex and the four distractors could be easily appealed as being correct. I believe if the question were appealed, it would be deleted from the exam. Recommend Mark review this one.
Open referenced question.
If EP-3 is provided then this is a direct lookup question.
This question was completely rewritten.
15 H
4 U
U U
U S
Question does not match KA statement in KA Catalogue. Distractors do not answer the question posed in the stem. Open ref Q.
KA addresses the environmental qualification of drywell equipment given a high drywell temperature. The question asks about level instrumentation response and the distractors focus on operator actions based on level indications.
Handing out the EPs makes this a direct lookup; since the reference material was not provided, it is not clear why -160 is considered the trigger point; Are the CAUTIONS part of the intended handout? The way the stem is worded, none of the options answer the question; doesnt test knowledge of operational implications of HIGH DRYWELL TEMP per KA (question focuses on operational implications of HIGH CONTAINMENT TEMP instead);
Open reference question.
This question was completely rewritten.
16 H
3 X
X U
X U
U U
S Question does not match KA statement in KA Catalogue. Open ref Q.
The KA is knowledge of surveillance procedures wrt low suppression pool water level. The question is focused on system response and operator action in response to a low suppression pool level.
Open referenced question.
Handout cues the answer; A is not credible; does not test knowledge of the surveillance procedure regarding low SP per KA.
This question was rewritten to address above comments.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 17 H
3 E
S is it necessary to say...due to swell in the stem?
This question was rewritten and due to swell was eliminated..
18 H
3 E
S Attachment title should be included in distractor A. Open ref Q.
Open referenced question.
This question was rewritten and is now a closed reference question.
19 H
F 1
2 U
U U
S The answer is a direct lookup in the reference provided. Open ref Q.
Still think this is a direct lookup - Recommend Mark review.
Open referenced question.
This question was rewritten and is now a closed reference question.
20 F
F 3
2 S
S Question was rewritten.
21 H
3 S
22 F
H 2
3 U
U S
Question does not match KA statement in KA Catalogue.
Question was rewritten.
23 F
H 2
3 S
S 24 H
F 3
3 U
U U
U S
Question does not match KA statement in KA Catalogue. Open ref Q.
Distractors are true false statements. The question does not meet the KA, however the correct distractor does. (IE the question stem is irrelevant.)
Open referenced question.
This question was rewritten.
25 F
H 3
3 S
S This question was revised.
26 H
F 3
3 S
27 H
H 3
4 S
S This question was rewritten.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 28 F
3 S
This question was rewritten.
ES-401, Rev. 9 2
Form ES-401-9 Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 29 F
H 2
3 U
U S
Question does not match KA statement in KA Catalogue.
Initial review was incorrect.
30 H
H 3
3 S
S This question was rewritten.
31 H
F 3
3 U
U U
S Negative logic is used in stem.
Still uses negative logic.
This question was rewritten.
32 F
F 2
2 U
U U
S S
This is a 10CFR55.43(2) question. Also the question does not match the KA in the KA Catalogue.
This question was rewritten.
33 H
H 3
3 U
U S
S Question does not match KA statement in KA Catalogue (does not include the mitigation part of the KA).
Closer to the KA, but a very poorly written question.
This question was rewritten.
34 H
H 3
3 U
U S
Question does not match KA statement in KA Catalogue (does not include the mitigation part of the KA).
This question was rewritten.
35 F
2 S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 36 H
F 3
2 U
U S
S Question does not match KA statement in KA Catalogue. This KA should probably be thrown out as there is no relationship between the SRMs and control rod programming (sequencing).
Distractors were modified.
37 F
F 3
2 S
S Open ref Q.
Open reference question - reference not required to answer question.
This question was rewritten as a closed reference question.
38 H
H 3
3 U
U U
S Question does not match KA statement in KA Catalogue. Also, this KA is a repeat of question 12.
This question was not changed.
This question was rewritten.
39 H
F 3
2 U
U U
S Question does not match KA statement in KA Catalogue. Open ref Q.
This question was rewritten. The KA is knowledge of the power supplies to ADS. The question gives the power supply and asks how the system will respond.
This question was rewritten.
40 H
H 3
4 U
s LOK is H vice F; A, C, and D are correct; This question was rewritten.
41 H
F 3
3 U
U S
S Question does not match KA statement in KA Catalogue.
This question was rewritten.
42 H
H 3
3 U
U U
S Question does not match KA statement in KA Catalogue Question does not address pump runout.
Question was not revised.
This question was revised.
43 H
H 3
3 U
U U
S Question does not match KA statement in KA Catalogue. The catalogue KA focuses on the FWLCS 3 element/single element switch.
Do not believe this matches the KA - question not revised - Recommend Mark review independently.
This question was rewritten.
44 F
H 3
3 S
S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 45 H
H 3
3 S
S This question was rewritten.
46 H
H 3
3 U
S answer cued from handout; Open reference question.
Open reference question.
This question was rewritten as a closed reference question.
47 F
H 3
3 S
S Open reference.
This question was rewritten as a closed reference question.
48 H
H 3
3 U
U S
S Two distractors could be argued to be correct. (Distractors C and D)
This question was rewritten.
49 H
H 3
4 S
S This question was rewritten.
50 F
F 3
2 S
S This question was rewritten.
51 H
H 2
2 U
U S
Open reference. A system drawing should not be provided to answer this question.
This question was rewritten as a closed reference question.
52 H
H 2
2 U
U S
too easy - answer is obvious; doesnt test knowledge of connections/
relationships btwn CCWS and RCS IN ORDER TO DETERMINE SOURCE of RCS leakage into CCWS per KA; This question uses reverse logic.
Question was not revised.
This question was rewritten.
53 H
H 3
3 U
U S
This question should be rewritten because in its present state it would be extremely easy to appeal and win. The referenced procedure does not state which would occur first but rather sets temperature limits.
Question was not revised.
This question was revised.
54 F
2 S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 55 F
H 3
3 U
U S
what is meant by...of rated core flow of one another? C is not credible; TS Bases are not RO-level; Tech Spec basis are for SRO only questions per 10CFR55.43.
Question was not revised.
This question was rewritten.
56 H
H 3
3 S
S This question was revised.
57 F
F 3
2 U
U S
S Question does not match KA statement in KA Catalogue. KA is focused on how RWCU is designed to maximize plant efficiency - question does not ask this.
New Question written.
This question was rewritten.
58 H
F 3
3 U
U S
handout narrows choices to B or D; C is not credible; doesnt test knowledge of connections/relationships btwn PRIMARY CONTAINMNT and RELIEF/SAFETY VALVES per KA; Question does not address the cause and effect relationship as required by the KA Question was not revised.
This question was rewritten.
59 H
H 3
3 U
U S
Distractors are long and unnecessarily complex.
Question was not revised.
This question was rewritten.
60 F
3 S
61 H
H 3
3 U
S The stem is overly complex with a lot of confusing information that is not relevant to the question.
This question was rewritten.
62 F
F 3
2 U
U S
Question does not match KA statement in KA Catalogue.
Question was not revised.
This question was revised.
63 H
3 S
S This question was rewritten.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 64 H
4 U
U U
S Question does not match KA statement in KA Catalogue. Open ref Q.
New Question, however specific determiner flaws cue all three distractors. Pt value is also incorrect (should be 2.7 not 2.9)
This question was rewritten.
65 F
3 S
U S
Open ref Q. Possible direct look-up.
Question was not revised.
This question was rewritten.
66 F
3 S
U S
Open ref Q. Possible direct look-up.
Question was not revised.
This question was revised to be a closed reference question.
67 H
3 S
U S
direct lookup; handouts not provided for review - cannot confirm correct answer; doesnt test ability to obtain/interpret station reference materials which contain PERFORMANCE DATA per KA; Open ref Q. Possible direct look-up.
Question was not revised.
This question was revised.
68 F
2 S
69 H
3 S
U S
handout cues answer; doesnt test ability to perform pre-startup procedures for the facility including operating those controls associated with plant equipment that COULD AFFECT REACTIVITY per KA; Open ref Q. Possible direct look-up.
Question was not revised.
This was made a closed reference question.
70 F
2 S
S 71 H
3 U
E S
No correct answer.
My mistake on review. Question still is written such that it may be difficult to read.
This question was revised.
72 F
1 2
U S
doesnt test knowledge of OPERATIONAL IMPLICATOINS of EOP cautions and notes per KA; Direct look-up, too easy. Open ref Q.
This question was revised to a closed reference question.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 73 F
3 E
E S
Stem needs further clarification.
Question was not revised. Is it acceptable by utility to have operators memorize cautions by number AND content?
This question was rewritten.
74 F
F 3
3 U
S KA Mismatch.
This question was rewritten.
75 F
3 S
76 H
4 U
U S
RO Knowledge This question was rewritten.
77 F
2 U
U S
RO Knowledge This question was rewritten.
78 H
3 U
U U
S RO Knowledge, K/A mismatch also.
This question was rewritten.
79 F
3 S
This question was revised.
80 F
3 U
U S
RO Knowledge This question was rewritten.
81 F
3 U
U U
S Negative perspective. Which is NOT required... K/A mismatch also This question was rewritten.
82 F
3 U
U S
RO Knowledge It was determined this is an SRO only question based on the references for the answer are the FSAR and technical specifications.
83 H
3 S
Possibly a direct look-up question?
This question was rewritten.
84 F
3 S
85 H
3 U
U S
RO Knowledge This question was rewritten.
86 H
3 S
This question was rewritten.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 87 H
2 U
U S
RO Knowledge This question was rewritten.
88 H
3 S
89 H
3 S
90 H
3 S
This question was rewritten.
91 H
3 S
92 H
3 U
U U
S RO Knowledge, K/A mismatch. Must write on A2 (b) portion of K/A.
This question was rewritten.
93 F
3 S
Possibly a direct look up question.
This question was rewritten as a closed reference question.
94 F
1 2
U U
S RO Knowledge, too easy. Better K/A match 2.4.28 This question was rewritten.
95 H
3 S
This question was rewritten.
96 H
3 U
U S
RO Knowledge This question was rewritten.
97 F
2 U
U S
Negative perspective. Which is NOT required...Better K/A match 2.2.11 This question was rewritten.
98 F
2 U
U S
Negative perspective. Which is NOT required...
This question was rewritten.
99 H
3 U
U S
RO Knowledge This question was rewritten.
100 H
3 S
Total Attached reference Questions = 33, RO = 19, SRO = 14.
Total Unsat Questions = 48.
Total KA Mismatch = 24 General Comments:
1.
Many of the KAs on the outline do not match the intent of the KAs in the KA Catalogue. The KAs in the outline appear to be learning objectives. The review was performed using the KAs in the catalogue.
2.
Many of the questions reviewed are overly complex making the question easier to challenge/appeal.
3.
Most of the 33 open reference questions are close to direct lookups and do not test the candidates knowledge.
4.
Some of the questions have multi-part distractors that are several lines long.
Second Precursory Review Data:
Note: SRO exam has not been sent in to the region for review as of Wednesday 3pm.
- 1. Total Unsat Questions = 25 (RO only)
- 2. Total K/A Mismatch Questions = 13 (RO only)
Originally 14 SRO questions were Unsat. Assuming no questions are changed on the SRO portion (which has yet to be submitted) the total Unsat value would be 39, so the number of Unsat questions has gone down by 11.
Tier/
Group Randomly Selected K/A Reason for Rejection 2/1 206000 High Pressure Core Injection (HPCI) - GGNS does not have a HPCI System for water inventory control.
2/1 207000 Isolation (Emergency) Condenser - GGNS does not have an Isolation Condenser for pressure suppression.
2/2 201002 Reactor Manual Control System (RMCS) - GGNS utilizes the BWR 6 Rod Control and Information System.
2/2 201004 Reactor Sequence Control System (RSCS) - GGNS utilizes the BWR 6 Rod Control and Information System.
2/2 201006 Rod Worth Minimizer (RWM) - GGNS utilizes the BWR 6 Rod Control and Information System.
2/2 214000 Rod Position Information System (RPIS) - GGNS utilizes the BWR 6 Rod Control and Information System.
2/2 215002 Rod Block Monitor (RBM) - GGNS utilizes the BWR 6 Rod Control and Information System.
2/2 230000 RHR/LPCI: Torus/Pool Spray Mode - GGNS does not have a Torus/Pool Spray mode of the RHR System.
1/1 600000 Plant Fire On Site - AK3 was selected for the Random Selection topic. This topic has 4 statements of which only AK3.04 has an importance of > 2.5 for a RO.
2/1 203000 RHR/LPCI: Injection Mode - K5.01 was eliminated due to the testable check valves for RHR are being disabled via plant change per an ER, thus the selection resulted in the only other K/A K5.02.
2/1 300000 Instrument Air - K6.04 has an importance of > 2.5, however GGNS does not have a Service Air Refusal Valve.
2/2 215001 Traversing In-Core Probe - At GGNS TIPS is only operated by Reactor Engineers. The only operations involvement with the system is protective tagging.
1/1 295027 295038 High Containment Temperature High Offsite Release Rate - Random selection for Generics was 2.2.34 Knowledge of the process for determining the internal and external effects on core reactivity. This K/A does not apply to these two Emergency/Abnormal Plant Evolutions. Random selection was redrawn.
3 Generics Random selection of 2.2.31 Knowledge of procedures and limitations involved in initial core loading is not applicable to GGNS.
1/1 295027 High Containment Temperature - Generics was selected for the topical area. 2.2.18 Knowledge of the process for managing maintenance activities during shutdown operations and 2.2.1 Ability to perform pre-startup procedures for the facility/including operating those controls associated with plant equipment that could affect reactivity, do not apply to high containment temperature in the realm of emergency level. There is not sufficient energy to raise Containment Temperature to the emergency level.
1/1 295001 Partial or Complete Loss of Forced Core Flow Circulation The initial random selection selected 2.3.2 Alara considerations. Section 2.3 of the Generics have limited applicability for this evolution. Attachment 2 section 1 sentence 4 allows elimination of these K/As without justification. The K/As listed in sentence 1 were numbered 1 - 16 and randomly selected to apply K/A 2.4.4.
1/1 600000 Initial topic called for AA2.16 Ability to determine and interpret as applied to a plant fire on site - vital equipment and control systems to be maintained and operated during a fire. The original outline identified the Main Transformers. The Main Transformers are only vital to Main Generator output. K/A mismatch was felt to be the case and changed the topic to basis for separating vital equipment from the Main Control Room during a fire in the Main Control Room to better fit the K/A based on 10CFR 50 Appendix R considerations.
1/1 295025 Initial K/A was K3.05, this is similar to K/A for 217000 A1.02. Reselected to 295025 K3.06 2/2 202001 Initial K/A was Generic 2.2.25 Tech Spec Bases. Not considered RO level knowledge. Reselected to Generic 2.4.11.
3 SRO 2.1.24 Protective Tagging is considered an RO function for written examinations. Reselected to 2.1.12 for SRO Only question.