ML101890111
| ML101890111 | |
| Person / Time | |
|---|---|
| Site: | Grand Gulf  |
| Issue date: | 06/25/2010 |
| From: | NRC Region 4 |
| To: | Entergy Operations |
| References | |
| 50-416/10-301 | |
| Download: ML101890111 (12) | |
Text
ES-401 GG-2010-06 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only C, CR, 41.7 - Stem needs enhancement to clarify that power is for operation of the valve. As written, there are 2 answers since technically 15P11 does supply power to the MOV motor space heater.
E Editorial - the Explanation contains a reference to LPCI vice LPCS 1 F 2 N Editorial - the Learning Objective related to this in GLP-OPS-E1200 is 7.3.
S Agree with comments.
Licensee will add text to make clearer.
Revised, change acceptable. Question is satisfactory.
A, CR, 41.7 S The stem needs clarifying so it cannot be misinterpreted to mean instrument air 2 F 2 N E piping from the discharge of the air compressor. If this were the case than distractor S B is correct. Suggest adding System pipingin the steam tunnel to the stem.
Disregard comment after discussion. Question is satisfactory Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 3 F 3 X N S B, CR, 41.10 4 F 3 X N S C, CR, 41.10 5 F 3 N S B, CR, 41.7 A, CR, 41.10 - DISCUSS: Wouldnt distracters C and D be plausible if you subtracted Loop A jet pump flows, vice Loop B?
E 6 H 3 X N Agree with comment - discuss with exam author.
S Licensee will review question.
Revised, changes acceptable. Question is satisfactory 7 F 4 N S B, CR, 41.10 8 H 3 N S C, CR, 41.7 9 H 2 N S D, CR, 41.7 D, CR, 41.7 10 H 3 X N S A revision was made based on comments provided by internal licensee review.
Question and changes reviewed, content is satisfactory.
11 H 3 N S D, CR, 41.7 12 F 2 N S C, CR, 41.7 13 H 3 X N S C, CR, 41.10 C, CR, 41.10 - Since the question stem tells you the level is in between the given E levels on the curve, why would 18.0 feet be selectable (answer B)?
14 H 3 X N S Licensee will reword question to resolve any issues.
Discussed basis for distractor B, will leave in the exam. Question is satisfactory.
15 F 3 N S B, CR, 41.10 16 H 3 X N S D, CR, 41.14 17 F 3 N S D, CR, 41.10 18 F 2 N S A, CR, 41.10 A, CR, 41.11 and .12 19 F 3 N S Changes were made as a result of an internal exam review. The revisions were reviewed, and the question is satisfactory. Page 2 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only D, CR, 41.10 - Should RO know minimum T.S. flowrate requirements for SLC pump?
Also, what are bases for distracters A & B (i.e. are they plausible?)
Editorial comment on Exam Outline - the outline says that this question is classified E as a F when it is a H.
20 H 3 X B S Discuss with exam author - are the expected flow rates the same as the minimum tech spec flow rates?
After discussion, questions are resolved. Editorial issue will be resolved.
Revision reviewed, question is satisfactory.
21 F 4 N S D, CR, 41.6 22 F 3 N S D, CR, 41.10 B, CR, 41.11 - This question structure is in conflict with NUREG-1021, Appendix B, Item C.1.h (reverse logic). Also, this is in conflict with NUREG-1021, ES-401, Item D.2.a, first paragraph (use of Tier 3 questions to test systems knowledge).
U Rewrite question so that it is not negatively stated. Per ES-401-E.2.d, this is an 23 F 3 X N E enhancement and therefore is not Unsat but needs revision. Given this is a tier 3 S question, recommend selecting a different KA.
Licensee will resolve issue.
Rejected K/A , selected new Tier 3 K/A for question. New question is satisfactory.
24 H 4 N S A, CR, 41.10 C, CR, 41.4 - Procedure 05-1-02-V-1 (Loss of Component Cooling Water), Section 3.1.3, calls for Answer D to be completed as part of its set of subsequent actions.
This and RWCU actions happen before adjustments to the Recirc pumps in Section 3.1.4. Further, in taking actions with the Recirc pumps, it says to reduce load on them to 67 lbm/hr, see if things stabilize, and if they dont, switch them to slow speed.
U I could see an applicant answering C because it is the part of the first action taken 25 H 4 X N here. I could also see the applicant arguing that answer D is incorrect because load S on the Recirc pumps should be reduced prior to trying the speed shift. Partially incorrect answer for answer D, partially correct answer on answer C.
Discuss with exam author.
Licensee will change Distracter D to resolve issue.
Distractor D and B have been revised, question is satisfactory.
A, CR, 41.4 26 F 3 N S None of the references for this question are provided. They are needed.
Discuss with exam author. Page 3 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Licensee will send needed materials.
Materials sent. The question is satisfactory.
A, CR, 41.1 and .9 -It says that the correct answer is C, but the explanation says the correct answer is A. Resolve.
E 27 H 3 N Discuss with exam author.
S Licensee will resolve issue.
Change made, question is satisfactory.
D, CR, 41.5 - GLP-OPS-C5104 page 11 defines APRM GAF as (actual thermal power/rated thermal power) /(APRM meter reading/100). A loss of RWCU input into the calculated thermal power would cause it to decrease per the explanation.
However, based on the above equation, the AGAF would actually be <1 not >1 per explanation. Even though AGAF<1, it would be non-conservative due to the failed computer input.
E Answer need clarification with lesson material.
28 H 3 N S Also, should initial condition state that the current AGAF is 1. If it is initially greater than 1, the decrease in calculated power due to RWCU input will lower AGAF, however it may stay above 1.
Agree with both comments.
Licensee will resolve issues.
Revisions made, question is satisfactory.
29 H 2 N S A, CR, 41.8 30 H 4 N S B, CR, 41.7 31 H 3 N S C, CR, 41.6 32 H 4 N S D, CR, 41.10 33 F 2 N S A, CR, 41.10 34 H 3 N S C, CR, 41.7 35 F 2 N S D, CR, 41.6 B, CR, 41.7 - This question is a fundamental knowledge one (F) vice higher E cognitive testing (H). It is testing knowledge of LPCI/Containment Spray initiation 36 H 2 N set points/permissives only.
S This is an editorial comment that has no impact on the actual question. Page 4 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Question revised, it is satisfactory.
37 F 2 N S A, CR, 41.3 E A, CR, 41.7 - Replace distractors C and D as they are not credible.
38 H 4 N S Changes to distracters C and D acceptable, question is satisfactory.
39 H 3 N S B, CR, 41.7 D, CR, 41.10 Answer C is correct i.a.w 05-1-02-III-1 para 3.4.3. The procedure says line-up S RWCU alternate SDC. It doesnt ask about existing decay heat. After placing RWCU 40 H 3 N E alternate SDC inservice, the procedure directs performance of answer D.
S Agree with comment.
Licensee will review question.
Revisions made, question is satisfactory.
41 H 3 N S A, CR, 41.12 B, CR, 41.3 and .5 - Is post-scram needed in the stem? It is contradictory with A which gives condition of ATWS. D is confusing and subjective as far as avoid pressure transients. SRVs are not SSCs that prevent (avoid) a transient, they E mitigate it. The distractor could be dismissed by this inaccuracy.
42 F 3 X N Suggest changing post-scram to following a condition requiring a scram. Discuss S
with exam author.
Licensee will resolve text issue.
Revisions made, question is satisfactory.
43 H 2 N S D, CR, 41.8 44 F 3 N S B, CR, 41.4 45 H 3 N S C, CR, 41.10 46 H 3 N S D, CR, 41.6 47 H 4 N S B, CR, 41.7 48 H 4 N S A, CR, 41.7 49 H 4 N S B, CR, 41.7 50 H 4 N S C, CR, 41.10 Page 5 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 51 H 3 N S A, CR, 41.7 52 F 3 N S B, CR, 41.7 53 F 3 N S A, CR, 41.7 B, CR, 41.7 - Move wait for the . to the stem since in all answers. Consider E selecting a more credible reset action for C and D (i.e. reset the RFPT locally) 54 H 3 N S This question Is S since this is a minor editorial comment - recommend making the S suggested change.
Revision made, question is satisfactory.
55 F 3 N S C, CR, 41.4 D, CR, 41.7 - 15AA and 16AB have identical BUV schemes, therefore modify C to read 15AA and 16AB are the only buses powered from offsite sources to improve credibility.
E Explanation line 3 - add a not after did 56 H 4 N S Agree with comments.
Licensee will review question.
Changes made, question is satisfactory.
C, CR, 41.4 - NUREG-1021, Appendix B, Item C.1.h discourages questions written in U this format.
57 F 2 X B E Revise this question so that it is not phrased negatively.
S Licensee will resolve issue.
Question revised, it is satisfactory.
C, CR, 41.4 - Remove Just from answer A Distractor B and D are similar.
E 58 H 3 X N Remove Just from distractor A and add only to the end of the same distractor.
S Distractors B and D are OK as written.
Revision made, question is acceptable. Question is New vice Bank.
C, CR, 41.4 - Need to add the justification for why answer D is incorrect.
E This is a minor editorial comment that has no affect on the question and is not 59 F 3 N S required by NUREG-1021. Agree with comment.
Revision made, question is satisfactory.
A, CR, 41.10 - A revision was made based on the results of an internal licensee 60 H 4 N S review. The revision was reviewed, and the question is satisfactory. Page 6 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 7 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only B, CR, 41.10 - A revision was made based on the results of an internal licensee 61 F 2 X N S review. The revision was reviewed, and the question is satisfactory.
B, CR, 41.10 S Answer A is not plausible since stem says CCW temp = SFP temp, therefore there is no differential to rapidly turn temperature.
62 H 3 N E Agree - Discuss with exam author.
S This issue is based on actual operating experience at plant. Question is satisfactory as written.
63 H 3 N S D, CR, 41.10 64 F 3 X N S B, CR, 41.10 B, CR, 41.4 - A & B - Order of sequencing between RHRP B and RHRP C is too specific for closed book test.
C & D - Not credible that identical pumps would have differing power requirements without more information in the stem (i.e. differing configurations)
U 65 H 3 X X N Agree - Suggest making C & D 200 KW and 200 KW instead of 200 KW and 800 KW S as currently written to improve credibility. The sequencing of B and C are reasonable memory questions so A and B are OK as written.
Licensee will resolve issue.
Question resolved. It is now satisfactory.
A, CR, 41.4 - Answer C is not plausible. Is there a situation where the applicant E could plausibly deduct that the bus voltage could be 60 VDC?
66 H 3 X B S Agree with comment.
Licensee explained answer, it is plausible. No further issues.
67 H 3 B S C, CR, 41.7 A, CR, 41.4 - This appears to be a SRO question, vice RO. Is it expected that ROs understand these controls?
This is an SRO only question since is testing on the refueling bridge interlocks.
E Recommend rejecting this KA.
68 F 4 N U Licensee will re-select K/A and address question.
S During validation, the licensee proposed a new question to address the existing K/A, instead of re-selecting the K/A. They were able to write a question testing RO knowledge of controls/indications in the control room during the new conditions provided. Proposed change is acceptable. Question is satisfactory. Page 7 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 8 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 69 H 3 X N S C, CR, 41.4 D, CR, 41.10 - Since the explanations for answers B and C say that there is no evidence that these effects can happen, and there are no requirements associated U with them, how can you say that they are plausible?
70 F 2 X N E Distractor C is plausible since some heat exchangers do have heatup rate limits associated with them. Distractor B is not plausible.
S Licensee will review and propose change to distracter B.
Change made to the distracter in question. Question is satisfactory.
A, CR, 41.10 - Tier 3 questions are expected to be based on plant-wide procedures, and not system specific. This question is system specific. See NUREG-1021, ES-E 401, Item D.2.a (first paragraph).
71 F 4 N Agree with comment - this needs to be a generic question.
S Licensee will review and resolve issue.
Changes made. Question is satisfactory. Now assigned to 55.41(b)(10).
72 F 3 N S A, CR, 41.10 73 H 3 X B S A, CR, 41.10 A, CR, 41.4 - Answer C suggests a rise in voltage on a bus when grounds are present. When there are additional loads for current draw, the voltage can only go down. This is an implausible distracter.
Editorial - In the Explanation, it would appear that the statement saying the correct answer is C should say A.
The stem indicated a local ground detector in alarm. It is common operator knowledge that a local alarm will also give an alarm in the control room (for the E majority of cases). A is the only answer that indicates an alarm in the control room 74 F 2 X N making it the logical answer. Answers do not discriminate adequate knowledge. This S question needs credible distractors.
Agree with comments.
Answer C is plausible based on observed level of knowledge of applicants based on LOK of ground detection.
Answer A question resolved by fix of editorial issue.
Licensee will review for further changes.
Question revised. It is satisfactory.
75 F 2 N S D, CR, 41.4 Page 8 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 9 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only D, CR, 43.2 - NUREG-1021, Appendix B, Item C.1.h discourages questions written in U this format.
76 F 3 X N E Rewrite question so that it is not written in a negative context.
S Licensee will resolve issue.
Question revised. It is satisfactory.
77 H 2 X X N S A, OR, 43.5 78 H 3 N S C, CR, 43.2 79 F 3 B S D, CR, 43.5 80 H 3 X N S B, OR, 43.5 81 H 4 N S A, OR 43.2 D, CR, 41.10 - Editorial: The attributed 10 CFR 55 section [55.43(b)(2)] doesnt match with this. That is for Tech Spec/TRM application. There should be a better 82 H 3 N S match.
Agree with comment.
Changed the attribute to 55.41(b)(10).
B, OR, 41.10 - Editorial: The attributed 10 CFR 55 section [55.43.(b)(10)] doesnt 83 H 2 N S exist. Review for better match.
Agree with comment. Changed to 55.41(b)(10).
84 H 3 X B S A, OR, 43.2 85 H 3 N S D, CR, 43.2 D, CR, 41.10 - Editorial: The attributed 10 CFR 55 section [55.43(b)(2)] doesnt match with this. That is for Tech Spec/TRM application. There should be a better 86 F 3 N S match.
Agree with comment. Changed to 55.41 (b)(10).
87 H 3 X X N S D, OR, 43.2 B, OR, 43.5 - Editorial comment - In the explanation, the last sentence says C is correct when B is the actual answer.
88 H 3 N S Correct the typo.
Typographical error resolved.
89 H 4 X N U C, CR, 43.5 - This question can be answered with systems knowledge of the effects Page 9 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 10 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E of instrument air loss. No special knowledge of decision making in EP procedures is S needed to come to the right answer. This is an RO question.
This is not an RO question as it is testing on decision making involving the EOPs given a set of conditions. Discuss with the exam author why distractor D is credible.
Licensee will review question and propose changes as needed.
Distracter D justification modified. Question is satisfactory.
C, CR, 43.2 - Answer A - Remove just and clarify to increase plausibility.
Is using RWCU in answer D plausible? Is it a means of decay heat removal that is credited? If it isnt, then the remainder of the answer is the same as answer C.
E Discuss Tech Spec bases.
90 F 2 X B S This question is OK as written. The RWCU is plausible since it is one of the alternate decay heat removal systems normally available. Agree with removing Just from distracter A.
Editorial change has been made.
91 F 4 N S D, CR, 43.7 B, CR, 43.4 - Add 10CFR55.43 reference E
92 F 3 N This question is S since the editorial comment has no effect on the question.
S Connected to 55.43(b)(4).
93 F 3 N S D, CR, 43.4 94 H 3 N S A, OR, 43.5 95 F 2 N S A, CR, 43.2 C, CR, 43.10 - Editorial: The attributed 10 CFR 55 section [55.43.(b)(10)] doesnt exist. Review for better match.
U Question does not address reactor/turbine pressure regulating system 96 H 3 X N S Agree with comment except it also does not address a safety related equipment.
Licensee will reject the K/A and reselect for new question.
Question has been revised. It is satisfactory.
97 H 3 N S C, OR, 43.5 B, OR, 43.5 - If the applicant establishes that the plant didnt meet the conditions to E enter EP-3 (Suppression Pool temperature greater than 95F) for answer C or D, then 98 H 4 X N S he/she automatically knows the other is wrong. One of these distracters should be revised. Page 10 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 11 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Agree with comment.
Licensee will resolve issue with question.
Distracter D changed. Question is satisfactory.
99 H 3 N S C, CR, 43.5 A, CR, 41.10 - Reference to 55.43(b)(4) is not correct.
E 100 H 3 N Question is S - Agree with comment.
S Referenced to 55.41(b)(10).
B= 5 F= 32 (43%) E= 21*
Additional Notes: *After review and comment resolution, all questions initially RO TOTALS: M= 0 H= 43 (57%) U= 3*
categorized as needing enhancement (E) or unsatisfactory (U) are satisfactory.
N= 70 B= 3 F= 8 (32%) E= 3*
Additional Notes: *After review and comment resolution, all questions initially SRO TOTALS: M= 0 H= 17 (68%) U= 1*
categorized as needing enhancement (E) or unsatisfactory (U) are satisfactory.
N= 22
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
- 2. Chief Examiner comments are indicated in blue.
- 4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 1* 43.1 = 0 41.2 = 0 43.2 = 10 41.3 = 2* 43.3 = 0 41.4 = 14 43.4 = 2 41.5 = 2* 43.5 = 9 41.6 = 4* 43.6 = 0 41.7 = 22 43.7 = 1 41.8 = 2 (3 SRO questions 10 CFR 55.43 designation TBD) 41.9 = 1*
41.10 = 27*
41.11 = 2*
41.12 = 2* Page 11 of 12 OBDI 202 - IOLE Process
ES-401 GG-2010-06 12 Form ES-401-9 41.13 = 0 41.14 = 1
- Questions written credited addressing more than one 10 CFR 41(b) section.
- 6. There are 10 questions with attachments provided.
- 7. This review was done by S. Garchow, S Hedger, and T. Pate the week of 5/17/2010.
- 8. The SRO exam contains 68% higher cognitive questions and this is above the 60% limit. Page 12 of 12 OBDI 202 - IOLE Process