ML101890111

2010-06-DRAFT-Written Exam Comments

ML101890111
Person / Time
Site:	Grand Gulf
Issue date:	06/25/2010
From:	NRC Region 4
To:	Entergy Operations
References
50-416/10-301
Download: ML101890111 (12)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9 GG-2010-06 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).

$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer

=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Attachment 8 Page 1 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia#/ units Back-wardQ= K/A SROOnly 1 F 2 N E S C, CR, 41.7 - Stem needs enhancement to clarify that power is for operation of the valve. As written, there are 2 answers since technically 15P11 does supply power to the MOV motor space heater.

Editorial - the Explanation contains a reference to LPCI vice LPCS Editorial - the Learning Objective related to this in GLP-OPS-E1200 is 7.3. Agree with comments.

Licensee will add text to make clearer.

Revised, change acceptable. Question is satisfactory.

2 F 2 N S E S A, CR, 41.7 The stem needs clarifying so it cannot be misinterpreted to mean instrument air piping from the discharge of the air compressor. If this were the case than distractor B is correct. Suggest adding "System piping-in the steam tunnel-" to the stem. Disregard comment after discussion. Question is satisfactory

ES-401 2 Form ES-401-9 GG-2010-06 Attachment 8 Page 2 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 3 F 3 X N S B, CR, 41.10 4 F 3 X N S C, CR, 41.10 5 F 3 N S B, CR, 41.7 6 H 3 X N E S A, CR, 41.10 - DISCUSS: Wouldn't distracters C and D be plausible if you subtracted Loop A jet pump flows, vice Loop B?

Agree with comment - discuss with exam author.

Licensee will review question.

Revised, changes acceptable. Question is satisfactory 7 F 4 N S B, CR, 41.10 8 H 3 N S C, CR, 41.7 9 H 2 N S D, CR, 41.7 10 H 3 X N S D, CR, 41.7 A revision was made based on comments provided by internal licensee review. Question and changes reviewed, content is satisfactory.

11 H 3 N S D, CR, 41.7 12 F 2 N S C, CR, 41.7 13 H 3 X N S C, CR, 41.10 14 H 3 X N E S C, CR, 41.10 - Since the question stem tells you the level is in between the given levels on the curve, why would 18.0 feet be selectable (answer B)?

Licensee will reword question to resolve any issues. Discussed basis for distractor B, will leave in the exam. Question is satisfactory.

15 F 3 N S B, CR, 41.10 16 H 3 X N S D, CR, 41.14 17 F 3 N S D, CR, 41.10 18 F 2 N S A, CR, 41.10 19 F 3 N S A, CR, 41.11 and .12 Changes were made as a result of an internal exam review. The revisions were reviewed, and the question is satisfactory.

ES-401 3 Form ES-401-9 GG-2010-06 Attachment 8 Page 3 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 20 H 3 X B E S D, CR, 41.10 - Should RO know minimum T.S. flowrate requirements for SLC pump? Also, what are bases for distracters A & B (i.e. are they plausible?) Editorial comment on Exam Outline - the outline says that this question is classified as a "F" when it is a "H."

Discuss with exam author - are the expected flow rates the same as the minimum tech spec flow rates? After discussion, questions are resolved. Editorial issue will be resolved. Revision reviewed, question is satisfactory.

21 F 4 N S D, CR, 41.6 22 F 3 N S D, CR, 41.10 23 F 3 X N U E S B, CR, 41.11 - This question structure is in conflict with NUREG-1021, Appendix B, Item C.1.h (reverse logic). Also, this is in conflict with NUREG-1021, ES-401, Item D.2.a, first paragraph (use of Tier 3 questions to test systems knowledge). Rewrite question so that it is not negatively stated. Per ES-401-E.2.d, this is an enhancement and therefore is not Unsat but needs revision. Given this is a tier 3 question, recommend selecting a different KA. Licensee will resolve issue. Rejected K/A , selected new Tier 3 K/A for question. New question is satisfactory.

24 H 4 N S A, CR, 41.10 25 H 4 X N U S C, CR, 41.4 - Procedure 05-1-02-V-1 (Loss of Component Cooling Water), Section 3.1.3, calls for Answer D to be completed as part of its set of subsequent actions. This and RWCU actions happen before adjustments to the Recirc pumps in Section 3.1.4. Further, in taking actions with the Recirc pumps, it says to reduce load on them to 67 lbm/hr, see if things stabilize, and if they don't, switch them to slow speed. I could see an applicant answering C because it is the part of the first action taken here. I could also see the applicant arguing that answer D is incorrect because load on the Recirc pumps should be reduced prior to trying the speed shift. Partially incorrect answer for answer D, partially correct answer on answer C. Discuss with exam author. Licensee will change Distracter D to resolve issue.

Distractor D and B have been revised, question is satisfactory.

26 F 3 N S A, CR, 41.4 None of the references for this question are provided. They are needed.

Discuss with exam author.

ES-401 4 Form ES-401-9 GG-2010-06 Attachment 8 Page 4 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnlyLicensee will send needed materials. Materials sent. The question is satisfactory.

27 H 3 N E S A, CR, 41.1 and .9 -It says that the correct answer is "C," but the explanation says the correct answer is "A." Resolve. Discuss with exam author. Licensee will resolve issue.

Change made, question is satisfactory.

28 H 3 N E S D, CR, 41.5 - GLP-OPS-C5104 page 11 defines APRM GAF as (actual thermal power/rated thermal power) /(APRM meter reading/100). A loss of RWCU input into the calculated thermal power would cause it to decrease per the explanation. However, based on the above equation, the AGAF would actually be <1 not >1 per explanation. Even though AGAF<1, it would be non-conservative due to the failed computer input. Answer need clarification with lesson material.

Also, should initial condition state that the current AGAF is 1. If it is initially greater than 1, the decrease in calculated power due to RWCU input will lower AGAF, however it may stay above 1. Agree with both comments. Licensee will resolve issues. Revisions made, question is satisfactory.

29 H 2 N S A, CR, 41.8 30 H 4 N S B, CR, 41.7 31 H 3 N S C, CR, 41.6 32 H 4 N S D, CR, 41.10 33 F 2 N S A, CR, 41.10 34 H 3 N S C, CR, 41.7 35 F 2 N S D, CR, 41.6 36 H 2 N E S B, CR, 41.7 - This question is a fundamental knowledge one ("F") vice higher cognitive testing ("H"). It is testing knowledge of LPCI/Containment Spray initiation set points/permissives only. This is an editorial comment that has no impact on the actual question.

ES-401 5 Form ES-401-9 GG-2010-06 Attachment 8 Page 5 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnlyQuestion revised, it is satisfactory.

37 F 2 N S A, CR, 41.3 38 H 4 N E S A, CR, 41.7 - Replace distractors C and D as they are not credible. Changes to distracters C and D acceptable, question is satisfactory. 39 H 3 N S B, CR, 41.7 40 H 3 N S E S D, CR, 41.10 Answer C is correct i.a.w 05-1-02-III-1 para 3.4.3. The procedure says line-up RWCU alternate SDC. It doesn't ask about existing decay heat. After placing RWCU alternate SDC inservice, the procedure directs performance of answer D. Agree with comment. Licensee will review question. Revisions made, question is satisfactory.

41 H 3 N S A, CR, 41.12 42 F 3 X N E S B, CR, 41.3 and .5 - Is "post-scram" needed in the stem? It is contradictory with "A" which gives condition of ATWS. "D" is c onfusing and subjective as far as "avoid pressure transients." SRV's are not SSCs that prevent (avoid) a transient, they mitigate it. The distractor could be dismissed by this inaccuracy. Suggest changing "post-scram" to "following a condition requiring a scram". Discuss with exam author. Licensee will resolve text issue.

Revisions made, question is satisfactory.

43 H 2 N S D, CR, 41.8 44 F 3 N S B, CR, 41.4 45 H 3 N S C, CR, 41.10 46 H 3 N S D, CR, 41.6 47 H 4 N S B, CR, 41.7 48 H 4 N S A, CR, 41.7 49 H 4 N S B, CR, 41.7 50 H 4 N S C, CR, 41.10 ES-401 6 Form ES-401-9 GG-2010-06 Attachment 8 Page 6 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 51 H 3 N S A, CR, 41.7 52 F 3 N S B, CR, 41.7 53 F 3 N S A, CR, 41.7 54 H 3 N E S S B, CR, 41.7 - Move "wait for the --." to the stem since in all answers. Consider selecting a more credible reset action for C and D (i.e. reset the RFPT locally) This question Is S since this is a minor editorial comment - recommend making the suggested change.

Revision made, question is satisfactory.

55 F 3 N S C, CR, 41.4 56 H 4 N E S D, CR, 41.7 - 15AA and 16AB have identical BUV schemes, therefore modify C to read "15AA and 16AB are the only buses powered from offsite sources" to improve credibility. Explanation line 3 - add a "not" after did Agree with comments. Licensee will review question.

Changes made, question is satisfactory.

57 F 2 X B U E S C, CR, 41.4 - NUREG-1021, Appendix B, Item C.1.h discourages questions written in this format.

Revise this question so that it is not phrased negatively. Licensee will resolve issue. Question revised, it is satisfactory.

58 H 3 X N E S C, CR, 41.4 - Remove "Just" from answer A Distractor B and D are similar. Remove "Just" from distractor A and add "only" to the end of the same distractor. Distractors B and D are OK as written. Revision made, question is acceptable. Question is "New" vice "Bank."

59 F 3 N E S C, CR, 41.4 - Need to add the justification for why answer D is incorrect.

This is a minor editorial comment that has no affect on the question and is not required by NUREG-1021. Agree with comment.

Revision made, question is satisfactory.

60 H 4 N S A, CR, 41.10 - A revision was made based on the results of an internal licensee review. The revision was reviewed, and the question is satisfactory.

ES-401 7 Form ES-401-9 GG-2010-06 Attachment 8 Page 7 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 61 F 2 X N S B, CR, 41.10 - A revision was made based on the results of an internal licensee review. The revision was reviewed, and the question is satisfactory.

62 H 3 N S E S B, CR, 41.10 Answer A is not plausible since stem says CCW temp = SFP temp, therefore there is no differential to "rapidly turn temperature." Agree - Discuss with exam author.

This issue is based on actual operating experience at plant. Question is satisfactory as written.

63 H 3 N S D, CR, 41.10 64 F 3 X N S B, CR, 41.10 65 H 3 X X N U S B, CR, 41.4 - A & B - Order of sequencing between RHRP B and RHRP C is too specific for closed book test. C & D - Not credible that identical pumps would have differing power requirements without more information in the stem (i.e. differing configurations) Agree - Suggest making C & D 200 KW and 200 KW instead of 200 KW and 800 KW as currently written to improve credibility. The sequencing of B and C are reasonable memory questions so A and B are OK as written. Licensee will resolve issue. Question resolved. It is now satisfactory.

66 H 3 X B E S A, CR, 41.4 - Answer C is not plausible. Is there a situation where the applicant could plausibly deduct that the bus voltage could be 60 VDC? Agree with comment.

Licensee explained answer, it is plausible. No further issues.

67 H 3 B S C, CR, 41.7 68 F 4 N E U S A, CR, 41.4 - This appears to be a SRO question, vice RO. Is it expected that RO's understand these controls? This is an SRO only question since is testing on the refueling bridge interlocks. Recommend rejecting this KA.

Licensee will re-select K/A and address question.

During validation, the licensee proposed a new question to address the existing K/A, instead of re-selecting the K/A. They were able to write a question testing RO knowledge of controls/indications in the control room during the new conditions provided. Proposed change is acceptable. Question is satisfactory.

ES-401 8 Form ES-401-9 GG-2010-06 Attachment 8 Page 8 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 69 H 3 X N S C, CR, 41.4 70 F 2 X N U E S D, CR, 41.10 - Since the explanations for answers B and C say that there is no evidence that these effects can happen, and there are no requirements associated with them, how can you say that they are plausible? Distractor C is plausible since some heat exchangers do have heatup rate limits associated with them. Distractor B is not plausible.

Licensee will review and propose change to distracter B.

Change made to the distracter in question. Question is satisfactory.

71 F 4 N E S A, CR, 41.10 - Tier 3 questions are expected to be based on plant-wide procedures, and not system specific. This question is system specific. See NUREG-1021, ES-401, Item D.2.a (first paragraph). Agree with comment - this needs to be a generic question.

Licensee will review and resolve issue. Changes made. Question is satisfactory. Now assigned to 55.41(b)(10).

72 F 3 N S A, CR, 41.10 73 H 3 X B S A, CR, 41.10 74 F 2 X N E S A, CR, 41.4 - Answer C suggests a rise in voltage on a bus when grounds are present. When there are additional loads for current draw, the voltage can only go down. This is an implausible distracter. Editorial - In the Explanation, it would appear that the statement saying the correct answer is C should say A. The stem indicated a local ground detector in alarm. It is common operator knowledge that a local alarm will also give an alarm in the control room (for the majority of cases). "A" is the only answer that indicates an alarm in the control room making it the logical answer. Answers do not discriminate adequate knowledge. This question needs credible distractors. Agree with comments.

Answer C is plausible based on observed level of knowledge of applicants based on LOK of ground detection.

Answer A question resolved by fix of editorial issue. Licensee will review for further changes.

Question revised. It is satisfactory.

75 F 2 N S D, CR, 41.4 ES-401 9 Form ES-401-9 GG-2010-06 Attachment 8 Page 9 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 76 F 3 X N U E S D, CR, 43.2 - NUREG-1021, Appendix B, Item C.1.h discourages questions written in this format.

Rewrite question so that it is not written in a negative context.

Licensee will resolve issue. Question revised. It is satisfactory.

77 H 2 X X N S A, OR, 43.5 78 H 3 N S C, CR, 43.2 79 F 3 B S D, CR, 43.5 80 H 3 X N S B, OR, 43.5 81 H 4 N S A, OR 43.2 82 H 3 N S D, CR, 41.10 - Editorial: The attributed 10 CFR 55 section [55.43(b)(2)] doesn't match with this. That is for Tech Spec/TRM application. There should be a better match. Agree with comment. Changed the attribute to 55.41(b)(10).

83 H 2 N S B, OR, 41.10 - Editorial: The attributed 10 CFR 55 section [55.43.(b)(10)] doesn't exist. Review for better match. Agree with comment. Changed to 55.41(b)(10).

84 H 3 X B S A, OR, 43.2 85 H 3 N S D, CR, 43.2 86 F 3 N S D, CR, 41.10 - Editorial: The attributed 10 CFR 55 section [55.43(b)(2)] doesn't match with this. That is for Tech Spec/TRM application. There should be a better match. Agree with comment. Changed to 55.41 (b)(10).

87 H 3 X X N S D, OR, 43.2 88 H 3 N S B, OR, 43.5 - Editorial comment - In the explanation, the last sentence says "C" is correct when "B" is the actual answer. Correct the typo. Typographical error resolved.

89 H 4 X N U C, CR, 43.5 - This question can be answered with systems knowledge of the effects ES-401 10 Form ES-401-9 GG-2010-06 Attachment 8 Page 10 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly E S of instrument air loss. No special knowledge of decision making in EP procedures is needed to come to the right answer. This is an RO question.

This is not an RO question as it is te sting on decision making involving the EOPs given a set of conditions. Discuss with the exam author why distractor D is credible. Licensee will review question and propose changes as needed.

Distracter D justification modified. Question is satisfactory.

90 F 2 X B E S C, CR, 43.2 - Answer A - Remove "just" and clarify to increase plausibility. Is using RWCU in answer D plausible? Is it a means of decay heat removal that is credited? If it isn't, then the remainder of the answer is the same as answer C. Discuss Tech Spec bases. This question is OK as written. The RWCU is plausible since it is one of the alternate decay heat removal systems normally available. Agree with removing "Just" from distracter A.

Editorial change has been made.

91 F 4 N S D, CR, 43.7 92 F 3 N E S B, CR, 43.4 - Add 10CFR55.43 reference This question is S since the editorial comment has no effect on the question. Connected to 55.43(b)(4).

93 F 3 N S D, CR, 43.4 94 H 3 N S A, OR, 43.5 95 F 2 N S A, CR, 43.2 96 H 3 X N U S C, CR, 43.10 - Editorial: The attributed 10 CFR 55 section [55.43.(b)(10)] doesn't exist. Review for better match. Question does not address reactor/turbine pressure regulating system Agree with comment except it also does not address a safety related equipment.

Licensee will reject the K/A and reselect for new question.

Question has been revised. It is satisfactory.

97 H 3 N S C, OR, 43.5 98 H 4 X N E S B, OR, 43.5 - If the applicant establishes that the plant didn't meet the conditions to enter EP-3 (Suppression Pool temperature greater than 95F) for answer C or D, then he/she automatically knows the other is wrong. One of these distracters should be revised.

ES-401 11 Form ES-401-9 GG-2010-06 Attachment 8 Page 11 of 12 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.

B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnlyAgree with comment. Licensee will resolve issue with question. Distracter D changed. Question is satisfactory.

99 H 3 N S C, CR, 43.5 100 H 3 N E S A, CR, 41.10 - Reference to 55.43(b)(4) is not correct.

Question is S - Agree with comment.

Referenced to 55.41(b)(10). RO TOTALS:

B= 5 F=32 (43%) E=21* Additional Notes: *After review and comment resolution, all questions initially categorized as needing enhancement (E) or unsatisfactory (U) are satisfactory.

M= 0 H=43 (57%) U=3* N= 70 SRO TOTALS:

B= 3 F=8 (32%) E=3* Additional Notes: *After review and comment resolution, all questions initially categorized as needing enhancement (E) or unsatisfactory (U) are satisfactory.

M= 0 H=17 (68%) U=1* N= 22

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is __

3__ on the RO exam and 3__ on the SRO exam.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 1* 43.1 = 0 41.2 = 0 43.2 = 10 41.3 = 2* 43.3 = 0 41.4 = 14 43.4 = 2 41.5 = 2* 43.5 = 9 41.6 = 4* 43.6 = 0 41.7 = 22 43.7 = 1 41.8 = 2 (3 SRO questions 10 CFR 55.43 designation TBD) 41.9 = 1* 41.10 = 27* 41.11 = 2*

41.12 = 2*

ES-401 12 Form ES-401-9 GG-2010-06 Attachment 8 Page 12 of 12 OBDI 202 - IOLE Process 41.13 = 0 41.14 = 1 *Questions written credited addressing more than one 10 CFR 41(b) section.

5. The answer distribution is: RO / SRO A = 20 / 6 B = 18 / 5 C = 19 / 6 D = 18 /8

6. There are 10 questions with attachments provided.

7. This review was done by S. Garchow, S Hedger, and T. Pate the week of 5/17/2010.

8. The SRO exam contains 68% higher cognitive questions and this is above the 60% limit.