ML20058H617

GG-2020-02 Draft Written Examination Comments

ML20058H617
Person / Time
Site:	Grand Gulf
Issue date:	02/10/2020
From:	Greg Werner Operations Branch IV
To:	Entergy Operations
References
Download: ML20058H617 (17)

Text

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 to 5 (easy to difficult) rating scale (questions in the 2 to 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

a. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

b. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

c. The answer choices are a collection of unrelated true/false statements.

d. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

e. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

a. The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

b. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

c. The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

d. The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

Place the answer letter here Place the 55.41 or 55.43 item here

8. At a minimum, explain any Unsat ratings (e.g., how the Appendix B psychometric attributes are not being met).

Ans

1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 CFR 8 Letter stem job #/ back- SRO- A/B/C/ 55.41x55 Q# (F/H) (1-5) cueing T/F cred dist partial minutia K/A B/M/N U/E/S Explanation focus link units ward only D .43y 1 F 2 B S B 41.10 ON 2012 NRC EXAM.

2 F 3 B S A 41.7 ON 2015 NRC EXAM.

Delete last sentence in stem and replace with "Circuit breaker 152-1511 should" Delete "Will" from A. Distractor C appears 3 F 3 X X N E B 41.8 implausible. How could the circuit breaker be available to energize bus 15AA but not be capable of remote or any local operation?

Revised. Q now SAT Delete "one" in stem. A is subset of D. Add "and stabilizes" after 4 H 2 X X N E B 41.5 "constant" in A to fix.Revised. Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 ON 2013 NRC EXAM. In stem, change "would" to "should." Add 5 F 2 X B E A 41.6 "without performing a calculation" at end of stem. Revised. Q now SAT ON 2015 NRC EXAM. In stem, change "it isolates" to "it should 6 F 2 X B E C 41.10 isolate." Delete "Isolates" from all four distractors. Revised. Q now SAT In stem, change "will" to "should."

7 H 3 X B E C 41.4 Revised. Q now SAT FIRST TEN. ON 2008 NRC EXAM. A not plausible (nothing happens; i.e. "none of the above").

8 H 3 X N U D 41.7 How could B and C ever occur (PV=nRT)? Replaced. Q now SAT.

9 F 3 B S B 41.2 ON 2007 NRC EXAM.

Delete periods at end of C and D.

10 F 2 X N E B 41.10 Revised. Q now SAT In stem, change the "a's" to "an's."

11 F 2 X N E D 41.7 Change "actions coincides with" to "actions should coincide with."

Revised. Q now SAT ON DEC 2017 NRC EXAM.

REFERENCE PROVIDED. In (1) 12 H 4 X B E B 41.14 add "should have" before "initially." In (2) change "is" to "should be." Revised. Q now SAT In stem, change (1) to "At what suppression pool temperature 13 F 3 X N E A 41.10 should the CRS call for SBLC injection?" Revised. Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 ON 2008 NRC EXAM. Correct answer obvious.Containment damage or challenge to equipment operability may occur at any time.

14 F 2 X B U A 41.10 (D is always correct for almost any given conditions) Replace? Q Replaced. In A and B, change "Enter" into "Entry into" Revised.

Q now SAT 15 F 2 B S D 41.10 ON MAY 2017 NRC EXAM.

In stem, add "the MINIMUM above 16 F 2 X N E A 41.10 which is" before "required."

Revised. Q now SAT 17 F 2 N S C 41.10 In stem, change "is to" to "should."

18 H 4 X N E B 41.1 Add "approximately" after "in."

Revised. Q now SAT Stem focus - change to "Per procedure 03-1-01-1, IOI, Cold Shutdown to Generator Carrying Minimum Load, to prevent high offsite release the Mechanical Vacuum Pumps (MVPs) should be 19 F 3 X X N E B 41.12 secured prior to exceeding a MAXIMUM of ___(1)___% power and the Steam Jet Air Ejectors (SJAEs) should be placed in service at approximately

_______(2)______psig." Revised.

Q now SAT Change "will" to "should" in stem.

Add "Per 05-1-02-I-4" before "If EFS" in stem. Subset issue - IF 496 Kv was correct, there would 20 F 3 X X N E C 41.10 be two correct answers.

Eliminates 496Kv as plausible.

Add "MINIMUM voltage" before "definition" in stem (should fix).

Revised. Q now SAT Change "will" to "should" in stem.

21 H 3 X N E B 41.10 Revised. Q now SAT First ten. Question OK - but avoid having second half of two 22 H 3 B S A 41.7 distractors the same unless using 2X2 format. a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 All distractors are grammatically incorrect (fix). Change "will's" to "should's" in distractors. Move 23 H 3 X X B E C 41.7 common distractor information to the stem. Revised. Q now SAT In stem, change "A RO" to "An RO." Distractor B not plausible.

How could the RO report level stable if the guage was maxed 24 H 3 X X N E A 41.9 out? Also, is there any plant parameter that ends its high range at "XX.3?" Need new B. In C, add "received" after "water." Revised.

Q now SAT REFERENCE PROVIDED. In stem, change (1) to just "When should automatic actions initiate?"

Change "will" to "should" in (2).

With provided reference, question answer is a direct look up (not 25 H 3 X X X N E A 41.7 believable that someone would ED while still having safe temperature values). OK if delete reference and make distractor temperatures closer to actual values. As written, correct answer is shortest time with highest temperatures.

Revised. Q now SAT Distractors B and C appear to employ "impossible physics."

26 H 3 X N E D 41.7 Please explain why not (or change). Replaced. Q now SAT In stem, change "from" to "for" and "will" to "should." In distractors, what is the difference between "operate all sump pumps" and 27 H 3 X X N E A 41.7 "manually start sump pumps?" If the same thing, delete that portion of answer. Revised. Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 Change "will" to "should" in stem.

In stem and in D, specify "wide 28 H 3 X X N E A 41.7 range reactor water level."

Revised. Q now SAT Change "will" to "should" in stem.

29 H 3 X N E A 41.7 Revised. Q now SAT Change stem to ask "What action(s) should be procedurally 30 H 3 X N E D 41.7 performed FIRST?" Need to add (s) to action since A and B are singular. Revised. Q now SAT Change "would" to "should" in stem. Second half of A and B, "system damage could occur," is almost always correct, even during normal system operation.

This auromatically cues that either 31 H 3 X X N U B 41.7 A or B is the correct answer and renders distractors C and D implausible. Second half of C and D don't appear plausible on their own. Replaced. Q now SAT ON 2007 NRC EXAM. Correct answer only one that has two parts (cause and effect), cueing it as the correct answer. Delete second half of C, as the applicant does not need to be told that the min flow valve closes, since this 32 F 3 X X B U C 41.7 would occur automatically. Also, D not credible - assuming that an inoperable light bulb would render a system inoperable is not plausible (need new D). How RO knowledge? Revised. Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 ON 2017 NRC EXAM. Applicant only needs to know second half of question to arrive at correct answer, i.e. if the >-217 level is known, A would only be correct if there were two correct answers, 33 F 3 X X B U C 41.8 thus eliminating it as a plausible disctractor. Restructure Q to ask for which meets MINIMUM requirements for flow and level.

Also, giant disparity between flow rates of A and B vice C and D (discuss). Replaced. Q now SAT 34 F 2 N S D 41.8 Change "will" to "should" in stem.

35 H 3 X N E D 41.9 Revised. Q now SAT REFERENCE PROVIDED. A and D do not appear plausible. No half scram is the same as "nothing happens" (i.e. "none of the 36 H 3 X B U B 41.8 above"). Can't see how anyone would come up with full scram (discuss). Revised. Q now SAT RPS seems a bit far-fetched.

37 F 2 X N E C 41.7 Discuss a better distractor for A.

Revised. Q now SAT B and D do not appear plausible.

Why would anyone think SRMs 38 H 3 X X B U C 41.7 would provide a rod insert block or that they are tied to the detector motor? Revised. Q now SAT 39 F 3 X N E B 41.7 Change second "is" to "should be" in stem. Revised. Q now SAT 40 H 3 N S D 41.7 Change "would" to "should" in stem. It appears that all one needs to know to answer this Q correctly 41 H 3 X N U C 41.7 is that a RCIC isolation signal will isolate RCIC. Q does not appear to have discriminating value.

Revised. Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 42 H 3 N S B 41.14 Change "is" to "should be" in 43 H 3 X N E B 41.9 stem. Revised. Q now SAT ON 2014 NRC EXAM. Change 44 H 3 X B E A 41.7 "would" to "should" in stem.

Revised. Q now SAT ON 2015 NRC EXAM. Change 45 F 3 X B E C 41.7 "will" to "should" in stem. Revised.

Q now SAT ON 2014 NRC EXAM. Change "will" to "should" in stem. Rad monitor groupings in distractors don't make sense. A has b and d, B and C have b and c, D has a 46 F 2 X X X B U C 41.11 and c. This cues b and c are probably in the correct answer.

Restructure distractors to eliminate cueing. Revised. Q now SAT Change "is" to "should be" in stem. 16AB is in three of four distractors, cueing it is probably in 47 H 3 X X X N E C 41.7 the correct answer. Need to restructure distractors such that the question is more like a 2X2.

Revised. Q now SAT 48 H 3 B S A 41.7 Change "is" to "should be" in 49 F 2 X B S D 41.7 stem. Revised. Q now SAT 50 H 3 N S B 41.7 ON MAY 2017 NRC EXAM.

Change "is" to "should be" in stem. There is absolutely no value in asking where parameters can be read in the control room on the written examination. The simulator 51 F 2 X X B U C 41.4 and operating test accomplishes this (eliminate that portion). This leaves only normal operating IA pressure, which in of itself doesn't have much dicriminating value (Replace Q). Replaced. Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 ON 2015 NRC EXAM. Delete "(1)"

in stem. Add "should" between 52 H 3 B E A 41.10 "that" and "still" in stem. Revised.

Q now SAT Change "will's" to "should's" in stem. Why would student think 53 H 3 X B E A 41.10 auto Xfer back to CST might exist (distractors B and D)? Revised. Q now SAT 54 H 3 N S C 41.7 In stem, change "are" to "should 55 H 3 X N E D 41.7 be." Revised. Q now SAT 56 F 2 N S C 41.7 Delete period at end of D Revised.

57 H 3 N E C 41.6 Q now SAT Change "will" to "should" in stem.

Add "on the indicator" at end of stem and eliminate "of the 58 H 2 X X N E A 41.7 indicator" in D. Change C to "unchanged" to be consistent with other distractors. Revised. Q now SAT 59 H 2 N S B 41.7 Don't see how A, B, or C could 60 F 3 X N U D 41.5 possibly be plausible. Replaced. Q now SAT Add "CST" to A and C to be consistent with B and D. If the 61 F 2 X N E D 41.7 CST has no level control system, what makes A and C plausible?

Revised. Q now SAT ON 2013 NRC EXAM. Add "Less than" at beginning of A to make it 62 H 2 X B E D 41.4 match "More than" in D. Revised.

Q now SAT 63 H 3 B S B 41.7 a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 Change "is" to "should be" in stem. C and D indicate "possible" control rod drop (almost anything is possible), cueing C or D is correct. It doesn't appear that 64 H 3 X X X N U D 41.7 performing a local single control rod scram is ever a corrective action, so how plausible? Renders D as the only plausible answer.

Replace? Restructered. Q now SAT Stem obviously indicates flooding.

Knowledge that merely stopping a pump will not isolate the flooding is LOD=1 knowledge and eliminates B and C as plausible.

The fact that all distractors have 65 H 3 X N U D 41.7 EOP entries except for A renders A implausible, leaving D as the only plausible answer. Q Replaced. Add "MAXIMUM" to (1); "minutes" misspelled in stem.

Revised. Q now SAT "Scram" in stem cues "reactor protection" in D immediately as the correct answer. Change "will" in stem to "should." Change Q to 66 F 2 X X N U D 41.10 require a different attachment?

Tier 3 Q - possible to make more generic? May have to replace K/A.

Q Replaced with Q on May 2017 NRC EXAM. Q now SAT 67 F 2 N S A 41.10 68 H 3 N S C 41.10 69 F 2 N S C 41.10 70 F 2 N S B 41.10 a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 ON 2008 NRC EXAM. Stem examples should be more specific.

In stem, move "during normal 71 F 2 X B E A 41.13 plant operations" to the beginning after "processes." Add "effluents" at the end of 2. Revised. Q now SAT First Ten. Revise stem: An event has occurred that requires entry into E3, "Containment Control,"

due to rising containment pressure. What is the MINIMUM containment perssure at which venting is procedurally required?

Change "will arise" to "should 72 F 2 X X N E C 41.13 occur." Delete "Before reaching" from all four distractors. Q replaced with Q of low or no discriminating value. Please retain original Q with above guidance. Original Q revised. Q now SAT 73 F 2 N S D 41.10 First Ten. B not plausible ("none of the above") C and D both correct as written (C is a correct statement and is a subset of D).

What drove the selection of the drywell pressure and the reactor 74 H 3 X X X N U D 41.5 water level parameters in the stem? Restructure distractors to have one true statement and three false ones. Revised. Tier 3 Q -

possible to make more generic?

May have to change K/A.

Replaced. Q now SAT 75 F 3 N S B 41.5 Why are Rev. 1 and Rev. 2 additions to stem necessary (discuss). In (1) and (2) in stem, 76 H 3 X X N E A 43.5 change "is" to "should be." How is distractor B(2) plausible? Revised.

Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 In distractor B, change "Restart" to "Restarting" to be consistent 77 H 3 X B E C 43.5 with other distractors. Revised. Q now SAT ON MAY 2017 NRC EXAM (root question). REFERENCE PROVIDED. Would like discussion on the differences 78 H 3 X M E B 43.5 between this Q and the 2017 Q to fully understand how it is a modified bank Q. Change "will" to "should" in stem. Discussed and revised. Q now SAT 79 F 2 B S B 43.2 ON 2013 NRC EXAM.

First ten. Discuss how this is SRO only knowledge. In stem, 80 F 3 X N E B 43.5 add "procedurally required" before "RPV." Revised. Q now SAT 81 F 3 N S C 43.4

1. (dispatch to RSP) in stem too obvious. Replace with Defeat LSS 82 F 3 X X N E C 43.5 to make distractors more credible? (discuss) Revised. Q now SAT What is you definition of ATWS 83 H 3 N S D 43.5 (discuss). Make sure applicable to Q. Discussed. Q SAT ON 2009 NRC EXAM. Q would be much better if the stem asked for which of the following is the MAXIMUM amount of time 84 H 3 B E B 43.2 allowed by tech specs to perform the surveillance and go with 4, 8, 12, and 24. Would still be a bank vice modified question because Q and A remain the same (just better distractors). Revised. Q now SAT First ten. REFERENCE PROVIDED. Capitalize 85 F 3 X N E C 43.5 "MIMUMUM" in stem. Add "procedurally" before "require."

Revised. Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 Revise (2) in distrators to be consistent (The CRS must call for 86 H 3 X N E A 43.5 only Att XXXX, and delete the "now's") Revised. Q now SAT In stem, change "DIRECTS" to "should DIRECT," and "is 87 H 3 X N S A 43.2 directing" to "should direct."

Revised. Q now SAT First Ten. In all four distractors 88 H 3 X X N E D 43.5 add "A-RPS" after "transfer." Q Replaced. Q now SAT 89 H 3 N S D 43.5 ON 2011 NRC EXAM.

REFERENCE PROVIDED.

90 H 3 B S C 43.5 Validate with licensee during validation week.

In stem, what is "CRS has been maximized for pressure?" In stem, 91 H 3 X N E B 43.6 change "action to be directed" to "required action." Revised. Q now SAT Change "will" to "should" in stem.

92 H 3 N E C 43.5 Revised. Q now SAT First ten. B and C don't appear plausible, as the procedure does not "go" with the associated action. This makes only A or D plausible. Restructure to have beievable actions with reasonable 93 H 3 X N U A 43.5 procedures to accompany those actions for B and C.Revised. Is Q still valid if only one switch failure triggers the alarm? Re-revised. Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 First ten. Somewhat confusing -

who are you referring to as "the SRO." SM? CRS? Both? Change stem to ask, "In accordance with procedure EN-OP-115, "Conduct of Operations," which of the following can ONLY be a resposibilty of an active licensed SRO? Distractors also need some reworking, as stem numbers 2 and 4 are both in three out of four disctractors, cueing that the 94 F 2 X X X N U D 43.5 correct answer is probably C or D.

Stem 3 should be revised to say "required to be in the main control room at all times when fuel is in the reactor" to make it more discriminating. Delete typo in stem. Revised; Stem 3 not revised per above and does not match the explanation. Re-revised. Q now SAT 95 H 2 N S B 43.5 First ten. Distractors need some reworking, as stem numbers 1 and 2 are both in three out of four disctractors, cueing that the 96 F 3 X X B U D 43.7 correct answer is probably C or D.

All distractors have "only" except for distractor D. Is this a bank or new question? Revised. Q now SAT ON DEC 2017 NRC EXAM.

REFERENCE PROVIDED. What 97 H 3 X B E B 43.5 reference needs provided?

Reference removed. Q now SAT 98 F 2 B S D 43.4 ON 2010 NRC EXAM.

REFERENCE PROVIDED.

99 H 3 N S C 43.7 Validate with licensee during validation week.

Change "will" to "should" in stem.

100 F 3 N E A 43.5 Revised. Q now SAT a Rev 2

ES-401 GG-2020-02 Written Examination Review Worksheet Form ES-401-9 Results Table RO LOK -H 40 Avg RO LOD 2.67 Flaws 10 CFR Distribution RO LOK-F 35 AVG SRO LOD 2.84 Stem focus 49 41.1 1 43.1 0 SRO LOK - H 16 Overall LOD 2.71 Cues 7 41.2 1 43.2 3 SRO LOK - F 9 T/F 0 41.3 0 43.3 0

%  % Cred Dist 39 41.4 3 43.4 2 RO Bank 26 34.67 SRO Bank 7 28 Partial 3 41.5 4 43.5 17 RO Mod 0 0 SRO Mod 1 4 job link 0 41.6 2 43.6 1 RO New 49 65.33 SRO New 17 68 units 0 41.7 33 43.7 2

% minutia 0 41.8 4 Total Bank 33 33 backward 0 41.9 3 Total Mod 1 1 KA 0 41.10 18 Total New 66 66 SRO-only 0 41.11 1

%  % LOD = 1 0 41.12 1 RO Sat 22 29.33 SRO Sat 9 36 41.13 2 RO Unsat 15 20 SRO Unsat 3 12 Answer Dist ( in %) 41.14 2 RO Edit 38 50.67 SRO Edit 13 52 RO-A 18 SRO-A 5

%  % RO-B 19 SRO-B 7 Total Sat 31 31 Total Unsat 18 18 RO-C 20 SRO-C 7 Total Edit 51 51 RO-D 18 SRO-D 6 a Rev 2

Question Overlap with the Previous Two NRC Exams QA form ES-401-6 item 4 requires 4 or less questions on RO exam from previous two NRC exams, SRO portion two or less questions from previous two NRC exams*

• if this is exceeded, the region shall call NRR to explain why/reasons.

RO overlap from previous two NRC exams 4

SRO overlap from the previous two NRC exams 2

Questions Requiring References to Answer 6-10 questions rule of thumb on SRO portion of exam are allowed to be open reference, while the RO exam is generally none RO Questions that require hand-outs 3

SRO Questions that require hand-outs 4