ML15007A178

2014-12 ES-401-9 Draft Written Exam Comments

ML15007A178
Person / Time
Site:	River Bend
Issue date:	12/12/2014
From:	Vincent Gaddy Operations Branch IV
To:	Entergy Operations
laura hurley
References
Download: ML15007A178 (20)

Text

ES-401 RB-2014-12 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only General Comments:

1. Please provide parent questions for any questions designated as modified so the extent of modification can be evaluated.

2. Explanations of why a distractor is plausible are encouraged.

3. Explanations of why a question is SRO-only are encouraged.

4. There are too many question stems that generically state that a transient has occurred or a bus is lost and that operators have entered a procedure.

A more effective and operationally valid way to test knowledge is to provide actual plant conditions and parameters and allow the candidates to analyze the integrated plant response.

5. There are many non SRO only questions on the SRO exam. Please refer to SRO-only guidance on NRC public website.

Questions need to delve deeper into AOPs and EOPs. Many questions examine basic entry conditions or elementary systems knowledge. These questions do not distinguish between a qualified and unqualified candidate.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E CFR 55.41.7, 2000 NRC exam Distractor A is not credible. ATWS/ARI is designed to reduce power to much lower than 73%.

Would Recirc Flow Control Valve Runback actually occur if level is 35 as noted in 1 F 3 X d B the stem? References state that this actuation occurs at Level 4 (30.8) with less than 3 feed pumps running. If the assumption is that at some point a feed pump was lost and RPV level dropped below 30.8 before recovering to 35, the stem should state as such.

S Licensee found different question in their bank with rev2 that has not been previously used on an NRC exam and it is now Sat.

2 H 3 c B S CFR 55.41.8, March 2014 NRC exam.

CFR 55.41.7, not used on previous NRC exam but pulled from RBS bank.

U Distractors B&C are not credible. With a basic understanding of the purpose of the turbine bypass system, these distractors can be easily disregarded. The thought that 3 H 3 X a B there could be a common hydraulic failure that would affect both the main turbine and the bypass system is not plausible.

S Licensee made changes to both distracters in rev1 of the question and it is now Sat.

CFR 55.41.1 U LOD 1: Question asks for basic response of plant after a Reactor Scram from 100%

power. Also GFES knowledge and therefore should not be on this exam.

I think A is also correct-it may not be the answer the question author was after but the 4 F 3 X X c N question remains, if you shift recirc pumps to slow speed does that cause a drop in level? I think it does, whether at power or immediately after a trip. It may not be as dominant after a trip as the FCV design feature on level control but it could be argued S as correct on a challenge, right?

Licensee wrote new question and as rev1 is now Sat.

5 F 3 a N S CFR 55.41.10 CFR 55.41.4, RBS audit exam 2010 (made some mods to two distractors during free 6 H 3 a B S review to make it Sat).

CFR 55.41.7, December 2008 NRC Exam.

E Distractor D is not credible. It is not plausible to think that the low air pressure alarm 7 H 3 c M would come in after FRVs lock-up on a loss of air.

S Changed all four distracters and question is now Sat as rev1. Page 2 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 8 H 3 c B S CFR 55.41.10, GG 2010 NRC exam.

CFR 55.41.7 E

Add only to the end of distractors B&D.

9 H 3 X c N It is unclear from the references provided why the second train would start on a low flow signal.

S Licensee made changes as recommended and question is now sat as rev1.

10 F 3 a B S CFR 55.41.6, 2008 RBS NRC exam.

CFR 55.41.7 U Distractors A&C are not credible. RCIC is designed to provide inventory control for an isolated RPV. It is not plausible to think that RCIC is designed to isolate on a full MSIV closure as this is the condition that RCIC is designed to address.

11 H 3 X d N Question states after the scram. Would RCIC eventually start on low level due to the closure of MSIVs and loss of reactor feed pumps? This would occur after the scram. If it is meant to ask what happens immediately after the scram, then the stem S should make it clear.

Licensee made changes to distracters A and C with rev 1 and question is now Sat.

CFR 55.41.10, HANDOUT.

U Distractors C&D are subsets/replicates of each other since both temperatures are 12 H 2 c N less than 145 deg.

S Licensee made requested changes with rev2 an question is now Sat.

U CFR 55.41.9 Distractors C&D are not credible. It is not plausible that a candidate would consider a 13 F 4 X b N temperature of 180 degrees to be greater than the scale or EQ of a suppression pool temperature instrument.

S Licensee made requested changes with rev1 an question is now Sat.

U CFR 55.41.2, March 2008 RBS audit exam.

Distractor A needs to read like answer C to prevent cueing and for symmetry. It should read something like continue to indicate level on-scale when actual RPV level went off-scale high.

14 F 3 X X X c N Distractor B is too short and should read something like fail off-scale low because boil off reduces the height of water in the reference leg.

I also think that this is borderline GFES (knowledge of basic instrument design and function) and to be more plant specific you could have gone with the 93 degrees MRT Page 3 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only value or an MIL value or something that really fits the operational implications AT RBS for high DW temp to be plant specific.

Stem is unclearly written.

Distractor A could be considered correct under certain conditions. The basis states that The direction and magnitude of the change depend upon the arrangement of the instrument runs. If the variable leg vertical run in an area is longer than the reference leg vertical run, the indicated level will tend to decrease as temperature in the area S rises.

NRC wrote new question as rev1 and it is now Sat.

15 F 2 a N S CFR 55.41.9 CFR 55.41.7, from RBS bank, not used on an NRC exam.

E Question: In the stem you state that an operator is directed to close the min flow valve but you dont state if it was actually closed. Stem needs to be clear on this 16 H 2 X X b M aspect.

S Licensee modified stem to include that valve is closed and included as part of an IST to ensure operationally valid. Question is now Sat as rev1.

E CFR 55.41.1 This looks like GFES to me and even your reference points to GFES. I think we need 17 H 2 X d N something else here for this question or at least change the reference to something other than GFES material.

S Licensee wrote new question rev1 and it is now Sat.

18 F 2 b N S CFR 55.41.10 CFR 55.41.10 E If this is knowledge of AOPs where is the AOP for the reference? IS AB-141-531 an 19 F 3 d X B AOP because it isnt labeled as such?

S Licensee showed AOP-0052 ties to AB-141 and so this meets the KA and is correct as written with text to show the tie in the worksheet., question is now Sat with rev1.

20 F 3 d N S CFR 55.41.5 U CFR 55.41.5 Stem needs to state At steady state conditions.. because in explanation section it even states that pressure changes in both directions and then stabilizes out at 21 H 3 X X a B original pressure. As written there is no correct answer. If you stated that the initial response is for power to rise and pressure to rise then that would be correct. Instead of remain steady for pressure in the answer and choices it would be better to use the words stabilize at its original value. This prevents a challenge on appeal Page 4 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only because it technically does not remain steady.

You need to check this on the simulator as well as USAR to ensure it is correct for the given conditions.

LOD 1: Distractors C&D are not credible. With the most basic knowledge of the effects of increasing recirc flow, distractors C&D can be immediately eliminated.

K/A mismatch: Question does not establish a high reactor pressure condition.

Stem does not require How will the EHC pressure control system respond to the condition? to be included.

Licensee could not write non GFES question on this KA so they resampled, pulled S question from Bank, used on NRC exam in 2008, rev1 and made requested changes and question is now Sat.

CFR 55.41.14, RBS bank, not used on NRC exam.

U Distractors B&C are not credible. There is no plausible reason to expect FRV leakage or a cooldown would occur with the given conditions.

22 H 3 X a M Does the stem require drywell pressure as a given parameter to answer the S question?

Licensee made requested changes with rev1 an question is now Sat.

CFR 55.41.10, RBS bank, not used on NRC exam.

U The only parameters that are even close to entry conditions are primary containment and drywell parameters. All RPV parameters are well inside their normal ranges.

Distractors A&B can be easily eliminated with the very basic knowledge that 970 psig 23 F 2 X c M and 20 inches are perfectly normal levels to be at post scram.

Licensee made requested changes with rev1 and stem needs minor edit to ensure S EOP-1 is not argued as correct (ie distracter B). Add after following in the stem current plant such that it is clear mode sw is out of run and question is now Sat.

CFR 55.41.9 U

LOD 1: It should be considered very basic knowledge that EOP-1 is directed to be entered to ensure that a shutdown has occurred. There are no conditions given to 24 F 3 X b N cause a candidate to potentially determine that EOP-1 should be entered for suppression pool cooling or pressure/level adjustments.

S Licensee wrote new question rev1 an question is now Sat.

U CFR 55.41.10 LOD 1: Distractors A&B are not credible. It is not plausible to believe that initiating 25 F 2 X X d N suppression pool cooling would allow for local temperature determination or affect containment activity levels.

Distractor C is partially correct if, as stated in the explanation, that reject may Page 5 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only eventually be required.

S Licensee made requested changes and added better justification to explanation section, rev1 an question is now Sat.

CFR 55.41.10 U

This is Unsat for KA mismatch because it does not match the reason you SCRAM with high SP level. If B were correct then it would be a match to the KA. It is a good basis question but not for this KA.

26 F 2 X a X B Distractor B is not credible. It should be considered very basic knowledge that EOP-2 is still applicable when the reactor is shutdown.

Still need answer A reworded Ensure adequate margin for SRV operation to prevent tailpipe damage.

S Licensee made recommended changes and question is now Sat with rev1.

CFR 55.41.10 RBS 2012 NRC Exam.

U Cueing makes this question unsat. The stem has area temperature in it and the only choice with area temp in it is the answer. D is not a credible distractor either.

Cueing: the stem directs the operator to monitor high area temperature, and the answer is to use an Area Temperature recorder.

27 H 3 X X d B Question is not Comprehension/Analysis, it examines fundamental knowledge.

Distractor D is not credible. The stem states that the CRS directs the operator to monitor and trend. This direction suggests a control room recorder is to be used.

S KA was resampled since couldnt write a good question for it.

Question is now Sat with rev1 and new KA and bank question from 2012 exam.

28 H 3 X d N S CFR 55.41.7 29 H 2 X c B S CFR 55.41.7, RBS NRC Exam 1997.

CFR 55.41.10 E Distractors need work.

Distractor C is not credible. There is no reason to believe that suppression pool level would affect pump bearing temps even if they were cooled by system flow.

30 F 2 X X b N Distractor D could be considered a second correct answer as it is a true statement that presents a reasonable answer to the question.

S NRC wrote new question as rev1 and it is now Sat.

31 H 4 X a N E CFR 55.41.7 Page 6 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only To avoid cueing, remove return to the suppression pool from the stem. State only that LPCS is running in the test mode at 5050 gpm. Allow the candidate to show knowledge of the flow path in test mode.

The second part of the stem needs clarification to ensure that the candidate understands that the leak to the drywell occurs after LPCS is already running in test mode.

S Licensee made changes and question is now sat as rev1.

CFR 55.41.8 RBS-NRC-577a bank, but not used on NRC exam.

U This is LOD = 1, too easy .Also this is GFES knowledge about pump operations. You could apply this question to most systems and it would be true.

32 H 2 X c B LOD 1: Question has no discriminating value and examines a very basic, fundamental understanding of HPCS. There is no reason to believe that pump flow would increase upon closure of the injection valve. Distractors A&C are not credible.

S Licensee found a different bank question and it is now Sat as rev1.

33 H 2 c N S CFR 55.41.10 CFR 55.41.10 U LOD = 1, too easy. Distractor D is not credible and it is too obvious as written that SLC A is not available. This makes the question unsat. Why not use the alarm in the stem for the breaker trip of the available pumps suction valve and require a cross connect operation through the other pumps (ie the B pump) suction valve? Also, using the bus power supply loss is more challenging than just a loss of the DG.

34 H 4 X b N LOD 1: Question only functions to examine if candidate can determine that SLC PP B has power when the diesel is on the bus - this is very basic knowledge. Distractor D is not plausible with power at 38%. Distractor C is not plausible with the basic understanding that SLC PP B is available.

S Wrote new question and question is now Sat.

35 F 2 d N S CFR 55.41.6 U CFR 55.41.2. HANDOUT.

Cueing from stem to answer (word bypass) , therefore it is unsat.

Also too easy as written.

36 F 2 X X a N Cueing: The question directs the operator to bypass the IRM. The correct answer is the only one of the four with a bypass switch involved.

The stem has too much extra information. All the question asks is how to bypass an S IRM. This knowledge is fundamental and provides no discriminating value. Page 7 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 8 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC wrote new question and it is now Sat as rev1.

CFR 55.41.2 Nov 2010 audit exam.

E You have the wrong letter marked for the proposed correct answer. A is correct not 37 F 2 a M C.

S Licensee made correction (rev 1) and it is now Sat.

CFR 55.41.6 E C and D not are not credible, are not part of the stem focus for the KA, and because they are repairable with edits it is an E. If you make a 2 X 2 with A and B and reword the stem slightly it will be a good question.

What does having all the SRMs full out in the stem have to do with the IRM s on 38 H 2 X X b M range 8 or above in the answer (ie is this needed in the stem to get to the answer)?

Distractor C is not plausible. The fact that a bypassed SRM will not give any type of rod block or scram signal is a basic, fundamental understanding of the term bypass.

S Licensee made recommended corrections and rev1 is now Sat.

E CFR 55.41.2 The stem of the question provides a systems-level overview of the RBM flow units and purposes of the core flow instruments. This is unnecessary. Distractor D is not 39 F 2 X X c N credible as steam/feed flow mismatch has no plausible relationship to core flow.

S Made recommended changes with rev1 of the question and it is now Sat.

40 F 3 b N S CFR 55.41.8 CFR 55.41.3 E

41 H 3 X b N A and B same length using manually open.

S Question is now Sat with rev1.

CFR 55.41.8 E Comment on the stem-need to be clear that the operator completed inhibit ADS as directed.

42 H 3 X X a N Why are the names you use here for manual initiation different than those in Q41 S (ADS B man initiation pushbuttons versus Div 1 manual ADS pushbuttons)?

Recommended changes made rev1 and is now Sat. Page 8 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 9 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only U CFR 55.41.3 This is answering the KA for knowledge of the reason for redundancy not the design or interlock feature which provides for redundancy. KA mismatch.

43 F 2 a X N You could ask the power supplies for the two different MOVs of a pair, accumulators for MSIVs, inboard versus outboard design criteria, etc.

LOD 1: Question tests very basic knowledge and has no discriminating value.

S Licensee wrote new question rev1 and it is now Sat.

CFR 55.41.14, RBS Bank not used on NRC exams.

U Is the SRV that stuck open the tailpipe temp you are monitoring? The stem does not say this. Also, in the stem you have ___ and will then slowly fall but distractor B has the temp start at 285F and goes up, which doesnt match the stem info and is therefore a non-credible distractor. This is stuff taught mostly in GFES.

44 H 3 X X b B The grammar of the question stem and the distractors do not match. Further, the correct answer states that temp starts at 285 and rises to 330, however the stem states that temperature will then slowly fall. Distractors appear to be randomly S chosen with no basis for why 330 would be selected as the initial temperature.

Licensee wrote new question and as rev 1 is now Sat.

45 H 3 b N S CFR 55.41.4 E CFR 55.41.7 46 H 3 X c N Stem: loss of RSS#1.

S Made correction and it is now Sat.

CFR 55.41.5.

U This is a good question if you dont provide a handout. All information you have in the answer choices should be known from memory. The note in this AOP to not parallel a DG to offsite power if it is not stable should be known from memory. The fact that P=IE and pumps need the same power means that they should know that amps go up if voltage is low and therefore this should also be known from memory.

47 H 4 X b N The synchroscope note also would require them to know from memory that an emergency start of the DG is required, so no handout should be allowed. If you gave them the handout, it would be a direct look-up question (where LOD=1) and it would not be allowed on the exam.

Licensee agreed to not provide the handout. Question is Sat with rev1 and with no S

handout.

E CFR 55.41.7 RBS audit exam 2008.

48 H 3 X X b B Remove normal at from the stem in two places unless this is actually how the Page 9 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 10 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S information is displayed on BYS-INV01A.

Made change and question is now Sat as rev1.

CFR 55.41.10, RBS bank, not used on NRC exam E How is it credible to have all lights OFF and be performing the test from the control room? What indication would you be looking at to complete the test if all three lights were off? This makes distractor A not credible. Also, since Green light OFF is in all four selections it is not relevant to the choices and per NUREG-1021 should be 49 F 2 X d B moved to the stem.

Licensee made changes to the stem to ensure it is known that it is done locally, not in the CR, but monitored in the CR, and this gives credibility to the A distracter. Kept Green in all four choices since it adds balance to the discussion and should be left in the answers in this case.

S Question is now Sat as rev1.

E CFR 55.41.8 The stem grammar and answer choices do not match. If it is intended that the 50 F 2 X d N answer be correct sequentially, then the stem should somehow state so.

S Made recommended changes and question is now rev1 and is Sat.

CFR 55.41.7, RBS bank, not used on NRC exam.

U Question does not adequately address the K/A. Question examines basic generator 51 H 3 c X M theory.

S Pulled different bank question then modified and with rev1 it is now Sat.

CFR 55.41.4 U

Distractors C and D are not credible with the same power for multiple compressors. Also, is it reasonable to use an MCC for the IAS compressor power distractor? It makes more sense to make them all the same service and vary the 52 F 3 X a N letters, such as NJS-SWG- 1F, 1G, and 1H, then NJS-SWG-1G, -1H, and 1F, then -

1E, -1J, -1K, and lastly -1J, -1E, and 1K or something like that.

Licensee made all four NJS-SWG with different letters as suggested and question is S

now Sat with rev1.

E CFR 55.41.7, RBS Bank, not used on NRC exam.

C and D distractors have non-credible sample panel in them. To fix this use the following selections:

53 F 3 X a B A. Recirc pumps, RWCU pumps, RWCU NRHX B. RHR pump seal coolers, RWCU pumps, RWCU NRHX Page 10 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 11 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S C. Recirc pumps, RHR pump seal coolers, Fuel Pool Coolers D. Fuel Pool coolers, RWCU pumps, RWCU NRHX Question is now Sat as Rev 1 with recommended changes.

CFR 55.41.6 E

Distractor A is not credible. The thought that isolating a CRD electrically would allow 54 H 3 X d N rod motion is not plausible.

S New distracter for A and question is now Sat as rev1.

E CFR 55.41.6, NRC Exam Dec 2008 55 H 3 b B Minor Explanation error on A - it needs a typo corrected (ie rod w/d blow ).

S Made change and it is now Sat.

CFR 55.41.2, NRC Exam 1995.

E Distractor D has negative cueing in it with the words an imminentRemove and 56 F 2 X c B make distractor D read:

D. adds negative reactivity in anticipation of reactor scram S

Made changes and question is now Sat with rev1.

57 H 3 d B S CFR 55.41.2, NRC Exam Dec 2008.

CFR 55.41.7, NRC Exam Feb 2003 E

Distractors B&C are too similar and can easily be eliminated by the candidate. Need 58 F 2 X a B to replace one of the two distractors.

S Changed distracter C to fuel zone recorder and question is now Sat as rev1.

CFR 55.41.8 U Stem reads awkwardly - need to separate the two parts of the question.

59 F 2 X X c N Distractors B&D are not credible. It is not plausible to think that a train of RHR would have a RHR pump and line fill pump powered from separate divisions.

S Licensee modified stem and two distracters such that off of same division with rev 1 question it is now Sat.

CFR 55.41.9, RBS bank, not used on NRC Exam U

Cueing in stem with word consume and answer igniter. None of the other systems 60 F 3 X X b M actually consume H2. Need to modify stem to take out the cueing word consume.

S Licensee modified question to remove cueing with rev1 and it is now Sat.

61 H 3 c B S CFR 55.41.5, NRC Exam June 1995 Page 11 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 12 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only CFR 55.41.5, RBS bank, not used on NRC exam U

KA match incomplete: You did not address the second part of the KA. For those questions where it cant be written to both, the NUREG states to write the question to the more challenging part of the KA (ie the b part). In this case of this question, it can 62 H 2 a X N be written to both and you can slightly modify this question to address what action would be taken to fix issue and therefore the question.

LOD 1: Question examines a very basic understanding of the EHC system.

S Accepted NRC written new question, which is Rev 1, which is now Sat.

63 H 3 a N S CFR 55.41.11 CFR 55.41.7 RBS bank, not used on NRC exam Please add units after 1.68 in answer B. Also, make format the same in all four E choices by adding return to distractors C and D to make them look like A and B.

To make this a balanced 2 X 2 make the choices the following:

A. (1) High negative D/P between.

(2) RPV level 1 (-143) after 60 sec time delay 64 F 2 X b M B. (1) High Drywell D/P (1.68 psid)

(2) RPV level 1 (-143) after 60 sec time delay C. (1) High negative D/P between.

(2) RPV level 1 (-143)

S D. (1) High Drywell D/P (1.68 psid)

(2) RPV level 1 (-143)

Licensee made suggested changes and as rev1 it is now Sat.

CFR 55.41.11 U LOD 1: Question examines very basic knowledge of control room ventilation.

65 H 2 X c N Distractors A&B are not credible. It is not plausible to think that outside intakes would open when control room HVAC goes on recirc.

S Rewrote question and adjusted explanations and with rev1 of the question THAT Tom wrote it is now Sat.

66 F 2 d N S CFR 55.41.10 E CFR 55.41.4, RBS NRC 2010 Exam 67 H 2 X b B Distractor C is not credible. There is no plausible reason to believe that the A ENS S switchgear would swap its source to another division based solely on a turbine trip. Page 12 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 13 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Created new distracter C and question is now Sat with rev1.

CFR 55.41.10 parent from RBS 2010 NRC exam U Reference states that Direct communication shall be maintained between the main control room and refueling platform personnel. Distractors A&B could be considered refueling platform personnel. Second reference states that a duty of the Fuel 68 F 4 X d M Handling Supervisor is to Maintaining constant communications with the Control Room during core alterations. This could mean that these are at least partially correct based on current wording of the question and answers.

S NRC wrote modified question from 2010 RBS NRC exam and with rev1 of this question it is now Sat.

69 F 2 c B S CFR 55.41.10, RBS NRC 2010 Exam CFR 55.41.10 U What can an RO authorize in the plant for equipment configuration changes? I am not sure that C is credible because of this.

70 F 3 X b N Distractors A&D are not credible. It is not plausible to believe that relying on a call-S back before the end of shift is acceptable configuration control.

Licensee fixed the issues above and the question is now Sat with rev1.

CFR 55.41.10 E

A is not credible given the stem conditions Distractor A is not plausible with switch in shutdown.

71 H 2 X a N Question examines fundamental knowledge - no analysis is required, therefore the question is marked as F not H as you have in the worksheet.

S Licensee accepted NRC written question and is now Sat as rev1.

72 H 3 c B S CFR 55.41.10, RBS NRC 2003 Exam 73 F 4 d B S CFR 55.41.10, RBS Bank, not used on NRC exam CFR 55.41.10, RBS Bank, not used on NRC exam U

This does not make sense to have a person on the fire brigade assigned to areas outside of the plant (ie JIC), making D not credible. This is an LOD=1 question as 74 F 2 X c N written. A better question is the responsibilities of various crew members during shutdown outside the control room, or which Aux Operators go to which locations/which tasks during extended blackout conditions, etc.

S Licensee agreed to use NRC new written question for this one, rev1 and it is now Sat.

75 F 2 c N S CFR 55.41.10 Page 13 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 14 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only CFR 55.43.5, RBS NRC 2010 Exam E

Per SRO only guidance this is knowledge of entry conditions for an AOP which is RO knowledge. I am NOT marking it unsat because it was previously approved on an NRC exam, albeit prior to the SRO-only guidance coming out. It must be changed 76 H 3 b X N completely to meet the new SRO-guidance.

Question also is LOD 1 and has stem cueing that directs the candidate to answers with 125 VDC.

S Licensee modified NRC proposed question and it is now Sat as rev2.

E CFR 55.43.5,RBS Bank, not used on NRC exam Need to structure the question around the SRO knowledge required to answer the 77 H 3 b X M question, which uses enclosure 26 flowchart to go to the next enclosure most appropriate for the given plant conditions.

S NRC modified question and is now Sat as rev1.

CFR 55.43.5 U

GG Bank used on GG 2013 audit exam, not used on NRC exam 78 H 2 a B LOD = 1, not allowed on the NRC exam because it is a direct lookup with reference provided.

S NRC wrote new question and is now sat as rev1.

CFR 55.43.5 E

Put 43.b.5 in worksheet (blank right now)

Stem needs to be clarified. Second sentence of first paragraph does not read clearly.

79 H 3 X b N Dont need reference to answer this question and with reference it helps in answering Q14 therefore No reference is allowed.

S Question is now Sat as rev 1.

CFR 55.43.5 E

You didnt perform the action you stated in the comments field to revision to incorporate procedural choice. into this question, therefore as written it looks like 80 H 3 X c X N RO knowledge of adequate core cooling can be used to answer this question.

Is distractor A correct for the time period when RPV level is -190 to -200? Adequate core cooling by steam cooling without injection exists when level is > - 200.

S NRC modified to make it SRO knowledge and is now Sat as rev1.

U CFR 55.43.5 81 H 2 d X N Direct look-up for even an RO. If you give them Att 1 of AOP-52 the only train available for the stem condition is RHR-A, which is only in answer B, which makes Page 14 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 15 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only this not SRO-only (no procedure selection is needed to answer this one) and LOD=1 S and not allowed on the exam.

NRC wrote new question and licensee didnt like so they used the NRC format and made a slight change to location for the fire and the question is now Sat as rev2.

CFR 55.43.2 U This question can be answered with RO knowledge of TS. You need to get to information below the line in TS to reach SRO-only knowledge.

Distractors B, C &D are not credible. There is no plausible reason to believe that the offsite power source is anything but inoperable. For B&D, there is no second aspect to the question that needs to be analyzed - if something is operable, then it is 82 F 3 X a X N obvious that no TS entry is required.

Question is not SRO-only unless there is an application of TS LCO requirements below the line. With the way the distractors are constructed, this question basically asks whether it is TS or not, which is RO knowledge.

S NRC wrote new question and licensee edited the new question and it is now sat as Rev2.

CFR 55.43.5 U BY knowing that three rods are out therefore reactor will not be shut down under all conditions without boron and this is also entry conditions for EOP-1A, both RO knowledge items, you can answer this question, therefore it is Unsat.

83 H 2 X b X N Distractors C&D are not credible. C states that the reactor cannot be determined to be shutdown, but that ATWS actions in EOP-1A are not required. D states that the reactor will remain shutdown, but directs ATWS actions anyway. These two S scenarios are not plausible.

NRC wrote new question and it is now Sat as rev1.

84 F 3 d N S CFR 55.43.2 U CFR 55.43.2 This one needs work to meet SRO-only guidance. RO is required to know EOP entry conditions and knowing that H2 is above 0.7% is entry conditions for EOP-2, so I can throw out B for sure and probably D as well since we are not near severe accident conditions yet and nothing else has been indicated to be tried in the stem conditions.

85 H 2 X X c X N This leaves only A and C for choices, both of which are EOP-2 and therefore no procedure selection is needed. See example IV in the Satisfactory SRO question guidance.

Distractor A is not plausible. How have operators determined that H2 concentration is 2 percent without H2 monitoring systems in service?

Distractor B could be considered correct if it is read to say continue in EOP-1 Page 15 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 16 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only (continue cooldown and depressurization) then initiate shutdown cooling (once interlocks are clear). This would not be an incorrect course of action even though H2 igniters should also be started. Answer C is the better answer, but B is not necessarily incorrect.

S NRC modified the question and as rev1 is now Sat.

CFR 55.43.2 U Can you manually/locally operate the RHR-A SDC suction valve E12-F006A, in Mode 4 with a leak in the drywell? If not then A is not correct and you need to work on the question. You also need to work on the words in the explanation. First sentence after 86 H 2 X X a N the semicolon has opening and closing together in the same sentence.

Distractors B&D are not credible. Manually closing the suction valve is the obvious answer. It is not plausible to believe that one could inject with the SDC suction valve S off of the recirc loop partially open.

NRC wrote new question rev1 and is now Sat.

CFR 55.43.5 2004 NRC exam.

E Per SRO only guidance this question can be answered knowing system knowledge and given the loss of RPS bus in the stem, but because it was on the 2004 NRC exam prior to the SRO-only guidance being released it is NOT being marked as Unsat but will require fixing or replacement. Also, entry conditions for an AOP are RO knowledge. It must be changed completely to meet the new SRO-guidance.

Another way to look at this question is this:if I know (and an Ro would know this) that no rods scram, then I can pick answer D because no other choice has this in the 87 H 3 d X B answer. I also missed this on the free review, so this is another reason why it is not marked as unsat for the submittal.

When there are only two RPS busses and you lose one, it is not plausible to believe that 1/4 of rods will scram. Distractors B&C are not credible.

Cueing in the stem leads the candidate to a loss of RPS bus procedure. Question:

What procedure do you enter for a loss of RPS Bus? Answer: Loss of RPS Bus.

Question is not SRO-only. Question can be answered with systems knowledge and S knowledge of AOP entry conditions.

NRC wrote new question rev1 and is now Sat.

U CFR 55.43.5, RBS bank audit exam, not used on NRC exam.

Why would a control rod block require a plant shutdown per TS? This affects 88 H 3 X b X N credibility of distractor C. RO knowledge will allow elimination of distractor A and D so this makes this question unsat for RO knowledge only required to answer.

Question is not SRO-only. All that is needed to answer the question is basic RO knowledge of APRM trip coincidence and the fact that you can bypass one APRM in Page 16 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 17 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only each division (Tech spec above the line knowledge).

Is there anything wrong with APRMs B&C or do they just read slightly <5% while others read slightly >5%? This needs to be made clearer.

With two answers containing Control Rod Block, it cues the candidate to selecting one of the two. Need 4 different conditions (scram, half scram, rod block, nothing) or 2/2 taken twice.

Tech specs need to be added as a reference if one of the answers has information S from below the line.

NRC wrote new question rev1 and is now Sat.

CFR 55.43.5 U Distractors B&D are not credible: Operators will have seen this situation (MSIVs closed with condenser not available) many times during their simulator training. It is 89 H 3 X X c N immediately obvious that stabilizing pressure with SRVs is the action to be taken.

S Knowledge of the specific setpoints for one SRV can be considered minutia.

NRC wrote new question rev1 and is now Sat.

CFR 55.43.2 First of all, the words in the TS you copied provides cueing-not your fault, but anyway U if you use unit in answer A then you need to use unit in the other three choices instead of plant to eliminate the cueing. Secondly, this is too easy for an SRO question. What does complete loss of D/G power mean? Either you cant power a vital bus or you can, whether by a D/G or not. Lastly, this could be challenged on 90 F 3 X a X N appeal with different words so you need to write something different on the LCO bases and offsite power, maybe evaluation of degraded voltage conditions and proper LCO reasons behind it would be the basis question you go after. Just a suggestion.

S LOD 1: Question presents material that should be tested as part of the stem. The answer is easily recognizable. Distractors C&D are not credible.

NRC wrote new question rev1 and is now Sat.

U CFR 55.43.5 You still need to work on procedure selection in the distractors. What other procedures would help the distractors be more credible besides a reactor trip or staying in GOP-0001? If there is another reactor engineering AOP that might be a 91 H 2 X X d X N good choice? Right now with RO knowledge you know the condition to enter AOP-61.

What if the required position was from 04 to 00 position, then C is a second correct answer but we still havent done a good job of having an applicant demonstrate procedure selection with SRO knowledge.

Distractor A is not credible. It is not plausible to believe candidate will determine a Page 17 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 18 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only scram is required with a single rod 4 notches out of sequence.

Question is not SRO-only. Answer can be determined with basic RO systems S knowledge and knowledge of AOP entry conditions.

NRC wrote new question rev1 and is now Sat.

CFR 55.43.5 HANDOUT.

U Still dont have procedure selection in the choices(all are the same procedure) -still RO knowledge.

92 H 2 X d X N Distractors present the question as being either F001 or F004, when the real question is F004 or both F004 and F001. Transferring resin to the holdup tank is not a plausible answer. Securing RWCU could not possibly be considered wrong in this condition, so it must be the correct answer.

S NRC wrote new question rev1 and is now Sat.

CFR 55.43.5 U

No procedure selection in the choices -RO knowledge. EOP bases is RO knowledge per SRO-only guidance.

High radiation levels will have no impact is not a plausible distractor.

Does the stem provide any conditions that would suggest that ED should be 93 H 3 b X N considered?

Is it really possible for the RWCU pump room to get to 210 degrees in this situation?

The EOP basis aspect to this question is too weak to consider it SRO-only. An RO can tell you that high radiation levels have an impact on personnel access and equipment operation.

S NRC wrote new question rev1 and is now Sat.

CFR 55.43.2. Dec 2010 NRC exam.

E Marked as edit because it was previously approved on NRC exam.

94 H 4 b B LOD 1: Direct lookup of the reference provided. No analysis is required.

S NRC pulled bank question from 2014 CNS NRC exam, question rev1 and is now Sat.

E CFR 55.43.7 NRC 2012 Exam Even though approved on previous NRC exam this is not discriminating at the SRO level. Only one procedure has a general title that could encompass ALL requirements 95 F 3 c N for fuel handling, and this is the answer. This is clearly cueing from the stem to the answer. Certainly not refuel logs, reactor vessel disassembly, or refueling platform operations would be selected (since it doesnt deal with fuel building moves) because none of these match the ALL in the stem with refueling moves. This must be replaced but it is not marked as Unsat because it was previously approved for an NRC exam. Page 18 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 19 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S NRC wrote new question rev1 and is now Sat.

CFR 55.43.3 96 F 2 d N S Not an F3 question as marked but an F2 level question.

CFR 55.43.1 RBS Bank 2010 audit exam, not used on NRC exam. HANDOUT U required with NRC written question to replace this question.

Not SRO-only: Generic LCO statements constitute RO knowledge. Knowledge of 97 H 4 c B the basis is not required to answer the question.

S LOD 1: Knowledge required to answer the question is elementary.

NRC wrote new question rev1 and is now Sat.

CFR 55.43.5 E

A is not credible because how can a value that requires entry into EOP-3 be listed 98 F 3 X d N inside the details of EOP-3? This makes A not credible.

S NRC wrote new question rev2 and is now Sat.

U CFR 55.43.5 RBS Bank 2012 audit exam, not used on NRC exam This is a trivial use of this KA with level of difficulty of 1. For an SRO it needs to be 99 H 3 a N deeper knowledge than the routine trip procedures. The SAMGs would be a good question or different procedures used with an EOP attachment or something like that S NRC wrote new question rev1 and is now Sat.

CFR 55.43.5 RBS 2007 NRC Exam E LOD 1: Question examines very basic knowledge of the E-plan PAR process and 100 F 4 c B provides no discriminating value. Not marked as unsat because it was previously S approved on the 2007 NRC exam but it must be repaired or replaced.

NRC wrote new question rev1 and is now Sat.

B= 25 (33.3%) F= 37 (49%) E= 25 RO TOTALS: M= 10 (13.3%) H= 48 (51%) U= 28 Additional Notes:

N= 40 (53.3%)

B= 5 (20%) F= 7 (28%) E= 9 SRO TOTALS: M= 1 (4%) H= 18 (72%) U= 14 Additional Notes:

N= 19 (76%)

GENERAL COMMENT

S: Page 19 of 20 OBDI 202 - IOLE Process

ES-401 RB-2014-12 20 Form ES-401-9

1. Bank / Mod / New questions are indicated in their respective column field above.

2. Chief Examiner comments are indicated in blue where multiple comments exist, otherwise they are in black.

3. Average difficulty is __ 2.65 ___ on the RO exam and _2.80__ on the SRO exam.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 0 41.2 = 0 43.2 = 2 41.3 = 1 43.3 = 1 41.4 = 1 43.4 = 4 41.5 = 7 43.5 = 16 41.6 = 6 43.6 = 1 41.7 = 36 43.7 = 1 41.8 = 2 41.9 = 2 41.10 = 15 41.11 = 0 41.12 = 3 41.13 = 2 41.14 = 0

5. The answer distribution is: RO / SRO A = 19 (25.3%) / 5 (20%)

B = 18 (24%) / 7 (28%)

C = 22 (29.3%) / 6 (24%)

D = 16 (21.3%) / 7 (28%)

6. There are 4 questions with attachments provided. Questions 12, 36, 92, and 94 require hand-outs Page 20 of 20 OBDI 202 - IOLE Process