ML102920654

NRC Comments on Draft Licensee-Developed Exam (Written & Operating Tests) (Folder 2)

ML102920654
Person / Time
Site:	Limerick
Issue date:	09/01/2010
From:	Caruso J Operations Branch I
To:	Exelon Nuclear
Hansell S
Shared Package
ML100560247	List: IR 05000352/2010302 ML101650598 ML102720677 ML102870021 ML102920654 ML102920658 ML102920661 ML102920664 ML102920667 ML102920669 ML102920673 ML102920676 ML103080887
References
TAC U01767
Download: ML102920654 (29)

Text

General Comment for all Questions: 1) List justification statements in order of how they appear in question; 2) Provide KIA statement along with KIA number to save man hours for reviewers to look up in KIA catalogue; 3) List reason why each distractor is plausible but incorrect it saves time reviewing and saves man hours in addition helps developer to ensure distracters are adequate.

How many licensed SRO and ROs validated this exam? Initaliy 2 SROs and 2 ROs and then later after revisions were made revalidated with lSRO and 1RO. Did you pre-brief the validators to make sure the distracters were plausible but incorrect? Yes Resolutions listed in Bold and Italics print ES-401 Written Examination Review Worksheet Form ES-401-9 1_ 2. 3. Psychometric Flaws 4. Job Content Flaws 5. other 6. 7_ B.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #f Back- Q= SRO 8/M/N UtEfS Explanation Focus Dist. Link units ward KIA Only

1. H 2 B S Q 1 - Ok

2. H 3 X N E 02 -1) tighten stem. Done

3. F 2 X X B E 03 -1) LOD=1-2 borderline; 2) just recall of max subcritical bank position and allowed to have most restrictive rod stuck out; 3) A&B distracters not plausible with 2 rods completely out of core; This question has overlap with SRO Q 25. Basically both questions are testing the same knowledge. Q rewritten to increase LOD and plausibility of distracters. SRO Q 25 replaced.

4. F 2 B E 04- 1) Q stem should be tightened - How HPCI &

RCIC will function based on what operator action or auto response ?; 2) It is confusing what 0 is asking RCIC manual by operator? By controlling what steam flow or discharge valve? Simplified stem and answer choices.

5. F 2 N S 05- 0 k

1. 2. 3. Psychometric Flaws 4. Job Content Flaws I 5. Other I 6. I 7. I 8.

I Q# LOK (F/H)

LOD (1-5) I Stem, Cues Focus

~

T/F I I cred'l Partial JOb-I Minutia #1 Dist.

Link I I units ward KIA Only I

Back-l Q= SRO I B/M/N I U/E/S I Explanation

6. I F I 1-2 I I I I I I I I I I I I N I E I Q 6 - 1) Borderline LOD. 2) Explain why distracters plausible but incorrect. C and D do not appear to be plausible because DW components are going to isolate on Hi DW pressure. Modified stem to make more challenging did not indicate that DW pressure went above 1.68psig

7. I F I I2 Ix I I I I N E Q7 -1) Stem could be simplified; 2) distracters B&C don't appear plausible. Question asking why reactor pressure is lowered and those distracters talking about SP temp? Modified stem and distractors

8. I I I F 2 I I I N U 08 - 1) Explain why C&D distracters and reasons plausible? Modified distracters improved KIA match Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.)

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated truelfalse statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

The question contains data wlth an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or {S)atisfactory?

n

8. At a minimum, explain any *U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only

9. F 2 X N E Q9- Explain why A&C distracters are plausible reasons for isolating a tripped pump? Revised stem and answer choices

10. H 3 X N E Q 10- Explain why B&D are plausible to allow level to get to the top of the fuel racks on bellows leak?

Revised B&D to improve plausibility.

11. H 2 X B E Q 11 - Explain why would "A&D" be plausible on a Unit 1 Hi Rad? Revised stem and distracters to focus on Unit 1

12. F 2 B S Q 12 - Ok

13. H 2 X B E Q 13 - C&D appear to be implausible to have Rx temp lower on a loss of SID RHR cooling?

Stem revised from valve closed to instrument line break to bypass valve improving plausibility for A&D.

14 H 3 N S Q 14 -Ok 15 H 3 B E Q 15 - A&D are very close in wording appears to be same choice? ADS valves vice all SRV's. TH. "A" does not make sense. It implies to allow operation vice operator taking action. Auto will go on set pressures and will not rotate around. Revised answer choices to clarify intent.

16 H 2 N S Q 16-0k 17 F 2 X N E Q17- 1) Add breaker number for D23 feeder breaker for clarification. Added breaker numbers to stem to add clarity and focus.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO BIMIN UlEtS Explanation Focus Dist. Link units ward KIA Only 18 H 2 X N S 018 - Why is the fire guide being provided as a reference? 04 has overlap. 04 states MSIVs are shut in CR evac this cues bypass would not be avail on CR evac helps to eliminate A&B choices. Fire Guide removed as a supplied reference and Q4 revised to avoid overlap.

19 F 2 N E 0191) This 0 needs work. The question is what actions per E-5 are required, but the reference given to the student is S32.3.A The correct answer is step 3.11 of E-5 which I do not think an RO needs to know from memory. 2) Do you have an RO LO that requires this knowledge from memory? Replaced with new question testing the same KIA.

20 H 2 X N E o 20 - Explain why C&D are plausible 525 seconds?

Why not use 420 seconds seems more plausible which would be correct for just low level without HI DW pressure? Distractors changed to improve plausibility.

21 H 2 M E 021- Recommend renaming 2nd column Vent Alignment. Modified stem order and renamed column.

22 F 2 B E 022- 1) Not really a modified 0 answer outcome the same. Changed to bank.

23 F 2 B S Q23-0k 24 H 2 X N S 024- 1) This seems deep in the EOP's for an RO question. 2} Do you have an RO LO that requires this knowledge from memory? Stem modified to focus on RO actions.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #t Back- Q= SRO BtM/N UlEtS Explanation Focus Dist. Link units ward KIA Only 25 H 3 N S 025- What makes 75.9 feet plausible? Seems like simply did not add 26.8 feet. This one might be better with 4 differen-Vt levels .. Also what is plausible about internal only if none of the levels put you in the unsafe region. Removed some imbedded info., changed to single column answer and distracters revised to improve plausibility.

26 F 2 X N E 0 1) Confused by stem "when selected

... indicates both control rods are moving slower than normal" ? 2) Are you implying that both rods are drifting out of the core? 3) The stem is not procedurally correct. With a rod drift alarm you would not press the rod drift reset. You would select that rod. The whole first section of the stem could be eliminated. It just confuses the situation.

Per the procedure step 2.2 if 2 rods are drifting out scram the reactor. Not dependant on reactor power. Stem simplified and focused. Actual rod numbers added to the stem.

27 F 1-2 X N u 027- 1) Explain bases for why A&B are plausible?;

2) Obvious set pOint question at 340F ED is required LOO borderline =1.

All distracters start with reactor shutdown ...

The C & 0 answers are saying the same thing. C is for normal but 0 just says before RPV depressurization. Need to say Emergency Blowdown to differentiate. Stem simplified and focused. Distractors changed to improve plausibility.

28 F 2 N S 028- distracters revised for readability.

~.

ES-401 2 Form ES-401*9

1. . Psychometric Flaws 4. Job Content Flaws 5. Other 8.

Q# I LOK (F/H) cred'l Partial I JOb-I Minutia I #I I Back-I Q= I SRO I B, Explanation Dist. Link units ward KIA Only 9-0k N I E I Q30-0k N I S IQ31-0k N I E I 032 1) Don't we need to know decay heat operating history to answer this question?

2) There is no mention of recirc pumps in the stem.

With recirc running there is forced circulation. The caution at step 3.0 says level must be greater than 60 inches(assuming upset level is LI-42-R605) to credit natural circulation as an alternative method of reactor coolant circulation. Add to stem No Recirc pumps running for clarification.

3) Do you have an RO LO that requires this knowledge from memory?

4) For this KIA being greater than 212 and having a loss of shutdown cooling would be a better way to test the knowledge of the pressure effects of a loss of shutdown cooling.

5) Caution can't credit natural circ with instrument tolerance answer may not be technically correct.

Stem rewritten so Rx pressurized.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

01 LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N UtE/S Explanation Focus Dist. Link units ward KIA Only

33. H 2 X B u Q33- 1) Please explain bases for distracters being plausible? e.g. are these answer based on math error traps?

2) Do not think it matches KIA and I would not expect RO's to know the formula for scram setpoints.

The sample plan KIA is K3.01 which is loss or malfunction. Question lists K4.07 trip biased setpoints.

3) Do you have an RO LO that requires this knowledge from memory?

Rep/aced with a new Q.

34 H 2 B S Q 34- Ok 35 H 3 N E Q35 - 1) The original question is ok. In general voltages should be matched

2) "A" distractor is not plausible to raise incoming voltage if incoming is already high. In the original Q the voltages were swapped so it made sense to raise voltage.

3) The procedural requirement is incoming 0 to 3 volts above running. I don't know if that is needed to be known from memory.

Question rep/aced with Sank Q.

36 H 3 N S Q-36-0k 37 F 2 N S Q37 -Ok 38 H 3 N E Q38-0k 39 F 2 X B S Q39 - 1) The wording of the question may need to be clarified. Stem wording clarified.

40 H 2 B s Q40- Ok 41 H 3 B S Q 41 - Ok

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

0# LOK LOD (F/H) (1-5) Stem Cues TlF Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N UtEtS Explanation Focus Dist. Link units ward KIA Only 42 H 2 X N u 042- 1) Please explain why CaD distracters are plausible with the conditions in the stem in indicating the SRVs are all closed - no indication of any open SRV.

2) C& D OT-114 states if a SRV was open would have you place both loops of cooling in service. By having the distracters only have one loop in service, it is not operationally valid. CaD should have both loops SP cooling in service.

Stem and distractors revised and clarified.

43 H 3 X N u 043- 1) Please explain why A&C are plausible when only need one ECCS pump to proceed to T-111, step 11?

2) SLC running in the stem that gives you one alternate pump running regardless of CS pumps so you have N+1 pumps. Also remove CRD, SLC, RCIC from stem.

3).May need to add one more electrical failure to get down to get down to one remaining pump so that distracters are more plausible .

4) The stem says the CRS has ordered vessel depressurization, therefore he is past the decision point about how many systems and pumps are running. .

He should be entering T -112 and performing concurrently. The answer that ED cannot proceed is out of place. The decision has already been made to do it. Recommend remove that statement from the stem

5) Consider reword of stem, << Given the above plant conditions: 1) What is the status of the Core spray system and does this satisfy the criteria in T-111 for performing an ED?"

Q replaced with a new Q testing same KIA.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #I Back- Q= SRO B/M/N UtE/S Explanation Focus Dist. Link units ward KIA Only 44 H 3 M S Q 44 - Ok 45 F 1? X N U Q45 - Please explain why C&D are plausible - what would causes pump{s} to stop running?

LOD 1 should replace Q Q replaced with a new Q testing same KIA.

46 H 3 X B S Q 46 - Ok 47 H 4 B s Q47 - Ok 48 F 2 N U Q 48 1) Would we expect an RO to know this without the tech spec. This is not a 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> or immediate sId TS.

2) C & 0 are saying the same thing - both are stating no LCO if you have one power supply. This Q needs work.

Q replaced with a new Q testing same KIA.

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only

49. F 1-2? X N E Q 49 - Stem replace "functions" with "power supplies to the We don't allow simple power supply questions.

U Low LOD this ok if it was one of 3-4 Low LOD Qs total on the exam. Q replaced with a Bank modified Q testing same KIA.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK (F/H) LO (1-5) OTIF Stem Cues m cre~Jpartial Job- Minutia #1 Back- Q= SRO BlMIN U/E/S Explanation Focus ~~ - -- --

Oist.

Link units ward KIA Only

50. H 3 N E Q Q is too easy with all of the info provided S increase the loading on the EDG in stem so "D" is correct. Stem modified to require operator to take action prior to placing pump in service distracters modified accordingly_

51. H 2 B E 051 - 1} Add Loops or Loop to answer choices and C&D add "core spray loop" Changes made.

52. F 1 X N E 052- 1) 0 too Simplistic LOD=1 doesn't test operator understanding. How about if one channel OOS then a channel failed. 3} distractor plausibility with one channel failed downscale.

Stem modified to include multiple failures.

Correct answer changed and recategorized bank modified.

53. H 3 X B E Q53 - 1) A & C don't appear to be plausible.

The aprm averages the Iprm inputs, and if an input goes down the channel will go down.

Stem modified to focus on indications RO would see and distracters modified to address stem changes and increase plausibility.

54 F 2 B S 054 - Ok 55 F 1-2 N U Q 55 -Simply knowledge that B power supply powers B RPS solenoid valves. LOD= 1. Q replaced with a new Q testing same KIA.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO BtMtN utEtS Explanation Focus Dist. Link units ward KIA Only 56 F 1 X N E Q56 -1) LOD=1 doesn't test operator understanding. Correct answer stands out. 2) implausible distracters e.g. B failed smoke detector never actuates system and C failed heat detector for a pre-action system? Clarified stem wording and revised distracters to make more plausible.

57 F 2 B s 057 -Ok 58 H 3 X B E Q58 - 1) explain basis for plausibility of distracters?

Plausibility of A & C. If the load limit is 94% why would the control valves go to 85.

Q replaced with a new Q testing a different KIA.

Revision??

59 H 2 M s Q59 - Ok 60 F 1-2 X B u Q60 - 1) Answer is obvious does not test operator understanding LOD=1.

2) B if your are trying to move the rod at elevated drive pressure without movement that would be indicative of a stuck rod.

3) B distractor and justification are backwards. One is elevated drive pressure and the other is low.

4) ) Doesn't meet definition of modified bank. Q stem is confusing?

Q replaced with a new Q testing a different KIA.

61 H 3 N E Q61 - ok - Modify the second part of the stem" ..and list the correct procedure the operators must enter?"

Change made.

--- ~~--

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# lOK lOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/MIN U/EIS Explanation Focus Dist. link units ward KIA Only 62 H 2 B E o 62 Like the original 0 better it actually tests the system response SP securing on a LOCA and LPCI initiation - the revised 0 only test % of system response. The KJa fits both versions. Used bank Q as suggested.

63 H 2 B S 063- Ok 64 F 2 N E 064 - Ok word "received" misspelled in stem. Fixed 65 H 3 X B E Q-65 1) with zero flow on outlet and rising pressure don't know how improving vacuum in A & Bare plausible Revised distracters to make more plausible 66 F 1-2 B S 0 Ok 67 H 2 B S 067 ok 68 F 2 X N U 068 Believe 0 is flawed with multiple correct answers stem too general. I think the 0 should be replaced. Replaced Q with New Q testing a different generic KIA

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. B.

Q# LOK lOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/MIN u/EIS Explanation Focus Dist. link units ward KIA Only

69. F 1-2 N U 069 - 1) I think this is a non license level PEO O. 2)

"D" looks like it could also be correct. 3) Stem cues correct answer in that string and missing tag alludes to equipment status tag. 0 may be ok but too many low LOD Os on exam about 3-5 max total on exam is goal.

Q replaced with a new Q testing same KIA.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues TlF Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N UlE/S Explanation Focus Dist. Link units ward KIA Only

70. F 2 B u 070 Appears to be an SRO level a see 55.43.3 and 55.43.5 referenced on O. Licensee staff stated RO level knowledge.

71. H 3 B S a 71 -Ok

72. F 2 X B a 72 - GET level. Distractors A&D may also be correct or partially correct? "C" seems obviously wrong. a may be ok but too many low LCD as on exam about 3-5 max total on exam is goal. Revised stem to operationally oriented non GET Q. Distractors were modified to address stem change and increase plausibility.

73. H 2 N S 073 - Ok 74 H 3 B S 074 - OK Changed to Higher order not memory level 75 F 2 B S 075 - Ok 76 H 3 X X N U a 76 - SRO 01. D is a second correct answer. All annunciators being denergized would include the subset in the stem.

o is not an SRO question. ON-122 states dispatch an operator to determine cause of loss of power. REFER to E-34 for troubleshooting. 55.43.b.5 requires selection of procedures. The procedure is given as a reference in the question and is not selected by the candidate. An RO could perform this step and should have the technical expertise to perform. Qreplaced with a Bank Q testing a different KIA.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

0If. LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #lsl~~ck- Q= SRO B/M/N U/EIS Explanation Focus Dist. Link units ward KIA Only 77 H 1 N U Q 77 - SRO Q2. The answer is a direct look up on the provided reference T-116 step RF-14. When 5 SRVs are open and RPV press is below 230 psig then continue. The next step is to slowly commence injecting. Modified Stem from stating loss of level indication, to including indications that must be used to diagnose. Changed Distracters to support answers where it is determined level is known. Added T-117 as a reference 78 H 2 X B E Q 78 - SRO Q3. Distractor D is not plausible because the TRM does not require a continuous fire watch.

Recommend 14 day LCO and then an hourly fire watch. Distractor C has a typo, should be lowered to 200psig. Distractor d has a typo. Should be watch instead of water. Also explanation d for D has the same typo. Comments incorporated.

79 H 3 X X N U Q 79 - SRO Q4. Technically Answer Band D are also correct. Narrow range 1R606C is +5 inches which is greater than MIL +3. Therefore per step 4.1 narrow range is usable. This would make Band D correct. The stem should not have RPV level is -100". The question writer used this fact to determine which instruments are usable. Where in a real situation the SRO would not know exactly what reactor level is and would need to determine which instruments are usable. Changed stem to remove information that conflicted with levels noted. Fixed typo in stem for level instrument.

80 L 3 M E Q 80 - SRO Q.5 Listed KIA is 295023 Refueling.

Question is a safety limit violation. KIA Mismatch.

Question 6 has same KIA. Should be KIA 295025 2.4.30 from sample plan. Fixed typo on KIA

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5, Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T1F Credo Partial Job- Minutia #I Back- Q= SRO B/M/N UlEtS Explanation Focus Dist. link units ward KIA Only 81 L 3 B S o 81 - SRO 06 Ok 82 L 2 X X N U 082 - SRO 0.7 The question lacks clarity. The stem does not mention what system or a TS number that is applicable. If the question was more focused then none of the distracters are plausible, The question seems to be testing if the candidate knows when a fuel leak happens which path (offgas) the elevated release is at and then what procedure is used. The isotopic off-gas procedure is the only one that measures off-gas.

Q rep/aced with a New Q testing a different KIA.

83 H 3 B E Q 83 - SRO 0.8 The question asks what action is required. The alarm response directs the operator to "consider". The wording should be improved.

Also 8&D list actual negative numbers rather than Greater than (positive) which is confusing? Stem revised to tighten focus. Distractors revised to improve plausibility.

84 H 2 X B U Q 84 - SRO 0.9 With suppression pool level at 23 feet, it appears that the suppression pool temperature limit is about 165 degrees. With temperature at 155 and slowly rising "8" is a true statement. It is acceptable. Also the correct answer D stating pressure MUST be maintained at or below 900 is not true. Also with the graph embedded in the a it is a DLO O. Maybe, we should provide T -102 as a reference as long as it doesn't cause other EOP type as to be DLO.

Imbedded picture changed to color attachment Change stem to place values on more restrictive part of curve. Distractors modified to improve plausibility,

1.

I LOK I 2. I 3. Psychometric Flaws 4. Job Content Flaws 5. Other I 6. I I

7. 8.

Q#

(F/H)

LOD (1-5) I Stem Cues T/F Focus Credo Partial Job- Minutia #1 Back-Dist. Link units ward I

Q= SRO B/MIN UlElS KIA Only Explanation 851 H 2 N E 10 85 - SRO 0.10. Made D HPCI and RCIC for consistency.

861 H 3 N E 10 86 - SRO 0.11. No KIA listed in documentation.

Added from sample plan KIA is 202001 A2.03 871 H I3 I IX I IN IE 10 87 - SRO 0.12. "B" is not a credible distractor.

If aU 4 SRM's are inoperable Placing one detector in the trip condition can not be the correct answer.

Revised "B" to be more plausible.

L I3 I IN IS 1088-SR00130K' S-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content 5. Other 6. 7. 8.

0# LOK LOD Flaws (F/H) (1-5) BIMI utEI Explanation Stem Cue T/F Cred Parti Job Minuti #1 Back 0= SR N S Focu s al Link a unit - KIA 0 s Dist. s ward Onl y

1. 2. 3. Psychometric Flaws 4. Job Content 5. Other 6. 7. 8.

Q# LOK LOD Flaws (F/H) (1-5) B/MI UlEI Explanation Stem Cue TIF Cred Parti ~ob- Minuti #1 Back Q= SR N S Focu s al Link a unit - KIA 0 s Dist. s ward Onl y

89 L 3 B E Q 89 - SRO Q14. No KIA listed in documentation.

From sample plan 218000 2.2.40 Also the reference to 55.43 b 5 is incorrect. B2 would be more appropriate.

The reference provided to the candidate are TS 3.3.3 and 3.5.1, however 3.4.2 and 3.5.1 appear to be the correct references. Comments incorporated.

90 H 3 X N E Q90 - SRO Q15 55.43 b2 is not appropriate. Should be 55.43.b5.

B&D - continue with startup would not be a plausible response to RPS trouble.

"8&0" revised and other comment incorporated.

91 H 3 N S Q 91 - SRO Q16 Not a 55.43.b6. Should be 85.

Ok 92 H 3 X X N U Q 92 - SRO 17. System question. Does not match 55.43b6? Not SRO level just system level Q.

KIA mismatch 2.1.20 is Ability to interpret and execute procedure steps. No procedure steps in question.

From sample plan 2.2.41 does match question.

Q replaced with a new Q testing the same KIA.

1. 2. 3. Psychometric Flaws 4. Job Content 5. Other 6. 7. B.

Q# LOK LOD Flaws (F/H) (1-5) B/MI UlEI Explanation Stem Cue TIF Cred Parti Job Minuti #1 Back Q= SR N S Focu s al Link a unit - KIA 0 s Dist. s ward Onl y

93 L 3 N S Q 93 - SRO 18 OK 94 H 2 N E Q 94 - SRO 19. No KIA listed.

From sample plan, 2.1.9 Ability to direct personnel activities inside the control room.

Should be 55.43.b5. B13 does not exist for SRO.

Changes incorporated.

95 L 3 B S Q 95 - SRO 20 OK 96 L 3 N E Q 96 - SRO 21 Step 4.B.4 states access will not be permitted to 313'. How can this situation arise if it is not permitted by procedure?

Typo "The fuel bundle is laying on "its" side in the transfer canal" Should B state In Drywell? If work was outside the OW then it may not be an issue.

Q rep/aced with a new Q testing the same KIA.

97 H 3 N E Q 97 - SRO 22 Ok to provide flow chart vice embedding in question. Changed to an attachment.

98 H 2 N S Q 98-SRO 23 OK 99 H 3 B S Q 99 - SRO 24 02 NRC exam. OK

1. 2. 3. Psychometric Flaws 4. Job Content 5. Other 6. 7. B.

Q# LOK LOD Flaws (F/H) (1-5) BIMI utEI Explanation Stem Cue T/F Cred Parti Job Minuti #1 Back Q= SR N S Focu s al Link a unit - KIA 0 s Dist. s ward Onl y

100 H 1 X X N U Q 100 - SRO 25. NOT SRO Need to add procedure selection to make 43.5 Technically disagree with answer. With one rod out the shutdown margin is adequate to maintain the reactor shutdown under all conditions without injecting boron. See Limerick definition.

This question has overlap with RO Q 3. Basically both questions are testing the same knowledge.

Q replaced with a new Q testing the same KI~"-

Licensee Submitted Exam to NRC COB 8/20. NRC started review 8/23/10

27 July 2010 Review Comments for T-75 Day Limerick Exam Outline Submittal The exam outline was hand delivered to the Regional office on Tuesday, 7/20 and comments were provided per telecom on 7/27/10. The outline was developed using a BWR Owners Group Random Generator software. The audit exam is being developed independently by a different staff person in a separate area with no communications in so far as the written exam. The NRC operating tests were developed with no overlap with the topics planned for the audit exam.

Scenarios The scenario outlines are all new. Requested that narratives be provided next week which include the proposed Critical Tasks for each of the scenarios.

Admin JPMs RO topic Shift Turnover Checklist will be designed for classroom administration - change "S" to uRn on outline.

RO topic to Activate Site Evacuation is planned for HI Rad situation with announcement suggested to include activation of emergency organization also SRO topic Shift Turnover Checklist A.1.1 will be designed for classroom administration change "S" to uRn on outline. This will include TS implications SRO topic Pre Control Rod Withdrawal Checks will also include TS implications.

Simulator JPMs Three proposed alt path JPMs involve securing equip due to hi vibs (b) or hi temp (c & e) and in addition only one new and one modified JPM which is at the minimum the rest from the bank suggested replacing RCIC JPM with a new alt path JPMs.

In addition, for the 5 JPMs designated for SROUs 2 of the 5 were from the last 2008 NRC exam.

Suggested replacing a or g.

Written SRO Outline The following KIA question topics did not appear to be SRO level topics: 1, 2, 3, 6, 8, 9, 10, 13, 14,17,94,95,98,96.

Written RO Outline Cautioned not to write questions involving simply asking what the power supplies are to a component directly ex. air compressors.

Limerick Operating Exam Review Comments Scenarios General Comments:

1. All required operator actions are required to be scripted for each event (Le., all switch manipulations, etc). Note: Verifications need not be scripted.

2. List required operator actions chronologically as they would be performed e.g. following a SCRAM the RO should report if all rods are not full in and start taking actions to insert these rods as appropriate.

3. For all CTs please consult NUREG 1021, Appendix 0, Simulator Test Guidelines, section 0.1 and list how each of the 4 criteria are being met for each CT (Le., safety significance, cuing, measurable performance indicators and performance feedback also failure criteria) in some cases I don't believe your listed CTs meet the criteria for safety significance.

4. Please highlight CTs and operator actions taken to respond and mitigate for each CT in the body of the script.

5. Please script in at the end of each scenario to the have the applicant in the role of CRS classify the event - please add the expected classification to the scripting.

6. In the body of the scenario guide please list the procedure title the first time it is referenced in each scenario e.g. T-213, 215, 216, 212, 221, 117, 270.

7. At the top of each event page please summarize expected failures- not required but helpful.

8. The scenarios overall lack diversity in that they short on both instrument failures and electrical failures that cause an integrated or dynamic plant challenge e.g. loss of safety busses or loss of off- site power.

9. Please enhance scenario summaries in front of scenario guides. Not required but it helps when reviewing the scenario just prior to administration. Typically scenarios have a narrative overview or executive summary towards the front of the scenario.

10. What is the origin of these scenarios are they new and or significantly modified?

11. Scenario 1, 3, 5 all involve a LOCA and exceeding PSP scenario appear to lack diversity e.g. no hi rad, fuel failure, offsite release or Loss of offsite power etc. Recommend replacing scenario #1 multiple EOP legs are not challenged and LOCA theme repeated on 3 of 5 scenarios.

Scenario #1 (LOCA, PSP)

1. Event 3 - list how much power is required to be reduced and how power would be expected to be reduced with just recirc or rods and recirc. Additionally, what is the plant response to this failure? Does the EHC backup regulator take control, and if so how much of a transient is it?

2. Event 3 - Is there a procedure that addresses EHC failures? Also the event describes maintaining pressure below 1053psig. Is there any chance a crew would fail to do this and the unit scrams. If that happened the next 3 component failures would be missed.

Might be better to have some of the single component failures done prior to this event if

there is a chance of a crew not being able to control pressure or choosing to manually scram to be conservative.

3. Event 3 - list what ARCs

4. Event 4 - RECW pump trips and start standby pump. Event 6 also CRD pump trips and start standby pump events should be more diversified. How about an instrument failure to replace on these malfunctions - e.x. RPV level controlling channel fails low so that affects FW control?

5. Event 5 - List all required operator actions for RO and BOP - no required actions to stabilize then no credit.

6. Event 6 - If sprays are still available, why will the plant exceed PSP? If PSP is not exceeded, then the crews will not ED and the second critical task will not be performed. The scenario would then only have one critical task and need 2 minimum.

7. Event 7 - Bold and highlight the critical tasks and the required operator actions to mitigate each CT.

B. Event 7 - List the critieria for spraying the DW and the valves that are opened. List the values when the PCP curve cannot be maintained and SE-10 actions.

9. Event 7 - D-1 looks like a LBLOCA not small break LOCA? Critical Tasks- The first critical task should not be before exceeding 340 degrees or 55 psig. Procedurally when suppression pool exceeds 7.5 psig. SP spray is required. The second critical task should also not be dependent on 55 psig. Suggest the criteria be if they exceed PSP for 3 minutes and have not taken actions to ED then they would not meet the critical task. Are you sure that you will have to ED, if the crew aggressively reduces pressure early on?

10. Simulator setup section. Do not see a failure for EHC pressure set.

Scenario # 2 (ATW 19%)

1. When was the last time Limerick administered a low power scenario 3-5% for an initial license exam? This scenario at 19% is not considered low power. We should have one low power scenario every other exam and we did not have on the last exam Oct OB.

2. The scenario does not have a D-1 outline form on the first page of it

3. Event 2 - would want to pull a handful of rods enough to demonstrate effective reactivity controls 5% change in Rx power.

4. Event 3 - List operator actions any back panel actions required?

5. Event 4- List all required RO and BOP actions? Are there any other actions required other than initiate CR HVAC chlorine isolation manually. If not need an electrical loss that involves integrated plant response and corrective actions. In scenario summary it states "once action for drifting rod is complete ... " I see no other reference to a drifitng rod on D-1 or in scenario guide.

6. Event 5 - Trip of both Recirc pumps - if this results in an auto P/F scram it would not count as a separate malfunction just a setup item for the major event ATWS? Even if not an auto scram (only at 19% power), the only actions are to scram the reactor which is unsuccessful. So I believe it should not count as a separate malfunction. It is set-up item for the ATWS. You need to evaluate if an additional malfunction should be added.

7. Event 6 -Implement TAR - define?

8. Event 6 - Bold and highlight the critical tasks and the required operator actions to mitigate each CT.

9. Event 6 - List all of the expected operator actions taken for T-213, 215, 216,212,221, 117, 270 in the body of the script and also list the procedure title the first time it is referenced in the script.

10. Event 6 - Script expected actions to lower level.

11. Event 7 - failure of RDCS is this a call out to the field?

12. Event 8 - turbine byp. valves fail use SRVs to control pressure. Note: Page 4 indicates that this malfunction will not be entered until after power is intentionally lowered making this malfunction meaningless - hands in the pocket. This event needs to be triggered when the A TWS is first initiated to make the event meaningful.

13. Critical task #1. This needs to be expanded. Is T-215 giving direction to electrically scram the reactor by denergizing RPS. Be specific in specifying what required and success criteria.

14. Critical task #2- What power will the plant be at after the recirc pump trip? If below 4%

(APRM downscales), then no need to lower level. And therefore task is not critical.

15. In section on malfunction/remote function/report for event 6, 7 and 8 there needs to be coordination. Event 6 says when level is lowered pull the scram fuses, event 7 when terminate and prevent is complete reset RDCS, which is only a minute or 2 before 50inches, and event 8 is at -25 inches. If the reactor scrams at -50 inches there will be almost no time to respond to drive rods or to control pressure with SRV's. Events 7 & 8 RCDS should be reset earlier to provide time to drive rods should occur earlier and turbine bypass valves should fail earlier before pulling scram fuses.

Scenario #3 (LOCA, 3 rods, PSP)

1. Event 1 - appears to be of little value

2. Event 2 - seems like no credit to RO since it appears the RO has no actions

3. Event 3 - List procedure titles the first time referenced for all procedures entered e.x OT-101, GP8.5, On-113.

4. Event 3- Also list all required BOP actions to restore DWCW, RECW and PCIG.

5. Event 4 - List steps take to reduce power. How much is power expected to be reduced.

6. Event 5 - It doesn't look like any creditable BOP actions required -- list all required operator actions.

7. Event 6 -- Event Description at top of page summarize that happens HPCI aux oil pump failure, loss of condensate/FW. Should list condensate pumps tripping in script.

List all required RO & BOP actions T101, T102, GP8.5, T-225, T-117, T240, SE-10.

Did not see a step to enter T -117 for ATWS. Also states enter T -112 when level drops to -161. T-117 Can level be restored and maintained above -186, if answer no then ED per T-112.

8. Event 7 - pg 18 recognize failure of 3 rods to scram should be listed chronologically as event occurs on page 16 when RO scrams reactor.

9. Event 8 - 012 failed to start what is the safety significance of this failure at this time if no operator actions taken - doesn't appear to affect the success path of the scenario?

10. Critical task 1 not sure of the safety significance of not performing this CT in a timely manner? List 4 factors for CT.

11. Critical task #3, not sure if spraying drywell after ED would meet definition of critical.

The max energy of the reactor has been transferred to the containment. Pressure will not exceed 55 psig.

12. LOCA via "1 a" Recirc is the same LOCA as scenario #1. In scenario 1 the setting on small LOCA is 50%, In this scenario it is 45% and a 1% large LOCA.

Scenario #4 (Steam leak, ATWS - ARI works)

1. Event 1 - list open inbd MSIM and scenario guide top of pg 10,11 show open outbd MSIVtypo?

2. Event 2 & 3 - would like to see ROs drive rods how much do we expect to decrease power for reactivity.

3. Event 4 - Trip TECW pump cook book limited testing of operating knowledge. Would prefer integrated plant electrical fault to add depth and diversity to exam or at least an instrument malfunction that initiates a transient of some sort requiring the crew to take actions to stabilize for BOP.

4. Event 6 - please list all required operator actions for T290 and 291 with procedure titles.

5. Event 7 - please move actions to place actions in chronological sequence on pg 17 after placing mode switch to sid.

6. Event 8 - Don't understand consequence of failure? What do the operators have to do different? List required operator actions to respond to failure.

7. Event 8 & 9 - List actions in chronological sequence e.g. Event 9 actions should be moved to bottom of pg 17 and top of 18 when 5 ADS valves are opened.

8. CT1 - don't understand what makes T-290 actions critical. Please list 4 factors qualifying this as a CT. Can we monitor secondary plant temps in the CR on SPDS or on panels?

Scenario #5 (LOCA, steam leak can't be isolated, PSP)

1. This scenario is similar to scenario 1 LOCA but is more challenging due to the failure of the ability to spray the OW.

2. No instrument failures in this scenario.

3. Event 3 - please list all required operator actions to decrease power per OT-104.

4. Event 4 - please list all required operator actions SE-10.

5. Event 5 - please list all required operator actions -list RWCU affected valves.

6. Event 6 same as scenario 1 and 3.

7. Event 7 & 8 - OW sprays place in chronological order before ED.

8. Critical Task #3. If OW sprays are unsuccessful causing to exceed PSP, then the OW sprays should not be a critical task. Even if the crew eventually gets to RHR service water to spray, if it is after the ED then not critical.

9. Critical Task # 2. Same as previous comment. Should not be based on 55 psig. Should be based on exceeding PSP.

General comment: Try to decide before Tuesday if any of the simulator JPMs can be performed in parallel (2 or 3 at a time) during exam week. You need to consider: if simulator setups are similar enough, proximity to other JPM(s) being performed in parallel and especially on faulted JPMs if they are going to cue the other applicant(s) that are in the simulator at the same time.

Simulator/In-plant JPMs

a. Ok

b. Only 2 critical steps marked in this JPM. There may be other critical task not marked. Step 9b lower speed - it looks like your intent that this doesn't resolve the problem and you need to secure the machine - please clarify.

c. Ok

d. Opening N2 makeup valve for 5 seconds does not test operator understanding and safety significance of task is marginal at best. Recommend replacing JPM with something that involves venting the DW due to high H2 orsomething safety significant.

e. Do we have an EDG overhead alarm in the CR that would accomplish the same purpose e.g. low lube oil or bus fault rather than having to call out to the field?

f. All the rest of the JPMs appear to be Ok.

RO Admin ..IPMs A.1.1 - Borderline LOD - does this really need to be done in the simulator. It will tie up too much simulator time. Please provide screen shots or whatever to facilitate performing in a classroom.

A.1.2 - non assigned A.2 - Steps 2 and 5 clarify which pages affected and provide a key.

A.3 - provide a key.

A.4 - Replace ..IPM. This is technically not a license level operator task - anyone could be assigned do make the announcement, also low LOD and it wastes simulator time and slows down and stretches out the exam unnecessarily. Replace with a second conduct of operations task designed for classroom administration.

SRO Admin ..IPMs A.1.1 - Borderline LOD - does this really need to be done in the simulator?

A.1.2 - Replace JPM - not license level task to review quais of equipment operators also low LOD - NRC does not regulate these activities. Recommend replacing with JPM to test work hour rules and or licensed operators quais covered by 10 CRR 55.

A.2 - initiating cue too leading - just review it - don't have to say for errors and TS concerns that should be obvious. Also after step 7 should list task standard summarize expectations.

A.3 - provide a key. Is anything wrong with the release if nothing wrong then nothing to evaluate?

AA - provide a key for notification form.

Changes made to LGS Exam materials after Prep Week 9128110 Control Room Systems

• a. CRD Hydraulic System I Perform Reactor, -Added note to setup to ensure reactivity flapper placed on top

. Startup, Alternate Path of rod insert and withdraw buttons

-Added additional direction on who would give equipment reports

- Changed activation of malfunction for clogging of alternate drive water filter" from 1 minute after placed in service to "at direction of examiner"

b. Manually place a RFP in Service, Alternate - Changed initial conditions for pump in standby from 60 to 65 Path minutes

- Added Evaluator note, During Pre-Brief: regarding PMS Computer Point KOOONSS

- Added Evaluator Cue to tell candidate: "Supervision directs you to use the manual method of removing a RFP from service" when providing candidate copy of S06.C.C

-Changed step 3.b from critical to non-critical

c. Transfer HPCI From Pressure Control Mode No changes required to Level Control Mode, Alternate Path

d. T-228, InertinglPurging Primary Containment This JPM was developed during prep week after the task was demonstrated in the simulator

e. Transfer D13 from 101 to 201 -Evaluator note was added that copy of S92.2.N (Shutdown Of the DG) may be given to candidate is requested

-Took out Local EO report for local ARC "consider tripping engine to avoid damage"

-Changes CRS Cue from "CRS direct you to place the D13 DIG in a safe condition: to "CRS directs you to perform ARC actions"

- Added additional action to notify CRS that Tap Changer is in Auto

- Clarified that Examiner will be providing Cues

f. Control Rod Exercise Test -This JPM was developed during prep week after the task was demonstrated in the simulator. Original JPM (same title did not adequately support safety function)

-Procedure used did not change

g. Comp Cooling Water I Align RECW for DW -Added note to cue sheet: (For this JPM, the initial conditions Cooling are to be used as stated in the JPM vs. actual simulator conditions)

-Added ANNs and turnover items for:

• B RECW pump overload

• ANN for RECW Heat exchanger low pressure on leak

-Clarified who provides reports

-Added note for evaluator regarding simulator indications for step 5, and changed step from critical to not critical.

-Added step for candidate to Verify no Group VillA isolation per PMS

-Added "valves closed" to step 8

-Added EO report if asked to look for leaks

-Clarified Examiner question in step 11

h. Manually Isolate the RE Enclosure -Evaluator note was added that copy of GP-8.2 "Manual Isolations" may be given to candidate is requested

-Clarified who provides reports

-Clarified Control board nomenclature in standards In-Plant Systems

i. Bypassing and Removing the *A RPS and -Added to initiating cue that S94.2.a prerequisites are UPS Static Inverter from Service complete NOTE: this JPM will be conducted on Unit One

j. Alignment of Equipment or Operation for No changes required Shutdown Cooling (Fire Safe Shutdown)

NOTE: this JPM will be conducted on Unit One

k. Scram and MSIV closure from the AER -Clarified when Examiner reports *OC611 Bay "B" report is given

-Added Examiner report for *OC611 Bay"C" NOTE: this JPM will be conducted on Unit Two RO Administrative Topics Shift Turnover Checklist -Clarified Cue as directed by NRC

-Re-organized order of handouts to put checklist first

-Added checks to items on the checklist that were going to be cues as complete and removed from cues.

Evaluate Valve Stroke Data

-New JPM developed after material submittal, validated during ST-6-107-200-0 prep week Review ST-6-047-370-1, Pre Control Rod -Removed two steps as directed by NRC Withdrawal Check -Added regions for rods

- Cue sheet revised to provide area for candidate to make comments which allows JPM to be performed in a classroom setting as directed by NRC.

Gaseous Effluent Release Rate Determination No changes required (2008 RO A3)

I SRO Administrative Topics Shift Turnover Checklist - Turnover checklist revised to acknowledge turnover items that are not available during performance of JPM.

- Cue sheet revised to provide area for candidate to make comments which allows JPM to be performed in a classroom setting as directed by NRC.

- JPM reorganized to allow examiner to hand out Initial conditions page with attachments added behind Initial Conditions page per NRC direction I Maintain Minimum Shift Staffing and Control -New ..IPM developed after material submittal, validated during prep week I Overtime Review ST-6-047-370-1, Pre Control Rod -Added regions for rods I Withdrawal Check Review and Approve a Liquid Rad Waste -Revised based on !\IRC comments after prep-week.

• Discharge Permit, (2008 SRO A3) -Added additional critical task related flow indicator.

NOTE: Needs to be validate with change Make an E-plan Classification, (0131 ) No changes to JPM required. Will provide colored keys for examiners SCENARIOS For all scenarios the following will be added / modified:

• All required operator actions

• Operator actions listed chronologically as they would be performed

• Critical tasks described in greater detail focusing on specific application in scenario, and highlighted in document

• CRS classify event scripted in at the end of each scenario

• Procedure titles listed the first time they are used Scenario 2 9/20 NRC Comments incorporated - Dave check CT Scenario 3 Added N-SRO on Event #1 per J.Caruso Added option for CT#2 so BID can be on PSP or Level.

Scenario 4 Deleted CT#1 (T-290)

Scenario 5 Added additional instrument failure "Pressure Instrument Fails High and RPS Failure (new event 4)