Text

2015-12 Draft Written Exam Comments - Resolved
	ML15351A432
Person / Time
Site:	Cooper
Issue date:	12/08/2015
From:	Vincent Gaddy; Operations Branch IV
To:	; Nebraska Public Power District (NPPD)
References
	50-298/15-012
	Download: ML15351A432 (16)
	v • d • e

ES-401 CN-2015-11 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Partial K/A M/M - Part (1) not related to K/A. See recommended changes to Q1 U provided on 9-24-15.

1 H 3 X N S Revised to determine applicable TSs required to be entered.

Corrected references, added are to stem and remove "AP" from LHGR in D.

2 F 3 N S 3 H 3 N S Candidate is required to analyze MT trip while operating at 90% impact on BPV 4 H 2 N S operation. Determining whether BPVs fast open or throttle open based upon the current power level and why is higher cognitive.

5 H 4 N S E [OPEN] Level of detail to deduce pressure = 1010psig etc. is difficult. Consider 6 H 3 X N placing needle exactly on a major marker, e.g. 1000 psig and 800 psig, and S increasing size of image provided. [Revised to be on major marker]

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). _Rev 1 Page 1 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Is it necessary to have the saturated steam table included on question? Steam Tables provided.

Enlarged meters and removed Steam Table operator aide.

Provided actual meter pictures with pressures indicating 1100. 1000, 900, & 800 psig per recommendation. Steam tables are provided as a standard reference and should not be included in the total reference count.

Specify Reactor Water Cleanup Non-Regen HX Modify stem to What REC load(s) is(are) manually isolated or secured IAW to account for the fact that I believe CRD pumps are merely secured, not isolated. If 7 F 3 N S thats wrong, disregard.

Modified stem to What REC load(s) is/are manually isolated and spelled out RWCU Non-Regen HX Inlet vs. Reactor Water Cleanup.

Spell out F/D 8 H 3 N S Comment incorporated - spelled out filter demineralizer.

9 H 2 N S Add in a FUEL POOL LOW LEVEL alarm to stem to make distractors A and B more discriminatory (if this makes sense with given conditions).

E 10 H 2 X N For part (2), modify to test on the source or method of emergency makeup water to S SFP required by 2.4FPC.

Added PMIS alarm point for low Fuel Pool level.

Structure of stem can be clarified. Modify question to read similar to:

1) Under given conditions, what is the Primary Containment pressure limit? [62.7 vs 56.8]

2) Which of the following is NOT a concern in derivation of Primary Containment Pressure Limits? [RPV vent valve operability vs. SRV operability]

Revised question to identify the LOWEST DW pressure which exceeds PCPL and the E component of concern if PCPL is exceeded to maintain K/A focus.

11 H 2 X N S Revised question to identify the LOWEST DW pressure which exceeds PCPL-A and the component of concern if PCPL-A is exceeded being SRV operability.

Requires determining the lowest DW pressure which exceeds PCPL with torus level in the normal range and determining the component of concern if limit is exceeded.

Comparing provided values to the actual setpoint for normal & high torus level requires recall of acquired knowledge and associating two or more pieces of data which is HCL. _Rev 1 Page 2 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 2014 NRC Exam 12 H 3 B S Question categorized New but a bank question is provided with references. Clarify.

Is a BANK question

[OPEN] Whether RX is Emx Depressurized just before or right when HCTL is exceeded appears non-discriminatory.

Revised question to eliminate reference and asked when ED is required and why if HCTL is exceeded.

U Requires determining when ED is required in relation to HCTL and why. Comparing 13 H 3 X N various other ED requirements which are required if a parameter is exceeded, prior to S

exceeding or cannot restored and maintained below along with eliciting a mental demand that requires a why response such that the examinee must derive the correct explanation tests at the comprehension level.

Both aspects of question are required RO knowledge IAW INT0080613 (OPS EOP Flowchart 3A - Primary Containment Control).

Part (2) cues the correct answer as worded. Modify similar to:

If Wide Range RPV instrument run temperature exceeds maximum run temperature, level indication __(2)___

E a. CAN be used for trending if above Minimum Indicated Level (MIL).

14 H 4 X N S b. CANNOT be used for trending, regardless of indicated level.

Comment incorporated - changed second part to can CAN be used for trending, only if above Minimum Indicated Level (MIL) and CANNOT be used for trending, regardless of indicated level.

[OPEN] Distractor A, HPCI, is problematic. Per explanation, it IS the system which is required to be secured first, although technically not for Vortex Limits ... What procedural guidance directs securing HPCI below 11? If it is found in an EOP warning, caution, or note, then the question can be modified to ask which component is procedurally required to be secured first.

Also, AMP-TBD00 page 14-16 contains a HPCI Vortex Limit graph, with limits well 15 H 4 X N S below 11 SP level. Why does this not apply?

HPCI is secured as an EOP step if SP level cannot be maintained above 11 feet due to turbine exhaust directly to SP air space. Vortex limit is for the HPCI pump which is never reached due to being secured due to turbine exhaust. Added HPCI step to references. No change to question.

[HPCI reqd to be secured due to steam discharge to torus well before vortex limits reached.]

16 F 2 B S Is the term natural circulation used at CNS? If not, just say thermal convection flow. _Rev 1 Page 3 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Natural Circulation is used - deleted thermal convection flow from the stem and explanation.

17 H 2 N S 18 F 2 N S Part (2) answer is cued in stem Remote actuator would not be located inside the DG1 Room.

U 19 F 3 X N Remote is away from the CO2 bottles which are located inside DG rooms. Revised S

to reflect more plausible location. Added DG room maps to references.

Comments incorporated. Added either & OR to B & D for answer clarity.

[OPEN] Does not meet criteria to be considered Modified. No pertinent conditions in stem were changed. Re-categorize as Bank or change a stem condition. From ES-401:

Select the remaining questions for the examination (nominally 11 for the RO and 4 for the SRO-only) from the facility licensees or any other bank, but significantly modify each question by changing at least one pertinent condition in the stem and at least one distractor. Changing the conditions in 20 H 3 B S the stem such that one of the three distractors in the original question becomes the correct answer would also be considered a significant modification. The intent or objective of the question does not necessarily have to be changed. Adding or deleting irrelevant information and making minor changes (e.g., the unit number, component train, or power level when it makes no difference) would not be considered a significant modification to the question.

Agree - this is a Bank question.

Question stem says SRV D controls pressure between 855 and 966.5 psig, but TS 3.3.6.3 says lower limit for closing is 835 psig. Deconflict.

E Also, the band for SRV D opening is 966.5 to 1010 psig. Is there other guidance 21 F 3 X M which narrows the SRV upper limit down to 966.5? Otherwise, modify question to S account for the opening range / closing range.

Procedure OI-8 provides SRV D pressure band of 855 psig to 992 psig - changed first part of Answers A & B to 992 psig to be consistent with actual setpoints.

Testing on whether RPV water level rises or falls with feed flow < steam flow is LOD=1. Recommend modifying to test on potential cause, or expected RR Pump U response as described in explanation.

22 H 2 X N S Removed SF & FF values and modified chart indication to reflect a constant SF/FF mismatch below 80% power.

Explanation corrected. _Rev 1 Page 4 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Correct answer says 900psig, but the explanation for distractor B says 800psig. SR 3.1.4.1 also says that the maximum scram insertion times occur at RX steam dome E pressure of 800psig. Clarify.

23 F 3 X B S Added IAW Procedure 2.2.8 in stem to eliminate SR conflict. Corrected distractor explanations.

The accumulator is operable otherwise scram insertion times would not be assured.

Distractors C and D: NPSH for RHR and CS pumps is negatively affected by lowering SP level / increasing temp, not increasing SP level. Replace with more credible distractors or provide amplifying example of drywell pressure consideration U

24 H 2 X N for NPSH. CSCS lesson plan states that There are no circumstances under which a S sub-atmospheric pressure could exist in the containment.

Revised question to test when and why DW sprays are secured due to high PC level.

Deleted s from Sprays.

Distractor B could also be considered correct. PSTG Step RR-2 Discussion states, A reactor scram is initiated to reduce the primary system discharge outside primary and secondary containments and to ensure that, if possible, the reactor is shut down before emergency RPV depressurization is initiated. RPV E

25 F 3 X N depressurization is initiated before a General Emergency is required, therefore the S SCRAM transitively ensures the reactor is shut down before a General Emergency is required.

Changed GE to Alert level (entry condition to EOP 5A) and added ONLY to after to eliminate inadvertent inclusion.

26 H 4 B S 2015 Retake Exam 27 F 2 N S Explanation implies that RHR inboard injection valves are open, but provided reference states If reactor pressure is > 436 psig, injection valves will not open.

28 H 2 N S Clarify.

Clarified in explanation that injection valves automatically open as reactor pressure lowers below 436 psig.

What is normal level control in Mode 4? Trying to determine how far off of normal

+35 is.

A more discriminatory way to test this knowledge: Instead of cueing in the stem that SDC valves isolate, make this information part of the requested answer. Additionally, need to 29 F 3 N S specify which SDC valves isolate.

Added Normal RPV level band while in Mode 4 is 80 to 100 inches to explanation. SDC valve names not required as these are required to be known as a normal response to lowering RPV water level. _Rev 1 Page 5 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Recommend providing sufficient additional information for applicant to deduce HPCI-MO-19 FW INJ VLV has not opened (normal drywell press and reactor water level) 30 H 3 B S Candidates are required to know system alignment based upon conditions provided - no changes.

Distractor A implausible that HPCI pump would be left running and its Aux Lube Oil pump taken to PTL. Replace A. with 4 Only.

E Distractor B doesnt make much sense to take Aux Lube Oil to start and then 31 F 2 X N S immediately to PTL. Make it 4, 2, 1.

Distractor A not changed due to being the correct answer if the HPCI turbine were at 0 rpm speed. Distractor B changed to 4, 2, and 1 ONLY.

Distractors A and C not plausible that applicant would think a room cooler provides bearing cooling for core spray pump.

Randomly re-selected KA (loss of AC power impact on Core Spray). New question developed to determine impact of loss of Startup & ESST transformers following a U LOCA and status of Core spray with RPV pressure greater than pump shutoff head.

32 H 2 X N S This question requires analyzing the impact of power loss on the Core Spray system under LOCA conditions and predicting system response with RPV pressure above the pump shutoff head. LOK is high cognitive.

Added following Bus energization to eliminate potential challenges associated with DG start time (14 sec) not being addressed in the time delay.

[OPEN] The BIIT Graph 8 shows a plot that is not consistent with plant conditions in stem - it shows RX Power = 14%, vice 25%. Same with HCTL graph - not consistent.

33 H 3 N S Eliminated BITT reference due to not being required to correctly answer question -

other parameters provided in stem to support plausibility associated with not requiring SLC initiation. Removed graphs from references. Added RCIC & CRD to stem for systems maintaining RPV water level to eliminate questions/assumptions of RCIC availability.

34 H 2 N S Agree that the question as-written meets the K/A, but confirm if this is an activity you expect the applicant to know from memory, i.e., whether they need to depress the U DISP or FUNC button, and then whether the data will show up in a bottom menu 35 F 4 X N requiring soft key use or in a list requiring arrow button use.

S Procedure 4.19 (Honeywell and Yokogawa Digital Recorders) sections have been classified as skill of the craft which supports operation from memory.

36 F 3 N S Place the choices in the order they would be encountered on a startup. _Rev 1 Page 6 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Comments incorporated - New Answer is C.

Part 1) is non-discriminatory. Consider edit to test on the allowed 2-loop operation range and corrective action Revised question to test impact of bypassing a downscale LPRM on APRM indication and which APRM indication requires calibration.

37 H 2 X N U Requires analyzing current ARPM indication with an LPRM downscale and predicting the resultant indication due to bypassing this LPRM and determining which indication requires calibration by comparing provided indications to allowable deviation from CTP is HCL.

Fixed distractors and correct answer is D.

Explanations for C and D are swapped.

38 F 3 N S Corrected explanations.

39 H 3 B S 40 H 4 B S 2012 NRC Fitzpatrick Exam.

Consider making the correct answer a Drywell Pressure Instrument, so that the correct answer is related to primary containment like the distractors, instead of an 41 F 3 B S outlier (albeit a correct one).

Comment incorporated - PC-PS-12A (Drywell High Pressure) is new correct answer.

Explanation is incomplete - cut off. Clarify what is meant in stem by complete loss of pneumatics nitrogen only? Could be interpreted to mean a loss of U

42 F 3 X N accumulators as well.

S Changed to complete loss of pneumatic supply to the accumulators to eliminate misinterpretation.

Modify the stem to more directly state What is the impact on RVLCS of a loss of all 43 F 2 N S valid level signal inputs?, if that is the intent.

Comment incorporated.

Provide a statement in stem that would require the applicant to infer the state of SGT fan 4-position switch on Panel K, such as :

The plant is operating at rated power with all systems in normal configuration.

44 F 2 X N S This would require the applicant to know that both trains of SGT are in auto, for Dist A and B.

The absence of an abnormal switch alignment in the stem provides system in normal 100% alignment - no change. _Rev 1 Page 7 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 8 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Change stabilized voltage to 3800VAC in stem, vice 3700. Makes it a <10% change 45 H 3 B S but still results in automatic response.

Comments incorporated.

46 F 3 B S 47 H 4 N S 48 H 2 X N S 49 F 2 M S 50 H 4 M S Part 2) is non-discriminatory. Suggest rephrase to ask, under current conditions, which of the following buttons will shutdown the DG, DIESEL GEN STOP/START switch to STOP on Panel C ONLY, or Either DIESEL GEN STOP/START switch to U

51 H 3 X N STOP on Panel C OR DG-SW-DG1(5L) LOCAL ENGINE STOP button. This tests S the functionality of the DG trip mechanisms based on DG1 Control Mode Selector switch position.

Comments incorporated - revised to remote ONLY and OR local.

52 F 2 B S 53 F 3 N S Part 1) non-discriminatory. The stem cues the applicant that the SR is not met, then U asks if the LCO is met.

54 H 3 X N S Revised to determine cause of HCU accumulator alarm and procedure used to correct.

55 H 4 N S More discriminatory option:

1) When keylock switch is returned to NORMAL, the RWM lamp will be (lit / dark).

2) In order to restore RWM to service IAW Proc 4.2 RWM, (Depress the E INOP/RESET pushbutton first, then depress the SYSTEM INITIALIZE pushbutton /

56 H 3 N depress the SYSTEM INITIALIZE pushbutton first, then depress the INOP/RESET S

pushbutton)

Revised question to determine if RWM light is lit or extinguished and modified second part to restore the RWM to service.

Added RWM aide disclaimer.

57 F 3 N S _Rev 1 Page 8 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 9 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 58 H 3 N S Applicants are being provided with vortex limits for question 15. So take out within vortex limits in stem.

59 H 3 N S Question is not evaluating NPSH or Vortex limits (No reference required) - no change 60 F 3 N S 61 H 3 B S The 2nd part of D appears correct (CBP B will trip after 12 sec) update the explanation.

62 F 3 B S Simplify the stem: When the Condensate Booster Pump B control switch is placed in START, the pump will _____

Comments incorporated.

63 F 2 N S 64 F 3 N S Partial K/A mismatch. System is Secondary CTMT, but 1st part tests on Primary U

65 H 3 X B CTMT. 2nd part OK.

S Improved explanation to clarify interrelation of primary and secondary containment.

[OPEN] Consider changing DW Temp = 400F, and then make distractor A = 110F (incorrectly used 400psig curve). Any implausibility with this?

E Changed distractor 85 inches to 110 inches to support plausibility associated with 66 H 2 N S incorrectly using the 400 psig curve. Correct answer changed to B.

C distractor corrected.

[DW temp =400F not plausible under current conditions]

67 F 3 N S 68 F 3 N S NRC 2014 Exam.

-In stem delete the first sentence and add IAW Procedure 0.26 (Surveillance Program), to the front of the 2nd sentence.

69 F 2 M S

-Cant be credited as Modified because no pertinent conditions in stem were changed. See response to Q20 above. Can be credited as Bank if still within limits from last 2 NRC exams. _Rev 1 Page 9 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 10 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Spell out WCCA.

Significantly modified the stem and changed the correct answer to now be considered Modified. Spelled out all position titles. Also A procedural change was made since last use that changes the correct answer, therefore revision qualifies as modified Question.]

Answers revised - A procedural change was made since last use that changes the correct answer, therefore revision qualifies as modified Question.]

70 F 2 N S 71 F 2 N S NRC 2015 retake exam.

72 F 2 B S Spell out the permit names.

Comment incorporated.

Are the terms PSTGs and EOPs synonymous?

73 F 3 N S PSTGs are EOP Bases.

74 F 2 N S NRC 2015 retake exam The term ONLY after B. is problematic because Yellow is not just a shutdown E condition, it also has a radioactive release potential. So it could potentially be argued as 75 F 3 M no correct answer.

S Revised question to distinguish prioritize alarm by color and identify which color alarm may require a plant shutdown or rad release.

Incorporated review comments and is now a Modified question.

For part 2), 42.5 is not a reasonable distractor for an SRO question. Recommend making the distractor 52.5 (the actual setting of RPV level high turbine trip), or better yet modifying part 2 to test on what parameter is actually being mitigated (MCPR vs something else This would be more discriminatory for an SRO question).

E 76 F 4 X N Also, provide us a reference that shows the actual setting for RPV level high turbine S trip is 52.5 vice 54.

Revised question to determine TS allowable water level and the thermal limit protected. Annunciator 9-5-2 and OI-8 provide high level setpoint. Added OI-8 reference.

[OPEN] Worksheet says no references provided but TS 3.3.1.1 is provided with 77 H 3 N S question 88, which affects distractors A and B. Therefore this is an open-reference question. _Rev 1 Page 10 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 11 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Is quarterly interchangeable with 92 days at CNS? Stem and answer use both.

Question itself it SAT, but Q88 needs to be replaced with a closed reference Q.

Removed TS 3.3.1.1 reference from exam. Provided TS 3.3.1.1 Condition A & C conditions as part of the stem. Added 92 days to stem.

Need better clarification about why shifting REC HXs using 5.2SW would not alleviate the condition. The highlighted text provided on worksheet references placing REC HX A in service (the failed HX in the stem of the question) per step 1.1.1, but it seems like the crew would instead go to step 1.1.2 which provides guidance to place REC HX B in service, which is the unaffected HX, and remove E REC HX A from service.

78 H 3 X B S Question asks for the REQUIRED procedure to mitigate. Entry conditions ONLY exist for 5.2REC and is therefore the only procedure required to be utilized IAW conduct of operations. Added highlight reference 2.0.1.2 and Corrected highlighted reference.

[Leave as-is; procedurally not allowed to enter distractors]

RO knowledge. 10CFR55.43.(B)5 requires both Assessment of facility conditions and selection of appropriate procedures. This question doesnt require applicant to do either he is told an instrument air leak is in progress, and told what the U

79 F 2 X N procedural transition is. Additionally, procedural transitions directly to major EOPs S (such as RX Scram) are generally not SRO knowledge, per ES-401 Att 2.

Revised question to determine when a scram is required based upon IA pressure and when Attachment 2 (IA Pressure Loss) is implemented.

[OPEN] Is it necessary to state that No decay heat removal systems are operating, 80 H 2 N S if they are told all SDC has been lost?

Provided to provide clarity during exam administration - no change.

[OPEN] Distractors A and B are both partially correct, because they are applicable and the question doesnt ask for the HIGHEST applicable emergency classification.

Not providing the location of damage to the fuel pool liner creates a lot of unknowns.

Stating that the damage is at some reference point that the applicant would have to E know is above Top of Active Fuel would be a good, discriminatory cue.

81 H 2 X N S Testing on whether RMA-RA-1 or RMP-RM-452 is the driving rad monitor for Alert would be a discriminatory question. As currently worded, they only need to know one or the other.

Revised to provide location of crack and ask the highest required emergency classification declaration. Changed foot to feet.

82 H 4 N S [OPEN] List of references to be provided states EOP Caution 1, but this is not listed on the SRO references sheet. Is this intended to be provided? May affect some _Rev 1 Page 11 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 12 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only questions on RO exam - Q14 for example.

Modify distractor C to reference the procedure directing drywell spray, like the other distractors.

Modify stem to What is the CRS first required action at this time?

Eliminated references.

Modified stem to include FIRST and C distractor to include IAW EOP 3A.

Removed DW temperature and provided DWSIL is NOT met and reactor pressure is in the unsafe region of the RPV Saturation Curve to eliminate reference from the question. Change DW pressure to Torus pressure to support DW spray plausibility.

83 F 3 N S

[OPEN] Why is it necessary to provide 5.7.11 Att 2 as a reference? What is the EPIP

/EPLAN training document, and does it cover this?

E Added Procedure 5.7.2 (Emergency Director EPIP) to Technical references (provided 84 H 2 N highlighted markup) which directs performing site evacuation IAW procedure 5.7.11 S

(which is not from memory). The MET data provided requires interpretation beyond the wind direction from arrows (elevation and actual degree wind direction).

Reference is needed to correctly answer this question.

Stem needs to define which EOPs have been entered to set the initial conditions.

Be consistent amongst the answers. C and D need to have IAW EOP-1A for condensate injection.

85 H 4 N S Move the condensate pump information under the following conditions exist SRO determines which EOPs are applicable based upon conditions provided - no change required. Moved condensate information to bulleted conditions and add IAW EOP 1A to answers C & D.

86 H 3 N S [OPEN]

Worksheet says no references provided but TS 3.3.1.1 is provided with question 88.

Procedure 2.1.1, Startup Procedure, Step 4.12, states that overlap should be performed prior to 106 and then position SRM detectors to maintain 103 to 105 cps. The data in the stem is inconsistent with procedural guidance (beyond E6) modify stem conditions such E that the inoperable IRMs are identified when they would be procedurally-expected to be.

87 H 3 X N TS 3.3.1.1 is no longer being provided as a reference.

S Conditions provided allow for determination of inoperable IRMs due to not achieving proper SRM/IRM overlap - No change.

[OK as-written. Forces applicant to make a procedural transition / operability call, Conditions provided allow for determination of inoperable IRMs due to not achieving proper SRM/IRM overlap]

[OPEN] Replace this question with a closed-reference question. There is too much 88 H 3 N E information available by providing TS 3.3.1.1 that could affect the rest of the exam. _Rev 1 Page 12 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 13 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S Removed TS 3.3.1.1 reference and changed second part of question to determine APRM operability by evaluating whether RPS trip capability is maintained or NOT.

Name the actual TS 3.3.2.2 in the stem.

Is there any time that an instrument IS considered operable when it is removed from service? NO It would be slightly more discriminatory to state in the stem I&C is performing SR E

89 H 3 X N 3.3.2.2.2, Perform Channel Calibration, [or 3.3.2.2.3] on one channel of Reactor S Feed Pump Turbine High Water Level Trip , rather than stating directly that the channel is being removed from service.

Modified stem to The CRS has authorized I&C to start SR 3.3.2.2 (Feedwater and Main Turbine High Water Level Trip Instrumentation) Channel Calibration on NBI-LT-52C at 0800.

Add in stem that Procedure 2.3_A-1 was entered and identified Get rid of the sentence Which one of the following completes.

U 90 H 2 X N RO knowledge. Systems knowledge of annunciator system, and direct entry into an AOP.

S Revised question to predict ground indication and determine the correct procedure utilized to perform ground isolation on sensitive panels IAW procedure 2.0.1. Added NO circuit analysis exists for this panel to eliminate operational confusion.

[OPEN] With TS 5.6.6 provided, this is direct lookup. If applicant is required to U implement 5.6.6 from memory, acceptable.

91 F 2 X N S Requires interpreting 14 days form LCO entry vs. Time CONDITION D is entered which is a common misconception - no change.

92 H 3 N S [OPEN]

Is the alarm setpoint for RMA-RA-13 the maximum normal operating value, 100 mr/hr? Yes Max normal op value is the trigger for isolating system, stem says alarm.

U Question is not SRO-only because distractors B and D only require systems 93 F 3 N S knowledge of RCIC supply line, and A and C only require major EOP-5A entry criteria. See ES-401 ATT 2, II.E SRO-Only guidance.

Revised question to allow SRO to make a decision on RCIC system operation based upon plant conditions.

94 F 3 N S Explanation discusses stem statements that are not included in actual stem (not actually needed - fix explanation)

E 95 F 2 N Add (the time the surveillance is first applicable) after 0000 Im not sure if a S

previous days surveillance counts as an initial performance or not, but this should obviate the question. _Rev 1 Page 13 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 14 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Corrected explanations and modified stem to state from initial applicability time.

K/A mismatch. Tests on application of TS 3.0.5, not the BASES of TS 3.0.5.

U Revised question to test TS bases knowledge of what activities are NOT allowed to 96 F 3 N be performed with an instrument returned to service under TS 3.0.5.

S Revised to incorporate recommended changes (activity which is permissible to return an inoperable instrument to service for IAW LCO 3.0.5 Bases).

97 F 2 N S Instead of May / May NOT say IS permitted to and IS NOT permitted to 98 F 2 N S Revised to state is/is NOT permitted to.

Answer C and D are the same. Develop another distractor for D.

E Revised to determine whether the EOP 3A is required to be re-entered/continued and if all 99 F 2 X N S legs/SP temp leg is required to be addressed. Provide actual SP temperature to meet KA and support continue in EOP 3A plausibility.

100 F 2 N S B = 16 F = 36 E = 13 Additional Notes:

RO TOTALS: M= 5 H = 39 U = 10 N = 54 S = 52 B= 1 F = 12 E= 9 Additional Notes:

SRO TOTALS: M= 0 H = 13 U= 5 N = 24 S = 11

GENERAL COMMENT

S OL Program Feedback #401.53 provides the following guidance on use of open-reference questions:

RO (75 items) = up to ~5% or 4 questions SRO (25 items) = up to ~20% - 25% or 5 - 6 questions The draft exam as submitted contains 6 RO open reference qs (8%), and 8 SRO open-references (32%). Reduce the open reference questions by 2 on each portion, or substantially justify why not.

Steam tables are provided as a standard reference (Q6) and eliminated references to Questions 13, 33, 77, 82, and 88. _Rev 1 Page 14 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 15 Form ES-401-9 RO exam is <50% higher cognitive level questions. Edit questions as necessary to achieve 50-60% higher cognitive level.

Edited/justified Questions 4, 11, 13, 32, 37, and 54 as being higher cognitive which makes the RO exam >50% comprehensive.

SRO Exam: 4 of 7 55.43 KSA categories are not sampled (.1, .3, .6, .7). While not every category is required to be sampled, if less than 50% are sampled it cannot be considered broad, thorough, and representative. Evaluate if any of the existing questions apply to unsampled categories, and if not, when re-writing Unsat questions take the unsampled items into consideration.

Question #91 evaluates required actions for not meeting administrative controls listed in TS Section 5 (55.43(b)(1). Question #83 could be appliced to 55.43(b)(6) - Knowledge of TS bases for reactivity controls even though being specifically identified to 43.2. Question #81 could be applied to 55.43(b)(7) - Fuel Handling facilities and procedures (Emergency Classifications) even though being specifically identified to 43.5. Added these items to the 10CFR Part Content Unsampled items were taken into consideration during question re-write.

RO Exam: 6 of 14 55.41 KSA categories are not sampled (.2, 3, 4, 11, 13, 14). Apply same considerations discussed above.

NUREG-1123 states 10CFR55.41 (b) items (1), (3), (7), & (14) are covered on the GFE - 1 and 14 are not covered on the site specific written exam. The K/As associated with the RO site-specific written examinations are derived from 10 CFR 55.41(b) items 2 through 13. 8 of 12 (66%) items sampled.

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner comments are indicated in black.

3. Average difficulty is __ 2.76 ___ on the RO exam and 2.64___ on the SRO exam.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 16 43.1 = 1 41.2 = 0 43.2 = 9 41.3 = 0 43.3 = 0 41.4 = 0 43.4 = 2 41.5 = 15 43.5 = 11 41.6 = 1 43.6 = 1 41.7 = 35 43.7 = 1 41.8 = 3 41.9 = 3 41.10 = 16 41.11 = 0 41.12 = 2 41.13 = 0 41.14 = 0

5. The answer distribution is: RO / SRO A = 22 (29%) / 5 (20%) _Rev 1 Page 15 of 16 OBDI 202 - IOLE Process

ES-401 CN-2015-11 16 Form ES-401-9 B = 17 (23%) / 7 (28%)

C = 18 (24%) / 8 (32%)

D = 18 (24%) / 5 (20%)

6. There are _6_4__ RO questions with handouts provided and __8 6___ SRO questions with handouts provided. _Rev 1 Page 16 of 16 OBDI 202 - IOLE Process