ML14241A707

02-Draft Written Exam Comments

ML14241A707
Person / Time
Site:	Arkansas Nuclear
Issue date:	02/27/2014
From:	Vincent Gaddy Operations Branch IV
To:	Entergy Operations
laura hurley
References
50-313/14-002, 50-368/14-002, ES-401, ES-401-9 50-313/OL-14, 50-368/OL-14
Download: ML14241A707 (24)

Text

ES-401 AN-2014-02 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only B41.8 E R0 - See DRAFT Written Exam Comments - Attachment 1

H 3 N E R1 - see DRAFT Written Exam Comments - Attachment btl E R2 - minor edits S R3 - Question is Sat D41.7 2

F 2 N E R0 - See DRAFT Written Exam Comments - Attachment btl S R1 - Question is Sat Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only A41.01 neg worded question 3

H 2 N E R0 - See DRAFT Written Exam Comments - Attachment btl S R1 - Question is Sat D41.8 4 E F 2 N R0 - See DRAFT Written Exam Comments - Attachment btl S R1 - question was determined to not be a K/A mismatch but was rewritten C41.8 [REF PROVIDED]

5 H 3 N E R0 - See DRAFT Written Exam Comments - Attachment btl S R1 - Question is Sat.

D41.7 6

H 3 N E R0 - See DRAFT Written Exam Comments - Attachment btl S R1 - Question is Sat D41.5 E R0 - See DRAFT Written Exam Comments - Attachment 7

F 3 N E R1 - incorporate editorial comments btl E R2 - correct question formatting S R3 - Question is Sat 8 A41.7 (2008)

H 4 M S btl Page 2 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only D41.7 (2008)

E R0 - See DRAFT Written Exam Comments - Attachment 9

H 3 B E R1 - Editorial changes btl E R2 - minor edits S R3 - Question is Sat A41.7 E R0 - See DRAFT Written Exam Comments - Attachment 10 F 3 N E R1 - change leak to rupture in first bullet btl E R2 - minor edits S R3 - Question is Sat C41.10 (2005)

R0 - Question is not operationally valid as written. You cant have the original question be correct where you emptied the entire SG into containment, got hi Pressure in containment and lose RCPs (and therefore must use Aux spray) and then in this question you have the same exact SG failure and dont get a hi containment U

pressure event? I would also recommend swapping A and B with C and D since this was used before with only one enhancement to the stem made on this version. Also, to strengthen the questions stem to prevent challenges, I would add to the stem IAW procedure EOP-2202.005..Tying it to the procedure step is what makes 1540 psia correct with the lookup value of 1543 psia, otherwise there could be argument that since 1543 is missing there is no correct answer. Also need to add 11 F 4 X N more information to the discussion section of the question, why distractor A is kdc credible but not correct, etc.

Also, unless you give them that MSIS had a malfunction, is it credible to assume the A SG emptied into containment? Look at its inventory, shouldnt have gone that low if MSIS worked correctly?

R1 - Recommend new question - see DRAFT Written Exam Comments -

U Attachment E R2 - editorial changes required - see DRAFT Written Exam Comments - Attachment.

E Question is now Sat.

S R3 - minor edits R4 - Question is Sat Page 3 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only D41.7 U R0 - This question is not discriminatory as written and also doesnt test the KA the way it was meant. If you are indirectly testing the MFIS interlock by asking what pump can be overrriden and started, dont cue them to it via loss of power supply, or it becomes a power supply question with cueing (in this case partial cueing on the 12 answer with 2A1 bus LO, no one would pick A or C, making them not credible). Also H 4 X X N kdc doesnt seem credible to ask if they know if you can feed a bad SG.

U R1 - see DRAFT Written Exam Comments - Attachment E R2 - change part 2 of A from A pump to B pump and update Notes accordingly E R3 - Editorial changes S R4 - Question is Sat 13 F 3 N S D41.5 kdc B41.7 (2008) 14 H 3 B E R0 - delete ONE (1) from stem, update References kdc S R1 - Question is Sat C41.7 U R0 - For this question, the KA is for the indicators and/or alarms, not the relays so the question needs to ask something along the lines of What lights are lit at the ESFAS status panel upon loss of 2RS-1?

15 Also, distractor A is not allowed by NUREG-1021.

H 3 X X X N kdc R1 - proposed question - see DRAFT Written Exam Comments - Attachment U

E R2 - minor edits required E R3 - LOD=3 S R4 - Question is Sat Page 4 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only D41.5 (2005 Modified) 16 U R0 - You are not asking knowledge of the reason if you give them the reason and just H 4 X X X M ask them to pick which train is applicable. Also, this piece of knowledge of ch 1-3 red kdc and ch 2-4 green is asked in a previous question.

S R1 - Question is now Sat C41.5 (2006)

U R0 - Reason for action is given in the stem. KA mismatch.

17 F 3 X B E R1 - Question replaced, now Sat. Format editing required.

kdc E R2 - change taxonomy to F, LOD=3, reorder answers S R3 - Question is Sat C41.4 18 H 4 N E R0 - Arrange answers shortest to longest. Correct answer stays the same.

kdc S R1 - Question is Sat 19 C41.7 kdc F 3 N E R0 - Arrange answers shortest to longest. Correct answer becomes C S R1 - Question is Sat D41.7 20 H 3 B E R0 - Arrange answers shortest to longest. Correct answer becomes D kdc S R1 - Question is Sat C41.5 R0 - KA mismatch-SU channels vs IR channels. Also, see DRAFT Written Exam 21 U H 3 X X N Comments - Attachment.

kdc U R1 - Question is still a K/A mismatch. Replace K/A S

R2 - Question is now Sat. Page 5 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only C41.8 22 F 2 N E R0 - Formating edits kdc S R1 - Question is Sat B41.7 23 H 3 N E R0 - The first blank should be does not meet or meets kdc S R1 - Question is Sat C41.10 R0 - It does not seem credible to have a distractor using the on-watch operator.

24 Does he/she abandon the watch? I think a better distractor is a qualified field F 2 N E operator.

kdc E R1 - Minor edits S R2 - Question is Sat A41.5 U R0 - KA mismatch-question does not address reasons for the actions. Also, see DRAFT Written Exam Comments - Attachment.

25 H 3 X X X N E R1 - minor edits, Question is Sat kdc E R2 - minor edits E R3 - minor edits S R4 - Question is Sat A41.7 E R0 - C and D have incorrect vent valve titles. Rearrange answers shortest to 26 F 2 N longest.

kdc E R1 - minor edits S R2 - Question is now Sat Page 6 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only A41.10 U R0 - SRO question.

27 H 2 X N U R1 - Still unsat, same reason.

tj U R2 - Still unsat, same reason S R3 - Question is now Sat C41.7 28 F 3 X N U R0 - LOD = 1. Submit new question.

tj S R1 - Question is now Sat C41.5 29 U R0 - LOD = 1 F 3 X N tj E R1 - Question is now Sat but needs minor edits S R2 - Question is Sat B41.7 E R0 - Is this the normal verbiage for a handswitch in neutral? Startup #2 Transformer handswitches are OUT of Pull-to-Lock (PTL) If not, it may be more straightforward 30 to say handswitches are in NEUTRAL F 3 N tj E R1 - formatting change E R2 - minor edits S R3 - Question is Sat A41.7 31 H 2 X N U R0 - See DRAFT Written Exam Comments - Attachment tj S R1 - Question is now Sat Page 7 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 8 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only B41.5 R0 - Distractors C and D are essentially the same. Is there a time when it would be U procedurally directed to close one but not the other Ctmt Sump isolation valve? If 32 not, replace one with a new distractor. Also, explain how this question meets both H 3 X N parts of the KA. See DRAFT Written Exam Comments - Attachment tj U R1 - see comments from Tech Review Rev 3 in DRAFT Written Exam Comments -

Attachment S

R2 - Question is now Sat B41.5 (2011 NRC exam)

R0 - what changed based on NRC comments?

E 33 R1 - see comments from Tech Review Rev 3 in DRAFT Written Exam Comments -

F 3 B E tj Attachment E

R2 - minor edits S

R3 - Question is Sat D41.7 34 F 2 N E R0 - minor edit tj S R1 - Question is Sat B41.7 R0 - Ensure the goes up and goes down is the correct response for the controller E output with respect to actual plant parameter being controlled.

35 Also, question listed as New in pedigree, but in historical notes it says NRC Exam F 3 N Bank QID # 1744 used on 2011 NRC Exam. Verify if this is New or Bank and adjust tj Bank total accordingly.

E R1 - minor edits (see Tech Review Rev 3) in DRAFT Written Exam Comments -

Attachment S

R2 - Question is Sat Page 8 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 9 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only B41.7 (2008)

E R0 - Revise stem to indicate actual values without giving them the diagnosis. See 36 DRAFT Written Exam Comments - Attachment.

H 3 M co E R1 - correct formatting of question E R2 - minor edits S R3 - Question is Sat A41.7 R0 - See DRAFT Written Exam Comments - Attachment 37 E F 3 N R1 - minor edits (see Tech Review Rev 3) in DRAFT Written Exam Comments -

co E Attachment S

R2 - Question is Sat B41.5 R0 - Part 2 of Distractors B and D are not credible. You cannot have a single failure U of active and passive components. See DRAFT Written Exam Comments -

Attachment 38 F 3 X X N R1 - Question not a good K/A match. See recommended new question in attached co U file.

E R2 - minor edits (see Tech Review Rev 3) in DRAFT Written Exam Comments -

S Attachment R3 - Question is Sat A41.7 (2006)

R0 - Verify that there is no procedural operator action that during the time from 30 39 E H 3 B minutes to 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> would allow for starting the LPSI pumps.

sh E R1 - minor edit S

R2 - Question is Sat 40 H 3 N S A41.4 sg Page 9 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 10 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only B41.5 R0 - Verify if it is expected knowledge for ROs to be able to determine if a system is 41 OPERABLE or not. If knowledge of the Tech Spec bases is required, or if their F 2 N S sh knowledge of the Limits and Precautions for this is SRO area, then this is a SRO question. Reference materials for question are RO, the Systems Training Manual and CS operations procedure. Question is Sat.

B41.7 E R0 - Since the EOP calls for use of feedwater pump 2P7B as priority over 2P7A, an applicant may assume that 2P7B is available for operation since nothing says that it 42 H 3 N has a problem. Therefore, the correct answer could be driven by how an operator sh controls 2P7B vice 2P7A.

E R1 - Added 2P-7B OOS to stem. Minor edits S R2 - Question is Sat C41.7 (2003 Modified)

E R0 - Bank or Modified: Does changing the initial power level (100% and 80%) affect the answer to the question compared to the source bank question?

43 R1 - Changed to Bank. No support information is provided to describe how High H 3 B E sh Level Override (HLO) feature functions in this situation (what S/G level it takes action at). This is needed to understand why the proposed correct answer is correct.

E R2 - minor edits S R3 - Question is Sat Page 10 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 11 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only C41.5 R0 - Given the scenario, to test the ability of the applicant to predict the impacts on U the MFW system by the trip alarm of Main Feed Pump A, some part of the question has to test: what is the status of feed in this situation? As part of the stem, the question tells the applicant that the Loss of Main Feedwater AOP is entered, so he/she is told that a) there was an actual Main Feed Pump trip, and b) he/she is told 44 the effect is that they are driven into the Main Feedwater AOP. This question does H 3 X N sh not test the applicants ability to address the a) portion of the K/A statement.

U R1 - see DRAFT Written Exam Comments - Attachment U R2 - formatting of question U R3 - see Tech Review, Rev 4 in DRAFT Written Exam Comments - Attachment . A not plausible S

R4 - Question is Sat D41.5 U R0 - For the first half of the question, the SPTAs appear to support that the minimum flow is to each S/G, and the questions Notes section support that. This doesnt match with the correct answer indicated. Second, the minimum flow needed to remove the expected decay heat is based on the assumption that at least one S/G is below 10% level. This condition is not addressed in the stem, so it appears that a 45 correct answer is hard to define.

H 2 X N sh The second half of the question lowers the level of difficulty on this overall question.

What is the level of discrimination if the applicant is being asked if after a reactor trip, with feedwater provided, there is more or less decay heat being generated with more time after reactor trip?

U R1 - rewrote question to raise LOD.

S R2 - Question is Sat Page 11 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 12 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only C41.5 U R0 - To enter Mode 4, RCS temperature has to be greater than 200F. Even with the calculation showing that the RCS would reach 200F in 15.3 minutes, the proposed correct answer is not correct. At 15 minutes, the temperature would be less than 200F. There is no correct answer as provided. Recommend using a different initial 46 condition RCS temperature so that an exact time in minutes for the calculation can be H 2 X N sh used (for example, with RCS at X degrees F, it would be greater than 200F at Y minutes, not Y.Z minutes).

U R1 - Question is now Sat. Also, see DRAFT Written Exam Comments - Attachment.

E R2 - corrected formatting of question S R3 - Question is Sat A41.7 47 U R0 - K/A in exam outline is 062 K2.01. This question is written to 063 K2.01.

F 2 X N sh E R1 - Changed question to match K/A. Question is Sat. Minor edits S R2 - Question is Sat A41.5 R0 - Faulted 2 x 2 logic. Also, see DRAFT Written Exam Comments - Attachment.

U 48 R1 - Revised question did not correct faulted 2 x 2 logic. See DRAFT Written Exam H 3 X N U sh Comments - Attachment.

E R2 - Question is now Sat but needs minor edits : Taxonomy=H, LOD=3 S

R3 - Question is Sat D41.7 (2009)

E R0 - In distracter B, does remotely operable mean that the breaker can be operated remotely (anything outside the Control Room), or in both remote manual and local 49 manual methods? Ensure that there are not 2 correct answers.

F 2 B sh E R1 - see DRAFT Written Exam Comments - Attachment E R2 - minor edit in Historical Comments S R3 - Question is Sat Page 12 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 13 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only B41.7 (2009)

U R0 - A could be argued as also being correct (the EDG would be expected to run for 37 minutes). The question needs to be reworded so distractor A is not correct. I suggest rewording the last sentence of the stem to something like, Based on these conditions, #2 EDG would reach the Tech Spec Minimum Day Tank volume in _____

50 minutes.

H 3 X M sg Also, screened this question to be a bank (not modified) since it tests on the same knowledge requirements to get correct (i.e. in order to answer both questions, all you need to know is the TS minimum value and do some basic math).

Also, see DRAFT Written Exam Comments - Attachment.

S R1 - Question is Sat A41.5 U R0 - K/A mismatch because K/A requires testing on what (a) the impact of increasing steam demand (due to normal operations or a malfunction) will have on the Main Steam Reheat System and (b) what to do about it. The question as written is focused 51 on the reverse; what will the impact of operating the Main and Reheat Steam System H 3 X X N be on the RCS primary. Bottom line is how the Main and Reheat Steam System will sg be impacted by increasing steam demand. Also, see document sent 1-20-14.

E R1 - Enhance Notes section E R2 - Correct formatting of question S R3 - Question is Sat B41.13 (2008)

E R0 - Suggest deleting the first line (Consider) and the two bullets below it as they 52 F 2 B are not needed to answer the question.

sg E R1 - minor edit S R2 - Question is Sat Page 13 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 14 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only B41.7 U R0 - K/A mismatch because the K/A requires testing on automatic actions associated with Service Water System emergency heat loads. The question, as written, does 53 H 3 X N not test on this but actually tests on manual actions contained in the SPTAs and sg somewhat on the AC electrical distribution system.

E R1 - minor edits S R2 - Question is Sat A41.4 54 E R0 - question formatting F 3 N sg E R1 - minor edit S R2 - Question is Sat B41.9 55 F 3 N E R0 - Add greater than to post RAS flow sg S R1 - Question is Sat C41.10 (ANO Bank) 56 F 2 M E R0 - see Tech Review Rev 4 in DRAFT Written Exam Comments - Attachment sg S R1 - Question is Sat C41.7 57 F 2 N E R0 - Arrange answers shortest to longest sg S R1 - Question is Sat C41.5 (ANO Bank) 58 F 2 B E R0 - minor edits sg S R1 - Question is Sat Page 14 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 15 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only B41.7 (ANO Bank) 59 H 3 B E R0 - minor edits sg S R1 - Question is Sat B41.9 E R0 - Recommend changing the VIOLENT reaction in the stem to detonation so 60 that it is clear.

F 3 B sg E R1 - see Tech Review Rev 4 in DRAFT Written Exam Comments - Attachment E R2 - minor edits S R3 - Question is Sat D41.9 (2008) 61 H 2 B E R0 - See Tech Review Rev 4 in DRAFT Written Exam Comments - Attachment sg S R1 - Question is Sat??

D41.11 62 F 3 N E R0 - minor edits sg S R1 - Question is Sat D41.7 E R0 - See DRAFT Written Exam Comments - Attachment 63 E R1 - Reworded per comments F 3 N tb E R2 - minor edits E R3 - minor edit S R4 - Question is Sat Page 15 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 16 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only A41.6 U R0 - Answers are subjective. What is small or large to one person might be different 64 to someone else, and the stem does not provide any indication as to what small or H 3 X X X N large means. Therefore, depending on the students assumptions about what is tb intended by small or large, any of the answers could be considered correct. See DRAFT Written Exam Comments - Attachment.

S R1 - Question is now Sat C41.7 65 F 2 N E R0 - rearrange answers shortest to longest tb S R1 - Question is Sat B41.10 U R0 - The facility plans to provide Attachment 9.1 to the applicants, including all notes as a reference. This is therefore a direct look-up and provides no discriminatory 66 value. Select new K/A F 1 X X N tb E R1 - see DRAFT Written Exam Comments - Attachment, question is now Sat E R2 - rearrange answers shortest to longest S R3 - Question is Sat A41.10 R0 - This is a Tier 3 K/A, but the question is a specific system knowledge question.

U Per NUREG-1021, ES-401 D.2.a, first paragraph, Tier 3 must maintain plant wide 67 F 2 X N focus and not become an extension of Tier 2.

tb U R1 - still Unsat, not Tier 3 question. See DRAFT Written Exam Comments -

Attachment.

S R2 - K/A changed, new question. Question is now Sat C41.10 (2011 NRC exam) 68 H 3 B E R0 - formatting of question tb S R1 - Question is Sat Page 16 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 17 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only C41.10 (1998) 69 U R0 - see DRAFT Written Exam Comments - Attachment.

F 2 X B tb U R1 - revised question, removed negatively worded question S R2 - Question is now Sat A41.10 E R0 - Stem of question says ALL items on the IPTE pre-job checklistare required to be. The procedure reference, however, states that items SHOULD be discussed.

70 F 3 N An applicant could argue then that no answers are correct because of the stem tb wording.

E R1 - change LOD=3 S R2 - Question is Sat D41.10 U R0 - See DRAFT Written Exam Comments - Attachment. Replace K/A and question 71 F 3 X B E R1 - correct spelling. Question is now Sat with enhancements.

tb E R2 - minor edits S R3 - Question is Sat C41.11 (2009)

E R0 - Based on question as worded, answer A could also be correct as it is listed as a possible long term action in Attachment 19. Either reword the distractor or the stem 72 to make answer A incorrect (i.e., add initial actions to stem Standard Attachment 19, F 3 M Control of Secondary Contamination, initial actions would direct.)

tb E R1 - minor edits E R2 - minor edits S R3 - Question is Sat Page 17 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 18 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only B41.12 (2011)

U R0 - TS 6.7.1.e states Except for individuals qualified in radiation protection procedures, or personnel continuously escorted by such individuals, entry into such areas shall be made only after dose rates in the area have been determined and 73 entry personnel are knowledgeable of them. Need to edit stem to more closely align F 3 X X M with this, as stem is unclear as to whether this requirement is met (specifically that tb entry personnel are knowledgeable). Someone could argue that distractor A is correct the way stem is worded. See DRAFT Written Exam Comments - Attachment.

E R1 - formatting of question. Question is now Sat S R2 - Question is Sat B41.10 (ANO Bank) 74 E R0 - Distractor D not credible, edit to make it more credible.

F 2 X M tb E R1 - See DRAFT Written Exam Comments - Attachment S R2 - Question is Sat B41.10 U R0 - Edit the question stem so that it doesnt ask what the CRS would direct, but ties it directly to the procedure, i.e., Per, OP-2203.042, the operators would..

Also, why are Distractors plausible? Nowhere in the reference procedure does it direct a plant shut down by rapid, normal or plant trip; only to monitor by other means 75 and refer to Tech Specs and E-plan.

H 3 X X N tb U R1 - see DRAFT Written Exam Comments - Attachment E R2 - Change question format and answers to make technically correct. Question is now Sat E R3 - arrange answers shortest to longest S R4 - Question is now Sat Page 18 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 19 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only B43.5 [REF PROVIDED]

E R0 - See DRAFT Written Exam Comments - Attachment 76 E R1 - Minor format of question needs fixed.

H 3 N smg E R2 - Does PLCS keep PZR level from reducing to 59%? Delete HPSI pumps from S stem, remove SIAS from Notes, expand Notes, correct PZR gal/%

R3 - Question is Sat C43.5 R0 - See DRAFT Written Exam Comments - Attachment 77 M U H 4 X R1 - Question was completely rewritten and is now Sat. Question is now considered smg N E to be a new question as it does not resemble the bank question originally referenced.

E R2 - Question is Sat C43.5 78 E R0 - See DRAFT Written Exam Comments - Attachment H 2 N smg E R1 - minor edits to stem, rewrite Notes section S R2 - Edits made, question is now Sat.

B43.5 E R0 - See DRAFT Written Exam Comments - Attachment 79 E R1 - All edits were made as recommended with the exception of one (see below).

H 4 N Overall the question is sat, however recommend revising the fourth bullet in the stem smg to read Now 2D-32B output breaker trips open. The current wording is awkward.

E R2 - Arrange answers shortest to longest.

S R3 - Question now Sat A43.2 80 E R0 - See DRAFT Written Exam Comments - Attachment H 3 N smg E R1 - Minor question format edits required.

S R2 - Question is now Sat. Page 19 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 20 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only C43.2 (ANO Bank) [REF PROVIDED]

U R0 - See DRAFT Written Exam Comments - Attachment 81 E R1 - see attached file H 3 X B sh E R2 - minor edit to B - circuits vs circuit E R3 - minor edits S R4 - Question is now Sat B43.2 (ANO Bank) 82 E R0 - See DRAFT Written Exam Comments - Attachment F 2 B sh E R1 - see attached file S R2 - Question is now Sat B43.5 (2008) 83 H 3 B E R0 - See DRAFT Written Exam Comments - Attachment sh S R1 - see attached file. Question is Sat B43.5 (2005)

U R0 - See DRAFT Written Exam Comments - Attachment 84 F 2 X N U R1 - DRAFT Written Exam Comments - Attachment sh E R2 - Worksheet updated to reflect original K/A . Question is Sat w/ minor edits.

S R3 - Question is Sat B43.5 (2005)

E R0 - See DRAFT Written Exam Comments - Attachment 85 H 3 M E R1 - Question formatting changes sh E R2 - minor edits S R3 - Question is now Sat Page 20 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 21 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only C43.2 [REF PROVIDED]

86 E R0 - See DRAFT Written Exam Comments - Attachment F 3 N kdc E R1 - minor edits S R2 - Question is now Sat.

D43.2 87 F 2 N E R0 - See DRAFT Written Exam Comments - Attachment kdc S R1 - Question is now Sat.

B43.5 E R0 - See DRAFT Written Exam Comments - Attachment 88 H 2 N E R1 - Enhance Notes kdc E R2 - minor edits S R3 - Question is Sat C43.5 U R0 - See DRAFT Written Exam Comments - Attachment U R1 - Did not change the 2 X 2 (which is not really a 2 X 2) and knowing the pump priorities you can pick out the answer A purely by knowing that the A EFW pump is 89 the first priority to get going for these conditions, which is RO only knowledge. Called H 3 X N licensee and they said that they forgot to change the answers so that it is a correct 2 kdc X 2 array, which would fix this issue. They will be sending an updated Q89 today.

E R2 - Formatting of question needs fixed. Question is now Sat E R3 - minor edits S R4 - Question is Sat A43.5 90 U R0 - See DRAFT Written Exam Comments - Attachment H 2 X N kdc U R1 - Discussed issues via phone conversation S R2 - Question is now Sat. Page 21 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 22 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only D43.5 91 U R0 - See DRAFT Written Exam Comments - Attachment H 4 X N tb U R1 - Question still does not address part b of K/A S R2 - Question is now Sat B43.5 92 H 2 X N U R0 - See DRAFT Written Exam Comments - Attachment tb S R1 - Question is now Sat B43.6 U R0 - How is the SRO-only? The procedures are SRO knowledge only but this tests 93 F 2 X N the FHES equipment. Also, see DRAFT Written Exam Comments - Attachment tb E R1 - see DRAFT Written Exam Comments - Attachment S R2 - Question is now Sat.

C41.10 Neg worded question 94 E R0 - See DRAFT Written Exam Comments - Attachment.

H 3 N tb E R1 - see DRAFT Written Exam Comments - Attachment S R2 - Question is now Sat A43.5 (NRC bank, not used)

U R0 - See DRAFT Written Exam Comments - Attachment 95 E R1 - Question is now Sat. Minor edits H 3 X B tb E R2 - Minor edits E R3 - Minor edits S R4 - Question is Sat Page 22 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 23 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only A43.3 U R0 - See DRAFT Written Exam Comments - Attachment 96 U R1 - Question is still Unsat because two of the distractors (A & B) are not plausible. If F 3 X X N configuration control is maintained, there would be no reason to ensure correct plant smg configuration before or after exiting any procedure. Comments made to Rev 1 - see DRAFT Written Exam Comments - Attachment.

S R2 - Question is now Sat C43.5 [REF PROVIDED]

97 E R0 - See DRAFT Written Exam Comments - Attachment H 3 N smg E R1 - minor edits S R2 - Question is Sat B43.4 U R0 - See DRAFT Written Exam Comments - Attachment U R1 - Question was replaced from the original but is still Unsat because it is a K/A 98 mismatch. K/A requires a question focused on radiation or contamination hazards F 4 X N smg that may arise and this question is focused on dose limits. Recommended question -

see attached file.

E R2 - Question is now Sat. Correct Notes section S R3 - Question is Sat D43.5 99 U R0 - See DRAFT Written Exam Comments - Attachment H 3 X N smg E R1 - New K/A selected. Question is now Sat. Change LOD=3 S R2 - Question is Sat B43.5 E R0 - See DRAFT Written Exam Comments - Attachment 100 H 3 N E R1 - Question format needs revised. Question is Sat.

smg E R2 - minor edit S R3 - Question is Sat Page 23 of 24 OBDI 202 - IOLE Process

ES-401 AN-2014-02 24 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Explanation Rev 4 (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only xx Used on previous 2 NRC examinations B= 16 F= 40 E = 57.3%

RO TOTALS: M= 8 H= 35 U = 37.3% Additional Notes:

N= 51 S = 5.3%

B= 4 F= 7 E= 52%

SRO TOTALS: M= 1 H= 18 U= 48% Additional Notes:

N= 20 S = 0%

GENERAL COMMENT

S: Page 24 of 24 OBDI 202 - IOLE Process