ML14140A123

04 Draft Written Exam Comments

ML14140A123
Person / Time
Site:	Waterford
Issue date:	04/02/2014
From:	Chris Steely Operations Branch IV
To:	Entergy Operations
laura hurley
References
50-382/14-004
Download: ML14140A123 (9)

Text

ES-401 W3-2014-04 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only (41.7) Only need to know second part of the question in order to answer correctly.

Suggest adding the second part of the answer into one of the C and D distracters and E vice versa of the C and D distracters into A and B for a true two by two. Are there H

any procedural guidance steps to perform the actions you indicate are the correct 1 2 B N actions for this situation (ie driving the output to 3% to reset the RTO)? Changed as recommended. Resetting the RTO is not in the offnormal, but is in the lesson S plan.

Fixed 2 H 2 D B S (41.5)

(41.10) K/A mismatch. The K/A is requiring knowledge of what is occurring in the U S/G during a small break LOCA and what is important about that. The question as 3 H 2 B N stated is requiring knowledge of ESFAS signals. The question was completely revised to match the K/A.

S Fixed Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 9 OBDI 202 - IOLE Process

ES-401 W3-2014-04 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only (41.10) KA mismatch Generic Section 2.4 is for emergency procedures and it has a lot of different topics on EOPs but this KA is asking for AOP knowledge as it relates to the LBLOCA system 011, so this KA probably should be rejected and changed if you cant find a good U AOP question for the LOCA conditions. You might consider steps in an AOP for loss of IA or something like that where valuable perform steps are directed but while you 4 F 3 D B are in the LOCA EOP. A field operators actions might be valuable from an AOP S perspective during a LB LOCA, not sure. You could resample the KA to something in EOP space within the G2.4 section. This might allow you to keep the question, too.

Randomly selected another K/A in the 2.4 field in EOP space. 2.4.20 was selected and was able to keep the question.

Fixed 5 F 3 A N S (41.6) 6 H 2 A B S (41.10) 2006 NRC Exam 7 H 1 D N S (41.10) Reference 8 H 2 D N S (41.7) 9 H 3 D N S (41.10) 10 F 2 A B S (41.10) 2010 NRC Exam 11 H 3 B N S (41.5) 12 F 2 D N S (41.10) 13 H 3 D B S (41.10) 14 H 3 B N S (41.10) 15 H 2 D N S (41.10) 16 H 2 C N S (41.10)

(41.8) Reference - If any is similar to the none of the above statement which is U not allowed per NUREG 1021. Also, ROs are only required to know above the line or one hour action statements which would make this question too easy. Question was 17 H 1 C N changed to include only TS 3.7.4 material above the line. The reference was S removed.

Fixed 18 F 2 D N S (41.10) Page 2 of 9 OBDI 202 - IOLE Process

ES-401 W3-2014-04 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E (41.10) 2007 NRC Exam - K/A requires knowledge of RCS temperature AND pressure changes for a dropped control rod. Changes made that include RCS 19 H 2 C B pressure.

S Fixed 20 H 2 B B S (41.10) 21 H 3 C N S (41.6) Reference (41.10) K/A mismatch. K/A requires knowledge of magnitude of release. Also this U

question would be LOD=1. Condenser route vs. Atmospheric Dumps 22 H 1 C N New K/A randomly selected and a new question written.

S Fixed 23 F 2 D B S (41.10) 2010 NRC Exam 24 F 3 A N S (41.4)

(41.10) Need only know the first part of the answer in order to decide correctly. Not proper 2 x 2.

Distracter C is not credible-if you bypass the IX flow is stopped not minimized. Also, to make the question more discriminating, I would use the entire statement for the reason in the answer selection, such as U

A. Start backup charging pumps to maximize L/D flow 25 H 2 A N B. Stop backup charging pumps to mimimize contamination of IX resin S

C. Bypass IX to prevent contamination of IX resin D. Place stby IX in service to remove more activated products Changed as recommended Fixed 26 H 3 B B S (41.10) 2008 NRC Exam 27 F 2 A B S (41.10) 2007 NRC Makeup Exam 28 F 3 B B S (41.10) 2007 NRC Makeup Exam (41.7) Editorial: Usually when you give a numerical valve (such as CV-169) you also E

provide its noun name in the stem of the question.

29 H 2 B N Changed as recommended S

Fixed 30 H 2 B N S (41.5 Page 3 of 9 OBDI 202 - IOLE Process

ES-401 W3-2014-04 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 31 F 2 C N S (41.2)

(41.7) D is a subset of A. Need interlock setpoint in the stem to prevent arguing A is E

correct. 368 psia is a better alternate pressure IAW the NUREG.

32 F 3 C N Incorporated the change as recommended.

S Fixed 33 H 2 A B S (41.3) 2007 NRC Makeup Exam 34 F 2 B B S (41.3) 2011 NRC Exam 35 H 2 C N S (41.10) 36 H 2 B N S (41.10) 37 H 2 A N S (41.5) 38 F 3 D N S (41.2) 39 H 2 B B S (41.4) 2012 NRC Exam 40 F 3 C B S (41.4) 2007 NRC Exam 41 F 3 A B S (41.4) 2012 NRC Exam 42 H 2 D B S (41.8) 2010 NRC Exam 43 H 2 C N S (41.10) 44 H 3 D N S (41.4)

(41.4) 2009 NRC Exam - Credible to use S/U FRV at 58% power? This affects two E distracters. The S/U FRV can maintain steam generator levels up to 22% flow 45 H 2 A B demand. The stem was changed to indicate Rx power at 40% to make the incorrect answers more plausible.

S Fixed 46 H 2 C N S (41.7)

(41.7 Why is distracter A not correct. The TG states The sequencer is required to be timed out in order to close the 32 feeder breakers. The operator should reset all of U the Pzr heaters and maintain pressure.. All heaters implies both proportional and 47 H 3 C N backup, so this makes A correct as well as C. Two correct answers Unsats the question.

S Changed the distractors to make only one answer correct. Resetting the heaters was added to all distractors. Fixed Page 4 of 9 OBDI 202 - IOLE Process

ES-401 W3-2014-04 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 48 H 2 C N S (41.13) 49 H 3 B B S (41.5) 2008 NRC Exam (41.10) 2008 NRC Exam - K/A mismatch. K/A requires knowledge of the E relationship between radiation intensity and exposure limits not reason for initiating a 50 H 2 A B survey. The question has been changed to a bank question that meets the K/A.

The new question comes from the 2010 exam.

S Fixed 51 F 2 C N S (41.10) 52 F 2 C N S (41.7)

(41.7) 2008 NRC Exam Compare to Q18 to ensure nothing given away. Nothing is 53 F 3 A B S given away. Question 18 only refers to essential air and not interlocks with the Instrument Air system. NRC agreed, no changes required.

54 F 2 C B S (41.10) 2010 NRC Exam 55 H 2 D B S (41.10) 2008 NRC Exam 56 H 2 A N S (41.6) 57 H 2 C B S (41.7) 2006 NRC Exam, Reference 58 H 3 B B S (41.6) 59 F 3 A B S (41.7) 2008 NRC Exam 60 F 2 C N S (41.2) 61 F 3 B B S (41.10) 2010 NRC Exam 62 H 3 A N S (41.7) 63 H 2 D B S (41.11) 2008 NRC Exam 64 F 2 B N S (41.4) 65 F 2 A N S (41.10) the NRC, question is SAT and no changes are required.

(41.10) 2006 NRC Exam - Prior to reactivation of the license a plant tour is required E by someone to the appropriate license level of the operator. This reads that only the 66 F 2 C B 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> are necessary which isnt true. Added in addition to conducting a plant tour into the stem.

S Fixed Page 5 of 9 OBDI 202 - IOLE Process

ES-401 W3-2014-04 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 67 F 2 C N S (41.10)

(41.7) Only need to know second part of answer to get both correct. Also, if the SRO E declares the Mode change then this question becomes unsat on K/A mismatch which 68 F 2 A N is knowledge of RO duties in the control room during fuel handling. Removed the two column format and made each answer one line.

S Fixed (41.10) Dont believe its credible to give a reactivity brief after the PMU addition.

E This affects two distracters. Changed the distractors for reactivity brief to notification of the RAB watch. The RAB watch is required to be notified if 69 F 2 B N starting or stopping major loads in his watchstation, but is not required in this S instance for cycling valves.

Fixed 70 F 2 B B S (41.10) 2011 NRC Exam 71 F 3 D B S (41.11) 2009 NRX Exam 72 F 2 A N S (41.11)

E (41.10) It seems by definition that Optimal would prohibit the use of two procedures at once. Lets discuss. Changed the question such that the word Optimal is 73 F 2 C N not used. The question know states recovery procedures.

S Fixed 74 F 3 A N S (41.10) 75 F 2 C N S (41.10) 76 H 2 B N S (43.5) 77 H 2 A N S (43.2) Reference 78 H 2 D N S (43.5) 79 H 3 C N S (43.5 80 F 3 C B S (43.2)

(43.2) 2009 NRC Exam E

There are multiple examples of cueing in the stem of this question. To fix it, give 81 F 3 A M them voltage, freq, load, MVAR values and have them select whether or not they S need to enter the TS and why. Removed the statement which is below the required value of TRM 3.8.1.1 for post-trip emergency loads from the stem. Page 6 of 9 OBDI 202 - IOLE Process

ES-401 W3-2014-04 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Fixed 82 H 2 C N S (43.5) 83 H 2 C N S (43.4)

U (43.5) RO level question. System knowledge. Agreed, a new K/A had to be 84 H 2 B N selected and a new question was written to match the new K/A.

S Fixed 85 H 3 A B S (43.5) 2009 NRC Exam 86 H 3 C B S (43.5) 2009 NRC Exam - Reference 87 H 2 D N S (43.5) 88 H 2 C B S (43.5) 2009 NRC Exam (43.5) Why would anyone pick B or D distracters for verifying flow, (assuming that the LPSI pump B started) when you gave them in the stem that its output breaker has no control power? A possible change might be send a local operator to manually U close the LPSI B breaker as long as this is not a possible correct answer. Changed B and D distractor per NRC request. Meeting flow curves was not a valid 89 H 3 C N distractor with no LPSI pumps running. Changed to locally closing the breaker using the pistol grip. The breaker can not be closed in using the pistol grip S even if control power was available, the breaker can only be tripped with the local pistol grip.

Fixed 90 F 3 C N S (43.5) 91 F 3 B B S (43.7) 2009 NRC Exam 92 F 3 B N S (43.2)

(43.5) Entry conditions to major AOP. RO level question. A decision must be made by the CRS if the Rx must be tripped on an overspeed, he must know E what is the content of OP-901-210 and the overall mitigation strategy. The reactor must be tripped if due to a loss of lube oil or high vibration. This 93 H 2 B N decision is usually made before AOP entry. No changes made due to this comment. Did change the question from a two column format to a one column S for better readability per NRC request.

Fixed 94 F 2 B B S (43.5) 2012 NRC Exam Page 7 of 9 OBDI 202 - IOLE Process

ES-401 W3-2014-04 8 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only (43.5) For distracters C and D why would someone pick natural circulation to E verify RCS is not water solid? This is not a misconception and has no physical bearing on natural circ and therefore is not a good distracter. Changed distractor C 95 H 3 A N and D to something more plausible as requested. The new distractors will require knowledge of RVLMS indication on QSPDS. Explained in the S explanations portion.

Fixed 96 F 2 B N S (43.3)

(43.2) 2010 NRC Exam - Distracter D stands out as an outlier due to the no testing requirement.

E Distracter C is not credible (why would you test the door that is inoperable to establish operability?) and D is not allowed by NUREG-1021.

97 F 3 A B Changed distractor C to indicate no surveillance testing required such that I S matches the D distractor. The D distractor was previously unlike the other 3 distractors Fixed 98 F 2 D N S (43.4) 99 F 3 D B S (43.2) 2010 NRC Exam 100 F 2 D N S (43.1)

B= 31 F= 34 E= 10 RO TOTALS: M= 0 H= 41 U= 6 Additional Notes:

N= 44 B= 8 F= 11 E= 4 SRO TOTALS: M= 1 H= 14 U= 2 Additional Notes:

N= 16

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is __ 2.27___ on the RO exam and __2.48_ on the SRO exam. Page 8 of 9 OBDI 202 - IOLE Process

ES-401 W3-2014-04 9 Form ES-401-9

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 1 41.2 = 3 43.2 = 6 41.3 = 2 43.3 = 1 41.4 = 7 43.4 = 2 41.5 = 5 43.5 = 14 41.6 = 4 43.6 = 0 41.7 = 12 43.7 = 1 41.8 = 2 41.9 = 0 41.10 = 36 41.11 = 3 41.12 = 0 41.13 = 1 41.14 = 0

5. The answer distribution is: RO / SRO A = 19 (26% )/ 5 (20%)

B = 18 (24%) / 7 (28%)

C = 22 (29%) / 8 (32%)

D = 16 (21%) / 5 (20%)

6. There are 7 questions with attachments provided. Page 9 of 9 OBDI 202 - IOLE Process