ML14111A276

2014-03-Draft Written Examination Comments

ML14111A276
Person / Time
Site:	River Bend
Issue date:	03/27/2014
From:	Vincent Gaddy Operations Branch IV
To:	Entergy Operations
laura hurley
References
50-458/14-03
Download: ML14111A276 (9)

Text

ES-401 RB-2014-03 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 2 N S A, 41.10, CR 2 H 2 N S B, 41.10, CR E B, 41.7, CR Edit distractor C to delete the word all 3 H 2 N S This correction has been made.

4 F 3 N S A, 41.4, CR 5 F 3 B S C, 41.6, CR, NRC 2010 6 H 3 N S D, 41.3, CR 7 H 3 N S A, 41.4, CR 8 F 3 N S C, 41.4, CR Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 9 OBDI 202 - IOLE Process

ES-401 RB-2014-03 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only D, 41.2, CR - Revise distractor B to read provide flow to nuclear instrumentation in E

9 F 2 E N the core.

S This correction has been made.

B, 41.10, CR - This question is unsat because distractors A, C, and D are not credible; only one of the listed individuals has the capability to see the criticality - the H 1 U 10 U B ATC. Suggest approaching the question from a rod interlock perspective such as the F 2 S one rod out interlock.

This question was replaced.

C, 41.9, CR - Answers C and D have a similar word structure. Edit answers A and B E

11 F 3 N to be similar (i.e. SRV tailpipe damage due to exceeding code allowable stresses).

S The question has been revised.

D, 41.5, CR Answers overlap, which is discouraged by NUREG-1021 Appendix B, E

12 H 3 E N Section C.2.j. Reword distractors and answer to remove overlap.

S The overlap was removed.

13 F 4 B S A, 41.7, CR C, 41.10, CR, NRC 2003 - Question is unsat due to K/A mismatch. K/A is a general K/A under Section 295027, Hi Containment Temperature, which is not tested as part U

14 F 3 U B of the question. Modify stem so that entry condition is instead due to high S containment temperature and not high containment pressure.

This question was revised.

15 H 3 B S C, 41.10, OR, NRC 1999 16 F 3 N S A, 41.8, CR E C, 41.6, CR - Revise stem so distractors A and B cannot be correct.

17 H 3 E N S This question was revised.

18 F 3 N S D, 41.10, CR C, 41.4, CR - Is this a question we would expect an operator to know from memory?

19 H 4 N S This question is ok as written.

A, 41.4, CR, NRC 2012 - I recognize this question was approved for use in the 2012 U NRC exam, however distractors B and D are not credible. How is it possible for the 20 H 2 U B generator to experience motoring or reverse power when it is tripped?

S Distractors B and D were revised. Page 2 of 9 OBDI 202 - IOLE Process

ES-401 RB-2014-03 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 21 F 3 N S D, 41.4, CR 22 H 3 N S D, 41.7, CR D, 41.9, CR - CAF Are there any EOP attachments that can be implemented to make distractors A, B, or C correct?

? Facility provided answer: No, there are no enclosures for overriding the level 2 23 H 3 N S isolation of the containment HVN valves and the only enclosure allowing normal venting is used during a severe accident procedure, which is not applicable in this situation. Question is therefore SAT.

24 F 3 N S A, 41.6, CR 25 F 3 N S B, 41.7, CR 26 H 3 N S C, 41.13, CR 27 F 3 N S D, 41.9, CR 28 H 3 B S B, 41.3, CR 29 F 2 N S B, 41.7, CR 30 H 3 N S A, 41.7, CR 31 H 3 N S C, 41.7, CR A, 41.7, CR - Need to provide more information in the stem to clarify which of the 4 E situations the diesel generator is in (per the technical reference). Per R-STM-309H, 32 H 3 E N situation 2 could result in the output breaker opening.

S This question was revised.

C, 41.6, CR - Where is the relief valve relative to the isolation valve? Also need 33 H 3 N S technical reference for review.

The stem is ok as written.

34 H 3 N S B, 41.6, CR 35 F 3 N S D, 41.6, CR 36 H 3 N S A, 41.6, CR, NRC 2003 37 H 3 B S A, 41.6, CR, NRC 2004 38 F 3 N S B, 41.6, CR Page 3 of 9 OBDI 202 - IOLE Process

ES-401 RB-2014-03 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only D, 41.5, CR - Eliminate either discharge pressure or turbine speed from the E

39 H 3 E N distractors. Revise the distractors accordingly.

S Turbine speed was removed from the distractors.

40 H 3 B S C, 41.7, CR, NRC 2007 41 H 3 N S B, 41.4, CR D, 41.5, CR - Distractor C overlaps with distractor B in that the range in B begins at E 425 degrees and the range in C ends at 426 degrees. Change distractor C to range 42 H 3 E N 350-424 degrees.

S The overlap was eliminated.

D, 41.5, CR - This is a KA mismatch because it does not test on design features U and/or interlocks which provide for 3 element control.

This question was revised.

43 H 3 U N The question was revised, but Im not sure how credible the responses are that S include reactor pressure. If those are credible, then the question is SAT.

Question was revised. Question is now SAT.

44 F 3 N S A, 41.5, CR A, 41.7, CR, NRC 2010 - The wording of the stem doesnt make sense to me - the E

45 H 3 E B two bullets in the stem need some type of explanation of what they represent.

S This question was revised.

B, 41.4, CR, NRC 1994 - Question is SAT, however we need the technical reference 46 H 3 B S describing FW pump power supplies. The page we were given was on FW regulating valves.

47 F 3 N S A, 41.4, CR 48 F 3 N S B, 41.4, CR 49 H 3 B S B, 41.4, CR 50 H 3 N S D, 41.7, CR 51 H 3 N S C, 41.7, CR 52 F 3 N S A, 41.4, CR 53 H 3 B S B, 41.4, CR 54 F 3 N S C, 41.6, CR Page 4 of 9 OBDI 202 - IOLE Process

ES-401 RB-2014-03 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only A, 41.6, CR -This is a KA mismatch because it does not test on the impact the uncoupled rod would have on the RCIS. Suggest rewriting on what RCIS indications F 2 U would be available to detect an uncoupled rod.

55 U N E Need to work on this one as the second part of the distractors are all different making H 3 S the first part unnecessary knowledge to answer the question.

The question has been revised to require knowledge of the first part in order to answer the question. Question is now SAT.

56 H 3 N S B, 41.6, CR A, 41.4, CR - This is a subset issue. If C is true, so is A, so the candidate can E

eliminate C. Same is true for D and B. Need to reword answer and distractors to correct this issue.

The revision did not correct the subset issue. A better way to fix the question would 57 F 3 E N be to simplify the stem: i.e. Which of the following valves isolate and what is the isolation temperature setpoint? Or similar.

Question has been revised, but need to add isolate to the distractors to prevent cuing.

S Question has been revised and is now SAT.

C, 41.7, CR - Although the question is SAT, the explanation section of the question needs to be corrected. It states that MOV-48A would close on LPCI initiation, which 58 H 3 N S is not true per the technical reference, and would change the correct answer if it was.

Explanation section has been revised.

59 F 3 N S B, 41.4, CR 60 H 3 N S D, 41.6, CR 61 H 3 N S A, 41.4, CR 62 F 3 N S C, 41.4, CR 63 F 3 N S B, 41.4, CR 64 H 3 N S B, 41.9, CR 65 F 2 N S A, 41.2, CR 66 F 2 N S C, 41.10, CR 67 F 2 N S D, 41.10, CR 68 F 3 N S D, 43.2, CR Page 5 of 9 OBDI 202 - IOLE Process

ES-401 RB-2014-03 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 69 F 3 B S C, 41.10, CR 70 F 3 N S D, 41.10, CR E A, 41.10, CR - Add procedurally to the stem: procedurally require for clarity 71 F 3 E N S This was corrected.

72 H 3 N S C, 41.5, CR 73 F 2 B S B, 41.10, CR D, 41.10, CR - Capitalize BOTH for emphasis. Also need the correct technical 74 F 2 E N S reference. Page we were provided did not have information on EOP responsibilities.

This was corrected.

75 F 2 B S A, 41.10, CR A, 43.2, OR - Because two loops of reactor recirc are operating, only condition A of U the provided tech spec can apply making distractors C and D not credible by direct look up. Not sure how credible distractor B is since it seems securing one loop of reactor recirc would make flow induced vibrations worse in the operating loop. Also, 1 K/A does not match. The K/A is to test procedures, not apply technical specifications.

76 H U U N There are two correct answers, A and C.

3 Per the facility, distractor A is not correct because it is not required. At this power level, the two flows are within the 10% of RATED core flow mismatch, and so A is not S required.

Question has been revised and is now SAT.

A, 43.5, CR - Per EOP-001, entry conditions are on RV level less than 9.7 inches, so E why would the applicant enter EOP-001 with RV level 12 inches and rising?

77 H 3 E N S This question is ok as written since it has setpoint setdown stated in the stem, indicating that RV level went below 9.7 inches.

78 H 3 N S B, 43.5, CR 79 H 3 N S C, 43.5, CR 80 F 3 N S D, 43.2, CR 81 H 3 N S D, 43.5, CR C, 41.10, CR - This is an RO question since it tests on a specific operator action and H U 82 3 U N does not involve a procedure selection.

F S This question was rewritten and is now SAT. Page 6 of 9 OBDI 202 - IOLE Process

ES-401 RB-2014-03 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 83 H 3 N S A, 43.5, CR 84 H 3 N S D, 43.5, CR D, 43.5, CR - Provide more information on what makes answer D the only correct answer. Hard card does reference operators to SOP-59, which would make B E partially correct. Most sites have in Conduct of Ops that they would follow up hard 85 H 3 E N card performance with the use of the appropriate procedure to verify actions.

S Facility provided information in the explanation as to why answer D is the only correct answer. Question is now SAT.

B, 41.6, CR - This is an RO question since the first part of each distractor is RO U

86 H 3 U B knowledge and can be used alone to correctly answer the question.

S This question was rewritten.

D, 41.2, OR - Can easily eliminate distractors A and B with the open reference E provided. Need to improve credibility of distractors A and B such that they cannot be 87 H 3 E N eliminated by reference.

S Question was rewritten. Question is now SAT.

88 H 3 N S A, 43.5, CR E A, 43.2, OR - Do not provide the bases document in the handout.

89 H 2 N S Bases were removed from the handout.

C, 43.5, CR - Revise stem so it includes the name of the associated AOP. Also E capitalize ONLY in answers and add ONLY to distractor D to make it similar to the 90 H 3 E N other choices.

S Question was revised, and is now SAT.

91 H 3 N S B, 43.5, OR A, 43.5, CR - Include names of procedure/enclosure in distractors. The first half of U this question is too similar to Question 11. Need to either change Q92 OR Q11.

92 H 3 U E N S Q92 was revised. After closer review, determined that there is enough difference between Q92 and Q11, so both are SAT as is.

C, 41.5, CR - This is RO because it is a basic systems question. Since it would be difficult to write a question to this K/A to the appropriate license level, we recommend U

rejecting the K/A and selecting a new one.

93 H 2 U N This revised question is still U because it is RO knowledge since it tests on what S specific actions to take.

Question has been rewritten and is now SAT. Page 7 of 9 OBDI 202 - IOLE Process

ES-401 RB-2014-03 8 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only D, 43.2, CR - The distractors that require the operator(s) to wait to be relieved when 1 they require hospitalization are not credible. Recommend replacing by asking how U

94 F U B many people below the minimum are allowed (1 less than minimum) and for how long S (2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />).

2 This question was revised.

95 F 2 N S B, 43.7, CR 96 F 3 N S A, 43.6, CR D, 41.10, CR - Distractors B and C are barely credible, suggest revising the stem so it reads something like The highest level of approval required to commence the E

activity is ___. Delete the procedure references.

97 F 2 E B Question revised, but still a cuing issue in that the three distractors all list specific S positions, while the answer does not.

Question replaced. Question is now SAT.

C, 43.4, CR, NRC 2012 - Why is it reasonable for the operators to know this from E memory?

98 F 4 E B S From the facility, this is a short term LCO and is expected to be known from memory, therefore, this question is SAT.

B, 43.1, CR - Distractors A and C are not credible since everyone knows the NRC U

99 F 2 U N does not care about company assets or generation of electricity.

S Question revised and is now SAT.

100 F 2 N S D, 41.10, CR B= 16 (21%) F= 37 (50%) E= 12 RO TOTALS: M= 0 (0%) H= 38 (50%) U= 5 Additional Notes:

N= 59 (79%)

B= 5 (20%) F= 9 (36%) E= 6 SRO TOTALS: M= 0 (0%) H= 16 (64%) U= 7 Additional Notes:

N= 20 (80%)

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N Page 8 of 9 OBDI 202 - IOLE Process

ES-401 RB-2014-03 9 Form ES-401-9

2. Chief Examiner follow-up comments are indicated in blue.

3. Average difficulty is _2.83 on the RO exam and 2.64 on the SRO exam. These values for this reviewer indicate the RO exam is average difficulty and the SRO exam is slightly easier than average.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 1 41.2 = 2 43.2 = 5 41.3 = 2 43.3 = 0 41.4 = 18 43.4 = 1 41.5 = 6 43.5 = 12 41.6 = 13 43.6 = 1 41.7 = 13 43.7 = 1 41.8 = 1 41.9 = 4 41.10 = 11 41.11 = 8 41.12 = 0 41.13 = 1 41.14 = 0

5. The answer distribution is:

A= 21 / 7 B= 18 / 5 C= 19 / 5 D= 17 / 8

6. There are 2/ 4 questions with attachments provided.

8. This exam was reviewed by S. Garchow on January 30 - February 4, 2014. Page 9 of 9 OBDI 202 - IOLE Process