ML13099A273

12 Draft Written Comments

ML13099A273
Person / Time
Site:	Grand Gulf
Issue date:	12/10/2012
From:	Chris Steely Operations Branch IV
To:	Entergy Operations
laura hurley
References
50-416/13-012, 50-417/13-012
Download: ML13099A273 (7)

Text

ES-401 GG-2012-12 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 2 N S A (41.8) 2 F 2 N S B(41.7) 3 F 2 N E D(41.4) This question borders LOD=1. Needs enhancement. Fixed. Now Sat.

4 H 2 N S C(41.8) 5 H 3 N S D(41.5) 6 H 3 N S B(41.7) 7 F 2 N S A(41.10) 8 F 2 N S A(41.7) 9 F 3 N S D(41.7) 10 F 3 N S B(41.7)

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process

ES-401 GG-2012-12 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 2 B S C(41.7) 12 F 2 B S C(41.10) NRC EXAM LOT-309 -

B(41.10) - Distracter C does not seem plausible since no information was given in 13 H 3 N E the stem concerning RPV. Fixed. Now Sat.

A(41.10) HANDOUT - K/A is PARTIAL OR COMPLETE LOSS of A/C. - Appears this question concerns a controlled de-energizing of a bus. Question should be more 14 H 3 B E casualty driven with operators responding to a loss of a bus, not pre-empting it. Also, appears there is a handout (drawing) with this question. This needs to be referenced on the table. Fixed. Now Sat.

15 H 3 B S D(41.7) 16 H 3 N S C(41.8)

C(41.10) HANDOUT - Bullet stem conditions. Distracter B is almost 17 F 2 N E indistinguishable on the chart from the correct answer. Fixed. Now Sat.

18 H 3 N S A(41.9) 19 H 2 N S B(41.5) 20 F 2 N S A(41.10) 21 H 3 B S C(41.10) 22 H 3 N S A(41.4) 23 H 3 N S C(41.7)

C(41.10) - Required by GG for operators to memorize frequency of surveillances?

24 F 2 N E Would rather see this question focused on actual procedure (per K/A) and not frequency. Fixed. Now Sat.

D(41.7) Distracter D must be revised as ROs are not required to make operability 25 F 2 N E determinations. Are there any interlocks that would prevent a start with the room cooler fan tripped? This affects plausibility of the other distracters. Fixed. Now Sat.

26 H 2 N S B(41.7) 27 H 3 B S A(41.5) 28 H 3 N S D(41.5) 29 F 2 N S B(41.7) Page 2 of 7 OBDI 202 - IOLE Process

ES-401 GG-2012-12 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only A(41.7) - Rewrite question to be more operationally valid. Meaning, give the applicant control room indications and phrase the question to require the student to 30 F 2 N E recall the power supply to a major D.C. load(s) based on an event in progress. Fixed.

Now Sat.

31 F 2 N S B(41.6) -

32 H 2 B S C(41.7) 33 H 3 N S B(41.5)

A(41.8) Replace rated power with a number. ROs not required to have TS action 34 F 2 X N U statements longer than 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> memorized. This effects three distracters. Fixed.

Now Sat.

35 H 2 B E D(41.4) - Replace rated power.

D(41.5) - Replace rated power. ROs not required to determine operability. This 36 H 2 X N U effects A and D. Fixed. Now Sat.

C(41.7) - Simple memorization. Make this a more operationally valid question.

37 F 2 N E Fixed. Now Sat.

38 H 2 N S D(41.7)

B(41.7) - Bullet the stem. One could argue that if the annunciator comes in at < 20 psig then technically 2.6 is correct as well since it is below 20. Recommend changing 39 F 2 N E 2.6 to something greater than 20 to preclude this. Would also need to use equal to since less than 30 (i.e. 19 psig) would be correct as well. Need to only have ONE correct answer. Fixed. Now Sat.

D(41.10) - 2011 NRC EXAM - The stem is somewhat confusing. Mode 2 at 17%?

40 F 2 B E Im assuming some event has happened to raise power from the Mode 2 range to 17%? Need to add this to the stem. Fixed. Now Sat.

41 F 3 N S A(41.10)

C(41.10) - Explanation for C says that MCPR must be at least 1.10. Answer C 42 F 3 N E says that MCPR is reading 1.10. How does this violate TS safety limit? Fixed. Now Sat.

43 F B S D(41.10)- HANDOUT -

44 F 3 B S B(41.12) - 2010 NRC EXAM -

45 F 2 B S A(41.12) Page 3 of 7 OBDI 202 - IOLE Process

ES-401 GG-2012-12 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 46 H 2 N S A(41.10) 47 F 2 N S C(41.5)

A(41.7) - 2011 NRC EXAM - Bullet the stem. Replace rated power. Add the word 48 H 2 B E immediate before operational impact in the stem. This will preclude B completely from being a correct answer. Fixed. Now Sat.

49 F 2 N S D(41.10) 50 F 3 N S A(41.10) 51 H 3 N S B(41.7) -

52 F 1 N S B(41.7) 53 F 2 B S C(41.10) - 2011 NRC EXAM 54 F 2 N S C(41.10) 55 F 2 N S D(41.7) 56 F 3 N S C(41.7) 57 H 2 N S A(41.5) 58 H 2 N S D(41.5) 59 H 2 N S B(41.7) 60 H 2 N S D(41.10) 61 H 2 N S A(41.10) 62 F 2 N S D(41.7) 63 H 2 N S C(41.7) 64 F 3 N S C(41.10) 65 H 2 N U A(41.5) - Not plausible that closing TCV will raise generator output. Fixed. Now Sat.

66 F 3 N S B(41.7)

C(41.7) K/A requires loss or malfunction of reactor vessel internals and how that 67 H 2 X N U would affect nuclear boiler instrumentation. This question discusses loss of a thermocouple and how it affects recirc pumps. Fixed. Now Sat. Page 4 of 7 OBDI 202 - IOLE Process

ES-401 GG-2012-12 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 68 F 2 N S C(41.7) 69 H 2 N S D(41.5 70 H 2 N S B(41.4) 71 H 2 N S D 72 F 2 B S D(41.10) 73 F 2 N S B(41.10)

C(41.10) - Replace rated power. Also, make this type of question more operationally 74 H 3 N E valid vice simple memorization which is discouraged in NUREG 1021. Fixed. Now Sat.

75 F 2 N S B(41.10)

D(43.2) - 2011 NRC EXAM - Only have to know 2 of 3 parts in order to determine 76 H 3 B E correct answer. Three part answers are very difficult. Recommend going with 2 by 2 style answer. Fixed. Now Sat.

77 H 3 N S A(43.2) 78 F 2 N S B(43.2) 79 H 2 N S A(43.10)

C(43.5) - HANDOUT - Change from plant equipment to safe shutdown equipment 80 F 2 N E Fixed. Now Sat.

81 H 2 N S B(43.5) 82 H 2 N S A(43.5) 83 F 2 B S B(43.3) 84 F 2 N S C(43.4) 85 F 3 N S D(43.2) 86 H 2 B S C(43.5) - HANDOUT -

87 H 3 N S A(43.2) 88 H 2 N S C(43.5) - HANDOUT -

89 H 3 N S D(43.5) - Page 5 of 7 OBDI 202 - IOLE Process

ES-401 GG-2012-12 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only D(43.2) - 2011 NRC EXAM - Replace rated power. K/A mismatch. K/A requires use 90 H 3 X B U of procedures to mitigate the consequences of the failure not knowledge of TS.

Fixed. Now Sat.

91 F 2 N S B(43.5) 92 F 2 N S A(43.5) - HANDOUT B(43.2) - K/A mismatch. K/A requires use of procedures to mitigate event. Fixed.

93 H 2 X N U Now Sat.

94 H 2 N S D(43.5)

C(43.5) - Entry conditions by itself would make this question unsat. Understand K/A 95 H 2 N E requires knowledge of entry level conditions but need to add some meat to this question that would make it SRO level knowledge. Fixed. Now Sat.

C(43.5) - Minimum shift manning is RO level knowledge. Also, LOD=1. Fixed. Now 96 F 1 N U Sat.

B(43.2) - HANDOUT - K/A mismatch. Use procedures to mitigate the event per the 97 H 3 X N U K/A. Fixed. Now Sat.

98 F 2 N U B(43.2) - RO level knowledge. Fixed. Now Sat.

99 H 2 N S A(43.5) 100 H 3 N S A(43.5) - HANDOUT B= 15 F= 26 E= 39 RO TOTALS: M= 0 H= 49 U= 6 Additional Notes:

N= 60 B= 4 F= 9 E= 10 SRO TOTALS: M= 0 H= 16 U= 8 Additional Notes:

N= 21

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner comments are indicated in blue. Page 6 of 7 OBDI 202 - IOLE Process

ES-401 GG-2012-12 7 Form ES-401-9

3. Average difficulty is __ ___ on the RO exam and ___ on the SRO exam.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 43.1 =

41.2 = 43.2 =

41.3 = 43.3 =

41.4 = 43.4 =

41.5 = 43.5 =

41.6 = 43.6 =

41.7 = 43.7 =

41.8 =

41.9 =

41.10 =

41.11 =

41.12 =

41.13 =

41.14 =

5. The answer distribution is: RO / SRO A = 13 (17%) / 6 (24%)

B = 16 (22%) / 7 (28%)

C = 21 (28%) / 5 (20%)

D = 25 (33%) / 7 (28%)

6. There are 9 questions with attachments provided. Page 7 of 7 OBDI 202 - IOLE Process