ML110970399

WT-2011-03 Draft Written Exam Comments

ML110970399
Person / Time
Site:	Waterford
Issue date:	03/21/2011
From:	NRC Region 4
To:	Entergy Operations
References
50-382/11-301
Download: ML110970399 (7)

Text

ES-401 WT-2011-03 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 1

H 3

N S

D/41.7 2

H 2

N S

A/41.7 3

H 3

N S

D/41.10 4

F 3

N S

C/41.10 5

H 4

N S

C/41.14 6

H 4

N S

A/41.3 7

F 3

N S

B/41.10 8

H 3

N S

B/41.7 9

F 2

B S

D/41.7

ES-401 WT-2011-03 2

Form ES-401-9 Page 2 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 10 H

3 N

S B/41.10 11 F

3 N

S A/41.10 12 F

2 x

M U

A/41.10 2007 RO NRC Retake Exam Q65 - Distractors C and D not credible. Not credible to affect Main Steam piping support when refilling a dry S/G. Fixed 13 F

2 N

S B/41.10 14 F

3 N

S A/41.7 15 F

3 x

B U

C/41.5, 41.10 Not credible to use the controller on CP-8 after the loss of power and when the stem indicates that the valve must be operated manually. Replace question. Fixed 16 H

3 B

S B/41.7 17 F

2 N

S D/41.7 18 F

3 B

S D/41.

19 H

3 N

S D/41.10 20 H

3 M

S D/41.10 21 H

3 N

S D/41.7 22 H

3 N

S C/41.6 23 F

3 N

S D/41.7 24 H

3 N

E C/41.10 Asking about operability which is SRO knowledge. Perhaps change it from violation of operability requirements to configurations are not allowed in Mode 1?

Fixed 25 H

3 N

S A/41.5 26 H

2 N

E B/41.10 Would there be any annunciator alarms locked in? If not, modify stem and conditions to give the applicant annunciator alarm(s) for this event. Will add annunciator PZR Level Hi/Lo to stem. Fixed 27 H

3 N

S D/41.7 28 H

3 N

U B/41.5 LOD=1 Fixed 29 H

3 N

S D/41.5 30 H

2 N

S A/41.10

ES-401 WT-2011-03 3

Form ES-401-9 Page 3 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 31 H

4 N

S C/41.7 32 F

3 N

S D/41.

33 H

2 N

U C/41.8 RCS pressure is 800 psia as given in the stem. Without even looking at the graphs the applicant will realize that there will be no flow via LPSI. How does that make Distractors B and D plausible? Fixed 34 F

2 N

S B/41.5 35 H

3 N

S A/41.7 36 H

3 N

S C/41.7 37 F

3 N

S B/41.7 38 F

2 N

S A/41.7 39 H

2 N

S C/41.7 40 H

3 N

S D/41.5 41 H

3 N

E B/41.5 Distractor A is not plausible. Doing nothing when a containment fan cooler has tripped with increasing containment pressure is simply not credible. Fixed 42 H

3 M

S D/41.5 43 F

3 N

S D/41.5 44 H

3 N

S A/41.7 45 H

3 M

S A/41.4 46 F

2 x

x M

U D/41.5 The K/A deals with testing the knowledge of the effects on AFW operations when there is feed line voiding and water hammer. This question is written to test the knowledge of what design feature is in the system to protect against water hammer.

Fixed 47 F

2 N

E C/41.7 Distractors A and B can be eliminated since rectifiers supply DC loads from AC. Distractor D can be eliminated since no DC Bypass supply exists. Distracters A and B have been frequently missed at W3. Will fix Distracter D. Fixed 48 H

3 N

E A/41.5 The Explanation section for the correct answer says a second ground in positive or negative direction may cause the DC bus to be inoperable. This is consistent with the text in the reference SD-DC (page 13). In the reference slide provided, it emphasizes that the concern is a second ground of opposite polarity, which isnt consistent with the other references. If the polarity of the second ground

ES-401 WT-2011-03 4

Form ES-401-9 Page 4 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only doesnt matter, then both answers A and C are correct. If it does matter, and it has to be a negative ground, then there is an issue with the reference materials.

As a recommendation, change the wording in the question A new to An additional.. Since the reported ground from the RAB is a new item, the applicant may be confused that this is what is being referred to in the question.

Remove reference to declaring equipment inoperable. Polarity does matter. Will remove inoperable statement. Will change new to additional. Reference material is ok. Fixed 49 F

3 N

E D/41.7 Distractor C assumes no operator action is required. This can be easily eliminated as an outlier from the other distractors. Fixed 50 H

3 N

S D/41.7 51 H

3 M

S A/41.7 From 2008 NRC Exam 52 H

2 N

E C/41.5 Distractor D stops Tower B fans while Distractors A-C start Tower B fans in fast speed. Distractor D can be eliminated by just knowing the first parts of Distractors A-C. Editorial note: substitute internals for intervals in Distractor B.

Fixed 53 H

2 B

S A/41.10 54 F

2 B

S C/41.7 55 H

2 N

E D/41.4 Distractors A and B not credible with the first part of the answer being that containment integrity is met with a containment penetration removed. Also, ROs not required to know TS action statements over one hour which eliminates Distractor C.

Mode 6 allows containment penetration to be removed as long as it is planned and there are contingency actions in place. This makes the first part of A and B credible.

It would not be met if it was unplanned and just discovered by a watchstander making their rounds. Second part of question will be changed to the action statement for.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> for required RO knowledge. Fixed 56 H

2 x

N U

C/41.4 T/F question - Stem information not required to answer question. Also K/A mismatch (K/A points to knowledge of physical connections and cause effect relationships between CRDS and CCW, not knowledge of required actions). Will revise. Fixed 57 F

2 N

S B/41.7 58 H

2 N

S C/41.6 59 F

2 B

S B/41.5

ES-401 WT-2011-03 5

Form ES-401-9 Page 5 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 60 F

2 N

S A/41.8 61 F

2 N

E B/41.7 Distractors A and D can be eliminated since its not credible to have spent fuel pool temperature lowering with the given failure. Fixed 62 H

3 N

S D/41.10 63 H

3 M

E A/41.5 2009 NRC Exam - Not credible to have distractors with S/G pressure increasing with increased steam demand. Fixed 64 F

2 N

S B/41.7 65 F

2 N

S C/41.5 66 F

2 N

E B/41.10 Distractors A and C not credible - Not credible to require safety glasses in the control room when compared with respirator glasses. Fixed 67 H

3 N

S D/41.10 68 F

2 N

S C/41.10 69 F

3 M

S B/41.10 70 F

2 N

E B/41.5 Second part of Distractors A and C should be a specified rate to seem plausible. Fixed 71 F

2 B

S D/ 41.12 2010 NRC Exam -

72 F

2 N

S C/41.12 73 F

1 B

U A/41.10 2007 RO Retake Q75 - LOD=1 Fixed 74 F

3 N

S B/41.10 75 F

3 N

S C/41.10 76 H

4 N

S C/43.5 77 H

3 N

S B/43.5 78 H

3 M

S A/43.5 2010 NRC Exam SRO Q16 -

79 H

3 N

S A/43.5 80 F

2 N

S C/43.2 81 F

2 x

N U

D/43.5 Distractors A and C not plausible considering there is no generator malfunction given in the stem. Fixed

ES-401 WT-2011-03 6

Form ES-401-9 Page 6 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 82 F

2 N

S A/43.5 83 F

2 N

S C/43.5 84 F

2 N

S B/43.5 85 F

2 N

S D/43. 2 86 H

3 N

S A/43.5 87 H

3 N

S B/43.5 88 F

2 N

S A/43.5 89 H

3 B

S C/43.5 90 H

3 N

S A/43.5 91 H

3 N

E B/43.5 No indication in the stem that pressurizer level is dropping so why would distractors C and D be plausible? Will add that PZR level is lowering slowly which is accurate for given conditions. Fixed 92 F

2 x

N U

D/43.2 Continuing refueling without audible count rate in Containment does not seem plausible (Distractors A and B) - Also, this is tied to Generic K/A 2.1.32.

Question S 5 (80) is written to this one as well. Recommend re-sampling and replacing one of them. Resample K/A. Fixed 93 H

2 N

U B/43.5 RO level question. Systems knowledge. Fixed 94 H

4 N

S A/43.6 95 H

3 N

U C/43.7 Not plausible to secure the Normal SFPHX and replace it with the Backup SFPHX with no indicated problems in the stem. Also, RO level knowledge. Fixed 96 H

4 N

S D/43.2 97 H

3 B

S B/43.4 Q24 2009 NRC Exam 98 H

4 N

S D/43.1 99 H

3 N

S C/43.5 100 H

4 N

S D/43.5 Editorial - pedigree of this question indicates this as 43.4 knowledge but 43.5 is correct.

RO TOTALS:

B= 9 F= 35 E= 12 Additional Notes:

ES-401 WT-2011-03 7

Form ES-401-9 Page 7 of 7 OBDI 202 - IOLE Process M= 8 H= 40 U= 7 N= 58 SRO TOTALS:

B= 2 F= 8 E= 1 Additional Notes:

M= 1 H= 17 U= 4 N= 22

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2.

Chief Examiner comments are indicated in blue.

3.

Average difficulty is __ 2.56 ___ on the RO exam and _2.84__ on the SRO exam.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 =

43.1 = 1 41.2 =

43.2 = 4 41.3 = 1 43.3 =

41.4 = 3 43.4 = 1 41.5 = 17 43.5 = 17 41.6 = 2 43.6 = 1 41.7 = 25 43.7 = 1 41.8 = 2 41.9 =

41.10 = 22 41.11 =

41.12 = 2 41.13 =

41.14 = 1

5.

The answer distribution is: RO / SRO A = 13 (17%)

/

6 (24%)

B = 16 (22%)

/

7 (28%)

C = 21 (28%)

/

5 (20%)

D = 25 (33%)

/

7 (28%)

6.

There are 15 questions with attachments provided.