ML100670654

AN2-2009-08-Draft Written Exam Comments

ML100670654
Person / Time
Site:	Arkansas Nuclear
Issue date:	08/07/2009
From:	Brian Larson Operations Branch IV
To:	Entergy Operations
References
Download: ML100670654 (7)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9 AN2-2009-08 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 1

H 3

B E

C, 41.5, NRC-2005, CR.

2 H

3 N

S D, 41.11, CR.

3 H

3 B

E A, 41.5, NRC-2005, CR. Provide steam tables as a reference. Done 4

H 3

M S

B, 41.3, CR.

5 F

3 N

S C, 41.5, CR. Need reference for procedure basis. Done 6

F 2

B S

D, 41.10, NRC-2005, CR.

7 H

2 B

E C, 41.14, CR Provide steam tables as a reference. Done 8

H 3

N S

B, 41.6, CR. Need ref for Mode definition. Done 9

F 3

B S

A, 41.5, NRC-1998,CR.

10 H

3 B

S A, 41.10, NRC-2002, CR.

ES-401 2

Form ES-401-9 AN2-2009-08 Page 2 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 11 F

2 N

S D, 41.8, CR.

12 H

3 N

E B, 41.5, CR. Provide steam tables as a reference. Done 13 H

2 N

E D, 41.7, CR. Add reference for Log Safety channels. Done 14 H

3 N

S B, 41.7, CR.

15 H

3 N

S C, 41.7, CR. Add ref for MSIS setpoint. Done 16 H

2 N

S A, 41.5, CR.

17 H

3 B

S B, 41.7, NRC-2005, CR.

18 H

3 N

S C, 41.10, CR.

19 H

3 B

S A, 41.5, NRC-2000, CR.

20 F

2 N

S B, 41.6, CR.

21 F

3 B

S D, 41.7, NRC-2005, CR.

22 H

3 N

E C, 41.9, CR. Re-word stem. Done 23 H

3 N

E A, 41.10, OR. Revise stem. Done 24 F

3 B

S B, 41.13, NRC-2005, CR.

25 H

3 N

S D, 41.7, CR.

26 H

3 B

S B, 41.10, NRC-2002, CR. Add ESD reference. Done 27 H

3 N

E A, 41.10, CR. Reword question / answers. Done 28 H

3 N

S D, 41.5, CR.

29 H

3 B

S C, 41.7, NRC-2002, OR.

30 H

3 N

S D, 41.7, CR.

31 F

3 N

U A, 41.7, CR. Overlap with Q25. Re-wrote question 32 F

2 M

E A, 41.3, CR.

33 H

3 N

E C, 41.5, CR. Change the phrase plant power to reactor power and re-word stem.

Done 34 F

3 N

E D, 41.7, CR.

ES-401 3

Form ES-401-9 AN2-2009-08 Page 3 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 35 F

3 N

S B, 41.7, CR.

36 F

3 N

S C, 41.7, CR. Missing p. 55 of reference. Provided 37 H

2 N

S A, 41.10, CR. Add ref for why SW is restored to CCW. Done 38 H

3 M

S D, 41.7, NRC-1998, CR.

39 H

3 N

S C, 41.7, CR.

40 H

3 M

E B, 41.7, NRC-2005, OR.

41 H

3 M

S C, 41.7, CR.

42 H

3 N

S B, 41.4, CR.

43 F

2 N

S D, 41.7, CR.

44 F

2 N

S A, 41.4, CR.

45 H

3 N

S C, 41.10, CR. Need ref for 485 gpm basis. Done 46 H

3 N

S D, 41.7, CR.

47 F

3 N

S B, 41.7, CR.

48 H

3 B

E D, 41.7, CR.

49 H

3 M

E A, 41.7, CR. Need to add local aspect for operation of the EDG. Done 50 H

3 N

E B, 41.8, CR.

51 F

2 X

M U

B, 41.13, NRC-1998, CR. NUREG-1021 recommends not using negatively phrased questions unless necessary. Re-write question. Done 52 H

3 N

S C, 41.10, CR.

53 F

2 N

S D, 41.4, CR.

54 F

3 M

E A, 41.9, NRC-2005, CR.

55 H

3 N

S C, 41.10, CR. Need ref for first part of question. Done 56 F

2 N

S B, 41.6, CR.

57 H

3 N

S A, 41.6, CR.

58 F

3 N

E B, 41.4, CR.

ES-401 4

Form ES-401-9 AN2-2009-08 Page 4 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 59 F

3 N

E C, 41.2, CR. Stem includes info not required. Re-wrote stem 60 H

2 N

E C, 41.9, OR.

61 F

3 X

B U

B, 41.6, NRC-2002, CR. Re-write question to RO level. Done 62 H

3 B

S D, 41.5, NRC-2000, CR.

63 H

3 N

S C, 41.13, CR. Need ACA reference. Done 64 H

3 N

S A, 41.4, CR.

65 F

3 N

S C, 41.10, CR.

66 F

2 N

E B, 41.10, CR. Does not test on a generic aspect with regard to temporary management directives. New KA and question 67 F

3 B

S A, 41.10, CR.

68 F

3 N

S D, 41.10, CR.

69 F

3 N

S D, 41.10, CR.

70 F

3 N

S A, 41.10, CR 71 H

2 B

S D, 41.12, NRC-2005, CR. Need page 2 of question worksheet. Done 72 F

3 X

M U

B, 41.13, CR. SRO-only question in that it tests on an LCO action step and its basis per the references provided. Replace question. Done 73 F

3 N

E C, 41.9, CR. Add procedure reference to stem. Done 74 F

2 N

S B, 41.10, CR. Re-wrote question to preclude overlap with revised Q100 75 F

2 M

S A, 41.10, CR.

76 H

3 N

S D, 43.5, CR.

77 H

3 B

S C, 43.5, NRC-2005, CR.

78 H

3 M

S B, 43.5, CR.

79 H

4 N

S A, 43.5, CR.

80 H

3 N

E A, 43.5, CR.

81 H

3 N

S C, 43.5, CR.

82 H

3 N

E B, 43.2, CR.

ES-401 5

Form ES-401-9 AN2-2009-08 Page 5 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 83 H

3 N

U B, 43.1, OR. Unsat because it is a direct lookup with the reference provided (1903.010, Attachment 2). Replace question. Done 84 H

3 M

S D, 43.5, CR.

85 H

4 N

E D, 43.5, CR.

86 H

3 X

N U

A, 43.2, CR. The indicated correct answer states initiate a shutdown to Hot Standby within one hour. This is technically incorrect; the required action is if operability cannot be restored within one hour then initiate action to be in Hot Shutdown within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />. Revise answers. Done 87 H

2 X

N U

C, 43.5, OR. This could be argued it is an SRO question, however, an RO should recognize the reactor would trip and this would be an entry to the STPAs. This makes the question an RO question. Replace question. Done 88 H

3 N

S B, 43.5, CR.

89 H

3 M

S D, 43.5, CR.

90 H

3 N

S C, 43.2, OR.

91 H

3 N

S A, 43.5, CR.

92 F

3 B

E C, 43.2, NRC-2002, CR. Technically, this LCO does not REQUIRE increasing the frequency of the water inventory balance. It provides two options; increasing the frequency or being in Hot Standby within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />. Since either is acceptable, it could be argued there is no correct answer. Re-word stem. Done 93 H

3 B

E C, 43.6, NRC-2005. Re-write to include second part of KA. Done 94 F

2 B

E B, 43.5, CR. Questions involving generic KAs should be generic in nature and not system specific. Re-write question. Done 95 F

2 N

E C, 43.2, CR. Questions involving generic KAs should be generic in nature and not system specific. Done 96 F

2 N

E D, 43.3, CR.

97 F

2 N

E D, 43.3, CR.

98 F

2 M

S A, 43.4, CR.

99 F

3 N

E A, 43.7, CR. Questions involving generic KAs should be generic in nature and not system specific. Replace KA. Done 100 F

2 X

N U

B, 43.5, CR. Because this question tests on a specific operator action and the basis

ES-401 6

Form ES-401-9 AN2-2009-08 Page 6 of 7 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation for that action, it is an RO question. Replace question. Done B= 17 (23%)

F= 32 (43%)

E= 21 (28%)

M= 10 (13%)

H= 43 (57%)

U= 4 (5%)

RO TOTALS:

N= 48 (64%)

S= 50 (67%)

Additional Notes:

B= 4 (16%)

F= 8 (32%)

E= 10 (40%)

M= 4 (16%)

H= 17 (68%)

U= 4 (16%)

SRO TOTALS:

N= 17 (68%)

S= 11 (44%)

Additional Notes: The number of higher cognitive questions is higher than normal.

This was discussed with the facility who agrees that this is acceptable.

GENERAL COMMENT

S:

1.

Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2.

Chief Examiner comments are indicated in blue.

3.

Average difficulty is __2.76_ on the RO exam and 2.80 on the SRO exam. Both these values are within the normal expected difficulty range for this reviewer.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 1 41.2 = 1 43.2 = 5 41.3 = 2 43.3 = 2 41.4 = 5 43.4 = 1 41.5 = 10 43.5 = 14 41.6 = 5 43.6 = 1 41.7 = 21 43.7 = 1 41.8 = 2 41.9 = 4 41.10 = 18 41.11 = 1 41.12 = 1 41.13 = 4 41.14 = 1

5.

The answer distribution is: RO / SRO A = 18

/ 6 B = 20

/ 6 C = 19

/ 7

ES-401 7

Form ES-401-9 AN2-2009-08 Page 7 of 7 OBDI 202 - IOLE Process

6.

T D = 18

/ 6 here are 4 questions on the RO exam and _3_ questions on the SRO exam with references provided.

7.

There are 21 questions on this exam from previous NRC exams at ANO-2 (11 were from the 2005 exam). While there are no questions from the previous 2 NRC exams (2006, 2008), this is an unusually high number of previous NRC exam questions.