ML100670654

AN2-2009-08-Draft Written Exam Comments

ML100670654
Person / Time
Site:	Arkansas Nuclear
Issue date:	08/07/2009
From:	Brian Larson Operations Branch IV
To:	Entergy Operations
References
Download: ML100670654 (7)

Text

ES-401 AN2-2009-08 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 B E C, 41.5, NRC-2005, CR.

2 H 3 N S D, 41.11, CR.

3 H 3 B E A, 41.5, NRC-2005, CR. Provide steam tables as a reference. Done 4 H 3 M S B, 41.3, CR.

5 F 3 N S C, 41.5, CR. Need reference for procedure basis. Done 6 F 2 B S D, 41.10, NRC-2005, CR.

7 H 2 B E C, 41.14, CR Provide steam tables as a reference. Done 8 H 3 N S B, 41.6, CR. Need ref for Mode definition. Done 9 F 3 B S A, 41.5, NRC-1998,CR.

10 H 3 B S A, 41.10, NRC-2002, CR.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process

ES-401 AN2-2009-08 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 11 F 2 N S D, 41.8, CR.

12 H 3 N E B, 41.5, CR. Provide steam tables as a reference. Done 13 H 2 N E D, 41.7, CR. Add reference for Log Safety channels. Done 14 H 3 N S B, 41.7, CR.

15 H 3 N S C, 41.7, CR. Add ref for MSIS setpoint. Done 16 H 2 N S A, 41.5, CR.

17 H 3 B S B, 41.7, NRC-2005, CR.

18 H 3 N S C, 41.10, CR.

19 H 3 B S A, 41.5, NRC-2000, CR.

20 F 2 N S B, 41.6, CR.

21 F 3 B S D, 41.7, NRC-2005, CR.

22 H 3 N E C, 41.9, CR. Re-word stem. Done 23 H 3 N E A, 41.10, OR. Revise stem. Done 24 F 3 B S B, 41.13, NRC-2005, CR.

25 H 3 N S D, 41.7, CR.

26 H 3 B S B, 41.10, NRC-2002, CR. Add ESD reference. Done 27 H 3 N E A, 41.10, CR. Reword question / answers. Done 28 H 3 N S D, 41.5, CR.

29 H 3 B S C, 41.7, NRC-2002, OR.

30 H 3 N S D, 41.7, CR.

31 F 3 N U A, 41.7, CR. Overlap with Q25. Re-wrote question 32 F 2 M E A, 41.3, CR.

C, 41.5, CR. Change the phrase plant power to reactor power and re-word stem.

33 H 3 N E Done 34 F 3 N E D, 41.7, CR. Page 2 of 7 OBDI 202 - IOLE Process

ES-401 AN2-2009-08 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 35 F 3 N S B, 41.7, CR.

36 F 3 N S C, 41.7, CR. Missing p. 55 of reference. Provided 37 H 2 N S A, 41.10, CR. Add ref for why SW is restored to CCW. Done 38 H 3 M S D, 41.7, NRC-1998, CR.

39 H 3 N S C, 41.7, CR.

40 H 3 M E B, 41.7, NRC-2005, OR.

41 H 3 M S C, 41.7, CR.

42 H 3 N S B, 41.4, CR.

43 F 2 N S D, 41.7, CR.

44 F 2 N S A, 41.4, CR.

45 H 3 N S C, 41.10, CR. Need ref for 485 gpm basis. Done 46 H 3 N S D, 41.7, CR.

47 F 3 N S B, 41.7, CR.

48 H 3 B E D, 41.7, CR.

49 H 3 M E A, 41.7, CR. Need to add local aspect for operation of the EDG. Done 50 H 3 N E B, 41.8, CR.

B, 41.13, NRC-1998, CR. NUREG-1021 recommends not using negatively phrased 51 F 2 X M U questions unless necessary. Re-write question. Done 52 H 3 N S C, 41.10, CR.

53 F 2 N S D, 41.4, CR.

54 F 3 M E A, 41.9, NRC-2005, CR.

55 H 3 N S C, 41.10, CR. Need ref for first part of question. Done 56 F 2 N S B, 41.6, CR.

57 H 3 N S A, 41.6, CR.

58 F 3 N E B, 41.4, CR. Page 3 of 7 OBDI 202 - IOLE Process

ES-401 AN2-2009-08 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 59 F 3 N E C, 41.2, CR. Stem includes info not required. Re-wrote stem 60 H 2 N E C, 41.9, OR.

61 F 3 X B U B, 41.6, NRC-2002, CR. Re-write question to RO level. Done 62 H 3 B S D, 41.5, NRC-2000, CR.

63 H 3 N S C, 41.13, CR. Need ACA reference. Done 64 H 3 N S A, 41.4, CR.

65 F 3 N S C, 41.10, CR.

B, 41.10, CR. Does not test on a generic aspect with regard to temporary 66 F 2 N E management directives. New KA and question 67 F 3 B S A, 41.10, CR.

68 F 3 N S D, 41.10, CR.

69 F 3 N S D, 41.10, CR.

70 F 3 N S A, 41.10, CR 71 H 2 B S D, 41.12, NRC-2005, CR. Need page 2 of question worksheet. Done B, 41.13, CR. SRO-only question in that it tests on an LCO action step and its basis 72 F 3 X M U per the references provided. Replace question. Done 73 F 3 N E C, 41.9, CR. Add procedure reference to stem. Done 74 F 2 N S B, 41.10, CR. Re-wrote question to preclude overlap with revised Q100 75 F 2 M S A, 41.10, CR.

76 H 3 N S D, 43.5, CR.

77 H 3 B S C, 43.5, NRC-2005, CR.

78 H 3 M S B, 43.5, CR.

79 H 4 N S A, 43.5, CR.

80 H 3 N E A, 43.5, CR.

81 H 3 N S C, 43.5, CR.

82 H 3 N E B, 43.2, CR. Page 4 of 7 OBDI 202 - IOLE Process

ES-401 AN2-2009-08 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only B, 43.1, OR. Unsat because it is a direct lookup with the reference provided 83 H 3 N U (1903.010, Attachment 2). Replace question. Done 84 H 3 M S D, 43.5, CR.

85 H 4 N E D, 43.5, CR.

A, 43.2, CR. The indicated correct answer states initiate a shutdown to Hot Standby within one hour. This is technically incorrect; the required action is if operability 86 H 3 X N U cannot be restored within one hour then initiate action to be in Hot Shutdown within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />. Revise answers. Done C, 43.5, OR. This could be argued it is an SRO question, however, an RO should 87 H 2 X N U recognize the reactor would trip and this would be an entry to the STPAs. This makes the question an RO question. Replace question. Done 88 H 3 N S B, 43.5, CR.

89 H 3 M S D, 43.5, CR.

90 H 3 N S C, 43.2, OR.

91 H 3 N S A, 43.5, CR.

C, 43.2, NRC-2002, CR. Technically, this LCO does not REQUIRE increasing the frequency of the water inventory balance. It provides two options; increasing the 92 F 3 B E frequency or being in Hot Standby within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />. Since either is acceptable, it could be argued there is no correct answer. Re-word stem. Done 93 H 3 B E C, 43.6, NRC-2005. Re-write to include second part of KA. Done B, 43.5, CR. Questions involving generic KAs should be generic in nature and not 94 F 2 B E system specific. Re-write question. Done C, 43.2, CR. Questions involving generic KAs should be generic in nature and not 95 F 2 N E system specific. Done 96 F 2 N E D, 43.3, CR.

97 F 2 N E D, 43.3, CR.

98 F 2 M S A, 43.4, CR.

A, 43.7, CR. Questions involving generic KAs should be generic in nature and not 99 F 3 N E system specific. Replace KA. Done 100 F 2 X N U B, 43.5, CR. Because this question tests on a specific operator action and the basis Page 5 of 7 OBDI 202 - IOLE Process

ES-401 AN2-2009-08 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only for that action, it is an RO question. Replace question. Done B= 17 (23%) F= 32 (43%) E= 21 (28%)

RO TOTALS: M= 10 (13%) H= 43 (57%) U= 4 (5%) Additional Notes:

N= 48 (64%) S= 50 (67%)

B= 4 (16%) F= 8 (32%) E= 10 (40%)

Additional Notes: The number of higher cognitive questions is higher than normal.

SRO TOTALS: M= 4 (16%) H= 17 (68%) U= 4 (16%) This was discussed with the facility who agrees that this is acceptable.

N= 17 (68%) S= 11 (44%)

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is __2.76_ on the RO exam and 2.80 on the SRO exam. Both these values are within the normal expected difficulty range for this reviewer.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 1 41.2 = 1 43.2 = 5 41.3 = 2 43.3 = 2 41.4 = 5 43.4 = 1 41.5 = 10 43.5 = 14 41.6 = 5 43.6 = 1 41.7 = 21 43.7 = 1 41.8 = 2 41.9 = 4 41.10 = 18 41.11 = 1 41.12 = 1 41.13 = 4 41.14 = 1

5. The answer distribution is: RO / SRO A = 18 / 6 B = 20 / 6 C = 19 / 7 Page 6 of 7 OBDI 202 - IOLE Process

ES-401 AN2-2009-08 7 Form ES-401-9 D = 18 / 6

6. There are 4 questions on the RO exam and _3_ questions on the SRO exam with references provided.

7. There are 21 questions on this exam from previous NRC exams at ANO-2 (11 were from the 2005 exam). While there are no questions from the previous 2 NRC exams (2006, 2008), this is an unusually high number of previous NRC exam questions. Page 7 of 7 OBDI 202 - IOLE Process