ML14184B149

2014-05 Draft Written Exam Comments

ML14184B149
Person / Time
Site:	Fort Calhoun
Issue date:	05/22/2014
From:	Vincent Gaddy Operations Branch IV
To:	Omaha Public Power District
laura hurley
References
50-285/14-005
Download: ML14184B149 (7)

Text

ES-401 FC-2014-05 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 3 N S C, 41.10, CR 2 H 3 N S B, 41.10, CR 3 H 3 N S C, 41.2, CR 4 H 3 B S D, 41.3, CR 5 H 3 N S B, 41.5, CR 6 F 3 N S D, 41.10, CR E B, 41.10, CR - Revise stem to read is the operator first directed to take 7 F 3 E N S Changes made.

8 F 3 B S C, 41.7, CR 9 H 3 N S C, 41.3, OR 10 H 3 N S A, 41.7, CR 11 H 3 N S A, 41.5, CR 12 F 3 N S C, 41.10, CR A, 41.10, CR - Change last part of stem to read Which of the following actions E

13 H 3 E N should be taken?

S Changes made.

14 H 3 B S C, 41.10, CR 15 H 3 N S D, 41.4, CR 16 H 3 N S A, 41.4, OR 17 H 3 N S B, 41.4, CR 18 H 2 N S B, 41.4, CR 19 F 3 N S D, 41.6, CR 20 H 3 N S A, 41.6, OR 21 F 3 N S C, 41.10, CR Page 1 of 7 OBDI 202 - IOLE Process

ES-401 FC-2014-05 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E B, 41.4, CR - Are these numbers reasonable for a from memory question?

22 F 3 N S S Numbers are reasonable.

23 F 4 N S A, 41.12, CR 24 H 3 N S D, 41.10, CR 25 F 3 B S A, 41.9, CR 26 H 3 N S A, 41.5, CR 27 H 3 N S B, 41.4, CR 28 H 3 N S C, 41.3, CR 29 H 3 N S A, 41.5, CR 30 F 3 N S C, 41.5, CR 31 H 3 N S D, 41.5, CR 32 F 3 N S A, 41.4, CR 33 H 2 N S A, 41.3, CR 34 H 3 N S C, 41.4, CR 35 H 3 N S D, 41.4, CR 36 H 3 N S B, 41.5, CR 37 H 3 N S C, 41.6, CR B, 41.13, CR - Insert missing space in the third bullet in the stem.

38 H 3 N S Fixed.

39 H 3 N S A, 41.9, CR C, 41.9, CR - This is a KA mismatch because it does not address the loss of service U

40 H 3 U N water on the CCS system.

S This question is ok as written.

41 H 3 M S B, 41.9, CR 42 F 2 N S B, 41.4, CR 43 F 3 N S B, 41.4, CR Page 2 of 7 OBDI 202 - IOLE Process

ES-401 FC-2014-05 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E E D, 41.5, CR - Does the stem need to say initially respond?

44 H 3 N S S Question revised.

45 F 3 N S C, 41.7, CR 46 F 3 N S B, 41.7, CR 47 F 3 B S B, 41.7, CR 48 F 3 N S A, 41.7, CR 49 F 3 B S C, 41.11, CR 50 F 3 N S B, 41.11, CR 51 H 3 N S A, 41.4, CR 52 H 3 N S C, 41.4, CR 53 H 3 N S B, 41.4, CR 54 F 4 N S C, 41.9, CR 55 H 3 N S D, 41.9, CR 56 F 3 N S B, 41.3, CR 57 F 3 B S C, 41.6, CR 58 F 3 N S D, 41.2, CR 59 F 2 B S A, 41.8, CR 60 F 3 B S C, 41.7, CR 61 H 3 B S A, 41.5, CR D, 41.4, CR - Revise the stem to ask at what ehc pressure does the standby pump E E 62 F 3 B auto start rather than the alarm setpoint.

S S Question has been revised.

63 F 3 B S D, 41.4, CR 64 F 3 N S D, 41.13, CR E C, 41.4, CR - Need to replace either this question or question 52 due to the overlap 65 F 3 B S in what is being tested. A replacement KA may be necessary. Page 3 of 7 OBDI 202 - IOLE Process

ES-401 FC-2014-05 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Resampled question.

66 F 2 N S D, 41.10, CR 67 F 3 N S B, 41.10, CR 68 F 2 B S B, 41.6, CR 69 F 3 N S B, 41.10, CR A, 41.10, CR - Remove so that it is returned to the correct E E 70 F 3 B cubicle in the correct position from distractor D so it is consistent with the others.

S S Changes made.

71 H 3 B S C, 41.12, CR 72 F 3 N S B, 41.12, CR 73 F 3 N S B, 41.10, CR 74 H 3 N S B, 41.10, OR 75 F 3 N S A, 41.10, CR 76 H 3 N S B, 43.5, CR 77 H 3 N S D, 43.5, CR 78 H 3 N S D, 43.5, CR 79 H 3 N S A, 43.5, CR 80 H 3 N S C, 43.5, CR E E A, 43.5, CR - Add procedure noun name to AOP-20 in distractor D.

81 H 3 N S S Noun name added.

82 H 3 N S D, 43.5, CR 83 H 3 N S B, 43.7, CR C, 43.2, OR - We do not expect applicants to know 30 day LCO actions from E

84 H 4 E N memory.

S Will provide the TS reference.

U C, 41.10, CR - This is an RO question because it tests on a specific action.

85 H 3 U N S Will rewrite. Page 4 of 7 OBDI 202 - IOLE Process

ES-401 FC-2014-05 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Was rewritten.

86 H 3 N S D, 43.5, CR 87 F 2 B S A, 43.5, CR E E C, 43.2, OR - Is this a reasonable from memory TS question?

88 F 3 N S S Facility believes this to be a reasonable memory question.

89 H 3 N S D, 43.5, CR 90 F 3 N S B, 43.5, CR D, 43.2, OR - We need to talk about this one.

91 H 4 N S Get back to this.

D, 43.2, OR - We need to talk about this one. Are operators required to know the E E contents of chemistry procedures from memory?

92 H 4 N S S The actions are contained in the AOP, not the chemistry procedure. The question is ok as written.

B, 43.5, CR - Distractors A and C are not credible since there doesnt appear to be U any fuel handling in progress from the information in the stem.

93 F 2 U N S Will revise the question.

Question was revised.

E E A, 41.10, CR - Why is this not RO knowledge?

94 F 3 N S S This is an SRO only duty - question is ok as written.

95 F 2 N S C, 41.10, CR E E C, 43.2, CR - Is this a reasonable from memory TS question?

96 F 3 N S S Facility believed this is a reasonable TS memory question.

E E D, 43.4, CR - There is no correct answer since this is not a normal entry.

97 F 3 N S S Per the emergency plan procedures, this question is ok as written.

98 F 3 N S D, 43.4, CR U U C, 41.10, CR - This is an RO question because it tests on a specific operator action.

99 F 3 N S S This is an SRO only function so question is ok as written.

100 F 3 B S B, 41.10, CR Page 5 of 7 OBDI 202 - IOLE Process

ES-401 FC-2014-05 Written Examination Review Worksheet Form ES-401-9 B= 19 (25%) F= 38 (50%) E= 5 (7%)

RO TOTALS: M= 4 (5%) H= 37 (50%) U= 1 (1%) Additional Notes:

N= 52 (70%) S= 69 (92%)

B= 2 (8%) F= 11 (44%) E= 7 (28%)

SRO TOTALS: M= 0 (0%) H= 14 (56%) U= 3 (12%) Additional Notes:

N= 23 (92%) S= 15 (60%)

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner follow-up comments are indicated in blue.

3. Average difficulty is _2.94 on the RO exam and 3.00 on the SRO exam. These values for this reviewer indicate the RO exam and the SRO exam are slightly higher than average difficulty.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 0 41.2 = 2 43.2 = 5 41.3 = 5 43.3 = 0 41.4 = 17 43.4 = 2 41.5 = 9 43.5 = 12 41.6 = 5 43.6 = 0 41.7 = 7 43.7 = 1 41.8 = 1 41.10 = 5 41.9 = 6 41.10 = 16 41.11 = 2 41.12 = 3 41.13 = 2 41.14 = 0

5. The answer distribution is:

A= 18 / 4 B= 22 / 5 C= 20 / 7 D= 15 / 9

6. There is/are 4 / 4 question(s) with attachments provided. Page 6 of 7 OBDI 202 - IOLE Process

ES-401 FC-2014-05 7 Form ES-401-9

8. This exam was reviewed by S. Garchow on April 7-8, 2014.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 7 of 7 OBDI 202 - IOLE Process