ML14181A856

2014-05-DRAFT Written Exam Review_Smg

ML14181A856
Person / Time
Site:	South Texas
Issue date:	05/01/2014
From:	Vincent Gaddy Operations Branch IV
To:	South Texas
laura hurley
References
50-498/14-005, 50-499/14-005
Download: ML14181A856 (7)

Text

ES-401 ST-2014-04 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Comment - Delete the Equations and Conversions Handout Sheet - Done 1 F 3 N S C, 41.4, CR The handout has been deleted.

2 F 2 N S C, 41.10, CR 3 F 2 N S B, 41.10, CR 4 H 3 N S A, 41.9, CR A, 41.4, CR - This is a KA mismatch because the question does not address a U U 5 H 3 N physical relationship or cause and effect between service water and CCW.

S S OK as written - it was semantics.

A, 41.10, CR - This is an RO question because it tests on interpreting tech specs.

E E Suggest changing the reference to a station procedure.

6 F 3 N S S Will correct.

The reference was changed to the training materials.

7 H 3 B S B, 41.11, CR 8 F 3 B S B, 41.6, CR 9 F 3 N S D, 41.10, CR 10 H 3 B S C, 41.5, CR 11 F 3 B S A, 41.8, CR 12 H 3 B S B, 41.6, OR 13 H 3 B S B, 41.4, CR C, 41.10, CR - This is a tier 3 question so it must be generic and not test on an E E action dependent on a specific scenario.

14 F 3 B S S After discussing, the immediate actions for the turbine not tripping are generic enough to be considered generic.

15 F 3 B S A, 41.10, CR 16 F 2 B S D, 41.5, CR 17 H 3 B S A, 41.10, CR Page 1 of 7 OBDI 202 - IOLE Process

ES-401 ST-2014-04 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 18 H 4 B S C, 41.5, CR 19 H 3 B S C, 41.7, CR 20 F 2 B S B, 41.4, CR 21 H 3 B S B, 41.4, CR 22 F 3 B S D, 41.7, CR 23 H 3 B S D, 41.3, CR 24 H 3 B S D, 41.3, CR 25 H 3 B S D, 41.4, CR 26 H 3 B S B, 41.4, CR 27 H 3 B S C, 41.6, CR 28 H 3 B S C, 41.5, CR 29 F 2 N S C, 41.10, CR 30 H 3 B S B, 41.11, CR 31 H 3 B S C, 41.12, CR 32 F 3 B S A, 41.11, CR 33 H 3 N S C, 41.4, CR C, 41.4, CR - One could plausibly argue there is no correct answer since damage E E could occur in as little as 4 minutes, not 3.

34 F 4 N S S Will correct.

This has been corrected.

35 H 3 N S N, 41.10, CR 36 F 3 N S N, 41.1, CR 37 F 3 N S C, 41.4, CR 38 H 4 N S D, 41.10, CR 39 H 3 N S C, 41.4, CR 40 H 3 N S A, 41.4, CR Page 2 of 7 OBDI 202 - IOLE Process

ES-401 ST-2014-04 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 41 F 3 N S A, 41.10, CR 42 H 3 N S B, 41.4, CR 43 H 3 N S D, 41.4, CR 44 H 3 N S B, 41.7, CR 45 H 3 N S A, 41.7, CR 46 H 3 N S C, 41.5, CR 47 F 3 N S D, 41.6, CR 48 H 3 N S D, 41.5, CR 49 H 3 N S A, 41.4, CR 50 F 3 N S B, 41.9, CR 51 H 2 N S A, 41.2, CR 52 F 2 N S A, 41.4, CR 53 F 3 N S B, 41.3, CR B, 41.10, CR - Revise the last sentence in the stem to read something like; Which of the following would be the FIRST time the AFW Storage Tank minimum technical E E specification volume would not be met. I dont think exceeding a minimum volume 54 H 3 N makes grammatical sense.

S S Will correct.

This has been corrected.

55 F 3 N S D, 41.5, CR A, 41.7, CR - This is a KA mismatch because it does not address the restoration of U U power part of the KA.

56 H 3 N S S Will fix.

This question was rewritten.

57 F 2 N S D, 41.12, CR 58 F 3 N S C, 41.4, CR 59 F 3 N S D, 41.4, CR 60 F 3 E N E D, 41.4, CR - Recommend adding ONLY to the end of all four distractors. Page 3 of 7 OBDI 202 - IOLE Process

ES-401 ST-2014-04 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S S Will fix.

This was corrected.

61 F 3 N S D, 41.12, CR 62 H 3 N S C, 41.4, CR A, 41.10, CR - This is a tier 3 question so it needs to be generic in nature. You may E E want to pick another KA.

63 F 2 N S S Will pick a new KA.

This question was rewritten.

64 H 3 N S B, 41.4, CR B, 41.5, CR - This is a KA mismatch because it does not address a design feature or U U interlock.

65 F 2 N S S Will pick a new KA.

This question was rewritten.

D, 41.4, CR - This is a tier 3 question so it needs to be generic in nature.

E E 66 H 3 N Will revise question.

S S This question was rewritten.

67 H 3 N S C, 41.7, CR 68 H 3 N S A, 41.5, CR 69 F 3 N S D, 41.3, CR 70 F 3 N S B, 41.4, CR 71 H 3 N S D, 41.4, CR 72 F 3 N S A, 41.4, CR 73 H 3 N S B, 41.4, CR 74 H 3 N S A, 41.10, CR 75 H 2 N S D, 41.10, CR 76 F 3 B S A, 43.2, CR A, 43.5, OR - Need the reference to evaluate this question.

77 H 3 N S Reference received. Page 4 of 7 OBDI 202 - IOLE Process

ES-401 ST-2014-04 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only This question is ok as written.

78 H 2 B S B, 43.5, OR 79 F 3 N S D, 43.2, CR 80 F 3 N S A, 43.7, CR 81 H 3 B S C, 43.5, CR 82 H 3 B S A, 43.5, CR 83 H 3 B S D, 43.5, CR 84 H 3 B S C, 43.2, CR C, 41.10, OR - This is an RO question because it tests on specific operator actions.

U U 85 H 3 B Will fix.

S S This question was rewritten.

86 H 2 B S A, 43.5, CR 87 H 3 N S C, 43.5, CR 88 H 3 N S C, 43.7, CR 89 H 3 N S D, 43.5, CR 90 H 3 N S A, 43.5, CR 91 H 3 N S D, 43.5, CR D, 43.5, CR - The stem asks effect on TOTAL steam dump flow of the failed valve It is intuitive flow goes down since the valve failed closed. Recommend changing the stem to read effect on TOTAL steam flow. I.e., grammatically the question seems to be asking about the steam flow through the failed closed valve.

E E 92 H 3 N Will fix.

S S Change Which of the following describes the effect on TOTAL steam flow of the failed valve AND the procedural actions the Unit Supervisor should take? to Which of the following describes the effect on TOTAL steam flow AND the procedural actions the Unit Supervisor should take?

93 H 3 N S B, 43.5, CR 94 F 3 N S C, 43.7, CR 95 H 3 N S C, 43.2, CR Page 5 of 7 OBDI 202 - IOLE Process

ES-401 ST-2014-04 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 96 H 3 N S A, 43.5, CR 97 F 3 N S B, 43.3, CR 98 H 4 N S D, 43.5, CR 99 F 3 N S C, 43.7, CR 100 F 2 N S B, 41.10, CR B= 25 (33%) F= 33 (44%) E= 12 RO TOTALS: M= 0 (0%) H= 42 (56%) U= 5 Additional Notes:

N= 50 (67%)

B= 8 (32%) F= 7 (28%) E= 1 SRO TOTALS: M= 0 (0%) H= 18 (72%) U= 1 Additional Notes:

N= 17 (68%)

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner follow-up comments are indicated in blue.

3. Average difficulty is _2.89 on the RO exam and 2.92 on the SRO exam. These values for this reviewer indicate the RO exam and the SRO exam are of average difficulty.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 1 43.1 = 0 41.2 = 1 43.2 = 4 41.3 = 4 43.3 = 1 41.4 = 26 43.4 = 0 41.5 = 9 43.5 = 14 41.6 = 4 43.6 = 0 41.7 = 6 43.7 = 4 41.8 = 1 41.9 = 2 41.10 = 17 41.11 = 3 41.12 = 3 Page 6 of 7 OBDI 202 - IOLE Process

ES-401 ST-2014-04 7 Form ES-401-9 41.13 = 0 41.14 = 0

5. The answer distribution is:

A= 18 /7 B= 18 /4 C= 18 /8 D= 21 /6

6. There is/are 1/2 question(s) with attachments provided.

8. This exam was reviewed by S. Garchow on April 2-3, 2014.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 7 of 7 OBDI 202 - IOLE Process