ML13029A068

2012-11-Draft Written Exam Comments

ML13029A068
Person / Time
Site:	River Bend
Issue date:	11/15/2012
From:	David Strickland Operations Branch IV
To:	Entergy Operations
Laura Hurley
References
ES-401, ES-401-9 50-458/12-011
Download: ML13029A068 (8)

Text

ES-401 RB-2012-11 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 2 N S C, OR, 41.5 Change A (1) to 61% to be more realistic. Change made.DGS 2 H 2 N S A, CR, 41.7 3 H 3 N S B, CR, 41.7 Reword responses C & D to reduce confusion. Change made. DGS 4 F 2 N S C, CR, 41.4 5 H 3 N S D, CR, 41.5 6 H 3 B S A, CR, 41.8, NRC2007 D, CR, 41.4, NRC2000 - This is a KA mismatch because the question is not focused 7 F 2 N S on CCW. It is focused on TBCW. Replace question. New Question is SAT. DGS 8 F 3 N S C, CR, 41.4 D, CR, 41.7 D is obvious answer due to noun name. Replace with another correct 9 F 3 B S answer less obvious. Change made. DGS Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 8 OBDI 202 - IOLE Process

ES-401 [[::JAF-2012-09|JAF-2012-09]] 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 10 F 3 N S B, CR, 41.7 A, CR, 41.8 In stem, replace LOCA with leak. Lower drywell pressure in stem.

11 H 3 N S Changes made. DGS A, CR, 41.3 Pressures too low in C (1) and D (1). Reuse pressures from A and B 12 F 2 N S with different explanation. D is also correct. Corrected, now SAT. DGS 13 F 3 B S B, CR, 41.9 14 F 3 N S D, CR, 41.8 15 H 4 B S B, CR, 41.7 B, CR, 41.5 Change A to lower due to increase in volume of steam space. Expand 16 H 4 N S new response. Corrected, now SAT. DGS B, CR, 41.6 - This is a KA mismatch because the CRD system is not safety related 17 H 3 N S and the KA is focused on safety related equipment. May need to select a new KA. To be replaced with SLC question. SLC question SAT. DGS 18 F 3 N S B, CR, 41.13 D, CR, 41.7 - This basically a True/False question. Stem information is required to 19 F 3 B S choose correct response. Sat as written.

A, CR, 41.4 Change C (2) and D (2) RPC level response to low level. Change made.

20 F 2 B S DGS A, CR, 41.5 - Recommend adding throttle to each of the distractors since none of 21 H 2 N S the valves will fully open or close. C and D responses changed from CIVs to turbine bypass valves. Changes made. DGS 22 H 3 B S D, CR, 41.4 C, OR, 41.10 Change Suppression pool temp in stem to 108 degrees. Change B to 23 H 2 N S Suppression pool reject. Correct explanation for B answer. Corrected, now SAT.

DGS 24 F 2 N S A, CR, 41.6 25 F 3 N S C, CR, 41.6 Replace D response. Change made. DGS 26 F 3 N S C, CR, 41.7 27 F 3 N S B, CR, 41.9 28 H 3 N S D, CR, 41.10 Page 2 of 8 OBDI 202 - IOLE Process

ES-401 [[::JAF-2012-09|JAF-2012-09]] 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 29 H 3 N S B, CR, 41.7 30 H 3 N S B, CR, 41.7 31 F 3 N S A, CR, 41.10 32 F 3 N S A, CR, 41.3 33 F 3 N S A, CR, 41.6 Make all answers Train A. Changes made. DGS C, CR, 41.6 - The distractors are True/False. Modify or replace question. Modified, 34 H 3 N S SAT. DGS 35 H 3 N S B, CR, 41.2 Question to be modified. Changes made. DGS 36 H 3 B S D, CR, 41.6, NRC1997 37 H 3 N S A, CR, 41.7 38 H 3 N S C, CR, 41.7 39 H 4 N S D, CR, 41.7 A, CR, 41.5 - The way the stem is written, it tells the applicant more SRVs must be opened to reduce pressure, i.e. pressure must be rising. Therefore there are only 2 40 H 3 N S potentially correct answers. Power level changed in stem to account for RCIC running. Review when modified. Change power level to 26% in explanation A.

Corrected, now SAT. DGS B, CR, 41.4 - The stem asks what actions should be taken and there is only one 41 F 3 N S distractor with more than one action; the correct one. Recommend deleting actions from the stem. Change A and D to be more realistic. Changes made. DGS.

D, CR, 41.4 - Suggest replacing two of the distractors with other parameters such as high containment pressure. If the applicant knows low RPV level doesnt auto start 42 F 2 N S SBGTS, then there is only one other choice. Level 2 does start SGT. Replace C response. Changes made. DGS 43 F 3 N S D, CR, 41.4 A, CR, 41.4 - Suggest defining failure in the stem as this term could be interpreted in a variety of ways (upscale, downscale, as is, etc). Suggest replacing failure with 44 H 3 N S loss. I did not give Angie this comment. Change to loss. Corrected, now SAT.

DGS 45 F 3 B S C, CR, 41.4 Page 3 of 8 OBDI 202 - IOLE Process

ES-401 [[::JAF-2012-09|JAF-2012-09]] 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 46 F 3 N S A, CR, 41.4 Change B (1) to residual magnetism. Change made. DGS 47 H 3 B S A, CR, 41.4 48 F 3 N S A, CR, 41.4 B, CR, 41.7 Add assumption in stem that no water is being lost. Changes made.

49 H 3 N S DGS 50 F 2 N S B, CR, 41.6 51 F 3 N S D, CR, 41.7 52 H 3 B S A, CR, 41.4 Response C to be rewritten. Changes made. DGS D, CR, 41.4 - The question needs to be written focused on CCW and not TBCW.

53 H 3 N S Rewrite. Question rewritten. DGS 54 F 2 N S D, CR, 41.4 55 H 3 N S A, CR, 41.6 56 F 3 N S C, CR, 41.6 C, CR, 41.7 Tell Angie to delete from the feedwater system in stem. Corrected, now 57 F 3 N S SAT. DGS A, CR, 41.7 Change second part of C and D response to E12-F068A full open.

58 H 3 N S Changes made. DGS 59 H 3 N S B, CR, 41.7 60 H 3 N S B, CR, 41.4 61 H 3 N S B, CR, 41.11 62 F 2 N S B, CR, 41.11 63 F 3 N S A, CR, 41.4 64 F 3 N S B, CR, 41.10 65 F 3 N S C, CR, 41.9 66 F 2 N S D, CR, 41.10 Change C (2) to could. Changed. DGS 67 H 2 N S C, OR, 41.10 68 F 2 N S A, CR, 41.7 Page 4 of 8 OBDI 202 - IOLE Process

ES-401 [[::JAF-2012-09|JAF-2012-09]] 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 69 F 4 N S B, CR, 41.10 C, CR, 41.10 - Distractor D is also correct. The question is not a generic Tier 3 70 F 2 N S question. Write question on channel check. New question SAT. DGS C, CR, 41.7 - The question is not a generic Tier 3 question. Write new question. Still 71 H 3 N S problems. B answer modified, now SAT. DGS 72 F 2 B S B, CR, 41.12 73 F 2 N S D, CR, 41.12 A, CR, 41.10 - This question involves the assessment of plant conditions and selection of a procedure making it an SRO question. However, since it only requires 74 H 3 M S knowledge of the EOP entry conditions it is ok for the RO exam. Change D response to remove EOP-1A. Change made. DGS 75 F 2 N S D, CR, 41.10 A, CR, 43.5 Too many bullets refer to jet pump problems. Eliminate one of them.

76 H 3 N S Change made. DGS 77 H 3 N S C, CR, 43.5 Change C and D responses. Changes made. DGS D, OR, 43.2 - The candidates should not need TS 3.4.11 as they should know the 78 H 3 N S cooldown limit from memory. AOP-00031 is not needed; provide the steam tables.

Remove TS handout. Removed. DGS 79 H 2 B S B, CR, 43.5 Change D response. Changed. DGS 80 H 3 B S B, OR, 43.5, NRC2010 A, CR, 41.10 - This is an RO question since it does not involve assessment of plant conditions or a procedure selection. It is testing on entry conditions and operator 81 F 3 N S actions. Secondly, the KA is 295024 (High Drywell Pressure) and the question does not involve high dry well pressure so it is a KA mismatch. Replace question.

Replaced. DGS D, CR, 43.5 Modify A and C responses to be less generic. Rev 1 was unacceptable.

82 F 3 B S Revert back to original question. DGS 83 F 2 N S A, CR, 43.2 84 F 2 N S B, CR, 43.5 C, CR, 41.10 - This is an RO question since it tests on EOP basis. Question sat as 85 F 3 N S written. Page 5 of 8 OBDI 202 - IOLE Process

ES-401 [[::JAF-2012-09|JAF-2012-09]] 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 86 H 3 N S D, CR, 43.5 B, CR, 43.5 No reference. Change time to make flow rate more realistic.Removed 87 H 3 M S reference and shortened time. DGS C, CR, 41.7 - This can be answered using RO system knowledge. Distractors A and B can be eliminated by knowing there will be no half scram and D can be eliminated 88 H 3 N S by knowing only one SRM can be bypassed. Rewrite question. K/A mismatch.

Question modified, now SAT. DGS 89 H 2 N S B, CR, 43.5 90 H 2 N S B, CR, 43.5 D, OR, 43.2 - Recommend not providing candidates with TS 3.8.1 since there is no need for this to answer the question. I.e., it is given in the stem the EDG is 91 H 3 N S inoperable so there is no need to assess it against the TS. Remove TS handout.

Changed. DGS A, CR, 41.7 - This can be answered using RO system knowledge. An RO should 92 H 3 N S know F028A is a Div I isolation valve and that the given leakage rate will cause an isolation. Select procedure makes it an SRO question.

93 F 2 N S A, CR, 43.2 B, OR, 43.2 - Recommend TS 3.8.1 not be handed out as it is not needed. Agree.

94 H 2 N S Removed from handout. DGS 95 F 2 N S C, CR, 43.7 96 H 3 N S A, CR, 43.2 D, OR, 43.2 - This is direct lookup question since the TS basis states cooler C is not 97 H 2 N S safety related and the TS LCO clearly states with one cooler inop the completion time is 7 days. No TS basis handout. Removed from handout. DGS 98 F 3 B S C, CR, 43.4 D, CR, 41.10 - This is RO knowledge since it does not test on assessment / selection 99 F 3 N S of a procedure. Second, it is not a generic question. Third, distractor A could also be argued as being correct. Reselect. Changed. DGS 100 F 3 B S B, CR, 43.1 B= 14 (19%) F= 42 (56%) E= 6 RO TOTALS: Additional Notes:

M= 1 (1%) H= 33 (44%) U= 6 Page 6 of 8 OBDI 202 - IOLE Process

ES-401 [[::JAF-2012-09|JAF-2012-09]] 7 Form ES-401-9 N= 60 (80%)

B= 5(20%) F= 10 (40%) E= 1 SRO TOTALS: M= 1(4%) H= 15 (60%) U= 6 Additional Notes:

N= 19 (76%)

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner follow-up comments are indicated in blue.

3. Average difficulty is _2.89 on the RO exam and 2.56 on the SRO exam. These values for this reviewer indicate the RO exam is more difficult than average and the SRO exam is easier than average.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 1 41.2 = 1 43.2 = 7 41.3 = 2 43.3 = 0 41.4 = 18 43.4 = 1 41.5 = 5 43.5 = 10 41.6 = 9 43.6 = 0 41.7 = 19 43.7 = 3 41.8 = 3 41.9 = 3 41.10 = 10 41.11 = 2 41.12 = 2 41.13 = 1 41.14 = 0

5. The answer distribution is: Changed distribution to 5, 8, 5 and 6 respectively, DGS A= 22 / 5 B= 20 / 11 C= 15 / 4 D= 18 / 5

6. There are 2/5 questions with attachments provided.

7. The SRO distribution of correct answers for distractor B is high. Recommend the author revises the correct distractor distribution. Angie to balance distribution more evenly. Balanced more evenly. DGS Page 7 of 8 OBDI 202 - IOLE Process

ES-401 [[::JAF-2012-09|JAF-2012-09]] 8 Form ES-401-9

8. This exam was reviewed by S. Garchow on September 18-19, 2012. Page 8 of 8 OBDI 202 - IOLE Process