ML13029A068

River Bend-2012-11-Draft Written Exam Comments

ML13029A068
Person / Time
Site:	River Bend
Issue date:	11/15/2012
From:	David Strickland Operations Branch IV
To:	Entergy Operations
Laura Hurley
References
ES-401, ES-401-9 50-458/12-011
Download: ML13029A068 (8)

Text

ES-401 RB-2012-11 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content). $ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory). $ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). $ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO

-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES

-401 Section D.2.f

. 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Attachment 8 Page 1 of 8 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws

5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 1 H 2 N S C, OR, 41.5 Change A (1) to 61% to be more realistic.

Change made.DGS 2 H 2 N S A, CR, 41.7 3 H 3 N S B, CR, 41.7 Reword responses C & D to reduce confusion.

Change made. DGS 4 F 2 N S C, CR, 41.4 5 H 3 N S D, CR, 41.5 6 H 3 B S A, CR, 41.8, NRC2007 7 F 2 N S D, CR, 41.4, NRC2000

- This is a KA mismatch because the question is not focused on CCW. It is focused on TBCW.

Replace question.

New Question is SAT. DGS 8 F 3 N S C, CR, 41.4 9 F 3 B S D, CR, 41.7 D is obvious answer due to noun name. Replace with another correct answer less obvious.

Change made. DGS

ES-401 [[::JAF-2012-09|JAF-2012-09]] 2 Form ES-401-9 Attachment 8 Page 2 of 8 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws

5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 10 F 3 N S B, CR, 41.7 11 H 3 N S A, CR, 41.8 In stem, replace 'LOCA' with 'leak'. Lower drywell pressure in stem.

Changes made. DGS 12 F 2 N S A, CR, 41.3 Pressures too low in C (1) and D (1). Reuse pressures from A and B with different explanation.

D is also correct

. Corrected, now SAT. DGS 13 F 3 B S B, CR, 41.9 14 F 3 N S D, CR, 41.8 15 H 4 B S B, CR, 41.7 16 H 4 N S B, CR, 41.5 Change A to lower due to increase in volume of steam space.

Expand new response.

Corrected, now SAT. DGS 17 H 3 N S B, CR, 41.6

- This is a KA mismatch because the CRD system is not safety related and the KA is focused on safety related equipment. May need to select a new KA.

To be replaced with SLC question.

SLC question SAT. DGS 18 F 3 N S B, CR, 41.13 19 F 3 B S D, CR, 41.7

- This basically a True/False question.

Stem information is required to choose correct response. Sat as written.

20 F 2 B S A, CR, 41.4 Change C (2) and D (2) RPC level response to low level.

Change made. DGS 21 H 2 N S A, CR, 41.5

- Recommend adding "throttle" to each of the distractors since none of the valves will fully open or close.

C and D responses changed from 'CIV's ' to 'turbine bypass valves.

Changes made. DGS 22 H 3 B S D, CR , 41.4 23 H 2 N S C, OR, 41.10 Change Suppression pool temp in stem to 108 degrees. Change B to Suppression pool reject.

Correct explanation for B answer.

Corrected, now SAT. DGS 24 F 2 N S A, CR, 41.6 25 F 3 N S C, CR, 41.6 Replace D response.

Change made. DGS 26 F 3 N S C, CR, 41.7 27 F 3 N S B, CR, 41.9 28 H 3 N S D, CR, 41.10

ES-401 [[::JAF-2012-09|JAF-2012-09]] 3 Form ES-401-9 Attachment 8 Page 3 of 8 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws

5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 29 H 3 N S B, CR, 41.7 30 H 3 N S B, CR, 41.7 31 F 3 N S A, CR, 41.10 32 F 3 N S A, CR, 41.3 33 F 3 N S A, CR, 41.6 Make all answers Train A.

Changes made.

DGS 34 H 3 N S C, CR, 41.6

- The distractors are True/False.

Modify or replace question.

Modified, SAT. DGS 35 H 3 N S B, CR, 41.2 Question to be modified.

Changes made. DGS 36 H 3 B S D, CR, 41.6, NRC1997 37 H 3 N S A, CR, 41.7 38 H 3 N S C, CR, 41.7 39 H 4 N S D, CR, 41.7 40 H 3 N S A, CR, 41.5

- The way the stem is written, it tells the applicant more SRVs must be opened to reduce pressure, i.e. pressure must be rising. Therefore there are only 2 potentially correct answers.

Power level changed in stem to account for RCIC running. Review when modified.

Change power level to 26% in explanation A.

Corrected, now SAT. DGS 41 F 3 N S B, CR, 41.4

- The stem asks what "actions" should be taken and there is only one distractor with more than one action; the correct one. Recommend deleting "actions" from the stem.

Change A and D to be more realistic. Changes made. DGS

. 42 F 2 N S D, CR, 41.4

- Suggest replacing two of the distractors with other parameters such as high containment pressure. If the applicant knows low RPV level doesn't auto start SBGTS, then there is only one other choice.

Level 2 does start SGT. Replace C response. Changes made. DGS 43 F 3 N S D, CR, 41.4 44 H 3 N S A, CR, 41.4

- Suggest defining "failure" in the stem as this term could be interpreted in a variety of ways (upscale, downscale, as is, etc).

Suggest replacing "failure" with "loss". I did not give Angie this comment. Change to 'loss'

. Corrected, now SAT. DGS 45 F 3 B S C, CR, 41.4

ES-401 [[::JAF-2012-09|JAF-2012-09]] 4 Form ES-401-9 Attachment 8 Page 4 of 8 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws

5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 46 F 3 N S A, CR, 41.4 Change B (1) to 'residual magnetism'.

Change made. DGS 47 H 3 B S A, CR, 41.4 48 F 3 N S A, CR, 41.4 49 H 3 N S B, CR, 41.7 Add assumption in stem that no water is being lost.

Changes made. DGS 50 F 2 N S B, CR, 41.6 51 F 3 N S D, CR, 41.7 52 H 3 B S A, CR, 41.4 Response C to be rewritten.

Changes made. DGS 53 H 3 N S D, CR, 41.4 - The question needs to be written focused on CCW and not TBCW.

Rewrite. Question rewritten. DGS 54 F 2 N S D, CR, 41.4 55 H 3 N S A, CR, 41.6 56 F 3 N S C, CR, 41.6 57 F 3 N S C, CR, 41.7 Tell Angie to delete 'from the feedwater system' in stem

. Corrected, now SAT. DGS 58 H 3 N S A, CR, 41.7 Change second part of C and D response to 'E12

-F068A full open'.

Changes made. DGS 59 H 3 N S B, CR, 41.7 60 H 3 N S B, CR, 41.4 61 H 3 N S B, CR, 41.11 62 F 2 N S B, CR, 41.11 63 F 3 N S A, CR, 41.4 64 F 3 N S B, CR, 41.10 65 F 3 N S C, CR, 41.9 66 F 2 N S D, CR, 41.10 Change C (2) to 'could'.

Changed. DGS 67 H 2 N S C, OR, 41.10 68 F 2 N S A, CR, 41.7

ES-401 [[::JAF-2012-09|JAF-2012-09]] 5 Form ES-401-9 Attachment 8 Page 5 of 8 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws

5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 69 F 4 N S B, CR, 41.10 70 F 2 N S C, CR, 41.10

- Distractor D is also correct. The question is not a generic Tier 3 question. Write question on 'channel check'.

New question SAT. DGS 71 H 3 N S C, CR, 41.7

- The question is not a generic Tier 3 question.

Write new question

. Still problems. B answer modified, now SAT. DGS 72 F 2 B S B, CR, 41.1 2 73 F 2 N S D, CR, 41.12 74 H 3 M S A, CR, 41.10

- This question involves the assessment of plant conditions and selection of a procedure making it an SRO question. However, since it only requires knowledge of the EOP entry conditions it is ok for the RO exam.

Change D response to remove EOP

-1A. Change made. DGS 75 F 2 N S D, CR, 41.10 76 H 3 N S A, CR, 43.5 Too many bullets refer to jet pump problems. Eliminate one of them.

Change made. DGS 77 H 3 N S C, CR, 43.5 Change C and D responses.

Changes made. DGS 78 H 3 N S D, OR, 43.2

- The candidates should not need TS 3.4.11 as they should know the cooldown limit from memory. AOP

-00031 is not needed; provide the steam tables.

Remove TS handout. Removed. DGS 79 H 2 B S B, CR, 43.5 Change D response.

Changed. DGS 80 H 3 B S B, OR, 43.5, NRC2010 81 F 3 N S A, CR, 41.10

- This is an RO question since it does not involve assessment of plant conditions or a procedure selection.

It is testing on entry conditions and operator actions. Secondly, the KA is 295024 (High Drywell Pressure) and the question does not involve high dry well pressure so it is a KA mismatch.

Replace question.

Replaced. DGS 82 F 3 B S D, CR, 43.5 Modify A and C responses to be less generic.

Rev 1 was unacceptable. Revert back to original question. DGS 83 F 2 N S A, CR, 43.2 84 F 2 N S B, CR, 43.5 85 F 3 N S C, CR, 41.10

- This is an RO question since it tests on EOP basis.

Question sat as w ritten.

ES-401 [[::JAF-2012-09|JAF-2012-09]] 6 Form ES-401-9 Attachment 8 Page 6 of 8 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws

4. Job Content Flaws

5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 86 H 3 N S D, CR, 43.5 87 H 3 M S B, CR, 43.5 No reference. Change time to make flow rate more realistic.Removed reference and shortened time. DGS 88 H 3 N S C, CR, 41.7

- This can be answered using RO system knowledge. Distractors A and B can be eliminated by knowing there will be no half scram and D can be eliminated by knowing only one SRM can be bypassed.

Rewrite question.

K/A mismatch. Question modified, now SAT. DGS 89 H 2 N S B, CR, 43.5 90 H 2 N S B, CR, 43.5 91 H 3 N S D, OR, 43.2

- Recommend not providing candidates with TS 3.8.1 since there is no need for this to answer the question. I.e., it is given in the stem the EDG is inoperable so there is no need to assess it against the TS.

Remove TS handout.

Changed. DGS 92 H 3 N S A, CR, 41.7

- This can be answered using RO system knowledge. An RO should know F028A is a Div I isolation valve and that the given leakage rate will cause an isolation.

Select procedure makes it an SRO question.

93 F 2 N S A, CR, 43.2 94 H 2 N S B, OR, 43.2

- Recommend TS 3.8.1 not be handed out as it is not needed.

Agree. Removed from handout. DGS 95 F 2 N S C, CR, 43.

7 96 H 3 N S A, CR, 43.2 97 H 2 N S D, OR, 43.2

- This is direct lookup question since the TS basis states cooler C is not safety related and the TS LCO clearly states with one cooler inop the completion time is 7 days.

No TS basis handout.

Removed from handout. DGS 98 F 3 B S C, CR, 43.4 99 F 3 N S D, CR, 41.10

- This is RO knowledge since it does not test on assessment / selection of a procedure. Second, it is not a generic question. Third, distractor A could also be argued as being correct.

Reselect. Changed. DGS 100 F 3 B S B, CR, 43.1 RO TOTALS:

B= 14 (19%) F= 42 (56%) E= 6 Additional Notes:

M= 1 (1%) H= 33 (44%) U= 6 ES-401 [[::JAF-2012-09|JAF-2012-09]] 7 Form ES-401-9 Attachment 8 Page 7 of 8 OBDI 202 - IOLE Process N= 60 (80%) SRO TOTALS:

B= 5 (20%) F= 10 (40%) E= 1 Additional Notes:

M= 1(4%) H= 15 (60%) U= 6 N= 19 (76%)

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N 2. Chief Examiner follow-up comments are indicated in blue. 3. Average difficulty is

_2.89 on the RO exam and 2.56 on the SRO exam.

These values for this reviewer indicate the RO exam is more difficult than average and the SRO exam is easier than average.

4. The 10CFR55.41/43 distribution is:

RO / SRO 41.1 = 0 43.1 = 1 41.2 = 1 43.2 = 7 41.3 = 2 43.3 = 0 41.4 = 18 43.4 = 1 41.5 = 5 43.5 = 10 41.6 = 9 43.6 = 0 41.7 = 19 43.7 = 3 41.8 = 3 41.9 = 3 41.10 = 10 41.11 = 2 41.12 = 2 41.13 = 1 41.14 = 0 5. The answer distribution is:

Changed distribution to 5, 8, 5 and 6 respectively, DGS A = 22 / 5 B = 20 / 11 C = 15 / 4 D = 18 / 5 6. There are 2/5 questions with attachments provided.

7. The SRO distribution of correct answers for distractor B is high. Recommend the author revise s the correct distractor distribution.

Angie to balance distribution more evenly. Balanced more evenly. DGS

ES-401 [[::JAF-2012-09|JAF-2012-09]] 8 Form ES-401-9 Attachment 8 Page 8 of 8 OBDI 202 - IOLE Process

8. This exam was reviewed by S. Garchow on September 18

-19, 2012.