ML101270243

2010-03 ES-401-9 Written Comment Resolution

ML101270243
Person / Time
Site:	Arkansas Nuclear
Issue date:	03/16/2010
From:	NRC Region 4
To:	Entergy Operations
References
50-313/10-03, 50-368/10-03
Download: ML101270243 (12)

Text

ES-401 AN1-2010-03 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 12 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 1

H 3

M S

Question missing from draft exam submittal. Marked as Unsat until I receive the question and pedigree. Submitted via e-mail with password on Thursday, 1/28/2010.

Sat once received.

2 F

2 X

N E

S If applicant knows kw rating of required heaters, then he/she will have the answer.

Marked as Higher level LOK, and LOD of 3, but I characterize it as fundamental since it is memory and LOD of 2. Change kw to be toggled or remove kw entirely to fix distracters. Could use 124 kw and 128 kw.

Licensee made changes as recommended 3

H 4

X X

N E

Would 45 gpm be considered a SBLOCA? Also, the distracters need symmetry wrt rate and time so that elimination is not based on knowing one piece of information and also the distracters should be based on calculation errors or common misconceptions about the sump, which are not explained in notes section.

ES-401 AN1-2010-03 2

Form ES-401-9 Page 2 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

Recommend using 91 gpm and 45 gpm for leak rates and 3 minutes and 6 minutes with explanations of assumptions of 45.4 gpi and 50% level and then half flow rate and double time.

Licensee changed to 3 and 8 minutes, and changed leak rates as recommended.

4 F

2 X

B E

S Recommend changing distractor C to Prevent overloading EDGs during transfer, and changing distractor D to read reduce radiation levels near RB spray piping Licensee made recommended changes.

5 F

3 B

S 6

H 2

X N

E S

D is not credible for given plant conditions, use P-66 Borated water recirc pump.

Licensee made recommended changes.

7 H

3 X

B E

S Distractors need words such as surveillance in D instead of checked to make more credible distracters.

Licensee made recommended changes.

8 H

4 N

S 9

H 4

B S

10 H

3 B

S 11 H

3 M

S 12 H

3 B

S 13 F

2 B

S 14 H

3 X

B E

S E

Distractor C is not credible due to balance and focus of stem not being on SG instruments but on ICS. Use distractor symmetry to get proper balance of information, such as toggling YO-1 and YO-2 and failing instruments low or no failure due to transfer.

Licensee made recommended changes.

During second review, changes made to distracters C and D for better symmetry.

ES-401 AN1-2010-03 3

Form ES-401-9 Page 3 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 15 H

3 B

S 16 H

3 X

B U

E E

S K/A mismatch - This K/A is knowledge of the reasons for auto alignment during ESFAS and this is not the focus of the question.

Modify stem and distracters in progress-review again once changes made.

Second review, still needs and and between energizedfor what reason?

Resubmitted question, now Sat.

17 H

2 B

E S

Stem could be improved to display K/A match and stem focus.

Licensee made rec. changes 18 H

3 X

N E

U U

S Both graph and question do not address leading/lagging.Why?

Redesign of question.

Second review done, now it is a direct lookup. Recommend using lagging instead of over-excited so that applicant will have to interpret the graph vs look-up.

Licensee made rec. changes 19 H

2 X

N E

S Need slight mods to distracters B and C. Reference doesnt support entire answer, ie CV-1235 valve operation.

Licensee made rec. changes 20 H

3 X

X N

S Reference is hard to read for overlap graphs.

21 F

2 X

N E

S Correct answer B to 15.4gpm, change distracters to A = 10.2, distracter C =

20.4, and distracter D = 30.8 Licensee made rec. changes 22 H

4 X

B U

S K/A mismatch-knowledge of the reasons is not addressed in the question.

Licensee made rec. changes by adding reasons portion to question.

23 F

2 B

S

ES-401 AN1-2010-03 4

Form ES-401-9 Page 4 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 24 F

2 X

X X

B E

S Stem appears to be incomplete. Distractor D not allowed since it is fictitious.

Reword distractor and modify stem.

Made rec. changes 25 H

3 M

S 26 H 3

X X

X N

U S

Stem is incorrect, wrong procedure referenced 1203.026 vs 1203.025. This is a selection of procedures question and is therefore SRO only. Also, when using procedures by number, you must include them by name, too ( such as 1203.045 and 1202.001,and 1203.040, 1203.013). I also disagree with the LOK classification. This is not fundamental but higher cognitive.

Rewrote question and resubmitted. Sat now.

27 H

3 N

E S

Need to clarify name for Code Safeties for Pressurizer (as opposed to SG Code Safeties) or use their name as mentioned in the bases, RCS Pressurizer Safety Valves.

Licensee made rec. changes 28 H

3 X

B U

S KA mismatch-because relationship of RCS flow to # RCPs is not the focus of the stem or answer.

Selected Bank question, now Sat.

29 H

3 M

S 30 H

3 N

U S

Originally thought that this was KA mismatch - I dont think that the TRM is part of the facility license and so this could be viewed as KA mismatch.

K/A is loosely defined to allow facility conditions, which unlike the TS call on a scenario, seems okay for TRM question.

31 F

2 X

B U

S Reference provided does not support answer.

Updated references to file. Now Sat.

32 F

3 X

M U

Reference provided does not support answer.

ES-401 AN1-2010-03 5

Form ES-401-9 Page 5 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

Licensee made rec. changes 33 F

2 X

N U

S KA mismatch-this question does not test on the loss or malfunction of a valve on the ECCS system. No Reference for correct answer.

Wrote New question.

Second review Sat.

34 H

3 B

S 35 H

3 X

N E

S Remove the word all from answer C because it could cue the applicant to it.

Made rec. changes 36 F

2 X

N E

S KA Mismatch - this question does not test on PZR PCS and impact on RCS but loss of MFW and stuck shut PZR spray valve and impact on the plant.

Utility explained, no changes needed and not KA mismatch.

37 F

3 X

N U

S DIstractors B and D are not credible. Permissive for Shutdown bypass being met would be credible. 2 out of 2 logic is not credible because it is not allowed.

Made rec. changes 38 F

3 X

X N

U E

E S

KA Mismatch - TS 1 hr entry conditions are not system limits and precautions. Also, the stem is negatively focused and this fact coupled with the three distracters all being correct choices with an OR statement in the TS and therefore are a collection of true/false statements, which is not allowed.

Wrote new question to match K/A. need to review.

Second review still needs edit of all distracters to substitute the words due to with because for balance and symmetry with answer. Question is now Sat.

39 F

2 B

S 40 F

2 B

S 41 F

2 B

S 42 H

3 X

B E

Distractor A is not credible as you would not initiate safety features actuation

ES-401 AN1-2010-03 6

Form ES-401-9 Page 6 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

systems and then continue to startup.

Licensee made rec. changes 43 F

2 B

S 44 F

2 X

X N

E S

Stem is incorrect, these choices are all parameters not interlocks. Question, the vibration trip reference states that this is normally bypassed during operation.

Licensee corrected stem 45 H

2 B

S 46 H

3 N

E S

Recommend changing correct answer to UV interlocks vice close permissives.

Licensee made rec. changes 47 F

2 M

S 48 F

3 B

S 49 F

2 N

S 50 H

3 N

S 51 F

2 B

S 52 H

3 B

E S

Distractor B not credible, Change DIstractor B to read Warm weather conditions cause low demand from ACW/SW.

Licensee made rec. changes 53 F

2 N

S 54 F

3 X

X B

E S

This question has cueing. All three distracters are manual actions and correct answer is an automatic action. Need to structure the choices so that they are balanced with no cueing., ie make one distractor a reasonable automatic action of similar length.

Licensee made rec. changes, also add 5 minutes into the trip to the stem.

ES-401 AN1-2010-03 7

Form ES-401-9 Page 7 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 55 H

3 B

S 56 H

2 B

S 57 H

2 X

X X

B E

S This question has minor cueing in it: The answer is much shorter than the three distracters and also the answer has a direct reference to a procedure and the other three do not, ie they look less formal and more casual as actions. Made some changes Licensee made some changes, question is now Sat.

58 H

3 N

E S

Remove flow from answers, use 50/150 psig with open/close for CV-1237.

Licensee made rec. changes 59 H

2 X

B E

S Distractor D explanation may be incorrect. Need to review this with utility to ensure they did not intend to mean.D is incorrect because the group 5 out limit lamp may be on (or off as appropriate) from other rods.

Licensee made rec. changes 60 H

3 B

S 61 H

3 X

B U

S KA Mismatch - this question does not address the operational implications of separate control and and protection circuits for NNIS. AN example of this is typically optical isolators or use of separate instruments for control of a parameter vs protection of the reactor or components, which would have its own instrument as well.

Licensee picked new K/A and then selected bank question.

Second review Sat.

62 H

2 B

S 63 F

3 B

S Originally thought KA Mismatch - the governor valves are not the same as the turbine stop valves are they? Answer is Yes.

At ANO the Governor and Throttle valves for the turbine are the stop valves.

64 H

2 X

N E

Modify stem to state stable water temp.

ES-401 AN1-2010-03 8

Form ES-401-9 Page 8 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

Made rec. changes 65 F

2 B

S 66 F

2 X

B E

S This question will need slight modification on basis of use of generic K/A for specific system questions.

Modified to be generic release question.

67 H

3 X

N S

This question will need slight modification on basis of use of generic K/A for specific system questions.

During discussion with utility it was determined that because multiple systems are involved for this K/A that it met the intent of generic T3 question so it remains as is.

68 H

4 X

N E

S This question will need slight modification on basis of use of generic K/A for specific system questions.

Because B & W is an ICS plant, which controls many systems in the primary and secondary and is not mutually exclusive in the K/A catalog, the licensee contended that it is also generic enough for the many systems it covers to count as a generic system precaution, therefore keeping this question as written.

69 F

2 B

S 70 H

2 N

S 71 H

3 N

S 72 F

2 X

N U

U S

Replace since JPM on OP test is exact copy of this question, Second Question written is also unsat as it contains cueing from stem has dose in it and the correct answer is the ODCM, so licensee wanted to select another K/A and I agreed.

Third Review Sat.

73 F

2 N

S 74 F

2 X

N E

S I am not sure if this question will need slight modification on basis of use of generic K/A for specific system questions.

New question after review EOP/AOP user guide.

ES-401 AN1-2010-03 9

Form ES-401-9 Page 9 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only Second review Sat.

75 F

2 X

B E

I am not sure if this question will need slight modification on basis of use of generic K/A for specific system questions.

Rejected K/A since could not write generic T3 question, redrew K/A. Selected a Bank Question. Remember to check reject form for this.

76 H

3 B

S 77 H

3 X

X X

N U

S If you know the TS entry condition (RO knowledge) you can answer the question.

Recommend changing stem to 65 inches from 85 inches, use second part of C and D in selection A and B and change the numeric values to A thru D to be 2 minutes, 2 minutes, 4 minutes, and 4 minutes, respectively, which still makes D the correct choice.

Made rec. changes 78 H

3 N

S 79 H

3 N

S 80 H

3 N

S 81 H

4 B

S 82 H

4 B

S 83 H

4 M

S 84 H

4 B

S 85 H

4 B

S 86 H

3 X

X N

E S

Reference doesnt support answer. Recommend modifying stem to include bleed off temperature 210F and stable where and stable is added to stem to prevent cueing to answer and also for balance.

Removed stable from RCP bearing temp in stem.

87 H

4 B

S 88 H

3 N

E Minor format issue with stem. Justification of answer should clarify that loss of condensate pumps causes loss of feedwater.

ES-401 AN1-2010-03 10 Form ES-401-9 Page 10 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

Made rec. changes 89 H

3 N

S 90 H

3 N

S 91 H

4 B

S 92 F

2 X

B E

S Classification is incorrect also, as this is memory or fundamental information retrieval question, and most common NI question for source range NIs.

Made rec. changes 93 H

4 N

S 94 H

4 B

S 95 F

3 N

S 96 H

3 X

X B

U S

This is a direct lookup question with TS 3.8.3 provided in handout.

This handout removed from question.

97 F

3 X

N U

Distractors C and D are not credible. Safety related equipment cannot logically be repaired with no work order and with tool pouch or FIN team work.

Modify stem and distracters in progress.

98 H

3 X

N E

S Modify stem to state A fuel assembly was stored for 6 months, then was to be moved to a different location in the spent fuel pool due to a mistake in its storage location. During movement the assembly becomes damaged Made changes to demonstrate greater than 190 days age of fuel, but simpler than way I rec above.

99 H

2 X

X B

E S

D is not credible. Where is the reference for fires? Are there any different considerations if fire is not out before certain time? 36 min is long time for fire.

Added reference for fire. Modified stem to support credibility.

ES-401 AN1-2010-03 11 Form ES-401-9 Page 11 of 12 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 100 H

3 X

N E

S Are B and C credible? Polishers and standby boiler dont seem to be a very high priority when a fire is ongoing.

SRO does direct these items sometimes, so licensee thinks they are credible.

RO TOTALS:

B= 39 F= 32 E= 28 Additional Notes: total unsat is 15 / 100,therefore exam was not unsat submittal. It took a first review with utility to agree on some characterizations as not unsat.

M= 6 H= 43 U= 12 N= 30 SRO TOTALS:

B= 11 F= 3 E= 7 Additional Notes:

M= 1 H= 22 U= 3 N= 13

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2.

All comments for this form are Chief Examiner comments.

3.

Average difficulty is _2.62 ___ on the RO exam and 3.28_ on the SRO exam.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 =

0 43.1 = 1 41.2 =

3 43.2 = 4 41.3 =

3 43.3 = 1 41.4 =

4 43.4 = 1 41.5 =

17 43.5 = 17 41.6 =

4 43.6 = 0 41.7 =

37 43.7 = 1 41.8 =

9 41.9 = 3 41.10 = 17 41.11 = 2 41.12 = 1 41.13 = 1 41.14 = 0

5.

The answer distribution is: RO / SRO A = 19 (25%)

/

5 (20%)

ES-401 AN1-2010-03 12 Form ES-401-9 Page 12 of 12 B = 21 (28%)

/

10 (40%)

C = 20 (27%)

/

5 (20%)

D = 15 (20%)

/

5 (20%)

6.

There are 5 questions with attachments provided: RO Exam Q18, Q30, Q70, and on SRO exam Q83 and Q91.