ML15148A369

2015-04 Draft Written Exam Comments

ML15148A369
Person / Time
Site:	Palo Verde
Issue date:	04/13/2015
From:	Vincent Gaddy Operations Branch IV
To:	Arizona Public Service Co
References
Download: ML15148A369 (9)

Text

ES-401 PV-2015-04 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 1

H 3

B N

E S

(41.5) Lets change the 16 inches in the stem to 14 inches which is just before the MFPs will trip. Technically at 16 inches the MFPs would still be available and one could argue how long the degrading will take to reach trip set point.

Fixed 2

H 3

C N

S (41.7) 3 H

3 D

N E

S (41.10)

Give indication of RCS subcooling in the stem of greater than 24F to ensure they have the information available to not stop all RCPs.

Fixed 4

H 4

C N

E (41.10) Would be psychometrically better if one more of the Can NOT distracters could be replaced with Can so that there is an equal number of each. Maybe add a Can be verified along with a statement that is incorrect thus making the distracter

ES-401 PV-2015-04 2

Form ES-401-9 Page 2 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

incorrect.

Loop dT is 25F not 30F as stated in explanation.

Specifically state the value for subcooling in explanation The Can statement has been missed by three people during validation and so will be kept. Other items will be fixed.

Fixed 5

H 3

B B

S (41.3) 6 H

2 B

N S

(41.8) 7 F

2 B

N E

S (41.7) Editorial: Spell out EW in stem. Make (2) an actual question in stem.

Fixed 8

F 3

B N

E S

(41.5) In the stem information we ask them in (1) what the status is of the back-up heaters and then in (2) we tell them that all heaters are energized. Perhaps in (2) we drop the first statement that all heaters are energized and just ask them the second part.

Fixed 9

H 3

B B

S (41.7) 2013 NRC EXAM 10 H

2 B

B E

S (41.10) Ensure values for RCS temp / pressure mean that it IS saturated.

Which of the following actions is required NEXT is probably too vague a question -

requires applicant to know exactly what part of the procedure they are currently in.

Do they have enough info to know that? How many HPSI pumps are currently running? Improve stem.

Fixed 11 H

2 C

M E

S (41.4) Since the correct answer is SG1 flow is < SG2 flow perhaps we should change the distracters to SG2 flow is < SG1 flow instead of being equal which seems easily discounted based on the conditions. We can discuss.

Fixed

ES-401 PV-2015-04 3

Form ES-401-9 Page 3 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 12 F

2 B

N S

(41.10) 13 H

2 A

N S

(41.7) 14 F

2 D

N S

(41.7) Is double sequencing an actual term / concern? If not could be considered not plausible distractor.

This is term widely used. Missed several times on validation.

15 F

3 B

N S

(41.7) Credibility of DG A tripping?

Loss of M-41 control power would trip the diesel. This is plausible.

16 H

3 A

N E

S (41.4) Spell out what NC and EW are. Same for all abbreviations in this exam. Also, explanation for questions B and C appear to be reversed from question page.

Fixed 17 F

3 C

B S

(41.8) 2010 NRC EXAM Spell out AFU.

Fixed 18 F

2 D

B S

(41.7) 19 H

2 B

N S

(41.6) 20 H

3 D

N S

(41.5) 21 H

2 A

M S

(41.10) 22 F

2 D

A B

E S

(41.7) 2010 NRC EXAM - K/A also requires knowledge of proper switch positions.

Fixed 23 F

2 D

N S

(41.10) 24 H

3 A

B S

(41.11) 2012 NRC EXAM 25 H

3 D

N E

S (41.7) Label the 2 parts (1) and (2) for clarity.

Answers discuss WCA-UV-62 but explanations discuss WCA-UV-61. Explain.

Fixed 62 is the valve not 61.

ES-401 PV-2015-04 4

Form ES-401-9 Page 4 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 26 H

2 B

N S

(41.7) 27 F

1 C

D N

U S

(41.10) LOD=1 Question Replaced. Now Sat.

28 F

2 A

B S

(41.5) 29 H

2 D

X N

U S

(41.6) K/A mismatch. K/A requires knowledge of how to prevent exceeding design limits not reason for changes in indications. K/A also requires incorporation of RCS pressure along with temperature.

Question Replaced. Now Sat.

30 H

3 C

N S

(41.7) Preassurizer Fixed 31 F

2 B

B S

(41.7) 32 H

2 D

N S

(41.7) 33 H

3 A

N S

(41.5) Why is it necessary to provide Condition 1 and Condition 2? Neither condition is actually referenced in the question.

These conditions are needed according to validators. High miss rate during validation.

34 F

2 C

B S

(41.4) 35 H

3 D

N E

S (41.10) B could be argued correct since tripping RX in 10 min is within 40 min.

Tweak the wording which one of the following actions is required.., or B. Trip the reactor and perform SPTAs within 10 minutes maximum Fixed 36 H

3 A

N S

(41.7) 37 F

3 A

N S

(41.7) 38 H

2 A

N E

(41.5) Distracter B not credible. No indications in stem that would lead applicant to think ESD is plausible.

ES-401 PV-2015-04 5

Form ES-401-9 Page 5 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

Fixed 39 F

1 B

A N

U S

(41.7) LOD=1 Question Replaced. Now Sat 40 F

2 A

B S

(41.4) 41 H

3 D

N S

(41.7) 42 F

2 C

B S

(41.7) 2013 NRC EXAM 43 F

2 D

N S

(41.4) 44 H

2 B

N S

(41.5) 45 F

3 C

N S

(41.8) 46 F

3 A

N S

(41.7) 47 H

2 D

N S

(41.7) 48 H

3 D

N S

(41.7) 49 F

B B

S (41.7) 50 H

2 A

B S

(41.7) 51 F

3 C

N S

(41.7) 52 H

3 B

B S

(41.7) 2013 NRC EXAM 53 F

3 D

N S

(41.7) 54 H

3 A

N S

(41.10) 55 H

2 B

B S

(41.7) 56 H

2 A

B E

S (41.5) Not credible to have SBCV fail open with no movement in temperature. This effects distracter C.

Fixed 57 F

3 B

N S

(41.7)

ES-401 PV-2015-04 6

Form ES-401-9 Page 6 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 58 F

2 B

N E

S (41.10) Editorial: 43.1 is for SRO only. This needs to be changed to (41.10). Would change (2) to read CEAs must remain above__ withdrawn by pulse counter Fixed 59 H

3 C

B S

(41.7) 60 F

2 B

N S

(41.7) 61 H

2 B

N S

(41.7) 62 H

2 B

B S

(41.7) Editorial: Change could be to are in progress Will fix.

63 F

2 X

A N

U S

(41.7) Not credible for the first action to place tripped pump to PULL TO LOCK.

This affects two distracters.

Question Replaced. Now Sat.

64 F

2 B

N E

S (41.7) E-3 step 69 doesnt appear to require that the H2 recombiners malfunction for TSC to direct operation of H2 purge Exh Sys - it just says If directed by TSC. Is applicant going to know TSCs decision making process?

Fixed 65 F

2 C

B S

(41.7) 66 H

2 A

M S

(41.10) Reference Provided 67 F

2 C

N S

(41.5) Partial K/A match. K/A requires knowledge of secondary chemistry limits as well. Suggestion would be to stick with the 2 x 2 format and use the second part to ask about the secondary.

Writing to the more difficult part of the K/A. This question is ok. Would be minutia to ask secondary plant chemistry question.

68 H

2 B

N S

(41.8) 69 F

2 D

N S

(41.10) 70 F

2 B

N E

(41.10) Editorial: This is a fundamental memory question not analysis.

ES-401 PV-2015-04 7

Form ES-401-9 Page 7 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

Improve answers: Process temperature 210F or higher; system pressure 500 Fixed 71 F

2 D

X N

U S

(41.12) TS bases are typically SRO knowledge.

Question Replaced. Now Sat.

72 F

2 B

N S

(41.11 73 F

2 A

B S

(41.10) 74 F

2 B

N S

(41.10) 75 F

2 D

M S

(41.10) 76 H

3 C

N S

(43.1) 77 H

3 C

N S

(43.5) 78 F

3 D

N S

(43.1) 79 H

2 D

N S

(43.5) 80 H

3 A

N S

(43.1) 81 H

2 C

N S

(43.5) 82 H

2 A

N S

(43.1) 83 H

3 C

B S

(43.5) 2010 NRC EXAM 84 H

3 B

N S

(43.5) Reference Provided 85 H

2 B

N S

(43.5) Reference Provided 86 H

2 C

N S

(43.5) Reference Provided 87 H

2 C

B S

(43.5) 2012 NRC EXAM 88 H

2 D

M S

(43.5) 89 H

2 D

N S

(43.1) Reference Provided 90 F

3 B

N S

(43.2)

ES-401 PV-2015-04 8

Form ES-401-9 Page 8 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 91 F

3 C

N S

(43.1) 92 F

2 B

N S

(43.7) 93 H

3 D

N S

(43.5) 94 F

2 D

N S

(43.1) 95 H

3 D

N S

(43.5) 96 F

2 A

N S

(43.1) 97 F

2 A

N S

(43.1) 98 H

3 C

N S

(43.4) 99 F

2 A

N S

(43.4) 100 H

2 B

N S

(43.5) Reference Provided RO TOTALS:

B= 21 F= 37 E= 16 Additional Notes:

M= 4 H= 38 U= 5 N= 50 SRO TOTALS:

B= 2 F= 8 E= 0 Additional Notes:

M= 1 H= 17 U= 0 N= 22

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2.

Chief Examiner comments are indicated in blue.

3.

Average difficulty is _ 2.36 ___ on the RO exam and _2.44__ on the SRO exam.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 1 43.1 = 9 41.2 = 0 43.2 = 1 41.3 = 2 43.3 = 0 41.4 = 4 43.4 = 2 41.5 = 9 43.5 =12

ES-401 PV-2015-04 9

Form ES-401-9 Page 9 of 9 OBDI 202 - IOLE Process 41.6 = 2 43.6 = 0 41.7 = 35 43.7 = 1 41.8 = 4 41.9 = 0 41.10 = 16 41.11 = 2 41.12 = 1 41.13 =0 41.14 = 0

5.

The answer distribution is: RO / SRO A = 17 (23%)

/

5 (20%)

B = 27 (36%)

/

5 (20%)

C = 13 (17%)

/

8 (32%)

D = 18 (24%)

/

7 (28%)

6.

There are 6 questions with attachments provided.