ML121080601

PVNGS-2012-03-DRAFT-Written Examination Comments

ML121080601
Person / Time
Site:	Palo Verde
Issue date:	03/16/2012
From:	Division of Reactor Safety IV
To:	Arizona Public Service Co
Thomas Farina
References
PV-2012-03, ES-401-9
Download: ML121080601 (9)

Text

ES-401 PV-2012-03 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 1

F 2

N S

CR, 41.7 2

H 3

E B

E S

CR, 41.14, NRC 2005 - Recommend changing distractors A and B so they are above 212 degrees to improve the credibility. Made distractors > 212 3

H 3

N S

CR, 41.7 Question replaced.

4 H

3 E

B E

S CR, 41.10, NRC 2010 - Recommend adding KA verbiage to pedigree. I believe an argument could be made that distractor C is also correct given the conditions provided require entering the EOPs. 55.41.(7) does not appear to apply. Added verbiage to KA statement. Changed to 55.41 (10). Changed EOP to ORP in distracter C.

5 H

3 N

S CR, 41.7 6

H 3

B E

S CR, 41.10, NRC 2010 - In K/A statement remove the word pump from loss of reactor coolant pump makeup. Edit question to state which of the following actions

ES-401 PV-2012-03 2

Form ES-401-9 Page 2 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only is procedurally required? Removed pump. Question edited.

7 H

2 E

B E

S CR, 41.14, NRC 2007 - Recommend rewriting stem so it reads; LPSI amps and flow are oscillating. This is an indication of:.

Remove bullet from aux spray and write Auxiliary Spray valve "B" fails open and the following conditions are observed: Removed pump. Question edited.

8 F

2 U

N U

S CR, 41.10 - This is a KA mismatch because it does not test on the reason for the EOP guidance actions, it tests on what actions to take.

Also, this K/A requires testing on CCW, meaning the closed loop system which cools primary components, (like NC nuclear cooling water). Question as written tests on loss of TCW.

Question replaced.

9 H

2 N

S CR, 41.10 -- Ensure that when exam is printed, applicants will be able to tell the color of the annunciator.

10 F

2 B

S CR, 41.3, NRC 2009 11 F

2 E

N E

S CR, 41.10 - The stem would be better written if it simply stated: Which of the following is the preferred method to control level in an isolated steam generator with a ruptured tube? Recommended changes made to stem.

12 F

2 E

B E

S CR, 41.7, NRC, NRC 2009 - Recommend rewriting stem to read: If there is a station blackout, the class (PK) batteries are designed to maintain rated voltage for UP TO:

Recommended changes made.

13 H

3 N

S CR, 41.7 - Improve explanation for why Distractor A is incorrect. Recommended changes made.

14 F

3 N

S CR, 41.7 - Include the system # and system with the k/a. Recommended changes made to KA statement.

15 F

3 U

N U

S CR, 41.10 - Need amplifying info How are Dist-A and Dist-C plausible if we are telling them that PKA-M41 control power has been lost? Why would applicant choose Battery A output as an answer, since the output is PKA-M41? Seems like its testing if the applicant knows what the Batt-A output is. Revised.

16 H

3 M

S CR, 41.10 -- Justification for Dist-B says correct. Dist-D change are to is.

Recommended changes made.

17 H

3 N

S CR, 41.5 18 H

3 N

S CR, 41.4 -- Change valves handswitches to valve handswitches Done

ES-401 PV-2012-03 3

Form ES-401-9 Page 3 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 19 H

3 B

S CR, 41.6, NRC 2008 -- Show the math for the correct answer in justification. Done 20 F

3 N

S CR, 41.6 -- Edit the stem to add context as to why SPTAs are being performed (i.e.

multiple inop CEAs, etc) Added trip initiator conditions to stem.

21 H

3 E

N E

S CR, 41.5 -- If distractors B,C,D are all system responses that would occur on a lowering pzr level, then they can all be eliminated by the applicant because if one was true then all three would be true. Re-wrote question to added low/high conditions to distractors and answer.

22 H

3 U

B N

U S

CR, 41.11, NRC 2008 - This is a KA mismatch because it does not involve a leak from the gaseous radwaste system. A primary pressure boundary leak doesnt constitutes an accidental gaseous radwaste release in the context of this K/A.

Replaced.

23 F

2 N

S CR, 41.10 24 F

4 E

N E

S CR, 43.2 - As written, Distractors A and D are also correct. While both valves may not be CTMT isolation valves, if only one of them listed is INOP then containment integrity would still be lost. Edit wording of the stem. SAT as originally written.

25 F

2 B

S CR, 41.10, NRC 2008 - Include the name of the AOP entered in the stem.

Recommended changes made to stem.

26 H

F 3

N E

S CR, 41.10 -- Include in accordance with procedure xxxx in question. Re-wrote stem to make recommended changes and stated Natl Circulation not met which required changing cognitive level to memory.

27 F

3 B

S CR, 41.10, NRC 2007 28 H

3 U

B U

S CR, 41.2, NRC 2008 - The distractors are not included in the question, it is incomplete. Revised.

29 H

3 B

S CR, 41.4, NRC 2005 30 H

3 B

S CR, 41.7 31 H

3 N

S CR, 43.2 32 F

3 B

S CR, 41.7 33 H

3 E

N E

S CR, 41.7 - Partial K/A mismatch. The question does not test on the second part of the K/A, (b) based on those predictions, use procedures to correct, control, or mitigate the consequences of those malfunctions or operations. Revised.

34 H

3 B

E CR, 41.7, NRC 2007 -- Dist-C: add, and NCN-P01A is stopped or something

ES-401 PV-2012-03 4

Form ES-401-9 Page 4 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only S

similar to differentiate it from Dist-D, if that is the intent. Revised.

35 H

3 N

S OR, 41.14 36 F

3 B

S CR, 41.6 37 H

3 N

S CR, 41.6 38 H

3 B

S CR, 41.7, NRC 2007 39 H

3 N

S CR, 41.7 40 H

3 N

S CR, 41.7 41 H

3 N

S CR, 41.7 42 F

3 N

S CR, 41.4 43 H

3 U

B U

S CR, 41.5, NRC 2005 -- Distractor B is implausible; colder water would not cause a heatup rate.

Distractor A: If there a LTOP valve on SDC loop A with a similar setpoint as loop B then this distractor is implausible.

Question replaced.

44 H

3 N

S CR, 41.7 45 H

4 N

S CR, 41.7 46 H

3 E

N E

S CR, 41.4 -- Line up the (1) and (2) into straight columns for ease of reading.

Not necessary to have 3 parts per answer make it 2 parts per answer. When you have 3-part answers in a four choice question, the applicant can deduce the right answer without knowing all parts. Revised.

47 F

2 N

S CR, 41.7 - Delete the s at the end of the stem.

48 H

3 B

S CR, 41.7, NRC 2007 49 H

3 N

S CR, 41.7 -- Line up the (2)s into a straight column 50 F

3 N

S CR, 41.7 -- The full K/A statement is not included on the worksheet.

51 F

4 U

N U

S CR, 43.2 - Include system number and name with K/A statement (073 PRM)

The 3 distractors are all 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> or greater action statements, which an RO is not expected to have memorized. Revised.

52 H

3 N

S CR, 41.10

ES-401 PV-2012-03 5

Form ES-401-9 Page 5 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 53 F

3 N

S CR, 41.4 54 F

3 B

S CR, 41.4, NRC 2007 -- The worksheet says this is a new question but that it also appeared on 2007 exam. Which is it? Correctly linked to 2007 exam. Added to justification why knowing location is given.

55 H

3 B

S CR, 41.10, NRC 2007 56 H

3 B

S CR, 41.6, NRC 2008 -- Add in accordance with procedure xxx to question, otherwise Dist-C could be argued correct.

57 F

2 E

N E

S CR, 41.10 - Distractor D is also correct as written. Change question to ask what the threshold is. Recommended changes made to stem.

58 F

3 B

S CR, 41.7, NRC 2008 59 F

3 B

S CR, 41.7, NRC 2010 -- Justifications for A and B are swapped.

60 F

3 E

N E

S CR, 41.7 -- Add and may remain so to Distractor A. Change its to their in Dist-B and -C. Recommended changes made.

61 F

2 N

S CR, 41.7 -- Align (2)s into a straight column.

62 F

3 N

S CR, 41.4 63 H

3 N

S CR, 41.4 -- Align (2)s into a straight column.

64 F

3 N

S CR, 41.11 65 F

2 E

N E

S CR, 41.7 -- Distractors C and D can be considered non-plausible because they are non-Class buses. Changed distracters C and D to class buses.

66 F

2 N

S CR, 41.10 67 F

2 E

N E

S CR, 41.10 -- Distractor D is all of the above, which should not be used. Add a period after Dist-C. Rewrote distracter D and added period to distracter C.

68 F

3 B

S CR, 41.10 69 F

2 N

S CR, 41.10 70 F

2 U

B N

U S

CR, 41.10, NRC 2007 - The distractors are a collection of True False statements.

Applicant can answer without reading the stem. Question Replaced 71 F

3 N

S CR, 41.12

ES-401 PV-2012-03 6

Form ES-401-9 Page 6 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 72 H

1 2

U U

B U

S CR, 41.12 - Based on the wording of the question, Dist-A is true - if you receive a Dose alarm youd be required to leave immediately. The stem is also not plausible in that the REP would not assign a Dose Rate alarm setpoint equal to the known rad level at the worksite. Revised.

73 F

2 E

M N

E S

CR, 41.10 - An argument could be made that distractor B is also correct.. Revised.

74 F

2 E

N E

S CR, 41.10 - An argument could be made that distractor A is also correct. Revised.

75 F

2 B

S CR, 41.10, NRC 2008 76 H

3 E

N E

S C, OR, 43.5 - Remove the EP code from the distractors as they cue the applicant where to look in the classification chart. Add the system name to K/A statement.

Removed EP codes and added name to K/A statement.

77 H

3 N

S B, CR, 43.5 78 H

3 N

S D, CR, 43.5 79 H

3 B

S A, CR, 43.5, NRC 2008 80 H

3 N

S B, CR, 43.5 81 H

3 B

S B, OR, 43.5, NRC 2009 82 F

4 N

S B, CR, 43.2 - Add system name to KA statement. Added system to K/A 83 H

3 N

S C, CR, 43.5 84 H

3 E

N B

E S

B, CR, 43.5 -- The stem gives plant conditions as they were at the onset of the event, but then asks the applicant to perform a diagnosis assuming SPTAs have been completed. For operational validity the applicant should know the conditions as they would be when the diagnosis is made, e.g., at completion of SPTAs. Changed stem to give current conditions. Added required procedures to answers.

85 H

3 E

B E

S C, CR, 41.10, NRC 2007 - Distractors A and B are subsets of C and D. A is partially correct. Revised.

86 H

3 U

N U

S B, CR, 43.5 -- Recommend adding concentrations in the primary coolant to the first part of each distractor for clarification, since there would be increased H2 concentration in the VCT. This is also a 3-part question, meaning that the applicant can deduce the answer without knowing the full answer. Revised.

87 H

3 N

S C, CR, 43.5

ES-401 PV-2012-03 7

Form ES-401-9 Page 7 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 88 H

3 U

N U

S A, CR, 41.10 - This is an RO question since it can be answered without knowing what procedure to reference, with just systems knowledge. I.e., if you know the controller response and how many RCPs to trip you can answer the question.

Revised.

89 H

3 U

N U

S B, CR, 41.10 - This is an RO question since it tests on a specific required operator action. Question is not testing on the second part of 55.43 (5), Assessment of facility conditions and selection of appropriate procedures during normal, abnormal, and emergency situations. Include system name in K/A statement. Revised.

90 H

3 E

B N

E S

C, CR, 43.5, NRC 2008 -- The crew should already be in the Alarm Response Procedure, so the only realistic transition would be to the Off-Normal Procedure. The ARP also directs you to go to the correct Off-Normal procedure. Provide values for IA header pressure for operational validity. Revised.

91 H

2 E

N E

S C, OR, 43.2 - Delete the TS bases from the provided references. We generally expect SROs to know the bases for the tech specs.

Justification for Dist-B appears incorrect. B would be correct if all 4 vent paths were inoperable.

Dist-D appears implausible if we are giving the applicant the TS, and there is no 7 day action statement in that TS. Revised.

92 H

3 U

U B

U S

A, CR, 43.5, NRC 2008 - This is an RO question since it can be answered without knowing what procedure to reference. I.e., if you know the SBCS response and what temperature to maintain, you can answer the question.

Distractors A and C say maintain Tc at 556, but the stem says that Tc dropped to 546 on trip - is this intended?

Distractor C is also correct, in that the Quick Open block IS the normal function for the SBCS under these conditions. Revised.

93 H

3 N

S D, CR, 43.2 94 F

2 B

S A, CR, 43.5, NRC 2010 95 H

2 B

S D, OR, 43.2, NRC 2008 - 55.43.(5) doesnt fit this question as written because there is neither assessment of facility conditions or procedure selection in this question.

96 F

2 B

S A, CR, 43.3 -- Add in accordance with procedure xxx to the stem. 10CFR55.43.(3) is probably a better category for this question.

97 F

3 B

S D, CR, 43.5, NRC 2007 -- 10CFR55.43.(4) isnt a good category to place this question in.

98 F

1 U

N U

B, OR, 43.4 - This question is a direct lookup. Replaced.

ES-401 PV-2012-03 8

Form ES-401-9 Page 8 of 9 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 2

B S

99 F

2 B

S C, CR, 43.5 100 F

2 B

S B, CR, 43.4 RO TOTALS:

B= 27 (36%)

F= 38 (51%)

E= 18 (24%)

Additional Notes: Question 28 is incomplete. Revised.

M= 1 H= 37 (49%)

U= 8 (11%)

N= 47 SRO TOTALS:

B= 12 (48%)

F= 7 (28%)

E= 5 (20)%

Additional Notes:

M= 0 H= 18 (72%)

U= 5 (20%)

N= 13

GENERAL COMMENT

S:

-Ensure the system number is included on all K/A listings.

-Ensure the justifications are readable in plain English, not just a set of incomplete statements.

-When a procedure number is given (e.g. 40AO-9ZZ02), you need to also give the name of the procedure (e.g. Excessive RCS Leakrate)

-In general, if asking for an answer that is found in a specific procedure, reference that procedure in the question (in accordance with xxx) unless it would cue the answer.

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2.

Chief Examiner comments are indicated in blue.

3.

Average difficulty is __ 2.76 on the RO exam and 2.76 on the SRO exam.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 43.1 = 0 41.2 = 0 43.2 = 4 41.3 = 1 43.3 = 1 41.4 = 8 43.4 = 2 41.5 = 3 43.5 = 15

ES-401 PV-2012-03 9

Form ES-401-9 Page 9 of 9 OBDI 202 - IOLE Process 41.6 = 5 43.6 = 0 41.7 = 25 43.7 = 0 41.8 = 0 41.9 = 0 41.10 = 22 41.11 = 2 41.12 = 2 41.13 = 0 41.14 = 3

5.

The answer distribution is: RO / SRO A = 18

/

5 B = 24

/

9 C = 11

/

7 D = 22

/

4

6.

There are 1 / 5 questions with attachments provided.

7.

There are 29 questions from previous NRC exams at Palo Verde. Three were from the 2010/2011 (last 2) exams. The NRC exam questions are all listed in the individual comment boxes.