ML12325A403

FC-2012-09 Draft Written Exam Comments Steely

ML12325A403
Person / Time
Site:	Fort Calhoun
Issue date:	09/21/2012
From:	Vincent Gaddy Operations Branch IV
To:	Omaha Public Power District
Laura Hurley
References
ES-401, ES-401-9 50-285/12-009
Download: ML12325A403 (7)

Text

ES-401 FC-2012-09 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 1

F 2

B E

B(41.10) Editorialreplace attempted with attempts. Fixed 2

H 3

N E

D(41.10) Give the applicant a bullet detailing exactly what subcooling is reading in the control room. This is an indication they would have. Also, distracter B really does not seem plausible to stop all charging pumps. Revise this distracter.

Consider making PZR level 60% and remove rapidly from bullet. Make action plural - What actions should be taken Fixed 3

H 3

B S

C(41.5) 4 F

3 B

S C(41.10) 2005 NRC EXAM 5

F 2

B S

D(41.3) 6 H

2 N

E B(41.7) Distracter A stands out as significantly different than the correct answer or the other distracters. This is psychometrically flawed. Also, plausibility is suspect for a pump to start on an alarm rather than some level deviation. Recommend changing distracter D as well due to significant difference to the other distracters numerically

ES-401 FC-2012-09 2

Form ES-401-9 Page 2 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only speaking. Make it closer to other distracters. If CCP starts at -2.9%, it will start at -

3.3% and -6%: more than one correct answer. Change question so that only -2.9%

will be correct. Fixed 7

H 1

N E

B(41.10) HANDOUT Simple graph exercise. LOD=1 Fixed 8

F 2

B E

D(41.7) Consider putting The Control switch for HCV-400A/C must be in the stem and out of all the answers. Fixed 9

H 3

M E

D(41.7) This is a two part answer. Must know probable cause of the failure AND the action to be taken. Only need to know one part in order to answer the question correctly. Fixed 10 H

3 B

S A(41.1) 2007 NRC EXAM 11 H

3 B

S B(41.11) 12 H

3 N

E A(41.10) Two part answer. Only need to know one part to answer the question correctly. Suggest making this a two by two. Stem - What action is directed, not actions are - only one action per answer. Also, should be and why is this action directed? Fixed 13 F

3 N

S C(41.10) 14 H

3 M

S B(41.8) 15 H

3 B

E B(41.10) 2009 NRC EXAM Put An additional vital instrument bus failure will result in in the stem and out of the answer. Fixed 16 H

3 N

S C(41.10) 17 H

3 B

S B(41.7) 2005 NRC EXAM 18 F

3 N

E A(41.10) Provide another bullet that states 4160V buses indicate whatever is expected under these conditions. This is an indication an applicant would have and would eliminate C and D as correct answers if the applicant is knowledgeable of the procedure. Fixed 19 H

2 N

S B(41.1) 20 H

2 M

S A(41.1) 2001/2002 NRC EXAM 21 H

3 B

E C(41.7) 1999 NRC EXAM Three part answer - only need to know two parts to answer correctly. Fixed 22 F

2 N

S B(41.8)

ES-401 FC-2012-09 3

Form ES-401-9 Page 3 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 23 F

2 X

N U

B(41.1) 1995 NRC EXAM K/A requires knowledge of the reason Fixed 24 F

2 X

N U

C(41.13) 2002 NRC EXAM K/A requires knowledge of the reason Fixed 25 H

3 M

S A(41.10) 26 F

2 B

S A(41.11) 2004 NRC EXAM 27 H

2 B

S A(41.10) 2005 NRC EXAM 28 H

3 B

S C(41.3) 1997 NRC EXAM 29 H

2 N

S B(41.7) 30 F

3 M

S B(41.5) 2001/2 NRC EXAM 31 H

2 X

N U

D(41.10) K/A requires knowledge of EOP entry conditions and immediate actions.

Fixed 32 F

2 B

E D(41.7) 2005 NRC EXAM Construct a scenario around this question. As written it is pure memorization. Needs to be operationally based/valid. Fixed 33 H

3 B

E C(41.10) 2009 NRC EXAM Distracter D more plausible if you trip one LPSI pump and not all the running LPSI pumps. D would be true when SIRWT level gets low enough - enhance stem to avoid D being true. A change minimize to reduce.

Fixed 34 F

2 N

S A(41.10) 35 H

2 M

E B(41.7) Same as question 32. Construct a scenario to test this knowledge. HCV-492A/B - is this for HX AC-1D? Should it be for HCV-491A/B? Fixed 36 H

3 N

S A(41.7) 37 F

2 B

S D(41.10) 2009 NRC EXAM 38 H

3 B

S C(41.6) 2009 NRC EXAM 39 H

3 B

S C(41.8) 40 H

2 B

S C(41.10) HANDOUT 41 F

2 N

S D(41.5) 42 H

3 B

S B(41.5) 2004 NRC EXAM 43 H

2 N

S B(41.4)

ES-401 FC-2012-09 4

Form ES-401-9 Page 4 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 44 F

3 M

E C(41.1) Question mark, not period, at end of stem. Fixed 45 F

2 N

S A(41.7) 46 F

2 N

S B(41.4) 47 F

2 N

S B(41.7) 48 F

2 B

E B(41.7) 2001/2 NRC EXAM Develop operational scenario to test this knowledge.

Fixed 49 H

3 M

E C(41.8) 2007 NRC EXAM Stopping LO-4 is technically correct per Attachment 6 of the procedure. Ask what action is first taken, per the procedure, and why. This will eliminate any potential second correct answers. B and D - LO-12B is the Emergency Seal Oil Pump per the ref, not DC Seal Oil Pump. Fixed 50 F

2 N

S C(41.5) 51 H

3 N

S B(41.11) 52 F

2 N

E A(41.12) Need question mark at end of stem. Fixed 53 F

2 N

E B(41.7) 2005 NRC EXAM Simple memorization. Develop scenario question. Fixed 54 H

3 B

E D(41.10) 2007 NRC EXAM Add bullets to annunciators in stem. Capitalize raw water in C. Fixed 55 F

2 B

S C(41.5) 1995 NRC EXAM 56 F

2 B

S A(41.3) 57 F

2 N

E D(41.3) Are all the MCCs 480V? I dont want to cue that in the stem if thats not the case. I want the applicant to have to understand they are powered by 480VAC and pick the correct P/S from a group that may contain non-480V power supplies. Fixed 58 H

2 B

S B(41.5) 2009 NRC EXAM, HANDOUT 59 F

3 N

E D(41.2) How does wiring fail open? Consider The electrical circuit for one 60 H

3 N

S D(41.10) 61 F

2 N

E B(41.7) K/A is predict/monitor changes in Containment Pressure. Fixed 62 F

3 B

S D(41.5) 2004 NRC EXAM 63 H

3 M

S A(41.4) 64 H

3 N

S D(41.12)

ES-401 FC-2012-09 5

Form ES-401-9 Page 5 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 65 F

2 M

E D(41.4) Memorization. Develop scenario so that the applicant has to analyze whether or not the valves are open/closed based on certain plant conditions. AOP-17 states I A system < 80 psig, not < or equal. Change stem. Fixed 66 F

2 M

S C(41.10) 2001 NRC EXAM 67 H

2 N

S A(41.10) Make sure not to include AOP-37 in ADAMS files!

68 F

2 N

S B(41.10) 69 F

2 B

E B(41.10) 2007 NRC EXAM Perhaps change this question to whom is responsible for the final approval. Fixed 70 H

3 N

E B(41.5) HANDOUT Fig 1-1 is mentioned in the question stem. Is this figure going to be a handout for the applicants? The question data sheet says no to a handout. Put To ensure that DNBR in stem. Fixed 71 F

3 B

S B(41.11) 2009 NRC EXAM 72 F

2 N

S D(41.12) 73 H

2 N

S C(41.10) 74 F

1 N

U A(41.10) LOD=1. Lets discuss whether or not we might need to resample this K/A.

Fixed 75 H

3 B

S A(41.10) 2009 NRC EXAM 76 H

2 M

S A(43.5) 2005 NRC EXAM 77 H

3 N

U C(43.5) Answer can be determined with RO level knowledge of AOP entry conditions.

Fixed 78 H

3 N

E B(43.2) HANDOUT Replace distracter A. Stands out as an outlier and is psychometrically unsound. Fixed 79 H

2 N

S D(43.5) 80 H

2 N

S D(43.5) 81 H

3 N

S A(43.5) 82 H

3 B

S A(43.5) 2004 NRC EXAM 83 F

3 N

S C(43.2) 84 F

2 N

S B(43.5)

ES-401 FC-2012-09 6

Form ES-401-9 Page 6 of 7 OBDI 202 - IOLE Process Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

B/M/N

7.

U/E/S

8.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 85 F

3 N

E A(43.5) Dont see plausibility of entering Loss of Containment Integrity with just the containment equipment hatch open. Revise distracter D. Fixed 86 H

2 M

E A(43.5) Remove AOP from stem. Make the applicant figure this out. Lets discuss whether or not this is RO level knowledge. Fixed 87 F

3 N

S A(43.5) 88 H

2 N

S D(43.5) 89 H

3 M

S C(43.5) 2010 NRC EXAM 90 H

3 N

S A(43.5) 91 H

2 N

U D(43.5) RO level knowledge. Fixed 92 H

2 N

E A(43.5) Entry condition to major EOP. RO level question. Also, only need to know one part of answer not both. Fixed 93 H

3 N

E D(43.5) Capitalize SPTA in stem. Add periods to answers. Fixed 94 F

2 B

E B(43.1) 1997 NRC EXAM Replace slipped on the stairs with the STA being unavailable to the crew due to an emergency medical condition or words to that effect. Fixed 95 F

2 N

S D(43.5) 96 F

2 N

E C(43.5) Reword this question so that the answer requires knowledge of the department manager or above attending the pre-job brief vice just knowing the appropriate procedure. This more accurately reflects the K/A of understanding the process on infrequent tests. Fixed 97 F

3 N

E B(43.2) Not a Tier 3 question. Fixed 98 F

2 B

S C(43.4) 2009 NRC EXAM 99 H

2 N

S C(43.5) 100 F

2 B

S D(43.1) 2005 NRC EXAM RO TOTALS:

B= 29 F= 36 E= 25 Additional Notes:

M= 11 H= 39 U= 4 N= 35 SRO TOTALS:

B= 4 F= 9 E= 8 Additional Notes:

ES-401 FC-2012-09 7

Form ES-401-9 Page 7 of 7 OBDI 202 - IOLE Process M= 3 H= 16 U= 2 N= 18

GENERAL COMMENT

S:

1.

All questions from previous NRC exams should be checked to see if there were any post-exam comments.

2.

Be consistent in capitalization (Control Room vs control room.

3.

Worksheets should include LOD for all questions.

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2.

Chief Examiner comments are indicated in blue.

3.

Average difficulty is __ ___ on the RO exam and ___ on the SRO exam.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 =

43.1 =

41.2 =

43.2 =

41.3 =

43.3 =

41.4 =

43.4 =

41.5 =

43.5 =

41.6 =

43.6 =

41.7 =

43.7 =

41.8 =

41.9 =

41.10 =

41.11 =

41.12 =

41.13 =

41.14 =

5.

The answer distribution is: RO / SRO A = 13 (17%)

/

6 (24%)

B = 16 (22%)

/

7 (28%)

C = 21 (28%)

/

5 (20%)

D = 25 (33%)

/

7 (28%)

6.

There are questions with attachments provided.