Text

2009-08-DRAFT Written Exam Review
	ML093200397
Person / Time
Site:	Wolf Creek
Issue date:	11/16/2009
From:	Apger G; NRC Region 4
To:	; Wolf Creek
References
	50-482/09-301, ES-401, ES-401-9 50-482/09-301
	Download: ML093200397 (8)
	v • d • e

ES-401 WC-2009-08 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

/

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 1

F 3

B S

D, CR, 41.7 (VC Summer 2006) 2 H

3 B

S A, CR, 41.5 (Callaway 2007) 3 F

3 B

S B, CR, 41.4 4

F 3

B S

B, CR, 41.7 (McGuire 2008) 5 H

3 N

S B, CR, 41.13 6

H 3

B S

B, CR, 41.7 7

F 3

N S

C, CR, 41.7 8

F 3

N S

B, CR, 41.7 Question was simplified to make it memory.

9 F

3 B

E B, CR, 41.10 (Watts Barr 2004) Remove - Of the following choices from the stem.

Changes made.

ES-401 WC-2009-08 2

Form ES-401-9 Page 2 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

/

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 10 H

3 B

S A, CR, 41.5 (VC Summer 2005) 11 H

3 N

S D, CR, 41.6 12 H

2 B

S B, CR, 41.7 (Beaver Valley 2002) 13 F

2 B

S C, CR, 41.7 (Diablo Canyon 2007) Aligned distractors.

14 F

3 E

N E

D, CR, 41.7 - Answer B could also be argued as being correct. Remove the lead-in for the answers. Remove performance of EMG C-0, Attachment C from all answers. Changes made. D is correct.

15 H

3 B

E A, CR, 41.10 (McGuire 2007) Add a ; after AFW system in the stem. Change made.

16 H

3 N

S C, CR, 41.7 17 H

3 B

S B, CR, 41.7 (Sequoyah 2007) 18 H

3 N

E C, CR, 41.7 Change stem to: Which of the following describes the status of the Containment Spray System? Changes made.

19 H

3 N

S D, CR, 41.4 20 F

3 M

E B, CR, 41.4 In each of the answers, change are under control of to are fully controllable using Changes made.

21 H

3 U

B U

S C, CR, 41.10 - This is a KA mismatch because the question does not involve any aspect of an emergency procedure as required. (Farley 2006) This question meets the KA since the question asks alarm setpoints and ARP actions as required by the KA.

22 H

3 N

S D, CR, 41.10 23 F

3 N

S A, CR, 41.4 24 H

3 B

S C, CR, 41.10 (Wolf Creek 2007) 25 F

3 N

S B, CR, 41.7 26 F

4 U

B U

B, CR, 43.2 - This is an SRO question because it tests on a technical specification application. Also, the LCO is greater than 1 hour1.157407e-5 days 2.777778e-4 hours 1.653439e-6 weeks 3.805e-7 months making it a non-memory question.

(McGuire 2008). Question was rewritten with a new KA (2.1.20). Now SAT.

27 H

3 U

N U

E A, CR, 41. 6 - This is a KA mismatch because it does not predict the impact on RPS.

Meets KA. Changed fifth bullet of stem.

ES-401 WC-2009-08 3

Form ES-401-9 Page 3 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

/

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 28 F

3 N

S C, CR, 41.10 29 H

3 B

S D, CR, 41.6 (Beaver Valley 2007) 30 H

3 B

S C, CR, 41.7 (Wolf Creek 2007) 31 H

3 N

S D, CR, 41.10 32 F

2 N

E D, CR, 41.4 Remove the last two bullets. Change stem to Which ONE of the following describes the operation of the Turbine Emergency Trip System (ETS) if the Main Turbine were manually tripped from the control room. Changes made.

33 F

3 B

S B, CR, 41.7 (Wolf Creek 2007) 34 H

3 U

B U

S B, CR, 41.4 - This is a KA mismatch because it does not test on liquid radwaste. OK as written.

35 H

3 N

S B, CR, 41.9 36 F

3 E

N E

D, CR, 41.4 - Recommend rewording the stem since it states the system has been returned to normal standby pressure implies the fire pumps have been shutdown.

Therefore, it could be argued there are no correct answers.

Change the last bullet to and the Fire Protection System pressure has returned to normal. Changes made.

37 F

3 N

S E

B, CR, 41.2 Change stem to: Which ONE of the following describes the individual inputs to the Core Subcooling Monitor, and the location(s) where these inputs may be monitored. Remove the word individual from all the answers. Changes made.

38 H

3 N

E B, CR, 41.4 Last bullet of the stem: change has failed to failed Change made.

39 H

3 E

B E

B, CR, 41.10 - With both EDGs supplying their respective busses, it is not a station blackout as required by the KA. Recommend changing stem so both EDGs are lost.

This may change the procedure that is referenced. OK as written.

40 H

3 B

S E

D, CR, 41.10 (North Anna 2008) Change in the stem: and includes the Basis to and the basis Change made.

41 H

3 M

S A, CR, 41.5 42 F

3 M

S B, CR, 41.10 43 H

3 U

M U

E A, CR, 41.10 - This is a KA mismatch because it does not test on a component or function associated with an uncontrolled depressurization of all steam generators.

This question is not unsat as it is describing the control of AFW components while

ES-401 WC-2009-08 4

Form ES-401-9 Page 4 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

/

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation excess cooldown is occurring during an uncontrolled depress of all SGs. Place a bullet in the stem for the status of total AFW flow. Change made.

44 H

3 B

S D, CR, 41.5 (VC Summer 2007) 45 F

3 B

S C, CR, 41.10 (VC Summer 2006) 46 H

3 N

S D, CR, 41.7 47 H

3 M

S A, CR, 41.10 Stem was simplified.

48 H

3 B

S A, CR, 41.10 (Robinson 2004) 49 H

3 N

S E

C, CR, 41.5 First bullet: change is to was. Remove the rest of the bullets. Add a bullet that states: A SGTR occurred in the A SG followed by a reactor trip 2 minutes later. Remove the first phrase of the question. It should start out: Which ONE Change Assume following transmitters to The following transmitters are in service. Simplified the question.

50 H

3 N

S D, CR, 41.9 Changed stem and answers.

51 H

3 N

S D, CR, 41.5 52 H

3 B

S B, CR, 41.5 (VC Summer 2007) 53 H

3 N

S A, CR, 41.10 54 F

3 M

S D, CR, 41.10 Rewrote question to fundamental.

55 H

4 U

N U

C, CR, 43.2 - This is an SRO only question because it is testing on the application of an LCO. Changed question and answers.

56 F

2 N

S A, CR, 41.10 57 F

4 B

S D, CR, 41.11 58 H

3 U

N U

S A, CR, 43.7 - This is an SRO question because it testing on the fuel handling equipment. Rewrite this question. Almost minutia. OFN used by ROs. OK as written.

59 F

4 B

S D, CR, 41.10 (Callaway 2005) Question was rewritten to fundamental.

60 H

3 M

S C, CR, 41.5 61 H

3 N

S D, CR, 41.11 62 F

3 N

S B, CR, 41.5

ES-401 WC-2009-08 5

Form ES-401-9 Page 5 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

/

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 63 F

3 B

S C, CR, 41.11 (Callaway 2007) 64 H

3 N

S D, CR, 41.5 65 F

3 B

S D, CR, 41.5 (Catawba 2004) Changed answers to change to fundamental.

66 H

2 N

S C, OR, 41.5 67 F

2 N

S C, CR, 41.10 68 F

2 N

S C, CR, 41.5 69 H

3 U

N U

S D, CR, 43.2 - This is an SRO question because it is testing on the application of TS.

OK as written.

70 H

3 N

S C, CR, 43.10 71 H

3 N

S E

A, CR, 41.11 Choice B, C and D: Remove the word reading. Changes made.

72 F

3 U

N U

D, CR, 41.4 - This is a KA mismatch because it does not test on a fire procedure.

Rewrite this question to test on general fire response in accordance with OFN KC-016. Stem and answers changed.

73 F

2 N

S A, CR, 41.10 74 F

3 N

S D, CR, 41.10 75 F

4 U

B U

A, CR, 41.10 - This is considered to be testing on minutia because it requires an attachment from an admin procedure be memorized. (North Anna 2008) Rewrite to test on general requirements contained in AI 15C-006. KA resampled (2.2.1).

Question rewritten.

76 H

3 N

S B, CR, 43.5 77 H

3 M

S B, OR, 43.5 Edited distractors A and C for clarity.

78 H

3 B

S C, CR, 43.5 (Sequoyah 2007) 79 H

3 N

S A, CR, 43.5 Changed the second bullet for clarity.

80 H

3 U

N U

S B, OR, 43.5 - This is testing on RO actions in the EOPs making it an RO question.

OK as written.

81 H

2 N

S B, CR, 43.5 82 H

3 E

B E

A, OR, 43.5 - There are insufficient references provided to evaluate this question.

ES-401 WC-2009-08 6

Form ES-401-9 Page 6 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

/

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation Also, this may be a direct lookup with the references provided to the candidate.

(Harris 2004) OK as written.

83 H

3 B

S C, CR, 43.5 (Harris 2008) 84 H

4 B

S D, CR, 43.5 (Beaver Valley 2007) 85 H

3 N

S A, CR, 43.5 86 H

3 B

S A, CR, 43.5 (WC 2007) Added Lpzr going up.

87 H

4 N

S A, CR, 43.5 Minor edits to the stem made.

88 H

3 N

S C, CR, 43.5 89 H

4 E

N E

D, CR, 43.5 - This question could not be technically evaluated due to a lack of references. The question KA is focused on notifications to external organizations but is made more difficult with a plant scenario that requires an evaluation using OFN attachments. This is a memory question with a 4 hour4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months NRC notification which we do not expect a candidate to know from memory. Recommend the question be rewritten. Providing Attachment E.

90 H

4 U

E B

U S

D, OR, 43.2 - This question provides the TS tables but not the LCO and the question requires the candidate to recall from memory a 4 hour4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months action statement. The TS basis is to limit secondary system pressure to 110% of SG design pressure in order to preclude a primary over pressurization. This means there is no correct answer.

Also, this is the third question on the exam with a KA involving G-2.1.25 (see Q66 and Q80) (McGuire 2008) Changed technical reference.

91 H

4 U

E N

U E

B, CR, 43.5 - This question tests the candidate on actions contained in two OFN attachments associated with an earthquake and therefore would not be expected to be memorized. Based on the conditions in the stem, it is more probable there is a leak in the circ water piping rather than a failure of the dam since vacuum is deceasing with a 2 inch decrease in lake level. (I.e., it is unlikely that a 2 inch lake decrease would result in decreasing vacuum.) Changed last bullet and distractor B.

92 F

3 E

E E

C, CR, 43.2 - This is a repeat KA (see Q84). From the documentation provided, it is not clear if this is the TS basis since the wording of the documentation provided describes the functional operation. These may not be the same. (Robinson 2007)

OK as written.

93 H

3 E

C, CR, 43.5 - The documentation provided did not allow for evaluating the technical accuracy of the question and/or distractors. Swapped values for last two bullets.

94 F

3 S

A, CR, 43.7 (Robinson 2007)

ES-401 WC-2009-08 7

Form ES-401-9 Page 7 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

/

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 95 F

2 U

N U

S A, CR, 41.10 - This is an RO question since it tests on the process to control annunciators which is a typical RO responsibility. OK as written.

96 F

3 E

N E

D, CR, 43.7 - The discriminating difference between distractors is the number of bolts required to be in the equipment hatch. Not sure it is a reasonable expectation the candidates have the number of bolts required be memorized. The docuemtation provided does not identify the source of the requirement (i.e., caution, prereq, etc.)

OK as written.

97 H

3 B

S B, CR, 43.5 (Diablo Canyon 2007) 98 F

2 U

N U

S C, CR, 41.10 - This is a question involving a routine procedure change which would make it RO knowledge. OK as written. Meets KA and JTA. ROs do not perform this procedure.

99 F

2 S

D, CR, 43.5 (Callaway 2007) 100 H

3 S

C, CR, 43.5 (McGuire 2007)

B= 29 F= 23 (31%)

E= 4 M= 7 H= 52 (69%)

U= 10 RO TOTALS:

N= 39 S= 61 Additional Notes:

B= 11 F= 6 (24%)

E= 5 M= 1 H= 19 (76%)

U= 5 SRO TOTALS:

N= 13 S= 15 Additional Notes:

GENERAL COMMENT

S:

1.

Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2.

Chief Examiner comments are indicated in blue.

3.

Average difficulty is __3.0 ___ on the RO exam and 3.0___ on the SRO exam. Both these values indicate the exam is more difficult than the average exam for this reviewer.

4.

The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0

/

43.1 = 0 41.2 = 4

/

43.2 = 2 41.3 = 0

/

43.3 = 0 41.4 = 9

/

43.4 = 0

ES-401 WC-2009-08 8

Form ES-401-9 Page 8 of 8 OBDI 202 - IOLE Process 41.5 = 13

/

43.5 = 19 41.6 = 3

/

43.6 = 0 41.7 = 17

/

43.7 = 2 41.8 = 0 41.9 = 2 41.10 = 22 41.11 = 4 41.12 = 0 41.13 = 1 41.14 = 0

5.

The answer distribution is: RO / SRO A = 15

/ 7 B = 20

/ 6 C = 17

/ 7 D = 23

/ 5

6.

There are 1 / 4 (RO/SRO) questions with attachments provided.

7.

This review was performed by S. Garchow the week of 7/20/2009.

8.

There are 33 questions on this exam from previous NRC exams; 22 on the RO exam and 11 on the SRO exam. Of these 33, three were used on the previous 2 exams at Wolf Creek.

9.

The RO exam contains 69% higher cognitive questions and this is above the 60% limit.

10.

The SRO exam contains 76% higher cognitive questions. This is significantly higher than the recommended limit of 60%.

11.

There are many questions, particularly on the SRO exam, that require the candidate to recall information from memory that this reviewer does not believe to be memory type information. In the reviewers opinion, this may lead to a high failure rate on the SRO Only portion of the exam based on the exam and not the candidates knowledge level. It is recommended this be discussed with the facilitys Operator Training Manager to ensure the memory questions match the facilitys training expectations and learning objectives.

12.

The vast majority of the questions on this exam are scenario based and many are unnecessarily complicated with regard to the selected KA. This is the primary reason the higher cognitive limits were exceeded.

13.

After final submittal, the following stats exist: UNSAT(RO/SRO) = 4 / 0. LOK H (RO/SRO) = 41 / 19. LOK F (RO/SRO) = 34 / 6.

ML093200397

Text

GENERAL COMMENT

Navigation menu

Search