ML113000289

2011-07-DRAFT Written Exam Comments

ML113000289
Person / Time
Site:	Diablo Canyon
Issue date:	10/27/2011
From:	NRC Region 4
To:	Pacific Gas & Electric Co
References
50-275/11-07, 50-323/11-07
Download: ML113000289 (7)

Text

ES-401 DC-2011-07 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 2 B S C, CR, 41.6 B, CR, 41.7 - This is a KA mismatch because even though the question references a U small break LOCA in the stem, it is irrelevant for answering the question. Pump flow 2 H 2 U B will increase with or without a SB LOCA when the recirc (min-flow) valve is closed.

S The CE discussed this one with the licensee. This is the best fit for the KA. OK as is.

3 H 3 B S A, CR, 41.5 B, CR, 41.10 - Recommend adding the word Immediately to the beginning of 4 F 3 M E distractors A and B. Agree. Fixed 5 F 3 B S D, CR, 41.5 6 F 4 N S B, CR, 41.7 Modified stem and answers for operational validity 7 H 3 B S B, CR, 41.7 8 F 3 B S D, CR, 41.11 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process

ES-401 DC-2011-07 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only C, CR, 41.10 - There is a typo in the second bullet in the stem. Change Fleed to 9 F 3 M S feed. Fixed, and modified A and B for validity.

10 H 3 N S A, CR, 41.7 D, CR, 41.7 - Even though the DG is in the droop mode of operation, typically raising the governor will result in raising the speed governors null point thereby raising engine speed and generator frequency. Check to ensure this is not the case at DC.

11 H 3 B S If the question is correct as stated, recommend changing distractor A to UP, SAME, SAME, UP and distractor B to SAME, UP, SAME, SAME to make them more credible. Fixed. Removed one of the columns.

12 F 4 N S C, CR, 41.7 Changed the question to PY-12 13 H 3 N S C, CR, 41.7 14 H 3 N S C, CR, 41.8 Shifted distracters for balance 15 H 3 B S A, CR, 41.4 C, CR, 41.4 - The stem of the question states the frequency oscillates and the correct answer describes the response of the voltage regulator. Technically, if there E is any response, it would be by the turbine governor valves rather than the voltage 16 H 3 E N regulator. Recommend changing the stem so that there is a small voltage S oscillation rather than a small frequency oscillation. Otherwise, it could be argued there is no correct answer or even distractor D is correct. OK as is. Shifted answer for balance.

17 H 3 N S A, CR, 41.10 18 H 2 B S C, OR, 41.10 19 H 3 B S C, CR, 41.5 20 F 3 N S B, CR, 41.4 21 H 3 M S D, CR, 41.14 22 H 3 N S C, CR, 41.13 23 F 2 N S C, CR, 41.9 24 F 3 B S C, CR, 41.10 25 F 3 N S C, CR, 41.7 26 F 3 B S C, CR, 41.2 Page 2 of 7 OBDI 202 - IOLE Process

ES-401 DC-2011-07 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 27 H 3 B S A, CR, 41.11 28 F 2 N S A, CR, 41.7 29 H 3 B S C, CR, 41.5 30 H 3 B S D, CR, 41.7 31 H 3 N S D, CR, 41.7 32 H 3 N S B, CR, 41.5 33 H 3 B S C, CR, 41.5 34 H 3 N S C, CR, 41.7 35 H 3 M S D, CR, 41.7 36 F 3 N S B, CR, 41.6 37 H 3 N S B, CR, 41.7 38 F 3 N S B, CR, 41.7 39 H 3 B S A, CR, 41.8 Changed stem for validity 40 F 2 N S A, CR, 41.7 41 F 4 B S C, CR, 41.1 42 H 3 N S C, CR, 41.7 U C, OR, 41.2 - This is an SRO question since it is testing on the application of a TS.

43 H 1 U N S Also, it is a direct lookup. Agree. Replace this question. Replaced.

44 H 2 M S B, CR, 41.5 45 H 3 B S C, CR, 41.7 46 H 3 N S B, CR, 41.7 C, CR, 41.7 - This is a KA mismatch because the question does not address the 47 H 3 B S effects on instrumentation and controls of switching power supplies. OK as is.

48 H 3 N S B, CR, 41.7 Question was modified for validity after validation.

49 H 3 B S D, CR, 41.7 50 F 3 B S D, CR, 41.4 Page 3 of 7 OBDI 202 - IOLE Process

ES-401 DC-2011-07 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 51 F 3 B S B, CR, 41.11 D, CR, 41.7 - The explanation for distractor C makes that distractor also correct. I E believe the explanation needs corrected.

52 H 3 E E E N S A - not plausible because stem states ASW1-2 is in AUTO. C is also correct because it causes D. Fixed.

53 H 3 N S D, CR, 41.4 54 F 3 N S C, CR, 41.7 D, CR, 41.9 - Distractor B is confusing; the explanation states it is incorrect because it is auctioneered high which seems to suggest it can be displayed if 55 F 3 E N S desired. If the highest reading can be selected, it could be reasonably argued this would be monitored since it is the most conservative. CAF. Question OK as is. The average temperature is displayed on the recorder.

56 F 3 N S A, CR, 41.3 57 H 3 N S D, CR, 41.10 D edited for clarity.

58 F 3 B S B, CR, 41.2 B

59 F 3 S C, CR, 41.9 Replaced Q because too many on last NRC.

N 60 H 3 N S B, CR, 41.9 61 F 2 N S D, CR, 41.10 62 H 4 M S A, CR, 41.11 63 H 3 B S B, CR, 41.11 64 F 2 N S A, CR, 41.7 65 F 3 N S A, CR, 41.4 66 F 2 B S C, CR, 41.10 A, CR, 41.10 - Suggest the third sentence in stem of the question be revised to read 67 F 2 B S According to procedure OP1.DC10, as a minimum Fixed 68 H 4 N S C, OR, 41.1 69 F 3 N S A, CR, 41.3 Page 4 of 7 OBDI 202 - IOLE Process

ES-401 DC-2011-07 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 70 F 3 N S A, CR, 41.10 71 F 2 N S B, CR, 41.12 Replaced with new. Too close to prior NRC 72 F 3 N S C, CR, 41.12 Replaced with closer tie to KA B, CR, 41.10 Discussed with licensee about similarity with 74. Asking two different 73 F 2 N S pieces of information..

74 F 2 B S C, CR, 41.10 75 F 3 N S B, CR, 41.10 76 H 3 N S B, CR, 43.5 77 F 3 B S B, CR, 43.2 78 H 3 N S A, CR, 43.5 79 H 2 N S D, CR, 43.5 80 H 3 N S C, CR, 43.5 81 H 3 N S B, OR, 43.5 82 H 3 N S D, OR, 43.5 83 H 3 N S D, CR, 43.5 84 H 3 B S D, CR, 43.5 85 H 3 N S C, CR, 43.5 86 H 3 B S C, CR, 43.5 87 H 2 N S A, CR, 43.5 88 H 3 N S A, CR, 43.5 89 H 3 B S B, CR, 43.5 E A, OR, 43.2 - As written, all 4 distractors are correct. Suggest changing the stem to 90 H 3 E N read The technical specifications direct the Core Spray Pump 1-2 be returned to an S operable status no later than:. DONE 91 H 3 N S D, CR, 43.5 92 H 3 N S C, CR, 43.5 Page 5 of 7 OBDI 202 - IOLE Process

ES-401 DC-2011-07 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 93 H 3 N S B, OR, 43.2 Changed to prevent cascading tech specs.

94 F 2 N S C, CR, 43.2 Replaced Q due to prior NRC 95 F 2 B S B, CR, 43.7 96 F 4 N S A, CR, 43.2 97 H 3 N S A, OR, 43.2 98 F 3 N S A, OR, 43.4 Replaced Q due to last NRC 99 H 3 B S B, CR, 43.5 100 H 3 N S B, CR, 43.5 B= 29 F= 36 E= 4 RO TOTALS: M= 7 H= 39 U= 7 Additional Notes:

N= 39 B= 7 F= 5 E= 2 SRO TOTALS: M= 1 H= 20 U= 0 Additional Notes:

N= 17

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. This review was performed by S. Garchow on May 16-17, 2011.

3. Chief Examiner comments are indicated in blue.

4. Average difficulty is __2.84_ on the RO exam and 2.88 on the SRO exam. Both these values are within the expected range for this reviewer.

5. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 2 43.1 = 0 41.2 = 3 43.2 = 6 41.3 = 2 43.3 = 0 41.4 = 5 43.4 = 1 41.5 = 7 43.5 = 17 41.6 = 2 43.6 = 0 41.7 = 24 43.7 = 1 Page 6 of 7 OBDI 202 - IOLE Process

ES-401 DC-2011-07 7 Form ES-401-9 41.8 = 2 41.9 = 4 41.10 = 13 41.11 = 5 41.12 = 2 41.13 = 1 41.14 = 1

5. The answer distribution is: RO / SRO A = 15 / 7 B = 19 / 8 C = 27 / 5 D = 14 / 5

6. There are 3 / 6 questions with attachments provided.

7. The SRO Only portion of the exam is skewed toward the 43.5, Procedure Selection, type questions with 68% of the questions. Consideration should be given to replacing some of these with different 10CFR43 topics.

8. The RO exam is skewed toward distractor C as being correct with 27 questions.

Changes made to exam submittal:

1. 39 - added to question is required to be taken

1. 48 - added to question is required to be taken

1. 52 - deleted extra space in A and s to pump in distractor B (any ASW pumps)

General change - changed references to diesel generators to D/G when abbreviated. Based on DCPP EOP Writers Guide. Page 7 of 7 OBDI 202 - IOLE Process