Text

2008-12-Draft Written Exam Comments
	ML090750879
Person / Time
Site:	River Bend
Issue date:	11/21/2008
From:	; NRC Region 4
To:	;
References
	Download: ML090750879 (8)
	v • d • e

ES-401 RB-2008-12 RO Written Examination Review Worksheet Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1N F 3 E B, CR, 41.5 Need reference that is not TS bases to be RO level question. Fixed 2N H 3 S B, CR, 41.7 3M F 3 S B, CR, 41.7 Need original question Done 4N H 2 S A, CR, 41.6 5B H 3 S A, CR, 41.7, N97 D, CR, 43.5 - It is SRO only because it requires the candidate to evaluate a given set 6B F 3 X U of plant conditions and make a procedure selection based on this assessment.

Submitted new question 7N H 3 S C, CR, 41.4 8N H 3 S D, CR, 41.4 9N F 2 S D, CR, 41.10 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 2 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 10N F 3 S C, CR, 41.10 11N F 3 S A, CR, 41.7 12N F 3 E B, CR, 41.4 - minor enhancements 13M H 3 S A, CR, 41.10, N03 need original question. done 14N F 4 S B, CR, 41.7 A, CR, 41.8 - The experienced test taker would probably select distractor A if the answer were not known since it is the lowest of the four distractors. Suggest the first 15N F 3 X E distractor be something less than 13 feet. Second, remove the greater than symbol from all four distractors since the minimum level is, in fact, 13 feet. revised 16B H 2 S A, CR, 41.7 17B H 3 S C, CR, 41.1, N99 - Need original question. Done - changed to Bank 18N H 3 S B, CR, 41.11 19N F 3 S B, CR, 41.4 20N H 3 S C, CR, 41.5 21N H 3 S D, CR, 41.4 22N H 3 S A, CR, 41.7 23N H 2 S D, CR, 41.4 24N F 3 X U C, CR, 41.10 - Revise to make RO level knowledge fixed 25N F 2 S D, CR, 41.6 C, CR, 41.11 N07 - Current reading of 100 mr/hr is not at the max normal op value 26B F 2 E as stated in distractor c. fixed 27N F 2 S C, CR, 4.7 28B H 3 E B, CR, 4.7, N00 - Distractor b contains a cue. fixed 29N H 3 E A, CR, 41.7 - Reword stem for clarification done 30N H 3 E D, CR, 41.5 - Distractors dont answer the question. Revised stem 31N F 2 S B, CR, 41.7 32B H 3 S B, CR, 41.7, N97 Page 2 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 3 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 33N H 3 S D, CR, 41.7 34N H 2 S A, CR, 41.7 35N F 2 S B, CR, 41.10 36M H 3 E B, CR, 41.6, N97 - need original question done - minor enhancements A, CR, 41.6, N00 - KA requires procedures to correct/control/mitigate Also, need 37M H 3 X U original question. fixed 38N F 2 S D, CR, 41.6 39B H 3 E D, CR, 41.7 minor enhancements - removed cue from stem 40N F 3 U C, OR, 41.7 Direct lookup with reference - revised, removed ref C, CR, 41.7 - Both G33-F004 in the stem and G33-F054 in the correct distractor C 41N H 3 E have the same noun name. Revise noun names to include RWCU pump suction and discharge valves. fixed 42N H 3 S C, CR, 41.7 43N H 3 S B, CR, 41.3 44N H 2 S D, CR, 41.13 45B F 2 S D, CR, 41.7, N07 46B H 3 E C, CR, 41.4 N97 - Clarify distractors C & D. done 47N H 3 S C, CR, 41.4 48N F 3 X E C, CR, 41.4 - Need to revise distractor wording 49N F 3 S A, CR, 41.4 50N F 3 S B, CR, 41.4 C, CR, 41.7 - The stem asks about changes in reactive load or KVA. KVA is not 51N H 3 X U reactive load; KVAR is reactive load. fixed 52N H 3 E A, CR, 41.4 - KA requires procedures to correct/control/mitigate fixed 53B H 3 S B, CR, 41.4, N04 54N H 3 S B, CR, 41.10 55B H 3 E C, CR, 41.6, N99 - Delete the second part (i.e. following due to) of the distractors Page 3 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 4 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only because it provides unnecessary cuing. Done 56N F 3 S C, CR, 41.7 57M H 3 E D, CR, 41.7, N03 - need original question done - minor enhancements to distractors 58N H 4 S A, CR, 41.5 A, CR, 43.7 - SRO only because it is testing on the refuel bridge and has no 59B F 4 X U correlation to actions/activities in the control room. Selected new KA and submitted new question A, CR, 41.10 - It could be argued there is more than one correct answer. It is correct that it is preferred to send the RPV energy to the main condenser to conserve the margin to the HCTL. It is also true that the reason for maintaining the margin to 60N F 3 X X U the HCTL is to preclude exceeding the PCPL. Therefore, distractors a and d could be argued to both be correct. Also, is EOP bases RO level knowledge? Submitted new question 61N H 2 S A, CR, 41.4 62B F 3 S A, CR, 41.4 N97 63N F 3 S C, CR, 41.4 64B F 4 S D, CR, 41.4, N00 65N F 3 S B, CR, 41.6 66B F 2 X U D, CR, 41.5 - Doesnt test ability to use the plant computer. Submitted new question C, CR, 41.10, N99 - Revise the question in the stem to What is required following 67B F 2 X E this evolution? fixed 68B F 4 S B, CR, 41.10 69N F 2 S C, CR, 41.10 - Need reference. Done C, CR, 41.10 - Distractor d could be considered a subset of distractor c since both of 70N F 3 X E them allow entry into the switchyard. Change distractors a and d to directly enter the switchyard... fixed 71N F 2 S B, CR, 41.10 72N F 2 S B, CR, 41.12 73B H 3 S D, CR, 41.7 N99 Page 4 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 5 Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 74B H 3 S B, CR, 41.10 N93 75N F 2 E B, CR, 43.7 - need learning objective done B= 18 F= 37 / 49% E= 18 RO TOTALS: M= 6 H= 38 / 51% U= 8 Additional Notes:

N= 51 S= 49 B= F= E=

SRO TOTALS: M= H= U= Additional Notes:

N= S=

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is 2.8.

4. The answer distribution is: RO / SRO A = 17 (23%)

B = 22 (29%)

C = 20 (27%)

D = 16 (21%)

5. There are zero RO questions with proposed references. Page 5 of 5 OBDI 202 - IOLE Process

ES-401 RB-2008-12 SRO Written Examination Review Worksheet Rev 1 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 76N H 3 S D, CR, 43.5 A, CR, 41.10 - Not SRO-only. Entry conditions to major EOPs and RPS trip setpoints is RO knowledge.

77N H 3 X U Note: add the word to between due and TSV in distracters b & d so they read correctly. submitted new question B, CR, 41.10 - This is an RO question since there is no procedure selection (all the 78N H 3 X U distractors reference the same procedure) and tests on AOP entry conditions and specific operator actions. submitted new question 79N H 3 S B, CR, 43.5 80N H 3 E A, CR, 43.5 - minor enhancements made B, CR, 41.5 - This is an RO question because it tests on operating limits, indications, 81N F 3 X U locations of component controls. Resubmitted new question Instructions