ML090750879

2008-12-Draft Written Exam Comments

ML090750879
Person / Time
Site:	River Bend
Issue date:	11/21/2008
From:	NRC Region 4
To:
References
Download: ML090750879 (8)

Text

ES-401 RB-2008-12 RO Written Examination Review WorksheetRev 1 Form ES-401-9 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).

$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer

=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Attachment 8 Page 1 of 5 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia#/ units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 1N F 3 E B, CR, 41.5 Need reference that is not TS bases to be RO level question. Fixed 2N H 3 S B, CR, 41.7 3M F 3 S B, CR, 41.7 Need original question Done 4N H 2 S A, CR, 41.6 5B H 3 S A, CR, 41.7, N97 6B F 3 X U D, CR, 43.5 - It is SRO only because it requires the candidate to evaluate a given set of plant conditions and make a procedure selection based on this assessment. Submitted new question 7N H 3 S C, CR, 41.4 8N H 3 S D, CR, 41.4 9N F 2 S D, CR, 41.10 ES-401 RB-2008-12 2 Rev 1 Form ES-401-9 Attachment 8 Page 2 of 5 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 10N F 3 S C, CR, 41.10 11N F 3 S A, CR, 41.7 12N F 3 E B, CR, 41.4 - minor enhancements 13M H 3 S A, CR, 41.10, N03 need original question.

done 14N F 4 S B, CR, 41.7 15N F 3 X E A, CR, 41.8 - The experienced test taker would probably select distractor A if the answer were not known since it is the lowest of the four distractors. Suggest the first distractor be something less than 13 feet. Second, remove the "greater than" symbol from all four distractors since the minimum level is, in fact, 13 feet. revised 16B H 2 S A, CR, 41.7 17B H 3 S C, CR, 41.1, N99 - Need original question. Done - changed to Bank 18N H 3 S B, CR, 41.11 19N F 3 S B, CR, 41.4 20N H 3 S C, CR, 41.5 21N H 3 S D, CR, 41.4 22N H 3 S A, CR, 41.7 23N H 2 S D, CR, 41.4 24N F 3 X U C, CR, 41.10 - Revise to make RO level knowledge fixed 25N F 2 S D, CR, 41.6 26B F 2 E C, CR, 41.11 N07 - Current reading of 100 mr/hr is not "at" the max normal op value

as stated in distractor "c."

fixed 27N F 2 S C, CR, 4.7 28B H 3 E B, CR, 4.7, N00 - Distractor "b" contains a cue.

fixed 29N H 3 E A, CR, 41.7 - Reword stem for clarification done 30N H 3 E D, CR, 41.5 - Distractors don't answer the question.

Revised stem 31N F 2 S B, CR, 41.7 32B H 3 S B, CR, 41.7, N97 ES-401 RB-2008-12 3 Rev 1 Form ES-401-9 Attachment 8 Page 3 of 5 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 33N H 3 S D, CR, 41.7 34N H 2 S A, CR, 41.7 35N F 2 S B, CR, 41.10 36M H 3 E B, CR, 41.6, N97 - need original question done - minor enhancements 37M H 3 X U A, CR, 41.6, N00 -KA requires "procedures to correct/control/mitigate-" Also, need original question.

fixed 38N F 2 S D, CR, 41.6 39B H 3 E D, CR, 41.7 minor enhancements - removed cue from stem 40N F 3 U C, OR, 41.7 Direct lookup with reference - revised, removed ref 41N H 3 E C, CR, 41.7 - Both G33-F004 in the stem and G33-F054 in the correct distractor C have the same noun name. Revise noun names to include RWCU pump suction and

discharge valves.

fixed 42N H 3 S C, CR, 41.7 43N H 3 S B, CR, 41.3 44N H 2 S D, CR, 41.13 45B F 2 S D, CR, 41.7, N07 46B H 3 E C, CR, 41.4 N97 - Clarify distractors C & D. done 47N H 3 S C, CR, 41.4 48N F 3 X E C, CR, 41.4 - Need to revise distractor wording 49N F 3 S A, CR, 41.4 50N F 3 S B, CR, 41.4 51N H 3 X U C, CR, 41.7 - The stem asks about changes in reactive load or KVA. KVA is not reactive load; KVAR is reactive load.

fixed 52N H 3 E A, CR, 41.4 - KA requires "procedures to correct/control/mitigate-"

fixed 53B H 3 S B, CR, 41.4, N04 54N H 3 S B, CR, 41.10 55B H 3 E C, CR, 41.6, N99 - Delete the second part (i.e. following "due to") of the distractors ES-401 RB-2008-12 4 Rev 1 Form ES-401-9 Attachment 8 Page 4 of 5 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation because it provides unnecessary cuing.

Done 56N F 3 S C, CR, 41.7 57M H 3 E D, CR, 41.7, N03 - need original question done - minor enhancements to distractors 58N H 4 S A, CR, 41.5 59B F 4 X U A, CR, 43.7 - SRO only because it is testing on the refuel bridge and has no correlation to actions/activities in the control room.

Selected new KA and submitted new question 60N F 3 X X U A, CR, 41.10 - It could be argued there is more than one correct answer. It is correct that it is preferred to send the RPV energy to the main condenser to conserve the margin to the HCTL. It is also true that the reason for maintaining the margin to the HCTL is to preclude exceeding the PCPL. Therefore, distractors a and d could be argued to both be correct. Also, is EOP bases RO level knowledge?

Submitted new question 61N H 2 S A, CR, 41.4 62B F 3 S A, CR, 41.4 N97 63N F 3 S C, CR, 41.4 64B F 4 S D, CR, 41.4, N00 65N F 3 S B, CR, 41.6 66B F 2 X U D, CR, 41.5 - Doesn't test ability to use the plant computer. Submitted new question 67B F 2 X E C, CR, 41.10, N99 - Revise the question in the stem to "What is required following this evolution?"

fixed 68B F 4 S B, CR, 41.10 69N F 2 S C, CR, 41.10 - Need reference. Done 70N F 3 X E C, CR, 41.10 - Distractor d could be considered a subset of distractor c since both of them allow entry into the switchyard. Change distractors a and d to "-directly enter the switchyard..."

fixed 71N F 2 S B, CR, 41.10 72N F 2 S B, CR, 41.12 73B H 3 S D, CR, 41.7 N99 ES-401 RB-2008-12 5 Rev 1 Form ES-401-9 Attachment 8 Page 5 of 5 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 74B H 3 S B, CR, 41.10 N93 75N F 2 E B, CR, 43.7 - need learning objective done B= 18 F= 37 / 49%

E= 18 M= 6 H= 38 / 51%

U= 8 RO TOTALS:

N= 51 S= 49 Additional Notes:

B= F= E= M= H= U= SRO TOTALS:

N= S= Additional Notes:

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is 2.8.

4. The answer distribution is: RO / SRO A = 17 (23%) B = 22 (29%)

C = 20 (27%) D = 16 (21%)

5. There are zero RO questions with proposed references.

ES-401 RB-2008-12 SRO Written Examination Review WorksheetRev 1 Form ES-401-9 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).

$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer

=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Attachment 8 Page 1 of 3 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia#/ units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 76N H 3 S D, CR, 43.5 77N H 3 X UA, CR, 41.10 - Not SRO-only. Entry conditions to major EOP's and RPS trip setpoints is RO knowledge.

Note: add the word "to" between due and TSV in distracters b & d so they read correctly. submitted new question 78N H 3 X UB, CR, 41.10 - This is an RO question since there is no procedure selection (all the distractors reference the same procedure) and tests on AOP entry conditions and

specific operator actions. submitted new question 79N H 3 S B, CR, 43.5 80N H 3 E A, CR, 43.5 - minor enhancements made 81N F 3 X U B, CR, 41.5 - This is an RO question because it tests on operating limits, indications, locations of component controls. Resubmitted new question

ES-401 RB-2008-12 2 Rev 1 Form ES-401-9 Attachment 8 Page 2 of 3 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 82N H 3 S D, CR, 43.5 83N F 3 E B, CR, 43.2 minor enhancements made 84N H 3 E B, CR, 43.5 minor enhancements made 85N H 3 E A, OR, 43.5 enhancements made 86N H 2 X X X U C, OR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditi ons. KA requires "Predict impact of high supp pool temp and use procedures to correct/control/mitigate." Referenced procedures and eliminated handout 87N H 3 E D, CR, 43.5 enhancements made 88N H 3 E C, OR, 43.5 minor enhancements made 89N H 3 E B, CR, 43.5 enhancements made 90N F 2 X X X U A, CR, 41.10 - This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. It is a KA mismatch because it does not address the "predict the impact" part of the KA. It is graded as difficulty level 1 because it is intuitive the first action would be to attempt to start the pump that failed to auto start, and distracters c and d don't make sense. submitted new question 91N H 3 X UC, CR, 41.10 - This is an RO question because it tests on systems knowledge and

specific operator actions to be taken in response to a set of conditions. revised question with procedure entry in all distractors 92N H 4 S C, CR, 43.2 93N H 2 E B, CR, 43.5 minor enhancements made 94N F 3 X U D, OR, 41.10/43.1 - Direct lookup with handout - too hard without handout.

submitted new question 95N F 3 S C, CR, 43.7 96N F 2 S C, CR, 43.3 97N F 3 E C, CR, 43.4 minor enhancements made 98N F 3 E A, CR, 43.4/5 minor enhancements made 99N F 3 S C, CR, 43.5 100N F 3 X U C, CR, 41.10 -This is an RO question because it tests on specific operator actions to be taken in response to a set of conditions. revised question with procedure entry in ES-401 RB-2008-12 3 Rev 1 Form ES-401-9 Attachment 8 Page 3 of 3 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation all distractors B= F= E= M= H= U= RO TOTALS:

N= S= Additional Notes:

B= 00 F= 10 E= 10 M= 00 H= 15 U= 8 SRO TOTALS:

N= 25 S= 7 Additional Notes:

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is 2.8.

4. The answer distribution is: A = 5 (20%) B = 7 (28%) C = 9 (36%)

D = 4 (16%)

5. There one question with proposed references.

6. CR = Closed Reference OR = Open Reference