ML090790342

From kanterella
Jump to navigation Jump to search
2009-02-Draft Written Exam Comments
ML090790342
Person / Time
Site: Diablo Canyon  Pacific Gas & Electric icon.png
Issue date: 02/06/2009
From: Brian Larson
Operations Branch IV
To:
Pacific Gas & Electric Co
References
50-275/09-301, 50-323/09-301
Download: ML090790342 (8)
v • d • e


Text

ES-401 DC-2009-02 RO Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 B S C, CR, 41.7 2 F 2 N S A, CR, 41.5 3 F 2 N S A, CR, 41.7 4 F 2 N S B, CR, 41.5 5 F 3 N S C, CR, 41.8 6 H 3 B S C, CR, 41.7 7 F 3 N S C, CR, 41.7 D, CR, 41.10 - KA requires testing on the impact and use of procedures and the 8 H 3 X N U question tests on operator action. Reword stem and distractors to read as operator actions and include part (a) of the KA. Revised 9 H 3 N S A, CR, 41.7 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 5 OBDI 202 - IOLE Process

ES-401 DC-2009-02 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 10 H 3 N S A, CR, 41.7 11 H 3 N S A, CR, 41.2 12 H 3 B S A, CR, 41.7 13 F 3 B S D, CR, 41.7 14 H 3 M S B, CR, 41.10 15 H 3 N S C, CR, 41.10 16 F 3 B S A, CR, 41.8 17 H 3 N S C, CR, 41.7 18 H 3 X B U B, CR, 41.5 - KA states power change, not steady state. Revised question 19 H 3 M E C, CR, 41.5 - Reword stem to clarify location of break Fixed 20 F 3 X N E A, CR, 41.7 - Add the TDAFW pump is running to the stem. Fixed B, CR, 41.7 - The correct distractor is unique and could be a cue; add a different 21 H 3 X B E battery as a distractor. Fixed B, CR, 41.7 - Distractor A could be argued as being correct since 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> is a subset of 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> (i.e. if it is designed to last 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />, then it is by definition designed to last 1 22 F 3 X B E hour). Chang stem to read: During a loss of all AC, what is the design lifetime of a vital 125 VDC battery during a design basis accident coincident with a loss of battery charger? Fixed A, CR, 41.7 - This question does not match the KA because it does not address the 23 H 3 X B U automatic loader as the KA requires. Replaced question 24 H 2 X N E A, CR, 41.11 - Add fail high to distractors B and D for clarification Fixed B. CR, 41.10 - Having b in all distractors eliminates it from being discriminatory.

25 F 3 M E Revised 26 H 3 N S A, CR, 41.7 27 H 2 N S A, CR, 41.4 28 F 3 N S B, CR, 41.7 29 F 3 B S D, CR, 41.6 30 F 2 N S B, CR, 41.7 Page 2 of 5 OBDI 202 - IOLE Process

ES-401 DC-2009-02 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 31 F 3 N S D, CR, 41.6 32 F 3 N S B, CR, 41.5 A, CR, 41.7 - Consider replacing KA to enable a better question/KA match.

33 H 3 X M E Replaced KA 34 F 3 M E C, CR, 41.11 - Reword A and add BOTH to B Done 35 H 3 B S B, CR, 41.4 Bank, not Modified Bank Updated C, CR, 41.7 - Does not focus on the operating controls for the ventilation system. It 36 H 3 X N U may be necessary to select a new KA. Replaced KA 37 F 3 B S B, CR, 41.4 38 F 3 B S A, CR, 41.4 39 H 3 N S D, CR, 41.5 40 H 3 N S B, CR, 41.5 41 H 3 N S C, CR, 41.7 D, CR, 41.10 - Does not address the locate and determine the indications are 42 H 3 X B U correct as required by the KA. Revised question 43 H 3 N S C, CR, 41.7 44 F 2 N S D, CR, 41.7 45 H 3 N S C, CR, 41.7 46 H 3 N S B, CR, 41.6 47 H 3 N S D, OR, 41.5 48 F 3 M E C, CR, 41.7 - Capitalize the word MINIMUM Done 49 F 3 N S A, CR, 41.4 - Confirm there is no conflict with In-Plant JPM Minor rev to JPM 50 H 3 N S D, CR, 41.10 51 H 3 N E D, CR, 41.7 - Add AOP to Tech Ref to show link from AOP to OP. Done 52 H 3 M S C, CR, 41.7 - Need a complete copy of the BANK question Provided 53 F 3 N S D, CR, 41.7 Page 3 of 5 OBDI 202 - IOLE Process

ES-401 DC-2009-02 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 54 H 3 N S D, CR, 41.4 - Enhance explanations Done 55 H 3 M U C, CR, 41.10 - This is JPM S3. Replace question. Done 56 F 3 B S A, CR, 41.10 57 F 3 B S B, CR, 41.10 - BANK, not MOD BANK Changed 58 H 3 N S B, CR, 41.6 59 H 3 N S B, CR, 41.2 60 H 3 X B E C, CR, 41.10 - Remove greater than statements. Fixed 61 H 4 B E D, CR, 43.2 - Add OP as Tech Ref. (DCPP 2007 RO Exam) Done D, CR, 41.10 - Is this a valid from memory question? SRO only? Licensee confirms 62 H 4 X X B E this is RO level knowledge.

63 F 3 X M E C, CR, 41.10 - Clarify difference between second and third bullets. Revised 64 F 3 B S B, CR, 41.10 C, CR, 41.10 - Answer A may be slow, but is it available? May need to add to stem 65 H 3 X B E Per FR-H.2, which is ? Fixed 66 F 3 X B E C, CR, 41.10 - Change Fire Fighters to Fire Department. Done 67 F 3 B S B, CR, 41.10 68 F 2 N S D, CR, 41.10 69 F 2 N S A, CR, 41.10 B, CR, 41.10 - Clarify in stem that tags are being hung by Operations personnel.

70 F 2 N E Done 71 F 2 B S D, CR, 41.10 72 F 2 M S B, CR, 41.10 73 H 3 N S D, CR, 41.5 74 F 3 N S A, CR, 41.10 75 H 2 N S B, CR, 41.10 Page 4 of 5 OBDI 202 - IOLE Process

ES-401 DC-2009-02 5 Form ES-401-9 B= 24 (32%) F= 34 (45%) E= 16 (21%)

RO TOTALS: M= 10 (13%) H= 41 (55%) U= 6 (8%)

N= 40 (55%) S= 53 (71%)

SRO TOTALS:

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
2. Chief Examiner comments are indicated in blue.
3. Average difficulty is _2.85 on the RO exam.
4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0 41.2 = 3 41.3 = 0 41.4 = 5 41.5 = 13 41.6 = 4 41.7 = 24 41.8 = 2 41.9 = 0 41.10 = 22 41.11 = 2 41.12 =

41.13 =

41.14 =

5. The answer distribution is: RO A = 18 (24%) /

B = 21 (28%) /

C = 19 (25%) /

D = 17 (23%) /

6. There is (are) 1 / question(s) with attachments provided (steam tables). Page 5 of 5 OBDI 202 - IOLE Process

ES-401 DC-2009-02 SRO Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 M S A, OR, 43.5 Hand out B, OR, 43.2 - Do not provide TS 3.0.3 because the applicants should know from 2 H 3 X N E memory the TS motherhood statements. Done Hand out D, CR, 43.5 - Add a reference that defines Go To and Refer To. Also add 3 H 3 M S reference that defines applicability of OP SD-1 in Modes 5 & 6. Done 4 H 3 N S B, CR, 43.5 5 H 3 B S D, CR, 43.5 6 H 3 N S C, CR, 43.5 7 F 2 B S C, CR, 43.2 - Revise Answer A explanation. Done 8 H 3 N S D, CR, 43.5 - looks like a new question, not a modified bank. Identified as NEW 9 H 3 N S D, CR, 43.5 - Revise Answer C explanation. Done Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 3 OBDI 202 - IOLE Process

ES-401 DC-2009-02 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 10 H 3 B S A, CR, 43.5 11 H 3 N E C, CR, 43.5 - Add procedure refs to Answers A and D. Done A, CR, 43.5 - Does not address the predict the impact part of the KA. Answer D 12 H 3 X X N U may not be plausible. Enhanced answers; modified D D, OR, 43.2 - Change stem to read taken to allow entering MODE 2? instead of taken to allow the startup to continue? then make necessary changes to all 13 H 3 X B U answers. Also, it could be argued that completing the actions contained in B would allow the startup to continue and therefore there are two correct answers. Revised Hand out.

D, OR, 43.2 - Dont include P&IDs as a handout. Done Explanations for B and D are 14 H 4 N E not clear. Revised Hand out.

A, CR, 43.5 - Answer B not plausible. Explanations not clear. Does not address the 15 H 3 X X N U predict the impact part of the KA. Revised D, CR, 43.7 - Reword stem to ask what indications the Refuel SRO would use.

16 F 3 N E Revised A, CR, 43.5 - This is considered an RO question in that it is testing on the entry 17 H 3 X N U conditions for an AP. Revised 18 H 3 N S D, CR, 43.5 C, CR, 43.2 - This is an RO question in that it is testing on basic rad worker 19 F 2 X N U knowledge. Replaced KA 20 H 2 M E A, CR, 43.5 - Similar to JPM S1. Select new KA if necessary. Revised with same KA D, CR, 43.3 - Answer A is not plausible. Replaced. Explanation for Answer B is not 21 F 2 X N U clear. Revised 22 F 2 N E A, CR, 43.4 - revise explanations to address both parts of answers Done 23 H 3 N S A, CR, 43.4 24 H 3 N E D, CR, 43.5 - Revise stem to four TECs and revise correct answer. Done 25 F 3 N E A, CR, 43.3 - Consider new KA. Replaced B= 4 (16%) F= 6 (24%) E= 8 (32%) Additional Notes: The exam review indicates greater than 60% comprehension level SRO TOTALS: M= 3 (12%) H= 19 (76%) U= 6 (24%) questions. Chief Examiner has discussed with licensee and both determined this as acceptable given randomly selected KAs.

N= 18 (72%) S= 11 (44%) Page 2 of 3 OBDI 202 - IOLE Process

ES-401 DC-2009-02 3 Form ES-401-9

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.
2. Chief Examiner comments are indicated in blue.
3. Average difficulty is 2.84 on the SRO exam.
4. The 10CFR55.41/43 distribution is:

43.1 = 0 43.2 = 5 43.3 = 2 43.4 = 2 43.5 = 15 43.6 = 0 43.7 = 1

5. The answer distribution is:

A = 7 (28%)

B = 6 (24%)

C = 4 (16%)

D = 8 (32%)

6. There are 4 questions with attachments provided. Make sure hand outs dont provide cues to any other question. Page 3 of 3 OBDI 202 - IOLE Process
Retrieved from "https://kanterella.com/w/index.php?title=ML090790342&oldid=2147833"