ML12325A403
| ML12325A403 | |
| Person / Time | |
|---|---|
| Site: | Fort Calhoun  |
| Issue date: | 09/21/2012 |
| From: | Vincent Gaddy Operations Branch IV |
| To: | Omaha Public Power District |
| Laura Hurley | |
| References | |
| ES-401, ES-401-9 50-285/12-009 | |
| Download: ML12325A403 (7) | |
Text
ES-401 FC-2012-09 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 2 B E B(41.10) Editorialreplace attempted with attempts. Fixed D(41.10) Give the applicant a bullet detailing exactly what subcooling is reading in the control room. This is an indication they would have. Also, distracter B really 2 H 3 N E does not seem plausible to stop all charging pumps. Revise this distracter.
Consider making PZR level 60% and remove rapidly from bullet. Make action plural - What actions should be taken Fixed 3 H 3 B S C(41.5) 4 F 3 B S C(41.10) 2005 NRC EXAM 5 F 2 B S D(41.3)
B(41.7) Distracter A stands out as significantly different than the correct answer or 6 H 2 N E the other distracters. This is psychometrically flawed. Also, plausibility is suspect for a pump to start on an alarm rather than some level deviation. Recommend changing distracter D as well due to significant difference to the other distracters numerically Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process
ES-401 FC-2012-09 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only speaking. Make it closer to other distracters. If CCP starts at -2.9%, it will start at -
3.3% and -6%: more than one correct answer. Change question so that only -2.9%
will be correct. Fixed 7 H 1 N E B(41.10) HANDOUT Simple graph exercise. LOD=1 Fixed D(41.7) Consider putting The Control switch for HCV-400A/C must be in the stem 8 F 2 B E and out of all the answers. Fixed D(41.7) This is a two part answer. Must know probable cause of the failure AND the 9 H 3 M E action to be taken. Only need to know one part in order to answer the question correctly. Fixed 10 H 3 B S A(41.1) 2007 NRC EXAM 11 H 3 B S B(41.11)
A(41.10) Two part answer. Only need to know one part to answer the question correctly. Suggest making this a two by two. Stem - What action is directed, not 12 H 3 N E actions are - only one action per answer. Also, should be and why is this action directed? Fixed 13 F 3 N S C(41.10) 14 H 3 M S B(41.8)
B(41.10) 2009 NRC EXAM Put An additional vital instrument bus failure will result 15 H 3 B E in in the stem and out of the answer. Fixed 16 H 3 N S C(41.10) 17 H 3 B S B(41.7) 2005 NRC EXAM A(41.10) Provide another bullet that states 4160V buses indicate whatever is expected under these conditions. This is an indication an applicant would have and 18 F 3 N E would eliminate C and D as correct answers if the applicant is knowledgeable of the procedure. Fixed 19 H 2 N S B(41.1) 20 H 2 M S A(41.1) 2001/2002 NRC EXAM C(41.7) 1999 NRC EXAM Three part answer - only need to know two parts to 21 H 3 B E answer correctly. Fixed 22 F 2 N S B(41.8) Page 2 of 7 OBDI 202 - IOLE Process
ES-401 FC-2012-09 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 23 F 2 X N U B(41.1) 1995 NRC EXAM K/A requires knowledge of the reason Fixed 24 F 2 X N U C(41.13) 2002 NRC EXAM K/A requires knowledge of the reason Fixed 25 H 3 M S A(41.10) 26 F 2 B S A(41.11) 2004 NRC EXAM 27 H 2 B S A(41.10) 2005 NRC EXAM 28 H 3 B S C(41.3) 1997 NRC EXAM 29 H 2 N S B(41.7) 30 F 3 M S B(41.5) 2001/2 NRC EXAM D(41.10) K/A requires knowledge of EOP entry conditions and immediate actions.
31 H 2 X N U Fixed D(41.7) 2005 NRC EXAM Construct a scenario around this question. As written it is 32 F 2 B E pure memorization. Needs to be operationally based/valid. Fixed C(41.10) 2009 NRC EXAM Distracter D more plausible if you trip one LPSI pump and not all the running LPSI pumps. D would be true when SIRWT level gets low 33 H 3 B E enough - enhance stem to avoid D being true. A change minimize to reduce.
Fixed 34 F 2 N S A(41.10)
B(41.7) Same as question 32. Construct a scenario to test this knowledge. HCV-35 H 2 M E 492A/B - is this for HX AC-1D? Should it be for HCV-491A/B? Fixed 36 H 3 N S A(41.7) 37 F 2 B S D(41.10) 2009 NRC EXAM 38 H 3 B S C(41.6) 2009 NRC EXAM 39 H 3 B S C(41.8) 40 H 2 B S C(41.10) HANDOUT 41 F 2 N S D(41.5) 42 H 3 B S B(41.5) 2004 NRC EXAM 43 H 2 N S B(41.4) Page 3 of 7 OBDI 202 - IOLE Process
ES-401 FC-2012-09 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 44 F 3 M E C(41.1) Question mark, not period, at end of stem. Fixed 45 F 2 N S A(41.7) 46 F 2 N S B(41.4) 47 F 2 N S B(41.7)
B(41.7) 2001/2 NRC EXAM Develop operational scenario to test this knowledge.
48 F 2 B E Fixed C(41.8) 2007 NRC EXAM Stopping LO-4 is technically correct per Attachment 6 of the procedure. Ask what action is first taken, per the procedure, and why. This will 49 H 3 M E eliminate any potential second correct answers. B and D - LO-12B is the Emergency Seal Oil Pump per the ref, not DC Seal Oil Pump. Fixed 50 F 2 N S C(41.5) 51 H 3 N S B(41.11) 52 F 2 N E A(41.12) Need question mark at end of stem. Fixed 53 F 2 N E B(41.7) 2005 NRC EXAM Simple memorization. Develop scenario question. Fixed D(41.10) 2007 NRC EXAM Add bullets to annunciators in stem. Capitalize raw 54 H 3 B E water in C. Fixed 55 F 2 B S C(41.5) 1995 NRC EXAM 56 F 2 B S A(41.3)
D(41.3) Are all the MCCs 480V? I dont want to cue that in the stem if thats not the 57 F 2 N E case. I want the applicant to have to understand they are powered by 480VAC and pick the correct P/S from a group that may contain non-480V power supplies. Fixed 58 H 2 B S B(41.5) 2009 NRC EXAM, HANDOUT 59 F 3 N E D(41.2) How does wiring fail open? Consider The electrical circuit for one 60 H 3 N S D(41.10) 61 F 2 N E B(41.7) K/A is predict/monitor changes in Containment Pressure. Fixed 62 F 3 B S D(41.5) 2004 NRC EXAM 63 H 3 M S A(41.4) 64 H 3 N S D(41.12) Page 4 of 7 OBDI 202 - IOLE Process
ES-401 FC-2012-09 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only D(41.4) Memorization. Develop scenario so that the applicant has to analyze 65 F 2 M E whether or not the valves are open/closed based on certain plant conditions. AOP-17 states I A system < 80 psig, not < or equal. Change stem. Fixed 66 F 2 M S C(41.10) 2001 NRC EXAM 67 H 2 N S A(41.10) Make sure not to include AOP-37 in ADAMS files!
68 F 2 N S B(41.10)
B(41.10) 2007 NRC EXAM Perhaps change this question to whom is responsible for 69 F 2 B E the final approval. Fixed B(41.5) HANDOUT Fig 1-1 is mentioned in the question stem. Is this figure going to 70 H 3 N E be a handout for the applicants? The question data sheet says no to a handout. Put To ensure that DNBR in stem. Fixed 71 F 3 B S B(41.11) 2009 NRC EXAM 72 F 2 N S D(41.12) 73 H 2 N S C(41.10)
A(41.10) LOD=1. Lets discuss whether or not we might need to resample this K/A.
74 F 1 N U Fixed 75 H 3 B S A(41.10) 2009 NRC EXAM 76 H 2 M S A(43.5) 2005 NRC EXAM C(43.5) Answer can be determined with RO level knowledge of AOP entry conditions.
77 H 3 N U Fixed B(43.2) HANDOUT Replace distracter A. Stands out as an outlier and is 78 H 3 N E psychometrically unsound. Fixed 79 H 2 N S D(43.5) 80 H 2 N S D(43.5) 81 H 3 N S A(43.5) 82 H 3 B S A(43.5) 2004 NRC EXAM 83 F 3 N S C(43.2) 84 F 2 N S B(43.5) Page 5 of 7 OBDI 202 - IOLE Process
ES-401 FC-2012-09 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only A(43.5) Dont see plausibility of entering Loss of Containment Integrity with just the 85 F 3 N E containment equipment hatch open. Revise distracter D. Fixed A(43.5) Remove AOP from stem. Make the applicant figure this out. Lets discuss 86 H 2 M E whether or not this is RO level knowledge. Fixed 87 F 3 N S A(43.5) 88 H 2 N S D(43.5) 89 H 3 M S C(43.5) 2010 NRC EXAM 90 H 3 N S A(43.5) 91 H 2 N U D(43.5) RO level knowledge. Fixed A(43.5) Entry condition to major EOP. RO level question. Also, only need to know 92 H 2 N E one part of answer not both. Fixed 93 H 3 N E D(43.5) Capitalize SPTA in stem. Add periods to answers. Fixed B(43.1) 1997 NRC EXAM Replace slipped on the stairs with the STA being 94 F 2 B E unavailable to the crew due to an emergency medical condition or words to that effect. Fixed 95 F 2 N S D(43.5)
C(43.5) Reword this question so that the answer requires knowledge of the department manager or above attending the pre-job brief vice just knowing the 96 F 2 N E appropriate procedure. This more accurately reflects the K/A of understanding the process on infrequent tests. Fixed 97 F 3 N E B(43.2) Not a Tier 3 question. Fixed 98 F 2 B S C(43.4) 2009 NRC EXAM 99 H 2 N S C(43.5) 100 F 2 B S D(43.1) 2005 NRC EXAM B= 29 F= 36 E= 25 RO TOTALS: M= 11 H= 39 U= 4 Additional Notes:
N= 35 SRO TOTALS: B= 4 F= 9 E= 8 Additional Notes: Page 6 of 7 OBDI 202 - IOLE Process
ES-401 FC-2012-09 7 Form ES-401-9 M= 3 H= 16 U= 2 N= 18
GENERAL COMMENT
S:
- 1. All questions from previous NRC exams should be checked to see if there were any post-exam comments.
- 2. Be consistent in capitalization (Control Room vs control room.
- 3. Worksheets should include LOD for all questions.
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
- 2. Chief Examiner comments are indicated in blue.
41.2 = 43.2 =
41.3 = 43.3 =
41.4 = 43.4 =
41.5 = 43.5 =
41.6 = 43.6 =
41.7 = 43.7 =
41.8 =
41.9 =
41.10 =
41.11 =
41.12 =
41.13 =
41.14 =
B = 16 (22%) / 7 (28%)
C = 21 (28%) / 5 (20%)
D = 25 (33%) / 7 (28%)
- 6. There are questions with attachments provided. Page 7 of 7 OBDI 202 - IOLE Process