ML14279A078
| ML14279A078 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 08/08/2014 |
| From: | Vincent Gaddy Operations Branch IV |
| To: | Pacific Gas & Electric Co |
| laura hurley | |
| References | |
| 50-275/14-OL, 50-323/14-OL | |
| Download: ML14279A078 (8) | |
Text
ES-401 CP-2010-03 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E (41.5) Indicate what Tc is in the stem information to ensure that LTOP applies. Mode 4 doesnt appear specific enough (200 to 350F) 1 H 3 D N S Fixed 2 H 3 C N S (41.1) 3 H 2 A N S (41.7) 4 H 2 A N S (41.10) 5 H 3 B B S (41.7) 2012 NRC EXAM E (41.4) Consider replacing D with Primary Water Storage Tank as it is the first alternate.
6 F 2 A N S Fixed Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 8 OBDI 202 - IOLE Process
ES-401 DC-2014-08 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 7 H 3 D B S (41.5) Reference 8 H 1 A N S (41.7)
E (41.7) Can determine the answer by just knowing that channel III bistable status light should be out. Consider using only II and III to go through all possibilities in order to 9 H 2 C N eliminate this as an issue.
S Fixed 10 H 2 A N S (41.7) 11 H 2 C N E (41.7) Only have to know the first part of C to come up with the correct answer.
12 H 2 C B S (41.5)
E (41.10) Need to have one more not expected to make this psychometrically sound and less leading to A being one of the wrong answers.
13 H 2 B N S Fixed 14 F 2 C B S (41.10) 15 H 3 D N S (41.4)
E (41.7) It would be better, psychometrically speaking, to have two Yes and two No in the answer choices.
16 H 2 B N S Fixed 17 F 2 A N S (41.1) 18 F 2 C B S (41.10) 19 H 3 D B S (41.7) 20 F 3 C N S (41.7) 21 F 3 A B S (41.8) 2012 NRC EXAM 22 H 3 B N S (41.7) 23 F 2 B B S (41.8) 24 F 3 B N E (41.11) Distracter A appears as an outlier as it is the only discharge continue Page 2 of 8 OBDI 202 - IOLE Process
ES-401 DC-2014-08 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only option. Look for another way to incorporate this as a distracter to enhance this question psychometrically.
S Fixed 25 F 2 C B S (41.7) 2010 NRC EXAM E (41.7) Modify stem to say The following events occur in the order listed:
Applicant could argue B not technically correct as written, since the first start signal did not occur when ASW pressure lowered to 35 psig, it occurred slightly before that 26 H 2 B N at the moment when ASW pressure lowered to 40.5 psig.
S Fixed 27 F 2 B B S (41.7) 2008 NRC EXAM (41.10) Change distracter D to start one CRDM fan. As starting all fans is not E
allowed this may clue the applicant immediately to this as a wrong answer.
28 H 2 A N S
Fixed 29 H 2 D N S (41.7) 30 H 2 A N S (41.7) 31 H 3 C N S (41.7)
E (41.2) Is distracter A credible? Not an input to SCMM?
32 F 2 D N S Fixed 33 F 2 A B S (41.7) 2010 NRC EXAM 34 H 3 C B S (41.7) 2012 NRC EXAM (41.6) C should be changed to start with the most accurate just like the distracters.
E Otherwise it stands out.
35 F 3 C N S
Fixed U (41.11) K/A mismatch - K/A requires knowledge of monitoring parameters to prevent 36 H 2 B N exceeding design limits. Question as written only asks how to minimize radiation Page 3 of 8 OBDI 202 - IOLE Process
ES-401 DC-2014-08 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only exposure.
S Fixed 37 H 3 C N S (41.7) 38 H 2 C N S (41.8) 39 H 2 D B S (41.5) 2005 NRC EXAM E (41.8) C and D dont seem credible with the absence of S/Gs as a possibility.
Perhaps this question would be better as a 2 x 2 with S/Gs in each.
40 H 2 B B S Fixed 41 F 3 B M S (41.8) 42 H 2 C N S (41.7 43 H 2 A N S (41.4) 44 H 2 B N S (41.5) 45 F 2 D N S (41.2) 46 F 2 C N S (41.10)
E (41.5) Maintaining AFW to one faulted S/G not credible. This affects two distracters.
47 F 2 C N S Fixed 48 H 3 D B S (41.7) 49 F 3 B B S (41.8) 2007 NRC EXAM 50 H 2 B N S (41.7) 51 H 2 D N S (41.7) 52 F 3 C N S (41.7) 53 F 3 A N S (41.10)
U (41.7) T/F - Does not appear that any stem information is needed to answer this 54 F 2 A N question. Page 4 of 8 OBDI 202 - IOLE Process
ES-401 DC-2014-08 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S Fixed 55 H 2 B B S (41.10)
(41.7) Provide further explanation how B is not true if T < 283 and P >435 E
shouldnt PORV open?
56 H 2 A N S
Fixed E (41.5) Change distracter D to 60 minutes in order for it not to seem as an outlier. Its significantly more time than the answer or the other distracters.
57 H 2 C B S Fixed E (41.5) Distracter D doesnt seem credible as the core is being cooled already by ECCS.
58 F 2 A N S Fixed 59 H 3 A N S (41.2)
E (41.7) Distracter A not credible for Cardox to dump immediately into a room where people could be present.
60 F 2 D N S Fixed 61 H 2 B N S (41.7) 62 H 3 D B S (41.10) 63 H 2 D B S (41.5) 2007 NRC EXAM 64 H 2 A N S (41.7) 65 F 2 C N S (41.5) 66 F 2 D B S (41.10) 67 F 2 B N S (41.10) 68 H 3 A N S (41.7) Page 5 of 8 OBDI 202 - IOLE Process
ES-401 DC-2014-08 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 69 F 2 D N S (41.10) 70 H 2 C N S (41.6) 71 F 2 B B S (41.10) 72 H 2 B B S (41.12) 2005 NRC EXAM 73 H 3 A B S (41.11) 74 F 3 C B S (41.10) 75 F 2 D B S (41.10)
E (43.2) There is no actual question in the stem. If REGEN HX outlet temp is rising, that means less charging flow going through HX, wouldnt you therefore expect Lpzr 76 H 2 C N to lower as a consequence?
S Fixed U (43.5) RO level question. Check the SRO Question Guidance. Plant parameters that require direct entry to major EOPs:, e.g., major Westinghouse EOPs are E0, 77 H 2 C N E1,, Functional Restoration Procedures are now allowed.
S Fixed 78 F 3 A B S (43.2) 2007 NRC EXAM 79 H 3 D N S (43.2)
E (43.5) 2009 NRC EXAM - RO level question. Entry level condition into ECA-0.0 Should have been caught in 2009.
80 H 2 D B S Fixed 81 H 3 C B S (43.5) 2009 NRC EXAM 82 H 2 D N S (43.2) 83 F 3 A N S (43.4) 84 F 3 A N S (43.1) 85 F 2 D N S (43.2) Page 6 of 8 OBDI 202 - IOLE Process
ES-401 DC-2014-08 7 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 86 H 3 B N S (43.2) Reference E (43.2) Add surveillance requirement to strengthen as SRO level question.
87 H 2 D N S Fixed 88 H 3 A N S (43.2) 89 H 3 D N S (43.5)
U (43.5) Entry level conditions into E-0 are RO level question.
90 H 2 A N S Fixed 91 H 2 B N S (43.4) 92 H 2 C B S (43.7) 2005 NRC Exam 93 H 2 B N S (43.1) 94 F 2 B N S (43.3) 95 F 2 A B S (43.3) 96 F 2 C B S (43.3) 2008 NRC EXAM 97 F 2 C N S (43.3) 98 F 2 D B S (43.4) 2007 NRC EXAM 99 F 3 B B S (43.4) 2009 NRC EXAM 100 F 2 B N S (43.5)
B= 26 F= 31 E= 16 RO TOTALS: M= 1 H= 44 U= 2 Additional Notes:
N= 48 B= 8 F= 11 E= 3 SRO TOTALS: M= 0 H= 14 U= 2 Additional Notes:
N= 17
GENERAL COMMENT
S: Page 7 of 8 OBDI 202 - IOLE Process
ES-401 DC-2014-08 8 Form ES-401-9
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
- 2. Chief Examiner comments are indicated in blue.
- 4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 2 43.1 = 2 41.2 = 3 43.2 = 8 41.3 = 0 43.3 = 4 41.4 = 3 43.4 = 4 41.5 = 10 43.5 = 6 41.6 = 2 43.6 = 0 41.7 = 30 43.7 = 1 41.8 = 6 41.9 = 0 41.10 = 15 41.11 = 3 41.12 = 1 41.13 = 0 41.14 = 0
B = 19 (25%) / 6 (24%)
C = 21 (28%) / 6 (24%)
D = 16 (31%) / 7 (28%)
- 6. There are 2 questions with attachments provided. Page 8 of 8 OBDI 202 - IOLE Process