ML063250071
| ML063250071 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 10/30/2006 |
| From: | Operations Branch IV |
| To: | Southern California Edison Co |
| References | |
| 50-361/06-301, 50-362/06-301, ES-401, Rev 9, ES-401-9 50-361/06-301, 50-362/06-301 | |
| Download: ML063250071 (10) | |
Text
ES-401, Rev. 9 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 3 S D-2 F 3 S B 3 H 3 S B 4 H 3 S C 5 H 3 S D-6 H 3 U(S) C- This question does not meet the KA because the question is testing on flowpath whereas the KA is focused on the function/purpose of major system components/controls. Added details in answer section of question to embellish justification of CBO as a fn of VCT. K/A and question matched but the question was written such that the function was not clearly identified.
7 H 3 U U (S) B- This question does not meet the KA because the question is testing on operator actions given a set of conditions whereas the KA is focused on the system purpose/function. Added details in answer section of question to embellish justification of question to K/A. K/A and question matched but the question was written such that the function was not clearly identified.
8 H 3 S D-
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
- The answer choices are a collection of unrelated true/false statements.
- The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401, Rev. 9 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 9 F 2 U U (S) A- This question does not meet the KA because the question is testing on trip setpoints whereas the KA is focused on the system compontents/controls purpose/function. New question drawn because this one was unsat and new a K/A was selected for this question because of oversampling of G2.1.28 K/A in outline. New K/A 012.K1.06.
10 F 3 U U (S) B- This question does not meet the KA because the question is testing on operator actions given a set of conditions whereas the KA is focused on operator responsibilities. New question drawn because this one was unsat and new a K/A was selected for this question because of oversampling of G2.1.2 K/A in outline. New K/A 012.K2.01.
11 H 3 S A-12 H 3 S D-13 H 3 S C-14 H 3 S A-15 F 3 S B-16 F 3 S B-17 H 3 S A 18 H 3 S D 19 F 2 S B 20 H 2 S D 21 F 3 S A-22 H 3 S C 23 H 3 S B
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 24 F 3 U U (S) B- This question does not meet the KA because the question is testing on system response to a given a set of conditions whereas the KA is focused on the system purpose/function. This question was SAT but needed embellishment in answer explanation because at SONGS the seawater is their service water. Utility added to explanation.
25 F 3 S B 26 H 3 S B 27 F 3 S B 28 H 3 U U (S) C- This question does not meet the KA because the question is testing on operator procedure use given a set of conditions whereas the KA is focused on operating/monitoring system valves. The question could not be written for this K/A for this facility so a new question was written based on a new K/A 103.A3.01.
29 H 3 S A 30 F 2 U U (S) A- This question does not meet the KA because the question is testing on the primary indication for reactor power whereas the KA is focused on what redundant indications are available for reactor power. Replaced question.
31 F 2 U U (S) A- This question does not meet the KA because the question is testing on system design whereas the KA is focused on the operational implications of the system design. Reviewer misunderstanding due to poor clarification of answer. Question meets K/A and it is operationally written for higher cognitive ability.
32 H 3 E E (S) C- The candidates will need steam tables provided. Noted.
33 F 3 E E E (S) B- It is not apparent this is a memory question and there is no learning objective provided to support the RO job-link. This may be an SRO only question. LO provided from utility to satisfy job link at SONGS for Ro to be done from memory.
34 H 3 S C 35 F 2 S B 36 H 3 S B
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 37 F 3 S D 38 H 3 S B 39 H 3 S C-40 H 3 U U (S) D-This question is a KA mismatch because the KA is focused on system limits and precautions as applied to a pressurizer vapor space accident where as the question is focused on operator actions in response to a pressurizer vapor space accident. Modified question to provide better alignment to K/A and provided more detailed explanation of answer in justification section.
41 F 2 S C 42 F 2 S C 43 F 2 E ? U U (S) B-The stem should include a statement on the status of the RCP seals as this is part of the conditional statement in the referenced procedure.
Question - This is not an immediate action, therefore is this a memory item per the learning objectives? Added Seal DPs are normal to stem and incorporated LO to demonstrate this is required from memory by SONGS training program.
44 H 2 S B-45 H 1 U U (S) B- It is intuitive that a trip of the running pump would require starting the standby pump. Second, the KA is focused on reasons to shift to an alternate RHR flowpath and shifting pumps does not constitute an alternate flowpath. Consulted with NRR (they agreed with reviewer) so utility replaced question and sampled new K/A, 025.AA2.02.
46 F 3 E E (S) C- This is not an immediate action and this does not appear to a memory item. A learning objective is not provided. Improved question and utility provided LO to author (contractor) for modified question.
47 H 3 U U (S) A- The KA asks for the reason a response is taken during a SGTR and the question asks for what action should be taken. The reason is in the answer, therefore this question is SAT.
48 H 3 S B-49 F 3 S B
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 50 H 3 E (S) D- The question should indicate steam tables will be provided. Noted.
51 F 3 U (S) A- The referenced procedure states the first action is to confirm a single vital bus is lost. This is not listed as a distractor in response to what action is taken first. Therefore there is no correct answer to the question. Modified stem and initial conditions to create one and only one correct answer.
52 H 3 S B-53 H 3 S D-54 H 3 S D 55 H 3 S C-56 F 3 S D 57 F 3 S B-58 F 3 S B 59 H 4 S B 60 F 3 S C-61 H 3 S B 62 H 3 S B 63 H 3 S B 64 F 3 S A 65 H 3 S D 66 H 3 S D 67 F 2 S B 68 H 3 S B 69 F 3 S C
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 70 F 3 S D 71 F 3 S D 72 F 3 U U (S) C- This does not appear to be a memory item unless there is a learning objective to support the question. Utility disagreed and provided LO for this question.
73 H 3 S A 74 F 3 S D 75 F 3 S D 76 H 3 U U (S) A-if I know that the leak is in letdown (RO knowledge) then I can answer the question (ie not SRO level). Modified question to incorporate SRO knowledge on procedure selection.
77 H 3 U U (S) C- This is a poorly written question. There are many assumptions that an applicant can make (such as what partially open means) in answering this question. Modified question to include percent open and prevent D from also being correct answer.
78 H 3 U U (S) A- Distractors C and D are not credible since the EDG is not running, there is no concern with the output breaker closing. Changed distractors C and D for credibility.
79 H 3 U U (S) B- This is not a memory item in that applicants are not expected to know tech spec action statements that are greater than 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> without a reference being provided. Utility disagreed with reviewer and provided LO to support requirements on this for SRO.
80 H 3 S D 81 H 3 S C
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 82 H 3 U U U U (S) C- This question has some problems. First two of the distractors have grammatical errors which means 99% of the time they can be considered incorrect by the test taker. Second, the KA is focused on the function of major system components and the question is on operator actions. Third, the stem has conflicting information with regard to the status of CEA #20. Changed K/A to G2.1.12 because of oversampling, because needed more TS type questions for SRO exam, and because could not develop SRO level question on this topic for this K/A. New Q.
83 H 3 U U (S) A- The KA is focused on operator responsibilities and the question is focused on operator actions. Modified A to include procedure and added items to other selections for symmetry as well as embellished explanation for responsibilities include actions taken by SRO.
84 H 3 S A 85 H 3 S A 86 H 3 U U (S) B- This question is focused on the operator actions which makes it an RO question. It should be focused on diagnosis and which procedure to implement in response to the given conditions. Removed RO knowledge items and modified selections to include SRO only information.
87 H 3 S D 88 H 3 E (S) D-pt value incorrect on exam and outline. Corrected.
89 H 3 U U (S) B- The question does not address the predict impact part of the KA.
Consulted NRR and every question does not have to address both parts of the A2 K/A, but the emphasis is to TRY to write it on both if possible.
90 H 3 E (S) A-symmetry of distractors. Modified selections for better symmetry.
91 H 3 U U (S) A- The question does not address the predict impact part of the KA.
Consulted NRR and every question does not have to address both parts of the A2 K/A, but the emphasis is to TRY to write it on both if possible.
92 H 3 S A-93 H 3 S D 94 H 3 E (S) D-from memory? Yes and LO provided.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 95 F 2 S C 96 F 3 S B 97 F 3 E (S) B- Remove the if before ONLY IF in distractor C. Done.
98 F 3 S 99 H 3 U U (S) C- These are all immediate actions and are RO knowledge. For SRO the distractors should be something like enter procedure xxx based on...
Threw out question because this K/A is RO level, selected G2.4.30 and wrote new question.
100 F 3 S RO H=41 U = 16 RO TOTALS After facility feedback RO TOTALS B=39 54.7% E = 4 U = 22% U = 23.3% ( 10/75 Q)
M=4 F=34 S = 54 E = 5%
N=32 45.3% S = 72%
SRO H=20 U = 10 SRO TOTALS After facility feedback RO TOTALS B=12 80% E = 1 U = 40% U = 28% (7/25 Q)
M=2 F=5 S = 14 E = 4%
N=11 20% S = 56%
NOTE: Bank questions are indicated by B; Modified are indicated by M; New are indicated by N; comment resolution is indicated in blue;
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only General Comments:
- 1. Answer Distribution:
a = 12 + 9 = 21 b = 30 + 5 = 35 c = 15 + 6 = 21 d = 18 + 5 = 23 Recommend reducing the number of B answers to less than 30..
- 2. Average Level of Difficulty (LOD) is: RO = 2.84 and SRO = 3.0. Both these values are within the expected norms.
- 3. Learning objectives were not provided for most of the questions, nor could they be located in the system descriptions. Therefore some questions were not clear as to whether they were memory items. These questions are highlighted in the review. LO provided.
- 4. The parent questions were not provided for modified questions, therefore all modified questions were counted as bank questions for the purpose of meeting the NUREG-1021 requirements. Modified question parents provided.
- 5. There is a very high percentage of scenario based questions (i.e. answer a question based on given a set of plant conditions) on the exam. The exam reviewer found over 70 percent of the questions are scenario based. (Observation) This is encouraged in the NUREG to support operational validity and raises cognitive level required to answer question.
- 6. The distribution of questions is statistically skewed toward the generic KAs. The exam has 34 questions (including the 17 Tier 3 questions) from the generic section of the KA catalogue and 18 (53%) of these are from generic section 2.1, Conduct of Operations. Thirteen exam questions are derived from 5 generic KAs from Section 2.1. A random review of 3 other NRC exams found that on average there were 27 generic KAs and 9 were from Section 2.1. Recommend reducing the number of questions from Section 2.1 that are presently on the RO exam. The contractor that wrote the exam explained that they used the rule in NUREG-1021, ES-401, Att-1 for K/A suppression, rule 1, that allows the suppression of all Generic K/As except for a select few for T1 and T2, which is allowed. The problem of too many G2.1.2, G2.1.27, and others was addressed by requiring the utility to resample other K/As for balance.
- 7. On the SRO only portion of the exam, there are 2 questions involving the application of the technical specifications whereas a typical exam has 6-8 questions of this nature. Secondly, there are no questions involving refueling. (Observation) In changing the sampling per item 6 above, required addition of TS question to shift balance.
- 8. The questions on the RO exam were within the guidelines for higher order questions (54.7%) and were higher then the guidelines for SRO (76%). The guideline is 50-60% of the questions should be at the comprehension/analysis level.
- 9. Twenty questions were sampled for determining if they had been on either of the previous 2 NRC exams and none selected were repeats.