ML112620074
| ML112620074 | |
| Person / Time | |
|---|---|
| Site: | Three Mile Island  |
| Issue date: | 07/26/2011 |
| From: | Caruso J Operations Branch I |
| To: | Exelon Nuclear |
| JACKSON D RGN-I/DRS/OB/610-337-5306 | |
| Shared Package | |
| ML110030671 | List: |
| References | |
| TAC U01829 | |
| Download: ML112620074 (13) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 1 F 1-2 X B E Since "C&O" are not actually RCP starting interlocks wouldn't it be more plausible to change these to actual interlocks like you did for "A"? Revised and reordered answer choices.
2 F 2 N S 3 H 2 B S 4 F 2 B S 5 H 2 B S 6 F 2 M S 7 F 2 X B U "8&C" do not appear to be plausible Rx power would not be increasing during the heat-up and pressure is increasing?
Rewrote Q to make choices more balanced and plausible 8 H 3 N S Note: TMI Written Exam Draft Received 7/8/11. Review completed and comments on the SRO portion of written provided 7/21/11. RO comments provided on 7/22-7/26/11.
/ & (jl/S/fts:::- U( r1I- 7)/0 J7-/ /~ 21 ;< 7 Z?) S-3 sr; (, (
) / / / / I /
8~ I F3I Ft!I J'f"I yz/ 9c.f.
Instructions
[Refer to Section 0 of ES*401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowiedge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LaD) of each question using a 1 5 (easy difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless infnrm::ltinn)
The stem or distractors contain cues (I.e., clues, specific determiners, phrasing, length. etc).
The answer choices are a collection of unrelated truelfalse statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (I.e., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the dosed reference test mode (I.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved KIA and those that are deSignated SRO-only (KIA and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES*401 Section 0.2.f.
- 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum. explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO 8/M/N UlE/S Explanation Focus Dist. Link units ward KIA Only 9 H 2 8 S 10 H 1-2 X X N U 1) KIA mismatch? Is the A D-ring part of the EeeS system if not than how are testing the EeeS piece affect of EeeS injection?
- 2) LOO=1 too easy testing temp increase affect on PZR leve
- 3) "A&C' not plausible that level lowers on increasing temperature.
KIA randomly selected and replaced with a bank Q.
11 F 2 M E 1) Looks like Memory level rather than "H" if you know that at 100# EDG will load. Redesignated memory level.
12 H 3 X X B U 1) KIA mismatch - this Q is not testing a loss (I.e., loss in this context would be due to something going wrong like a fire short) or malfunction (this is normal and expected not a malfunction)?
- 2) "C&O" are not plausible since if there was a discharge to the RCOT temp would be higher. "8" also appears implausible with the spray valve open temp would be decreasing quicker than >5F/min Modified Q and distracters to make more plausible and match KIA 13 H 2 N S 14 F 3 X 8 E 1) Does the LO state that ROs need to know this information from memory?
- 2) Knowing this from memory appears to be borderline minutia. Do you want the NRC to deny a license if your applicants get this wrong?
Extensively revised Q to address concerns
r===
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
a# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- a= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 15 F 1-2 X N U 1) LOD=1 too easy does not test operator understanding
- 2) "A&C" are not plausible if applicants are properly trained they should know the breaker will not close and the lights should be off.
Wrote new Q testing same KIA 16 F 3 B S 17 H 3 M S 18 H 3 B S 19 F 2 M E It might be argued that as written "0" may not be a technically correct answer with a lag time of 30 - 60 minutes RM-L-1 may not show an upward trend for some time? Modified all answer choices to add clarity 20 H 3 B S 21 F 1-2 X N U 1) LOD=1 does not test operating understanding. If the applicant knows that you wouldn't trip the Rx at 120 or 130F which is too basic then the correct answer is obvious. 130F is pretty low and shouldn't jeopardize plant safety in any way.
- 2) Distractors are not credible to trip the Rx at 120 - 130F.
Changed Stem and question wording, agreed to change two distracters "B" & "c" and shorten :f'I distracter reordered "B" and "e" now "A" and "B".
22 H 3 N S 23 H 2 X N E 1) "Aft is not credible that with operation in the low NPSH portion of the curve.
- 2) Remove the words "References Provided" from the question. The applicants should know how and when to apply this reference and this makes the a more operationally valid since that is way it is in the CR Revised STEM and modified "A" re-ordered "A" and "B" reversed positions.
24 H 3 B S
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. a.
Q# LOK LOD
~---
(F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/MIN UtE/S Explanation Focus Dist. Link units ward KIA Only 25 F 2 N S 26 H 2 N E 1) Are system flow rates required information for the operators to know from memory and do you have a LO to support this?
Revised Q to get away from straight recall of flow rates to testing understanding of system response.
27 H 1-2 X N U 1) LOD=1 is not testing operating understanding too fundamentaL There is nothing in the stem of the Q to indicate that the MSIVs should have been given an auto closed signal. The stem also indicates that valve was mid positioned which would provide the expected position light status as indicated in choice "A"
- 2) Given the conditions in the stem and very fundamental plant knowledge as stated above the distracters are not credible.
- 3) General comment do not list on any question "Reference Provided" for reasons stated earlier make more operationally oriented.
Replaced Q with new Q testing same KIA.
28 H 3 M S 1) In this case power is not lost and therefore doesn't transfer auto or manually. This would be a tighter KIA match if the question actually tested a manual or auto transfer of power? Q is testing auto transfer logic and when it is blocked which meets spirit and intent of the KIA.
29 H 2 X N U 1) For a stuck rod you would expect choice "N to be correct.
- 2) Distractors are not credible if the applicants are trained on very basic system ops.
Revised A&C to better test operator understanding.
30 H 3 N S 31 F 3 N S
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 32 H 3 X N E 1) Question stem needs to be better focused this is really a two part Q. first part can you verify NC and second how do you verify NC established with these conditions.
Revised stem and answer choices.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 33 F 2 B S 34 H 3 N S 35 H 3 N S 36 H 3 B S 37 F 2 M S 38 F 3 B S 39 H 3 B S 40 H 2 X B E "A" is not a credible distrator with even very basic system knowledge Revised distractor "A" to be more plausible and tightened stem
ES*401 Written Examination Review Worksheet Form ES*401*9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #t 8ack Q= SRO 8tM/N utEtS Explanation Focus Dist. Link units ward KIA Only 41 H 3 N S 42 F 2 N E Revise stem 'Which of the following identifies the pumps the operator "is required to start by procedure?" Done 43 H 3 8 E On the surface this appears to be an SRO question. I think basically the recognition of no FWIEFW flow available and CETs increasing to secure one RCP per loop makes sense and maybe that makes it appropriately RO level. Replaced Q using same KIA 44 H 1-2 X N E 1) This appears to be very easy Q as written. I guessed this with no plant kno\Alledge. I looked at which were the largest electrical loads and considered that most likely.
- 2) Credible distracters - One would expect any loads coming off vital buses to be less susceptible. The question could be revised to pick at least two other important large loads.
Revised "D" to make more credible.
45 H 3 N S 46 H 3 X N E "cn distractor doesn't appear credible with no indication of RCS inventory challenges? Revised "e" 47 H 2 X X N E 1) "AU is not a credible distractor you wouldn't normally trip the Rx for a loss of letdown.
- 2) Please add AOP 0231, step 3.5 bases, pg 7, 2 nd paragraph that discusses closing EG-V-15 and possible damage to EDG.
Revised A to make more credible.
48 H 3 8 S 49 H 3 N E Add the word "to" 2.25 MWe to each answer Made change 50 F 2 N S
51 H 3 N S 52 H 2 N E 1) There is another Q on the exam that test Loss of Heat Transfer (LOHT)? Replaced 0#43 53 F 2 X N U Although a, b, and c are directed by the procedure, that does not make them a credible distractor. The candidate should recognize that SCM is low. Would any of the actions in a, b, or c ever be a correct answer for a low SCM condition? Revised distracters to provide more credible choices.
54 F 3 B S 55 F 2 N S 56 H 4 B E Explanation beefed up to include normal system ES alignment for 1C pump is always ES selected. 1B-E is normally aligned for MU and seal injection. Procedure step 4.2.4.B states, "The ES selected pump: MU-P-1C or MU-P-1B-E." If MU-P-1C was not put in PTL, is the question as worded still correct? By procedure it would appear that you can't assume that both MU P-C and MU-1B-E were both in PTL?
57 H 3 N S I 58 F 3 N S
Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO Focus Dist. Link units ward KJA Only 59 F 3 X N U How are C and D plausible distracters. If the MU recirc valves are opened, how could this prevent an overfill on a spurious ESAS actuation? Is there any physical way that the PZR could be overfilled by having these M/U recirc valves open?
Rewrote new replacement Q testing same KIA.
60 H 3 X M E *C&D" don't appear credible basic system knowledge to know that 14A & 14B are parallel injection paths and only one path is needed.
Modified A&B to balance answer choices plausibility and added to justifications.
61 H 3 M S 62 F 2 N E A little concerned that someone could argue in post exam space that A&C are equivalent answers. Modified A&C to address concern 63 F 3 N S 64 F 2 N S 65 H 3 B S
,,66 F 2 B S 67 H 3 B S 68 H 2 X N E 1) "A" is obviously not credible since it represents the definition of a critical RX? Revised "A" to be more plausible 69 F 1 X N U 1) LOD=1, C is the only answer that is common sense without knOwing TMI procedure. i.e place defiecincy tag on the thing broke and place a status tag on the switch that is out of position.
- 2) The distracters are not plausible.
Rewrote Q to address concerns 70 F 2 B S 71 F 2 N S 72 H 3 N S 73 H 2 B S
74 F 2 B E KIA matched since IAAT is also a term used in EOPs extensively. Licensee beefed up explanation of KIA match and IAA T statements use in EOPs and AOPs. AOP use more appropriate for ROs 75 F 3 B S 76 F 3 N E Is the generator capability curve required to answer this questions or this recall of some thumb rules? Will provide the generator capability curve as a reference and consider adding H2 pressure for generator in stem.
77 H 3 B S 78 H 3 N S 79 H 2 X N E Revised justification statements.
80 F 1-2 X N U Appears to be RO system level Q. If you look-up saturated temp for 720 psig then you know Sub- cooled temp and knowing what is required SCM and whether it is adequate is RO level knowledge.
Revised both stem and answer choices to include procedure selection. If applicant selects wrong procedure will get answer wrong now SRO level.
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
00 LOK LOO (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/MIN utE/S Explanation Focus Oist. Link units ward KIA Only 81 H 2 X N E 1) Explain why "A&B" are plausible to restore buses 10 or IE to normal power supply with a loss of offsite power in progress?
- 2) This question can be answered with basic RO system level knowledge in that MU pump B tripped and MU is needed for RCP seal leakage therefore it really comes down to "C or 0" and the normal MU pumps are "A&B" so with "B" tripped start "A".
Revised stem to delete MU pump trip and to make Q more challenging and SRO level.
82 F 2 N S 83 H 3 X M U Don't understand how "A & S" are plausible to enter fuel handling procedure when there are no fuel handling or fuel bridge problems described in the stem? Added condition in stem with assembly can't ungrapple but consider not fully in core?
84 H 3 X N U 1) "C&O" continuing to steam the OTSG with High Rads release to "dilute also
- 2) not isolating the S/G until less than 1,000 psig to prevent lifting relief should be RO system level knowledge?
Modified distracters and stem.
85 H 2 X N E 1) "0" appears to be implausible close valves "as needed"?
86 H 3 N S 87 H 1-2 X X N E If know location of valve can eliminate "N since in uninhabitable area, "8" if know system you will know this action wont isolate flow path, "C" is the minimum action to isolate EFW flow to the affected OTSG.
Made a 2X2 format to make choices more plausible.
" " " - ------ - ""L-"""""" --------- """ '-""""""
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 88 F 1-2 X X N U Can be answered with RO system level knowledge only e.g. if know set points for pressure and boron concentration which should be pretty fundamental RO system level knowledge can eliminate A. B.
C fairly easily with some certainty even if you just remembered the set point for pressure only could easily delete answer B&C. Per the SRO only level guidance this would be above the line RO TS knowledge.
This is memory level not Higher Order recall of 2 simple set points Replaced with a new TS Q testing same KIA.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/MIN utE/S Explanation Focus Dist. Link units ward KIA Only 89 H 2 X X N E 1) uB" states complete maintenance within 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> is that ok in this case? If you take 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> to restore EFW 2B TS 3.0.1 would still need to be implemented for the 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> making uB" incorrect as worded.
Revised and tightened stem 90 F 2 N S 91 F 2 X N S 92 H 1-2 X N U 1) This question is RO TS above the line required knowledge by our guidance since RO would be expected to know that 2 fire pumps are required to meet TRM requirements.
- 2) "B" does not appear plausible to establish a backup system in 24 hrs with even very basic knowledge of the fire suppression system?
- 3) "C" does not appear plausible with even very basic knowledge of the fire suppression system to have to sId with only one fire pump inop?
- 4) LOD=borderline Improved Q by modifying stem to increase level of difficulty resulting in a different answer 93 F 2 N S
...- . _. I ..~... ---- ~ . .... ~
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #! Back- Q:: SRO B/MIN utEtS Explanation Focus Dist. Link units ward KIA Only 94 F 1-2 X N U 1) LOD:: too easy
- 2) "A&B" are not credible distracters moving new fuel need SRO
- 3) Memory level - just knowing when SRO is required 95 F 2 N S 96 F 2 X N E 1) "0" explain plausibility? Don't believe that it is plausible for anyone to believe that a discharge can be delayed indefinitely without initiating a new permit?
- 2) Recommend changing answer to 9:59 the latest time and change other answer choices for parallelism.
Added justification for "D" and revised answer choices. I 97 H 2 B E No security codes declared on a credible threat? Revised stem to a "White" threat 98 F 2 N S 99 H 3 N S I 100 H 2 N S