ML090290231
| ML090290231 | |
| Person / Time | |
|---|---|
| Site: | Calvert Cliffs  |
| Issue date: | 10/30/2008 |
| From: | Brian Haagensen Operations Branch I |
| To: | Calvert Cliffs |
| Hansell S | |
| Shared Package | |
| ML082600339 | List: |
| References | |
| 50-317/08-302, 50-318/08-302 | |
| Download: ML090290231 (17) | |
Text
Written Exam Inventory Comments:
- 1. The revised sample plan does not contain WA descriptions. Please submit a sample plan that has these descriptions included, not just the WA numbers. Please use the format in ES 401-1/2 forms.
ES 401 page 4 Enter the KIA statement numbers, a brief description of each topic, the topics importance ratings for the license level of the exam (use the RO and SRO ratings for the RO and SRO-only portions, respectively), and the point totals (system, category, group, and tier) on the examination outline. The proposed point totals for each group and tier must match the number specified on Forms ES-401-1 and ES-401-2, as applicable.
ANS: The form submitted is not a revised ES401-1/2. The original ES401-1/2 remains valid with no changes to any WAS.
- 2. Please include nomenclature for the learning objectives. The objective number alone does not provide enough information for our review.
ES401 page 8 A technical reference and a cross-reference to the facility licensees examination question bank, if applicable, shall be noted for every question. If the facility licensee has a learning obiective applicable to the question, it should also be referenced. However, the absence of a learning objective does not invalidate the question, provided that it has an appropriate WA and technical reference. Refer to ES-201 for additional instructions regarding documenting the source of questions on facility-written examinations.
ANS: Adding learning objectives will take about 10 hours1.157407e-4 days <br />0.00278 hours <br />1.653439e-5 weeks <br />3.805e-6 months <br /> of time. Do not take this time.
Instead, I will ask you to provide lesson plan objectives if we challenge the validity of any questions.
- 3. Please provide copies of any modified questions that you have used in this exam. As an alternative, you may describe the changes that you made to the original bank question for all modified questiions.
ES40 1-5 requires:
Question Source: Bank #
Modified Bank # (Note changes or attach parent)
New A N S : This information will be provided in the hard copy submittal.
- 4. Please provide a history of the last time the question was use on an NRC exam - if applicable.
ES401-5 requires:
Question History: Last NRC Exam A N S : We did not take any questions fiom the past 2 NRC exams.
- 5. Please provide the cognitive level of each question or a guide that allows us to determine your cognitive level numbering scheme vs. our cognitive level categories in ES 401-5.
Question Cognitive Level:
Memory or Fundamental Knowledge Comprehension or Analysis ANS: I discovered the key in the electronic files. This is not a problem.
6 . You are not required to submit a plausibility analysis -but if you do not, we will have to do it ourselves. This will add time and effort to the review. We will have to establish plausibility for each distracter and document this in our form for those questions which are not obvious.
To facilitate the review process, examination authors should consider providing a brief explanation of why the answer is correct, and each of the distractors is plausible but incorrect. This optional practice increases the efficiency of the examination review process and promotes the detection and correction of problem questions before the examinations are administered.
I did not provide this comment to Calvert Cliffs.
- 7. Please include the WA nomenclature to the questions to make review easier. You are not required to add WA nomenclature to the question. But if you do not add this nomenclature, we will have to do add it ourselves. This will add time and effort to the review.
A N S : We will add this information.
Calvert Cliffs SRO ES 401-9 SRO EXAM
~~
~~
- 2. LOD 3. Psychometric Flaws 4. Job Content Flaws Ref Req 5. Other
- 6. 7.
BlMlN UIEIS Stem Cues TIF Partial Link Minutia #I units Back-Job- Q=WA SROonly Focus ward 7. Explanation Add IAW OP-7. Do they have to know this from memory? NI replace question Replaced question '%"is not plausible -
see suggested enhancement Calverl believes the suggestedenhancement may be a second correct answer. They want to use the original distracter ' X "
because this was the correct answer prior to a recent change to the procedure. Draft a plausibirity analysis to 3 N Y Y M S establish the case for why A i s plausible. - DONE Ok This is not an SRO level question -can write an SRO level question of you test tech specs - MADE CHANGES TO ADD 3 N Y Y B S TECH SPECS TO DISTRACTERS OK -
3 N Y Y B S Why are C and D plausible? C IS OK. REVISED D.- ~
Why provide section 3.0 of tech specs? We normally expect applicants to know this section. OK - REMOVED TS 3 Y Y Y N S SECTION 3.0 from handout 3 N Y Y N S Change to "directed by the CRS" - REVISION OK Why would EOP-7 be a plausible choice? See suggested changes to address issues. Diagnostic flow chart was not provided with reference package. RECOMMENDED CHANGES TO QUESTION - WILL GET BACK TO ME- Mad Changes to distracters C and D as discussed. Add 2 Y Y Y N S plausibility analysis. ADDED -OK I would consider this question to be written at the higher cognitive level. Why address USFAR 14 conditions in stem?
What does this mean? REMOVE REFERENCE TO UFSAR.
3 N Y Y B S CHANGE TO "Higher cog level" DONE C and D not plausible - shutdown margin not challenged unde these conditions. See proposed revisions to the question.
1:I This should be higher level of knowledge. CONSIDER REVISING QUESTION TO MAKE C AND D MORE 3 N Y Y B S PLAUSIBLE - Revised C and D to make more plausible.
B. C and D are not plausible as written (when isolated from A)
A is important - B, C and D are minor additional actions. See suggested changes to improve plausibilty using same information. Also note that another question provided a copy of the RAD Waste discharge permit - eliminate this reference or verib that this does not give this away. REVISE QUESTION TO CHANGE - Revisions made as requested.
85 10 3 3
--- N
--_________-. Y Y N S Need to add the word "verify" to distracter B N Y Y N S 45 Day 1I1 3/2009 Page 1
SRO EXAM SRO ES 401-9 Calvert Cliffs Page 2 1/13/2009 45 Day
m m
W a
Calveti Cliffs SRO ES 401-9 SRO EXAM FINAL Page 1
Calvert Cliffs SRO ES 401-9 SRO EXAM I
Q#
tesi
- 2. LOD I 3. Psychometric Flaws
- 4. Job Content Flaws 6.
BlMlN 7.
UIEIS Q#
- t:i Partial Minuti;
!I unit:
Back-ward - 7. Explanation Replace this question. A, C and D are not plausible. Question is not at the SRO level. WA is very broad. REPLACED QUESTION - new question: add A.8,C.D above distracters to correlate matching. Should this be F LOK?
Distracters A, B, and D not plausible (A=l right, B=1 right, C=2 right). See suggested revisions. ACCEPTED 86 11 M S REVISIONS - OK SRO level question? Why not test what you have to do about it? Stop the startup - continue etc? REVISED QUESTION -
see suggested revisions for A and C for distracter 87 12 N s p arallelism (B and D say shutdown the reactor).
88 13 M 89 14 N S Add reference for V-04 to package - OK Sam Hansel1 -
please provide a lesson plan objective or a statement thal this is appropriate level of knowledge for the SRO to knov from memory i.e. not minutea. STATEMENT PROVIDED LESSON PLAN OBJECTIVES ADDED BUT NOT DiRECTL' 90 15 N S ON POINT. -OK Good question - but the diagnostic flow chart was provided fo another SRO question. Does this reference give away this question? Direct lookup? DIAGNOSTIC FLOW CHART 91 16 N S WILL NOT BE PROVIDED TO APPLICANTS Why is B palusible? Add names of AOPs to answers.
92 17 N S REVISED B, added AOP names 93 18 N S 94 19 B S I lAdd in controlling procedures to A, B and D MADE 95 20 B s IREVISIONS AS~EQUESTED- O K IVery close to a question on the last NRC exam - replace? NO did not use any questions from last NRC exam. Not SRO level. WROTE TO MATCH THE KIA. NO OTHER SRO LEVEL QUESTION CAN BE WRITTEN TO TEST THIS KIA.
DO NOT MEET CRITERIA FOR SUPPRESSING THE KIA -
96 21 M 97 22 N 98 23 N Can we add in the selectionof an AOP in the distracters?
Meets WA but does not test at SRO level. DONE -delete the 99 24 - N S word "setpoint" in A and C. OK 100 25 - -- M S 1 otal 3.0 0 0 0 0 0 0 0 2 0 0 25 25 Sum F 16.0% FUNDAMENTAL KNOWLEDGE U B Bank = 6 0 UNSAT 84.0% HIGHER LEVEL KNOWLEDGE E M Modified = 5 0 Enhancement required 100.0% S N New = 14 25 SATISFACTORY FINAL Page 2
m 0
m a
Z w gm
Calvert Cliffs RO ES 401-9 RO Exam Ref
- 1. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
- 2. LOD Req Q# LOK B/M/N (FIH)
Stem Cues TIF Focus Partial t!; Minutia #/units 2;- Y/N Q=KIA SROonly NRC 7.
UIEIS 8. Explanation Does not test the cause of the alarm or the efffect of the alarm? May be OK with KIA match analysis? Not a modified question - cannot see difference between original and modified versions. PROVIDE CORRECT VERSION OF THf 1 H 2 X N N? N M S ORIGNAL QUESTION.
A, B and C are not plausible. Add 2 actions to take to 2 EOF 2 H 3 a, b, c ? Y N N U bases to get a 2x2 matrix question. No reference provided.
3 F 2 N Y N B S
, 4 H 2 d N N N B U Does not test the 2nd part of the KIA. Why is D plausible?
See recommended enhancement changes to C and D MAKI
, 5 H 3 N Y N B E CHANGES See recommended enchancements to A and B for parallelisn 6 F 3 N Y N B E of answers. WILL CONSIDER C is not plausible see recommended changes WILL 7 F 3 C ? Y N N E CONSIDER MAKING CHANGE TO C 8 F 2 N Y N B S A and B are direct lookups with reference. Not very piausibk Not a modified question - need to modify the stem and one distracter. - REF ES401 page 7 CHANGE TO BANK -
9 F 1 a, b Y Y N B U CHANGE B TO 72 HOURS - ASK SAM tO F 2 N Y N B S 11 F 4 N Y N N S 12 H 2 a, c N Y N B S 13 H 2 N Y N B S 14 H 3 a, b N Y N B U A and B not plausible A not plausible - no info provided on radiation levels. D not plausible -can rule out D the same way as B + RLEC-2 not plausible. Is this RO level of knowledge? -YES -
DETERMINATION OF SAFETY FUNCTIONS IS RO RESPONSIBILITY. RLEC 2 IS PLAUSIBLE - SEE PLAUSIBILITY ANALYSIS IN EXAM COMMENTS - WILL MAKE RECOPMMENDED CHNAGES TO ENHANCE THIS 15 H 3 a, d Y Y N B E QUESTION 16 H 2 N Y N B S 17 H 3 d N Y N N E Why is D plausible? REVISE D A and B are not plausible VALIDATION RESULTS SHOW A 18 H 3 N Y N B S AND B ARE DISCRIMINATORY
,I9 F N B S Why are A and D plausible? Recommend changes to A and 20 H 3 a, d N Y N B S D. - ADDED PLAUSIBILITY ANALYSIS FOR A AND D.
Why is D plausible? ADDED PLAUSIBILITY ANALYSIS - Se
~ 21 H ~~ 3 ---
~ d ------ N - Y - N ~-B S suggested changes. OK ON CHANGES A is not plausible - entire question is simplistic. CHANGE A 22 H 2 a Y Y N B U TO569F 45-DAY Version Page 3
Calved Cliffs RO ES 401-9 RO Exam 45-DAY Version Page 4
Calvert Cliffs RO ES 401-9 RO Exam 45-DAY Version Page 5
Calvert Cliffs RO ES 401-9 RO Exam 45-DAY Version Page 6
r ml*l "
I mm 2 zz z I
z s >z z I
E Y (12
Calvert Cliffs RO ES 401-9 RO Exam
- 2. LOD
- 3. Psychometric Flaws
- 4. Job Content Flaws II 6.
RIMIN II I I
(1-5)
Partial Job-Link Minutii U unit SRO only NRC 7.
UlElS 8. Explanation A not plausible - no info provided on radiation levels. D not plausible - can rule out D the same way as B + RLEC-2 not plausible. Is this RO level of knowledge? - YES -
DETERMINATION OF SAFETY FUNCTIONS IS RO RESPONSIBILITY. RLEC 2 IS PLAUSIBLE - SEE PLAUSIBILITY ANALYSIS IN EXAM COMMENTS - WILL MAKE RECOMMENDED CHANGES TO ENHANCE THIS IQUESTION Changes made to D added plausibility 3 N B S -
analysis OK 2 N B S 3 N N S I Why is D plausible? REVISED D - OK IA and B are not plausible VALIDATION RESULTS SHOW A 3 N B S AND 6 ARE DISCRIMINATORY OK -
- N B S Why are A and D plausible? Recommend changes to A and D. - ADDED PLAUSIBILITY ANALYSIS FOR A AND D. -
3 N B S OK Why is D plausible? ADDED PLAUSIBILITY ANALYSIS - SE 3 N B S suggested changes. OK ON CHANGES A is not plausible -entire question is simplistic. CHANGE A 2 --N B S I T 0 569F DONE -OK 3 --N N n J
I Need to revise D to be more plausible -what about the condensate pump? You do not address the auto start of the standby condensate pump in c or d. One can assume the standby condensate pump has started. See suggested revisions. REVISE QUESTION TO 80% POWER - REVISE D AS SUGGESTED - ADD PLAUSIBILITY ANALSYIS FOR 3 --N B S D. DONE-OK 4 --N N S Too simplistic -does not test operational implications of leak rate decreasing. REVISE OR REPLACE - REVISED 1
C and D not plausible - C is plausible - see plausibility analysi
- D is not plausible - agreed to revise question to recommended changes if we changed the U to an E. OK
- 2 FINAL 3
9/3/2008 I S Distracters A, B and D seem to test the condition where discharge pressure < some value (125 psig). This is OKfor one distracter - but not all 3. If the pump is deadheaded, mini flow protection must be provided. REMOVE STEM ABOUT LOW FLOW RATES - THEN DISTRACTERS A and B ARE OK - REVISE C -ADDED DISTRACTER ANALYSIS TO JUSTIFY THE PLAUSIBILITY OF "C" Page 2
Calvert Cliffs RO ES 401-9 RO Exam I
4.Job Content Flaws 5. Other 6.
P# - - B/M/N I -
Job-Link -
- Iunit! SRO only NRC 7.
U/E/S 8.Explanation A, C and D are just not plausible. A and C were plausible.
RO/ATC not allowed to do much of anything but See 29 - N S proposed revisions. Accepted proposed revisions OK -
simplistic - A, and Dare not plausible ADD PLAUSIBLITY 30 B S ANALYSIS- ADDED OK -
Question does not meet critieria for significant modification -
stem has not be modified. Does not match WA - generic radiation control - REPLACE WITH EWP QUESTION -
REPLACED New question: "D" is not plausible 50 -
REM does not correlate to a plausible error. CHANGED 31 B S "D"-OK A and C are not plausible - A - would not dilute to criticallity under any conditions. THIS IS THE CORRECT PROCEDURE FOR PHYSICS TESTING AFTER REFUELING - ONLY COMING OUT OF OUTAGE WITH A FRESH CORE. C -. REPLACE C WITH PROPOSED 32
- B S DISTRACTER OK 33 I N__ -. S .Revise stem to include procedure completion. DONE OK -
34 N S No reference to be provided? CORRECT - OK n E 35 36 37 N 38 N B S 39 N B S 40 N B S 41
- N B S 42
- N B S How is the "emergency backup" part of WA tested? REVISE 43 N B S QUESTION - REVISED OK -
Should be scored a fundamental knowledge? Memorization 44 N B S 2 signals. CHANGE TO F LoK - OK 45 N 46 N B S 47 N B S See proposed revision to distracter C WILL MAKE CHANGE 48 N B S OK 49 N M S Distracter B is somewhat implausible. See proposed revisior 50 N B S WILL MAKE SUGGESTED CHANGES - DONE OK -
51 N B S FINAL 9/3/2008 Page 3
Calvert Cliffs RO ES 401-9 RO Exam P# LOK 1.
- 2. LOD (1-5) -
- 3. Psychometric Flaws I 4. Job Content Flaws Ref (FIH)
-- Stem Focus Why is B plausible - CAC inlet valve is shut? UNDER CERTAIN CIRCUMSTANCES, THERE IS A REQUIREMEN FOR FLOW TO BE THROUGH SOME SYSTEMS FOR CONTAINMENT INTEGRITY - IF IN MODE 5, THEN B I S 52 H 3 S CORRECT (CONTAINMENT CLOSURE) -OK See proposed enhancements. No need to provide release form if changes are made. Do we need to give them the circ water pump alingment? - made suggested chnages circ -
53 H 3 S water pump alignment is not required -OK 54 F 3 S Why is B plausible? See proposed revisions - ACCEPTED 55 H 3 S REVISIONS -OK 56 F 3 57 H - 4 S Distracters A and B involve the same error. Change A to rei level lowers at 3 inches per minute. Reorder distracters as indicated. Which one is more plausible. MAKE REVISIONS 58 H 3 S AS RECOMMENDED - DONE OK -
59 H 3 S Why are A and B plausible? See plausibilityanalysis. OK See recommended changes to improve question - MADE CHANGES TO BAND C UNABLE TO READ SUMP LEVE 60 H 3 S SO D STANDS AS IS OK -
Why is D plausible? PZR TEMP IS USED IN ACCIDENTS 61 F 2 S Should be memory level. CHANGE TO MEMORY LEVEL 0 62 H 2 S See suggested change - more plausible. Requires applicant to know the answer to the 2nd Dart of the auestion. WILL 63 F 3 S MAKE THIS CHANGE DONE OK -
Why is " B plausible? See recommended change to A and E 64 H 3 S CHANGES MADE OK -
A is not plausible if D (which is the answer) is listed. This question has not been signficantly modified. Only distracter I has changed (was 4 CACs instead of 3 CACs - REVISE A REPLACED QUESTION New Question: state the title of 65 F 2 S -
AOP-9A DONE -OK Change to fundamental level of knowledge? MORE COMPLICATED THAN FIRST THOUGHT -AIR SUPLY COMES FROM DOWN STREAM - COMPLEX 66 H 3 S INTERACTION -THIS IS HIGHER COGNITIVE - OK Why is C plausible? C IS PARTIALLY CORRECT - BUT NC SUFFICIENT - ADD IN Plausibility analysis for C DON E -
67 F 2 S OK A is not plausible. Does not test part b of WA - mandatory -
modified A - changed each answer t o test part B of KIA.
68 H 3 FINAL 9/3/2008 Page 4
Calvert Cliffs RO ES 401-9 RO Exam Q#
I.
LOK 2izr 3. Psychometric Flaws 4. Job Content Flaws Ref Req
- 5. Other 6.
BlWN
( W Stem Focus Cues T/F E,' Partial t:i Minutia # I units Back-ward YIN Q=WA SROonly NRC 7.
UlElS 8. Explanation A and D not very plausible. See suggested revisions.
Alternatively, test if the isolation valve closes automatically. A was changed but D was not changed. Either change D 01 draft plausibility analysis. ADDED PLAUSIBILITY 69 F 4 ? Y N N S ANALYSIS OK -
D can be argued as true - can also be not plausible - see suggested change MAKE SUGGESTED CHANGE DONE -
,70 F 3 N Y N B S OK 71 H N Y N N S Change to fundamental level of knowledge? THEY CONVINCED ME THAT THE ONLY WAY TO ANSWER THIS QUESTION IS TO KNOW WHAT SYSTEMS ARE CAPABLE OF FEEDING INTO THE SFP - REQUIRES 72 H 3 N Y N N S SYSTEM KNOWLEDGE 73 F N Y N B S Why is "C"plausible? Revised C OK -
74 H N Y N B S 75 H 3 N Y N B S Total 75 2.7 0 0 0 0 0 0 0 0 1 6 0 75 75 75 Sum FINAL 9/3/2008 Page 5