ML030580203

NRC Comments on the Operating Test and Written Examination Including Forms ES-401-9 and ES-401-10 for the Duane Arnold Examination - November 2002

ML030580203
Person / Time
Site:	Duane Arnold
Issue date:	11/18/2002
From:	Lanksbury R NRC/RGN-III/DRS/OLB
To:
References
50-331/02-301
Download: ML030580203 (14)

Text

NRC COMMENTS ON THE OPERATING TEST AND WRITTEN EXAMINATION INCLUDING FORMS ES-401-9 AND ES-401 -10 FOR THE DUANE ARNOLD EXAMINATION - NOVEMBER 2002

COMMENT AND RESOLUTION FOR THE DUANE ARNOLD NOVEMBER 2002 PROPOSED OPERATING TEST JPM COMMENTS

1) COMMENT: The JPM title should be removed from the INITIAL CONDITIONS / INITIATING CUES page of the JPM's.

RESOLUTION: Done

2) COMMENT: In JPM A.1.b Respond to a RBM Rod Block annunciator. Change this JPM so that the core limit for are between the Technical Specification limit and the Station Administrative limit.

RESOLUTION: Done

3) COMMENT: In RO JPM A.4 Site Evacuation While Escorting A Visitor, this JPM should be noted to be time critical. The visitor must be evacuated within 30 minutes of the alarm.

RESOLUTION: Done

4) COMMENT: In JPM B. 1. e Containment vent irrespective of radioactive release, shouldn't this JPM be time critical?

RESOLUTION: No the activity would be limited by containment pressure rather than a time limit.

SCENARIO COMMENTS Only a few requests that additional information be added to the Expected Student Response selections, which was done.

FOR WRITTEN EXAMINATION COMMENTS SEE FORM ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ IBack- Q= SRO U/E/S Explanation

[DiFocus Dist. J unitsjward

[Link K/A Only 001 H 2 V V U To eliminate distracters #1/'B' and #2/'C' verses answer is recall of which F S valves close on turbine trip (also, are these valves normally closed following a turbine trip or do they change position). The K/A specifies "reasons" for Ext. steam/MSR isolation as applies to Turbine Trip (i.e.

prevent turbine OS following trip), question as written illicit's information about "what happens" and "what causes it to happen" following a turbine trip. RESOLUTION: Author will provide a new question that meets KA.

(DONE new question is at fundamental LOK) 002 F <3 S 003* H 2 S 004 H <3 S 005 H 2 S 006* H 3 E First sentence says see "Support Material Booklet," stem says see next S page. Answer options refer to valve control switch numbers, due to angle of picture and reproduction, these numbers are not clear. Ensure applicants understand panel portion pictured represent conditions after all conditions in stem are met. RESOLUTION: Author will make changes to stem and provide clearer picture. (DONE)

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are desiginated SRO-onlV (K/A and license level mismatches are unacceptable).

6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

DAEC RO Written Examination Review Worksheet Page 2 of 9 Form ES-401-9 (R8, Sl)

Q# 1.

LOK 2.

LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation

__ __Focus __ _Dist. _ Link __units ward K/A Ony_ __ _ _ _ _ _ _ _ _ _ _ _ _ _

007 F 2 LI U LOK, question = memory of loss of feedwater heating especially since H S distractors not plausible due to valve positions, i.e. valves in their normal positions (for the conditions identified) with no loss of heating.

RESOLUTION: LOK = H from stem info must identify type of failure, and must demonstrate good system knowledge to eliminate at lease 2 of 3 distracters.

008 F 2 S 009* F 2 V E Justification for answer states "PCPL....determined to be 53 psig and 95 S ft water" does stem need to include a containment level to be correct?

RESOLUTION: No action to be taken would be independent of water level per EOPs.

010 F 2 S 011 H 3 E Separate questions 11 & 12 so there are not two ATWS questions next S to each other i.e. make 12 number 37. RESOLUTION: Author will do.

(DONE) 012* H 3 S 013 H 3 S 014 F 2 S 015" F 2 U LOK, although the basis for the correct answer assumes higher level H S knowledge to combine the use of the fire pump and the RHR system, all of the distractors can be eliminated by the simple recall that each require unpowered pumps or fans to accomplish their mission. RESOLUTION:

After discussion with Author agreed that more knowledge than indicated above would be needed to eliminate distracters and to pick the correct answer.

016 F 3 S 017 H 3 V E K/A = Knowledge of annunciators alarms and indications / and use of the S response instructions. Add ann xx-xx in alarm? Position that ARP required actions being performed are immediate operator actions required to be performed from memory. Procedure pages attached are insufficient to make that determination. RESOLUTION: After discussion with author agreed operator training would allow this question to be answered. Will add reference to appropriate annunciators in stem.

(DONE) 018" F 2 S

DAEC RO Written Examination Review Worksheet Page 3 of 9 Form ES-401-9 (R8, Si)

Q# 1.

LOK 2.

LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

(F/H) (1-5) Stem ICues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus J_____Dist. JLink units ward KIA Only] _

019 H 3 U LOK = Fundamental memory recall on impact of transfer actions.

F S RESOLUTION: Will be considered to be a fundamental question.

020 H 3 V U If an applicant assumed SBGT running due to the transient, does an in S progress offsite release become feasible and distractor #3 ('D') true.

Change stem from off-site release to unmonitored or ground level release. COMMENT Question refers applicant to Support Material Booklet. A picture of the PCIS mimic is presented and referenced to question #20. The applicant is required to identify the status of the systems listed in the distractors, due to angle of picture and reproduction, portions of picture are not clear enough to read the valve label tags or mimic system labels. RESOLUTION: Author will change stem to refer to an "unfiltered release" or "release done unfiltered" etc. and will clarify picture.(DONE) 021* H 2 S 022 F <3 U LOK F verses H. RESOLUTION: H based on need to combine H S knowledge of service air and CRD to answer question also was H on 1999 NRC exam.

023 H 3 S 024* H <3 S 025 H 3 S Question 27 provides a cue to question #25, where fact that EOP 2 directs entry into EOP 1 goes unstated and is part of reasoning/comprehension that make question #25 higher level.

RESOLUTION: So many EOPs direct entry to EOP 1 that this was not considered a cue.

026 H <4 V U If only fact applicant remembers is -23" penalty, then choices down to 'A' S and 'D' suggest change distractor #2/'C' to 177" RESOLUTION : D is a good distractor because it is possible that under the conditions of the stem an operator could conclude none of the signals are reliable.

E Change stem for "Which value" to "What value"(DONE)

S 027* H 3 V U Question 27 provides a cue to question #25, where fact that EOP 2 S directs entry into EOP 1 goes unstated and is part of reasoning/comprehension that make question #25 higher level.

Change stem to "at what torus water level is scram required?"

RESOLUTION: SAME AS QUESTION #25 028 F <3 S

DAEC RO Written Examination Review Worksheet Page 4 of 9 Form ES-401-9 (R8, Si)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus __ Dist. Link ____ nits Iward- __________________ON______

029 F <2 V U Due to fact that physical location recognizable from detectors noun S name, credilility of all distractors significantly diminished. Distractors A &

B. (LOD) (Will applicants have a copy of EOP-3} RESOLUTION:

Distracter found to be credible based on their plant locations.

030* F 3 V U Distractor #3/'D', applicant can make an unstated assumption that is not S contradicted by stem. [i.e., MSL Hi-Hi Rad causes a GroupI Isol which causes entry into EOP 1 which requires verification of isolations.) (Will applicants have a copy of EOP-3)

Suggest change stem to include "and direct entry into EOP-3" RESOLUTION: At DAEC this rad level will not cause a SCRAM and therefor no EOP1 entry.

031 F 2 S 032 F 3 S 033* H 2 V E Given the panel picture the stem contains more information than is S needed, or too much needless information, remove info about status of Group 4 PCIS and let applicants figure this out from panel picture.

RESULUTION: will remove unnecessary information.(DONE) 034 H <2 V U Given that the stem states: Mode Switch in "Refuel" and "control rod S withdrawal," any distractor that does not contain something about the One-Rod Permissive lacks credibility. (include in distractor 'D' something about Rod Out Permissive. See question #086) Note on A, One Rod out

& following fuel movement sheets RESOLUTION: Will make suggested change. (DONE) 035 H 3 S Note: Explanation of distractors 2 and 3 not clear. RESOLUTION:

Clarified in discussion with author 036* H <3 S 037 F <3 S 038 H <4 S 039* H 2 S 040 H <4 U LOK, applicant has to analyze the conditions which result in a Scope F S Tube Lock-up verses the identified Elect/Mech stops and determine which applies for the given conditions. RESOLUTION: The question will be considered fundamental, the fact that the stem says this is an instantaneous fail to maximum makes it memory.

041 H 3 S

DAEC RO Written Examination Review Worksheet Page 5 of 9 Form ES-401-9 (R8, S)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD I_ _ A (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus 1 Dist. Link units ward K/A Ony_

042* H 3 S 043 H 2 S 044 F 3 S 045* F 2 S 046 F 2 S 047 F 2 S 048* F <2 U If IPOI 5 is an off normal procedure then those immediate operator H S actions are required to be performed from memory, thus LOK = F. Giver the Support Material Booklet item for question #48 that clearly shows SRM detectors have not been inserted into core, LOD is <2 RESOLUTION: These actions are not required to be performed from memory, the candidate will need to analyze information in the stem and the picture provided to answer the question.

049 F <2 V U Can not consider distractors #1/PA' and #3/'B' as credible for S discriminating competent operator. Is RWM still enforcing @ 85% power, plus RWM does not interface with APRM power, RBM does.

RESOLUTION: Author will change distractors #1/'A' and #3/'B or explain the tie between the RWM an the APRMs. (DONE distractors changed to refer to RBM rather than RWM) 050 H 2 S 051

• H <3 S 052 F 3 U LOK = F, straight memory/recall for the identified failure. The stem coul H S be reduced to 1) statement of annunciator in alarm and 2) chose correct impact on RCIC. RESOLUTION: OK as is candidates will need to analyze information in the stem to eliminate answers that would be correct if RCIC was operating in a different mode.

053 H 2 S 054* H 3 S 055 H 3 V U Picture in Support Material Booklet shows a failure to restore the Group E 3 isolation {N2 supply to SRVs & MSIVs). Is distractor #1/A' also true if S N2 is NOT available {relief function verses safety function)?

RESOLUTION: This was an editorial error by the author the word "relief" should be "safety" when this change is made question will be sat.

I (DONE)

DAEC RO Written Examination Review Worksheet Page 6 of 9 Form ES-401-9 (R8, Si)

2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q4 1.

LOK (F/H)

LOD (1-5) Stem Cues T/F t Job- Minutia I

1. / Back- Q= SRO

[RO U/E/S Explanation Focus Dist. Link units ward K/A Only 056 H 2 E LOD rated as 2 because of lower probability of selection of distractors.

S What would the impact be of changing distractor #1/PA' to read "Drywell pressure dropped below 1.8 psig but currently is indicating 2.1 psig."

RESOLUTION: OK as is author provided more insight as to why distracters were credible.

057* F 3 V U LOK = F, straight memory/recall for the identified failure. E on this issue H S because previous NRC exam. Why is 'D' not correct, reference AOP 358 "Automatic Actions" lists 'A' SBGT Initiation, all other items on this list are true for the given condition. RESOLUTION: Not memory the correct answer is not as obvious as first though candidate will need to understand PCIS logic. Also distractor "D" references SBDG not SBGT.

058 H 3 V U Agree distractor #3/'D' is not a correct answer because it states LLS is S cycling normally and PSV 4407 is not open. If I read things correctly distractor #2/'C' is correct except for the fact that PSV 4407 is not open.

1. 2/'C' does not contain enough information to prevent the applicants from choosing it and being correct though not completely correct for the conditions. Bottom line 'A' is the most correct but 'C' is also correct though it does not express the complete situation. RESOLUTION:

Author will add the phrase "and LLS is cycling normally." to distractor C.

(DONE) 059 H 2 E Stem should state "As reactor power is raised to full power,..."

S Distractor #1 and #3 minimally credible, suggest #3 read "When Turbine load reaches 550 MWE reactor pressure will continue to increase and an RPS Scram will occur." RESOLUTION: Author will do. (DONE) 060* F 2 S 061 F 2 S 062 F 2 V U Find distractors #2 and #3 minimally credible due to placing SBGT in S manual then expecting an action to initiate an auto start. Replace one of the two. Suggest "trip RB ventilation and SBGT auto start on bldg Dp."

Also add to 'A' "Take or verify HS 5814B in AUTO and ."for symmetry.

RESOLUTION: Distractors 2 and 3 ok since similar to actions taken in other situations. Author will add phrase "Take or verify HS 5814B in AUTO" to distractor A.(DONE) 063* F 2 S 064 F 2 V U Find distractors #2 and #3 minimally credible due to placing Governor S Mode (Droop) Switch to "PARALLEL" in both cases with a "loss of off site power" given in the stem. Suggest 1) dispatch an operator to investigate 2) check gov oil level, 3) fuel supply, 4) ok as long as engine is not black stacking. RESOLUTION: Author will replace distractor C.

(DONE)

DAEC RO Written Examination Review Worksheet Page 7 of 9 Form ES-401-9 (R8, Si)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD I I I (F/H) (1-5) Stem Cues IT/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus J Dist. Link units ward K/AIOnly 065 F 2 S 066* H 3 S 067 H 2 S 068 H 2 S 069* H 2 V U LOD <2 - A & B credibility. Possible fix, change stem to "None selected S and rod is at position 36, uncoupled and stuck." RESOLUTION: Author will make suggested change. (DONE) 070 F 2 S (what input signal does RBM use to begin enforcing) ANSWER: APRM 071 H 3 S 072' H 2 V U To correct 1) remove statement from stem about outside shaded area, E and 2) define what can be done without OSS permission (is there S enough so this answer not automatic). RESOLUTION: Yes there are a number of actions that once directed can be repeated without OSS permission (torus cooling, water level control, etc.). therefor answer is not given away. Author will remove statement from stem about outside shaded area. (DONE) 073 H 3 E Remove one of the "started" from 'D' RESOLUTION: Will do. (DONE)

S 074 F 2 U LOK=H Need to analyze information given in stem to answer.

H S RESOLUTION: Author agrees will change to higher order.

075* H 2 S 076 H 2 V U Find distractors #1 and #2 minimally credible, no possible impact on S conditions described for B Condensate Pump. Clean up scram issues.

RESOLUTION: Distractors credible because there is a feed reg valve at the discharge of each feed pump then a common feed header at DAEC.

077 H 3 U LOK = F, Stem can be reduced to "trainee closes breaker when F S synchroscope at 3 o'clock position. What happens? Then straight memory/recall, no analysis. RESOLUTION: Author agrees this question fundamental.

078* H 2 S 079 H 3 S

DAEC RO Written Examination Review Worksheet Page 8 of 9 Form ES-401-9 (R8, Si)

11. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia 1 #/ Back- Q= SRO U/E/S Explanation Focusl _ Dist. Link lunits ward K/A Only 080 F >4 V U Knowledge of power supply not an AOP immediate action to be S performed from memory. Unless can show a more specific objective would have to consider minutia. Question requires the recall of knowledge that is too specific for the closed reference test mode.

RESOLUTION: Operators are trained to understand electrical distribution system at a level to allow answering of this question.

081 H 3 V U K/A "and part" states "use procedures to correct, control, or mitigate the S consequences of those abnormal conditions." Second part of answer does not meet this requirement, just says "TS directs actions for 2r' cnm operability..." Change answer to plant shutdown is required by TS.

RESOLUTION: Will replace reference to TS with a procedure reference.

(DONE) 082 H 3 S 083 H 3 S 084* F 3 S 085 F 2 S 086 H 2 E Diagram not required and if indications are used on question #034 will be S a conflict. RESOLUTION there will be no conflict given the way question 34 was changed.

087* F <2 V U 1) Unless can show a more specific objective and detailed training given S on IE specifics would have to consider minutia. Question requires the recall of knowledge that is too specific for the closed reference test mode. 2) If can resole #1 then straight memory. 3) LOD, distractors minimally credible, suggest 1) pressure oscillations observed at higher power levels, 2) on #3 remove GEMAC, leave it at all level instruments will deviate from each other. RESOLUTION: Will replace distractor D with something more credible. (DONE) 088 F 3 S NOTE: picture not clear on where 'E' is pointing. RESOLUTION: OK because a color picture will be used on the actual exam.

089 F 2 V U The 2 part K/A states "Ability to explain and apply system limits and precautions." Question meets 2"d part but not first. RESOLUTION:

Author will change answers to include explainations.

090* F 2 U LOK=H, question requires recall the requirement then analyze H S RESOLUTION: Author agreed LOK=H based on need to do analysis to answer.

091 F 2 S

DAEC RO Written Examination Review Worksheet Page 9 of 9 Form ES-401-9 (R8, $1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F ICred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation FocusI j _ [Dist. Link units ward K/A On___

092 F 2 S 093 H <2 V U LOD is <2 due to ability of distractors to discriminate competent S operator. RESOLUTION: Can not develop a question with LOD >1 author will pick new KA and provide new question. (DONE) 094 H 2 S 095 F 2 S 096 F 2 S 097 F 3 S 098 F 2 V E On 1999 RO NRC Exam. Distractor #2, looking at reference seems S reason for 'C' is to allow 'B' under all conditions. Plus A also correct.

RESOLUTION: EOP bases clearly makes B wrong. Author will take another look at distractor A to see if it needs changing. (DONE distractor A changed) 099 F 2 S 100 F 2 S Currently LOK H Currently "U's" =

Currently "E's" =

• Questions used for initial review

ES-401 SRO ONLY QUESTIONS 2 Form ES-401-9 (R8, S)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 Q# LOK LOD [ t I 1 IrI 7.

(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ IBack- Q= SRO U/E/S Explanation Focus [ Dist. Link units ward K/A On__

33 H 3 V U Distracters 2/b and 3/d are "correct actions" (the question) even though S they do not include the action of completing the Group 4 isolation.

RESOLUTION: Distractors OK because Group 4 isolation will prevent RHR start.

38 H 3 V E The stem asks for "required actions" the listed correct action is an S optional action for the OSS. Change to "Which of the following would be a correct action to take and the a correct bases for the action?"

RESOLUTION: EOPs do REQUIRE a scram under these conditions.

39* H 2 V U It is hard to believe that a Tech Spec Bases would call out a work S approval level therefor diatracters b and d can be eliminated. Is there a more believable reference we could use? RESOLUTION: Author will replace reference to Tech Spec Bases with a procedure reference.

(DONE) 42 H 4 S 48 F 3 S 50* H 3 S 51 H 2 V V U Wouldn't an RO be expected to know enhanced level monitoring was S necessary under these conditions? RESOLUTION: Question will be Ichanged to test ability to select correct procedure. (DONE) 60 F 3 V U Wouldn't an RO be expected to know this same level of information if it S is from a annunciator procedure? RESOLUTION Because this question requires the SRO to decide on a course of action in an emergency the question was accepted.

62* F 2 U This appears to be a simple lookup question for 2 trains of SBGT out of H S service. RESOLUTION The question was accepted because the stem contains information that must be evaluated to determine the SBGT system status.

64 H 2 S

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-IMinutia / [Back- SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 72 H 4 S 73* H 3 S 74 H 2 S 75 H 2 S 76* H 3 I V V U This does not appear to be an SRO level question. This level of system S knowledge could also be expected of ROs. Also answers do not KA no "EXPLANATION" provided. RESOLUTION: A new SRO question that addresses KA will be provided. (DONE) 81 H 3 S 82 F 2 S 83* F 2 S 85 F 2 S 86 F 2 S 89* H 2 S 90 F 2 S 91 F 2 U LOD for this question is too low. This is a simple lookup question with S little discriminating value. RESOLUTION Because this is a judgment V call and because this question is a bank question that was excepted for the 1998 exam the question was accepted.

94* F 2 S 98 F 2 S 37 H 3 S

• Questions used for initial review

( ( (

ES-401 i-,lf.,,lr'rlrrl tiT I.,,ll*il/'*/"il'fltt It #/1^

.. . , I rV/Ab Form ES-401 -10 Tier / Group IRandomly Selected K]A-Reason for Rejectlon i, Z, y 2 2,

,7 NUREG-1021, Revision 8, Supplement 1 46 of 46