ML090290231

NRC Comments on Draft Licensee-Developed Exam (Written & Operating Tests) (Folder 2)

ML090290231
Person / Time
Site:	Calvert Cliffs
Issue date:	10/30/2008
From:	Brian Haagensen Operations Branch I
To:	Calvert Cliffs
Hansell S
Shared Package
ML082600339	List: IR 05000317/2008302 ML082760115 ML083430602 ML083580024 ML090220477 ML090260359 ML090290231 ML090290235 ML090290239 ML090290246 ML090490182
References
50-317/08-302, 50-318/08-302
Download: ML090290231 (17)

Text

Written Exam Inventory Comments:

1. The revised sample plan does not contain WA descriptions. Please submit a sample plan that has these descriptions included, not just the WA numbers. Please use the format in ES 401-1/2 forms.

ES 401 page 4 Enter the KIA statement numbers, a brief description of each topic, the topics importance ratings for the license level of the exam (use the RO and SRO ratings for the RO and SRO-only portions, respectively), and the point totals (system, category, group, and tier) on the examination outline. The proposed point totals for each group and tier must match the number specified on Forms ES-401-1 and ES-401-2, as applicable.

ANS: The form submitted is not a revised ES401-1/2. The original ES401-1/2 remains valid with no changes to any WAS.

2. Please include nomenclature for the learning objectives. The objective number alone does not provide enough information for our review.

ES401 page 8 A technical reference and a cross-reference to the facility licensees examination question bank, if applicable, shall be noted for every question. If the facility licensee has a learning obiective applicable to the question, it should also be referenced. However, the absence of a learning objective does not invalidate the question, provided that it has an appropriate WA and technical reference. Refer to ES-201 for additional instructions regarding documenting the source of questions on facility-written examinations.

ANS: Adding learning objectives will take about 10 hours1.157407e-4 days <br />0.00278 hours <br />1.653439e-5 weeks <br />3.805e-6 months <br /> of time. Do not take this time.

Instead, I will ask you to provide lesson plan objectives if we challenge the validity of any questions.

3. Please provide copies of any modified questions that you have used in this exam. As an alternative, you may describe the changes that you made to the original bank question for all modified questiions.

ES40 1-5 requires:

Question Source:

Bank #

Modified Bank #

(Note changes or attach parent)

New A N S : This information will be provided in the hard copy submittal.

4. Please provide a history of the last time the question was use on an NRC exam - if applicable.

ES401-5 requires:

Question History: Last NRC Exam A N S : We did not take any questions fiom the past 2 NRC exams.

5. Please provide the cognitive level of each question or a guide that allows us to determine your cognitive level numbering scheme vs. our cognitive level categories in ES 401-5.

Question Cognitive Level:

Memory or Fundamental Knowledge Comprehension or Analysis ANS: I discovered the key in the electronic files. This is not a problem.

6. You are not required to submit a plausibility analysis -but if you do not, we will have to do it ourselves. This will add time and effort to the review. We will have to establish plausibility for each distracter and document this in our form for those questions which are not obvious.

To facilitate the review process, examination authors should consider providing a brief explanation of why the answer is correct, and each of the distractors is plausible but incorrect. This optional practice increases the efficiency of the examination review process and promotes the detection and correction of problem questions before the examinations are administered.

I did not provide this comment to Calvert Cliffs.

7. Please include the WA nomenclature to the questions to make review easier. You are not required to add WA nomenclature to the question. But if you do not add this nomenclature, we will have to do add it ourselves. This will add time and effort to the review.

A N S : We will add this information.

Calvert Cliffs

2. LOD 3

3 3

3 3

2 3

3 3

3 SRO ES 401-9 SRO EXAM

~~

~~

3. Psychometric Flaws

4. Job Content Flaws Ref Req

5. Other

6.

7.

BlMlN UIEIS Q=WA SROonly

7. Explanation Add IAW OP-7. Do they have to know this from memory? NI replace question Replaced question - '%"is not plausible -

see suggested enhancement Calverl believes the suggested enhancement may be a second correct answer. They want to use the original distracter ' X "

because this was the correct answer prior to a recent change to the procedure. Draft a plausibirity analysis to N

Y Y

M S

establish the case for why A is plausible. - DONE Ok This is not an SRO level question -can write an SRO level question of you test tech specs - MADE CHANGES TO ADD N

Y Y

B S

Why are C and D plausible? C IS OK. REVISED D.-

~

Why provide section 3.0 of tech specs? We normally expect applicants to know this section. OK - REMOVED TS Job-Minutia #I units Back-ward Cues TIF Partial Link Stem Focus N

Y Y

B S

TECH SPECS TO DISTRACTERS - OK Y

Y Y

N S

SECTION 3.0 from handout N

Y Y

N S

Change to "directed by the CRS" - REVISION OK Why would EOP-7 be a plausible choice? See suggested changes to address issues. Diagnostic flow chart was not provided with reference package. RECOMMENDED CHANGES TO QUESTION - WILL GET BACK TO ME-Mad Changes to distracters C and D as discussed. Add I would consider this question to be written at the higher cognitive level. Why address USFAR 14 conditions in stem?

What does this mean? REMOVE REFERENCE TO UFSAR.

Y Y

Y N

S plausibility analysis. ADDED -OK N

Y Y

B S

CHANGE TO "Higher cog level" DONE C and D not plausible - shutdown margin not challenged unde these conditions. See proposed revisions to the question.

This should be higher level of knowledge. CONSIDER REVISING QUESTION TO MAKE C AND D MORE N

Y Y

B S

PLAUSIBLE - Revised C and D to make more plausible.

B. C and D are not plausible as written (when isolated from A)

A is important - B, C and D are minor additional actions. See suggested changes to improve plausibilty using same information. Also note that another question provided a copy of the RAD Waste discharge permit - eliminate this reference or verib that this does not give this away. REVISE QUESTION TO CHANGE - Revisions made as requested.

N Y

Y N

S Need to add the word "verify" to distracter B N

Y Y

N S

1::I 85 10 45 Day 1 I1 3/2009 Page 1

SRO EXAM SRO ES 401-9 Calvert Cliffs 1/13/2009 45 Day Page 2

m W m a

Calveti Cliffs SRO ES 401-9 SRO EXAM FINAL Page 1

Calvert Cliffs Q#

86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 otal F

SRO ES 401-9 tesi Q#

11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 SRO EXAM t:i I

Minuti; 0

0

4. Job Content Flaws

3. Psychometric Flaws

2. LOD I

6.

BlMlN -

M N

M N

N N

N N

B B

M N

N N

M 25 6

5 14

7.

UIEIS

7. Explanation Replace this question. A, C and D are not plausible. Question is not at the SRO level. WA is very broad. REPLACED QUESTION - new question: add A.8,C.D above distracters to correlate matching. Should this be F LOK?

Distracters A, B, and D not plausible (A=l right, B=1 right, C=2 right). See suggested revisions. ACCEPTED S

REVISIONS - OK SRO level question? Why not test what you have to do about it? Stop the startup - continue etc? REVISED QUESTION -

see suggested revisions for A and C for distracter s

p a

rallelism (B and D say shutdown the reactor).

Partial -

Back-ward -

!I unit: -

S Add reference for V-04 to package - OK - Sam Hansel1 -

please provide a lesson plan objective or a statement thal this is appropriate level of knowledge for the SRO to knov from memory - i.e. not minutea. STATEMENT PROVIDED LESSON PLAN OBJECTIVES ADDED BUT NOT DiRECTL' S

ON POINT. -OK Good question - but the diagnostic flow chart was provided fo another SRO question. Does this reference give away this question? Direct lookup? DIAGNOSTIC FLOW CHART WILL NOT BE PROVIDED TO APPLICANTS Why is B palusible? Add names of AOPs to answers.

REVISED B, added AOP names S

S S

S I

lAdd in controlling procedures to A, B and D MADE s

IREVISIONS AS~EQUESTED -OK IVery close to a question on the last NRC exam - replace? NO did not use any questions from last NRC exam. Not SRO level. WROTE TO MATCH THE KIA. NO OTHER SRO LEVEL QUESTION CAN BE WRITTEN TO TEST THIS KIA.

DO NOT MEET CRITERIA FOR SUPPRESSING THE KIA -

Can we add in the selection of an AOP in the distracters?

Meets WA but does not test at SRO level. DONE -delete the word "setpoint" in A and C. OK S

25 Sum S

1 0

U E

S 0

0 2

0 0

B Bank =

M Modified =

N New =

3.0 0

0 16.0% FUNDAMENTAL KNOWLEDGE 84.0% HIGHER LEVEL KNOWLEDGE 0

UNSAT 0

Enhancement required 25 SATISFACTORY 100.0%

FINAL Page 2

Z wg m

m 0

m a

Calvert Cliffs

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

Ref B/M/N Req

2. LOD

1.

Q# LOK (FIH)

NRC

7.

UIEIS

8. Explanation Stem Cues TIF Partial t!; Minutia #/units 2;-

Y/N Q=KIA SROonly Focus Does not test the cause of the alarm or the efffect of the alarm? May be OK with KIA match analysis? Not a modified question - cannot see difference between original and modified versions. PROVIDE CORRECT VERSION OF THf 1

H 2

X N

N?

N M

S ORIGNAL QUESTION.

A, B and C are not plausible. Add 2 actions to take to 2 EOF 2

H 3

a, b, c

?

Y N

N U

bases to get a 2x2 matrix question. No reference provided.

3 F

2 N

Y N

B S

, 4 H

2 d

N N

N B

U Does not test the 2nd part of the KIA. Why is D plausible?

, 5 H 3

N Y

N B

E CHANGES See recommended enhancement changes to C and D MAKI See recommended enchancements to A and B for parallelisn C is not plausible see recommended changes WILL 6

F 3

N Y

N B

E of answers. WILL CONSIDER 7

F 3

C

?

Y N

N E

CONSIDER MAKING CHANGE TO C 8

F 2

N Y

N B

S A and B are direct lookups with reference. Not very piausibk Not a modified question - need to modify the stem and one distracter. - REF ES401 page 7 CHANGE TO BANK -

9 F

1 a, b Y

Y N

B U

CHANGE B TO 72 HOURS - ASK SAM tO F

2 N

Y N

B S

11 F

4 N

Y N

N S

12 H

2 a, c N

Y N

B S

13 H

2 N

Y N

B S

14 H

3 a, b N

Y N

B U

A and B not plausible A not plausible - no info provided on radiation levels. D not plausible -can rule out D the same way as B + RLEC-2 not plausible. Is this RO level of knowledge? -YES -

DETERMINATION OF SAFETY FUNCTIONS IS RO PLAUSIBILITY ANALYSIS IN EXAM COMMENTS - WILL MAKE RECOPMMENDED CHNAGES TO ENHANCE THIS RESPONSIBILITY. RLEC 2 IS PLAUSIBLE - SEE 15 H

3 a, d Y

Y N

B E

QUESTION 16 H

2 N

Y N

B S

17 H

3 d

N Y

N N

E Why is D plausible? REVISE D 18 H

3 N

Y N

B S

AND B ARE DISCRIMINATORY A and B are not plausible VALIDATION RESULTS SHOW A

, I 9 F

N B

S Why are A and D plausible? Recommend changes to A and Why is D plausible? ADDED PLAUSIBILITY ANALYSIS - Se A is not plausible - entire question is simplistic. CHANGE A 20 H

3 a, d N

Y N

B S

D. - ADDED PLAUSIBILITY ANALYSIS FOR A AND D.

~ 21 H

~~

3

~

d N

Y N

B S

suggested changes. OK ON CHANGES

---

----- ~-

22 H

2 a

Y Y

N B

U TO569F RO ES 401-9 RO Exam 45-DAY Version Page 3

Calved Cliffs RO ES 401 -9 RO Exam 45-DAY Version Page 4

RO ES 401-9 RO Exam Calvert Cliffs 45-DAY Version Page 5

Calvert Cliffs RO ES 401 -9 RO Exam 45-DAY Version Page 6

z s I E Y (1 2 I

ml*l I

m m 2

z z z

> z z

r

Calvert Cliffs RO ES 401-9 RO Exam

2. LOD (1-5) -

3 2

3 3 -

3 3

2 3

3 4

1 2 -

3

4. Job Content Flaws I

6. I I

3. Psychometric Flaws I RIMIN I I

NRC

7.

UlElS

8. Explanation SRO only A not plausible - no info provided on radiation levels. D not Partial -

U unit -

Job-Link Minutii plausible - can rule out D the same way as B + RLEC-2 not plausible. Is this RO level of knowledge? - YES -

DETERMINATION OF SAFETY FUNCTIONS IS RO PLAUSIBILITY ANALYSIS IN EXAM COMMENTS - WILL MAKE RECOMMENDED CHANGES TO ENHANCE THIS IQUESTION Changes made to D - added plausibility analysis - OK RESPONSIBILITY. RLEC 2 IS PLAUSIBLE - SEE N

B S

N B

S N

N S

N B

S AND 6 ARE DISCRIMINATORY - OK N

B S

Why are A and D plausible? Recommend changes to A and D. - ADDED PLAUSIBILITY ANALYSIS FOR A AND D. -

N B

S OK N

B S

suggested changes. OK ON CHANGES N

B N

N Why is D plausible? ADDED PLAUSIBILITY ANALYSIS - SE A is not plausible -entire question is simplistic. CHANGE A N

B N

N I Why is D plausible? REVISED D - OK IA and B are not plausible VALIDATION RESULTS SHOW A S

I T 0 569F DONE -OK n

I J

Need to revise D to be more plausible -what about the condensate pump? You do not address the auto start of the standby condensate pump in c or d. One can assume the standby condensate pump has started. See suggested revisions. REVISE QUESTION TO 80% POWER - REVISE D AS SUGGESTED - ADD PLAUSIBILITY ANALSYIS FOR S

D. DONE-OK S

Too simplistic -does not test operational implications of leak rate decreasing. REVISE OR REPLACE - REVISED C and D not plausible - C is plausible - see plausibility analysi

- D is not plausible - agreed to revise question to recommended changes if we changed the U to an E. OK I

Distracters A, B and D seem to test the condition where discharge pressure < some value (125 psig). This is OKfor one distracter - but not all 3. If the pump is deadheaded, mini flow protection must be provided. REMOVE STEM ABOUT LOW FLOW RATES - THEN DISTRACTERS A and B ARE JUSTIFY THE PLAUSIBILITY OF "C" OK - REVISE C -ADDED DISTRACTER ANALYSIS TO S

9/3/2008 FINAL Page 2

Calvert Cliffs

6.

B/M/N NRC N

B B

B RO ES 401-9

7.

U/E/S

8. Explanation A, C and D are just not plausible. A and C were plausible.

RO/ATC not allowed to do much of anything but See proposed revisions. Accepted proposed revisions - OK simplistic - A, and Dare not plausible ADD PLAUSIBLITY Question does not meet critieria for significant modification -

stem has not be modified. Does not match WA - generic radiation control - REPLACE WITH EWP QUESTION -

REPLACED - New question: "D" is not plausible - 50 REM does not correlate to a plausible error. CHANGED A and C are not plausible - A - would not dilute to criticallity under any conditions. THIS IS THE CORRECT PROCEDURE FOR PHYSICS TESTING AFTER REFUELING - ONLY COMING OUT OF OUTAGE WITH A FRESH CORE. C -. REPLACE C WITH PROPOSED S

S ANALYSIS-ADDED - OK S

"D"-OK S

DISTRACTER OK RO Exam I

N__

N n

-. S S

. Revise stem to include procedure completion. DONE - OK No reference to be provided? CORRECT - OK E

B B

B B

B B

B S

S S

S S

S How is the "emergency backup" part of WA tested? REVISE Should be scored a fundamental knowledge? Memorization 2 signals. CHANGE TO F LoK - OK QUESTION - REVISED - OK S

B B

B M

B B

S S

S OK S

See proposed revision to distracter C WILL MAKE CHANGE Distracter B is somewhat implausible. See proposed revisior S

S WILL MAKE SUGGESTED CHANGES - DONE - OK I

P#

I 29 30 -

31 -

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 -

4. Job Content Flaws

5. Other Job-Link -

I unit! -

SRO only N

N N

N N

N N

N N

N N

N N

N N

FINAL 9/3/2008 Page 3

Calvert Cliffs P#

52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 RO ES 401-9

1.

LOK (FIH)

H H

F H

F H

H H

H F

H F

H F

H F

H RO Exam S

S Why is B plausible - CAC inlet valve is shut? UNDER CERTAIN CIRCUMSTANCES, THERE IS A REQUIREMEN FOR FLOW TO BE THROUGH SOME SYSTEMS FOR CONTAINMENT INTEGRITY - IF IN MODE 5, THEN B IS See proposed enhancements. No need to provide release form if changes are made. Do we need to give them the circ water pump alingment? - made suggested chnages - circ water pump alignment is not required -OK CORRECT (CONTAINMENT CLOSURE) -OK FINAL S

S

2. LOD (1-5) -

3 3

3 3

3 4 -

3 3

3 2

2 3

3 2

3 2

3 Why is B plausible? See proposed revisions - ACCEPTED REVISIONS -OK Stem Focus -

S S

S S

S I

Ref

3. Psychometric Flaws

4. Job Content Flaws AS RECOMMENDED - DONE - OK Why are A and B plausible? See plausibility analysis. OK See recommended changes to improve question - MADE CHANGES TO BAND C - UNABLE TO READ SUMP LEVE SO D STANDS AS IS - OK Why is D plausible? PZR TEMP IS USED IN ACCIDENTS Should be memory level. CHANGE TO MEMORY LEVEL 0 See suggested change - more plausible. Requires applicant to know the answer to the 2nd Dart of the auestion. WILL 9/3/2008 S

S S

S S

MAKE THIS CHANGE DONE - OK Why is " B plausible? See recommended change to A and E A is not plausible if D (which is the answer) is listed. This question has not been signficantly modified. Only distracter I has changed (was 4 CACs instead of 3 CACs - REVISE A REPLACED QUESTION - New Question: state the title of Change to fundamental level of knowledge? MORE CHANGES MADE - OK AOP-9A - DONE -OK COMPLICATED THAN FIRST THOUGHT -AIR SUPLY COMES FROM DOWN STREAM - COMPLEX INTERACTION -THIS IS HIGHER COGNITIVE - OK Why is C plausible? C IS PARTIALLY CORRECT - BUT NC SUFFICIENT - ADD IN Plausibility analysis for C - DON E OK A is not plausible. Does not test part b of WA - mandatory -

modified A - changed each answer to test part B of KIA.

S Distracters A and B involve the same error. Change A to rei level lowers at 3 inches per minute. Reorder distracters as indicated. Which one is more plausible. MAKE REVISIONS Page 4

Calvert Cliffs RO ES 401-9 RO Exam I.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

Ref Req Back-YIN BlWN NRC

7.

Q#

LOK 2izr UlElS

8. Explanation Q=WA SROonly Cues T/F E,'

Partial t:i Minutia #I units ward

(

W Stem Focus A and D not very plausible. See suggested revisions.

Alternatively, test if the isolation valve closes automatically. A was changed but D was not changed. Either change D 01 draft plausibility analysis. ADDED PLAUSIBILITY D can be argued as true - can also be not plausible - see suggested change MAKE SUGGESTED CHANGE DONE -

69 F

4

?

Y N

N S

ANALYSIS - OK S

OK

,70 F

3 N

Y N

B 71 H

N Y

N N

S Change to fundamental level of knowledge? THEY CONVINCED ME THAT THE ONLY WAY TO ANSWER THIS QUESTION IS TO KNOW WHAT SYSTEMS ARE CAPABLE OF FEEDING INTO THE SFP - REQUIRES 72 H

3 N

Y N

N S

SYSTEM KNOWLEDGE 73 F

N Y

N B

S Why is "C" plausible? Revised C - OK 74 H

N Y

N B

S 75 H

3 N

Y N

B S

Total 75 2.7 0

0 0

0 0

0 0

0 1

6 0

75 75 75 Sum FINAL 9/3/2008 Page 5