ML032460726

Draft NRC Comments on Written Examination

ML032460726
Person / Time
Site:	FitzPatrick
Issue date:	07/21/2003
From:	Williams J NRC/RGN-I/DRS/OSB
To:	Berry P Entergy Nuclear Northeast
Conte R
References
50-333/03-301
Download: ML032460726 (10)

Text

ES-401 Written Examination Form ES-401-9 Review Worksheet - FitzPatrick - ~uiy2003

3. Psychometric Flaws

4. Job Content Flaws I 5. Other

6.

UIEIS S

U -

E

7.

Explanation Stem ocus -

5ack-Q=

Nard T KIA I

X

red.

Dist. -

I Y

Y Y

Y Y

Y N

Y N

F I

N LOD=I The stem tells the operator that there is ten minute limit and tha is the correct answer.

Changed question to have operator answer what effect the reduced power has on dose and what actions the operator should take.

H H

3 N

2 B

2 B

X X

STEM Focus better on time for EDG to start and load and all equipment function as design.

(b) activation of the timer will not trip the bus.

Stem Revised Change to actuation of...

X X

E STEM should clarify one large load or several for one large DC load will not trip charger.

(b) - remove one hour.

Focused stem in accordance with procedure guidance Removed time limits in all distractors Question chanEed.

H E

(b) could also be correct.

(d) which state that BPV operate as designed and the distractor states must be operated manually.

No - 27%power no cram arrays Remove TBVfrom stem or (d)

S U

(b) is implausible because county fire control coordinator LOK=F This is a fundamental level of knowledge -when can control be returned to the control room.

KIA The question just asks when can control be returned to the control room.

Question rewritten - (b) is plausible - LOK = H - W A Matches F

H -

F 2

B 2

N 2

N S

I H

3 N

J I DO WA - Q does not ask about administrative control?

NOT (b) - is an implausible distractor OUNT Randomly desekcted/re-selected W A -New Question is SAT NRC provided guidance not to reject KIA during development.

Q#

1.

2.

3.

4.

5.

6.

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q= SR U/E/S Explanation Focus Dist.

Link units ward WA 0

On1 Y

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

Check the a propriate box if a sychometric flaw is identified:

!he stem lacks sukcient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

One or more tkpwmedistractors is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

Check the a ropriate box if a job content error is identified:

%e question is not linked to the job requirements @e., the question has a valid WA but, as written, is not operational in content).

The question requires the recall of knowledge that IS too specific for the closed reference test mode (Le., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

Check auestions that are SamDled for conformance with the approved WA and those that are desinnated SRO-only (WA and license level mismatches are unacceptable Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7.

At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

U / DO NOT COUNT There were several items that were UNSAT but we did not count them as UNSAT toward the total number of UNSAT for the 20% report count. The reason is as follows:

I.

2.

In some cases the license call and asked to change WA because they could not match. The NRC told Entergy to keep the same WA. This resulted in an UNSAT and WA swap after the exam was reviewed.

ES 501 section E.3.a, states in part Note: NRC-validated questions, JPMs,

, and scenario events that required replacement or substantial modification will not be counted unless the reason for the current unacceptable flaw was caused by the facility licensee since the time the test item was previously approved by the NRC. (For example, the questions reference changed but the question was not revised accordingly.)

In all cased these questions were corrected before the exam was given.

33 of 34 NUREG-I 021, Draft Revision 9

ES-401 2

Form ES-401-9 Q#

LOK (F/W 10/11 x F x 11/12 H 12/13 i F

13/16 =-P H

17/20 I H 19/22 v F

20123 Ix F x 21/24 F 22/25 +

H :--

5. Other Y

N Y

Y Y

N Y

Y Y

Y Y

Y Y

Y Y

Y Y

Y Y

Y Y

6. I

7.

UlElS Explanation U

LOD=I the applicant must just look at the handout provided and BTUIHr. Should give him RHR-FPC as well.

Question chanped to increase LOD, LOD=3 U

KIA 1. is refueling accident - if criticality accident is in FSAR then it is 01 as an ACCIDENT. 2. Does not address reason.

Is part of accident analysis, Question revised to address the reason.

U LOD=I This the applicants to review only one leg of the EOPs.

Changed to review 2 legs of EOPs, LOD = 3 S

(a), (b) are implausible because they will not shutdown the reactor.

SRO This is not an SRO only question, it does not meet criteria 5.

Question revised to evaluate condition and take appropriate actions.

Revised to account for exam overlap with audit exam., initial question SAT.

S I

U LOD= 1 EP-1 is given to the applicants and the answer is worded the same as the procedure.

Question revised.

LOK=F Should be Fundamental because the question only requires the knowledge of the level at which SRV Tailpipe is uncovered.

S Changed LOK to F.

S I

S S

S LOK=F changed to LOK = F S

S E

(b) - is not a plausible distractor Distractor was Changed E

STEM should also state that bundle integrity maintained?

Stem Revised.

Q#

24/28 25/29

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

LOK LOD (FIH) (1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q= SRO UlElS Explanation Focus Dist.

Link units ward WA Only H

2 B N

U KIA states to protect against high radiation, there is no discussion on high temperature.

Question modified to address high radiation / Temperature Increased to LOK = H.

H 3 N Y

S

ES-401 3

Form ES-401-9

32/44 F

2 N 33145 F

2 N 34/46 H

3 N 35/47 F

1 N 36/48 H

2 N 37/49 F

2 N

RO lnly

3. Psychometric Flaws I

4. Job Content Flaws I 5. Other I 6.

I

7.

UIEIS Explanation U

(a) - what component does the 10600 bus feed in reference to HPCI?

(d) - Loss of secondary containment how does it effect HPCI?

I0600 changed to 10500; changed secondary containment to SBGT.

Link Y

Y Y

Y Y

Y Y

S U

CUE Bullet 2 says reactor is shutdown under all conditions, not needed if ARI.

Psychometric - question does not provide sufficient counter balance.

The applicant only needs to know GREEN to get correct answer.

Question has been revised and now need to understand procedure -

LOD = 2.

S S

ES-401 4

Form ES-401-9 E

S S

X -

X (b) should be changed to "not full in" (b) changed to "not full in."

LOK=F Why is H, simple question of what other system uses this tap.

Changed to LOK=F.

6.

U/E/S U

U / DO NOT

>OUNT U

7.

Explanation (c) could also be correct.

Revised Question.

WA - actions to take when core instablitlites are identified.

H - because he must diagnose core instabilities and take action.

Deselected and re-selected KLA NRC provided guidance not to reject WA during development.

LOD=I Based on the print the LOD is I (a direct lookup)

CUE The print gives the answer away / and also may give the answer away to 40/53. SRO have this reference and RO do not.

(d) This could also result in a correct answer / PCI LCO Question revised. No reference needed and now LOD = 2.

U U / DO NOT

>OUNT S

~~

~~

E (d) could be correct (TSG-8)?

I Revised (d) to s&te from control room - must be done locally.

CUE (a) is the only one that is NOT ABSOLUTE.

K/A - Q is on secondary containment, not the PClS / NQS.

Distractors changed to obtain balance and remove CUE.

Deselect and re-selected UA.

NRC provided guidance not to reject WA during development.

S I

II S

S S

I II

3. Psychometric Flaws I

4. Job Content FI Back-ward -

7.

Explanation

~~

Nonsymetric - WA is a two part WA and to correctly answer the question the applicant only has to know that The breakers can be tripped manually Revised to make question symmetric (d) - run at reduced load is not plausible.

(d) has been revised