ML031190614

Draft - NRC Comments

ML031190614
Person / Time
Site:	Indian Point
Issue date:	04/01/2003
From:	Wilson F Entergy Nuclear Northeast
To:	Conte R NRC/RGN-I/DRS/OSB
Conte R
References
50-247/03-301 50-247/03-301
Download: ML031190614 (9)

Text

ES-401 IP#2 Written Examination Form ES-401-9 Note: Resolutions to comments in bold and italics Review Worksheet completed 2/4/03

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD I

I (F/H)

(1-5)

Stem Cuesl T/F Cred. Partial Job-Minutia

1. / B Q= SRO U/E/S Explanation FocusJ I

Dist.

Link

ward K/A Only 12 F

2 Y

S 317 F

2 X

Y E

SHUTDOWN BANKS ARE NOT USED FOR TAVE CONTROL, YET YOU HAVE THIS AS THE CORRECT ANSWER. TOO APPEALABLE.

Question got replaced because of the potential exam comprimise.

119

_iI 19 H 2

Y Sl

'23 F

Y S

question was similar to an In-plant JPM that was subsequently l

I

_replaced.

24 H

2 Y

E EXPLANATION? NO FIELD FLASH, WHAT ABOUT START AIR? Air l_

_.start needs to be valved in also revised wording on 2 distractors

'28 F

Y S/?

NON DISCRIMINATORY. Modified question to Increase discriminatory level 29 F

2 Y

S l31 F

2 N?

S?

This question was accepted as a borderline K/A match. See licensees comment

32 F

2 E

modify one distracotr to make more symmetrical. "C" modifed 34 F

2 y

S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

I

• The answer choices are a collection of unrelated true/false statements.

I More than one distractor is not credible.

l

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

I The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

I

• The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

I

• The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

l

• The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onl, (K/A and license level mismatches are unacceptable).

6.

Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7.

At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

I I

I

ES-401 Written Examination Form ES-401-9 Review Worksheet

11.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD

-l 1

(F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

/

Back-0= SROUIEIS Explanation

-38ocu2Focus Dist.

jLink junits ward K/A Only

'38 IF 12 V

S

-39 F 2

=

S a4 F

2 Y

E Clarify stem or answer. Clarified "A" answer

'51 H

2 i S

,66 F

2 y=

Is 68 H 2

=Y s

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undajmental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if ajob content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

l5 Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onhv (K/A and license level mismatches are unacceptable).

6.

Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7.

At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

I.

NUREG-1021, Revision 8, Supplement 1 44 of 46

NUREG-1021, Revision 8, Supplement 1 44 of 46 ES-401 Written Examination Form ES-401-9 Review Worksheet

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD (F/H)

(1-5)

Stem Cuesl T/F Cred. Partial Job-Minutia

1. 1 Back-UE SROU/E/S Explanation Focus Dist.

Link units ward K/AjOny _

R77 H 2

Y S

R79 F 2

X Y

E Clarify distractor "A" to ensure not correct. "A" was modified R81 F

2 S

82 F 1

S/E question does not discriminate adequately. Revised question to require knowledge of additional sources.

83 H 2

Y E

MANDATORY "e"; DON'T GIVE SET-POINT NUMBERS IN C,D - GIVE NUMBERS WITHIN THE REQUIRED ACTION RANGE. Accepted question as wriffen after further discussion.

84 F 2

X Y

U Old question replaced since not considered operational for NRC

__exam reviewed replacement question acceptable.

R86 IF 12 1

y I

89 H 2

=

=y

{93 F 2

_Y E

designated answer not correct. Revised answer key "C" is correct.

97 F 1

V U

Q: WHY DO YOU SHUT THIS VALVE? Replaced question A: TO ISOLATE THE LINE NON DISCRIMINATORY 99 F 1-2 Y

S questioned has low discriminatory value. Accepted question after further question.

ES-401 Written Examination Form ES-401-9 Review Worksheet J 1 2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

(FIH)

(1-5)

Stem Cues T/F ICred. Partial Job-lMinutia I #/ I Back-Q= lSRO U/E/S Explanation I

Focus Dist. l Link 1 units ward K/A Only 75 F

1 Y

N U?

NOT SRO ONLY. MATTERS ONLY IF COUNTED AS SUCH replaced question 378 H

2 Y

Y S

80 H 3

Y y

S 381 H

1 Y

Y U

Direct lookup question if only a portion of the procedure provided.

Modifled stem and licensee will provide entire procedure.

382 F

2 y

y S

85 F

2

=

N U

not an SRO level question. Question replaced.

86 F

2 X

Y N

U not licensed operator responsibility. Replaced question.

88 H

2 Y

S 94 F

2 Y

Y S

99 F

2 E

Enhanced distractor1B" 10 H 2

X Y

V U

Make distractors more plausible. Revised distractors NOTE: REVIEWED ALL REPLACEMENT/REVISED QUESTIONS COMPROMISE.

1-25 SUBMITTED BASED ON POTENTIAL EXAM

2/3/03 Note: Revised 3/5/03 to reflect resolution of comments.

IP2 OPERATING EXAM COMMENTS Simulator scenarios

1.

Scenario #1 - 1) add another critical task "Manual reactor Trip" too simplistic following loss of all feed - does not discriminate adequately. 2) Critical tasks not highlighted in script and don't have performance criteria defined - see appendix "D". 3) Booth operator responses not scripted in most cases. 4) No TS call defined in script. 5)Typo on outline for event #7 & 8 one of these should read "22" MDAFW trips 6) What should the SM designate as an appropriate rate of power decrease, pg 5. 7)Event 2: Why RO credit, not BOP? 8)Event 3: Does BOP get credit for positioning defeat switches and tripping bistables? 9)Event 4: Not clear actions taken by RO and BOP. 11) Step 4.6.1 energize all B/U heaters. 12) Add valve number designations where appropriate. 13) Event 7:

BOP must start 21 MDAFW pp manually, but not clear where in applicant actions Actions regarding trip of AFW pumps as well as FRH.1, step 2 and 9 not listed. This is true in general for all malfunctions on all the scenarios.

General:

1) ES-D-2 is too general, not specific for each applicants expected actions. Need to list all required operator actions.

2) Event 1 is 7 pgs, rest of ES-D-2 is 12 pgs.

NOTE: SCENARIO REVISED FOLLOWING VALIDATION WEEK TO RESOLVE ALL COMMENTS (E.G., REQUIRED OPERATOR ACTIONS ADDED AND CTs BETTER DEFINED WITH PERFORMANCE CRITERIA. THIS SCENARIO WAS RE-DESIGNATED AS THE SPARE FOLLOWING VALIDATION WEEK.

2.

Scenario #2-1) Critical tasks not highlighted in script and don't have performance criteria defined - see appendix 'D". 2) Booth operator responses not scripted in most cases. 3) Was should the SM designate as an appropriate rate of power decrease, pg 5.

4) First Stage shell pressure PT-412A might be a more interesting failure than PT-412B as currently proposed. 5) Actions regarding trip of CCW and failure to start of standby pump not scripted. This is true in general for all malfunctions on all the scenarios. Need to list all required operator actions. 6) state basis for reactor trip required ( i.e. RCP vibration). See other highlighted questions and comments.

Event 2: Need more info on how this affects the plant, RO/BOP actions.

Event 3: Does BOP control CCW?

Page 16: Not clear where RCP thermal barrier is addressed in Attachment 1, for CREW actions.

Event 4: What are actions by RO/BOP?

Event 6: Why does RO trip turbine?

Event 7: Why does RO get credit?

When are the Rx and turbine tripped, not clear?

NOTE: SCENARIO REVISED FOLLOWING VALIDATION WEEK TO RESOLVE ALL COMMENTS (E.G., REQUIRED OPERATOR ACTIONS ADDED AND CTs BETTER DEFINED WITH PERFORMANCE CRITERIA.

3.

Scenario #3 - need to fix level of detail in script similar comments to other scenarios.

Event 1: How long to transfer bus sections 1-4, need to wait that long?

Event 2: BOP credit for tripping bistables, defeat switches?

Event 3: What actions by RO/BOP for credit? Note that it is #23 S/G.

NOTE: SCENARIO REVISED FOLLOWING VALIDATION WEEK TO RESOLVE ALL COMMENTS (E.G., REQUIRED OPERATOR ACTIONS ADDED AND CTs BETTER DEFINED WITH PERFORMANCE CRITERIA.

JPM SI: (IP#2 only)

Initial conditions: What is Rx power and SUR?

Perf. Step 1: Says start at procedure step 4.35 but it is signed off. Appears applicant is starting at step 4.38, but is confusing. Where is Graph NI-1 for step 4.35? Is step 4.36 done yet or does applicant need to complete it?

Perf. Step 2: Procedure step 4.36 says to switch both recorders to IR channels, not one IR and one PR. Not clear what proc. step this is.

Per. Step 3: Is this a precaution? Where found?

Perf. Step 4: a. How block SR trips? b. This is not in the procedure. Is no step for switching NR-45 recorder per the proc. The Cue to perform SR voltage check is not in proc.

Perf. Step 5: not in procedure.

Perf. Step 7: This is not in procedure, skill of craft?

Perf. Step 8: Is this an immediate action or will applicant look at AOI.

Generally, procedure lacks specific direction for operator.

JPM S2:

What are COLs in the initiating conditions?

Perf. Step 4: What is the point of the NOTE?

Perf. Step 8&10 are not in procedure.

Perf. Step 20: Why pressure 635-660#, should be 660# per CRS direction?

JPM S3 (SROIIRO):(IP#2 only)

Perf. Step 2: Transition to step 18, not 17. How does applicant determine if spray is available and effective?

Step 3: How does applicant verify PORVs available?

Step 4: PORVs were verified available in step 3, how does applicant determine they are now unavailable?

Step 5: Step a. RNO actions should be individual Perf. Steps. How tell if each of these actions are completed correctly?

Step 6: Should RCS subcooling be criterion for stopping spray?

Step 7: Where is this in procedure?

Step 8: What is expected pressure and RCS subcooling? Specify necessary termination criteria Termination Cue makes no sense, the Perf. Steps go beyond establishing Aux spray flow.

JPM S3 (SROU):(IP#2 only)

Initiating Cue should be Step 18 RNO, not Step 19.

Step 2: Can't do this in E-3 step 19.

Same comments as above for Step 2-6.

JPM S4:(IP#2 only)

Note which S/G has the SGTR.

Step 2: More specific on how to meet the Standard.

Step 3: More specific on how to meet Standard.

Step 5: Why isn't this a Critical Task, to stop cooldown at target temp?

JPM S5:

Step 1: What is "high" pressure ( and temp?

Step 2: In comment, what is reason for alarm?

Step 4: b. Need cue for valves 1786&1 787 open.

Step 5: What is happening to pressure, is no cue?

Is no Perf. Step for proc. Step 4.1.5, ie when pressure decreases close 516.

Step 6: If JPM is complete when terminate venting, don't need this step.

NOTE: THIS JPM WAS REPLACED VALIDATION WEEK - LICENSEE COULDN'T GET THIS JPM TO OPERATE WELL ON SIMULATOR.

JPM S6:

Initiating cue and task standard contradict each other (bus 5&6 vs. bus 5 only)

Step 2: typo Step 3: What is status of feed from 1 3W93, doesn't say?

Step 4: What should applicant do about caution tags?

Step 6: Typo Is no perf step for procedure step 13.f.

Generally, need to word Perf. Steps like proc. Steps for clarity.

JPM P1: (IP#2 only)

Step 2-6 and 8: Add valve no.

Step 7: Typo JPM P2:

Step 3: Standard, how check valve open?

Termination cue: Typo What is applicant controlling, pressure on guage P-2 or S/G press?

JPM P3: THIS JPM WAS REPLACED BECAUSE PLANT OPERATORS ENTERED THE AREA TO PERFORM WORK DURING VALIDATION IN THE PLANT AND OBSERVED EXAMINER VALIDATION PRIOR TO BEING NOTICED.

Step 3: Add "for 23 battery charger" Step 5: Add wording 'MCC 26C Bkr 3F" Step 8: Why adjust voltage, is adjustment necessary? Simulate DVM reading or get one?

Stepl0: Is adjustment required or not?

Step 11: Cue applicant once he locates amp meter. What does the note mean? 4.2.1(6) typo.

Step 12: Why go to step 4.2.1(7) since amps are greater than130? Will current decrease to less than 130 - need a cue? There is no perf step for proc. Step 4.2.1.6(e).

GENERAL NOTE: All the part B. JPMs were revised to incorporate andlor resolve all identified comments.

Administrative JPMs A.1.b for ROs and SROs ensure adequate exam item do review in simulator.

RO A3 JPM not discriminating and not license relevant. Revise task e.g. Provide RWP to perform a task in a high rad area and contaminated area. If Ops has special quals that they can do their own surveys discuss limitations & controls. Make task more relevant to a licensed operators duties.

SRO, A2 in addition add step to determine operability and TS LCO concerns. This will make more relevant to SRO license. Task as currently written more applicable to RO.

SRO, A4 make one of the proposed JPMs a PAR to better examine the spectrum of EPIP areas and avoid predictability on succeeding exam days. This will also be necessary for IP#3 for the same reasons.

NOTE: Administrative JPMs were revised to incorporate these comments with the exception of SRO A.4, for consistency in examining between the units 2 & 3 made a decision to go with just a classification JPM which the Branch Chief approved.