ML031130102

Draft - NRC Comments

ML031130102
Person / Time
Site:	Hope Creek
Issue date:	04/01/2003
From:	Reid J Public Service Enterprise Group
To:	Conte R NRC/RGN-I/DRS/OSB
Conte R
References
50-354/03-301
Download: ML031130102 (11)

Text

ES-401 Written Examination Form ES-401-9 Review Worksheet

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

LOK

( LOD (F/H) l(11-5)

S e

usT/F Cred.

IPartial Job-W~nta/Bc-Q RUIS Explanation

_ IFocus JDist.

Link

___units ward K/A Ol H

2 X

E (1) Distractor'd' cues the individual to TS which provides the answer.

distractor ad" has been modified to remove the TS reference.

H 1

U This is a low level of difficulty because (1) the individual will easily identify that TS for refueling only requires the SRM alarm (2) distractor b&d are not required for refueling operations.

Question was rewrote to look at a loss of off site power.

H 3

X U

If 2 bundles double counts, then 4 bundles will give criticality, not another doubling. CR2/CR1=(1-Keff1)/(1-Keff2)

This has been conrected. Local criticality on next 2 bundles.

H 2

S H

2 S

F 3

S stem can be more clearly worded.

OTHER THAN... has been removed and replace with do to high dose rates...

H 2

S Minor stem rewording.

from the core has been removed from the stem.

H 3

S Site specific learning objective states from memory and the applicant is given P&IDs.

P&ID are allowed per the second reference.

H 2

X U

Distractor a & c are non-plausible because RACS is not leaking due to level increasing.

Stem modified to dispatched operator to verify level is increase or decrease -all distractors plausible.

H 2

S NOTE:

These questions when reviewed had one or more distractors that could have been eliminated by the use of provided reference material. The questions were modified or the reference material was removed if it was not required for other questions. These were not considered UNSAT because ES-401 states that reference material does not give the answer away, not eliminate a distractor. Additional guidance will be provided in Rev. 9.

ES-401 Written Examination Review Worksheet Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

lQ#

LOK LOD_____

(F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back.

Q= SRO U/E/S Explanation Focus Dist.

Link

_units ward K1 A Only 11 H

2 X

U (1) 2 non plausible distractors.

Distractors a&b and part of stem were changed.

12 H

2 S

13 H

2 S

14 F

2 S

15 F

2 S

(1) Question does not contain anything about secondary containment Dp? (2) Distractor b should have by the OM only, instead of by the OM.

Secondary Cont. Dp is a place holder - generic K/A met not miss match.

Distractor b is ok because states MUST, by procedure MUST is not required - le distrator is incorrect 16 F

2 S

(1) Question does not contain anything about secondary containment Dp? (2) Learning Objective states with references that the applicant will not have.

Secondary Cont. Dp is a place holder - generic K/A met not miss match.

The or statement allows no reference&

17 F

2 E

l (1) The stem states the license, TS will have a copy of the license.

Secondary Cont. Dp is a place holder - generic K/A met not miss match.

The stem has been modified to remove the cue to look at the license.

18 H

2 S

(1) Question does not contain anything about secondary containment Dp?

Secondary Cont. Dp is a place holder - generic K/A met not miss match.

19 H

2 S

(1) Question does not contain anything about secondary containment Dp?

Secondary Cont. Dp is a place holder - generic K/A met not miss match.

20 F

2 E

(1) One non plausible distractor - a, (2) verify from memory even though LO says references.

Distractor 'a' will be changed to channel fastener to make plausible.

The appropriate references have been provided.

ES-401 Written Examination Form ES-401-9 Review Worksheet

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD I

(F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO U/EIS Explanation Focus I

Dist.

Link units ward K/A IOnly _

21 H

2 S

(1) Add using normal plant procedures to stem.

Stem was modified.

22 H

2 X

E (1) Wording change on distractor - b; (2) Non plausible distractor on -d Distractor 'b" changed to control blade to read better.

Distractor "d" changed to include one good SRM.

23 H

2 E* (1) Move assigned to "A" APRM out of distractors. (2) TS for Q4 will remove 2 of the four distractors for this question that no material needed for exam.

lA"APRM is removed from Distractors to stem.

Table 3.3.1-1 has been modified.

24 H

2 S

(1) additional clarifying information to the Explanation of answer.

I I

Information added on all RBV fans running.

25 F

2 S

(1) Verify that TS or ECG will not aid in answering question.

TS or ECG will not give away answer and TS and ECG sections not required for exam removed from exam reference list.

26 F

2 S

(1) Verify that TS will not remove distractors c & d.

TS or ECG will not give away answer and TS and ECG sections not required for exam removed from exam reference list.

27 H

2 S

(1) Distractor d is non plausible - correct prior to use.

Distractor 'd" is plausible and must be eliminated by infonmation in stem - Mode 4. In mode 5 this is a plausible method of Decay heat removal.

28 F

2 E* (1) TS will answer question. There should be no references for this question and TS Table 3.3.2-1 will be provided. (2) verify from memory even though LO says references TS Table 3.3.2-1 trip function 7 answers question. TS Table removed from reference list.

Panel drawing is included as a reference.

29 F

2 E* (1) TS reference 3.8.2.1 Action A states that 250 VDC battery supplies HPCI & RCIC. This may allow distractor ad" to be eliminated.

TS 3/4.8.2, DC Sources has been removed a reference.

30 F

2 S

(1) verify from memory even though LO says references.

Pendent drawing will be given.

ES-401 Written Examination Form ES-401-9 Review Worksheet

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD__

(F/H)

(1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q=

SRO U/E/S Explanation Focus Dist.

Link units ward K/ Inl 31 F

2 U (1) Verify that TS will not assist in answering the question.

32 I

Section 3.9.1 has been removed from the TS package.

32 F

2 S

33 F

2 S

34 F

2 E (1) Change distractor b to make more incorrect. (2) Do they need the procedure as per the L.O.?

_The questions was replaced.

35 F

2 S

36 F

2 X

X U (1) The question does not address the impact on fuel pool cooling and cleanup. (2) If immediate operator actions then giving them the procedure no. could be a cue.

Question was changed to match K/A.

37 H

2 S

38 F

2 S

39 H

2 S--

40 H

2 5

ES-401 Written Examination Form ES-401-9 Review Worksheet 1

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

(F/H) 1(1-5)

Stem ICues T/F ICred. Partial Job-Minutia

1. /

Back-Q=

SRO UIEIS Explanation Focus Dist.

ILink j l units[ward K/A

_Only_l_-l 41 F

2S 42 F

2 S

43 F

2 S

44 H

2 S

45 H

2 E

(1) focus the distractors on Depletion of U-238 and Pu-240.

Distractor ac" will be changed to build up of PU-240 to increase the LOD.

46 H

1 U

(1) Will not discriminate Question has been changed.

47 F

2 5

48 H

2 S

(1) Add 'both' for clarity both has been added 49 F

2 5

50 F

2 5

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onl' (K/A and license level mismatches are unacceptable).

6.

Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7.

At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

I I

I NUREG-1021, Revision 8, Supplement 1 44 of 46

December 31, 2002 MEMORANDUM TO:

Exam File PSEG LSRO -Hope Creek FROM:

Alan Blarney U -

Senior Operations Engineer

SUBJECT:

COMMENTS ON HOPE CREEK SENIOR REACTOR OPERATOR LIMITED TO FUEL HANDLING EXAM OUTLINE.

The purpose of this memorandum is to document comments on the Hope Creek Station exam outline received on Tuesday, December 23, 2002. The exam outline was reviewed by A. Blamey, Senior Operations Engineer. The comments were provided to A. Faulkner, PSEG on December 31, 2002. The comments are as follows.

Modified ES-201-2, Examination Outline Quality Checklist Section 1.

Written

b.

There are currently four K/A that have no questions selected in the Tier 2 (system Section). When replacing items in the plant systems area sample more of the K/A to reduce the number that have 0 selected.

ES401-1, Item 1 states that at least two topics from each K/A must be selected.

In the case of a LSRO this can not be done; however, an attempt should be made to obtain as many K/A as possible.

c.

Overemphasizes of plant systems.

ES-401-1, Item 3, avoid selecting more than 2 or 3 topics from a given system.

Even though this is an LSRO exam and the amount of material is more limited than for an SRO exam, it appears that the following areas are overemphasized.

Control rod drive (5 questions - 10% of Exam)

Af1 / G2 295022, Loss of CRD Pumps - CRDM High temp.

T1 / G2 295022, Loss of CRD Pumps - Reactivity Control

,22 / G2 201001, CRD Hydraulics - pressure indication A'2 / G3 201003, Control Rod & Drive - CRD Hydriodic System T2 / G3 290002, Reactor Vessel Internals - CRD Mechanism RHR - Shutdown Cooling (5 question - 10% of Exam)

-T1 / G2 295021, Loss of Shutdown Cooling - Feed and Bleed T1 / G2 295021, Loss of Shutdown Cooling - Alternate Methods - NOT RWCU p2f1 / G2 295021, Loss of Shutdown Cooling - Local operation T2 / G2 205000, RHR SDC - Recirculation System T2 I G2 205000, RHR SDC - System Isolation Recommend Replacements Ti I G1 295031, Reactor Low Water Level T2 / G1 215005, APRM / LPRM T2 I G1 261000, Standby Gas Treatment System T2 I G2 201002, Reactor Manual Control System T2 / G2 214000, Rod Position Information System Category 4, Health Physics ES-701.D.1, under sample states that "Health physics and radiation protection for fuel handling activities and general employee responsibilities. These questions may include administrative procedures associated with radiation protection."

These question listed under this section do not pertain to health physics and / or radiation protection.

G.2.1.4 Knowledge of shift staffing requirements G.2. 1.10 Knowledge of conditions and limitation in the license G.2.4.30 Knowledge of events that need to be reported.

Change one item to met the criteria and move the other 2 items to the procedure section. This would increase the procedure section to 44% and reduce the HP I RP section to 10%, which is allowed in accordance with ES-701.D.1 (within the +

1-5%).

Section 2.

Discussion No Comments Section 3.

Walk Through - Job Performance Measures (JPM)

a. (3)

No JPMs are duplicated from the applicants audit exam.

Since the audit exam has not been developed this must be verified prior to administering the exam. PSEG is aware that the JPMs exams can not be a duplicated from the audit exam.

tsk bcckA*

b. (1)

ES-701.E.2. states "that instead of distributing systems among the safety function groupings in the K/A catalog as discussed in ES-301, this category of the LSRO operating test shall evaluate the applicant on 5 systems selected from Attachment 1 as follows: one from decay heat removal (DHR) or auxiliary (AUX), one from instrument and control (IC) or radiation monitoring (RM), and three from fuel handling equipment (FHE)."

S.1 Should come from IC I RM. The JPM is on NSSS Isolation valves, not IC I RM.

b. (2)

Which one of the tasks is conducted at a low power condition?

b. (4)

One of the in-plant tasks must test the applicant's response to an emergency or abnormal condition. Which one uses EOP or APs.

S. Z_ R-c<:

Duplication A. 1.a Applies working hour restrictions - ensure that written question G.2. 1.4 does not test same material.

A.1.b This JPM reviews RHR flow paths, a system that is over sampled. Should look at core loading / verification JPM.

S.2 Align RWCU for Alternate Heat Removal. The written question should not use RWCU for an alternate heat removal system.

Section 4.

General

+/-

c.

Ensure that K/A ratings are at least 2.5. Must add the K/A importance on Cat. B of the JPMs.

d.

Check for duplication and overlap among exam sections.

The outline appears acceptable; however, the NRC can not make a final determination until the exam is drafted and the individual questions can be reviewed.

e.

Check the entire exam for balance of coverage.

The outline appears balanced; however, can not make a final determination until the exam is drafted and the individual questions can be reviewed.

Cat. B Job Performance Measures (JPM) 215004 A1.04, SRM/IRM Rod Block Bypass P

Revise the initiating cue.

2003-NRC-LSRO-S2, RWCU decay heat removal T;

Need a copy of procedure HC.OP-AB.RPV-0009.

Add less than 200 degrees to 5.9.1, standard.

2003-NRC-S3, Bent Mast Step 5.8.17 has the examiner order the applicant to "Stop Hoist Travel,"

we should not give applicant orders when operating plant equipment.

-,The LSRO on the bridge will give this order.

'The step will be changed to blade guide being removed to inserted.

t'he standard EXAMINER CUE will have spotter observed slack cable as well.

,,Remove CUE: Carry out actions for a bent mast. This is what the applicant must identify for the alternate path, we will not cue the alternate path key.

Revise attachment 1.

2003-NRC-S4, Monorail Aux Hoist Controls Functional Test Remove specific step from Initiating Cue 2003-NRC-S5, Automatic Fuel Movement Must co~plete and validate when modification is complete.

ES-301-4 Simulator Scenario Quality Checklist General Comments Complete and submit ES-3014, Scenario Quality Checklist for the scenarios.

L.

Complete and submit ES-301-6, Competencies Checklist for the scenarios.

Scenario 2 Change scenario 2 to withdrawal of blade guide W/o adequate support / movement fuel to vessel I dropped assembly evacuate DW, Refuel floor / classify event.

January 31, 2003 MEMORANDUM TO:

Exam File PSE&G Hope Creek FROM:

Alan Blamey Senior Operations Engineer

SUBJECT:

COMMENTS ON HOPE CREEK LSRO EXAM.

The purpose of this memorandum is to document comments on the Hope Creek LSRO exam received on Monday, January 27, 2003. The written exam was reviewed by A. Blamey, Senior Operations Engineer, J. D'Antonio, Operations Engineer, and H. Williams, Senior Operations Engineer. The comments are as follows.

ES-401-7 Written Examination Quality Checklist See ES 401-9, "Written Exam Review Worksheet," for comments.

What TS section will the applicants be given and what hand out will they have for the test? (Item 8)

See Form ES-401-9 Will these handouts give answers away? (Item 8)

See Form ES-401-9 ES-301-3 Operating Test Quality Checklist We will need an LSRO on the bridge during movement. (What about verifiers?)

This issue has been reviewed with HC and an LSRO will be on the bridge during movement of the bridge.

Very that the JPMs have the required reviews as per the PSEG program.

Very that the audit exam has no overlap with the NRC exam.

Repetition from previous operating exams is not greater than 30%.

20% - 1 Walk-Through JPM is duplicated from the last NRC exam. (JPM S.1)

Cat. A Admin Section General Comments Many of the task standards are the same for steps. These should be specific to the step.

Specific Comments Al-1, Working Hours Limitations Provide copy of procedure NC.NA-AP.ZZ-0005.

Put standards in for each step.

t3 Revise JPM to have Form-1 filled out.

Al-2, Verify Log Requirements for Resuming Fuel Moves 6

UNSAT This admin JPM simply has the applicant obtain a surveillance and very that the last time the surveillance was completed is outside the 18 months. This JPM will not allow the examiner enough information to evaluate the applicant.

The JPM will be revised to provide additional technical verifications for the operator.

A2, Fuel Movement Sheet Changes Provide copy of spent fuel pit map.

This is not available and will not be necessary for the operator to review.

I_

Step 5.6.2.B has to many cues on when evaluating shutdown margin.

Remove examiner cue - he has the sheet to review.

A4dd dctevniite& that &DM is met and gees te step4 56.5. 2E, AMJ ge-c 4-A3-1, Question UNSAT This is a direct look up question. Change to closed reference or modify question.

Question changed to closed reference.

A3-2, Question Change to closed reference.

A4-1, Questions This question looks like direct look up questions.

This is not a direct look up question because the individuals have to understand that they will need to frisk at one location and then move to another location for accountability.