ML050690405

01-2005 - Draft Exam Comments

ML050690405
Person / Time
Site:	Arkansas Nuclear
Issue date:	01/27/2005
From:	Gody A Operations Branch IV
To:	Forbes J Entergy Operations
References
50-368/05-301 50-368/05-301
Download: ML050690405 (11)

Text

S:\\DRS\\OB\\EXAMS\\ANO \\UNIT2\\ANO 2-INIT EXAM-01-2005\\ANO2-01-2005-NRC D RAFT EXAM C OMMEN TS\\Appendix-e-Admin Review.wpd

- Page 1 -

APPENDIX E - REG ION IV OPERA TING TEST JO B PER FO RM AN CE ME ASUR E Q UA LITY REVIEW MA TRIX JPM#

1.

Dyn (D/S) 2.

LOD (1-5)

3. Attributes

4. Job Content Errors 5.

U/E /S 6.

Explanation (See below for instructions)

IC Focus Cues Critical Steps Scope (N/B)

Over-lap Job-Link Minutia 1A1 (R)

S 2

S 2A1 (R)

S 3

S 3A2 (R)

S 3

X E

Critical Steps 2 and 3 are one-of-the same. Consider adding another critical step that is unrelated to the other findings.

4A3 (R)

S 2

S 1A1 (S)

S 2

S 2A1 (S)

S 2

S 3A2 (S)

S 3

X E

Same comment as 3A 2 (R).

5A4 (S)

S 2

S Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1.

Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.

2.

Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.

3.

Check the appropriate box when an attribute weakness is identified:

The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.

The JPM does not contain sufficient cues that are objective (not leading).

All critical steps (elements) have not been properly identified.

Scope of the task is either too narrow (N) or too broad (B).

Excessive overlap with other part of operating test or written examination.

4.

Check the appropriate box when a job content error is identified:

Topics not linked to job content (e.g., disguised task, not required in real job).

Task is trivial and without safety significance.

5.

Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

6.

Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

S:\\DRS\\OB\\EXAMS\\ANO \\UNIT2\\ANO 2-INIT EXAM-01-2005\\ANO2-01-2005-NRC D RAFT EXAM C OMMEN TS\\Appendix-e-JPM R eview.wpd

- Page 1 -

APPENDIX E - REG ION IV OPERA TING TEST JO B PER FO RM AN CE ME ASUR E Q UA LITY REVIEW MA TRIX JPM#

1.

Dyn (D/S) 2.

LOD (1-5)

3. Attributes

4. Job Content Errors 5.

U/E /S 6.

Explanation (See below for instructions)

IC Focus Cues Critical Steps Scope (N/B)

Over-lap Job-Link Minutia a.

D 3

X E

Change initiating cue to state: "Perform actions in Inadvertent CIAS AO P, 2203.039, Steps 11 through 15." This will provide an end point for the applicant.

b.

D 4

X X

E Define HPSI pump "B" as 2P89B in Step 10. The JPM ends after >40 GPM flow is established, however, the procedure continues on through procedure Step 16. W e need to define the end point for the applicant such as; adding the steps to be performed in the initiating cue (e.g., Steps 2 through 13).

Procedure Step 13 states to use train A with Train B as the contingency step. W hat directs the applicant to use the "B" train HPSI pump? Should Steps 2, 9, and 14 be critical steps?

c.

D 4

X E

Do not give the applicant the procedures, have them identify what procedures are needed. Rewrite Step 4 to be consistent with Procedure Step 7.1.4.

d.

D 4

S e.

D 4

S f.

D 3

X X

E Since the first step of the JPM is to obtain O perations manager approval prior to performing this task, it can be assumed that the plant is in Mode 1 or 2. You need to add the plant mode to the initial task conditions. Provide more information in JPM S tep 7 to be consistent with procedure Step 12.7. Provide a cue to answer procedure Step 12.10.

g.

D 3

X E

Add examiner cue(s) to JPM S tep 8 that will address procedure Steps 7.3.1 and 7.3.2. Change examiner's note in JPM to a new step that is consistent with procedure Step 7.4.1. Add an examiner cue to address the status of 2RE-2330 in procedure Step 7.4.5.

h.

D 3

X E

Add the word "contingency" before "step 13.C" in the "Initial Task Conditions". JPM Step 8 not clear -

what is 2C-60 A & B?

i.

S 3

X E

Delete "...Exhibit 1 starting with Step 2.0." as applicant's should be able to determine where to start in the procedure.

j.

S 4

X E

How can these actions be performed without performing Steps 1 and 2? N eed to bold the examiner's cues. Change examiner's Note at the end of the JP M to state that the FW Ps will be tripped by other operators. The Initiating cue should s tate "...and close the MSIV's locally."

k.

S 4

X U

Appears to be an admin JPM.

- Page 2 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1.

Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.

2.

Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.

3.

Check the appropriate box when an attribute weakness is identified:

The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.

The JPM does not contain sufficient cues that are objective (not leading).

All critical steps (elements) have not been properly identified.

Scope of the task is either too narrow (N) or too broad (B).

Excessive overlap with other part of operating test or written examination.

4.

Check the appropriate box when a job content error is identified:

Topics not linked to job content (e.g., disguised task, not required in real job).

Task is trivial and without safety significance.

5.

Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

6.

Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

S:\\DRS\\OB\\EXAMS\\ANO\\UNIT2\\ANO2-INIT EXAM-01-2005\\ANO2-01-2005-NRC DRAFT EXAM COMMEN TS\\APPENDIX N SCENAR IO REVIEW.wpd

- Page 1 -

APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen Set 1

ES 2

TS 3

Crit 4

IC 5

Pred 6

TL 7

L/C 8

Eff 9

U/E/S 10 Explanation (See below for instructions) 1 X

E Need time line.

2 X

E Need time line.

3 X

X E

Need time line. TS involved are not clear, e.g., Event #4 only says to "Refer to Tech Specs...", need to be more specific. Also need another TS reference.

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1.

ES: ES-301 checklists 4, 5, & 6 satisfied.

2.

TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.

3.

Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.

4.

IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.

5.

Pred: Scenario sequence and other factors avoid predictability issues.

6.

TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.

7.

L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.

8.

Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.

9.

Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

10.

Provide a brief description of problem in the explanation column

OPERATING TEST GENERIC REVIEW NOTES Add procedure steps to the PERFORMANCE CHECKLIST column of the JPMs.

Make valve and/or pump indications consistent, i.e., "red light ON, green light OFF" for all JPMs instead of "red light ON above handswitch." that is used in some JPMs.

Form 301-5 for SRO-I doesn't show the total for the TS category.

Need an electronic copy of the "Suppressed K/As" table.

ES-401 Written Examination Review Worksheet Form ES-401-9 Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other 6.

U/E /S 7.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 1

H 3

Y S

2 H

2 Y

S 3

H 4

Y S

4 H

3 Y

S 5

F 4

Y S

6 H

2 x

Y E

4 GPM CBO total for all RCPs? C hange dist c to 58, d to 56 7

F 3

Y S

8 F

2 Y

S 9

H 4

x Y

E add to complete the question: to restore RCS pressure, IAW AO P 2203.028, PZR SYS M ALFUNC TION. Dist a, c: place a period. after RC S pressure, delete remaining. D ist b: Period after 4626B, delte remaining.

10 F

2 Y

S 11 H

1 x

Y U

GFE S - data in question stem is not needed to answer the question.

12 H

3 x

Y E

change Dist C to 1175, MAIN 13 F

2 Y

S 14 F

3 Y

S 15 F

3 x

Y U

There is no requirement in the procedure to trip within 3 minutes, procedure says to trip locally as soon as possible.

16 H

2 Y

S 17 H

4 Y

S 18 H

3 x

Y U

Dist c: not credible 19 H

3 x

Y S

ES-401 Written Examination Review Worksheet Form ES-401-9 Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other 6.

U/E /S 7.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 20 F

3 N

U Question is operational and does not match KA 21 H

2 Y

E dist d: delete at 22 F

2 Y

S 23 F

2 x

Y E

delete gamma sensitive in the stem.

24 H

5 Y

S 25 F

3 x

Y E

Dist c: delete attempt to. D ist d: could be argued as correc t?

26 H

3 Y

S 27 F

2 N

U T/S question, not a controls manipulation Q.

28 H

2 Y

S 29 F

2 Y

S 30 F

3 Y

S 31 H

3 Y

S 32 H

2 x

Y U

Dist d not credible... change to normal charging?

33 H

2 x

Y E

Delete in stem Q uench tank and Press urizer 34 H

4 Y

S 35 F

2 Y

S 36 H

3 x

Y E

Second bullet in stem, change now goes to went, change are to were. Last bullet in stem, add now after is. D ist d: change to containment atmosphere.

37 F

2 Y

S

ES-401 Written Examination Review Worksheet Form ES-401-9 Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other 6.

U/E /S 7.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 38 F

4 Y

E be consistent - operating/operational in stem. Delete operating in all distractors and answer. Dist a: change is to was. Dists b,c,d: change must to can. In stem, change is required.... to can be taken at the time to remove the Operating Bypass 39 H

3 Y

S 40 H

3 x

Y E

dist d not credible. Dist c: delete stable, replace with below T /S region of acceptable operations 41 F

3 x

Y E

In stem, change 6% level to RAS setpoint. O therwise, this cues information for a prior Q.

42 H

2 Y

S 43 H

2 Y

S 44 H

3 x

Y E

Sim plify the question stem... How will the EFW system automatically respond, then delete automatically form each dist and ans.

45 F

2 x

Y S

46 F

3 Y

S 47 F

2 x

Y E

Change dist c to None, NOUE then rearrange distractors in logical sequence.

48 F

2 Y

S 49 F

2 x

Y E

(unloaded) in s tem is not clear 50 H

2 Y

S 51 H

3 x

Y E

d: delete in the intake structure 52 H

3 x

Y E

breathing air status is not needed in the distractors.

53 F

2 x

Y E

change O PEN to all CAPS in s tem. T his question partiallly ques prior question #52 54 F

2 N

U question does not address K/A

ES-401 Written Examination Review Worksheet Form ES-401-9 Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other 6.

U/E /S 7.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 55 H

4 x

Y E

change 60 to 45 in dist c and d. Plant evacuation not credible. Consider -

determine time to boil?

56 F

2 x

Y E

Change atmoshere to containment 57 F

2 Y

S 58 H

2 Y

E change b: as RCS reaches saturation, change a: will remain stady untill, then count rate will rise rapidly.

59 H

4 Y

S 60 H

3 Y

S 61 H

4 x

Y E

No need for handout (Fig 3.9.2) OK for RO?

Dist a is the Tech Spec answer - need to say that in the stem; otherwise dist d is arguably correct. Do not hand out T/S, no ref needed.

62 H

3 Y

S 63 F

3 Y

S 64 F

3 x

Y E

dist C does not answer the question 65 F

2 x

Y U

GFES rad question 66 F

2 Y

S 67 F

2 N

U This KA is not easily tested on written... replace question/KA 68 F

2 Y

S 69 H

3 Y

S 70 F

2 x

Y E

Dist d is not credible.

71 H

2 Y

S 72 H

2 Y

S 73 H

2 Y

S

ES-401 Written Examination Review Worksheet Form ES-401-9 Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other 6.

U/E /S 7.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 74 F

2 Y

S 75 F

2 Y

S 76 H

3 Y

Y E

Dist d: add imm inent containment spray 77 H

2 Y

Y S

78 H

3 Y

Y S

79 H

3 Y

N U

Definition of Natural Circilation 80 H

3 Y

N U

System Interaction Knowledge - RCS/SDC 81 F

2 Y

N U

System Knowledge - Instrument Air 82 H

2 Y

Y S

83 H

3 Y

N U

Sytem Knowledge - radwaste discharge 84 H

3 Y

Y S

85 H

3 Y

Y S

86 H

4 Y

N U

RCS dilution calc, performed by ROs 87 H

3 Y

Y S

88 H

3 Y

Y S

89 F

2 x

Y N

U Dist b & c: reword, not credible. Knowledge of equipment control procedures is at RO level 90 F

3 Y

Y S

91 H

3 Y

Y S

92 H

3 Y

Y S

93 H

3 Y

Y S

ES-401 Written Examination Review Worksheet Form ES-401-9 Q#

1.

LOK (F/H) 2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other 6.

U/E /S 7.

Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only 94 F

2 Y

N U

Question at RO level 95 F

2 Y

Y S

96 H

3 Y

Y S

97 F

2 Y

Y S

98 H

2 Y

Y S

99 H

2 x

Y Y

E "A" in answer does not m atch "A" in distractor 100 H

3 x

Y Y

E Consider handout of other EALs - fire, security, as distractors... otherwise it is close to direct look up.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following c oncepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distrac tors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conform ance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6.

Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7.

At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).