ML062680166

Aug. 2006 Exam, NRC Comments on Draft Licensee-Developed Exam (Written & Operating Tests) (Folder2)

ML062680166
Person / Time
Site:	Ginna
Issue date:	09/27/2006
From:	Ginna
To:	Todd Fish Operations Branch I
Sykes, Marvin D.
References
50-244/06-301
Download: ML062680166 (9)

Text

Written Examination Review Worksheet (Ginna RO)

Form ES-401-9 ES-401 7

F 2

S 8

H 3

X E

Question and distractor b modified

1.

2.

3. Psychometric Flaws

4. Job Content Flaws (FIH)

(1-5) Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Focus Dist.

Link units ward Q#

LOK LOD 9

H 2

S Ginna agreed LOK is H vs F 1

O F

2 X

E Question re-ordereure-worded

~~~~

5. Other

6.

7.

Q= SRO U/E/S (Only identified flaws will be marked on the form below)

WA Only Explanation E

Time wan sDecified: distractor d. clarified

~~

E Stem modified and procedure referenced in stem E

Question modified for clarity and referenced procedure in stem 1 [ 6 1 F ( 2 (

1 1

1 1

1 1

1 1

1 1

1 S

I Grammar change types of failures Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

2.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3.

Chew the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

The question is not linked to the job requirements (Le., the question has a valid WA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

4.

Check the appropriate box if a job content error is identified:

5.

6.

Check questions that are sampled for conformance with the approved WA and those that are desianated SRO-only (WA and license level mismatches are unacceptable)

Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7.

At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

44 of 46

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

7.

Q#

LOK LOD Explanation (F/H)

(1-5) Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q= SRO U/E/S (Only identified flaws will be marked on the form below)

Focus Dist.

Link units ward WA Only 1 1 F 2

X E

Question re-worded 12 H

2 S

13 H

3 S

1 4 H 2

X E

Question re-worded and added procedure reference in stem 1 5 H 2

X E

Question modified for clarity and specified time span 16 H

2 S

1 7 F 2

X E

Question modified and procedure referenced In Stem 1 8 H 3

X E

Question shortened

' 1 9 H 2

X E

"would be" deleted & replaced with "is;" Ginna agreed LOK is H vs F.

20 F

3 S

21 H

2 S

2 2 H 3

X E

Question modified - deleted 'slowly" 2 3 H 2

X E

Question modified to reference procedure in stem 2 4 H 3

X E

Question shortened 25 F

3 X

U WA mismatch: procedural actiondresponses deleted 26 H

3 S

2 7 H 3

X E

Question wording re-ordered 28 H

2 X

U All distractors were correct; focused question by referencing procedure 2 9 F 2

X E

Question was shortened and restructured 3 0 H 3

X E

Q changed to F vs H; refers to procedure in stem; distractor c shortened 45 of 46

ES-40 1 2

Form ES-401-9 45 of 46

ES-401 2

Form ES-401-9 6 8 H 3

X E

Added "if any;" modified distrtactors

, 6 9 F 2

X E

Added procedure reference in stem 51 H

2 52 F

2 17011-1 1 2 I x I E

Deleted second half of distractor b I

I 5 3 H 2

X I

I 5 5 H 5 6 H 5 7 H

7.

Explanation (Onlv identified flaws will be marked below) 3 X

E First two bullets deleted; question re-ordered 3

X E

First two sentences deleted: "effect" vs "affect" 3

X E

Added "if any;" amplified distractor d; I

Only ---

S S

E Question modified and distractors re-ordered 58 5 9 H 54 H

4 S

I I

I I

I I

I I

I I

I I

I I

F 3

S 4

X E

"infoperable" capitalized in distractors 1

60 H 3

S 6 1 H 3

X E

Time frame specified: "vacuum" added: were vs where I

62 F

2 I s I

6 3 H 64 2

X H

2 E

S Stem sentence broken up; added "if any" 6 5 F 66 45 of 46 3

X E

"from" vs of" a loss of AC power F

2 S

6 7 F 2

X I

I E

Question shortened to fill-in-the-blank-type

ES-40 1 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws Q# LOK LOD (F/H) (1-5)

Stem Cues T/F Cred. Partial Job-Minutia

1. /

Back-Focus Dist.

Link units ward 7 1 F 2

X 7 2 F 2

X 7

3 F

2 X

74 F

2 75 H

2

~

5. Other

6.

7.

Q= SRO U/US (Only identified flaws are marked below)

KIA Only Explanation E

Re-worded question E

Modified question and added procedure reference in stem will require vs would warrant; modified distractor a E

S S

~~~~~~

45 of 46

ES-401 Written Examination Review Worksheet (Ginna SRO)

Form ES-401-9

~

I#

4 I

1 2

3 Psychometric Flaws 4 Job Content Flaws 5 Other 6

7.

LOK LOD Explanation (F/H)

(1-5) Stein Cues T/F Cred. Partial Job-Minutia

1. /

Back-Q= SRO U/E/S (Only identified flaws will be marked below)

Focus Dist.

Link units ward WA Only 1

H 2

S 2

H 3

S 3

H 2

S E

Question replaced -similar to JPM; too much EP on written exam 5

H 3

X U

WA mismatch; replaced 6

H 2

S 7

H 2

S 8

H 2

X E

Capitalize in/operability 9

H 3

S 1 0 H 3

X S

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

2.

3.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated trudfalse statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

The question is not linked to the job requirements (Le,, the question has a valid WA but, as written, is. not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

4.

Check the appropriate box if a job content error is identified:

5.

6.

7.

Check questions that are sampled for conformance with the approved WA and those that are desiqnated SRO-only (WA and license level mismatches are unacceptable).

Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

At a minimum, explain any U ratings (e.g., how the Appendix 8 psychometric attributes are not being met).

44 of 46

2 Form ES-401-9 ES-401 1

2

3. Psychometric Flaws

4. Job Content Flaws 5 Other 6

2#

LOK LOD (F/H)

(1-5) Stem Cues T/F Cred Partial Job-Minutia

1. /

Back-Q= SRO UlUS Focus Dist Link units ward WA Only 11 H

2 S

12 F

2 S

13 H

2 S

1 4 H 3

X 1

E 15 H

2 S

E 1 6 H 2

X 1 7 H 3

X E

1 8 H 3

X E

1 9 H 2

X E

2 0 H 2

X E

2 1 F 3

X E

22 H

2 S

2 3 F 2

X E

24 H

2 S

25 H

3 S

7.

Explanation (Only identified flaws are marked below)

Ginna agreed LOK is H vs F Added if any Misspelled organiztion Restructured question; modified distractor d for parallelism Added if any and MCB controls and indictations shall vs should Added procedure reference in stem; Ginna agreed LOK is H vs F Added TS reference in stem Ginna agreed LOK is H vs F Added MINIMUM and reworded

Changes to the Operating exam resulting from NRC Review

1) Operating Exam Outlines and Quality Checklists a) FormES-301-1 SRO i) Corrected JPM number for second JPM from JS341.001 to JS343.006 i) Corrected JPM number for a. to JR012.013 i) Revised forms to reflect that only 2 SRO-U candidates will be taking the exam

2) In-Plant JPMs a) JN086.002 Secure Plant Ventilation During a Fire.

i) Added a statement to the initial conditions specifying that the system was in operation.

i) Added a reference giving valve locations to be used by the examiner in case a candidate needs a location cue.

ii) Add word Close to beginning of element for steps 3-8 specify action for the examiner.

c) JN071.002 Release the A Gas Decay Tank i) Steps 2 and 3 Changed Close to Verify Closed. Procedure step is verification; they are expected to be closed.

ii) Steps 2 and 5 added a cue to indicate that these components have holdhlock b) Form ES-301-2 RO c) Forms ES301-5 and 6 b) JC035.006 Perform Attachment Ruptured Steam Generator Part B

-4 tags installed.

3) Simulator JPMs a) JROO 1.O 13 Realign a Misaligned RCC i) Step 13 Change to Not Critical b) JR004.012 Burp the VCT to remove Hydrogen i) Change step 2 to Critical ii) Change step 6 to Not Critical iii) Modify setup to have 60 GPM letdown orifice in service iv) End JPM at step 24 when A 0 sent to investigate c) JR006.002 Nitrogen Makeup to Accumulator i) Steps 8 and 15 are not critical ii) Step 9 is critical iii) Add note to step 11 for booth operator to set N2 press to 2500 psig d) JR103.007 Manually Close CNMT Valves on CI Failure i) Provide List of valve that must be closed in check off format.

e) JR064.002 Start and Load EDG i) Start JPM a current step 22 ii) Fix Setup tom actuate both SI Trains f) JR012.001 Defeat a Failed P E R Pressure Channel i) End JPM at step 12

4) Administrative JPMs a) JR343.005 Control of LCOs for equipment 00s i) Added attachment 1 to clarify what part of A-52.4 to perform ii) Added a CR number to use to fill out the attachment i) Specified which parameter are critical data for determining sathnsat for data recording ii) Made the step for transmitting the data NOT CRITICAL c) JS343.006 Loss of Safety Function Determination i) Corrected Typo in step 1 d) JS 1 19.00 1 Verify Equipment Tagout Boundary i) Reworded Initiating Cue to clarify task e) JS340.021 Event Classification i) Added note to Initiating Cue to indicate that this task is time critical.

ii) Added EAL 6.1.3 as a second possible classification for scenario 06-1-1

• -d b) JC119.001 Complete EPIP 1-5 Status report (no PPCS available)

5) Simulator Scenarios a) Scenario 06-1-1 i) Increase the ramp time for the P E R spray valve failure from 30 to 300 seconds ii) Correct a typo in the transition step in event 9 i) Adjusted ramp rate on fuel failure to give quicker response.

b) Scenario 06-01 -2