Text

Draft - NRC Comments
	ML023250476
Person / Time
Site:	Beaver Valley
Issue date:	09/30/2002
From:	Hynes C; FirstEnergy Nuclear Operating Co
To:	Conte R; NRC/RGN-I/DRS/OSB
	Conte R
References
	50-334/02-301 50-334/02-301
	Download: ML023250476 (11)
	v • d • e

ES-401 BV1 Written Examination Form ES-401-9 Common Questions Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia V

I Back- Q= SRO U/E/S Explanation FocusJ I Dist. Link Iunits Iward K/A IOnly I F 2 Y N S 2 H 2 Y N S 3 H 2 Y N E delete recommendation to trip p/p S F 2 Y N S 5 F 2 Y N E delete "... expected 5 F 2 Y N S 7 H 2 Y N S 3 H 3 Y N S H 2 Y N S 10 F 2 Y N E delete "... expected..."

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onl, (K/A and license level mismatches are unacceptable).

6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. At a minimum, explain any "U"ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD

-T (F/H) (1-5) Stem Cues TIF 1 Cred. Partial Job- Minutia #/ Back- Q= SRO U/EIS Explanation SFocus __j Dist, Link Iunits ward K/A I On__

1 F 2 Y N S 2 F 2 X Y N E background info regarding subcooling and ICCM is unnecessary: delete this part of the stem 3 F 2 Y N S 4 F 2 Y N S 5 H 2 Y N S 6 F 2 Y N S 7 H 2 Y N E delete "... expected ... "

8 F 2 Y N S 9 F 2 1Y N E delete"... expected..."

20 H 2 Y N S 21 F 2 Y N E delete"... expected..."

22 F 2 Y N S 23 H 2 Y N S 24 H 2 Y N S

?5 H 2 Y N S

?6 H 2 Y N S

_7 F 2 Y N IS 28 F 2 Y N E add procedure reference 9 H 2 Y N E temperature has initial effect, then power response O F 2 Y-__ N S 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #1 Back- Q= SRO U/E/S Explanation Focus J I Dist. Link J Junits ward K/A I Only 31 H 2 Y N S

ý2 F 2 Y N S 33 F 2 Y N E add procedure reference

ý4 H 3 Y N S Z5 H 2 X Y N E add "minimum" just before ... amount of boric acid 37 H 2 Y N E add procedure reference 38 H 2 Y N S 59 F 2 Y N S 0 F 2 Y N E add procedure reference t1 F 2 Y N E add procedure reference

$2 H 2 Y N S 43 H 2 Y N S 14 H 2 Y N S

$5 H 2 X Y N E add " minimum" to "... approximate time ..."

16 F 2 Y N S

$7 H 2 Y N S 8 H 2 Y N S 9 F 2 Y N S 52 H 2 Y N E ask which area is probable fire location

ý3 F 2 Y_ N S 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 2 Form ES-401-9 1J

. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5)

Focus J Stem Cues T/F Cred. Partial Job- Minutia Dist. - Link

/ Back-units ward Q= SRO U/EIS KIA" Only Explanation 54 H1 2 Y N S 55 HI 2 Y N S 56 H 3 Y N E replace "... stop the SI p/ps ... " with "... terminate SI ... "

58 F 2 Y N S 59 H 2 X Y N E delete that the crew suspects PORV problem; delete ... expected .. "

ask what indications confirm PORV is partially stuck, etc.

o0 H 2 Y N S 31 H 2 Y N S 32 F 2 Y N E answer includes a basis: add to Q 33 F 2 1Y N S 34 H 3 Y N E delete "... appropriate 35 H 2 Y N S 38 F 2 Y N S 39 H 2 Y N S o0 F 2 Y N S 34 H 2 Y N E add"... if any ,."to stem 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 BV1 Written Examination Form ES-401-9 RO Only Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Link Minutia units Dist. Partial Job- IBack- Q= SRO U/E/S (F/H) Stem Cues

- _Dj T/F Cred.

iist. #1 _ _ _ _ _

(F/H)(1-)(1-5' ocus_

ward K/A Only Q#

LOKs I Expanaio

ý6 F 2 Y N S o0 H 3 Y N S 51 F 1 Y N U not discriminating; modify or replace 57 H 3 Y N E delete "... should" 36 H 3 Y N S 37 H 3 Y N S SIF 12 Y N E delete u...expected..."

Y N E delete "...expected..."

1 F 2 3 H 2 Y N S 4 F 2 Y N S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

the closed reference test mode (i.e., it is not required to be known from memory).

The question requires the recall of knowledge that is too specific forinconsistent The question contains data with an unrealistic level of accuracy or units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

are unacceptable).

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches

6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. At a minimum, explain any "U"ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q SRO U/E/S Explanation FocusI Dist.

D - Link = units ward K/A r5 H 2 V Y N E delete "...expected..." Also, level drop is not credible - provide at least one more credible distractor r6 H 2 Y N E delete "...expected..."

77 F 2 Y N S 78 1H 2 f IY N E add procedure reference 9 F 2 -Y N E valves isolate on high rad alarms: provide at least one more credible distractor 30 F 2 Y N S 31 F 2 Y N E delete "...expected..."

ý2 F 2 Y N U essentially same type of Q as #83; replace either 82 or 83 33 F 2 Y N U see comment above 35 F 2 Y N S 36 H 2 Y N S 37 H 3 x Y N E provide a reasonable time span to "following the loss of power" 38 F 2 Y N S 39 F 2 Y N E set up stem to provide a loss of some piece of equipment as the effect on the SAC

)0 F 2 Y N S

)I F 2 Y N S

)2 1F 2 Y N E add procedure reference

)3 F 2 Y N S 4 F 2 Y N E provide list of annunciator groups; ask examinee to pick priority

)5 H 2 _ N S 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LODrrrt (F/H) 1(1-5)IStem Cues T/F ~Cred. ~Partial Job- Minutia~ #/IBack-I Q= ~SRO U/EJS Explanation Focus I I Dist. Link Iunits ward jOnly IK/A 36 F 2 Y N S

)7 F 2 Y N S

)8 F 2 X Y N E reasons in distractors C &D are the same; modify at least one

)9 F 2 Y N1 E delete ... expected ... "

100 H 2 Y N S 45 of 46 NUREG-1021, Revision 8, Supplement 1

ES-401 SRO-Only Written Examination Form ES-401-9 Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 1 6. 7.

Q# LOK LOD (F/H) I(1-5) Stem Cues TIF Cred. Partial Job- Minui #/ Bak Q= SRO U/E/S Explanation Focus I Dist. I Link I units ward A IOnl I 36 F 2 Y Y S o0 F 2 Y Y S 51 H 2 Y Y S 57 H 3 Y Y S 36 H 2 Y Y S 37 F 2 Y N S System Q, ID'd as 55.41 by facility r1 H 2 Y V E Suggest distractor C be 60 gpm 72 H 2 Y lY S 3 H 2 Y N S System Q, ID'd as 55.41 by facility 4 H 2 Y Y S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if ajob content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

NUREG-1021, Revision 8, Supplement 1 44 of 46

ES-401 Written Examination Form ES-401-9 Re.vie. Wnrk.*hi_=_

11. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem CuesI T/F Cred. Partial Job- Minutia #1 Back- Q SRO U/ES Explanation Focus - 1 Dist. I Link j runits I ward _ _ _ __ __y 75 IH 2 Y Y S 76 F 2 Y N S System Q, ID'd as 55.41 by facility 77 F 2 Y Y S 78 H 2 Y Y E Replace should w/required T9 H 2 Y Y S 0 H 2 Y Y S 31 H 2 Y Y S 32 H 1 X Y Y U DIRECT LOOKUP IN PROVIDED REFERENCE, QUESTION BASICALLY TELLS YOU ITS INOP 33 F 1 N N U KNOW WHAT "ARPI" IS, ELIMINATE 3 DISTRACTORS KA IS WHAT IS PURPOSE, YET Q IS HOW IT WORKS 35 F 2 Y Y S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.