ML030070137

Draft - NRC Comments

ML030070137
Person / Time
Site:	Millstone
Issue date:	10/25/2002
From:	Caruso J NRC/RGN-I/DRS/OSB
To:	Wilson M Dominion Nuclear Connecticut
Conte R
References
50-336/03-301, ES-401, ES-401-9 50-336/03-301
Download: ML030070137 (8)

Text

OPERATING EXAM COMMENTS 0 SRO Admin, JPM use "SRO Shift Turnover" as a spare JPM.

0 Revise SRO Admin JPM, "Shift Staffing Requirements" to determine which operators to call based on watchbill information provided.

• Revise SRO Admin JPM, " SRO Review and Apporve a Radiaoactive Liquid Waste Release Permit" to add additional error.

0 Revise RO JPM "RO Shift Turnover" initiating cue to make more focused as to what logs are to be reviewed.

0 JPM, B.1.c, "A" EDG operability Test", all steps up to step #12 cue are completed - these are call out items anyway little evauation value.

0 JPM, B.1.f, "Respond to RCP seal rpoblems" revise initating cue to make less leading eg respond to panel indications.

0 General Comment - Scenario Guides need to be revised to provide all required actions scripted out.

• Sim Guide, ES02LII, revise scenario to force applicant into once through cooling.

• Make guide #3 the spare.

• Revise guide #4 to add a malfunction that causes the computer displays PPC to lockup.

ES-401 Written Examination Millstone 2 Form ES-401-9 Note: Resolutions to comments in bold and italics Review Worksheet Initial Review completed 9/26/02

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 6. 7.

Q# LOK LOD (F/H) (1--5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q $190 U/E/S Explanation IFocus Dist. JLink" I _ Iunits ward JKI IOn-ly I I______________________

1B F 2 E Modified. Error on answ. Justification shows "A"as incorrect. (How was question Mod) correctedJsutification 2B H 3 S Bank 3B H 3 S New 4B H 3 S New 5B H 3 S New 6B H 3 S New 7R H 3 X X U Modified. Stem cue. "A"correct also. "C" not crediible Remove RCS P/T Curve from the stem unnecessary cue. Revised stem and 2 distractors.

8B F 2 S Modified 9B F 3 S New 1OR H 3 S Bank Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

• The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

More than one distractor is not credible.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check the appropriate box if the sampled question does not match the approved K/A or an SRO-only question is not at the SRO level.

6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. For any "U"ratings, at a minimum, explain how the Appendix B psychometric attributes are not being met.

NUREG-1021, Revision 8, Supplement 1 1 of 7 ES-401 Millstone 2 Form ES-401-9 1.

LOK LOD

2. 3. Psychometric Flaws r

4. Job Content Flaws I

• 6. 7.

Q#

(F/H) (1-5) IStem Cues T/F Cred. Partial Job- Minutia #/ Back- ý5RQl U/E/S Explanation Focus _____jDist. j Link lunits Jward RIA4 .Onty.,I I_________________________

1IB F 3 S Modified 12B F 2 S New 13B H 3 S New 14B F 2 S Bank 15R F 2 S New 16R F 2 X U New. How is "explain covered? Also the reference provided is all sect 3 of TS? Modified stem and distractorto fit W/A.

17R H 3 S Modified 18B F 2 S New 19B F 2 S Bank 20B H 3 S New 21 B H 3 S New 22B H 3 S New 23B H 2 S New 24B H 2 S New 25B F 2 S New 26B F 2 S New 27B F 2 S New 28B H 3 S New 29B F 3 S New 30B H 3 S New

2 of 7 NUREG-1021, Revision 8, Supplement 1 ES-401 Millstone 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws Ofther 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/FqCred. Partial Job- Mutia # o Back-.iooo s Explanation Focus IDist. Link units ward _ __

31B H 3 X U New. Given small steady rise, how can rapid rise be credible "A&D"?

Revised distractorto avoid cuing 32B F 1-2 X U New distractors not plausible and LOD low not idscriminating Replaced question and used same K/A 33B H 3 S New 34B H 3 S Modified? This question not significantly modified per guidance in ES401 treat as a bank 35B H 3 S New 36B F 3 S Modified 37R H 2 S New 38B H 4 S Modified 39R H 3 S New 40B F 2 S Modified 41 B H 3 S New 42R H 3 S New 43R H 3 S New 44R F 2 S Bank 45R H 3 S New 46B H 3 S Modified 47B H 3 S New 48R H 3 S New 49B H 3 S New 50R F 2 S Modified 3 of 7 NUREG-1021, Revision 8, Supplement 1

ES-401 Millstone 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5, Other1 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues TIF Focus _ J Cred. Partial Job- Minutia Dist. Link

1. / Back-units I ward 1 .* 1 SRO onty I U/E/S Explanation 5IR H 3 S New 52B F 3 S New 53B H 3 E New. Change "a"to "c"InJustiflcation.

54R H 3 S New 55B F 3 S Modified 56R F 2 S Modified 57R F 2 S New 58B F 2 S New 59B F 2 S New 60R H 2 S New 61B F 2 E New how are safety inj. Valve interlocks tested with this question throttling a valve is not an interlock Revised answer choice to tit K/A and balance choices to avoid cuing 62R H 3 E New. Minorrevisi9ons 63B H 3 X U Modified. No malfunction of the Pzr LCS. Revised stem to add malfunction to match K/A 64B H 3 S New 65B F 2 S New 66B H 3 S New 67R F 2 S New Elect supply question okay in limited number 68B F 2 X U New How is this question testing containment purge system? Replaced questionsame system different K/A 69B F 2 E Bank. Injustification, change "c" to "d" Corrected Typo 70B H 3 X U New"A,B,C" distractors not plausible TDS, pH, and power lose to rad

-I__ Iu monitor Replaced "A&B" distactors 4 of 7 NUREG-1021, Revision 8, Supplement I

ES-401 Millstone 2 Fnrrn F::-4fl1-.q V

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 57.!Wt 6.

Q# LOK LOD (F/H) (1-5) Stem ICues IT/F ICred. Partial Job- IMinutia #/ Back- Q:, SRO U/E/S Explanation Focus I Dist. Link units ward __A Only 71B F 2 S Bank 72B H 2 E New How would the operator tell that condenser air in leakage/air removal flow doubled in the CR? If this can not be determined then how is this a valid question and not minutia? Revised stem to make more operationallyoriented.

73B F 2 S New 74B H 3 S New 75B H 3 S Bank 76B F 1-2 X U Bank "A&B" distractors not plausible also LOD low does not discriminate Revised distractor 77B F 3 S Modified 78B F 1-2 X U New distractors not plausible also LOD low does not discriminate Replaced question 79B F 2 S New. Could "c" be correct? Reverifled not correct 80B H 3 S Modified 81B F 2 S New 82R F 2 S New 83B F 2 E New revise one distractor to be simular to "D" correct answer stands out also edit stem Revised "C" distractor 84R F 3 X S New 85B F 3 S New 86R H 3 S Bank 87R H 3 S New 88B F 2 E New. "Changing" should be "increasing" to fit theanswer Edit completed 89B H 3 X U/E New. K/A does not cover "ability" to obtain.... Accepted as an approveddeviation per recentguidance

Q# LOK LOD (F/H) (1-5) Stem Cues T/Fl Cred. Partial Job- Minui #/u IBack- Q, SRO U/E/S r r - 1 Focus jj III Dist. Link I II I!

units ward KIA OnIt Ne Explanation 90B 1 F 12 X U New This appears to be minutia. Is this required knowledge from t-I

=~~~~~~~ ~I ~~~~~~~~~~

..... L....1......L...L....I__Imemory Replaced question using same KWA.

91B F 2 S New 92B H 2 S New 93B F 1-2 U New LOD is low does not discriminate. This Is get level not license specific Replaced question using same K/A 94B H 2 S New 95B F 2 S New 96B F 3 S New 97B F 1-2 S Bank 98B F 1-2 S Modified 99B F 2 X U Bank the kIa tests knowledge of fire prot. Procedures. To answer this question all I need to remember is the titles of the procedures it doesn't really demonstrate any knowledge of the actual procedure. Question replaced using same K/A.

00B H 3 S Bank SRO ONLY 7S H 2 S New 10S F 3 X U New. Ability to determine not covered. Stem revised "What is the bases discussed in TS bases..." Question revised to match K/A 15S H 3 S New 16S H 3 S New 17S H 3 E? New Is it necessary to tell them in the stem what reference to use?

Removed proceduralreference from stem 21S H 3 S Bank 37S H 1 -2 X X U New LOD=1 with reference provided For a TS question give them the situation and let them find the correct answer better tested in an admin

_____JPM or scenario. Revised stem and distractors

ES-401 Millstone 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws I r 6. 7.

Q# LOK LOD I

(F/H) (1-5) Stem Cues T/F

~Focusi _J_

Cred. Partial Job- Minutia #/ Back-Dist. j Link _ _Iunitslward Q,ý SRO U/E/S WKA4Only __ __________________

Explanation 39S H 3 X S New Revise one of the distractors to also include containment purge.

The correct answer stands out and is the only choice also splitand make separate bullets for "Containment Purge and radmonitors" Revised distractos 42S H 2 S New 43S H 2 S New 44S H 2 S New with the curve provided as part of TS as a reference TS bases not provided 45S H 2 S New 48S H 3 S New 50S F 2 S New Will sect 3 of TS be provided as a refence for the exam? Did you indicate which questions require a reference to answer?

51S H 1-2 X U New It could be argued that "D"is not an incorrect answer. Charign pumps are providing SI flow and even though it is extra information it is also at least partially correct. Revised distractor 54S H 3 S New 56S H 3 S New 57S F 1-2 X U New This is really a basic system question and not SRO only and does not discriminate it is a LOD=1. Replaced question using same K/A 60S F 3 S New 62S H 2 X U Bank This is LOD=1 with reference provided. For a TS question give them the situation and let them find the correct answer better tested in an admin JPM or scenario. Rewrote question 67S H 3 S New 82S H 3 S New 84S H 3 X U New. Question deals with knowledge rather than ability. Retained question and sitedproperK/A treatedas site specific lesson learned question.

86S H 2 E New One distractorrevised 87S H 3 S New