ML053330596

08-2005 - Initial Draft Written Examination Comments

ML053330596
Person / Time
Site:	Palo Verde
Issue date:	08/31/2005
From:	NRC Region 4
To:	Arizona Public Service Co
References
50-528/05-301, 50-529/05-301, 50-530/05-301
Download: ML053330596 (6)

Text

ES-401PV-08-2005 Written ExaminationForm ES-401-9Review Worksheet Q#1.LOK(F/H)2.LOD (1-5)3. Psychometric Flaws4. Job Content Flaws5. Other6.U/E/S 7.ExplanationStemFocusCuesT/FCred.Dist.PartialJob-LinkMinutia#/unitsBack-ward Q=K/ASRO Only1H3-S 2B*H2-SNRC2003 RO#2; doesn't qualify as "new" - must count as "bank";

3B*H3-Edoesn't qualify as "modified" - must count as "bank"; wording in C and Dis confusing;4H3X-UC not credible and has typo; 5BH3-SNRC2003 RO#5 (same KA also)6H2-S 7BH3-S8F2X-UB and D are both correct; 9B*F3-ENRC2003 SRO#4; doesn't qualify as "modified" - must count as "bank";10H3-SInstructions[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1.Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable). 3.Check the appropriate box if a psychometric flaw is identified:The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).The answer choices are a collection of unrelated true/false statements.One or more distractors is (are) not credible.One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4.Check the appropriate box if a job content error is identified:The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).The question requires reverse logic or application compared to the job requirements. 5.Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6.Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 7.At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met). 33 of 34 NUREG-1021, Draft Revision 9 ES-401 2Form ES-401-9 Q#1.LOK(F/H)2.LOD (1-5)3. Psychometric Flaws4. Job Content Flaws5. Other6.U/E/S 7.ExplanationStemFocusCuesT/FCred.Dist.PartialJob-LinkMinutia#/unitsBack-ward Q=K/ASRO Only 11BH3-S 12BH3-S 13BH3XX-Udoes the stem have enough information to exclude C? Typo in C (toomany "only"s);

14BH1X-Utoo easy - direct lookup; B not credible; 15BH3-S 16BF3-S 17BH3-SNRC2001 RO#36 (same KA also);

18BH3X-U"Loss of MFW" KA topic has already been sampled by Q12, so eitherQ12 or Q18 must be replaced with an unsampled topic (15/17 or 57);

add oF to temperature in stem; C and D are not credible;19H*3X-ULOK is H vice F; this is not RO knowledge (LO doesn't match Q either);

20F2 S21H3-S 22BF2-S23F3-S 24MH3XXUdoesn't test knowledge of REASONS... per KA; D is correct also; 25BH3-S 26BH3X-UD is also correct; 27BH*3-ELOK is H vice F; explanation in comments doesn't match Q; 28BF3-S 29BH3-S30H3-S34 of 34NUREG-1021, Draft Revision 9 ES-401 2Form ES-401-9 Q#1.LOK(F/H)2.LOD (1-5)3. Psychometric Flaws4. Job Content Flaws5. Other6.U/E/S 7.ExplanationStemFocusCuesT/FCred.Dist.PartialJob-LinkMinutia#/unitsBack-ward Q=K/ASRO Only31H2-Eis D the only "potential concern"?;

32H3X-Ueliminate "pressurizer level..." from all 4 distractors; 33BF2-S 34BH2-Echange starting pressure to 1385 psia and the answer would be ~265 0 F;KA in comments doesn't match outline; 35B*F3X-Udoesn't qualify as "modified" - must count as "bank"; B is not credible; 36BF2-S 37BH3-SNRC2003 SRO#3; 38BH3-S 39BH3-S 40B*F3XUdoesn't qualify as "modified" - must count as "bank"; doesn't testknowledge related to ESFAS logic per KA; 41BH3-S 42BF3-S 43BH3-Estem cues answer to Q1 - replace "valves 7 & 8" with "atmospheric";

44BH3-ENRC2003 SRO#17; replace second semicolon on C and D with "to";45H3XX-UA isn't credible; add "only" to B and C; 46BH3-S 47BH3XUdoesn't test knowledge of effect of loss of ac will have on MAJORSYSTEM LOADS per KA; 48MF2X-UD is not credible;49F3-S 50H3XUdoesn't test knowledge of effect of loss/malfunction of EDG on ESFASper KA;34 of 34NUREG-1021, Draft Revision 9 ES-401 2Form ES-401-9 Q#1.LOK(F/H)2.LOD (1-5)3. Psychometric Flaws4. Job Content Flaws5. Other6.U/E/S 7.ExplanationStemFocusCuesT/FCred.Dist.PartialJob-LinkMinutia#/unitsBack-ward Q=K/ASRO Only51F4XX-Udo RO's have to know these LCO's by memory?

52H*3-ELOK is H vice F; 53BF3-S54H3-S 55BH3-S 56BF3-S57F*1X-ULOK is F vice H; reword without negative stem; answer is obvious; 58BF2-S 59B*H3-Edoesn't qualify as "modified" - must count as "bank";

60BF2-S61H3-S 62BF4X-Uis this an RO learning objective?63H3-S 64MF3-S65F3-S 66F2XXUdoesn't seem operationally important for an RO exam; doesn't test abilityto operate plant communication systems per KA; answer is not supported by reference provided; 67B*F3-Edoesn't qualify as "modified" - must count as "bank";

68BF2-Echange "downpowered" to "powered" in D; 69BF3-S34 of 34NUREG-1021, Draft Revision 9 ES-401 2Form ES-401-9 Q#1.LOK(F/H)2.LOD (1-5)3. Psychometric Flaws4. Job Content Flaws5. Other6.U/E/S 7.ExplanationStemFocusCuesT/FCred.Dist.PartialJob-LinkMinutia#/unitsBack-ward Q=K/ASRO Only 70BF2-S 71BF2X-UD is not credible;72H3X-Uadd "only" to A and B; 73BF2-S 74BF2-SNRC2001 RO#99 (same KA also);

75BF3X-Uis this RO knowledge? it looks like an SRO LO and Q; S1BH3X-XUNRC2003 RO#63;C and D not credible; this is RO knowledge(55.41.b.10);S2H3X--Eis the D also correct?

S3MH3--SS4H3--S S5H3--Ereplace "it's" with "its";

S6H3X-XUA is a subset of D; this is RO knowledge (55.41.b.10);

S7H3-XUthis is RO knowledge (55.41);

S8BH3XXUdoesn't test ability to determine/interpret CHARGING SUBSYSTEMFLOW INDICATOR AND CONTROLLER for PZR level control malfunctions per KA; this is RO knowledge (55.41);

S9BH3-XUthis is RO knowledge (55.41.b.10);S10BH3-XUthis is RO knowledge (55.41.b.10);S11BH3---SS12H3XX--Ushouldn't amps be more than "slightly above zero" for this condition? A isnot credible;S13H3X--UC is a subset of D;34 of 34NUREG-1021, Draft Revision 9 ES-401 2Form ES-401-9 Q#1.LOK(F/H)2.LOD (1-5)3. Psychometric Flaws4. Job Content Flaws5. Other6.U/E/S 7.ExplanationStemFocusCuesT/FCred.Dist.PartialJob-LinkMinutia#/unitsBack-ward Q=K/ASRO OnlyS14H3XX--Uchange "rapid" to "rapidly" in stem; A is also correct; S15H3--S S16H2XX--Uas worded, there are no correct answers; S17H*2X--Eestablish symmetry in distractors between A to C and B to D; S18H3X--Ehow much of the EAL tables are being provided? Appears to be a directlookup;S19BH3X--UNRC2003 SRO#18 (same KA also);currently, there are no correct answers; eliminate "must" fromthe end of the stem, put "must" in A, "may" in C and D, and replace "wait..." with "but only with TSC approval" in C;S20BF3XXUshould the graph be provided?; this is knowledge of a plant procedure,not 1-hour TS per KA; this is RO knowledge (55.41.b.10);S21BF3--SS22BH2--SS23BF2X--UB and D not credible;S24BF2XXUKA on Q doesn't match outline; this is RO knowledge (55.41.b.10);S25H2X--Ehow much of the reference will be provided (and will this be a directlookup)?General Comments:1. How was outline developed? It looks like KA statements wereselected to match bank Qs.

2. There were no question histories provided (no indication of when any of the Bank Qs might have appeared on previous NRC exams).

3. 7 repeats from last two NRC exams on RO exam (2 on SRO) - non of which identified by licensee.NOTE: Bank questions are indicated by B; Modified are indicated by M;comment resolution is indicated in blue;34 of 34NUREG-1021, Draft Revision 9