ML18079A812

Final Qa/Related Forms - Form ES-401-6 and Form ES-401-9 (Folder 1)

ML18079A812
Person / Time
Site:	Limerick
Issue date:	03/27/2018
From:	Operations Branch I
To:	Brian Fuller Exelon Generation Co
Shared Package
ML17138A091	List: IR 05000352/2017302 ML17151A980 ML18038B410 ML18038B436 ML18067A416 ML18067A417 ML18067A430 ML18079A546 ML18079A812 ML18088A662 ML18088A664
References
U01956
Download: ML18079A812 (10)

Text

ES-401 Written Examination Quality Checklist Form ES-401-6 Facility: Limerick Date of Exam: 12/18/17 Exam Level: RO JZ] SRO ~

Initial Item Description a b* c*#

1. Questions and answers are technically accurate and applicable to the facility.

y~ .-,.1/(! ( f

2. a. NRC K/As are referenced for all questions.

b.

c.

Facility learning objectives are referenced as available.

Correct answer explanation and distractor analysis provided (ES-401, D.2.g) T/!J 1.1/::f f

3. SRO questions are appropriate in accordance with Section D.2.d of ES-401

'14/J  ;,Jlt! f 4 The sampling process was random and systematic. (If more than four RO or two SRO questions were repeated from the last two NRC licensing exams, consult the NRR/NRO OL program office). 1/JJ /; fj,'r!:/*/I f='

5. Question duplication from the licensee screening/audit exam was controlled as indicated below (check the item that applies) and appears appropriate.

/,. The audit exam was systematically and randomly developed, or

_ the audit exam was completed before the license exam was started, or

_ the examinations were developed independently, or

) ( the licensee certifies that there is no duplication, or r-

_ other (explain).

6. Bank use meets limits (no more than 75% from the bank, Bank Modified New at least 10% new, and the rest new or modified); enter the actual RO/SRO-only question distribution(s) at right. 32/8 9/1 34/16 ,;'~

7. Between 38 and 45 questions of the questions on the RO Memory C/A exam and at least 13 questions of the questions on the SRO-only portion of the exam are written at the comprehension/analysis level (see ES-401, D.2.c); enter the actual RO/SRO-only question distribution(s) at right.

31/4 References/handouts provided do not give away answers or aid in the elimination of 44/21

1(/t(

lr. -l, r

~

8.

distractors. /1 ;/i((

9. Question content conforms to specific KIA statements in the previously approved examination outline and is appropriate for the tier to which they are assigned; deviations are justified. /.,y/ft f

10. Question psychometric quality and format meet the guidelines in Appendix B.

'Ii. /

/; .li(i { F

11. The exam contains the required number of one-point, multiple-choice items; the total is correct and agrees with the value on the cover sheet. /1 ,i!t1 f" Printed Name/Signature Date

a. Author -[;J/ A !l,u /IJ/tJ.1/11

b. Facility Reviewer(*)

. I./~

/." . ,<.I; L ,,

c. NRC Chief Examiner(#) 11/z.z/tz

d. NRC Regional Supervisor 1¥22-/17

( /

\..__..../"

Note: The facility reviewer's initials or signature are not applicable for NRG-developed examinations.

1. Independent NRC reviewer initials items in Column "c"; chief examiner concurrence is required,

LES-401 Limerick FY18 Initial -- Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 2 y B s 2 H 3 y B s 3 H 2 y N s 4 F 3 y B s 5 H 3 y N s 6 F 2 y M s 7 H 2 y N s 8 F 2 y B s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 *range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 y N s 10 H 3 y B s 11 H 3 y N s 12 H 2 y N s 13 F 2 y B s 14 H 2 y N s 15 H 3 y N s 16 H 3 y N s 17 H 4 y N s 18 H 3 y N s 19 F 2 y N s 20 H 3 y M s 21 F 2 y N s 22 F 3 y B s 23 H 4 y M s 24 H 3 y B s 25 H 3 y B s 26 H 2 y N E 27 H 3 y N s 28 F 3 y N s

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #! Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 F 2 y B s 30 F 2 y B s 31 F 3 y N s 32 H 3 y B s 33 H 2 y B s 34 F 2 y N s 35 F 2 y N s 36 F 3 y N s 37 H 3 y B s 38 H 3 y M s 39 H 3 y N s 40 F 2 y M s 41 F 2 y N s 42 F 2 y B s 43 H 3 y N s 44 H 3 y B s 45 H 2 y M s 46 H 2 y B s 47 H 3 y B s 48 H 3 y B s

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 H 3 y M s y

50 H 2 B s y

51 H 2 B s y

52 H 2 B s y

53 H 3 N s y

54 F 3 N s y

55 F 2 B s y

56 H 2 B s y

57 F 2 B s y

58 F 3 B s y

59 F 3 B s y

60 F 3 N s y

61 F 2 B s y

62 H 2 N s y

63 H 2 N s y

64 H 2 N s y

65 F 3 N s y

66 F 2 B s y

67 H 2 N s y

68 F 2 M s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only y

69 F 2 N s y

70 F 1 B u LOD=i GET Knowledge, Replaced Q y

71 F 2 B s y

72 H 3 N s y

73 F 3 M s y

74 H 2 B s y

75 H 2 X B E Implausible distracters, enhanced distracters to improve plausibility y

76 H 3 y N s y

77 H 3 y N s y

78 H 2 y B E y

79 H 2 y N s y

80 F 2 y N s y

81 H 3 y N s y

82 H 3 y N s y

83 H 2 y M E y

84 H 2 y N E Doesn't meet SRO only requirement to select a procedure, fixed N

85 H 2 X y N u KIA MM, multiple implausible distracters, fixed y

86 H 2 X y N u Multiple implausible distracters, fixed y

87 H 2 y M s y

88 H 2 y B s

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only y

89 H 2 y B s y

90 H 3 y N s y

91 F 3 N M E Doesn't meet SRO only reqmt to select procedure, fixed y

92 H 3 y N E Doesn't meet SRO only reqmt to select procedure, fixed y

93 H 2 X y N u Multiple implausible distracters, fixed y

94 F 2 y N s y

95 F 2 y N s y

96 H 2 X y B u Multiple implausible distracters, replaced Q y

97 F 2 y N s y

98 H 2 y B s y

99 F 2 y N E y

100 F 3 y B s

Question 4 (1740951) -Added Transfer switches do not need to be in the "Emergency" for HPCI key to work.

Question 5 (1741393) -Added more information around bypass flow path to the explanation of the question.

Question 16 (1797416) - changed trips to TRIPs Question 17 (2029824) - Removed "s" from minutes and changed occurring to occur.

Question 26 (1799092)- Changed question away from asking "Confirming Indication". Changed from

" ... what is an additional confirming indication identified by procedure E-2AY185?" to " ... what annunciator/indication serves as an additional confirmation identified by procedure E-2AY185?" and changed column 2 header from 11 Additional confirming indication" to 11 Additional confirmation". This was due to the concern that the confirming indication of E-2AY185 is three items and the annunciator listed is only one of the three items.

Question 29 (1799143) - Changed Logic to logic.

Question 35 {2029750) -Added 11 using 2A RHR" to stem (Suppression Pool Spray is in service)

Question 37 (1799196) - Removed first 11 is" from distractor D Question 43 (1799263) - Corrected answer explanation as a result of removing the table from the stem of the question.

11 Question 45 (1799264) -Added the word 11 the" to distractor A. Swap to the 2B ... "; added noun name for HV-51-2F009, RHR SHUTDOWN CLG SUCTION INBRD PCIV {INBOARD); and noun name of ON-121, Loss of Shutdown Cooling.

Question 47 (1799329) - made Not all capitals ... NOT in distractor C.

11 Question 48 (1799332) - Removed, Provided reference(s): E-5, Grid Emergency and GP-5 note 3.1.60 (p16 of rev 175)" from the beginning of the answer explanation.

Question 53 {1799469) -Added the basis for the l51 lowering, to clarify the second column distractor.

Question 55 (1801599) - re-structured the last portion of the stem to draw more attention to the annunciator and the SRM count rates Question 61 {1845774) - Cleaned up explanation and bulleted stem.

Question 62 (1845775) - Change power level to 95% and Circ water temperature to 90 degrees.

Changed temp results to 139 (right) and 141 (wrong). Updated explanation. Changed outside air temperature to 87 °F to keep the same 3 °delta.This was changed at the request of Brian due to plausibility.

Question 67 (1845800) - Updated plausibility explanation for distractors Band C. removed " ... or believes that provided 2 SRMs are Operable core alterations can continue regardless of quadrant application (i.e. fails to recall LCO 3.9.2.c)" and replaced with " ... incorrectly uses attachment 1 (SRM locations, Fuel Assembly Location, etc.)

Question 70 (1846054) - Question was deemed GET and has been replaced.

Question 73 (1846438) - fixed multiple spelling errors in the explanation section.

Question 75 (1846505) - Some distractors were deemed implausible. Change the distractors to only use volume of Cavity and storage pit as distractor and correct volume. Used flow rate for LPCI (10,000 GPM) and flow rate of a loop of Core Spray (6,350 GPM). Lastly, move starting cavity level to 475".

Question 76 (1743064) -Added"-" to 2A-Y160 to stem. Changed left column for A and B, to " ... Loss of 2A-Y160". Made "Loss" "loss." Added " ... and unavailable" to the third bullet.

Question 78 (1744190) - Removed some information from TSO portion of stem. Changed from "The TSO notifies LGS that grid disturbances have resulted in a voltage limit violation for the 500 kV Substation with voltage at 497 KV." To "The TSO notifies LGS that the 500 kV Substation voltage is 497 KV." Added to the explanation that the voltage in the stem would be the result of a grid disturbance.

Question 80 (1744192)-Added Given S55.1.D and S55.2.A. .. to the stem. Added" ... per S55.1.D" to second column for answer A and C.

Question 83 (2027489) - generated new SRO flow chart and fixed K/A reference based on rejected K/ A.

Question 84 (1744249)-Verified words for first column A and B. Added Per T-101 and Per T-112 to the first column. Given: T-101 and T-112 and noun names added.

Question 85 (1740796)- reworked question to eliminate implausible distractors.

Question 86 (1749051) - Removed "1 year= 8760 hours0.101 days <br />2.433 hours <br />0.0145 weeks <br />0.00333 months <br /> and" and fixed explanation for A and B to less than 10%

Question 87 (1749131) - embedded explanation for the requirement to use the synch switch.

Question 91 (1759803) -Added per T-103 and per ON-102 to answers and added Given: ... to the stem.

Question 92 (1763353) -Added per S43.1.A, OT-112 and OT-200 Appendix 1 to the stem and answers.

Question 93 (2029740) - Fixed alarm from Div 4 to Div 3. Change explanation from "misapplies" to "incorrectly recalls."

Question 96 (Bank Question 555189) - distractor C was deemed not plausible, replaced question with bank question 556506. Removed day 3 of 30-day LCO from the stem and replaced distractor B with a more plausible distractor.

Question 99 (1795598) - added more to tie SE-8 and EALs Search for "V160":

1. 16 - Loss of 1B-Y160 and its effect on T-221, T-251, and T-245

1. 21-1B-Y160 confirming indication and reason for restoring PCIG

1. 25 - Power supplies to 1A-V160 and 1B-Y160

1. 76 - Loss of 2A-V160 impact on Shut Down Cooling Search for "V185":

1. 26 - Two power supplies to 185

1. 28 - Event procedure for RBM and whether bypassing the RBM is required.