ML17003A282

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML17003A282
Person / Time
Site:	Beaver Valley
Issue date:	10/24/2016
From:	NRC Region 1
To:	Division of Reactor Safety I
Shared Package
ML16076A416	List: IR 05000334/2016302 ML16089A007 ML16292A483 ML16342C521 ML17003A276 ML17003A277 ML17003A278 ML17003A280 ML17003A281 ML17003A282 ML17003A283 ML17003A284 ML17003A285
References
U01922
Download: ML17003A282 (6)

Text

ES-401 BEAVER VALLEY 1 Written Examination Review Worksheet Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 2 N s 2 H 2 N s 3 H 2 N s 4 F 3 N s 5 H 2 x N E Typo in 3rd bullet (Reactor vs React). Add "ONLY" to A 1 & B 1. 6 H 2 N s 7 F 2 N s 8 H 2 x 0 B* E Replace o. Instructions

[Refer to Section O of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOO) of each question using a 1 -5 (easy-difficult) rating scale (questions in the 2 -4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified: . The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) . . The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) . . The answer choices are a collection of unrelated true/false statements . . The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable . . One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) . 4. Check the appropriate box if a job content error is identified: . The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) . . The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) . . The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) . . The question requires reverse logic or application compared to the job requirements . 5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f. 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #I Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 H 2 x N E Pictures need to be bigger/clearer.

10 F 2 B N E B is a weak distractor.

Replace. 11 F 2 x B E Clarify 3rd bullet and reword 2nd Q. 12 H 2 B* s 13 H 2 x N E Delete 3rd bullet. 14 F 2 x N E Add"ONLY"toA1

&C1. 15 F 2 B s 16 H 2 x N E Add IAW FR-H.1 to 2nd Q. Ensure only one possible answer. 17 F 2 B/D B u B1/D1 not plausible for ECA-1.1 condition.

Replace. 18 F 3 x N E Replace "reason" with "coincidence and logic". 19 F 2 x N E Reduce wording. 20 F 2 N s 21 F 2 x N E Minimize wording in stem. 22 F 3 c N E Modify first part of C (i.e., perform power reduction).

23 H 3 N s 24 F 2 B s 25 H 2 B s 26 H 2 x M E Remove recirc sptay pumps from choice D. 27 F 2 B s 28 H 2 B s ES-401 3 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 2 x B E 1st set of bullets not necessary.

Delete. 30 H 2 x B E Minimize/reword for clarity. 31 H 2 x B,C B u Just say blockage occurs in Hx. Replace B & C. 32 F 2 x N E Remove "coincidence/coincident" from Q. Add "signal".

33 H 2 B s 34 H 2 N s 35 F 3 x x N u Eliminate bullets. Address T/F statements.

Reword. 36 H 2 x B E Make answer be something other than "Steam is Saturated".

37 H 2 x N E Change Q to diagnose spray valve being open. 38 F 3 B s 39 H 3 B s 40 F 2 N s 41 F 2 B,D B u Eliminate 4th item. Not plausible that that valve would close. 42 F 2 B N E Steam dumps not addressed by TS. Replace with TS-related item. 43 H 2 x B E Delete parenthetical phrase. 44 F 2 x N E Add wording to A to have the same length as other distractors.

45 F 2 N s 46 H 2 B s 47 H 2 N s ,,j8 H 2 N s ES-401 4 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO 8/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 F 3 8 s 50 F 3 x N E Add "at least" to 2nd part to eliminate possible subset issue. 51 H 2 M s 52 F 2 x N E Re-word question for simplicity.

53 H 2 N s 54 H 2 8 s 55 F 2 8 s -56 H 2 N s 57 H 2 N s 58 H 2 x N E Reword Q for simplicity.

59 F 2 8 s 60 F 2 x N E L vs H. Specify time frame. 61 H 2 N s 62 H 3 x N E Clarify statement about ineffective control room actions. Replace C. 63 F 2 N s 64 H 2 x N u There are 4 or 5 other GFE-type Q's on exam. Reselect.

65 F 2 N s 66 F 2 x 8 E Capitalize a word in each distractor.

67 H 3 N s hJ38 F 2 x 8 E Spell out RPS.

ES-401 5 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #I Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 69 F 2 B s 70 F 2 B s 71 H 2 B s 72 F 2 B E 73 F 2 D N E D not plausible.

74 H 3 B s 75 H 2 N s H=39->52%

F=36->48%

U=5->6.7%

76 H 2 x N E Provide indications of RCP trip criteria.

Minimize wording. 77 H 2 N s 78 H 2 x N u Q can be answer with systems knowledge.

Reword or reselect.

79 H 3 B s 80 H 3 x N E Delete "3.5.1". Add "at latest" to specify time frame. -. 81 H 2 .. B s 82 H 2 x N E System Q with and EAL twist. Reword or reselect.

83 F 2 N s 84 H 2 B s 85 H 2 B s 86 F 2 N s ES-401 6 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 87 F 2 N s 88 H 2 C,D B u Reducing AFW flow not plausible.

Replace C & D 89 H 2 N 90 H 2 x N E Plant in Mode 5 (vs 4). Enter Mode 4 (vs 3). 91 F 1 x B u Low discriminatory value. KAMM -no mention of Ni's. 92 H 2 x N u KAMM. Q should be about a LOCA vs SGTR. 93 H 2 x x N E Reword to avoid stating that ASDV is open. Minimize wording. 94 F 2 B s 95 H 2 B s 96 H 2 x B E Replace 3'd bullet with a condition that requires assessment.

97 H 2 x B E Capitalize wording in B. 98 L 2 D N E Replace D -"1 only". 99 L 2 x B E Typo 7th bullet -fuel vs Fuel. 100 H 2 B s