ML15103A566

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML15103A566
Person / Time
Site:	Seabrook
Issue date:	03/23/2015
From:	David Silk Operations Branch I
To:	Entergy Nuclear Operations
Shared Package
ML14254A318	List: IR 05000443/2015301 ML15076A091 ML15098A055 ML15098A059 ML15098A101 ML15098A126 ML15103A565 ML15103A566 ML15103A567 ML15103A569 ML15104A069 ML15119A306 ML15119A310
References
TAC U01915
Download: ML15103A566 (6)

Text

ES-401 SEABROOK Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 2 N s 2 H 2 N s 3 F 3 N s 4 H 2 x x B u Fix wording in B; Negative testing on KA. Replace Q.

5 F 2 D B E Replace/modify D 6 H 3 B s 7 H 3 D N E Modify D 8 F 2 B s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 H 2 B s 10 H 2 B s 11 H 2 x N E Delete ?1h bullet 12 H 3 x B E Create 3 columns to clean up choices.

13 F 2 A B u Is A technically correct? Modify A.

14 H 3 x N E Put rods in manual.

15 F 4 N s 16 H 2 x D N E Replace D 17 H 3 N s 18 H 2 B s 19 H 3 M s 20 H 2 B s 21 H 2 x B E Typo in C - bypasses vs bypass 22 F 1 N u Low LOD. Modify distractors to be more discriminating.

23 H 2 B s 24 F 4 x N E Modify B, C, D.

25 H 4 B s 26 F 4 B s 27 H 2 B s 28 H 2 x B E Make first bullet a complete sentence.

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 29 H 2 x N E Indicate xenon-free conditions.

30 F 2 B s 31 H 1 N u Low LOO. Replace Q.

32 F 2 x B E Modify A to include a numerical value.

33 F 3 C,D M u Why check pressure to check if cooling is needed? Replace C&D.

34 F 2 B s 35 H 1 N u Low LOO. Replace Q.

36 H 2 N s 37 H 2 B s 38 H 3 B s 39 H 3 x B E Is ih bullet necessary? Delete. Add CNMT pressure.

40 H 3 x B E Typo in 3rd bullet- Tech vs tech 41 H 2 M s 42 F 2 N s 43 H 2 x N E Capitalize .Qharging 44 H 3 B s 45 H 2 M s 46 H 2 N s 47 F 3 x B E Change "does" to "will" in B, C, D

~ ..t'~,8 H 2 N s

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 H 2 x B E What is purpose of 3'd bullet? Delete 50 H 2 B s 51 H 2 B s 52 F 2 M s 53 H 2 N s 54 F 3 x x B u Provide initial plant conditions? KAMM - IAS I CAS. Reselect.

55 H 2 B s 56 H 2 B s 57 F 3 x B E Possible subset issue - reword Q.

58 H 3 x N E Make bullets full sentences.

59 F 2 x N E Create choices grounded in factual data. 2,3,4,5. Add IAW.

60 F 2 D N E Replace D - provide input 61 F 2 N s 62 F 3 N s 63 H 3 B s 64 H 3 N s 65 H 2 N s 66 H 3 N s 67 F 2 B s bn~8 F 3 B s

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 69 H 2 x B E Replace EOL with burn-up data.

70 F 2 B s 71 H 2 B s 72 H 2 B s 73 F 2 C,D M u Replace C & Dor modify Q. Subset issue due to "less than."

74 F 2 B s 75 F 2 x C,D B u Provide initial conditions. Clarify C & D for plausibility 76 F 3 N s 77 H 2 x N E Add ECA-1.2 to another distractor.

78 H 2 A B E Replace A- why go to E-1 for EFW flow issue?

79 H 2 B s 80 H 2 A N E Replace A 1

81 H 2 x B E Delete 5 h bullet to avoid confusion.

82 H 2 x D N E Modify D.

83 H 2 x D N E Add "basis" to stem. Replace D.

84 H 3 B s 85 H 2 B s 86 H 2 x c N E Remove "should" from stem. Replace C.

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 87 H 2 B s 88 H 2 x B u Modify/replace. Can be answered with systems knowledge 89 H 2 B s 90 H 3 N s 91 H 2 B s 92 H 2 x B u Backward subject matter regarding KA. Replace or reselect KA.

93 H 2 B s 94 H 2 x N E Delete unnecessary words. Reverse procedure & reason.

95 H 2 B,D B u Replace 2"d part of B & D 96 F 2 D N E Replace D 97 F 2 B s 98 H 2 B B E Not plausible to not rescue someone 99 F 2 B s 100 F 3 B s