ML14304A660

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML14304A660
Person / Time
Site:	Calvert Cliffs
Issue date:	10/21/2014
From:	Christopher Lally Operations Branch I
To:	Calvert Cliffs
References
TAC U01886
Download: ML14304A660 (6)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KJA Only 1 H 3 X M E D does not seem plausible based upon stem conditions. Also, Explanation of why D is incorrect does not match. Nothing in provided references describes why the correct answer is correct.

2 H 3 X B E Remove 'expected'. Provided reference does not support plausibility of A and B; D plausibility questionable as it is a conflicting statement. A potential second correct answer if 1/3 more power is max power 3 H 3 N s 4 H 2 N s 5 H 3 X N ' E Questionable plausibility of D. Site to provide clarification 6 H 3 B E Add the word 'tube' to component description in answer C 7 H 3 B s 8 F 2 N s Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy- difficult) rating scale (questions in the 2-4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check theThe appropriate box if a job content error is identified:

question is not linked to the job requirements (i.e., the question has a valid KJA but, as written, is not operational in content) . i

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

i The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KJA and those that are designated SRO-only (KJA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

~~-

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 X B E B could also be considered a correct answer because it will accomplish the desired outcome. Suggest re-wording stem to add

'is required next in accordance with procedure' to exclude B since other steps are performed prior to it.

10 F 2 X N s Could this question be viewed as minutia? Is this procedure part of the training program?

11 H 3 N E Basis for correct answer is incorrect, should say 'does not prevent' 12 F 3 N s 13 F 2 X N S/U Site to provide justification how this highlights unit difference, does not appear to meet KIA 14 F 2 N s 15 H 3 X M s 16 F 2 B s 17 F 1 B s 18 H 3 B s 19 H 3 N s 20 F 3 N E Answer D-change RFM to RFP 21 H 3 N E Answer C component description should match its description in the procedure 22 H 3 B s 23 F 3 X N u Based upon stem conditions, does not meet KIA 24 F 3 N E Procedure should be in stem to be more specific and also to prevent possible post-exam comments 25 H 3 X N S/U Potentially SRO level question unless site documents that this is expected RO level knowledge 26 H 3 B s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 27 H 3 B s 28 F 2 B s 29 F 2 N s 30 H 3 X N E D not plausible if minimum system knowledge applied 31 H 3 N s 32 F 2 N s 33 H 3 X N s 34 H 3 N s 35 F 3 N s 36 H 3 N s 37 H 3 N E Too little margin between CCW pressure (approx 80 psia or 95 psig) and RCS pressure at 100 psi a) 38 H 3 X M E Unsure how A is plausible. Both controllers cannot go in the same direction and even if they did, pressure would not rise.

39 H 3 B E Remove 'lowered', let the applicants determine this. Wouldn't backup heaters be on for this insurge?

40 F 2 B s 41 H 3 N s 42 F 2 X N E A could also be viewed as correct, as it bounds B. Need to enhance stem to ensure only B is correct. Give stem conditions of containment pressure XX# and rising, RCS pressure XXXX# and falling. At which pressure would you first expect... ... ?

43 F 2 X M s This appears beyond RO license level and potentially beyond SRO knowledge level as most training programs train on TS setpoints, not TS setpoint limits. Unsat for job link unless site states that this is expected level of knowledge for RO.

44 F 2 B s 45 H 3 N s __ , __

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO 8/M/N U/E/S Explanation Focus Dis!. Link units ward KIA Only 46 F 3 N s 47 F 3 N s Is this borderline minutia? Does site expect RO license candidates to possess this level of knowledge?

48 H 4 B s 49 F 3 B s 50 F 3 N s 51 H 3 N s 52 H 3 N s 53 F 2 X B u Question does not meet KIA 54 F 3 B s 55 F 3 B s 56 F 3 N s 57 H 3 N s 58 H 3 N s 59 F 2 B E Add some more diversity to the distracters; they all have the same power supply 60 F 3 N E Suggest reviewing this question. 4% LFL is borderline GFE and potentially allows for easy elimination of C and D 61 F 3 B s 62 F 2 N s 63 H 3 B s 64 H 3 N s 65 F 2 N s 66 F 2 B s 67 H 3 B s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N UIE/S Explanation Focus Dist. Link units ward KIA Only 68 H 3 N s 69 F 2 N s 70 H 3 B s 71 F 2 B s 72 F 2 N s 73 H 3 N s 74 F 3 B s 75 F 2 B s 76 H 4 N s Remove stem statement about using provided references 77 H 3 N s 78 F 3 B s 79 H 3 N s 80 H 3 B s 81 H 3 N s 82 H 2 M s 83 H 3 B s Remove stem statement about using provided references 84 H 3 N s 85 H 3 N s 86 H 3 X N u Questionable KIA match 87 H 3 N s 88 F 3 B s 89 H 3 B s Remove stem statement about using provided references 90 H 3 N s I 91 H 3 N s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 92 H 3 B s 93 F 3 N s 94 F 2 B s 95 F 2 X B u Equivalent of 'above the line' TS knowledge; no SRO level of knowledge 96 F 2 B s 97 F 3 X B u KIA mismatch 98 H 3 B s 99 H 3 N s 100 H 3 B s Remove stem statement about using provided references