ML15281A141

NRC Comments on Draft Licensee-Developed Exam (Written) (Folder 2)

ML15281A141
Person / Time
Site:	Millstone
Issue date:	08/14/2015
From:	David Silk Operations Branch I
To:	Dominion Nuclear Connecticut
Shared Package
ML15072A211	List: IR 05000423/2015301 ML15078A430 ML15243A191 ML15281A139 ML15281A140 ML15281A141 ML15281A142 ML15281A143 ML15281A144 ML15281A146 ML15281A147 ML15281A148 ML15357A489
References
ES-401-9, TAC U01903
Download: ML15281A141 (6)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 H 3 B s 3 H 2 N s 4 H 3 N s 5 H 2 B s 6 F 2 N s 7 H 3 N s Editorial stem changes: Provide conditions to indicate SIG are not available. Delete plant mode reference.

8 F 3 B s

" Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 B E Minor editorial stem change to delete redundant statements.

10 F 2 B s 11 H 3 N E Minor editorial change to change the tense in the stem and answer choices.

12 H 2 M s 13 H 3 B s 14 H 2 B E Minor editorial change to stem and answer choices to reduce the question from two parts to one.

15 H 3 B s 16 H 3 N N u KAMM; Revise Q to have the plant in ECA-1.1 17 F 2 B s 18 H 2 N E Minor editorial change to stem to add "minimum" time requirement for the bus transfer time. Also editorial stem change to delete redundant statements.

19 F 2 B s 20 H 3 N s 21 F 2 B s 22 F 2 B s 23 H 3 B E Minor editorial change to stem and answer choices to reduce the question from two parts to one.

24 H 3 M s 25 H 2 M E Minor editorial change to delete redundant statements and change "would" to "will" in the stem.

26 H 2 B E Minor editorial stem change to delete redundant statements.

27 H 3 M s 28 F 2 B s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KJA Only 29 F 2 N E Minor editorial change to stem to delete redundant statements and associate the question with appropriate plant reference procedure.

30 H 2 x N u 'B' and 'D' second part of the answer choices are not plausible distractors; replace the plausible distractors with new ones.

31 H 3 N s 32 H 2 N s 33 H 2 N N u KJA MM; redraw based topic not part of normal plant specific evaluation.

34 F 2 B E Editorial stem change to remove breaker specific reference.

35 H 2 B s 36 H 2 B E Editorial stem change to add time reference, and enhance answer choices c and d to tail pipe temperature scale.

37 F 2 B E Make 'C' answer choice more plausible.

38 H 3 N s 39 F 3 B E Editorial stem change to add reference document.

40 N s 41 F 2 B s 42 F 2 B E Editorial stem change to add initial plant conditions in stem, and remove valve ids in answer choices to increase LOO.

43 F 2 B s 44 H 2 B E Editorial stem change to remove "if at all".

45 H 2 N E Editorial stem change to add timeframe reference.

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 F 2 B E Minor editorial stem change to delete redundant statements.

47 H 3 N s 48 H 2 N s 49 H 2 B s 50 F 2 B E Minor editorial stem change to better match KIA.

51 H 2 B s 52 F 2 N E Minor editorial stem change to delete unnecessary statements.

53 F 2 B s 54 F 2 B s 55 F 2 B E Minor editorial stem change to delete redundant statements.

56 H 2 B E Minor editorial stem change to lower case words.

57 H 2 B s 58 H 2 N E Minor editorial stem change to provide indication that annunciator "just" came in to distinguish hi vs hihi setting.

59 F 2 N s 60 F 2 B s 61 F 1 B u LOO - 1; Develop New Q per KIA.

62 H 2 B E Minor editorial stem change to remove "likely" from stem.

63 H 3 N s 64 F 2 N s 65 F 2 B E Replace answer choice 'B' with more plausible distractor.

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5} Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 2 B s 67 F 2 B s 68 F 2 N N u KIA MM; Replace Q with new.

69 F 2 N N u KIA MM; reselect new KIA based on topic not part of normal job function.

70 H 2 B E Minor editorial stem change to add Pzr solid parameters.

71 F 2 B s 72 F 2 N B u KIA MM; Develop New Q per KIA.

73 H 2 N s 74 H 2 B E Minor editorial stem change to add timeframe and replace 'B' plausible distractor.

75 F 2 B s 76 H 3 y B s 77 H 3 y B E Editorial: Make 'B' more plausible; 530 psig; delete RCP trip criteria 78 H 3 y N s Editorial: Change the stem question to make it SRO related 79 H 3 y B E Editorial: B answer choice - change grammar to "is not" 80 H 4 y N E Add output bkr closure to part C & D to make it more plausible 81 H 3 y B E May need to evaluate to remove 3rd bullet or come up with another way of saying manual start of Charging pump was unsuccessful 82 H 3 y B s 83 H 3 y B s 84 H 3 y B s 85 H 3 N y N u KIA MM; the Q does not test inadvertent CIA. Replace Q.

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 3 y B s Editorial - Add KIA match info related to CVCS to the explanation.

87 H 3 y B s 88 H 3 y M s 89 F 3 y N E Add word "only" to choice 'C' 90 F 2 y B E Add the noun name for 3LWS-RE70, if needed.

91 H 3 y N E Editorial; Match A & B wording to match description of "P-A" 92 H 3 y N s 93 H 3 y N E Recommend not provide table 3.7.3 and change the answer choices C & D to match A & B with 25.5% power to make it more plausible 94 H 3 y B E Add AFW status in the stem. Change feed setpoint to 21 % (the procedure refers to 21%); delete the first word of ALL answer choices (Attempt & Immediately) 95 F 2 y B E LOW LOD; see if that can be enhanced or be replaced.

96 F 2 y N s 97 H 3 y M E Delete in stem "The SM will notify RE" (2"d bullet) to make C the only correct answer 98 F 3 N y B u KIA MM; Replace Q.

99 H 3 y B s 100 F 2 y N E Editorial - Change part 2 of 'C' to "manually discharge the C02 system" to balance out the answer choices.

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N s 2 H 3 B s 3 H 2 N s 4 H 3 N s 5 H 2 B s 6 F 2 N s 7 H 3 N s Editorial stem changes: Provide conditions to indicate SIG are not available. Delete plant mode reference.

8 F 3 B s

" Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

.. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information) .

.. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc) .

The answer choices are a collection of unrelated true/false statements .

. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable .

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem) .

4.

..Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content) .

.. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory) .

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons) .

The question requires reverse logic or application compared to the job requirements .

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 9 F 2 B E Minor editorial stem change to delete redundant statements.

10 F 2 B s 11 H 3 N E Minor editorial change to change the tense in the stem and answer choices.

12 H 2 M s 13 H 3 B s 14 H 2 B E Minor editorial change to stem and answer choices to reduce the question from two parts to one.

15 H 3 B s 16 H 3 N N u KAMM; Revise Q to have the plant in ECA-1.1 17 F 2 B s 18 H 2 N E Minor editorial change to stem to add "minimum" time requirement for the bus transfer time. Also editorial stem change to delete redundant statements.

19 F 2 B s 20 H 3 N s 21 F 2 B s 22 F 2 B s 23 H 3 B E Minor editorial change to stem and answer choices to reduce the question from two parts to one.

24 H 3 M s 25 H 2 M E Minor editorial change to delete redundant statements and change "would" to "will" in the stem.

26 H 2 B E Minor editorial stem change to delete redundant statements.

27 H 3 M s 28 F 2 B s

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KJA Only 29 F 2 N E Minor editorial change to stem to delete redundant statements and associate the question with appropriate plant reference procedure.

30 H 2 x N u 'B' and 'D' second part of the answer choices are not plausible distractors; replace the plausible distractors with new ones.

31 H 3 N s 32 H 2 N s 33 H 2 N N u KJA MM; redraw based topic not part of normal plant specific evaluation.

34 F 2 B E Editorial stem change to remove breaker specific reference.

35 H 2 B s 36 H 2 B E Editorial stem change to add time reference, and enhance answer choices c and d to tail pipe temperature scale.

37 F 2 B E Make 'C' answer choice more plausible.

38 H 3 N s 39 F 3 B E Editorial stem change to add reference document.

40 N s 41 F 2 B s 42 F 2 B E Editorial stem change to add initial plant conditions in stem, and remove valve ids in answer choices to increase LOO.

43 F 2 B s 44 H 2 B E Editorial stem change to remove "if at all".

45 H 2 N E Editorial stem change to add timeframe reference.

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 46 F 2 B E Minor editorial stem change to delete redundant statements.

47 H 3 N s 48 H 2 N s 49 H 2 B s 50 F 2 B E Minor editorial stem change to better match KIA.

51 H 2 B s 52 F 2 N E Minor editorial stem change to delete unnecessary statements.

53 F 2 B s 54 F 2 B s 55 F 2 B E Minor editorial stem change to delete redundant statements.

56 H 2 B E Minor editorial stem change to lower case words.

57 H 2 B s 58 H 2 N E Minor editorial stem change to provide indication that annunciator "just" came in to distinguish hi vs hihi setting.

59 F 2 N s 60 F 2 B s 61 F 1 B u LOO - 1; Develop New Q per KIA.

62 H 2 B E Minor editorial stem change to remove "likely" from stem.

63 H 3 N s 64 F 2 N s 65 F 2 B E Replace answer choice 'B' with more plausible distractor.

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5} Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 66 F 2 B s 67 F 2 B s 68 F 2 N N u KIA MM; Replace Q with new.

69 F 2 N N u KIA MM; reselect new KIA based on topic not part of normal job function.

70 H 2 B E Minor editorial stem change to add Pzr solid parameters.

71 F 2 B s 72 F 2 N B u KIA MM; Develop New Q per KIA.

73 H 2 N s 74 H 2 B E Minor editorial stem change to add timeframe and replace 'B' plausible distractor.

75 F 2 B s 76 H 3 y B s 77 H 3 y B E Editorial: Make 'B' more plausible; 530 psig; delete RCP trip criteria 78 H 3 y N s Editorial: Change the stem question to make it SRO related 79 H 3 y B E Editorial: B answer choice - change grammar to "is not" 80 H 4 y N E Add output bkr closure to part C & D to make it more plausible 81 H 3 y B E May need to evaluate to remove 3rd bullet or come up with another way of saying manual start of Charging pump was unsuccessful 82 H 3 y B s 83 H 3 y B s 84 H 3 y B s 85 H 3 N y N u KIA MM; the Q does not test inadvertent CIA. Replace Q.

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #I Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 86 H 3 y B s Editorial - Add KIA match info related to CVCS to the explanation.

87 H 3 y B s 88 H 3 y M s 89 F 3 y N E Add word "only" to choice 'C' 90 F 2 y B E Add the noun name for 3LWS-RE70, if needed.

91 H 3 y N E Editorial; Match A & B wording to match description of "P-A" 92 H 3 y N s 93 H 3 y N E Recommend not provide table 3.7.3 and change the answer choices C & D to match A & B with 25.5% power to make it more plausible 94 H 3 y B E Add AFW status in the stem. Change feed setpoint to 21 % (the procedure refers to 21%); delete the first word of ALL answer choices (Attempt & Immediately) 95 F 2 y B E LOW LOD; see if that can be enhanced or be replaced.

96 F 2 y N s 97 H 3 y M E Delete in stem "The SM will notify RE" (2"d bullet) to make C the only correct answer 98 F 3 N y B u KIA MM; Replace Q.

99 H 3 y B s 100 F 2 y N E Editorial - Change part 2 of 'C' to "manually discharge the C02 system" to balance out the answer choices.