ML20303A144

NRC Comments on Draft Licensee-Developed Exam (Written) ES-401-9

ML20303A144
Person / Time
Site:	Millstone
Issue date:	09/18/2020
From:	Dominion Nuclear Connecticut
To:	NRC Region 1
JAckson, D.
Shared Package
ML19105A205	List: ML20302A234 ML20302A235 ML20303A011 ML20303A012 ML20303A014 ML20303A016 ML20303A032 ML20303A144 ML20307A010 ML20307A019 ML20307A064
References
CAC 000500
Download: ML20303A144 (7)

Text

ES-401 Millstone Unit 2 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F? 2 X N E Remove most and replace with an objective term.

2 H 3 X M E Comma needed for clarity in first sentence after lift.

3 F 3 X B N E Add alarm in stem to meet KA. B not plausible - why add water.

4 F 2 X X N U Fix stem to address T/F situation. Replace will with is required in A&D and cannot be in B&C. Q was rewritten.

5 F 2 X N E Reword part 1 Q. Fix column heading with 1 & 2. .

6 H 3 N S 7 H 3 B S 8 F 2 C N E Reword Q for clarity and add that a trip signal was generated..

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

• The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

• The answer choices are a collection of unrelated true/false statements.

• The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

• The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

• The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

• The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 9 H 2 N S 10 H 2 X N E Specify SG1/SG2 in choices.

11 H 3 B S 12 F 1 N U Low LOD. Reselect KA. New Q was provided.

13 H 3 N S 14 H 3 X N E Clarify that VA-20 lost power and shorten choice D.

15 H 3 X B S Remove () from choices.

16 H 2 X N E Insert that in part 1.

17 F 2 N S 18 H? 2 X M E Delete 2nd column and provide KA justification.

19 F? 2 X N E Remove best and reword to make objective.

20 H 1 X X X N U Low LOD. Backward logic. Distractors are not plausible. Rewrite Q.

21 F? 2 X N E Add info that CNMT purge is progress.

22 F 2 C N E Change C to gpd vs gph.

23 F 2 X X N E Eliminate first two lines. Reword Q and switch columns.

24 F? 2 X X N E Reword Q for clarity. State plant condition first.

25 H 3 M S 26 F 2 X M E Choice C - start all remaining charging pumps.

27 F? 2 N S 28 F 1 X A,D N U Low LOD. B is leading. A&D implausible. Reselected KA.

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 29 H 3 X X N E Reword Q to present malfunction and then ask for impact.

30 H 2 X M U KAMM No actions is not using procedures to mitigate. Q rewritten.

31 H 2 X X X B E Add if any to avoid cue.

32 H 3 M S Explain justifications for C & D.

33 H? 3 N S 34 F 2 N S 35 H 2 N S 36 F 3 N S 37 H 2 N S 38 H 2 X M E Eliminate spray column.

39 F 4 N S 40 H 3 B S 41 H 3 X X N E Reword Q for clarity and ease of reading.

42 H 2 X N E Typo - future vs futures.

43 H 2 M S 44 H 3 N S 45 H 4 X X M U KAMM. Subsets A&B and C&D. Reselect KA.

46 F 3 X M E Ensure C is wrong. D doesnt seem plausible. Modify Q for clarity.

47 H 3 X M E Reword Q to elicit proper response.

48 F 3 X N E Add regarding DC buses and loads to Q.

444

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 49 H? 2 B S 50 H 2 X B M E Provide power levels to make B plausible.

51 F? 1 M U LOD 1. GET type Q. Reselected KA.

52 H 2 N S 53 F 2 N S 54 F 2 X N E Do not provide info about SOVs. Reword Q to assess loss of DC.

55 F 3 X M E Make a statement about TS being followed.

56 H? 2 X M E Add procedure to stem.

57 H 2 M S 58 H 2 X A N E Best should be avoided. Change choice A.

59 F 2 X B M E Replace A or B with borated water source. Use preference in stem.

60 F 2 X N E Insert a lead-in line for Q.

61 H 3 B S 62 H 3 X N E Remove could. Replace with will to make Q more definitive.

63 F 2 B S .

64 F? 3 X N E Add question marks.

65 H 3 A/C B E Put Unit 3 in one other choice.(C?)

66 H? 2 X X N E Modify Q to be applicable for RO quals.

67 F 4 X B E Delete first line and ask about TS staffing requirements.

68 F? 3 A B E Reword A2 to avoid subset issue with B.

696CC

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 69 H 3 X B? B E Ask per procedure. Verify that B is incorrect.

70 H? 3 N S 71 F 2 X B E Reword to include operator activity in CNMT.

72 F 2 B S 73 F? 2 S B S 74 F 2 N S 75 H 2 X M E Make Q singular vs plural. Provide alarm in stem to meet KA.

N = 43; M = 18; B = 14 NRC :F = 34 (45.3%), H = 41 (54.7%); Licensee F = 31 (41.3%),

H = 44 (58.7%). F?/H? indicates disagreement w/ licensee.

76 H 2 X X N U Answered by systems knowledge. KA reselected. Q replaced.

77 H 3 X N E Add per procedure or per TS to end of choices.

78 H 2 X A N E Reword Q. Make A & C symmetrical.

79 H 2 X X N U Answered by system knowledge. Reword to select and take action.

80 H 3 A,C X N U KAMM - Q does not test instrument bus. Q was rewritten.

81 H 3 B S Previous NRC exam Q.

82 H 2 B S 83 H 2 N S 84 H 3 X N E Reword stem.

85 H 2 X M E What are vs Describe. Add procedures to A&C.

86 H 2 X N E Add ONLY to B and D.

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 87 H 3 N S 88 H 2 X B E Add per TS to stem.

89 H 3 X B E Specify procedural location. Faulty 2x2. Just ask when.

90 H 3 X A N E Reword Q wrt RCS temperature limits. A not plausible. Clarify B.

91 H 2 X X N U Systems knowledge and AOP entry conditions. Reword to select procedure and take actions.

92 H? 2 N S 93 H 3 X N E Typo - delete first in.

94 H 2 X B E Faulty 2x2. Ask for only the first part.

95 F 3 X N E Remove phrase about TS bases from stem.

96 F 2 B E Same knowledge was tested on Q90. Q as written was acceptable.

Replace Q to avoid overlap with Q 90.

97 F? 3 X N U Q not SRO level.

98 F 2 D N E Make D be DSEO.

99 H 2 X N E Add worsened. Use a valve for calculated leak rate. SG not faulted.

100 F? 2 X N E Rework Q to be more discriminating.

+ - From previous two exams

- met the criteria for a modified question

• - did not meet the criteria for a modified question RO - 6/75 UNSAT = 8.0%

SRO - 5/25 UNSAT = 20.0%