ML22179A249

2018 Prairie Island Ile ES-401-9 Rev 11 Written Exam Comments-Final

ML22179A249
Person / Time
Site:	Prairie Island
Issue date:	10/16/2018
From:	Randy Baker NRC/RGN-III/DRS/OLB
To:	Nuclear Management Co
Baker R
Shared Package
ML17165A317	List: IR 05000288/2018301 ML18333A306 ML22179A239 ML22179A243 ML22179A245 ML22179A247 ML22179A249 ML22179A250 ML22179A254 ML22179A256
References
Download: ML22179A249 (9)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO (F/H) (1-5) Focus Dist. Link units ward K/A Only B/M/N U/E/S Explanation

General note: Majority of the questions to not provide justification for why the correct answer is correct. This is required in support of the distractor analysis.

NRC: Why would Distractor B be considered plausible?

1 H 3 N S Response: Plausible if applicant thinks Train A SI only removes power to the UV coil.

Question is SAT.

2 F 2 N S NRC: 1 out-of 2 twice. Question is SAT as written.

NRC: 1 out-of 2 twice. [Reference Provided-1E-1 pg. 3]

3 H 3 N S Response: Providing the reference requires the applicant to work through the procedure. Question is SAT as written.

4 H 3 B S NRC: Question is SAT as written.

5 H 2 N S NRC: {Correct K/A IR=4.0}

Response: Agreed - Corrected K/A IR. Question is SAT as written.

Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3. Check the appropriate box if a psychometric flaw is identified:

• Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length).

• T/F: The answer choices are a collection of unrelated true/false statements.

• Cred. Dist>: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

• Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

• Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

• Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).

• #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• Backward: The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only. (K/A and license-level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).

Page 1 of 9 ES-401 2 Form ES-401-9

Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO (F/H) (1-5) Focus Dist. Link units ward K/A Only B/M/N U/E/S Explanation

NRC: [2014 ILT Audit Exam] How does this question relate to a PZR PCS 6 F 2 N B U malfunction? (K/A match)?

S Response: Agreed - Modified Distractor A to read... if PRZR press is less than 2235. Question is now SAT.

NRC: [2014 ILT Exam] Removing Boron will add negative reactivity?!

7 H 2 B S Response: Modified question to remove first part and reorder the distractors.

Question is SAT.

8 H 3 N E NRC: {Correct K/A IR=3.1} [Reference Provided-Steam Tables]

S Response: Agreed - Question is now SAT.

9 H 2 N S NRC: Question is SAT as written.

10 H 3 B S NRC: Question is SAT as written.

11 H 2 M S NRC: {Original Bank question provided} [2016 ILT Exam Significantly Modified]

Question is SAT as written.

12 F 2 N E NRC: {Correct K/A IR=3.7}

S Response: Agreed - Question is now SAT.

NRC : [K/A IR=2.2*; Justification: Important at facility because of loss of remote 13 F 2 N S operating capability (from Control Room) for Safeguards equipment.]

Question is SAT as written.

14 F 3 N S NRC : Question is SAT as written.

15 H 2 M S NRC: {Need to see the original Bank question to ensure modified.}

Response: Agreed - Question is Modified and SAT.

16 H 2 B S NRC : [2017 LOR Exam] Question is SAT as written.

17 F 2 N S NRC : Question is SAT as written.

18 F 2 B S NRC : [2016 ILT Audit Exam] Question is SAT as written.

19 F 3 B S NRC: [2014 ILT Audit Exam] {Add K/A IR=3.6}

Response: Agreed - Question is SAT as written.

20 F 3 N S NRC: Question is SAT as written.

21 H 2 N E NRC: [Reference Provided-Fig. C1-20]

S Response: Removed the reference. Question is now SAT.

22 F 2 N S NRC: Question is SAT as written.

Page 2 of 9

ES-401 3 Form ES-401-9

Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO (F/H) (1-5) Focus Dist. Link units ward K/A Only B/M/N U/E/S Explanation

NRC: [2017 LOR Exam] {Correct K/A IR=3.1} [Reference Provided-pg. 16 of 2C1.3 23 H 3 B S AOP1 and Fig. C1 -10B]

Response: References provided to permit calculation. Agreed - Question is SAT.

NRC: [Reference Provided-pg. 10 of 1FR-C-1]

24 H 2 N S Response: Providing the reference requires the applicant to work through the procedure. Agreed - Question is SAT.

25 F 2 N S NRC: Question is SAT as written.

NRC: [2014 ILT Exam (Similar)] [Reference Provided-1E-0 pg. 5]

26 H 3 B S Response: The reference is provided to preclude reference to minimum AWF flow requirements. Agreed - Question is SAT.

27 F 3 N S NRC: 1 out -of 2 twice. Question is SAT as written.

28 F 2 N E NRC: {Correct K/A IR=3.9}

S Response: Agreed - Question is now SAT.

29 H 3 N S NRC: Question is SAT as written.

30 H 3 B S NRC: 1 out -of 2 twice. [2013 LOR; 2017 LOR Re-Exam] Question is SAT as written.

31 H 3 N S NRC: 1 out -of 2 twice. Question is SAT as written.

32 F 2 N S NRC: 1 out -of 2 twice. Question is SAT as written.

33 H 2 N E NRC: 1 out-of 2 twice. {Correct K/A IR=2.9}

S Response: Agreed - Question is now SAT.

34 H 2 N E NRC: [Reference Provided-1C-14 AOP 1 pg. 5]

S Response: Removed The reference (Immediate action.) Question is now SAT.

35 H 2 N S NRC: Question is SAT as written.

36 H 3 B E NRC: 1 out-of 2 twice. [2014 ILT Audit Exam] {Correct K/A IR=3.6}

S Response: Agreed - Question is now SAT.

37 F 2 N E NRC: 1 out-of 2 twice. {Correct K/A IR=3.9*}

S Response: Agreed - Question is now SAT.

38 H 2 N S NRC: Question is SAT as written.

39 H 3 M S NRC: 1 out-of 2 twice {Original Bank question provided} [2016 ILT Exam Significantly Modified] [Reference Provided-NF-86186-3,-4]

Page 3 of 9

ES-401 4 Form ES-401-9

Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO (F/H) (1-5) Focus Dist. Link units ward K/A Only B/M/N U/E/S Explanation

Response: Applicants required to work through the logic diagrams. Agreed -

Question is SAT.

NRC: {Original Bank question provided.} {Correct K/A IR=3.9 * } [2016 ILT Exam 40 H 3 M E S Significantly Modified]

Response: Agreed - Question is now SAT.

NRC: {Original Bank question provided.} [2016 ILT Exam Significantly Modified]

41 H 3 M E S [Reference Provided-1E-0 Attachment L pg. 4]

Response: R emove the reference. Agreed - Question is now SAT.

42 H 2 N E NRC: {Correct K/A IR=3.2*}

S Response: Agreed - Question is now SAT.

43 F 2 N S NRC: Question is SAT as written.

NRC: Which comes first the reactor or the turbine trip? Is there only 1 correct 44 F 2 X X N U answer? Is it actually B? (Does the turbine trip cause the Reactor trip at power?)

S Response: Agreed - The stem was reworded to ask what the result will be of a loss of Both MFWP, therefore only one answer is correct. Question is now SAT.

45 H 2 N S NRC: 1 out-of 2 twice. Question is SAT as written.

NRC: [Reference Provided-B20.2 pg. 18]

46 H 2 N S Response: Providing the reference requires the applicant to determine the normal line-up. Agreed - Question is SAT.

47 H 3 N E NRC: 1 out-of 2 twice. {Correct K/A IR=2.8*}

S Response: Agreed - Question is now SAT.

48 F 2 N S NRC: 1 out -of 2 twice. Question is SAT as written.

49 F 2 N E NRC: 1 out-of 2 twice. {Correct K/A IR=2.7*}

S Response: Agreed - Question is now SAT.

50 H 2 N E NRC: 1 out-of 2 twice. [Reference Provided-1C20.7 pg. 12-15]

S Response: R emove reference. (Validate Stem ICs.) Agreed - Question is now SAT.

51 H 2 M E NRC: This question does not appear to be modified.

B S Response: Agreed - This is a B question. Question is now SAT.

52 H 2 N S NRC: 1 out-of 2 twice. Question is SAT as written.

53 H 3 M S NRC: {Original Bank question provided.} [2014 ILT Exam Significantly Modified]

Question is SAT as written.

Page 4 of 9

ES-401 5 Form ES-401-9

Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO (F/H) (1-5) Focus Dist. Link units ward K/A Only B/M/N U/E/S Explanation

54 H 2 N E NRC: 1 out-of 2 twice. {Correct K/A IR=2.8*}

S Response: Agreed - Question is now SAT.

NRC: [2015 LOR Exam] [Reference Provided-XH-1-242]

55 H 3 B S Response: Providing the reference requires the applicant to work through the logic diagram. Agreed - Question is SAT.

NRC: [Reference Provided-C47012-0507]

56 H 3 N S Response: Reference provided to support plausibility of distractors. Agreed -

Question is SAT.

57 F 2 N S NRC: Question is SAT as written.

58 H 3 N S NRC: Question is SAT as written.

59 F 3 N E NRC: {Correct K/A IR=2.9}

S Response: Agreed - Question is now SAT.

60 F 2 N S NRC: Question is SAT as written.

H NRC: This appears to be a simple limit value memory question not Higher Cog.

61 F 2 N E S Is Distractor D (30 Hg) plausible for the facilitys physical location?

Response: Agreed - changed Distractor D to 25. Question is now SAT.

62 H 2 M S NRC: {Original Bank question provided.} [2016 ILT Exam Significantly Modified]

Question is SAT as written.

NRC : [ Reference Provided-SP1783.1A pg. 6]

63 F 3 N S Response: Providing the reference requires the applicant to determine which annunciators are expected. Agreed - Question is SAT.

NRC: 1 out -of 2 twice. {Original Bank question provided.} [2017 LOR Exam]

64 H 4 M S [ Reference Provided-PINGP 0045, C25 Fig. 1 ]

Response: Matches K/A! Agreed - Question is SAT.

65 F 2 N S NRC: Question is SAT as written.

NRC: 1 out-of 2 twice. {Original Bank question provided.} [2014 ILT Exam 66 H 3 M S Significantly Modified] [ Reference Provided-Fig. C1-8, TS 3.1.4 pg. 1]

Response: Reference provided to allow calculation for K/A. Agreed - Question is SAT.

E NRC: 1 out-of 2 twice. {Original Bank question provided.} [2016 ILT Exam 67 H 3 B M S Significantly Modified] This question is modified; why is it listed simply as a B question?

Page 5 of 9

ES-401 6 Form ES-401-9

Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO (F/H) (1-5) Focus Dist. Link units ward K/A Only B/M/N U/E/S Explanation

Response: Agreed - Question is now SAT.

NRC: Asking the question in the per TS 3.6.6 Bases... puts this into the SRO-Only arena. Is there another RO knowledge document that also discusses why this is 68 F 2 X N U S done?

Response: Agreed - the purpose is referenced to the procedural requirements.

Question is SAT.

NRC: 1 out-of 2 twice. Again, is there somewhere else that defines what qualified backup equipment is, other than the TS bases? This could also be SRO-Only. {Fix 69 H 3 X N U S the grammar in the stem.}

Response: Agreed - The stem was modified to add load test of portable charger and provided the installation procedure. Question is SAT.

70 H 2 N S NRC: Question is SAT as written.

NRC: How does knowing the general location of an annunciator, demonstrate 71 F 2 N N U knowledge of the process used to track an inoperable annunciator? (K/A match)?

S Response: Stem was modified to instruct applicant to follow required procedure.

Question is SAT.

NRC: 1 out-of 2 twice. [Reference Provided-RWP 180051 Tasks 101/203]

72 H 3 N S Response: Applicant must decide which RWP is appropriate and then implement requirements to answer the question. Agreed - Question is SAT.

73 F 3 B S NRC: [2016 ILT Exam] Question is SAT as written.

74 F 2 N S NRC: 1 out -of 2 twice. Question is SAT as written.

75 F 3 N S NRC: {Fix the grammar in the stem. Remove the first in.} Question is SAT as written.

NRC: How does performing the next Step in 1ES-1.1 (which would be performed regardless) relate to verifying the existence of Natural Circulation for this question?

(K/A Match)? [Reference Provided-1ES-1.1] Why is the reference required for an 76 H 3 N Y N U S SRO to answer this question?

Response: Changed distractor to include action if natural circ ulation is not verified, requiring the use of the reference to support the correct answer. Now matches the K/A. Question is SAT.

NRC: 1 out-of 2 twice. [Reference Provided-TSs-3.5.1 and 3.5.4] Why are the TSs 77 H 3 Y N E S provided as references for an SRO to answer this question?

Response: Agreed - Removed the SRs from the references. Question is now SAT.

78 H 2 Y N S NRC: Question is SAT as written.

79 H 2 Y B E NRC: { Correct K/A Statement for 2.2.40 vs 2.2.22}

Page 6 of 9

ES-401 7 Form ES-401-9

Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO (F/H) (1-5) Focus Dist. Link units ward K/A Only B/M/N U/E/S Explanation

S Response: Agreed - Corrected. Question is now SAT.

80 H 2 Y N S NRC: 1 out of 2 twice. Question is SAT as written.

NRC: How does selecting which contingency procedure to enter relate to the ability to identify post-accident instrumentation for this question? (K/A Match)? [Reference 81 H 3 N Y M U Provided-1E-1 pg. 9-12]

S Response: The applicant must diagnose the ICs, using the EOP reference to select the proper transition point. Agreed. Changed K/A to random selection - 2.4.4. Re-worked the question to match the K/A. Agreed - Question is SAT.

82 H 2 Y N S NRC: Question is SAT as written.

NRC: [Reference Provided-PINGP 1576]

83 H 2 Y N S Response: Reference does not answer any additional questions and requires the applicant to review all possibilities. Agreed - Question is SAT.

84 H 3 Y N S NRC: [Reference Provided-F5 Appendix A Zone 50]

Response: Reference provided to better match K/A. Agreed - Question is SAT.

85 H 3 Y N S NRC: 1 out of 2 twice. Question is SAT as written.

NRC: Perhaps it would be better to reword the stem to state that The Unit Supervisor 86 H 2 N E will direct transition into... [Reference Provided-1ES-1.2]

S Response: Removed the bases section of the reference procedure. Agreed -

Question is now SAT.

87 H 3 Y N E NRC: 1 out-of 2 twice. [Reference Provided-PINGP 577 and F3-2.1 pg. 6-F 17]

S Response: R emove the F3 reference. Agreed - Question is now SAT.

88 H 2 Y N S NRC: [Reference Provided-1ES-1.2]

Response: See Question 86. (Already Provided) Question is SAT as written.

NRC: [Reference Provided-1FR-H.1 pg. 3-10]

89 H 3 Y N S Response: Providing the reference requires the applicant to work through the procedure. Agreed - Question is SAT.

NRC: {Correct K/A IR=4.3} [Reference Provided-1ECA-0.0 pg. 6-7]

90 H 3 Y N E S Response: Providing the reference requires the applicant to demonstrate how to implement the NOTE. Agreed - Question is now SAT.

91 H 2 Y N S NRC: Question is SAT as written.

92 H 2 Y N E NRC: [Reference Provided-C20.3 AOP1 Fig. 5 and TS 3.8.1]

S Response: Remove the SRs. Fig. OK. Agreed - Question is now SAT.

Page 7 of 9

ES-401 8 Form ES-401-9

Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO (F/H) (1-5) Focus Dist. Link units ward K/A Only B/M/N U/E/S Explanation

NRC: 1 out-of 2 twice. [2016 ILT Exam] [Reference Provided-H4 T able 2.2]

93 H 3 Y B S Response: Reference requires applicant to diagnose and implement corrective actions. Agreed - Question is SAT.

94 H 2 Y M E NRC: {Correct K/A IR=4.4} [Reference Provided-C35 AOP6]

S Response: Removed the system diagram. Agreed - Question is now SAT.

95 H 2 Y N S NRC: [Reference Provided-PINGP 1125 pg. 20-21]

Response: Reference provided to match the K/A. Question is SAT as written.

NRC: [Reference Provided-FP-OP-RSK-01 pg. 18]

96 H 3 Y N S Response: Table provided as reference required for calculating answer. Window dressing (ICs) provided for Operational validity. Agreed - Question is SAT.

97 H 2 Y N S NRC: 1 out -of 2 twice. Question is SAT as written.

98 H 2 Y B S NRC: [2016 ILT Audit Exam] Question is SAT as written.

NRC: 1 out -of 2 twice. [Reference Provided-C34 AOP1]

99 H 3 Y N E S Response: Limit the procedure provided to the first 20 pages. Agreed - Question is now SAT.

100 H 2 Y B S NRC: [2014 ILT Audit Exam] Question is SAT as written.

B=14 F=30 E=21 RO TOTALS: M=10 H=45 U=5 Additional Notes: H = 38-45; N 8; B 56 N=51 B=4 F=0 E=8 SRO TOTALS: M=2 H=25 U=2 Additional Notes: H 13; N 3 ; B 18 N=19

GENERAL COMMENT

S:

1. Question numbers highlighted as xx are sample test items the facility submitted for pre-review.

2. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

3. Questions from the previous 2 NRC Exams: _ 3 (8 )_ (RO)/ _ 1 (2)_ (SRO) [Max (without OLB concurrence) is 4 RO and 2 SRO.]

4. Average difficulty is ___2.44____ on the RO exam and ___2.44____ on the SRO exam.

Page 8 of 9

ES-401 9 Form ES-401-9

Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018

5. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 41.8 = 43.1 =

41.2 = 41.9 = 43.2 =

41.3 = 41.10 = 43.3 =

41.4 = 41.11 = 43.4 =

41.5 = 41.12 = 43.5 =

41.6 = 41.13 = 43.6 =

41.7 = 41.14 = 43.7 =

6. The answer distribution is:

RO / SRO A = 0 (17%) / 0 (24%)

B = 0 (22%) / 0 (28%)

C = 0 (28%) / 0 (20%)

D = 0 (33%) / 0 (28%)

7. There are _13_ (RO)/ _16_ (SRO) questions with references/ attachments provided.

Page 9 of 9