ML20267A398

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML20267A398
Person / Time
Site:	Seabrook
Issue date:	09/07/2020
From:	Thomas Setzer NRC Region 1
To:
Shared Package
ML19105A186	List: IR 05000443/2020301 ML19352E303 ML20203M068 ML20254A128 ML20254A272 ML20267A158 ML20267A392 ML20267A398 ML20267A399 ML20268A322 ML20268B437 ML20268C197 ML21025A403 ML21025A405 ML21026A228
References
CAC 000500
Download: ML20267A398 (9)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q LOK LOD Source Status 8. Explanation (F/H) (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward Focus Dist Link K/A Only K/A mismatch, distractor A is correct also. Rewrote Distractor A.

Changed The reactor is still critical to Power Range channels are 6%.

1 H 3 X X B U This indicates that the heat being removed is equivalent to maximum decay heat load. The reactor is subcritical however, 6% power exceeds the required 5% to transition out of FR-S.1 at step 7. This sets the question up so that the student must still utilize knowledge of step 8, but better aligns with the K/A for decay heat only and not heat from power operation.

2 H 3 B S 3 H 3 N E Changed distractors to normal and prohibited 4 H 3 N E Edited stem to what is the first action Changed stem statement to What is the cause of these 5 H 3 B E indications.

K/A not matched well. Added bullet in the stem to discuss alarm.

6 H 3 X B E Added details to stem about alarm response manual.

Subset issue with distractors A and C. Rewrote question with edits.

7 H 2 N S 8 F 2 N E Edited stem to remove bullets and just ask question directly.

Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3. Check the appropriate box if a psychometric flaw is identified:

• Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• Cues: The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).

• T/F: The answer choices are a collection of unrelated true/false statements.

• Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.

• Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).

4. Check the appropriate box if a job content flaw is identified:

• Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

• Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).

• #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

ES-401 Written Examination Review Worksheet Form ES-401-9

• Backward: The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only. (K/A and license-level mismatches are unacceptable.)

6. Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.

7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q LOK LOD Source Status 8. Explanation (F/H) (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward Focus Dist Link K/A Only 9 H 3 N E Changed question to station blackout conditions.

10 F 2 N S 11 F 3 N S 12 H 3 N S 13 F 2 N S K/A Mismatch. Question rewritten to ask for the reasons 14 H 3 X B U why pertaining to actions in EOPs for loss of instrument air.

15 F 3 B S 16 H 3 B E Typo in stem on RH-V22 corrected 17 F 2 B E Rewrote distractor A to make more plausible.

18 H 3 B E Typo form - edited to from 19 F 2 N S 20 F 3 M E C is implausible. Rewrote C 21 F 3 B S Edited stem to next and edited distractor A to continue to 22 H 3 N E lower A is correct also. Changed distractor trip the reactor to Insert control rods to 10 steps above the RIL.

23 H 3 X N U Added in accordance with OS1000.07, Approach to Criticality?.

24 F 2 B S Upped the difficulty by having candidate calculate subcooling 25 H 2 B E needed for question instead of being given the subcooling.

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q LOK LOD Source Status 8. Explanation (F/H) (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward Focus Dist Link K/A Only Changed question from Can one CCP be stopped and why, or why not? to Can one CCP be stopped at this time and why, or why not?

Changed C from: No. Subcooling must be greater than the value required in step 13c. This ensures adequate subcooling will be maintained after one CCP is stopped.

to: No. Subcooling must be greater than the value required in step 13c Changed D from: No. Subcooling requirements apply until hot leg temperatures are below 250 F. Then adequate subcooling will be maintained after one CCP is stopped.

to: No. Subcooling requirements apply until hot leg temperatures are below 250 F.

C2 and D2 implausible - equalizing boron is not connected 26 F 3 X N U with PTS. Rewrote C2 and D2.

27 H 3 B E Stem edited to check that the penetrations are isolated 28 F 3 B S B is implausible, a mistake should not result in a positive 29 F 2 B E outcome. Rewrote question.

30 H 3 B S A is not plausible Changed RCS Temp column to Stable in 31 H 3 E U distractor A3.

32 F 3 X N U C and D implausible. Edited C1 and D1 33 H 3 N S 34 F 3 N S Edited stem to automatic and made distractors look more 35 F 3 B E similar 36 H 3 B S

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q LOK LOD Source Status 8. Explanation (F/H) (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward Focus Dist Link K/A Only There is no time element in the stem to know when things are occurring, so A could be correct also. Edited stem and distractors to include heaters and PORVS.

37 H 3 X B U Changed question from PT-455 failing low to 457 failing low with an alternate channel alignment. This avoids overlap with Sim Scenario #1.

38 F 2 B S 39 H 2 M S 40 F 2 B S K/A mismatch, question does not ask about bus power 41 H 3 X B U supplies. Rewrote question.

K/A mismatch, question does not ask about bus power 42 H 3 X N U supplies. Rewrote question.

43 F 3 B S Stem and distractors read awkwardly. Rewrote stem and 44 H 3 B E distractors to include due to.

45 H 3 B S 46 H 3 B E Simplified the stem to less bullets.

Question does not ask about D/G load limits. Rewrote 47 F 2 X B U question.

48 F 3 B E Typo, changed stem to de-energized 49 F 3 N E Added IAW procedure to stem 50 F 3 B S 51 F 3 N S 52 F 3 B E A is implausible. Rewrote A

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q LOK LOD Source Status 8. Explanation (F/H) (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward Focus Dist Link K/A Only 53 H 2 B S Rewrote distractors to ask only 1 part instead of 2 part 54 H 3 B E dstractors. Rewrote question to include both Phase A and B.

55 H 3 B S A2 and B2 are not plausible as AFD is not a concern at 56 F 3 X N U startup. Rewrote distractors.

57 H 3 B S 58 H 2 N S 59 H 3 B E Edited distractors to two columns instead of three Question required to be rewritten. Original question had 60 F 2 N E implausible distractor and poor stem focus 61 H 3 B S 62 H 2 B S 63 F 3 N E Added plausibility statements for A, B, and C.

64 H 3 N S 65 H 2 N S 66 F 2 B E Edited stem from could to would 67 F 3 N S Increased the difficulty by changing the distractors to 68 H 2 N E unidentified leakage from pressure boundary 69 F 2 B S

ES-401 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q LOK LOD Source Status 8. Explanation (F/H) (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward Focus Dist Link K/A Only 70 H 3 B S 71 F 3 B S The question is too system specific for Tier 3. Question has 72 F 3 N E been rewritten.

73 F 3 B S The question is too system specific for Tier 3. Question has 74 H 3 B E been rewritten.

75 H 3 N S 76 H 3 B S Rewrote question so that SRO must select recovery 77 F 3 X N U procedure.

A2 and C2 are not plausible. Rewrote question based on 78 H 3 X N U further feedback on using reference Distractors C and D do not reference procedures like A and B 79 H 3 N E do. Rewrite stem to include procedure name and edited all distractors.

80 H 2 N S B and D are implausible. There is nothing in the stem on loss of emergency coolant, so ECA-1.1 is not plausible. Rewrote 81 H 2 X N U distractors. Added and in accordance with that procedure, why to question stem.

K/A mismatch. Rewrote question to test ability to diagnose 82 H 3 X N U rod on bottom.

83 H 3 N E Edited distractors referring to TS 3.3.1 action 6 (B and D) 84 H 3 B S Added per ES-0.2, Natural Circulation Cooldown to 85 H 2 N S question stem.

ES-401 8 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q LOK LOD Source Status 8. Explanation (F/H) (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward Focus Dist Link K/A Only Edited stem to reflect prior to stopping D RCP. Also edited 86 H 2 N E distractors to remove stop the D RCP 87 H 3 B S 88 H 3 X N U Not SRO only as a TS bases question. Rewrote question 89 H 3 N S 90 H 3 X N U K/A not matched. Rewrote question.

Edited distractors to remove the statement Verify that the 91 H 3 B E containment ventilation systems are isolated, since this is in all 4 distractors.

Changed:

Radiation Monitor RM-6549, Lo Range Spent Fuel Pool goes into high alarm. What procedure will be entered (1) and what actions will be taken to minimize personnel exposure (2)?

92 H 3 N E to:

Plant conditions:

• Radiation Monitor RM-6549, Lo Range Spent Fuel Pool is in high alarm.

• Spent Fuel Pool Level is 25.5 feet.

Added in FR.H-1, Response to Loss of Secondary Heat 93 H 3 N E Sink? to question stem.

B and D are implausible because they are CROs. Rewrote 94 F 2 X N U distractors 95 H 3 N E Rewrote distractors to 2x2 format, 2 column 96 F 3 B S 97 H 3 B S 98 F 2 B S 99 H 2 B S

ES-401 9 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q LOK LOD Source Status 8. Explanation (F/H) (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward Focus Dist Link K/A Only 100 H 3 N E Edited stem to state when ECA-0.0 was entered (at step 5)