ML23200A241

Form 2.3-5 Written Examination Review Worksheet

ML23200A241
Person / Time
Site:	Fermi
Issue date:	06/12/2023
From:	NRC/RGN-III/DORS/OB
To:
Bryan Bergeon
Shared Package
ML22007A060	List: ML23200A239 ML23200A240 ML23200A241 ML23200A242 ML23200A255 ML23200A256 ML23200A257 ML23212A873 ML23213A030
References
Download: ML23200A241 (1)

Text

Form 2.3-5 Written Examination Review Worksheet

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 1

H 3

N S

NRC:

1)

No facility learning objective listed.

Licensee:

1)

An objective link has been made.

2 H

3 N

M S

NRC:

1)

Question is modified from 2010 Exam Q48 and/or 2006 Exam Q57.

Licensee:

1)

Changed question pedigree to Modified.

2)

Added original question (Q48 from 2010 Exam) to folder containing modified questions.

3 H

2 N

S 4

H 3

N S

5 H

2 B

S 2019 NRC Retake Exam NRC:

1)

Two 10CFR 55.41 references listed.

Recommend b(7).

Licensee:

1)

Removed reference to 55.41 (b)(5).

6 F

3 N

E NRC:

1)

Two 10CFR 55.41 references listed.

Recommend b(7)

2)

Is the DC MCC area fire zone 3L? If it is, there is no reason not to clarify that in the stem.

Licensee:

1)

Removed reference to 55.41 (b)(5).

2)

Added information to stem to clarify that the DC MCC area (Zone 14) is a 3L zone.

NRC:

1)

Question is SAT.

7 H

2 N

S NRC:

1)

Two 10CFR 55.41 references listed.

Recommend b(8)

Licensee:

1)

Removed reference to 55.41 (b)(10) 8 H

2 X

N U

NRC:

1)

Question stem is using a cause and effect relationship - the East SAC must be started at 95 pisg to prevent outboard MSIV closure. However, the 20.129.01 bases do not provide a pressure at which

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level MSIVs start drifting closed; only that outboard MSIV closure occurs on a loss of IAS. It appears that starting a SAC at 95 or 85 psig would accomplish preventing MSIV closure. Not convinced a correct answer exists.

Licensee:

1)

A new question version has been written.

NRC:

1)

Question is SAT.

9 H

3 B

S 2017 Exam NRC:

1)

Two 10CFR 55.41 references listed.

Recommend b(7)

Licensee:

1)

Removed reference to 55.41 (b)(10) 10 F

2 B

S 2019 Exam 11 H

3 B

E 2017 Exam NRC:

1)

If the stem states the crew enters EOPs on high DW temp, a given temp should be provided in the stem, as its pertinent information. Bank question (2017 Q11) included DW temp in the stem; why did this one not?

Licensee:

1)

It looks like the question was cleaned up to make it easier to read by breaking the paragraph(s) up into individual sentences separated by hard returns. Its hard to say, but perhaps that is how the DW temp was removed from the stem. Regardless of the reason, the DW temp information has been restored.

NRC:

1)

Question is SAT.

12 F

2 X

X N

U S

Free Look NRC:

1)

1. 1 in the stem is an obvious answer and D can be easily eliminated. B21-R623A, Post Accident Monitoring Recorder is clearly an instrument used to monitor reactor pressure and would be required by the Post Accident Monitoring LCO. Cueing eliminates 1 distractor but does not provide for the answer. Question is an E

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

2)

Question sources state NEW, however, answer explanation states it is modified from Q84 from the 2020 NRC Exam. This paragraph in the answer explanation can be removed, as this is a NEW question.

Licensee:

1)

Corrected implausible distractor.

2)

Removed paragraph from answer explanation and double-checked question is classified as NEW.

NOTE that Instrument 3 was changed during validation due to comments from validators that the original Instrument 3 was implausible since it is located outside the MCR and nothing in the stem indicated that control room evacuation was necessary.

NRC:

1)

Move instruments to below question stem and rephrase stem to ask which instrument(s) below would you use..

2)

What is the AOP/EOP procedure tie to this question such that it fits in tier 1? Appears to be a tier 2 question.

Licensee:

1)

Moved instruments to below the question stem and rephrased question as requested.

2)

Added information to the stem and to the question explanation to make a stronger tie to the EOP that would have been the controlling procedure for the conditions given in the stem (29.100.0, RPV Control EOP).

NRC:

1)

Question is SAT.

13 F

3 N

E NRC:

1)

Consider referencing 20.000.25 for a failed SRV, as the AOP directs placing the mode switch to shutdown at 110 deg. While PC Control EOP provides that direction (its not entered until 95 deg torus temp), the AOP does as well, and the question stem could easily reference the AOP. E due to minor enhancement to meet tier 1 requirements.

2)

Question appears modified from the 2003 Exam Q39 Licensee:

1)

Added 20.000.25, Failed SRV to both the stem and explanation of the question.

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

2)

This question was developed independently, and without referring to any bank question. Although similar to Q39 on the 2003 exam, that question does not exist in the current exam bank (it is so old, it only exists as a word document) so it was not referenced during development of this exams question. The Fermi exam team believes the question is NEW based on this statement from NUREG-1021 ES-4.2 (page 4 of 29) second bullet from the top:

Generally, if a question is created without referring to a bank question, it can be considered a new question.

NRC:

1)

Question is SAT.

14 H

2 N

S 15 H

3 B

E 2019 Retake Exam NRC:

1)

Reactor Scram AOP should be more formal - The crew has entered 20.000.21, Reactor Scram, and has performed all Immediate Actions.

Licensee:

1)

Revised stem as suggested.

NRC:

1)

Question is SAT.

16 H

3 B

E 2020 Exam (previous 2 exams)

NRC:

1)

Include valves numbers; Open the CRD Pressure Control Valve should read Open C1152-F003, CRD Drive/Clg Water PCV.

Want to ensure proper equipment nomenclature referenced (unless these valves also have this informal nomenclature in the control room, procedures, etc).

2)

Consider capitalizing (or otherwise emphasizing) valves positions -

THROTTLE OPEN Licensee:

1)

Valve numbers have been included and nomenclature changed to accurately represent the wording used in the procedure, 29.ESP.04, and to be more formal.

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

2)

We considered making this change but decided to stick with the format used for exam submittal because it is consistent with the format used in the procedure (29.ESP.04) as is our response to Comment 1) above.

NRC:

1)

Question is SAT.

17 H

3 B

E NRC:

1)

Could not find the question sourced on a previous NRC exam, however, it appears modified from 2006, SRO Q3.

2)

Distractor D to defeat RPS automatic logic trips is the first step directed in both Section 4.0, scram reset and manual scram re-initiation (distractor C) and section 6.0, pen scram test switches (distractor D).

Therefore, if D is wrong, C is also wrong.

Why is distractor D not referencing Section 4.0 for Scram Reset and Manual Scram Re-initiation since that is the section referenced in the explanation?

Licensee:

1)

We found this question in our Bank but could not find evidence of it being used on a previous NRC exam, which is why we called it Bank. We usually put this in the Question Explanation section but failed to do so. Recommend calling it Bank for this reason. This was added to the question explanation to clarify its source.

2)

Distractor D (defeat RPS logic trips) was replaced with a new distractor (vent CRD over-piston volumes) to remove the appearance of a sub-set issue.

NRC:

1)

Agree that question is bank. Question is SAT.

18 F

2 B

S 2019 Retake Exam 19 F

2 X

B U

2020 Exam (previous 2 exams)

NRC:

1)

Question is a tier 2 systems question and can be answered only with systems knowledge - no AOP/EOP tie to meet tier

1. Should have some tie to the Plant Fire

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level AOP (also discussed in the selected learning objective).

2)

Consider adding to the answer explanation

- only 1 pre-action sprinkler system exists within the plant, protecting the Warehouse A loading dock.

Licensee:

1)

A new question has been written that focuses on the relationship between installed fire suppression equipment and entry into the Plant Fires AOP to make the question a Tier 1 match.

2)

N/A since a new question has been written.

O/V Week comment:

1)

Need to spell out title of AOP in stem.

Licensee:

1)

Added proper title of AOP to stem.

NRC:

1)

Question is SAT.

20 F

2 B

S 2018 Exam NRC:

1)

Two 10CFR 55.41 references listed.

Recommend b(5)

Licensee:

1)

Removed second 10CFR 55.41 reference.

21 H

3 B

E 2017 Exam NRC:

1)

RPV level of 178 and lowering should have lowering emphasized similarly to the previous parameter trends given in the bulleted conditions.

Licensee:

1)

Emphasized lowering for 178 for consistency.

NRC:

1)

Question is SAT.

22 H

3 X

X N

U NRC:

1)

Specify which EOPs entered.

2)

The question stem allows for subset issues in the answer/distractors (distractor B is partially true). If the stem asked What are the next SEQUENTIAL actions the CRLNO should perform?, the answer/distractors should be fine as written.

Licensee:

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

1)

Specified EOP entered in the stem.

2)

Reworded question as recommended.

O/V Week:

1)

Capitalize the word MAXIMIZE in the stem.

Licensee

1)

Capitalized MAXIMIZE in stem.

NRC:

1)

Question is SAT 23 H

3 N

S NRC:

1)

Is there any validation via references that RR seal purge remains unisolated?

Considering including referencing M-5703-1 in answer explanation.

Licensee:

1)

Note 1 of 20.106.01 indicates that starting the pump will impact RR seal purge pressures, which implies the seal purge line does not get isolated prior to pump start. Reference to M-5703-1 was removed because it doesnt show how Ref Leg backfill is isolated.

24 H

F 3

X N

E S

NRC:

1)

Per 29.100.01 SH 2, if torus level > 570 ft, the preferred vent path is via the DW. The explanation references that the torus is correct as torus level is < 570 ft. This is not given in the stem, so how is the applicant to assume this given the plant conditions?

2)

The K/A is high suppression pool level. In this case, the vent path being the torus is due to low torus level.

3)

Question is low-cog.

Licensee:

1)

The question, as written, was meant to be generic in nature. Since it is on the RO exam, we need to be careful that we dont put numbers in the stem that would require them to evaluate and choose the correct procedure path, since that is SRO knowledge. Instead, the question is simply asking, of the two vent paths, which one is preferred (the Torus) and when (generically speaking) is that preferred path no longer available (when TWL is high enough that the preferred vent path becomes submerged). No changes were made to the question for this reason.

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

2)

Changing from the preferred vent path (the Torus) to the less-preferred vent path (the Drywell) occurs when TWL is >570 feet.

This is the point where the Torus event path becomes submerged and is extremely high in the Torus, which matches the K/A, which is knowledge of implications of High Suppression Pool Water Level on Venting the suppression pool.

3)

Changed Question to Fund.

NRC:

1)

Question is SAT.

25 H

2 N

S NRC:

1)

Two 10CFR 55.41 references listed.

Recommend b(10)

2)

Reference trip criteria for RBHVAC in distractor D explanation.

Licensee:

1)

Removed second 10CFR 55.41 reference.

2)

Added trip criteria for RBHVAC in distractor D explanation.

NRC:

1)

Question is Sat.

26 F

F H

2 3

N M

U S

Free Look NRC:

1)

Question appears to be a tier 2 (systems) question rather than a tier 1 (emergency procedure) question. Question can be answered with system level (and general plant layout) knowledge and no knowledge of emergency / abnormal procedures.

Licensee:

1)

A new K/A was selected and new question written for Q26.

NRC:

1)

No learning objective listed.

2)

Question is modified from 2018 exam Q25.

3)

Question is high-cog.

Licensee:

1)

Made objective link.

2)

Changed question source to Modified.

3)

Changed to High Cog.

NRC:

1)

Question is Sat.

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 27 H

3 N

E O/V Week:

1)

Due to the poor validation and rarity of operation of the switch in #1, is it possible to write another version of this question that hits the K/A?

Licensee:

1)

New question version written.

NRC:

1)

New version of question deleted the switch in #1. Modified version is SAT.

28 H

3 B

S 2019 Exam 29 F

2 X

N E

S NRC:

1)

Question is similar to 2017 exam Q32 but is different enough to be considered New.

2)

Distractor D is easily eliminated for RO applicants, as they are not required to know 72-hour action requirements from memory.

Licensee:

1)

Will maintain question source as New.

2)

This question is also taken by SRO applicants who might select this option because of their more in-depth knowledge of LCO Actions beyond 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />. We will need to discuss this further as Distractor D is plausible for SRO applicants. Also, an RO applicant could fail to recall from memory (this is a closed-book question requiring knowledge of TS Conditions and Actions) all of the combinations of INOPERABILITIES that would require entry into LCO 3.0.3. If the examinee concluded that this combination did not require entry into LCO 3.0.3, then they could plausibly conclude that D is the correct response.

NRC:

1)

Agree with licensee comment #2. Question is SAT.

30 H

3 B

S Free Look NRC:

1)

Distractor explanations show A,C,D, while the correct answer is C. Correct lettering in distractor explanation.

Licensee:

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

1)

Lettering for distractor explanations was corrected.

Please note that minor changes have been made to this question, following validation, to improve readability.

NRC:

1)

Question is SAT.

31 H

F 3

B S

2019 Exam NRC:

1)

Question is low-cog Licensee:

1)

Changed Cog Level to Fund 32 H

3 B

S 2019 Exam 33 H

3 N

E NRC:

1)

Question is modified from 2021 Exam Q38, however, question is modified such that the question is New.

2)

Recommend adding ONLY to distractor B (Only LPRMs 1 and 3 lose power)

Licensee:

1)

Question source remains New.

2)

Added ONLY to distractor B.

NRC:

1)

Question is SAT.

34 F

2 N

S NRC:

1)

Two 10CFR 55.41 references listed.

Recommend b(6)

Licensee:

1)

Removed second 10CFR 55.41 reference.

35 H

3 B

S 2019 Retake Exam 36 F

2 X

N U

E Free Look NRC:

1)

Distractor C maintain power is relatively subjective. Recommend adding stable to the distractor.

Licensee:

1)

Added stable to Distractor C.

NRC:

1)

MOP19 Section 5.12 states that if THI is occurring, place the mode switch to SD, however, it also states that if the PF map is in the exit region, insert CRAM rods.

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level Therefore, while B may be the most correct answer, A is also correct. Can we modify the stem to make A not partially correct?

Licensee:

1)

Changed the stem to ensure A, insert the CRAM Array, is not also correct.

NRC:

1)

Question is SAT.

37 H

3 B

E 2015 Exam?

NRC:

1)

Could not find this question in the administered 2015 exam. Verify sourcing.

2)

Recommend adding statement &

procedure reference to distractor A explanation regarding if an auto action does not occur, make it happen (MOP01, enclosure E) - this would help to show that, while A would accomplish the goal to secure RCIC, it is incorrect because the operator must take manual actions to accommodate for the failed auto actions.

Licensee:

1)

Could not locate this question on the 2015 exam either. Removed link to 2015 exam, but kept as Bank.

2)

Added reference to MOP01, enclosure E to the explanation for Distractor A and additional information to explain why A is incorrect.

NRC:

1)

Question is SAT.

38 H

3 B

S 2019 Exam 39 F

2 B

S NRC:

1)

Two 10CFR 55.41 references listed.

Recommend b(7)

Licensee:

1)

Removed second 10CFR 55.41 reference.

NRC: Question is Sat.

40 H

3 M

E S

NRC:

1)

Consider referencing alarm tiles for MSL high flow channel trips.

Licensee:

1)

Adding the alarms to the stem would add unnecessary clutter to the question as it is

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level information not needed to answer the question. Also, since this is a Tier 2 question, it can be answered with systems knowledge and is not required to be linked to an abnormal procedure.

NRC:

1)

Discussed and agree with licensee.

Question is SAT.

41 F

3 X

N E

Free Look NRC:

1)

Answer explanation lists distractors A, C, and D. D is the correct answer. Answer explanation has a labeling error - should be A, B, and C (currently show distractor B as C and distractor C as D).

2)

Based on the distractor explanation for distractor C (incorrectly labeled as distractor D), SRV G could be powered from alternate power via manual action.

While the applicant is directed not to make assumptions based on information not provided in the question stem, is this a reasonable assumption to make, based on procedure progression in a ELAP (of Div 2 only) event? Need to ensure partial answers are fully eliminated. It doesnt appear that its reasonable to add a no operator actions taken type statement to the stem.

Licensee:

1)

Corrected distractor explanation labeling error.

2)

The CE brought up a valid concern. During validation two (of 5) operators did select C.

During review, they did mention that power was still available and thought that might make C correct (under certain conditions). I read to them the statement in the NUREG about not making assumptions not stated in the stem (as mentioned above) and, although they accepted this explanation, they were not completely satisfied.

Therefore, the question has been re-worded to add the statement if NO actions are taken outside the Main Control Room to eliminate C as a possibly correct response.

NRC:

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

1)

Recommend reordering C or D to read similar (see below) to allow applicant to more easily highlight the differences:

C: H, E, R, P, J, A and G D: H, E, R, P, J, A, and B

2)

Clarifying statement added to the stem resolved the potential for 2 correct answers Licensee:

1)

Re-ordered the list of SRV in Distractor D as recommended.

2)

This version did not have the same problem as we saw during validation of the original version, so we agree that the change to the stem resolved the conflict.

NRC:

1)

Question is SAT.

42 F

2 N

S 43 F

2 N

S 44 H

3 X

N E

NRC:

1)

Stem states that the condenser pressure high alarm is received (indicating its rising), but then local pressure reading shows pressure is lowering, with nothing in the stem indicating why the trend reversed.

Operational validity concern.

2)

While the condenser pump may not be required for emergency operation, the pump is still required for operation, as it was designed to run and cycle on level.

Applicant could argue this and that there is no correct answer. Recommend modifying answer A to eliminate this potential.

Licensee:

1)

Per ARP 2D62, the alarms comes in at 3 Hg Vacuum, decreasing. The trend in the stem states pressure (vacuum) is 2Hg Vacuum and lowering. Vacuum is lowering (i.e., pressure is rising) so the stem is correct.

2)

Revised Answer A to eliminate the potential argument that no right answer exists.

NRC:

1)

Question is SAT.

45 H

3 N

S

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 46 H

F 2

B S

2019 Retake Exam NRC

1)

Question is low-cog Licensee:

1)

Changed Cog level to Fund.

47 H

3 M

S 48 F

2 N

E NRC:

1)

Insert commas before then in B, C, D Licensee:

1)

Commas added.

NRC:

1)

Question is Sat.

49 H

4 X

N U

S NRC:

1)

The question answers/distractors all state that P50-R801 would indicate 90 psig and lowering, however, the stem states that 7D51 is in, which comes in at 85 psig and lowering on that same indicator. How is the question stem operationally valid?

Licensee:

1)

Lowering NIAS pressure as read on P50-R801 is only one of several initiating devices that can cause 7D51 to alarm. It can also be caused by rising Dryer D/P at 3.0 psid, which is what is being simulated in this question. As Dryer D/P rises, NIAS pressure will lower. At the point in time represented by the question, NIAS pressure has not yet lowered to 85 psig.

Note that this question has been validated twice and operational validity was not raised by the Operators. Recommend that the question stays as-is and its status be changed to Sat.

O/V Week:

1)

Emphasize the fact the Dryer D/P is RISING (all caps)

Licensee:

1)

Changed RISING to all caps in the stem.

NRC:

1)

Agree with licensee comment #1. Question is SAT.

50 H

H 3

3 N

N S

E Free Look NRC:

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

1)

Consider spelling out Temperature Control Valve for TCV.

Licensee:

1)

This question was completely re-written following validation on 2/16/2023. This will require another look by the CE.

NRC:

1)

Is high area air temperature (in the question) the same as RRMG air temperature (referenced in the SOPs)?

Are these terms synonymous (the key word being area which does not appear in the SOP)? Applicant could argue D is incorrect since there is no area air temp trip listed in the SOP and that B is correct.

Licensee:

1)

Distractors B and C, and correct Answer D, have been revised to include RRMG for the lube oil and air temperature trips to be consistent with the SOP.

NRC:

1)

Question is SAT 51 H

2 X

N U

NRC:

1)

Applicant may ask if P43-F400 is open (as it should fail open on low level).

2)

Tier 4 generic question (marked as K/A mismatch - meets K/A, just not in tier 2).

Recommend modifying the question to ask what the concern is and what design feature the pumps have (a low-level trip at 6 in the tank per 23.128) to protect the pump - or something similar.

3)

Answers/distractor should have indications similarly ordered for ease of indication of differences.

Licensee:

1)

A new question version was written that should address all three comments above.

2)

See above.

3)

See above.

O/V Week: New question is sat, with some changes needed:

1)

Change stem per discussion (break up paragraph into sentences).

2)

Correct distractors to remove redundant responses in each distractor Licensee:

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

1)

Changed stem from paragraph to sentences.

2)

Revised distractors per discussion.

NRC:

1)

Question is SAT 52 F

2 N

S 53 F

2 N

S 54 H

3 B

S 2013 Exam 55 F

2 X

N E

S NRC:

1)

Why is the North RWCU recirc pump A not used as a distractor (breaker 72F Pos 2C)?

Seems like this would be a much more plausible distractor than current distractor B.

Licensee:

1)

Each of the power supplies listed in the 3 distractors are power supplies to components in the RWCU system, making each one of them plausible. Since this version of the question is what has been validated, the Fermi exam team is reluctant to make this change since a more plausible distractor is subjective. Note that, during validation, 9 operators took this question. Four got it correct, while 4 answered C and one selected B, indicating that B is a plausible choice.

NRC:

1)

Question is Sat.

56 F

2 B

S 2019 Retake Exam 57 H

2 X

N E

NRC:

1)

Distractor A is not plausible - tripping sump pumps on a sump hi-hi level makes no logical sense.

Licensee:

1)

The explanation for Distractor A was insufficient to explain why it is plausible.

The explanation has been updated to read the following:

Plausible because ODE-10 requires entry into 29.ESP.27, Torus Leak isolation, for a

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level "Large Torus Leak." 29.ESP.27 in turn requires the Torus Area Floor Drain Sump Pumps to be turned OFF. This is to satisfy the requirements of ODE-10, EOP Expectations, to preserve a volume of water in the Torus room to maintain viable Low Pressure ECCS (Core Spray) under the worst-case torus leak scenario. Per Design Calc DC-4689, "If all of the water in the Torus leaks into the Torus Room, level will equalize at a Torus level of approximately -97 to -99.5 (assuming Torus level starts at +2 to -2). This is well below the Vortex limit for RHR (LPCI), but near the Vortex limit for Core Spray." The candidate may incorrectly determine that the correct action to take at this time is to trip (turn off) the sump pumps for this reason. However, these actions are only taken for a large torus leak, the symptoms of which have not been provided in the stem of the question. The correct action at this time is to trip and isolation the TWMS system as it may be a source of leakage that may prevent the need to enter and perform actions in 29.ESP.27.

O/V Week:

1)

Change stem to state leak is from a source other than the Torus.

Licensee:

1)

Changed stem to state that the leak is from a Fire Header running through the Torus Room.

NRC:

1)

Question is SAT.

58 F

2 N

S 59 H

3 N

B E

S 2015 Exam NRC:

1)

Direct overlap with Scenario 1, Event 5.

Scenario event include a loss of the in-service pressure regulator to fail low and the system auto swaps to the backup regulator, requiring lower the pressure setpoint on the backup regulator by 3.5 psig. Question is SAT, but not in terms of overlap - marked as Enhancement.

Recommend revising this question. 2017

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level Exam Q62 or 2015 Exam Q61 may work as replacements.

Licensee:

1)

Question has been replaced with Q61 from the 2015 Exam per the above recommendation.

NRC:

1)

Replacement bank question is SAT.

60 H

3 N

S 61 H

3 N

M E

NRC:

1)

Question appears modified from 2009 Exam Q65 - questions asks direct effect from pushing the Manual Iso PB, while this Q61 is asking both direct and indirect impacts.

2)

Recommend highlighting / emphasizing the 5 minute timeframe.

3)

Recommend rephrasing what would you expect to see to something such as which of the following indications represents the appropriate Secondary Containment Isolation Damper status?

Licensee:

1)

Changed question source to Modified.

2)

Emphasized the 5 minute timeframe using ALL CAPS to be consistent with other emphasis used throughout this exam.

3)

Rephrased question per CE recommendation above.

NRC:

1)

Question is SAT 62 F

2 B

S Grand Gulf 2010 Exam 63 H

3 N

E NRC:

1)

Recommend adding procedures to distractors (start Div 2 chiller per 23.413?).

D being the most detailed answer appeared to be the most logical choice, just due to the details in the answer, especially in comparison to B.

2)

Some reference should be made to the HVAC high temp trip setpoint in the answer explanation. May want to add an initial condition regarding temperature in the stem.

Licensee:

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

1)

Added procedure references to the distractors for consistency.

2)

The actual temperatures are superfluous and not readily available to the operating crew. The crux of the question is that the ARP allows for an attempt to reset the chiller after it trips on low oil pressure, preventing the need to swap divisions of CCHVAC. Providing temperature information would unnecessarily clutter the question and could confuse the candidates.

We are reluctant to add this information to the question since it was not included during validation and this concern was not brought up during validation.

NRC:

1)

Question is SAT.

64 F

2 N

S 65 H

F 2

N E

S NRC:

1)

MOP01 3.11.2.3.a.1.b states that, regarding peer checks, the CRS must be present in the At Controls Area for all back panel peer checks. It seems this is implying the P603 can provide those (despite step 3.11.2.3.a.1 contradicting that) however, step 3.12.1.2 allows the P603 to leave the at the controls area, provided the CRS is aware.

2)

Question appears to be low-cog Licensee:

1)

This did not come up during validation and the Operators had no problem with the question as it is consistent with how activities are conducted in the MCR. Step 3.12.1.2 is referring to the reactor controls area (aka front horseshoe) and not the At Controls area (area inside the Control Room panel access barrier ropes (See Enclosure A, Control Room Layout).

Additionally, per MGA24 Peer-checking (PC) is a series of actions by two individuals working together at the same time and place, before and during a specific action, to prevent an error by the performer which would prevent conducting the peer check from the transition from the At Controls to the Back Panel area.

2)

Changed LOK to Fund.

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level NRC:

1)

Agree with licensee comment #1. Question is SAT.

66 F

2 M

E NRC:

1)

Distractor A explanation states steam dome pressure is addressed in LCO 3.4.4; this is the RCS operational leakage LCO, which does not discuss steam dome pressure. Appears to be a copy / paste error from the bank question. Reference should be LCO 3.4.11, Reactor Steam Dome Pressure.

Licensee:

1)

Changed TS in Distractor A explanation from 3.4.4 to 3.4.11.

NRC:

1)

Question is SAT.

67 F

3 B

S NMP 2020 Exam 68 F

3 X

B E

S 2019 Retake Exam NRC:

1)

Applicant may contest theres not enough information to answer the question, as the details of the RWP are not known, and 27.000.05 states that normal operator rounds metho for inspection high radiation. Is there a way to ask the question, IAW 27.000.05, how will you direct the NO to perform the inspection?

That makes it more concrete.

Licensee:

1)

This is stated in the second sentence, which states You are preparing to brief an operator who is assigned to inspect equipment located in a HIGH RADIATION AREA, while performing rounds, per 27.000.05, Operator Rounds, which should sufficiently alleviate this concern.

During validation no Operators expressed a similar concern, and this question is Bank from the 2019 Retake Exam with no issues noted during review of that exams comments.

NRC:

1)

Agree with licensee comment #1. Question is SAT.

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 69 H

2 B

S 2017 Exam 70 F

3 N

S 71 F

2 N

S 72 H

2 B

S 73 H

3 B

S 74 F

2 B

S 75 F

2 B

S 76 H

F H

3 3

N B

S S

2020 Exam (previous 2 exams)

Free Look NRC:

1)

Is it necessary to provide bullets stating RCS heat input > heat losses (ie RCS heating up)? Is this to allow for Mode 4 transition plausibility (and part 1 distractor) if the applicant misunderstood the Mode definitions?

Licensee:

1)

A new K/A was requested and approved for this question due to poor validation and an extensive amount of time spent trying to write a valid question. A Bank Question was added.

NRC:

1)

Replacement question is SAT, however, the question is high cog, not log cog.

Licensee:

1)

LOK level changed to High.

NRC:

1)

Question is SAT.

77 H

3 N

B S

2020 Exam (previous 2 exams)

NRC:

1)

Question is the same as 2020 Exam Q89.

Bank question, not new

2)

Two 10CFR 55.41 references listed.

Recommend b(2)

Licensee:

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

1)

In our bank this question had a note that it was removed from the 2019 Retake Exam.

We failed to recognize that it was then used on the 2020 exam. Sorry for missing this. It has been changed to Bank. This is the 2nd Previous 2 question on the SRO portion of this exam, which is the maximum allowed by NUREG-1021 Form 2.3-4. We will work to ensure no more questions are added to this exam from the previous 2 NRC exams.

2)

Removed second 10CFR 55.41 reference.

NRC:

1)

Question is SAT.

78 F

3 N

S 79 H

3 N

E References provided NRC:

1)

Referenced provided 29.100.01 Sh 6, note 4 provides the answer for part 2 of Q13

2)

Reference provided 29.100.01 Sh 6, note 11 provides the answer to Q24

3)

Recommend providing only the HCL curve, or at minimum, redact the notes of Sh 6.

Curve sheet provided on the 2013 exam did not include the notes - only the curves.

Licensee:

1)

See below.

2)

See below.

3)

A new version of Sheet 6 has been created with the Notes section redacted. This has been placed in the References folder that will be printed for this exam. Also, Questions 79 and 86 have been updated to reflect that Sheet 6, without the Notes Section, will be provided with the exam.

NRC:

1)

Question is SAT and overlap concern alleviated.

80 H

3 X

N U

NRC:

1)

All actions 1-4 should have been completed prior to 125 deg TWT. > 110, Action 1 is completed. With action 1 completed, action 3 is completed. With TWT > 120, action 4 is completed. With the stem phrasing if TWT is 125 deg and the SRV is still stuck open, which of the above

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level sets of actions are applicable? The applicant could argue none, since those actions should have already been completed.

2)

While a unit shutdown is not valid for the given conditions, within the bounds of 20.000.25, the question stem does not ask that. Validate that a unit shutdown is not procedurally valid - does any ARP action require a SD due to the condition?

Licensee:

1)

Added clarification to the stem to ensure the examinee understands that 20.000.25 has been entered. Clarified question statement to address the NRC CEs concern.

2)

The clarification provided to address part (1) above should also address this concern. Since Action 5, Commence plant shutdown IAW 22.000.03 / 22.000.04 is in only one distractor (B) and it accompanies Action 2, place RHR in Torus Cooling only; if only these 2 actions were directed by the CRS, then both the AOP and Tech Specs would be violated by the CRS, showing lack of competence on the examinees part and reason for missing this question.

NRC:

1)

Question is SAT.

81 H

3 N

S 82 H

3 B

E 2019 Retake Exam NRC:

1)

Add reference regarding what meets SDM (MSBWP) to the answer explanation (and to distractor B).

Licensee:

1)

Added reference regarding MSBWP to the answer explanation and to Distractor B.

NRC: Question is Sat.

83 H

3 B

S 2019 Exam 84 H

3 B

S References provided 2019 Exam

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 85 H

F 3

3 X

N N

U NRC:

1)

Appears to be a K/A mismatch - question does not address the knowledge of the organization of the operating procedures network for normal, abnormal and emergency evolutions.

Licensee:

1)

A search of ADAMS was conducted for K/A G.2.4.5. A new version of the question has been written that will hopefully be a better K/A match. Q89 on the Seabrook 2020 exam was used as an idea to write the new question, although the Fermi question is significantly different enough to be called New vice Modified. The new question, if accepted, would be LOK = Fund, which is needed because right now the SRO only portion of the exam is at 16 High Cog questions. NUREG-1021 ES-4.2, Page 2 allows for approaching 60% (i.e. 15 Questions) at the High Cog Level for the SRO exam.

O/V Week:

1)

Emphasize NON-primary system for C and C.

Licensee:

1)

Added emphasis to NON-primary system for C and D.

NRC:

1)

Replacement question is SAT.

86 H

3 M

N E

Free Look References provided NRC:

1)

Torus pressure does not provide a trend, whereas all other parameters do.

2)

Distractor explanation lettering is incorrect.

D is the correct answer, but distractors show B, C, D (should be A, B, C). Correct distractor lettering.

3)

Distractor B seems to be easily eliminated as part 2 states to rapidly lower RPV pressure, while A and C are to open 5 SRVs Choice is less descriptive to accomplish the same function and is easily eliminated Licensee:

1)

Added and steady to Torus Pressure.

2)

Corrected distractor lettering. Apologies for these errors. We are trying to figure out

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level why the report does this sometimes and not others.

3)

The Part (2) response for distractor B was changed to match the other distractor Part (2)s, to be consistent and prevent easily eliminating this option.

NRC:

1)

Notes on provided reference29.100.01 Sh 6 should be redacted, as discussed under Q79.

2)

With ALL distractors (A,B,C) for part 2 being ED via SRVs, the applicant can determine the correct answer by correctly eliminating distractor A of Q79 - which requires use of the HCL curve. Overlap concern. Question is not unsat, but needs modified to eliminate the overlap.

3)

Question is modified such that the question can be considered new.

Licensee:

1)

A new version of Sheet 6 has been created with the Notes section redacted. This has been placed in the References folder that will be printed for this exam. Also, Questions 79 and 86 have been updated to reflect that Sheet 6, without the Notes Section, will be provided with the exam

2)

It appears the overlap concern only exists for Distractor A, because it is the distractor that is associated with the HCL curve (if I am understanding the NRCs comment).

Therefore, a new Distractor A has been written, to eliminate this overlap with Q79.

The new distractor is associated with the basis and actions for approaching the PCPL.

3)

Changed from Modified to New.

NRC:

1)

Question is SAT and overlap concern alleviated..

87 H

H 4

3 X

N N

U NRC:

1)

Question only hits the (a) part of the K/A and does not hit the (b) part on how to use procedures to correct, control or mitigate.

ES-4.2.C.2. Question can be answered with only TS knowledge - no knowledge of procedures used to correct, control, or mitigate the transient are necessary to correctly answer the question.

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

2)

Recommend highlight the difference between ACTIVE and TRACKING in C &

D.

Licensee:

1)

A new question has been written for this K/A.

NRC:

1)

Replacement question is SAT.

88 H

3 N

M S

NRC:

1)

Question is modified from 2012 Exam Q69.

Licensee:

1)

Changed question source to Modified.

89 F

2 N

S 90 H

3 X

N M

E NRC:

1)

How is distractor D plausible? Why would the applicant determine IAS is lost when the stem states that IAS header pressure is 100 psig and stable?

2)

Question is modified from 2018 Exam Q78.

Licensee:

1)

New distractor D written to address plausibility.

2)

Changed question source to Modified.

NRC:

1)

Question is SAT.

91 F

2 N

E S

Free Look NRC:

1)

Recommend adding ONLY to the beginning of C and D, such that potential subset issues can be eliminated.

2)

Plurality issue in stem - first sentence states 1 fuse blew, but then operators replaced fuses. Is more than 1 fuse replaced if only a single fuse blows?

Licensee:

1)

ONLY was added to the beginning of Distractors C and D.

2)

Corrected issue in stem by changing fuses to fuse throughout (it was stated several times).

NRC:

1)

Question is SAT.

92 F

H 3

N S

NRC:

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level

1)

Consider rephrasing Distractor A to include Operable

2)

Question is high cog.

Licensee:

1)

Changed distractor A as suggested.

2)

Changed LOK to high.

NRC:

1)

Question is SAT.

93 F

H 3

B S

NRC:

1)

Question is high cog, not low cog.

Licensee:

1)

Changed LOK to High.

94 H

3 X

X N

U References provided NRC:

1)

The answer is correct for the most limiting TRM action required, however, 3.12.8 is also required to be entered. The question stem does not ask for the most limiting actions, only what is applicable. Consider rephrasing stem to ask Which of the following is the most limiting TRM required action?

2)

Is 3.12.5 RA A.1 (continuous fire watch w/suppression equipment staged) more limiting than 3.12.8 RA A.1 (continuous fire watch)? Applicant could argue no correct answer, as both are correct and both equally limiting in terms of time required.

Additionally, distractor A and B both have ONLY, while C and D do not. Answer D is partially true.

Licensee:

1)

A new question version has been written to address both concerns.

2)

A new question version has been written to address both concerns.

O/V Week:

1)

Spell out Control Room Supervisor in the stem.

2)

Change wording to the question per recommendations.

Licensee:

1)

CRS spelled out.

2)

Changed question wording.

NRC:

1)

New revision of question is SAT.

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 95 F

2 N

S 96 F

2 B

S Columbia 2019 Exam 97 F

H 3

N E

NRC:

1)

Swap A & B and swap C & D to allows part 2 being in chronological order. This would make the answer B instead of A.

2)

No learning objective listed.

3)

Question is high cog Licensee:

1)

Swapped answer and distractors as directed. The correct response is now B.

2)

Objective link established.

3)

Changed LOK to high cog.

NRC:

1)

Question is Sat.

98 F

3 X

N E

Free Look NRC:

1)

Recommending rephrasing stem to replaced approaching. This is a subjective term and an applicant could argue that rad levels rising up through 1R/hr (able to be observed on the ARMs, Ch 7, 8, 9, 10, 11, 31) can be considered approaching the MSO.

2)

Distractor explanations for C and D state the MSO value is 5MR/hr. Should be 5R/hr.

Licensee:

1)

Changed wording to ask how they would determine the limit was exceeded.

2)

Corrected units in Distractors C and D.

NRC:

1)

Question is SAT.

99 F

2 N

S 100 H

2 M

E S

References provided NRC:

1)

Is the manhole damaged considered a safety SSC per SA8.1? If so, the Alert may be a concurrent EAL, but not the declared EAL as the SAE is higher. If it is a safety SSC, recommend adding that discussion to the answer explanation. Additionally, consider adding to the stem that you are the SM and rephrasing the stem to ask

Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level Which of the following is the highest EAL threshold met? or something similar.

Licensee:

1)

There are no safety significant components (as defined by EP-101) adjacent to the Aux Boiler House. If the examinee makes this connection, without such information stated in the stem of the question, this would be in conflict with the guidance given during the pre-exam briefing and could lead to an incorrect classification lending to the plausibility of some of the other distractors.

None of the candidates taking the exam are currently SM qualified. SROs may be called upon to make a classification in the absence of the SM or may be asked to back up the SMs EAL classification. For this reason, it is unnecessary (and unrealistic) to state that the examinee is the SM in the stem.

Since only one EAL classification is currently met, changing the phrasing of the question may add unnecessary ambiguity to the question. The concerns presented by the NRC were not voiced during validation so the Fermi exam team is reluctant to make these changes.

NRC:

1)

Discussed with licensee that applicants, while not SM qualified, are still responsible for making EAL calls and can be tested to the highest license level, including that as a SM (reporatibility, EALs, etc). Question is SAT

Form 2.3-5 Instructions for Written Examination Review Worksheet Refer to ES-4.2 for the definitions of terms used in this worksheet for the written examination.

Review each question (Q) as submitted and as subsequently revised and document the following in the associated worksheet columns:

1. Enter the level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) from 1 (easy) to 5 (difficult); mark direct lookup questions (applicant can directly determine the answer from the provided reference) as LOD 1. A question is (U)nsatisfactory if it is LOD 1 or LOD 5.

3. Check the appropriate box if a psychometric flaw is identified:

Stem Focus: The stem lacks enough focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information). This is an (U)nsatisfactory question.

Cues: The stem or one or more answer choices contains cues (e.g., clues, specific determiners, phrasing, length). This is an (U)nsatisfactory question.

T/F: All of the answer choices are a collection of unrelated true/false statements. This is an (U)nsatisfactory question.

Cred. Dist.: The distractors are not credible; single implausible distractors require (E)nhancement, and more than one noncredible distractor in the same question results in an (U)satisfactory question.

Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem). This is an (U)nsatisfactory question.

4. Check the appropriate box if a job content flaw is identified:

Job Link: The question is not linked to the job requirements (i.e., the question has a valid knowledge or ability (K/A) but, as written, is not operational in content). This is an (U)nsatisfactory question.

Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory). This is an (U)nsatisfactory question.

1. /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). This is an (U)nsatisfactory question.

Logic: The question requires backward or reverse logic or application compared to the job requirements. This is an (U)nsatisfactory question.

5. Check the first box if a K/A mismatch flaw exists. Check the second box if the question is flawed because it is written at the wrong license level. Either condition results in an (U)nsatisfactory question.

6. Enter the questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-4.2.

7. Based on the review performed in steps 2-5, mark the question as (U)nsatisfactory, in need of (E)nhancement, or (S)atisfactory.

8. Fully explain the reason for any (U) in column 7 (e.g., how the psychometric attributes are not being met).

9. Save the initial review comments and detail subsequent comment resolution so that each exam-bound question is marked by an (S) on this form.