ML20241A244

4-CN-2020-04 Written Exam Comments

ML20241A244
Person / Time
Site:	Cooper
Issue date:	08/04/2020
From:	Greg Werner Operations Branch IV
To:	Nebraska Public Power District (NPPD)
References
Download: ML20241A244 (16)

Text

ES-401 CN-2020-04 Written Examination Review Worksheet Form ES-401-9

1. 2. 4. Job Content 6. 7. 8.

3. Psychometric Flaws 5. Other Q# LOK LOD Flaws (F/H) (1-5) B/M/ U/E/S Explanation N

Stem Focus Cred. Dist. Job-Link Backward SRO-Only Cues T/F Partial Minutia #/units Q=K/A If the setpoint above which PCV-609 can be opened is 77 psig, then make the correct answer B = 77 psig. Otherwise you are at risk of an applicant arguing there is no correct answer. Change stem to Which of the following is the Instrument Air header pressure setpoint above which E Distractor A explanation describes an alarm at 60 psig but the distractor is 75 psig, 1 H 3 X X N S which appears not to have a relation to the explanation.

Distractor C explanations describes an alarm at 80 psig but the distractor is 83 psig.

Make the same.

Distractor D: Discussion is about 85 psig, distractor is 87 psig. Make same.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3. Check the appropriate box if a psychometric flaw is identified:

• Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• Cues: The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).

• T/F: The answer choices are a collection of unrelated true/false statements.

• Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.

• Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

4. Check the appropriate box if a job content error is identified:

• Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

• Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).

• #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• Backward: The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only. (K/A and license-level mismatches are unacceptable.)

6. Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.

7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).

Attachment 8_Rev 3 Page 1 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 2 Form ES-401-9 kg - For Q#1, changed stem wording as specified and changed Answer A to 60 psig, C to 80 psig, and D to 85 psig and updated answers justifications accordingly per CE comments.

[NRC OK]

Add = to given stem parameters.

2 H 3 N S kg - added = to stem parameters in first two bullets per CE comment.

[NRC OK]

3 H 3 N S [NRC pre-review - OK]

Dist B: Modify to Initially rises higher and stabilizes greater than 102%

E Dist D: Modify to Initially drops lower and stabilizes less than 98%.

4 H 3 X N S kg - Modified distractors originally designated B and D per CE comments.

[NRC OK]

5 F 3 N S 6 H 3 N S 7 F 3 N S 8 F 3 B S NRC 2015 44 NRC 2018 53 9 F 3 N S This counts as a new question, not modified.

kg - Designated as NEW on worksheet per CE comment.

10 H 3 B S Distractor C Explanation: SRM C is at 6E4 cps, not 5E4 cps.

kg - Changed 5E4 to 6E4 in correct answer explanation and in distractor C justification 11 H 3 N S per CE comment.

[NRC OK]

12 H 3 N S 13 H 3 N S 14 H 3 N S Same as previous comments, if the reset setpoint is 54 inches, make the correct answer 54 inches, not an arbitrary value above or below it. Make the distractor 59 inches to track with the explanation.

E 15 H 3 X X X N S Modify stem part 2 to Which of the following Narrow Range reactor water levels is the highest value below which depressing the displayed REST button(s) will cause BC comments: The picture looks like a representation vs actual picture. Recommend replacing with an actual picture of control board. Also I think Lesson Number:

Attachment 8_Rev 3 Page 2 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 3 Form ES-401-9 COR002032-02 (pg 73 of 85), Line (h), Initiating Device/Setpoint is incorrect. It should be >+54.0. The sign is turned the wrong way, because you only want the turbine and RFP to trip on water level >54.

kg - Changed Part 2 stem to Which of the following Narrow Range reactor water levels is the highest value below which depressing the displayed RESET button(s) will cause the tripped logic channel(s) to reset? and changed Part 2 correct answer to 54 and Part 2 distractor to 59 per CE comments.

Regarding use of the subject representation from simulator Thunderview mimic, images from simulator control panel mimics are often used in written questions at CNS to ensure consistent clarity. Sometimes brightness of actual panel photos varies due to simulator lighting or camera flash. This image was specifically reviewed for clarity when printed on white paper, as it will be during exam administration. Validators and reviewers agreed the status of the lights depicted on the image was clearly evident.

This question was validated by 14 validators and 3 Operations representatives, and none misinterpreted the intent of the image. This image is being retained in this question.

Regarding the setpoint 54 in the lesson plan, it is the TS allowable value. The sign is correct. CNS lists TS allowable values in lesson plans.

[NRC OK]

If 72 psig is the recognizable value for a distractor, then make the distractor 72 psig vice 70 psig.

16 F 3 X X X N E kg - Changed Part 1 distractor to 72 psig per CE comment.

[NRC OK]

17 H 3 B S Distractor B is not plausible that the DG WOULD NOT have to be tripped for the surveillance, but WOULD have to be tripped for an automatic start.

kg - rewrote question, replacing premise of part 2 based on CE comment that original form made distractor B implausible.

[NRC] Is B plausible that the DG would be tripped but the output breaker would remain closed, i.e. motoring the DG?

18 H 3 X N E kg - Based on CE comment that distractor B was implausible, rearranged Part 1 and 2 stems so part 1 tests knowledge of response of DG output breaker to receipt of a LOCA signal and part 2 tests knowledge of response of DG1 engine to high vibration signal with an automatic start signal present.

[NRC - OK]

19 H 3 X N E Add ONLY to C and D part 1 Attachment 8_Rev 3 Page 3 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 4 Form ES-401-9 S Added ONLY to answers B and D part 1 per CE comments.

[NRC OK]

20 H 3 N S Arrange the question as 1) and 2) for clarity.

E 21 H 3 X N S Arranged question into Parts 1 and 2 per CE comments.

[NRC OK]

22 H 2 M S NRC 2018 50 Confirm that a bus voltage drop from 265 VDC to 249 VDC over 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> is systematically reasonable for this condition.

kg - Based on CE comment, since could not located sufficient empirical data to E support stem condition of bus voltage lowering from 256 VDC to 249 VDC over a one 23 H 3 X N S hour period, reworded stem to only state Indicated 250 VDC Bus 1A voltage has lowered several volts, since charger supply voltage is higher than battery terminal voltage due to the charger supplying bus loads.

[NRC OK]

Change LOD = 2.

24 F 2 N S kg - Changed LOD to 2 per CE comment. [NRC OK]

Change to Memory/Fundamental cognitive level.

25 F 3 N S kg - Changed to Memory/Fundamental cognitive level per CE comment. [NRC OK]

Modify part (1) to, As reactor pressure continues to lower, which one of the following..

kg - Modified part (1) to, As reactor pressure continues to lower, which one of the 26 H 3 N S following.. per CE comment.

[NRC OK]

27 H 3 N S Change LOD = 2 28 H 2 N S kg - Changed LOD to 2 per CE comment. [NRC OK]

29 H 3 N S NRC 2018 65 Remove bolding from stem words.

Remove only from C and D since there is no option for all 3 valves to be in required E

30 H 2 X M S position.

kg - Removed bolding from in and required in stem and removed ONLY from answers C and D per CE comments.

[NRC OK]

Attachment 8_Rev 3 Page 4 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 5 Form ES-401-9 NRC 2015 62 Provide explanation of what the modification was.

31 H 3 M S Recent plant mod.

kg - Added description of design change CED 6024460 to correct answer explanation.

[NRC OK]

Modify stem to during a LOCA, complete the following:

Concerned that D part (2) could be argued correct, since scramming the reactor does limit the heat input to the RCS and thereby the energy that must be absorbed by primary ctmt.

kg - Modified stem to Regarding the reasons for a manual scram due to high drywell pressure during a LOCA, complete the following: per CE comment.

Regarding concern that D part 2 may be argued as correct, the stem asks for the reason stated in procedure 2.4PC for the scram action. The scram action is a manual action employed before the automatic initiation setpoint is reached. The distractor 32 F 3 X N E states the action is to avoid initiation of safety systems at the setpoint. It is assumed Drywell pressure will reach the initiation setpoint if it cannot be controlled below 1.5 psig by other mitigative strategies, such as by venting or by reducing power to reduce the heat input from Reactor Recirc pumps. Scramming would not avoid reaching the initiation setpoint (1.84 psig) of safety systems as stated in the Part 2 distractor.

Therefore, replaced part 2 distractor with To avoid automatic isolation of the Primary Containment vent pathway. The PC vent pathway employs valves PC-MO-231 and PC-AO-246. These valves isolate at 1.84 psig Drywell pressure. Scramming would not avoid isolation of the PC vent pathway.

[NRC OK]

Consider adding to answers, based on sensed reactor/equalizing header pressure 33 H 3 N S Added sensed to each answer based on CE comments.

[NRC OK]

34 F 2 B S NRC 2015 18 35 F 2 M S NRC 2017 28 NRC 2014 17 If Emergency In is an informal term, make the correct answer Emergency Rod In which is the formal term. Same with Continuous In vs Continuous Rod In.

E 36 F 3 X B S Change to Memory/Fundamental kg - In answers, changed Emergency In to Emergency Rod In and changed Continuous In to Continuous Rod In, and changed to memory/fundamental level per CE comments.

Attachment 8_Rev 3 Page 5 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 6 Form ES-401-9

[NRC OK]

Specify the title of EOP-5A in stem.

Operational validity would drive this question to be asked in the form of Which of the following procedures are required to be entered, if any?

After making the above structural change, part A would be more plausible if it referenced Alarm Response Procedures XX only.

kg - Added title to EOP-5A in stem and restructured stem and answers per CE E comments, except did not include the phrase if any, since it would be implausible that 37 H 3 X N S Alarm Cards would not be entered with an amber and a red alarm given in the stem.

(The original version of this question had been structured to make the question more concise for readability, it was structured so that procedures could be listed only once, in the stem, versus being repeatedly listed in the answers.)

[NRC] Remove only from the end of D.

kg - Removed ONLY from end of answer D per CE comments.

[NRC OK]

The explanation does not track with the conditions in the stem of the question, specifically, JP LOOP FLOW NBI-FI-92A is 5 Mlbm/hr not 3 Mlbm/hr, AND Total Core Flow is 32 Mlbm/hr, NOT below 24 Mlbm/hr. Unsure if this question is valid given inconsistency between justification and actual question. Is this a New question as listed or is it a Modified question?

kg - This is a new question. The first version drafted was to test knowledge with core U flow below 29.5 mlbm/hr, with <26 Mlbm/hr in the active Recirc loop. During validation, 38 H 3 X X N S the question was changed to test knowledge with core flow above 29.5 mlbm/hr. The following sentence was inadvertently left in the correct answer explanation from the old version; Since indicated total core flow is below 24 Mlbm/hr, the 3 Mlbm/hr indicated for the idle jet pump loop is actually forward flow produced by natural circulation driving head.

So, removed that sentence from the correct answer explanation.

[NRC OK]

NRC 2015 45 What is the necessity of specifying 2 minutes for a grid UV in the stem? Many of the justifications seem to rely on recognition of time requirements for 1st level and 2nd level E UV time delays, (5 sec, 7.5 sec), so it would appear to be make the distractors more 39 H 3 X X B S plausible if the elapsed time was closer to these values.

kg - Per CE comment to reduce elapsed time in stem, changed time to 30 seconds, which accommodates 12.5 seconds for UV relays to trip plus 14 seconds (max. time per design) for DG to reach rated speed and voltage.)

Attachment 8_Rev 3 Page 6 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 7 Form ES-401-9 Explanation discusses 5.5 seconds as the time delay for 2nd level UV, but the attached procedures describe 2nd level UV as a 7.5 sec +/- 0.8. Include the following excerpts in explanation:

1.1.1 There are two levels of undervoltage protection at CNS. The first level is a loss of voltage protection which is designed to actuate at conditions indicative of a grid voltage rapidly collapsing to zero volts (i.e., bus voltages < 2870V). Relays which actuate are a time undervoltage relay with inverse time characteristics (i.e., lower the voltage, the faster the actuation).

CNS Technical Specification setting for these relays is 2300V +/-

5% in < 5 seconds. This window sectionalizes an area of the inverse time relay curve which corresponds to the median of voltage levels tested during surveillance testing.

1.1.2 Second level of undervoltage protection is for sustained degraded (low) voltage conditions. This system is designed to respond to a static low voltage condition and will actuate whenever the bus voltage drops below 3880V +/- 52V for a time period of 7.5 +/- 0.8 seconds. This static setting is also the CNS Technical Specification setting.

kg - Included the above excerpts in the correct answer explanation per CE comment.

[NRC OK]

40 H 3 N S [NRC pre-review - OK]

41 F 3 N S Modify stem to some version of Which of the following procedures are REQUIRED to be entered for this condition? Concerned that the term EVERY could result in a no E correct answer argument.

42 F 3 X N S List the name of EOP-5A in stem and answers consistent with the other procedures.

kg - Changed stem wording to Which of the following procedures are REQUIRED to be entered for this condition? and added EOP-5A title to answers per CE comments.

Attachment 8_Rev 3 Page 7 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 8 Form ES-401-9

[NRC OK]

43 H 3 B S NRC 2018 57 If the lowest level allowed by TS is >= -2.0 inches, make that the correct answer, vice -

E 1.9. Modify D to -5.0 inches.

44 F 3 X N S Modified correct answer C to -2.0 and distractor D to -5.0 per CE comments.

[NRC OK]

45 F 2 N S 46 F 3 N S 47 F 2 N S 48 H 3 N S

[NRC pre-review - Part 1 was not OK, revised]

49 F 3 N S Modify to LOD=3 due to nuance of part 1 answer.

kg - Changed LOD to 3 per CE comment.

50 F 2 N S 51 H 3 N S Is there another 250 VDC component that is not affected by the failure of Bus 1B that can be used as a distractor vice Core Spray B?

E kg - The only other distractor choices are RCIC and LPCI Loop A, which may be more 52 H 3 X N S implausible than Core Spray B, since they are Div 1 powered. No change has been made as of 3-13-20.

[NRC OK as-is]

NRC 2015 25 Confirm that part 2 is required RO knowledge per CNS accredited ILT program.

kg - This is required RO knowledge. The related RO objective is from lesson plan INT008-06-17 [EOP Flowchart 5A Secondary Containment and Radioactivity Release 53 F 3 X M S Control], enabling objective EO-7: Given plant conditions and EOP flowchart 5A, SECONDARY CONTAINMENT CONTROL and RADIOACTIVITY RELEASE CONTROL, state the reasons for the actions contained in the steps.

[NRC OK]

54 H 3 N S [NRC pre-review - OK]

Include title of EOP-3A in stem.

E Modify stem to for high Suppression Pool temperature is the same as the TS 55 F 2 X N S 3.6.2.1 Modify distractor B to which requires immediate plant shutdown.

Attachment 8_Rev 3 Page 8 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 9 Form ES-401-9 kg - Added EOP-3A title to stem, changed stem wording to temperature is the same as the TS 3.6.2.1, and changed distractor B to which requires immediate plant shutdown. per CE comments.

[NRC OK]

56 F 2 N S Include title of EOP-1A in stem.

Stem part (2), Which one of the following valves receiveS 57 F 2 X N S kg - In stem, added title for EOP-1A and changed receive to receives per CE comments .

[NRC OK]

58 H 3 N S 59 F 2 N S 60 H 3 M S NRC 2017 52 61 F 3 N S E Recommend providing a power level in stem.

62 H 3 X N S kg - Added Reactor power is ~3% to stem per CE comment. [NRC OK]

63 F 3 N S 64 H 3 N S 65 H 3 M S NRC 2017 54 66 H 3 N S 67 F 2 B S NRC 2017 67 68 F 2 B S NRC 2015 75 NRC 2017 72 Modified questions typically need to have 1 stem condition and 1 distractor changed, or 1 stem condition changed such that it changes the correct answer. In this case no stem or distractors have changed, but due to a procedural change, there is a new correct answer. I believe that the procedural change meets the intent of a stem change (i.e. the background information has changed), and that this question meets 69 F 2 M S the intent/spirit of a modified question. Litmus test: If an applicant just reviews past 2 exams, and answers this question based on knowledge of the 2017 exam question, they will get it WRONG. It is in the NRCs and public interest that the facility test applicants on recent plant modifications, to include modifications to administrative practices kg - 8 of 14 validators missed this question due to obsolete knowledge.

Attachment 8_Rev 3 Page 9 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 10 Form ES-401-9 70 F 2 B S NRC 2018 74 71 F 2 B S NRC 2015 72 72 F 3 B S NRC 2017 70 73 F 2 N S 74 F 3 N S [NRC pre-review - OK]

75 F 2 B S NRC 2015 66 Im not enthusiastic about part (2) construction, as it isnt testing the expected knowledge in the most direct manner. Better to eliminate the 1925 line of the stem and for the question to ask, what is the LATEST time by which the NRC is required to be notified to satisfy reporting requirements? or some variation thereof.

This is not really testing on any knowledge of the Reactor Water Level Control System the stem is just telling the operator that something related to Reactor Water Level Control resulted in a required report to the NRC. As such, this question is written as a Tier III question. If the question gave the applicant a water level, or some aspect of RPV level control that required the applicant to determine the who/when/what/ of U

76 H 3 X X X N S reporting requirements, that would meet the K/A.

kg - I need help or another K/A for this!!!

[NRC 3-11] Replace with:

2.4.31 Knowledge of annunciator alarms, indications, or response procedures.

l (CFR: 41.10 / 45.3)

IMPORTANCE RO 4.2 SRO 4.1 kg - Replaced previous K/A G2.4.30 with G2.4.31 provided by CE and replaced question per CE comments. Updated ES-401-1 and ES-401-4.

[NRC OK]

77 H 3 N S NRC 2015 77 78 H 3 M S Explanation should summarize how the question was modified.

kg - Added explanation of how question was modified to the explanation. [NRC OK]

E Why is one page of the reference in question worksheet blacked out? Nothing is 79 H 3 X N S blacked out on the reference handout.

Attachment 8_Rev 3 Page 10 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 11 Form ES-401-9 kg - There is no intentional black out and the master word document is not blacked out. This must have occurred during uploading the document.

Are applicants intended to recall from memory the TS Allowable values for RBM upscale trip setpoints from the COLR?

Ask TJ if he will allow TS Table 3.3.2.1-1 as a provided reference Remove the ONLY from A and C part (2), as the qualifier FIRST in stem covers that.

Removed ONLY from part 2 of answers A and C per CE comment.

[NRC] Id be willing to give applicant the COLR page with Low/Int/High/Downscale Trip Setpoint limits and values redacted, but not inclined to give Table 3.3.2.1-1.

kg - Examinees are expected to know from memory RBM setpoints IAW licensed operator objectives 4a and 4b from lesson plan COR002-24-02. This question was validated without providing COLR as a reference. Therefore, COLR will not be provided.

[NRC OK]

[NRC pre-review modified]

-The way this question is structured is not really assessment based which would be expected for an SRO question. Reformat this question to include system parameters in the stem (DW pressure, RPV level, etc), and have the SRO assess conditions for procedure entry (which alone may not be SRO only if it is an AOP or major EOP), and select the appropriate mitigative strategy based on the plant conditions. Use the decision points you covered in your explanation of your answer and distracters as your guide.

Per CE comments, restructured question by providing plant conditions in stem, so the SRO must assess conditions and exercise detailed procedural knowledge in selecting the procedure containing mitigative steps and, also, assessing conditions required for 80 F 3 X X N E performance of a procedural Critical Step.

S [NRC] Given that Drywell pressure is 5.5 psig - well past the 1.84 psig setpoint that scrams the reactor, initiates core spray, SGT, isolations, etc., it is not credible that an applicant will not realize there are indications of a drywell leak. Provide indications that are not so extreme (pre-scram) that would still allow the applicant to diagnose need to fire squib valve. Change (is/is NOT) to (is/is NOT YET) for increased plausibility.

kg - The SRO knowledge being tested in Part 2 is not so much whether the examinee can discern whether a leak in the drywell exists, but whether the examinee knows specific detail of procedure content regarding exercising a conditional step. The examinee must know in order to direct firing the squib, a coolant leak must exist. An examinee may believe there are additional/other conditional requirements, such as indication of an offsite release above the Alert level, before squib actuation is Attachment 8_Rev 3 Page 11 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 12 Form ES-401-9 permitted, since that action is severe in that it destroys the TIP cable, is irreversible, and would be very complicated to repair. The test is not whether the examinee can determine a LOCA exists, but rather, given conditions that represent a LOCA exists, does the examinee know what to do with that information. It would be difficult to make the stem conditions below the 1.84 psig setpoint, since the TIP penetration would not be automatically isolated below the setpoint. Requiring the examinee to project in the future what should occur and trying to convey in the stem isolation does not occur might be confusing to the examinee.

[NRC] The questions justification for plausibility says The examinee who does [not]

realize there are indications of a reactor coolant leak in the drywell may choose this answer. It is not plausible given the conditions that an applicant wouldnt recognize a leak. If the intent is actually to test whether the applicant knows that a leak is a condition for firing a squib valve, then Id like something else in the stem to increase the plausibility that it is not yet required, such as a reactor water level still in the normal range (i.e. a small, slow leak), statement that no offsite release is currently in progress, etc.

kg - Added reactor water level is 30 inches, slowly rising to stem and enhanced part 2 distractor plausibility justification.

[NRC] Modify is/is NOT to is/is NOT YET and that will be sufficient.

kg - Changed is/is NOT to is/is NOT YET for part 2.

[NRC OK - Get rid of is/is NOT YET in the stem since it is in the answer choices]

NRC 2015 99 81 F 2 M S Explanation should summarize how the question was modified.

kg - Added explanation of how question was modified to the explanation.

This could be turned into an interesting Admin JPM.

82 H 3 N S kg - Acknowledged Question is credited to 10CFR55.43.10, which does not exist.

83 H 3 N S kg - replaced (10) with (5) [NRC OK]

Attachment 8_Rev 3 Page 12 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 13 Form ES-401-9 84 H 3 N S Modify part (2) to require applicant to choose between two different procedural transitions (with the correct answer still being EOP 6A). Better meets the SRO only attribute of assessment and selection of mitigative procedure.

85 H 3 X X N E Restructured Part 2 to require selection of a procedure with which to mitigate plant conditions per CE comment.

[NRC OK]

[NRC pre-review - REPLACED]

86 F 2 B S NRC 2015 79 Add ..occurred to end of 1st statement in stem.

I believe this is LOD = 4.

87 H 4 N S kg - Added occurred to end of first sentence in stem and changed LOD to 4 per CE comments. [NRC OK]

88 H 3 M S NRC 2018 82 89 H 3 N S [NRC pre-review - OK]

90 H 3 N S 91 H 3 N S [NRC pre-review - OK]

92 H 3 N S 93 H 3 N S For increased discriminatory value, replace one of the distractors by adding 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> to one of the times later than 0800, to assess whether applicant understands from which time the DET starts. Your choice which one.

kg - Per CE comment, replaced distractor B, formerly 1000, with 1600, to represent E

94 H 3 X N S the 6-hour DET being applied from the time of instrument channel trip and revised distractor justification.

[NRC OK - In this case I actually will ask you to rearrange answers from earliest to latest time.]

kg - Rearranged answers earliest to latest per CE comment. [NRC OK]

Modify part 1 to 1300 and 1400, to exactly match the time limit, per previous E comments.

95 F 3 X X N S kg - Modified part 1 to 1300 and 1400, to exactly match the time limit per CE comment. [NRC OK]

96 F 3 N S [NRC pre-review - REPLACED]

97 H 2 B S NRC 2018 98 Attachment 8_Rev 3 Page 13 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 14 Form ES-401-9 98 H 3 B S NRC 2017 94 99 F 3 N S 100 F 3 B S NRC 2015 95 Attachment 8_Rev 3 Page 14 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 15 Form ES-401-9 GENERIC COMMENTS Good explanations / justifications for answers and distractors. Made review easier. Kg - Acknowledged For handout sheets, label each separate handout with the question it is intended for. Ensure that ROs are not being given handouts for SRO exam. Kg - Acknowledged. Specific exam packets are assembled for each individual with either a RO or an SRO exam coversheet with their name on it. Only the appropriate references, either RO or SRO, are included with the examinees exam. The packets with pencils, markers, calculators, steam tables, etc. are placed in individual flap folders. We will label the exam references with the associated question number.

Review RO exam to see if any of the questions as-constructed can be credited towards some of the unused 10CFR55.41 categories, specifically 41.1, 41.2, 41.3, 41.9, 41.13, 41.14.

kg -

Credited the following questions to 10CFR55.41(1): Q#4, 51 Credited the following questions to 10CFR55.41(2): Q#38, 54, 66 Credited the following questions to 10CFR55.41(3): Q#38, 58, 61 Credited the following questions to 10CFR55.41(9): Q#44, 49, 55 Credited the following question to 10CFR55.41(14): Q#25 There are _2_ RO questions with handouts provided and __5___ SRO questions with handouts provided. _Rev 3 Page 15 of 16 OBDI 202 - IOLE Process

ES-401 CN-2020-04 16 Form ES-401-9 Results Table RO LOK -H 40 0.53 Avg RO LOD 2.73 Flaws 10 CFR Distribution RO LOK-F 35 0.47 AVG SRO LOD 2.92 Stem focus 18 41.1 2 43.1 2 SRO LOK - H 18 0.72 Overall LOD 2.78 Cues 0 41.2 3 43.2 7 SRO LOK - F 7 0.28 T/F 1 41.3 2 43.3 0

%  % Cred Dist 10 41.4 12 43.4 1 RO Bank 13 17.3 SRO Bank 4 16 Partial 7 41.5 6 43.5 14 RO Mod 8 10.7 SRO Mod 3 12 job link 0 41.6 7 43.6 0 RO New 53 70.7 SRO New 18 72 units 0 41.7 22 43.7 1

% minutia 0 41.8 2 Total Bank 17 17 backward 0 41.9 3 Total Mod 11 11 KA 1 41.10 13 Total New 71 71 SRO-only 2 41.11 1

%  % LOD = 1 0 41.12 1 RO Sat 56 74.7 SRO Sat 18 72 41.13 0 RO Unsat 1 1.33 SRO Unsat 1 4 41.14 1 RO Edit 18 24 SRO Edit 5 20 Answer Dist (%)

%  % RO-A 19 SRO-A 7 Total Sat 74 74 Total Unsat 2 2 RO-B 16 SRO-B 7 Total Edit 23 23 RO-C 22 SRO-C 6 RO-D 18 SRO-D 5 Note: Sat/Unsat percentages reflect original draft submittal. All questions administered were Sat. _Rev 3 Page 16 of 16 OBDI 202 - IOLE Process