ML23307A112
| ML23307A112 | |
| Person / Time | |
|---|---|
| Site: | McGuire, Mcguire  |
| Issue date: | 11/03/2023 |
| From: | NRC/RGN-II |
| To: | Duke Energy Carolinas |
| References | |
| 50-369/23-301, 50-370/23-301 50-369/OL-23, 50-370/OL-23 | |
| Download: ML23307A112 (1) | |
Text
ES-3.2 Administrative Topics Outline Form ES-3.2-1 (Rev_032823)
NUREG-1021, Revision 12 Facility:
McGuire Date of Examination:
07/2023 Examination Level:
RO Operating Test Number:
N23-1 Administrative Topic (see Note)
Type Code*
Describe activity to be performed Conduct of Operations M, R 2.1.4 (3.3)
Knowledge of individual licensed operator responsibilities related to shift staffing, such as medical requirements, no-solo operation, maintenance of active license status, 10CFR55, etc.
JPM:
Determine License Status Conduct of Operations M, R 2.1.7 (4.4)
Ability to evaluate plant performance and make operational judgments based on operating characteristics, reactor behavior, and instrument interpretation.
JPM:
Calculate QPTR with an Inoperable Power Range Instrument Equipment Control D, R 2.2.41 (3.5)
Ability to obtain and interpret station electrical and mechanical drawings.
JPM:
Determine Leak Isolation Boundaries Radiation Control D, R 2.3.12 (3.2)
Knowledge of radiological safety principles and procedures pertaining to license operator duties.
JPM:
Predict Radiation Levels While Responding to a Damaged Spent Fuel Pool NOTE:
All items (5 total) are required for SROs. RO applicants require only four items unless they are retaking only the administrative topics (which would require all five items).
- Type Codes & Criteria:
(C)ontrol room, (0) (S)imulator, (0) or Class(R)oom (4)
(D)irect from bank ( 3 for ROs; 4 for SROs & RO retakes) (2)
(N)ew or (M)odified from bank (> 1) (2)
(P)revious 2 exams ( 1; randomly selected) (0)
ES-3.2 Administrative Topics Outline Form ES-3.2-1 (Rev_032823)
NUREG-1021, Revision 12 RO Admin JPM Summary A1a This is a Modified Bank JPM. The operator will be told that they are a Licensed Reactor Operator who is currently assigned to the day shift staff for a special project, and not routinely standing watch in the Control Room, that the assignment has continued since 8/1/22, that during this time their license became inactive, and was reactivated on 12/1/22, and that their supervisor has directed them to stand watch as needed to maintain your license active. They will be provided with a work history for the last three months and directed to complete any paperwork necessary to track the maintenance of their license, and to identify the MINIMUM requirements that must be met going forward to maintain their license active (Including number of watches, required watch positions, and any time restraints). The operator will record the completion of the watches stood as the U2 OATC on 2/3/23 and U1 BOP on 2/9/23 and 2/10/23 on Attachment 1 of AD-OP-ALL-0107 in accordance with the attached KEY, and identify that 2 additional 12-hour shifts must be stood as either the U1 RO, U1 BOP, U2 RO or U2 BOP by 3/31/23 to maintain the license active.
A1b This is a Modified Bank JPM. With the plant at 74% power, the operator will be told that the Unit 1 OAC failed and is not operating, and that PR-41 has been removed from service.
The operator will be directed to calculate QPTR in accordance with PT/1/A/4600/21A (Loss of Operator Aid Computer while in Mode 1) and to identify all additional Technical Specification LCOs, if any, that are NOT met. The operator will be expected to calculate the QPTR (Per Attached Key) and determine that Technical Specification LCO 3.2.4 is not met for two quadrants.
A2 This is a Bank JPM. The operator will be told that Unit 1 is operating at 100% power, that the crew suspects a leak in the Aux Building and has entered Case II of AP/1/A/5500/10, NC System Leakage Within the Capacity of Both NV Pumps, and that an AO has just reported that there is a large packing leak on 1NV-151A (NV Pumps Recirculation Valve).
The operator will be directed to identify the closest leak isolation boundary valves for this leak, identify which, if any, of these valves need to be re-positioned from their current position, and to identify the Breaker for any electrically operated leak isolation boundary valve that may need to be operated. The operator will be expected to review the Flow Diagram of Chemical and Volume Control System (NV) and determine the closest leak isolation boundary valves for this leak, review OP/1/A/6200/001E and determine the boundary valves that need to be re-positioned, and review OP/1/A/6200/001E and identify the Breaker for 1NV-150A in accordance with the Attached KEY.
A3 This is a Bank JPM. The operator will be given a set of conditions reflecting a damaged and leaking Spent Fuel Pool with a full core off-loaded, where attempts of makeup have failed, but are expected to be successful within three hours. The operator will also be told that the Control Room and local Spent Fuel Pool Level and Temperature instrumentation are unavailable, that they have been assigned to locally monitor the Aztec Level Gauge in accordance with Generic Enclosure 32 (Monitoring Unit 1 SFP Level and Temperature),
and the OSC Coordinator has directed you to report when the Spent Fuel Pool Level is at the SLC Limit (-33 3/4 inches), and provided with Aztec Level Gauge readings showing current level and the rate that the Spent Fuel Pool level is lowering. The operator will be directed to (1) Predict when the Spent Fuel Pool level will lower below the SLC Limit, (2)
Refer to Enclosure 13 (Spent Fuel Pool Radiation Level vs. Water level Above Fuel) of AP/1/A/5500/41 (Loss of Spent Fuel Pool Cooling or Level), and determine the expected
ES-3.2 Administrative Topics Outline Form ES-3.2-1 (Rev_032823)
NUREG-1021, Revision 12 radiation levels three hours from now, based on the last known leak rate, and (3) Assuming that the ability to provide makeup water to the Spent Fuel is NOT restored, predict when the fuel will start to uncover. The operator will be expected to predict that the SLC Limit will be reached on the Spent Fuel Pool within 10+/-1 minutes, determine that the expected radiation level three hours from now is 117 Rem/hour, and predict that if water is not added to the Spent Fuel Pool the fuel will start to uncover in four hours.
ES-3.2 Administrative Topics Outline Form ES-3.2-1 (Rev_041523)
NUREG-1021, Revision 12 Facility:
McGuire Date of Examination:
07/2023 Examination Level:
SRO Operating Test Number:
N23-1 Administrative Topic (see Note)
Type Code*
Describe activity to be performed Conduct of Operations M, R 2.1.25 (4.2)
Ability to interpret reference materials, such as graphs, curves, tables, etc.
JPM:
Perform a Thermal Margin Determination Conduct of Operations M, R 2.1.4 (3.8)
Knowledge of individual licensed operator responsibilities related to shift staffing, such as medical requirements, no-solo operation, maintenance of active license status, 10CFR55, etc.
JPM:
Determine License Status Equipment Control D, P, R 2.2.12 (4.1)
Knowledge of surveillance procedures JPM:
Perform Daily Surveillance Items Checklist Radiation Control D, R 2.3.14 (3.8)
Knowledge of radiation or contamination hazards that may arise during normal, abnormal or emergency conditions or activities JPM:
Determine Emergency Exposure Controls During a General Emergency Emergency Procedures/Plan D, R 2.4.44 (4.4)
Knowledge of emergency plan implementing procedures protective action recommendations.
JPM:
Evaluate and prepare an Updated PAR NOTE: All items (5 total) are required for SROs. RO applicants require only four items unless they are retaking only the administrative topics (which would require all five items).
- Type Codes & Criteria:
(C)ontrol room, (0) (S)imulator, (0) or Class(R)oom (5)
(D)irect from bank ( 3 for ROs; 4 for SROs & RO retakes) (3)
(N)ew or (M)odified from bank (> 1) (2)
(P)revious 2 exams ( 1; randomly selected) (1)
ES-3.2 Administrative Topics Outline Form ES-3.2-1 (Rev_041523)
NUREG-1021, Revision 12 SRO Admin JPM Summary A1a This is a modified Bank JPM. The operator will be told that Unit 1 was shutdown 14 days ago for a mid-cycle outage after 200 days of operation, that Unit 1 is currently in Mode 5 with the NC system is 130°F and A Train ND in service; and that preparations are being made to lower NC system level to 67 inches above Hot Leg Centerline per Enclosure 4.1 (Draining the NC System) of OP/1/A/6100/SD-20 (Draining the NC System). The operator will be directed to complete Attachment 12.6 of OMP 5-8 (Shift Supervision Turnovers) to determine the new thermal margin with NC system level at 67 inches above Hot Leg Centerline and make the appropriate notifications (Complete all paperwork). The operator will be expected to determine the Thermal Margin and complete Attachment 12.6 (Thermal Margin Determination) and Attachment 12.7 (Shutdown Assessment Status) of OMP 5-8 (Shift Supervision Turnovers) in accordance with the provided KEY.
A1b This is a Modified Bank JPM. The operator will be provided with the work histories and qualifications of three Licensed Operators that have off-shift assignments at the plant. The operator will be directed to determine if the license for each operator has met the requirements to remain active for the 1st quarter of 2023, and if not identify what action must be taken. The operator will be expected to determine that SRO Lannon meets the requirements to have the SRO License remain active; SRO Sherman must complete one 12-Hour Shift as the SM, CRS, U1 or U2 OATC, or BOP between now and 2400 on 12/31/22 to have the SRO License remain active; and that RO Miller must complete two 12-Hour Shifts as the U1 or U2 OATC or BOP between now and 2400 on 12/31/22 to have the RO License remain active.
A2 This is a Bank JPM. The operator will be told that Unit 1 is in Mode 1 at 100% power, given several initial plant conditions and told that Attachment 1, Daily Surveillance Items Checklist, of PT/1/A/4600/003B, Daily Surveillance Items, has been completed. The operator will be directed to evaluate the completed Attachment 1 of PT/1/A/4600/003B (Daily Surveillance Items Checklist) per Step 7.3, identify all Technical Specification/SLC required ACTION, as well as all other actions that must be taken. The operator will be expected to review the completed Attachment 1, Daily Surveillance Items Checklist and associated Equipment Problem Identification Form and verify that the applicable surveillance items meet specified acceptance criteria. For surveillance items NOT meeting Acceptance Criteria, all required action will be identified per the attached KEY.
Randomly selected from the 2020 MNS NRC Exam.
A3 This is a Bank JPM. The operator will be told that a General Emergency has been declared and that there are reports of an injured non-ambulatory person on-site. As the SM (Emergency Coordinator), the operator will be required to select two rescuers, from among seven potential rescuers, and dispatch them to the injured individual by using AD-EP-ALL-0205 and completing Attachment 1, (Emergency Exposure Authorization), of AD-EP-ALL-0205.
ES-3.2 Administrative Topics Outline Form ES-3.2-1 (Rev_041523)
NUREG-1021, Revision 12 A4 This is a Bank JPM. The operator will be placed in a post-accident condition with a Large Break LOCA and a release from the Containment. The operator will be told that a General Emergency has been declared, and provided with the initial Protective Action Recommendation (PAR). The operator will be given a subsequent set of plant conditions and meteorological data, and asked to prepare an EXPANDED PAR and submit the Emergency Notification Form to the Emergency Coordinator for approval. The operator will be expected to evaluate meteorological conditions and determine the Expanded PAR for the current conditions and complete the Emergency Power Plant Emergency Notification Form per the provided KEY within 15 minutes.
ES-3.2 Control Room/In-Plant Systems Outline Form 3.2-2 (Rev 042023)
NUREG-1021, Revision 12 Facility:
McGuire Date of Examination:
7/2023 Exam Level (circle one):
Operating Test No.:
N23-1 Control Room Systems@ (8 for RO; 7 for SRO-I; 2 or 3 for SRO-U)
System / JPM Title Type Code*
Safety Function A.
035 S/G Steam Generator System [035 A4.06 (4.2)]
Isolate faulted S/G 1D in accordance with E-2 S, N, A, L 4P B.
013 ESF Actuation System [013 A4.01 (4.4)]
Respond to Failed ESF Actuation S, P, D, A, EN 2
C.
010 Pressurizer Pressure Control System [010 A4.02 (3.6)]
Remove Pressurizer Heaters from Service S, D, A, 3
D.
APE 026 Loss of Component Cooling Water [APE 026 AA1.02 (3.5)]
Respond to High VCT Temperature S, D 8
E.
061 Auxiliary/Emergency Feedwater (AFW) System [061 A2.07 (4.0/3.5)]
CA Suction Source Realignment S, M, A 4S F.
016 Non-Nuclear Instrumentation System [016 A4.01 (3.3)]
Respond to an Alternate Action on a VCT Level Channel S, D 7
G.
027 Containment Iodine Removal System [027 A4.01 (2.8)]
Perform the 1A Annulus Ventilation Operability Test S, D, EN 5
H.
APE 056 Loss of Off-Site Power [056 AA1.02 (3.6)]
Restore Normal Power to 1ETB and Unload the 1B EDG S, M 6
In-Plant Systems:* 3 for RO; 3 for SRO-I; 3 or 2 for SRO-U I.
001 Control Rod Drive System [001 A4.08 (3.3)]
Start up the 1A Motor Generator Set D, L 1
J.
033 Spent Fuel Pool Cooling System [A2.03 (3.9/3.7)]
Makeup to the Unit 1 Spent Fuel Pool During a Station Blackout D, R, E 8
K.
APE 040 Steam Line Rupture [040 AA1.10 (3.9)]
Closing 1AS-11 Using Local Controller D, E 4S
ES-3.2 Control Room/In-Plant Systems Outline Form 3.2-2 (Rev 042023)
NUREG-1021, Revision 12
- 1.
Select safety functions and systems for each JPM as follows:
Refer to Section 1.9 of the applicable knowledge and abilities (K/A) catalog for the plant systems organized by safety function. For pressurized-water reactor operating tests, the primary and secondary systems listed under Safety Function 4, Heat Removal from Reactor Core, in Section 1.9 of the applicable K/A catalog, may be treated as separate safety functions (i.e., two systems, one primary and one secondary, may be selected from Safety Function 4). From the safety function groupings identified in the K/A catalog, select the appropriate number of plant systems by safety functions to be evaluated based on the applicants license level (see the table in step 1).
For RO/SRO-I applicants: Each of the control room system JPMs and, separately, each of the in-plant system JPMs must evaluate a different safety function, and the same system or evolution cannot be used to evaluate more than one safety function in each location. One of the control room system JPMs must be an engineered safety feature.
For SRO-U applicants: Evaluate SRO-U applicants on five different safety functions. One of the control room system JPMs must be an engineered safety feature, and the same system or evolution cannot be used to evaluate more than one safety function.
- 2.
Select a task for each JPM that supports, either directly or indirectly and in a meaningful way, the successful fulfillment of the associated safety function. Select the task from the applicable K/A catalog (K/As for plant systems or emergency and abnormal plant evolutions) or the facility licensees site-specific task list. If this task has an associated K/A, the K/A should have an importance rating of at least 2.5 in the RO column. K/As that have importance ratings of less than 2.5 may be used if justified based on plant priorities; inform the NRC chief examiner if selecting K/As with an importance rating less than 2.5. The selected tasks must be different from the events and evolutions conducted during the simulator operating test and tasks tested on the written examination. A task that is similar to a simulator scenario event may be acceptable if the actions required to complete the task are significantly different from those required in response to the scenario event.
Apply the following specific task selection criteria:
At least one of the tasks shall be related to a shutdown or low-power condition.
Four to six of the tasks for RO and SRO-I applicants shall require execution of alternative paths within the facility licensees operating procedures. Two to three of the tasks for SRO-U applicants shall require execution of alternative paths within the facility licensees operating procedures.
At least one alternate path JPM must be new or modified from the bank.
At least one of the tasks conducted in the plant shall evaluate the applicants ability to implement actions required during an emergency or abnormal condition.
At least one of the tasks conducted in the plant shall require the applicant to enter the radiologically controlled area. This provides an excellent opportunity for the applicant to discuss or demonstrate radiation control administrative subjects.
If it is not possible to develop or locate a suitable task for a selected system, return to step 2 and select a different system.
- 3.
For each JPM, specify the codes for type, source, and location:
- Type Codes Criteria for R / SRO-I / SRO-U (A)lternate path 4-6 (4) /4-6 (4) / 2-3 (3)
(C)ontrol room (D)irect from bank 9 (8) / 8 (8) / 4 (4)
(E)mergency or abnormal in-plant 1 (2) / 1 (2) / 1 (1)
(EN)gineered Safety Feature 1 (2) / 1 (2)/ 1 (1) (Control Room System)
(L)ow-Power / Shutdown 1 (2) / 1 (2) / 1 (2)
(N)ew or (M)odified from bank including 1(A) 2 (3) / 2 (2 ) / 1 (1)
(P)revious 2 exams 3 (1) / 3 (1)/ 2 (1) (Randomly Selected)
(R)CA 1 (1)/ 1 (1) / 1 (1)
(S)imulator
ES-3.2 Control Room/In-Plant Systems Outline Form 3.2-2 (Rev 042023)
NUREG-1021, Revision 12 JPM Summary JPM A This is a New JPM. The operator will be told that a reactor trip and safety injection has occurred due to a fault on the 1D S/G inside containment and the crew has completed applicable steps of EP/1/A/5000/E-0 (Reactor Trip or Safety Injection) and transitioned to EP/1/A/5000/E-2 (Faulted Steam Generator Isolation). The operator will be directed to isolate the 1D S/G in accordance with E-2, beginning at step 9. The operator will be expected to isolate the 1D S/G by closing 1CF-26AB, 1CA-38B, 1CA-42B and 1SM-101 and then respond to a failure of 1CA-42B to close by closing 1CA-40B and dispatching an operator to locally close 1CA-42B (Alternate Path).
JPM B This is a Bank JPM. The operator will be told that Unit 1 has experienced an ATWS and an inadvertent Train B Safety Injection actuation, and the crew is presently performing EP/1/A/5000/FR-S.1 (Response to Nuclear Power Generation/ATWS). The operator will be directed to perform Enclosure 3 (Subsequent S/I Actions) of EP/1/A/5000/FR-S.1 (Response to Nuclear Power Generation/ATWS), while the crew continues with the actions of EP/1/A/5000/FR-S.1. The operator will be required to manually actuate the A Train of SI which is expected to start the 1A RN Pump, the operator will observe that the 1A RN Pump has tripped (Alternate Path). The operator will be expected to complete the required actions of Enclosure 3 including manually actuating A Train of SI and Phase A CIS, stopping all NC pumps, resetting A Train SI and Sequencer and dispatching an operator to stop the A Diesel Generator using the Emergency Stop Pushbutton. This JPM appeared on the 2022 Initial License Exam and was randomly selected for the 2023 Exam.
JPM C This is a Bank JPM. The operator will be told that plant power has just been raised to 100% per OP/1/A/6100/003 (Controlling Procedure for Unit Operation). The operator will be directed to remove Pzr Heater Groups A, B and D from service per Enclosure 4.6 (Operation of Pzr Heaters) of OP/1/A/6100/003. The operator will be expected to remove the A, B and D Pzr Heater Groups from service in accordance with the Enclosure. After the Pzr Pressure Master has been placed in MANUAL and its output has been adjusted, the Pzr Variable Heaters (Group C) will fail (Alternate Path). The operator will operate Pzr Heater Group(s) as needed before MCB Annunciator 1AD-6, C6 alarms.
JPM D This is a Bank JPM. The operator will be told that Unit 1 was at 100% power when a leak developed in the KC System, that the crew has entered AP/1/A/5500/21 (Loss of KC or KC System Leakage) and has completed actions through Step 9. They will be told that MCB Annunciator 1AD-7, D1, VCT HI TEMP, has just alarmed, making Foldout Page item
- 5 applicable. The operator will be directed to perform the actions of Enclosure 6 of AP/1/A/5500/21 (Loss of KC or KC System Leakage), while the crew continues with the AP. The operator will isolate Letdown, and align the suction of the NV Pumps to the FWST.
JPM E This is a Modified Bank JPM. The operator will be told that Unit 1 has just tripped from 100% power, due to seismic activity, that the crew is now implementing EP/1/A/5000/ES-0.1 (Reactor Trip Response), and that the CA Storage Tank has developed a leak, and level has lowered to 1.5 feet. The operator will be directed to perform EP/1/A/5000/G-1, Generic Enclosure 20 (CA Suction Source Realignment), while the crew continues with ES-0.1. The operator will realign the suction of the CA Pumps from the non-safety related to the safety-related source (RN). During this action, the operator will recognize that RN Supply to the 1A MDCA Pump cannot be established (Alternate Path) and stop the pump.
ES-3.2 Control Room/In-Plant Systems Outline Form 3.2-2 (Rev 042023)
NUREG-1021, Revision 12 JPM F This is a Bank JPM. The operator will be told that the plant is operating at 100% power, that VCT Level instrument 1NVP-5760 has failed high, and that VCT level has been stabilized. The operator will be directed to respond to an Alternate Action on a VCT Level Channel in accordance with Attachment 1 (Removing/Returning VCT Level Channel From/To Service) of OP/1/A/6102/003 (Distributed Control System Operation). The operator will be expected to use Section 3.2 of Attachment 1 and select 1NVP-5761 for operation.
JPM G This is a Bank JPM. The operator will be told that Unit 1 is operating at 100% power, Unit 1 VE System is aligned for Engineered Safeguards Operation, and PT/1/A/4450/003 A (Annulus Ventilation System Train 'A' Operability Test) is on the Operations schedule for today. The operator will be directed to perform PT/1/A/4450/003 A (Annulus Ventilation System Train 'A' Operability Test). The operator will be expected to place the 1A VE Fan in Recirculation Mode with the cross connect from B Train closed. The 1A VE Fan will be shut down after flow verification and returned to normal alignment.
JPM H This is a Modified JPM. The operator will be told that Unit 1 was operating at 100% power when the normal power breaker to 1ETB was inadvertently opened, the 1B EDG started and re-energized the bus and sequenced loads onto 1ETB as expected, the crew entered AP/1/A/5500/07, Loss of Electrical Power, Case II, Loss of Normal Power to Either 1ETA or 1ETB; and they are currently at Step 85. The operator will also be told that an investigation has revealed that the breaker was inadvertently opened, that the breaker is ready to be re-closed, and that the crew is attempting to return 1ETB to normal power and shutdown the 1B D/G. The operator will be directed to restore 1ETB to normal power and separate the 1B D/G from the Grid from the Control Room per OP/1/A/6350/002 (Diesel Generator), Attachment 4 (1B D/G Shutdown). The operator will be expected to parallel 1ETB, with 1ATD, and then unload the 1B D/G.
JPM I This is a Bank JPM. The operator will be told that a unit start-up is in progress on Unit 1 and that all requirements for entering Mode 2 are met. The operator will be directed to start the 1A M/G Set per OP/1/A/6150/008 (Rod Control) Enclosure 4.1 (M/G Startup) for post maintenance inspection. The operator will be expected to start the 1A M/G Set per OP/1/A/6150/008.
JPM J This is a Bank JPM. The operator will be told that the site has experienced a sustained Station Blackout, Unit 1 is implementing Enclosure 31 (Monitoring Unit 1 Spent Fuel Pool Level and Temperature) of EP/1/A/5000/ECA-0.0 (Loss of All AC Power), the Unit 1 Spent Fuel Pool Level is MINUS 1.5 Feet, the Unit 1 OAC and several instruments, including the U1 FWST Level indication are NOT available, and that they are an available operator in the Control Room. The operator will be directed to coordinate with the BOP and makeup to the Unit 1 Spent Fuel Pool by performing Step 11.E and 11.F of Enclosure 31 of EP/1/A/5000/ECA-0.0 (Loss of All AC Power); and raise Spent Fuel Pool level to 0 inches.
The operator will be expected to align the needed valves to makeup to the Unit 1 Spent Fuel Pool by performing Step 11.E of Enclosure 31 of EP/1/A/5000/ECA-0.0, recognize that pool level is at the desired level, and secure the lineup per step 11.F of Enclosure 31.
ES-3.2 Control Room/In-Plant Systems Outline Form 3.2-2 (Rev 042023)
NUREG-1021, Revision 12 JPM K This is a Bank JPM. The operator will be told that Unit 1 is at 100% power and Unit 2 is at 50% power, that a Steam Break subsequently occurred on the Unit 1 SM Equalization Header causing a Reactor Trip and Main Steamline Isolation, and that the crew is currently implementing both EP/1/A/5000/ES-0.1 (Reactor Trip Response) and AP/1/A/5500/01 (Steam Leak). The operator will be directed to CLOSE 1AS-11 PER Step 2 of EP/1/A/5000/G-1 (Generic Enclosures), Enclosure 4 (Closing 1AS-11 Using Local Controller). The operator will be expected to CLOSE 1AS-11 using Step 2 of Generic, thereby stopping the backflow of steam from Unit 2 via the AS System.
Form 3.4-1 Events and Evolutions Checklist Facility: McGuire Date of Exam: 07/31/23 Operating Test No.: N23-1 A
P P
L I
C A
N T
E V
E N
T T
Y P
E Scenarios N23-1-3 (Day 1)
N23-1-6 (Day 2)
N23-1-5 (Day 3)
N23-1-4 (Day 4)
T O
T A
L M
I N
I M
U M*
POSITION POSITION POSITION POSITION S
R O
A T
C B
O P
S R
O A
T C
B O
P S
R O
A T
C B
O P
S R
O A
T C
B O
P RO I
U SRO-U 1
RX 1
1 0
NOR 1
1 1
1 1
I/C 2,3,4,5 4
4 4
2 MAJ 6,7 2
2 2
1 Man. Ctrl 1
1 0
TS 2,3 2
0 2
2 RO 1
RX 1
1 1
1 0
NOR 1
1 1
1 1
I/C 3,5 2,5 4
4 4
2 MAJ 7
6,7 3
2 2
1 Man. Ctrl 3,5,8 1
4 1
1 0
TS 0
2 2
RO 2
RX 1
1 1
1 1
0 NOR 1
1 1
1 1
I/C 3,5 2,5 4,5 6
4 4
2 MAJ 7
6,7 6,7 5
2 2
1 Man. Ctrl 3,5,8 1
4,5 6
1 1
0 TS 0
2 2
Form 3.4-1 Events and Evolutions Checklist Facility: McGuire Date of Exam: 07/31/23 Operating Test No.: N23-1 A
P P
L I
C A
N T
E V
E N
T T
Y P
E Scenarios N23-1-3 (Day 1)
N23-1-6 (Day 2)
N23-1-5 (Day 3)
N23-1-4 (Day 4)
T O
T A
L M
I N
I M
U M*
POSITION POSITION POSITION POSITION S
R O
A T
C B
O P
S R
O A
T C
B O
P S
R O
A T
C B
O P
S R
O A
T C
B O
P RO I
U RO 3
RX 1
1 1
1 0
NOR 1
1 1
1 1
I/C 3,5 2,5 1,3 6
4 4
2 MAJ 7
6,7 5
4 2
2 1
Man. Ctrl 3,5,8 1
3 5
1 1
0 TS 0
2 2
RO 4
RX 1
1 1
1 0
NOR 0
1 1
1 I/C 1,3 4,5 4
4 4
2 MAJ 5
6,7 3
2 2
1 Man. Ctrl 3
4,5 3
1 1
0 TS 0
2 2
RO 5
RX 1
1 1
1 0
NOR 0
1 1
1 I/C 1,3 4,5 4
4 4
2 MAJ 5
6,7 3
2 2
1 Man. Ctrl 3
4,5 3
1 1
0 TS 0
2 2
SRO-I 1
RX 0
1 1
0 NOR 1
1 2
1 1
1 I/C 2,3,4,5
,6 3,4,5 8
4 4
2 MAJ 7
6,7 3
2 2
1 Man. Ctrl 4,8 2
1 1
0 TS 4,5 2
0 2
2 SRO-I 2
RX 1
1 1
1 0
NOR 1
1 1
1 1
I/C 2,6 2,3,4,5 6
4 4
2 MAJ 7
6,7 3
2 2
1 Man. Ctrl 2
1 1
1 0
TS 3,4 2
0 2
2
Form 3.4-1 Events and Evolutions Checklist Facility: McGuire Date of Exam: 07/31/23 Operating Test No.: N23-1 A
P P
L I
C A
N T
E V
E N
T T
Y P
E Scenarios N23-1-3 (Day 1)
N23-1-6 (Day 2)
N23-1-5 (Day 3)
N23-1-4 (Day 4)
T O
T A
L M
I N
I M
U M*
POSITION POSITION POSITION POSITION S
R O
A T
C B
O P
S R
O A
T C
B O
P S
R O
A T
C B
O P
S R
O A
T C
B O
P RO I
U SRO-I 3
RX 0
1 1
0 NOR 1
1 2
1 1
1 I/C 2,3,4,5
,6 3,4,5 8
4 4
2 MAJ 7
6,7 3
2 2
1 Man. Ctrl 4,8 2
1 1
0 TS 4,5 2
0 2
2 SRO-I 4
RX 1
1 1
1 0
NOR 1
1 1
1 1
I/C 2,6 2,3,4,5 6
4 4
2 MAJ 7
6,7 3
2 2
1 Man. Ctrl 2
1 1
1 0
TS 3,4 2
0 2
2 SRO-I 5
RX 0
1 1
0 NOR 1
1 2
1 1
1 I/C 2,3,4,5
,6 3,4,5 2,4 10 4
4 2
MAJ 7
6,7 5
4 2
2 1
Man. Ctrl 4,8 2,4,6 5
1 1
0 TS 4,5 2
0 2
2 SRO-I 6
RX 1
1 1
1 0
NOR 1
1 1
1 1
I/C 2,6 2,3,4,5 6
4 4
2 MAJ 7
6,7 3
2 2
1 Man. Ctrl 2
1 1
1 0
TS 3,4 2
0 2
2 SRO-I 7
RX 0
1 1
0 NOR 1
1 1
1 1
I/C 1,2,3,4 3,5 6
4 4
2 MAJ 5
6,7 3
2 2
1 Man. Ctrl 3,5,8,9 4
1 1
0 TS 1,3 2
0 2
2
Form 3.4-1 Events and Evolutions Checklist Facility: McGuire Date of Exam: 07/31/23 Operating Test No.: N23-1 A
P P
L I
C A
N T
E V
E N
T T
Y P
E Scenarios N23-1-3 (Day 1)
N23-1-6 (Day 2)
N23-1-5 (Day 3)
N23-1-4 (Day 4)
T O
T A
L M
I N
I M
U M*
POSITION POSITION POSITION POSITION S
R O
A T
C B
O P
S R
O A
T C
B O
P S
R O
A T
C B
O P
S R
O A
T C
B O
P RO I
U SRO-I 8
RX 0
1 1
0 NOR 1
1 1
1 1
I/C 2,4 2,3,4,5 6
4 4
2 MAJ 5
6,7 3
2 2
1 Man. Ctrl 2,4,6 3
1 1
0 TS 2,3 2
0 2
2 SRO-I 9
RX 0
1 1
0 NOR 1
1 1
1 1
I/C 1,2,3,4 3,5 6
4 4
2 MAJ 5
6,7 3
2 2
1 Man. Ctrl 3,5,8,9 4
1 1
0 TS 1,3 2
0 2
2 SRO-I 10 RX 0
1 1
0 NOR 1
1 1
1 1
I/C 2,4 2,3,4,5 6
4 4
2 MAJ 5
6,7 3
2 2
1 Man. Ctrl 2,4,6 3
1 1
0 TS 2,3 2
0 2
2 SRO-I 11 RX 0
1 1
0 NOR 1
1 1
1 1
I/C 1,2,3,4 3,5 6
4 4
2 MAJ 5
6,7 3
2 2
1 Man. Ctrl 3,5,8,9 4
1 1
0 TS 1,3 2
0 2
2
Facility:
McGuire Exam Date: 2023 1
JPM # or title 2
Type (S/P/A) 3 ALT (Y/N) 4 LOD (1-5) 5 JPM Errors 6
U/E/S 7
Explanation LOD REF IC TSK CUE CS TL General comments:
A S
Y 2-4 S
B S
Y 2-4 S
C S
Y 2-4 S
JPM Step 23 may not be critical. Typically a monitor step with no actions would not be critical; however, the action taken as a result of adequately monitoring would be critical (I.E. energizing a heater bank).
Facility Response:
Agreed, not sure why this was marked as a critical step. Asterisk removed. SLM 5/30/23 D
S N
2-4 S
E S
Y 2-4 S
F S
N 2-4 S
G S
N 2-4 S
H S
N 2-4 S
I P
N 2-4 S
J P
N 2-4 S
Facility:
McGuire Exam Date: 2023 1
JPM # or title 2
Type (S/P/A) 3 ALT (Y/N) 4 LOD (1-5) 5 JPM Errors 6
U/E/S 7
Explanation LOD REF IC TSK CUE CS TL K
P N
2-4 S
General During on-site prep the exam team will review all critical steps and task standards to ensure they are satisfactory.
RO A1a A
2-4 E
S Only the first watch states the unit mode (No Mode). For all the other watches, the student must assume the mode. Mode may need to be stated for each watch.
Instead of stating No Mode the plant conditions that represent No Mode should be stated - I.E. Core has been offloaded to facilitate a repair.
Facility Response:
Changed to describe conditions versus giving mode for No MODE and did this for all stated watches (ie power level). SLM 5/30/23 If I understand your student handouts correctly, the handout that they get will not include any cues as to which procedure contains the requirements. If my understanding is correct, then I have no concerns with this aspect of administration.
Facility Response:
Facility:
McGuire Exam Date: 2023 1
JPM # or title 2
Type (S/P/A) 3 ALT (Y/N) 4 LOD (1-5) 5 JPM Errors 6
U/E/S 7
Explanation LOD REF IC TSK CUE CS TL That is correct, the handout has no information as to which procedure has the required information to perform the SLM.
5/30/23 RO A1b A
2-4 E
S Telling them to go to two decimal places is not operationally valid. An acceptance band is required and their answer must be within the acceptance band. They need to know how many decimal places to use if the procedure does not tell them. If they are comparing against a Tech Spec at two decimal places, then wouldnt they have to do the calc to at three decimal places so that they are comparing the same significant figures? I.E. 1.021 is not within the Tech Spec Limit.
Facility Response:
Removed reference to two decimal places in cue. SLM 5/30/23 Why does the Initiating Cue tell them which procedure step to use? Should they not be able to figure out which step to use?
Facility Response:
I agree they should know. However, there are 15 steps prior to the step they will start at, none of which address the task standard.
Some require SRO signatures such as ERDS and TS calls, etc. It could take as much as 30 minutes to evaluate some of these steps (Fire
Facility:
McGuire Exam Date: 2023 1
JPM # or title 2
Type (S/P/A) 3 ALT (Y/N) 4 LOD (1-5) 5 JPM Errors 6
U/E/S 7
Explanation LOD REF IC TSK CUE CS TL watches, etc) and someone signing as an SRO to get to the step to calculate QPTR.
2-4 S
RO A3 2-4 S
SRO A1a A
2-4 S
SRO A1b A
2-4 S
Provide the conditions for the applicant to determine the Mode of operation. I.E. Instead of stating No Mode, state fuel has been removed. Provide a power level for Mode 1.
Provide an activity that is in progress that is indicative of Mode 5 or 6 (I.E. mid-loop installing nozzle dams). This would require a re-labeling of the column for Mode - perhaps the columns would be more appropriately named Plant Status?
Facility Response:
Made changes as suggested. SLM 5/30/23 SRO A2 A
2-4 S
SRO A3 A
2-4 S
SRO A4 A
2-4 E
S Does the Initiating Cue need to specifically prompt them to evaluate the weather conditions? Shouldnt the students know they
Form 2.3-3 Operating Test Review Worksheet (JPMs)
Facility:
McGuire Exam Date: 2023 1
JPM # or title 2
Type (S/P/A) 3 ALT (Y/N) 4 LOD (1-5) 5 JPM Errors 6
U/E/S 7
Explanation LOD REF IC TSK CUE CS TL need to evaluate the weather as part of an expanded PAR?
Facility Response:
Removed meteorological from the cue. SLM 5/30/23
Form 2.3-3 Instructions for Completing the JPM Table
- 1. Enter the JPM number and/or title.
- 2. Enter the type of JPM(S)imulator, (P)lant, or (A)dministrative.
- 3. Enter (Y)es or (N)o for an Alternate Path JPM.
- 4. Rate the level of difficulty (LOD) of each JPM using a scale of 1-5 (easy-difficult). A JPM containing less than two critical steps, a JPM that tests solely for recall or memorization, or a JPM that involves directly looking up a single correct answer is likely LOD = 1 (too easy). Conversely, a JPM with over 30 steps or a JPM that takes more than 45 minutes to complete is likely LOD = 5 (too difficult).
- 5. Check the appropriate block for each JPM error type, using the following criteria:
LOD = 1 or 5 is unsatisfactory (U).
REF: The JPM lacks required references, tools, or procedures (U).
IC: The JPM initial conditions are missing or the JPM lacks an adequate initial cue (U).
CUE: The JPM lacks adequate evaluator cues to allow the applicant to complete the task, or the evaluator cues are subjective or leading (U).
TSK: The JPM lacks a task standard or lacks completion criteria for a task standard (U).
CS: The JPM contains errors in designating critical steps, or the JPM lacks an adequate performance standard for a critical step (U).
TL: The JPM validation times are unreasonable, or a time-critical JPM lacks a completion time (U).
- 6. Mark the JPM as unsatisfactory (U), satisfactory (S), or needs enhancements (E). A JPM is (U) if it has one or more (U) errors as determined in step 5. Examples of enhancements include formatting, spelling, or other minor changes.
- 7. Briefly describe any JPM determined to be unsatisfactory (U) or needing enhancement (E). Save initial review comments and detail subsequent comment resolution so that each exam-bound JPM is marked by a satisfactory (S) resolution on this form.
Form 2.3-3 Operating Test Review Worksheet (Scenarios)
Facility: McGuire Scenario: 1 Exam Date: 2023 1
Scenario Event ID/Name:
2 Scenario event errors 3
U/E/S 4
Explanation Realism/
Credibility Performance Standards Verifiable Actions Critical Task TS 1
S 2
S 3
S 4
S 5
S 6
S 7
x S
CT1: Minor wording change - Isolate the Main Turbine from the SGs by manually closing the governor valves to avoid reaching Orange Path on NC Integrity (Any T-cold < 250F)
CT2: Minor wording change - isolate the faulted SG to avoid Orange Path on NC Integrity.
Facility Response:
Made changes as suggested. SLM 5/30/23 8
S 9
S
Form 2.3-3 Instructions for Completing the Scenario Table
- 1. For each scenario, enter the scenario event names and descriptions.
- 2. Review the individual events contained in each scenario, and identify and mark event errors:
The scenario guide event description is not realistic/credibleunsatisfactory (U).
The scenario guide event description lacks adequate crew/operator performance standardsneeds enhancement (E).
The scenario guide event description lacks verifiable actions for a credited normal event, reactivity event instrument/component malfunction, or technical specification (TS) event (or a combination of these) (U).
The scenario guide event description incorrectly designates an event as a critical task (i.e., a noncritical task labeled as critical or a critical task labeled as noncritical). This includes critical tasks that do not meet the critical task criteria (i.e., the critical task does not have a measurable performance standard) (U).
The scenario guide event description incorrectly designates entry into TS actions when not required or does not designate entry into TS actions when required (U).
- 3. Based on the outcome in step 2, mark the scenario event as unsatisfactory (U), satisfactory (S), or needs enhancements (E). An event is (U) if it has one or more (U) errors as determined in step 2. Examples of enhancements include formatting, spelling, or other minor changes.
- 4. Briefly describe any scenario event determined to be unsatisfactory (U) or needing enhancement (E). Save initial review comments and detail subsequent comment resolution so that each exam-bound scenario event is marked by a satisfactory (S) resolution on this form.
Facility: McGuire Scenario: 2 Exam Date: 2023 1
Scenario Event ID/Name:
2 Scenario event errors 3
U/E/S 4
Explanation Realism/
Credibility Performance Standards Verifiable Actions Critical Task TS 1
S 2
S 3
S 4
S 5
S 6
S 7
S 8
S 9
S
Facility: McGuire Scenario: 3 Exam Date: 2023 1
Scenario Event ID/Name:
2 Scenario event errors 3
U/E/S 4
Explanation Realism/
Credibility Performance Standards Verifiable Actions Critical Task TS 1
S 2
S 3
S 4
S 5
S 6
S 7
S 8
S 9
S
Facility: McGuire Scenario: 4 Exam Date: 2023 1
Scenario Event ID/Name:
2 Scenario event errors 3
U/E/S 4
Explanation Realism/
Credibility Performance Standards Verifiable Actions Critical Task TS 1
S 2
S 3
S 4
S 5
S 6
S Minor wording enhancement on CT2 -
flow must be established to avoid reaching 700F.
Facility Response:
Made changes as suggested. SLM 5/30/23 7
S 8
S 9
S
Facility: McGuire Scenario: 5 Exam Date: 2023 1
Scenario Event ID/Name:
2 Scenario event errors 3
U/E/S 4
Explanation Realism/
Credibility Performance Standards Verifiable Actions Critical Task TS 1
S 2
S 3
S 4
S 5
S 6
S 7
S 8
S
Facility: McGuire Scenario: 6 Exam Date: 2023 1
Scenario Event ID/Name:
2 Scenario event errors 3
U/E/S 4
Explanation Realism/
Credibility Performance Standards Verifiable Actions Critical Task TS 1
S 2
S 3
S 4
S 5
S 6
S 7
S 8
S
Final Sample Plan is a combinaon of the List of Rejected K/As and the Original Dra Sample Plan.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level In the first column I have placed a P when the question requires emergency or abnormal procedure knowledge. This is to check for balance of coverage to ensure that adequate procedure knowledge is tested on the exam as a whole.
Facility Response: 28 Qs annotated with a P (26 Tier 1 K/As)
(+)/(-) are also used in the first column to ensure when binary question choices are provided (I.E. will / will NOT) that an acceptable balance exists between the positives and negatives.
Facility Response: RO Section 21 (+), 20 (-). SRO Section 10 (+),
5 (-)
Questions relying on a provided reference to students (REF): 11 Even though display of an instrument reading has nothing to do with a plant reference, NRR has instructed examiners to count these as a reference? Also, there is no limit on the number of actual references, as well as no limit on operationally valid displays of instrument readings.
Preliminary Statistics for Form 2.3-4:
Facility Response: Current stats in RED Bank RO: 37/27 SRO: 14/13 Mod RO: 20/30 SRO: 6/7 New RO: 18 SRO: 5 High LOK RO: 43/41 SRO: 17/16 Fundamental LOK RO: 32/34 SRO: 8/9 Reminders for Chief Examiner:
Check balance of coverage for procedures - Tier 1. (There were 28 Qs on the RO exam that required emergency/abnormal procedure knowledge to arrive at the correct answer - abnormal procedures included Tech Specs, ARPs, etc., that address abnormal conditions.)
Check for (+)/(-) balance. (Close to an even split for the binary choice formats - I.E. on the (is) / (is not) format of questions.
Check for acceptable use of the word ONLY.
Check for questions that ask for what an operator will do - if it is IAW a procedure it may be okay - otherwise, they should test what is required.
Provided References need to be reviewed to ensure they do not contain cues for unrelated questions.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level Ensure not too many bank questions come from any single exam.
NUREG-1021, Rev 12, ES-4.2, Page 4 of 29 If the bank contains more than one question that fits a specific K/A statement, randomly select from among the available questions unless there is an appropriate basis for selecting a specific question (e.g., a particular question has a higher cognitive level, has better discrimination validity, is more operationally oriented, or addresses site-specific priorities.
8 bank questions were used from the 2018 NRC exam, 8 from the 2020-1 exam, 6 from the 22-1 Audit, and 6 from the 2020-1 Audit exam. There is a hard limit on the number of questions that can be used from the previous two exams; however, the NUREG does not prescribe hard limits on the exams beyond the last two. The above ES-4.2 requirement strives to ensure that when bank questions are selected outside of the most recent two exams, that a generally random process is used so as not to allow the students to focus on an insufficiently small exam bank. Statistically, based on the bank questions selected from exams mentioned previously, it would seem unlikely that 28 questions would get randomly selected from the those exams.
If the audit exams are not published for student study, then no modifications need to be made for the 22-1 and 20-1 bank questions.
Rather than go through an exercise of understanding how authors linked questions to K/As and randomly select from among them, it may be more productive to modify some of the questions from 2020-1 and 2018-1 to reduce the impact of bank questions being over sampled. We could try to modify those that are most easily transitioned to a modified question. That may be enough to alleviate any concern of the applicants being able to target a small exam bank and get a large percentage of the exam correct.
Statistically, if the above NUREG requirement was followed, it would be unlikely that so many questions would be taken from those three exams if they were randomly selected from the bank.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level Facility Response: Chief examiner was not aware that there were two exams in 2020. Therefore, the comment from above for 8 questions from the 2020 exam in not valid as they are spread across the 2020 exam and the 2020 retake exam.
There were 8 bank questions used from the 2018 NRC exam.
Questions 25, 37 and 59 have been modified.
There were 6 bank questions used from the 22-1 Audit. Q33 was replaced and Q56 was modified.
There were 6 bank questions used from the 20-1 Retake Audit. Q8 and Q31 were modified and Q52 was replaced.
Comment addressed with changes made. MAB 1
(+)
(-)
H 2-4 B
(CNS 2003 NRC Q27)
Lea S
006K4.05 (T2/G1) Item 41 Q is sat.
2 H
2-4 B
(MNS 2013 NRC)
Laska S
061A4.02 (T2/G1) Item 58 Q is sat.
3 H
2-4 M
(MNS 20-1R NRC)
Lanyi S
064A1.09 (T2/G1) Item 62 Double check the referenced Bank question. I looked up the 20-1 NRC exam and did not see the question. Perhaps my search was flawed?
4 REF
(+)
H 2-4 M
(MNS 22-1 NRC)
Bates S
078A4.02 (T2/G1) Item 67 Q is sat, 5
H 2-4 x
B (MNS 2010 NRC)
Aiello E
S 004A1.04 (T2/G2) Item 39 Stem Focus: The last bullet in the initial set of conditions states, The OATC reducesto reduce Pressurizer level. Can this entire bullet be deleted? They have already been told that the SLIM is in MANUAL and then flow is at 18 gpm.
Incorporated. MAB Stem Focus: Does the actual pressurizer level when the SLIM is placed in MANUAL need to be stated? There was a Master Controller malfunction which may have driven level high (we dont know because the question does not say). The student may have to make an assumption of where level was when 18 gpm was established - otherwise perhaps level may still b egoing down at the 5 minute point. Discussed. Sat. MAB
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level 6
H 2-4 B
(MNS 2013 NRC)
Laska S
022A3.02 (T2/G1) Item 49 Q is sat.
7 REF
(-)
H 2-4 x
x N
E S
026K4.08 (T2/G1) Item 53 Does Spray ever automatically align? I am assuming that for a LBLOCA that containment spray would automatically initiate. At what pressure does Spray initiate during a LOCA? Did pressure peak at a higher value and is now trending down? What information is provided in the question to know that they are in ES-1.3 and manually aligning Spray for recirc? No auto spray initiation at MG. MAB Cred. Dist.: The containment spray system should always be manually aligned if manual alignment is required. Potentially the better way to ask is: whether or not, based on the stated conditions, spray will automatically align? Incorporated. MAB Perhaps these comments can be resolved with discussion, or perhaps some detail needs to be added to ensure students do not have to make assumptions. Also potentially need to address the idea that manual alignment of equipment when required is always a correct thing to do. Minor mods made. Sat. MAB 8
(P)
H 2-4 x
B (20-1R Audit)
M E
S 013A2.01 (T2/G1) Item 48 Cred. Dist.: The first part question would be a better test of knowledge if it simply asked: A reactor trip (WILL) / (WILL NOT) occur if pressurizer pressure lowers to 1945 psig. Incorporated.
MAB The reason the above is a better test is that an applicant must know and apply the interlock. The original wording provides the applicant multiple ways to eliminate distractors or just knowlng that containment pressure has not interlock. OK MAB 9
REF
(-)
H F
2-4 N
E S
039A3.03 (T2/G1) Item 54 LOK: Is the question more of a (F)undamental LOK vs. a comprehensive level question? Beyond simply knowing the setpoints for PORVs and Safeties, is any other knowledge needed
- or is any real analysis needed? Changed to fundamental knowledge. OK MAB 10
(-)
H 2-4 B
(18-1 MNS NRC)
Bates S
008A3.01 (T2/G1) Item 44 Q is sat.
11
(+)
H 2-4 B
S 003K5.03 (T2/G1) Item 37 Q is sat.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level (18-1 MNS NRC)
Bates 12 REF H
1 x
M (22-1 Audit)
B U
S 004A1.02 (T2/G1) Item 38 Cred. Dist.: C(2) and D(2) are not credible. It is not credible to think that Tave will lower when diluting. Suggest replacing the second part of the question. Q modified. Sat. MAB 13 REF
(-)
(-)
H 2-4 N
S 005K6.07 (T2/G1) Item 40 Q is sat.
14 F
2-4 M
(18-1 Audit)
S 007K4.05 (T2/G1) Item 42 Q is sat.
15 F
2-4 B
(22-1 MNS Audit)
S 007K6.11 (T2/G1) Item 43 PRE-REVIEW Q Q is sat based on pre-review by Dan.
16 F
2-4 x
N E
S 010G2.1.23 (T2/G1) Item 45 Stem Focus: Much of the information in the stem appears to be unnecessary. Can the question be asked in a more straight-forward manner? I.E. When 2NC-32B or 2NC36B fail to automatically open, OP/2/A/6102/005 directs the operator to manually open them at ______ psig. Incorporated. MAB 17
(-)
H 2-4 M
(22-1 MNS NRC)
Bates S
010K3.01 (T1/G1) Item 46 Q is sat.
18 REF H
2-4 x
B (GFE)
M E
S 012 - 191002K1.08n(T2/G1) Item 47 Questions in Tier 2 should test a site-specific piece of knowledge when possible. For example - testing the response of the pressurizer pressure control or level control. Be careful of overlap with other questions. Site-specific aspect added. MAB Cue: You can apply knowledge from each part of the question to help answer the other part when contemplating the weight of the water columns on the dP cell. Q modified. MAB 19 F
2-4 N
S 025K2.01 (T2/G1) Item 50 Q is sat.
20 H
2-4 N
S 039A3.01 (T2/G1) Item 55 Q is sat.
21 REF F
2-4 M
S 059G2.2.22 (T2/G1) Item 56 Q is sat.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level (22-1 MNS Audit) 22 F
2-4 N
S 059K1.06 (T2/G1) Item 57 Q is sat.
23 F
2-4 N
S 062K5.12 (T2/G1) Item 59 Q is sat.
24 (P)
F 2-4 M
(20-1R MNS NRC)
Lanyi S
063A2.01 (T2/G1) Item 61 Q is sat.
25
(-)
H 2-4 B
(18-1R MNS NRC)
Bates M
S 073A2.01 (T2/G1) Item 63 PRE-REVIEW Q Q is sat.
Q now modified. MAB 26 F
2-4 N
S 073K3.06 (T2/G1) Item 64 Q is sat.
27 F
2-4 B
(2015 MNS Audit)
S 076K2.08 (T2/G1) Item 65 Q is sat.
28 F
2-4 B
(2008 MNS Audit)
S 103K3.04 (T2/G1) Item 69 PRE-REVIEW Q Q is sat.
29 (P)
H 2-4 B
(18-1 MNS NRC)
Bates S
002K6.05 (T2/G2) Item 71 Q is sat.
30 REF H
M 2-4 x
x N
M (22-1 MNS NRC)
U S
016A2.02 (T2/G2) Item 73 Cred. Dist.: The second half of the question appears to be direct lookup. Students are told that one of the four channels fails and the second half provides the reference that tells the applicant what the DNB spec is with 3 good channels. Suggest either not providing the reference or replacing the second half.
The block diagram provides information how the system works.
This does not appear to be appropriate. We can discuss to ensure I correctly understand why this may be appropriate if you believe that I may not have the correct perspective.
Q was replaced. MAB
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level 31 H
F 2-4 B
(20-1R MNS Audit)
M S
029K6.10 (T2/G2) Item 75 Q is sat.
Q modified. Is the modifed question more of a Fundamenral/Memory level of knowledge? MAB LOK changed to F. MAB 32
(-)
F 2-4 B
(18-1R MNS NRC)
Bates S
034K5.02 (T2/G2) Item 76 Q is sat.
33 H
2-4 B
(22-1 MNS Audit)
M (2019 CNS NRC)
S 035A4.11 (T2/G2) Item 77 Q is sat.
Replacement Q is sat also. MAB 34
(+)
F 2-4 M
(18-1 MNS Audit)
S 041K3.02 (T2/G2) Item 78 35 REF H
2-4 x
N E
S 045K4.07 (T2/G2) Item 79 Stem Focus: Are all of the indications presented in the stem clear to the students. As you did with the generator breakers, do the other indications require clarification? Facility double checked indication clarity and stated that no clarifications were required.
MAB 36
(-)
F 2-4 B
(20-1R MNS NRC)
Lanyi S
055K1.04 (T2/G2) Item 80 Q is sat.
37
(+)
(+)
H 2-4 B
(18-1 MNS NRC)
Bates M
S 068A1.04 (T2/G2) Item 81 Q is sat.
Modified Q is still sat. MAB 38 (P)
?
F 2-4 B
(2009 MNS Audit)
E S
008AK3.05 (T1/G1) Item 2 PRE-REVIEW Q LOK: Is the question a fundamental LOK. It looks like they just need to do a memory level comparison of four parameters - but there is not really any high cognitive application of knowledge.
Changed to F LOK. MAB
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level 39 (P)
(-)
(+)
H 2-4 B
(20-1 Retake MNS NRC)
Lanyi E
S 009EK3.22 (T1/G1) Item 3 I could not find this question in the 20-1 MNS NRC exam. What was the question number on the 20-1 exam? Q was from the 20-1 Retake. MAB 40 (P)
(+)
(+)
H 2-4 M
(22-1 MNS NRC)
S 011EK1.06 (T1/G1) Item 4 Q is sat.
41 (P)
F 2-4 M
(20-1R MNS Audit)
S 015AA2.10 (T1/G1) Item 5 Q is sat.
42 (P)
H 2-4 B
(20-1R MNS NRC)
Lanyi E
S 022AK3.01 (T1/G1) Item 6 What was the question number from the 20-1R MNS NRC exam? I could not find a matching question from that exam. Q was from the 20-1 Retake. MAB 43 (P)
H 2-4 B
(20-1 MNS NRC)
Lanyi S
026AK2.01 (T1/G1) Item 8 Q is sat.
44 (P)
(+)
H 2-4 B
(22-1 MNS Audit)
S 027G2.4.2 (T1/G1) Item 9 Q is sat.
45 REF (P)
H 2-4 N
S 029EA1.23 (T1/G1) Item 10 Q is sat.
46 (P)
(-)
H 2-4 N
S 038G2.3.12 (T1/G1) Item 11 Q is sat.
47 (P)
H 2-4 B
(20-1R MNS Audit)
S 054AK1.02 (T1/G1) Item 13 Q is sat.
48 (P)
F 2-4 M
(2016 CNS NRC)
Meeks S
055EK2.09 (T1/G1) Item 14 Q is sat.
49 H
2-4 M
(20-1R MNS NRC)
S 056AK2.11 (T1/G1) Item 15 Q is sat.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level Lanyi Note that no procedure knowledge is required to arrive at the correct answer. This is now allowed - but the balance of the exam will be evaluated for balance of coverage to ensure emergency/abnormal procedures are adequately tested. No resolution required. MAB 50 (P)
F 2-4 B
(20-1R MNS NRC)
Lanyi S
057G2.1.19 (T1/G1) Item 16 Q is sat.
51 (P)
F 2-4 M
(22-1 MNS Audit)
S 058AA2.02 (T1/G1/) Item 18 Q is sat.
52 (P)
F H
2-4 x
B (20-1R MNS Audit)
(2016 MNS Audit Retake)
U S
062AA1.08 (T1/G1) Item 19 Cred. Dist.: It is difficult to see how choosing between an isolation valve and a control valve allows for plausible distractors. Consider modifying the first half to: IAW AP-20, 1RN-187B (IS)/(IS NOT) required to be used to establish. This will eliminate the comparison between a control valve and an isolation valve.
Replacement Q is sat. MAB 53 (P)
F 2-4 B
(20-1 MNS NRC)
Lanyi S
WE04EK1.04 (T1/G1) Item 22 Q is sat.
54 (P)
(+)
H 2-4 N
S WE11EA2.04 (T1/G1) Item 23 Q is sat.
55 (P)
H 2-4 N
S WE05EA1.07 (T1/G1) Item 24 Q is sat.
56 (P)
F 2-4 B
(22-1 MNS Audit)
M S
003AA1.08 (T1/G2) Item 25 Q is sat.
Modified Q is also sat. MAB 57 (P)
(+)
F 2-4 B
(2016 CNS NRC)
Meeks S
005AA2.06 (T1/G2) Item 26 Q is sat.
58 (P)
(-)
H 2-4 M
(2008 MNS Audit)
S 024AK2.10 (T1/G2) Item 27 Q is sat.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level 59 REF (P)
(+)
H 2-4 B
(18-1 MNS NRC)
Bates M
S 032G2.1.2 (T1/G2) Item 28 Q is sat.
Modified Q is also sat. MAB 60
(+)
H 2-4 M
(20-1R MNS Audit)
E S
032G2..2 (T1/G2) Item 29 Whether a reactor trip signal is generated has nothing to do with what AP-16 says - it is based on plant design. Suggest deleting reference to AP-16 in the second question statement. Incorporated MAB Note that no procedure knowledge is required to arrive at the correct answer. This is now allowed - but the balance of the exam will be evaluated for balance of coverage to ensure emergency/abnormal procedures are adequately tested.
No resolution required. MAB 61 (P)
H 2-4 M
(2004 CNS NRC)
Laska S
036AK3.04 (T1/G2) Item 30 Q is sat.
62
(-)
H 2-4 x
M (2005 MNS NRC)
Hopper E
S WE01EK1.05 (T1/G2) Item 33 Is it possible to have a LOCA in a specific location where the indicated RCS temperature could be going down? Or is the fact that RCS pressure is above injection shutoff head enough to rule out a LOCA? Just double checking that there is only one correct answer. RCS P raised to 1850 psig. Sat. MAB 63 (P)
(+)
(+)
H 2-4 M
(2017 CNS NRC)
S WE15EA2.04 (T1/G2) Item 34 PRE-REVIEW Q Q is sat.
64 F
2-4 M
(19-1 RNP NRC)
S G2.1.29 (T3) Item 84 Q is sat.
65
(-)
(+)
F 2-4 B
(21-1 CNS NRC)
KrchBm S
G2.1.37 (T3) Item 85 Q is sat.
66 (P)
H 2-4 B
(22-1 MNS NRC)
Bates S
G2.2.35 (T3) Item 87 Q is sat.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level 67 F
2-4 N
S G2.2.13 (T3) Item 86 Q is sat.
68 (P)
(+)
(-)
F 2-4 B
(2017 CNS NRC)
Bates S
G2.3.11 (T3) Item 90 Q is sat.
69 F
2-4 B
(21-1 CNS NRC)
KrchBm S
G2.4.3 (T3) Item 92 Q is sat.
70 F
1 x
B (GFE)
M (2023 TP NRC)
E S
192006K1.13 (T4) Item 95
[Did you misinterpret my GFE examples that I sent a few weeks ago?]
Cred. Dist.: A and D are not plausible due to the interplay between the first and second parts. Its extremely basic to know that xenon is a poison, especially at the end of the program.
Suggest using the following (modified from Turkey Point 2023):
Following a reactor trip from 100% power:
Shutdown margin 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> after reactor trip will be (HIGHER THAN /
THE SAME AS) shutdown margin immediately following the reactor trip - AND - Xe-135 concentration 20 hours2.314815e-4 days <br />0.00556 hours <br />3.306878e-5 weeks <br />7.61e-6 months <br /> after reactor trip will be (HIGHER / LOWER) than the Xe-135 concentration immediately following the reactor trip.
Question rated as E only due to being on a GFE exam.
Suggestion incorporated. MAB 71 F
1 B
(GFE)
N E
S 192007K1.05 (T4)
LOD: unacceptably low for this stage in the program.
Q rated as E only due to being on a GFE exam.
Suggest the following:
Reactor trip from 100% RTP and MOL equilibrium condition Crew performing ES-0.2 IAW ES-0.2, the minimum boration required prior to cooldown is:
A.
CSD Boron + 150 ppm, Xe credit allowed.
B.
CSD Boron + 150 ppm, Xe credit not allowed.
C.
CSD Boron + 150 pcm, Xe credit allowed D.
CSD Boron + 150 pcm, Xe credit not allowed Suggestion incorporated. Likely more of a F LOK. MAB
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level 72 F
1 B
(GFE)
E S
192008 (T4) Item 97 LOD unacceptably low for this stage in the program.
Q rated as E only due to being on a GFE exam.
Cred. Dist.; Distractors are not plausible.
Suggest the following:
IAW (Insert Procedure Name for Startup Procedure)
Critical data (is / is NOT) required to be taken as soon as indications are indicative of a critical reactor - AND - the highest (SR / IR) nuclear instrument reading is required to be recorded as part of the critical data. Suggestion partially accepted. Q now Sat.
MAB 73 F
2-4 N
S 193003K1.12 (T4) Item 98 Q is sat.
74 F
2-4 B
(GFE)
S 193004K1.11 (T4) Item 99 Q is sat.
75 H
2-4 N
S 193009K1.08 (T4) Item 100 Q is sat.
76 H
2-4 x
B (22-1 MNS Audit)
E S
007EA2.02 (T1/G1) Item 1 License Level: System knowledge can be used to eliminate C(1) and D(1). Suggest writing the first part as follows: Based on the conditions above, an ATWS (IS) / (IS NOT) in progress.
The second part is not SRO-only because it simply tests an action and the reason that an action is performed. Even ROs should know the reason they are taking certain actions.
Making the suggested change for the first part tests whether the student knows what an Anticipated Operational Occurrence is and whether the conditions presented in the stem represent an ATWS, which could be considered SRO-only level knowledge.
Comments incorporated. MAB 77 F
2-4 M
(20-1 MNS NRC)
S 025AA2.15 (T1/G1) Item 7 Q is sat.
78 H
2-4 M
(22-1 MNS NRC)
S 040G2.4.16 (T1/G1) Item 12 PRE-REVIEW Q Otherwise, this Q is sat.
79
(-)
H 2-4 x
B U
S 055G2.1.8 (T1/G1) Item 15 License Level: The SRO justification lacks an important component, which is procedure selection. The analysis of information must then be used to make a procedure selection.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level One possible solution would be to ask something related to E-Plan during a Station Blackout that involves activities outside the control room - such as with the emergency response organization. Initial suggestion did not relate to SBO. Replaced with acceptable alternative. Q now sat. MAB 80 F
2-4 x
M U
S 065AA2.12 (T1/G1) Item 20 Cueing / Cred. Dist.: The first part of the question tells the student that it is important to close the MSIVs before pressure gets too low
- that can be used for the second part - it is important to close the valves to meet its safety function.
Suggest: writing half of the question to provide a low accumulator pressure (but VI is still ok) and ask if the low accumulator pressure makes the MSIV inoperable.
Then for the second half ask: AP-22 (DOES) / (DOES NOT) provide direction to close MSIVs when pressure reaches a certain value.
The first part tests at the SRO level for operability of MSIVs when accumulator air pressure is low, thereby hitting the K/A at the SRO level.
Comments incorporated. MAB 81
(+)
H 2-4 B
(20-1R MNS NRC)
Lanyi S
077G2.1.20 (T1/G1) Item 21 Q is sat.
82
(-)
F 2-4 X
N U
S 059AA2.09 (T1/G2) Item 31 Cue: By providing the reference, no knowledge is being tested on the first part of the question. I was able to figure out the answer with no training - all I had to do was read the provided information.
Reference removed. MAB 83 H
2-4 N
S 074EA2.12 (T1/G2) Item 32 PRE-REVIEW Q Q is sat.
84 H
2-4 B
(17-1 CNS Audit)
S WE16G2.4.21 (T1/G2) Item 35 Q is sat.
85
(+)
H 2-4 N
S WE08 G2.1.1 (T1/G2) Item 36 Q is sat.
86 H
2-4 x
M (22-1 MNS NRC)
E S
025G2.2.25 (T2/G1) Item 51 Stem Focus: More precise wording is needed for the first part.
They are in, or have entered, a Tech Spec when thy meet the applicability (arguably). It is more precise to ask if the conditions of the LCO are met. Or perhaps whether actions of the LCO are
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level required. Also, it is not meaningful to have LCO 3.6.13 in every answer choice. It is more clearly written as: Actions of LCO 3.6.12 (ARE)/(ARE NOT) required. Incorporated. MAB 87 H
2-4 B
(2005 MNS NRC)
S 026G2.2.37 (T2/G1) Item 52 Q is sat.
88 H
2-4 B
(2009 MNS Audit)
S 062G2.2.36 (T2/G1) Item 52 Q is sat.
89 H
2-4 N
S 076A2.04 (T2/G1) Item 66 PRE-REVIEW Q Q is sat.
90
(+)
H 2-4 B
(20-1 MNS NRC)
Lanyi S
078A2.02 (T2/G1) Item 68 Q is sat.
91 H
2-4 B
(18-1 MNS NRC)
Bates S
001G2.4.12 (T2/G2) Item 72 Q is sat.
92 H
2-4 x
N E
S 014A2.02 (T2/G2) Item 72 PRE-REVIEW Q License Level: AOP entry symptoms are not met for AP-14. The second part of the question can be answered by only knowing AOP entry. The first part of the question can be answered using systems knowledge.
Q rated as E because it had previously been reviewed.
Suggest: Consider a Tech Spec application question beyond one hour actions. Or potentially an operability determination at the SRO level?
Comments incorporated into a new Q. MAB 93
(+)
H 2-4 B
(22-1 MNS NRC)
Bates S
017A2.02 (T2/G2) Item 74 PRE-REVIEW Q Q is sat.
94
(-)
(+)
F 2-4 B
(2016 MNS Audit)
S G2.1.42 (T3) Item 82 Q is sat 95 F
2-4 B
S G2.1.15 (T3) Item 83 Q is sat.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level
(20-1 MNS Audit) 96
(+)
F 2-4 M
(21-1 CNS NRC)
S G2.2.18 (T3) Item 88 Q is sat.
97
(+)
F 2-4 B
(22-1 MNS Audit)
S G2.2.5 (T3) Item 89 Q is sat.
98
(-)
(-)
H 2-4 M
(20-1R MNS NRC)
S G2.3.14 (T3) Item 91 Q is sat.
99
(+)
F 2-4 B
(20-1R MNS Audit)
S G2.4.26 (T3) Item 93 Q is sat 100 H
2-4 B
(2015 MNS Audit)
S G2.4.38 (T3) Item 94 Q is sat.
Form 2.3-5 Written Examination Review Worksheet Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. K/A Use Flaws
- 6.
Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job Link Minutia #/Units Logic Q-K/A License Level
Instructions:
Refer to ES-4.2 for the definitions of terms used in this worksheet for the written examination. Review each question (Q) as submitted and as subsequently revised and document the following in the associated worksheet columns:
- 1.
Enter the level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) from 1 (easy) to 5 (difficult); mark direct lookup questions (applicant can directly determine the answer from the provided reference) as LOD 1. A question is (U)nsatisfactory if it is LOD 1 or LOD 5.
- 3.
Check the appropriate box if a psychometric flaw is identified:
Stem Focus: The stem lacks enough focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information). This is an (U)nsatisfactory question.
Cues: The stem or one or more answer choices contains cues (e.g., clues, specific determiners, phrasing, length). This is an (U)nsatisfactory question.
T/F: All of the answer choices are a collection of unrelated true/false statements. This is an (U)nsatisfactory question.
Cred. Dist.: The distractors are not credible; single implausible distractors require (E)nhancement, and more than one noncredible distractor in the same question results in an (U)satisfactory question.
Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem). This is an (U)nsatisfactory question.
- 4.
Check the appropriate box if a job content flaw is identified:
Job Link: The question is not linked to the job requirements (i.e., the question has a valid knowledge or ability (K/A) but, as written, is not operational in content). This is an (U)nsatisfactory question.
Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory). This is an (U)nsatisfactory question.
- /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). This is an (U)nsatisfactory question.
Logic: The question requires backward or reverse logic or application compared to the job requirements. This is an (U)nsatisfactory question.
- 5.
Check the first box if a K/A mismatch flaw exists. Check the second box if the question is flawed because it is written at the wrong license level. Either condition results in an (U)nsatisfactory question.
- 6.
Enter the questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-4.2.
- 7.
Based on the review performed in steps 2-5, mark the question as (U)nsatisfactory, in need of (E)nhancement, or (S)atisfactory.
- 8.
Fully explain the reason for any (U) in column 7 (e.g., how the psychometric attributes are not being met).
Save the initial review comments and detail subsequent comment resolution so that each exam-bound question is marked by an (S) on this form.
LOK (F/H)
LOD SF Cue T/F Cred.
Dist.
Par Job Link Minutia