ML24026A165

NRC Comments on Draft Op Test and Written: Forms 2.3-3 and 2.3-5

ML24026A165
Person / Time
Site:	Beaver Valley
Issue date:	01/10/2023
From:	Energy Harbor Nuclear Generation LLC
To:	Todd Fish NRC/RGN-I/DORS/OB
References
EPID L-2023-OLL-0011
Download: ML24026A165 (1)

Text

Facility: Beaver Valley Unit 1 Exam Date: 10/2/23 1

JPM # or title 2

Type (S/P/A) 3 ALT (Y/N) 4 LOD (1-5) 5 JPM Errors 6

U/E/S 7

Explanation LOD REF IC TSK CUE CS TL S1: 1CR-089 S

N 3

E Enhanced task standard tolerance band: 100 gallons +/- 5 gallons of primary makeup water has been added to the RCS at a rate of 50 GPM

+/- 5 GPM.

S2: 1CR-505 S

Y 4

S S3: 1CR-659 S

Y 2

E Enhanced task standard: Accumulator A is OPERABLE as defined by no low pressure or high-level annunciator lit.

S4: 1CR-690 S

Y 2

E Enhanced task standard: RCS bleed and feed is initiated via the Reactor head vent valves due to PORV, PCV-1RC-455D, failure to open.

S5: 1CR-661 S

Y 3

E Enhanced task standard tolerance band: PRT temperature between X and Y, PRT level between A and B, with PRT pressure > 0.5 PSIG.

S6: 1CR-660 S

Y 2

S S7: 1CR-662 S

Y 2

E Designated step 10 critical.

S8: 1CR-155 S

N 2

E Enhanced task standard: CCR discharge header pressure > 100 PSIG as observed on PI-1CC-100.

P1: 1PL-074 P

N 3

S P2: 1PL-153 P

N 2

S P3: 1PL-178 P

N 3

X U

LOD = 1. Wrote new JPM (in SF6).

RO A1.1: 1AD-002 A

N 3

X U

No procedure guidance for task. Selected new JPM.

RO A1.2: 1AD-055 A

N 2

E Enhanced task standard tolerance band:

Power band +/- 0.1%.

RO A2: 1AD-049 A

N 2

S

Form 2.3 -3 Operating Test Review Worksheet (JPMs)

Facility: Beaver Valley Unit 1 Exam Date: 10/2/23 RO A3: 3AD-025 A

N 3

E Enhanced task standard tolerance band: Dose received acceptable range 23 mrem to 24 mrem.

SRO A1.1: 1AD-048 A

N 3

S SRO A1.2: 1AD-019 A

N 2

S SRO A2: 1AD-039 A

N 2

S SRO A3: 1AD-038 A

N 3

S SRO A4: 1AD-046 A

N 2

S

Form 2.3-3 Instructions for Completing the JPM Table

1. Enter the JPM number and/or title.

2. Enter the type of JPM(S)imulator, (P)lant, or (A)dministrative.

3. Enter (Y)es or (N)o for an Alternate Path JPM.

4. Rate the level of difficulty (LOD) of each JPM using a scale of 1-5 (easy-difficult). A JPM containing less than two critical steps, a JPM that tests solely for recall or memorization, or a JPM that involves directly looking up a single correct answer is likely LOD = 1 (too easy).

Conversely, a JPM with over 30 steps or a JPM that takes more than 45 minutes to complete is likely LOD = 5 (too difficult).

5. Check the appropriate block for each JPM error type, using the following criteria:

LOD = 1 or 5 is unsatisfactory (U).

REF: The JPM lacks required references, tools, or procedures (U).

IC: The JPM initial conditions are missing or the JPM lacks an adequate initial cue (U).

CUE: The JPM lacks adequate evaluator cues to allow the applicant to complete the task, or the evaluator cues are subjective or leading (U).

TSK: The JPM lacks a task standard or lacks completion criteria for a task standard (U).

CS: The JPM contains errors in designating critical steps, or the JPM lacks an adequate performance standard for a critical step (U).

TL: The JPM validation times are unreasonable, or a time-critical JPM lacks a completion time (U).

6. Mark the JPM as unsatisfactory (U), satisfactory (S), or needs enhancements (E). A JPM is (U) if it has one or more (U) errors as determined in step 5. Examples of enhancements include formatting, spelling, or other minor changes.

7. Briefly describe any JPM determined to be unsatisfactory (U) or needing enhancement (E). Save initial review comments and detail subsequent comment resolution so that each exam-bound JPM is marked by a satisfactory (S) resolution on this form.

Form 2.3-3 Operating Test Review Worksheet (Scenarios)

Facility: Beaver Valley Unit 1 Scenario: 1 Exam Date: 10/2/23 1

Scenario Event ID/Name:

2 Scenario event errors 3

U/E/S 4

Explanation Realism/

Credibility Performance Standards Verifiable Actions Critical Task TS 1

S 2

S 3

S 4

E DNB TS may not be applicable, depending on crews response to this event. Added new TS-related event (RWST Level Transmitter).

5 S

6 S

7 S

8 S

Form 2.3-3 Operating Test Review Worksheet (Scenarios)

Facility: Beaver Valley Unit 1 Scenario: 2 Exam Date: 10/2/23 1

Scenario Event ID/Name:

2 Scenario event errors 3

U/E/S 4

Explanation Realism/

Credibility Performance Standards Verifiable Actions Critical Task TS 1

S 2

S 3

S 4

S 5

S 6

S 7

S 8

S 9

S 10 X

E This event deleted because there is no BV procedure that directs MSL isolation in response to steam dump valve failure to close.

Form 2.3-3 Operating Test Review Worksheet (Scenarios)

Facility: Beaver Valley Unit 1 Scenario: 3 Exam Date: 10/2/23 1

Scenario Event ID/Name:

2 Scenario event errors 3

U/E/S 4

Explanation Realism/

Credibility Performance Standards Verifiable Actions Critical Task TS 1

S 2

S 3

E Event not needed (Event 5 and 6 meet requirements for TS-related events).

4 S

5 S

6 S

7 S

8 X

U CT associated with this event not valid: Low power starting point of scenario does not generate enough decay heat energy to ever challenge Subcriticality and RCS Integrity. (CT-12 deleted) 9 S

10 S

Form 2.3-3 Operating Test Review Worksheet (Scenarios)

Facility: Beaver Valley Unit 1 Scenario: 4 Exam Date: 10/2/23 1

Scenario Event ID/Name:

2 Scenario event errors 3

U/E/S 4

Explanation Realism/

Credibility Performance Standards Verifiable Actions Critical Task TS 1

S 2

S 3

S 4

E Added additional component malfunction (C) for ATC: Control rods fail to insert in AUTO.

5 S

6 S

7 S

8 S

9 S

10 S

Form 2.3-3 Instructions for Completing the Scenario Table

1. For each scenario, enter the scenario event names and descriptions.

2. Review the individual events contained in each scenario, and identify and mark event errors:

The scenario guide event description is not realistic/credibleunsatisfactory (U).

The scenario guide event description lacks adequate crew/operator performance standardsneeds enhancement (E).

The scenario guide event description lacks verifiable actions for a credited normal event, reactivity event instrument/component malfunction, or technical specification (TS) event (or a combination of these) (U).

The scenario guide event description incorrectly designates an event as a critical task (i.e., a noncritical task labeled as critical or a critical task labeled as noncritical). This includes critical tasks that do not meet the critical task criteria (i.e., the critical task does not have a measurable performance standard) (U).

The scenario guide event description incorrectly designates entry into TS actions when not required or does not designate entry into TS actions when required (U).

3. Based on the outcome in step 2, mark the scenario event as unsatisfactory (U), satisfactory (S), or needs enhancements (E). An event is (U) if it has one or more (U) errors as determined in step 2. Examples of enhancements include formatting, spelling, or other minor changes.

4. Briefly describe any scenario event determined to be unsatisfactory (U) or needing enhancement (E). Save initial review comments and detail 8subsequent comment resolution so that each exam-bound scenario event is marked by a satisfactory (S) resolution on this form.

Form 2.3-5 Written Examination Review Worksheet - BV1 - 2023 Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 1

F 2.0 Y

RO N

E Provided hole size and leak location, per BVPS Executive Volume, RCP Trip/Restart, generic information document, to better define that the accident is a small break LOCA.

2 H

3.0 Y

RO B

S 3

F 3.0 Y

RO B

S 4

H 3.0 Y

RO B

S 5

H 3.0 Y

RO N

S 6

H 2.0 Y

RO N

S 7

H 3.0 Y

RO N

S 8

F 2.8 Y

RO B

S 9

F 3.0 Y

RO N

S 10 H

2.8 Y

RO N

S 11 F

2.0 Y

RO N

S 12 H

3.0 Y

RO B

S 13 H

2.0 Y

RO N

S 14 F

2.9 Y

RO N

S 15 F

2.3 X

Y RO B

E Distractor explanation for C is opposite of that given in the answer choices. Fixed distractor.

16 F

3.0 Y

RO B

S

Form 2.3-5 Instructions for Written Examination Review Worksheet Refer to ES-4.2 for the definitions of terms used in this worksheet for the written examination. Review each question (Q) as submitted and as subsequently revised and document the following in the associated worksheet columns:

1. Enter the level of knowledge (LOK) as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) from 1 (easy) to 5 (difficult); mark direct lookup questions (applicant can directly determine the answer from the provided reference) as LOD 1. A question is (U)nsatisfactory if it is LOD 1 or LOD 5.

3. Check the appropriate box if a psychometric flaw is identified:

Stem Focus: The stem lacks enough focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information). This is an (U)nsatisfactory question.

Cues: The stem or one or more answer choices contains cues (e.g., clues, specific determiners, phrasing, length). This is an (U)nsatisfactory question.

T/F: All of the answer choices are a collection of unrelated true/false statements. This is an (U)nsatisfactory question.

Cred. Dist.: The distractors are not credible; single implausible distractors require (E)nhancement, and more than one noncredible distractor in the same question results in an (U)satisfactory question.

Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem). This is an (U)nsatisfactory question.

4. Check the appropriate box if a job content flaw is identified:

Job Link: The question is not linked to the job requirements (i.e., the question has a valid knowledge or ability (K/A) but as written, is not operational in content). This is an (U)nsatisfactory question.

Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory). This is an (U)nsatisfactory question.

1. /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). This is an (U)nsatisfactory question.

Logic: The question requires backward or reverse logic or application compared to the job requirements. This is an (U)nsatisfactory question.

5. Check the first box if a K/A mismatch flaw exists. Check the second box if the question is flawed because it is written at the wrong license level. Condition results in an (U)nsatisfactory question.

6. Enter the questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-4.2.

7. Based on the review performed in steps 2-5, mark the question as (U)nsatisfactory, in need of (E)nhancement, or (S)atisfactory.

8. Fully explain the reason for any (U) in column 7 (e.g., how the psychometric attributes are not being met).

9. Save the initial review comments and detail subsequent comment resolution so that each exam-bound question is marked by an (S) on this form.

Form 2.3-5 Written Examination Review Worksheet Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 17 F

3.8 Y

RO N

S 18 H

4.0 Y

RO N

S 19 F

3.5 Y

RO N

S 20 H

3.3 Y

RO N

S 21 F

3.0 X

Y RO B

E AOP-1.6.4 no longer specifically called out as a transition out of ES-0.1. AOP-1.6.4 removed from answer A.

22 F

2.3 Y

RO N

S 23 H

2.0 Y

RO N

S 24 F

2.0 Y

RO N

S 25 H

3.4 Y

RO N

S 26 H

2.0 Y

RO B

S 27 H

3.0 Y

RO B

S 28 F

2.3 Y

RO B

S 29 H

4.0 Y

RO N

S 30 H

2.9 Y

RO N

S 31 L

4.0 Y

RO N

S 32 H

3.9 Y

RO B

S

Form 2.3-5 Written Examination Review Worksheet Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 33 H

3.3 Y

RO N

S 34 H

2.3 Y

RO B

S 35 H

3.0 Y

RO B

S 36 H

4.0 Y

RO N

S 37 H

3.3 Y

RO B

S 38 H

3.0 Y

RO N

S 39 H

2.0 Y

RO N

S 40 F

3.0 Y

RO N

S 41 H

3.0 Y

RO N

S 42 F

2.0 Y

RO N

S 43 F

3.0 Y

RO N

S 44 F

2.0 Y

RO N

S 45 H

2.1 Y

RO B

S 46 H

2.9 Y

RO N

S 47 F

3.0 Y

RO N

S 48 F

2.0 Y

RO N

S

Form 2.3-5 Written Examination Review Worksheet Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 49 F

2.0 Y

RO B

S 50 H

2.5 Y

RO N

S 51 H

4.0 Y

RO N

S 52 F

3.3 Y

RO N

S 53 F

3.0 Y

RO N

S 54 H

2.8 Y

RO N

U Significant overlap with Question 32. Randomly selected new K/A and wrote new Q.

55 F

3.5 Y

RO N

S 56 H

3.0 Y

RO N

S 57 H

3.0 Y

RO B

S 58 F

3.0 X

X Y

RO B

U ROs are trained on fuel handling equipment, but do not operate fuel handling equipment. Randomly selected new K/A and wrote new Q.

59 H

3.0 Y

RO B

S 60 F

3.3 Y

RO N

S 61 F

3.0 Y

RO N

S 62 H

3.0 Y

RO N

S 63 H

2.0 Y

RO B

S 64 F

4.0 Y

RO B

S

Form 2.3-5 Written Examination Review Worksheet Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 65 H

3.0 Y

RO B

U Not valid Tier 3, too plant specific. Randomly selected K/A and wrote new Q.

66 F

2.0 Y

RO B

S 67 F

2.0 Y

RO B

S 68 F

3.0 Y

RO B

S 69 F

2.0 Y

RO N

S 70 F

2.0 Y

RO B

S 71 F

3.0 Y

RO N

S 72 F

3.0 Y

RO B

S 73 H

3.0 Y

RO B

S 74 H

3.0 Y

RO B

S 75 F

3.0 Y

RO B

S

Form 2.3-5 Written Examination Review Worksheet Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 76 H

3.0 Y

SRO N

S 77 H

3.0 Y

SRO N

S 78 F

4.0 Y

SRO B

S 79 H

3.0 Y

SRO B

S 80 H

2.7 Y

SRO B

S 81 F

3.0 Y

SRO N

S 82 H

4.0 Y

SRO N

S 83 H

3.0 Y

SRO B

S 84 H

4.0 Y

SRO N

S 85 F

3.3 Y

SRO N

S 86 F

3.0 Y

SRO N

S 87 F

3.0 N

SRO N

U Not SRO Only. Question uses above the line information to answer, therefore RO knowledge. Keep K/A and wrote new Q.

88 H

3.2 X

X Y

SRO B

U Similar knowledge required to answer Q76 - SI Termination Criteria. Both Q76 and Q88 are vapor space leaks, too much overlap. Randomly selected new K/A and wrote new Q.

89 F

3.0 Y

SRO N

S 90 F

3.0 Y

SRO N

S 91 H

1.0 N

SRO B

U LOD = 1.0, not SRO Only. Question can be answered using RO knowledge. Randomly selected new K/A and wrote new Q.

Form 2.3-5 Written Examination Review Worksheet Q#

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. K/A Use Flaws

6. Source (B/M/N)

7. Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist.

Partial Job Link Minutia

1. /Units Logic Q-K/A License Level 92 H

3.0 Y

SRO B

S 93 H

4.0 Y

SRO N

S 94 F

3.0 Y

SRO B

S 95 H

1.0 N

SRO B

U LOD = 1.0 (Reference provided DLU, shows limits not met).Not SRO Only. Not Tier 3 Generic. Randomly selected new K/A and wrote new Q.

96 F

2.0 Y

SRO B

S 97 H

2.0 Y

SRO B

S 98 H

3.3 Y

SRO B

S 99 H

3.0 Y

SRO B

U Not Tier 3 Generic. Randomly selected new K/A and wrote new Q.

100 H

2.0 Y

SRO B

U Not Tier 3 Generic. Randomly selected new K/A and wrote new Q.