Text

Cw 2019-03 ES-301-7 Op Test Comments with Resolutions
	ML19099A375
Person / Time
Site:	Callaway
Issue date:	03/13/2019
From:	Greg Werner; Operations Branch IV
To:	; AmerenUE
References
	50-483/19-03 50-483/OL-19
	Download: ML19099A375 (29)
	v • d • e

Note: This is a header page to separate the draft and final resolved versions of the ES-401-9 (Draft Written Exam Comments Form) and the ES-301-7 (Draft Operating test Comments Form) for the Adams package. The first file for both the draft written and draft operating test forms contains only the draft comments with stats for the as submitted draft materials. This header page separates the draft comments from the Final comment files that contains final resolutions of those items for both of these forms.

ES-301 Operating Test Review Worksheet Form ES-301-7 Rev. 11 Facility: Callaway Plant Exam Date: March 4, 2019 1

2 3

Attributes 4

Job Content 5

6 Admin JPMs ADMIN Topic and K/A LOD (1-5)

U/E/S Explanation I/C Cues Critical Scope Overlap Perf.

Key Minutia Job Link Focus Steps (N/B)

Std.

General comments:

1.

JPM standards should include a statement that all critical steps are completed satisfactorily.

2.

Sometimes the JPM step, standard, and notes sections are confusing. The step is always the actual step from the procedure, the std is what you expect them to do, what they see happen, and any transition piece (if required for alt path). Communications between applicant and CRS, field operators, etc, should be in the notes or in separate lines between the steps of the JPM.

3.

All SRO applicant class composition

4.

Admin JPMs that have a KEY should be included for each admin JPM and correctly marked as KEY for A4, as an example.

RA1 N/A-all SRO applicant class composition RA2 N/A-all SRO applicant class composition RA3 N/A-all SRO applicant class composition RA4 N/A-all SRO applicant class composition A1 (SRO)

Determine if core alterations can begin (2.1.40) 2 X

E S

Communications bullet should be less obvious than this (moved up into the group).

Minor issue - spelling error for Operable on cue sheet (two locations, cut and paste).

Task standard needs to be updated.

Licensee made requested repairs and JPM is now Sat.

A2 (SRO)

Determine Reportability (2.1.18) 2 S

A3 (SRO)

Review work scheduled and determine TS and risk

( 2.2.17) 3 S

A4 (SRO)

Review release permit and 3

X X

U Task standard does not match critical steps.

The three errors listed in the task standard are different than the 3 errors in JPM

ES-301 Operating Test Review Worksheet Form ES-301-7 Rev. 11 determine ODCM limits (2.3.6) standard step 1. Need to reconcile this before validation week. Also need to have A4 Key written at the top of the key.

A5 (SRO)

Determine EAL (2.4.41) 3 S

Need to have A5 Key written at the top of the key.

1 2

LOD (1-5) 3a I/C 3b Cues 3c Critical 3d Scope 3e Overlap 3f Perf.

3h Key 4a Minutia 4b Job Link 5

U/E/S 6

Explanation Simulator/In-Plant Safety Function and K/A JPMs General comments for all JPMS

1.

Task standards all need work. You should have a synopsis of the JPM in one or two sentences. For example, Upon completion of this JPM, the applicant will have isolated air to AB-PV-1 and AB-PV-4, which closes AB-PV-1 but not AB-PV-4. The applicant must then close AB-PV-4 by closing the manual isolation valve AB-0007 in order to close the two failed open dump valves.

2.

Use applicant in the task standard throughout the exams instead of operator or candidate

3.

At the end of each task standard it needs to state that.and completed all critical steps correctly for the legal tie.

S1 1

3 X

X E

Notice for the outline that the SRO-U cannot perform P2 because it is in SF1 and so is S1, and S1 is the only low power JPM in your set. which means you have to have the SRO-U perform one additional sim JPM and not perform P2. An easy fix is for the Upgrades to perform S7 (SF9),

which is new and alt path, which is also needed to give the upgrades 3 total alt path JPMs.

Task standard needs work-it should state something like: During rod withdrawal for startup, rod N-7 drops into the core, requiring the applicant to insert negative reactivity by either inserting rods (how much though?), by tripping the reactor, or by starting an emergency boration (what is the standard for this ie which valves and how much). The

ES-301 Operating Test Review Worksheet Form ES-301-7 Rev. 11 task standard should also state that all critical steps in the procedure must be successfully completed.

For step 5.2.7.a (JPM step 5) it should be If the applicant attempts to adjust MFP or its settings, then cue that..

Question-on step 5.2.7.a (step 6 of JPM) is there a standard for rod withdrawal in small increments? How much is too much?

For JPM step 18, what is the standard for inserting rods, std for tripping reactor, and std for beginning emergency boration? Switches need to be included in the steps here also.

S2 2

3 X

X E

Task standard needs work-it should state something like: When swapping to B CCP, its shaft shears, requiring the applicant to restore charging with either the NCP by doing xyz or by starting the A CCP. The task standard should also state that all critical steps in the procedure must be successfully completed.

It appears that there are two distinct paths for success: 1) the NCP, which is already running but at reduced flow, can be adjusted somehow, or 2) start the A CCP which was in standby. It also should be part of the task standard and a critical step to secure the B CCP since it is damaged. The JPM should have these clear paths for success. Step 24 of the JPM states that the performance std for the NCP is shown above but I dont see it.

S3 3

3 X

U S

Unsat as submitted. The task std appears to be for an alt path JPM which this JPM is not.

To be operationally valid where would the level be if it went just below the TS value or is this after maintenance where the level got much lower than normal operating band?

Licensee put it in initial conditions that maintenance was completed. Also modified the JPM task standard and it is now Sat.

S4 4P 3

X E

Task std edits (Upon completion of this JPM, the applicant borated the A RHR train in preparation for placing it in service and completed all critical steps correctly.

S5 5

3 X

E Task std edits (Upon completion of this JPM, the applicant placed the A hydrogen analyzer in service and completed all critical steps correctly.

S6 7

3 X

X E

Task standard needs values or a range of values based on JPM setup. Use applicant in the std, not operator. Put complete all critical steps correctly in it also.

ES-301 Operating Test Review Worksheet Form ES-301-7 Rev. 11 Step 17 is confusing because it cant be critical if the step states to put it in the required position as directed by CRS and then the CRS is not going to tell him what position to put it in. If he tells him what to put it in then it would be cueing of the critical step, so I would have CRS tell him to put it back in auto and have the step be not critical.

S7 9

3 X

E Task std needs work. Since it is alt path, it should be applicant is responding to high ARM annunciator 62B when it is noticed that CRVIS should have actuated and did not, the applicant actuates one train of CRVIS and completes all critical steps correctly.

S8 N/A N/A since there are no RO applicants P1 4S 3

X X

E Task standard needs work. See example above in general comments for it. Also, is providing Area 5 by direction in the cue a cue to the applicant on the location (ie is it directed in the procedure this way)? If it is not directed this way in the procedure then shouldnt the applicant figure out where to go to force the valves closed?

Use applicant in the task standard throughout the exams instead of operator and completes all critical steps correctly.

P2 1

3 E

Change from candidate to applicant in JPM content (Task std) and completes all critical steps correctly.

P3 2

2 E

Add to task std and completes all critical steps correctly.

ES-301 5

Form ES-301-7 Rev. 11 Instructions for Completing This Table:

Check or mark any item(s) requiring a comment and explain the issue in the space provided using the guide below.

1.

Check each JPM for appropriate administrative topic requirements (COO, EC, Rad, and EP) or safety function requirements and corresponding K/A. Mark in column 1.

(ES-301, D.3 and D.4)

2.

Determine the level of difficulty (LOD) using an established 1-5 rating scale. Levels 1 and 5 represent an inappropriate (low or high) discriminatory level for the license that is being tested. Mark in column 2 (Appendix D, C.1.f)

3.

In column 3, Attributes, check the appropriate box when an attribute is not met:

The initial conditions and/or initiating cue is clear to ensure the operator understands the task and how to begin. (Appendix C, B.4)

The JPM contains appropriate cues that clearly indicate when they should be provided to the examinee. Cues are objective and not leading. (Appendix C, D.1)

All critical steps (elements) are properly identified.

The scope of the task is not too narrow (N) or too broad (B).

Excessive overlap does not occur with other parts of the operating test or written examination. (ES-301, D.1.a, and ES-301, D.2.a)

The task performance standard clearly describes the expected outcome (i.e., end state). Each performance step identifies a standard for successful completion of the step.

A valid marked up key was provided (e.g., graph interpretation, initialed steps for handouts).

4.

For column 4, Job Content, check the appropriate box if the job content flaw does not meet the following elements:

Topics are linked to the job content (e.g., not a disguised task, task required in real job).

The JPM has meaningful performance requirements that will provide a legitimate basis for evaluating the applicant's understanding and ability to safely operate the plant. (ES-301, D.2.c)

5.

Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)nhancement, or (S)atisfactory? Mark the answer in column 5.

6.

In column 6, provide a brief description of any (U)nacceptable or (E)nhancement rating from column 5.

Save initial review comments and detail subsequent comment resolution so that each exam-bound JPM is marked by a (S)atisfactory resolution on this form.

ES-301 6

Form ES-301-7 Rev. 11 Facility: Callaway Plant General comments on scenarios Exam Date: March 4, 2019 1

2 3

4 5

6 7

8 9

10 Event Realism/Cred.

Required Actions Verifiable actions LOD TS CTs Scen.

Overlap U/E/S Explanation General Comments for all scenarios

1.

The Initial conditions need to have BOC/MOC/EOC on them. If they were all snapped from one time in core life, then we have to change the snaps and the scenario guides in order to meet the NUREG. Scenario 2 should be at BOC since it is a startup/low power.

2.

Events need to have the noun descriptors at the top of each page for all four scenarios.

3.

Highlight EOP/FRS procedures in Gray when first used and for transitions. You have a separate line item for them which is great but using a table shading property to color the entire line in gray makes it easier to see the transitions for the examiner.

4.

Measurable Performance indicators for CT table should have the actual switches manipulated for success of the CT. As an example, lets say that a NSSS Group isolation fails to isolate AFW as expected in scenario x, so the applicant must manually close the inbd and outbd isolation valves for AFW. Both of these valves should be listed in the CT table.

5.

For parameters to record for grading purposes, we will need to work thru the required parameters to capture for scenarios for grading during validation week.

6.

Need the generic statement for post scenario CTs added to the CT table. It should be placed at the bottom of the CT table for all scenarios and state:

NOTE: (Per NUREG-1021, Appendix D) If an operator or the Crew significantly deviates from or fails to follow procedures that affect the maintenance of basic safety functions, those actions may form the basis of a CT identified in the post-scenario review.

ES-301 7

Form ES-301-7 Rev. 11 Facility: Callaway Plant Scenario: 1 Exam Date: March 4, 2019 1

2 3

4 5

6 7

8 9

10 Event Realism/Cred.

Required Actions Verifiable actions LOD TS CTs Scen.

Overlap U/E/S Explanation 1

2 S

2 3

S 3

3 E

Words are confusing for DNB TS entry (page 15 near the bottom of page). If pressure goes below 2195# then the TS is required to be entered.

4 3

S 5

3 E

Typo page 23 near top for alarms (alarm unervoltage is undervoltage).

Secondly, the note just below this alarm list should state that all condensate pumps have tripped not all condensate trips have tripped.

6 3

3 E

On page 24 you have to list the equipment that must be manipulated to emergency borate (You have it in the CT table I believe) or if they use control rods (same thing). Put CT-52 under event description on the D1 form.

7 3

E Put CT-6 and CT-16 under the event description on the D1 form.

8 3

S The B sequencer malfunction is a separate malfunction and is counted as an 8th event in the attributes table, but it is acceptable to place it with others if they are triggered at the same time after the major. This note is mainly for accounting purposes with the D-1 form, the attributes table, and the counting form ES-301-7.

ES-301 8

Form ES-301-7 Rev. 11 Facility: Callaway Plant Scenario: 2 Exam Date: March 4, 2019 1

2 3

4 5

6 7

8 9

10 Event Realism/Cred.

Required Actions Verifiable actions LOD TS CTs Scen.

Overlap U/E/S Explanation 1

3 S

Total page count is off (it is several pages more than 34 for the total).

2 3

S 3

3 S

4 3

S 5

3 S

6 3

S Place CT-2 below event description on D1.

7 3

E Place CT-5 below event description on D1. Place a note for CT-16 that it may not be counted towards the minimum CT count if it is run after another scenario is run that contains this same CT, depending on order and selection of scenarios used during exam week. It is still a CT, however.

ES-301 9

Form ES-301-7 Rev. 11 Facility: Callaway Plant Scenario: 3 Exam Date: March 4, 2019 1

2 3

4 5

6 7

8 9

10 Event Realism/Cred.

Required Actions Verifiable actions LOD TS CTs Scen.

Overlap U/E/S Explanation 1

3 S

2 3

S 3

3 S

4 3

S Place CT-1 below event description on D1.

5 3

E Place CT-17 below event description on D1. Place a note for CT-16 that it may not be counted towards the minimum CT count if it is run after another scenario is run that contains this CT, depending on the order and selection of scenarios used during exam week. It is still a CT, however.

6 3

S D SG fault-not counted as an event but it is counted as a malfunction.

7 3

S Failure of automatic steam line isolation-not counted as an event but it is counted as a malfunction.

8 S

Failure of D MSIV to close with fast close PB-not counted as an event but it is counted as a malfunction.

ES-301 10 Form ES-301-7 Rev. 11 Facility: Callaway Plant Scenario: 4 Exam Date: March 4, 2019 1

2 3

4 5

6 7

8 9

10 Event Realism/Cred.

Required Actions Verifiable actions LOD TS CTs Scen.

Overlap U/E/S Explanation 1

3 S

2 2

S 3

3 S

4 3

S 5

2 S

Place CT-24 below event description on D1.

6 3

E Place CT-23 below event description on D1. Make sure this is not just a failure to start, that it includes some valve manipulations also, such as driving down and resetting the trip throttle valve, or manually opening the steam admission valves.

7 2

S

ES-301 11 Form ES-301-7 Rev. 11 Instructions for Completing This Table:

1 Use this table for each scenario for evaluation.

2 Check this box if the events are not related (e.g., seismic event followed by a pipe rupture) OR if the events do not obey the laws of physics and thermodynamics.

3, 4 In columns 3 and 4, check the box if there is no verifiable or required action, as applicable. Examples of required actions are as follows: (ES-301, D.5f) opening, closing, and throttling valves starting and stopping equipment raising and lowering level, flow, and pressure making decisions and giving directions acknowledging or verifying key alarms and automatic actions (Uncomplicated events that require no operator action beyond this should not be included on the operating test unless they are necessary to set the stage for subsequent events. (Appendix D, B.3).)

5 Check this box if the level of difficulty is not appropriate.

6 Check this box if the event has a TS.

7 Check this box if the event has a critical task (CT). If the same CT covers more than one event, check the event where the CT started only.

8 Check this box if the event overlaps with another event on any of the last two NRC examinations. (Appendix D, C.1.f) 9 Based on the reviewers judgment, is the event as written (U)nacceptable (requiring repair or replacement), in need of (E)nhancement, or (S)atisfactory? Mark the answer in column 9.

10 Record any explanations of the events here.

In the shaded boxes, sum the number of check marks in each column.

In column 1, sum the number of events.

In columns 2-4, record the total number of check marks for each column.

In column 5, based on the reviewer's judgement, place a checkmark only if the scenario's LOD is not appropriate.

In column 6, TS are required to be 2 for each scenario. (ES-301, D.5.d)

In column 7, preidentified CTs should be 2 for each scenario. (Appendix D; ES-301, D.5.d; ES-301-4)

In column 8, record the number of events not used on the two previous NRC initial licensing exams. A scenario is considered unsatisfactory if there is < 2 new events. (ES-301, D.5.b; Appendix D, C.1.f)

In column 9, record whether the scenario as written (U)nacceptable, in need of (E)nhancement, or (S)atisfactory from column 11 of the simulator scenario table.

ES-301 12 Form ES-301-7 Rev. 11 Facility: Callaway Plant Exam Date: March 4, 2019 Scenario 1

2 3

4 5

6 7

8 11 Event Totals Events Unsat.

TS Total TS Unsat.

CT Total CT Unsat.

% Unsat.

Scenario Elements U/E/S Explanation 1

8 0

2 0

3 0

0 E

See above comments for edits 2

8 0

2 0

0 E

See above comments for edits 3

8 0

2 0

3 0

0 E

See above comments for edits 4

7 0

4 0

2 0

0 E

See above comments for edits Instructions for Completing This Table:

Check or mark any item(s) requiring comment and explain the issue in the space provided.

1, 3, 5 For each simulator scenario, enter the total number of events (column 1), TS entries/actions (column 3), and CTs (column 5).

This number should match the respective scenario from the event-based scenario tables (the sum from columns 1, 6, and 7, respectively).

2, 4, 6 For each simulator scenario, evaluate each event, TS, and CT as (S)atisfactory, (E)nhance, or (U)nsatisfactory based on the following criteria:

a.

Events. Each event is described on a Form ES-D-2, including all switch manipulations, pertinent alarms, and verifiable actions. Event actions are balanced between at-the-controls and balance-of-plant applicants during the scenario. All event-related attributes on Form ES-301-4 are met. Enter the total number of unsatisfactory events in column 2.

b.

TS. A scenario includes at least two TS entries/actions across at least two different events. TS entries and actions are detailed on Form ES-D-2. Enter the total number of unsatisfactory TS entries/actions in column 4. (ES-301, D.5d)

c.

CT. Check that a scenario includes at least two preidentified CTs. This criterion is a target quantitative attribute, not an absolute minimum requirement. Check that each CT is explicitly bounded on Form ES-D-2 with measurable performance standards (see Appendix D). Enter the total number of unsatisfactory CTs in column 6.

7 In column 7, calculate the percentage of unsatisfactory scenario elements:

8 If the value in column 7 is > 20%, mark the scenario as (U)nsatisfactory in column 8. If column 7 is 20%, annotate with (E)nhancement or (S)atisfactory.

9 In column 9, explain each unsatisfactory event, TS, and CT. Editorial comments can also be added here.

Save initial review comments and detail subsequent comment resolution so that each exam-bound scenario is marked by a (S)atisfactory resolution on this form.

2 + 4 + 6 1 + 3 + 5100%

ES-301 13 Form ES-301-7 Rev. 11 Site name: Callaway Plant Exam Date: Exam Date: March 4, 2019 OPERATING TEST TOTALS Total Total Unsat.

Total Total Unsat.

Explanation Edits Sat.

Admin.

JPMs 5

1 2

2 Sim./In-Plant JPMs 10 1

9 0

Scenarios 4

0 4

0 Op. Test Totals:

19 2

15 2

10.5%

Satisfactory submittal.

Instructions for Completing This Table:

Update data for this table from quality reviews and totals in the previous tables and then calculate the percentage of total items that are unsatisfactory and give an explanation in the space provided.

1.

Enter the total number of items submitted for the operating test in the Total column. For example, if nine administrative JPMs were submitted, enter 9 in the Total items column for administrative JPMs.

For scenarios, enter the total number of simulator scenarios.

2.

Enter the total number of (U)nsatisfactory JPMs and scenarios from the two JPMs column 5 and simulator scenarios column 8 in the previous tables. Provide an explanation in the space provided.

3.

Enter totals for (E)nhancements needed and (S)atisfactory JPMs and scenarios from the previous tables. This task is for tracking only.

4.

Total each column and enter the amounts in the Op. Test Totals row.

5.

Calculate the percentage of the operating test that is (U)nsatisfactory (Op. Test Total Unsat.)/(Op. Test Total) and place this value in the bolded % Unsat. cell.

Refer to ES-501, E.3.a, to rate the overall operating test as follows:

satisfactory, if the Op. Test Total % Unsat. is 20%

unsatisfactory, if Op. Test Total % Unsat. is > 20%

6.

Update this table and the tables above with post-exam changes if the as-administered operating test required content changes, including the following:

The JPM performance standards were incorrect.

The administrative JPM tasks/keys were incorrect.

CTs were incorrect in the scenarios (not including postscenario critical tasks defined in Appendix D).

The EOP strategy was incorrect in a scenario(s).

TS entries/actions were determined to be incorrect in a scenario(s).

Note: This is a header page to separate the draft and final resolved versions of the ES-401-9 (Draft Written Exam Comments Form) and the ES-301-7 (Draft Operating test Comments Form) for the Adams package. The first file for both the draft written and draft operating test forms contains only the draft comments with stats for the as submitted draft materials. This header page separates the draft comments from the Final comment files that contains final resolutions of those items for both of these forms.

ES-301 Operating Test Review Worksheet Form ES-301-7 Rev. 11 Facility: Callaway Plant Exam Date: March 4, 2019 1

2 3

Attributes 4

Job Content 5

6 Admin JPMs ADMIN Topic and K/A LOD (1-5)

U/E/S Explanation I/C Cues Critical Scope Overlap Perf.

Key Minutia Job Link Focus Steps (N/B)

Std.

General comments:

1.

JPM standards should include a statement that all critical steps are completed satisfactorily.

2.

Sometimes the JPM step, standard, and notes sections are confusing. The step is always the actual step from the procedure, the std is what you expect them to do, what they see happen, and any transition piece (if required for alt path). Communications between applicant and CRS, field operators, etc, should be in the notes or in separate lines between the steps of the JPM.

3.

All SRO applicant class composition

4.

Admin JPMs that have a KEY should be included for each admin JPM and correctly marked as KEY for A4, as an example.

RA1 N/A-all SRO applicant class composition RA2 N/A-all SRO applicant class composition RA3 N/A-all SRO applicant class composition RA4 N/A-all SRO applicant class composition A1 (SRO)

Determine if core alterations can begin (2.1.40) 2 X

E S

Edit - Communications bullet should be less obvious than this (moved up into the group).

Minor issue - spelling error for Operable on cue sheet (two locations, cut and paste).

Task standard needs to be updated.

Licensee made requested repairs and JPM is now Sat.

A2 (SRO)

Determine Reportability (2.1.18) 2 S

A3 (SRO)

Review work scheduled and determine TS and risk

( 2.2.17) 3 S

A4 (SRO)

Review release permit and 3

X X

U Unsat - Task standard does not match critical steps. The three errors listed in the task standard are different than the 3 errors in

ES-301 Operating Test Review Worksheet Form ES-301-7 Rev. 11 determine ODCM limits (2.3.6)

S JPM standard step 1. Need to reconcile this before validation week. Also need to have A4 Key written at the top of the key.

Licensee made requested repairs and JPM is now Sat.

A5 (SRO)

Determine EAL (2.4.41) 3 S

Need to have A5 Key written at the top of the key.

1 2

LOD (1-5) 3a I/C 3b Cues 3c Critical 3d Scope 3e Overlap 3f Perf.

3h Key 4a Minutia 4b Job Link 5

U/E/S 6

Explanation Simulator/In-Plant Safety Function and K/A JPMs General comments for all JPMS

1.

Task standards all need work. You should have a synopsis of the JPM in one or two sentences. For example, Upon completion of this JPM, the applicant will have isolated air to AB-PV-1 and AB-PV-4, which closes AB-PV-1 but not AB-PV-4. The applicant must then close AB-PV-4 by closing the manual isolation valve AB-0007 in order to close the two failed open dump valves.

2.

Use applicant in the task standard throughout the exams instead of operator or candidate

3.

At the end of each task standard it needs to state that.and completed all critical steps correctly for the legal tie.

S1 1

3 X

X E

Notice for the outline that the SRO-U cannot perform P2 because it is in SF1 and so is S1, and S1 is the only low power JPM in your set. which means you have to have the SRO-U perform one additional sim JPM and not perform P2. An easy fix is for the Upgrades to perform S7 (SF9),

which is new and alt path, which is also needed to give the upgrades 3 total alt path JPMs.

Task standard needs work-it should state something like: During rod withdrawal for startup, rod N-7 drops into the core, requiring the applicant to insert negative reactivity by either inserting rods (how much though?), by tripping the reactor, or by starting an

ES-301 Operating Test Review Worksheet Form ES-301-7 Rev. 11 S

emergency boration (what is the standard for this ie which valves and how much). The task standard should also state that all critical steps in the procedure must be successfully completed.

For step 5.2.7.a (JPM step 5) it should be If the applicant attempts to adjust MFP or its settings, then cue that..

Question-on step 5.2.7.a (step 6 of JPM) is there a standard for rod withdrawal in small increments? How much is too much?

For JPM step 18, what is the standard for inserting rods, std for tripping reactor, and std for beginning emergency boration? Switches need to be included in the steps here also.