ML18270A089
| ML18270A089 | |
| Person / Time | |
|---|---|
| Issue date: | 09/24/2018 |
| From: | Patrick Raynaud NRC/RES/DE/CIB |
| To: | |
| Michael Benson | |
| Shared Package | |
| ML18270A088 | List: |
| References | |
| Download: ML18270A089 (10) | |
Text
WRS Validation Criteria Public Meeting Patrick Raynaud RES/DE/CIB 09/24/2018
WRS NUREG Proposal on Validation Metrics and Acceptance Criteria
- Proposed Quality Metrics
- Root mean square error on WRS: RMSEWRS
- Root mean square error on the 1st derivative of the WRS: RMSED1
- Average difference up to the initial crack depth of interest: diffavg
- Developed based upon flaw growth studies
- Based on average hardening law recommendation 09/24/2018 WRS Public Meeting 2
=
1
=1
2
=
1 2
=2 1
2
=
1 0.1
=1 0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0 100 200 300 400 500 600 700 a/t time [months]
Circumferential Flaw Growth Mean A
B C
D E
F G
WRS Validation Metrics Bases Circumferential Crack Growth (1/2)
- Acceptable circumferential flaw growth:
- Time to leakage prediction
- Very large scatter
- Proposal: take the top 3 predictions (top 50%)
Participant RMSEWRS RMSED1 diffavg Crack Growth D
29 409
-10 Acceptable E
32 366 11 Acceptable G
52 512 14 Acceptable B
79 523 92 Not Acceptable A
59 374
-55 Not Acceptable C
28 300
-31 Not Acceptable F
46 624
-20 Not Acceptable 09/24/2018 WRS Public Meeting 3
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0 20 40 60 80 100 120 a/t time [months]
Circumferential Flaw Growth Mean A
B C
D E
F G
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0 100 200 300 400 500 600 700 a/t time [months]
Circumferential Flaw Growth Mean A
B C
D E
F G
WRS Validation Metrics Bases Circumferential Crack Growth (2/2) 09/24/2018 WRS Public Meeting 4
- Which predictions are acceptable?
- Changes depending on the time considered for crack growth
- Explored 10 and 20 years of crack growth
- Acceptance criteria range need to be widened for 10 or 20 years of growth: 60 years allows for better discrimination between acceptable and unacceptable predictions
?
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0 100 200 300 400 500 600 700 a/t time [months]
Axial Flaw Growth Mean A
B C
D E
F G
WRS Validation Metrics Bases Axial Crack Growth (1/2)
- Acceptable axial flaw growth:
- Time to leakage prediction
- Very large scatter
- Proposal: take the top 3 predictions (top 50%)
Participant RMSEWRS RMSED1 diffavg Crack Growth D
49 417 47 Acceptable E
26 330 13 Acceptable G
68 525 63 Acceptable B
74 717 114 Not Acceptable A
51 524
-37 Not Acceptable C
34 331
-56 Not Acceptable F
50 591
-30 Not Acceptable 09/24/2018 WRS Public Meeting 5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0 20 40 60 80 100 120 a/t time [months]
Axial Flaw Growth Mean A
B C
D E
F G
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0 100 200 300 400 500 600 700 a/t time [months]
Axial Flaw Growth Mean A
B C
D E
F G
WRS Validation Metrics Bases Axial Crack Growth (2/2)
- Which predictions are acceptable?
- Changes depending on the time considered for crack growth
- Explored 10 and 20 years of crack growth
- Acceptance criteria range need to be widened for 10 or 20 years of growth: 60 years allows for better discrimination between acceptable and unacceptable predictions 09/24/2018 WRS Public Meeting 6
?
Other Metrics Investigated RMSE on Second Derivative Looking at second derivative as a test for concavity/convexity Did not improve differentiation between good and bad predictions 09/24/2018 WRS Public Meeting 7
D2 Axial Stress / Circ Flaw Growth Participant RMSEWRS RMSED1 RMSED2 diffavg Crack Growth Axial WRS D
29 409 8130
-10Acceptable Metric Acceptance E
32 366 6458 11Acceptable RMSEWRS
55 G
52 512 10025 14Acceptable RMSED1
520 B
79 523 9207 92Not Acceptable RMSED2
10050 A
59 374 5882
-55Not Acceptable diffavg
15 C
28 300 6414
-31Not Acceptable diffavg
-15 F
46 624 12475
-20Not Acceptable D2 Hoop Stress / Axial Flaw Growth Participant RMSEWRS RMSED1 RMSED2 diffavg Time to leakage Hoop WRS D
49 417 8569 47Acceptable Metric Acceptance E
26 330 5583 13Acceptable RMSEWRS
70 G
68 525 10951 63Acceptable RMSED1
550 B
74 717 14028 114Not Acceptable RMSED2
11000 A
51 524 15332
-37Not Acceptable diffavg
0 C
34 331 7421
-56Not Acceptable diffavg
65 F
50 591 13245
-30Not Acceptable
=
1 4
=3 2
2
=
1 42 +2 2 + 2
Other Metrics Investigated Truncated RMSE Calculate RMSE up to x/t = T Calculation performed for T=0.1, 0.2,, 0.9 Generally no improvement observed 09/24/2018 WRS Public Meeting 8
T=0.6 Axial Stress / Circ Flaw Growth Truncated Participant (RMSEWRS)T (RMSED1)T diffavg Crack Growth Axial WRS D
32 407
-10Acceptable Metric Acceptance E
35 444 11Acceptable (RMSEWRS)T
60 G
57 526 14Acceptable (RMSED1)T
550 B
67 386 92Not Acceptable diffavg
15 A
58 403
-55Not Acceptable diffavg
-15 C
27 286
-31Not Acceptable F
47 748
-20Not Acceptable T=0.6 Hoop Stress / Axial Flaw Growth Truncated Participant (RMSEWRS)T (RMSED1)T diffavg Time to leakage Hoop WRS D
56 438 47Acceptable Metric Acceptance E
27 361 13Acceptable (RMSEWRS)T
75 G
73 518 63Acceptable (RMSED1)T
525 B
84 661 114Not Acceptable diffavg
0 A
59 514
-37Not Acceptable diffavg
65 C
35 385
-56Not Acceptable F
63 718
-30Not Acceptable
=
1
=1
2
=
1 1
=2
2
Other Metrics Investigated Weighted RMSE Apply a weight function on the error to give more importance to low x/t values
- Calculation performed for W=1, 2, 5, 10 Acceptance criteria range for RMSEWRS W and RMSED1 W can be narrowed
- BUT: acceptance criteria on diffavg controls which predictions are acceptable or not
- Not worth the additional computational effort 09/24/2018 WRS Public Meeting 9
W=5 Axial Stress / Circ Flaw Growth Weighted Participant (RMSEWRS)W (RMSED1)W diffavg Crack Growth Axial WRS D
10 134
-10Acceptable Metric Acceptance E
16 256 11Acceptable (RMSEWRS)W
20 G
20 212 14Acceptable (RMSED1)W
260 B
33 156 92Not Acceptable diffavg
15 A
25 98
-55Not Acceptable diffavg
-15 C
12 77
-31Not Acceptable F
24 439
-20Not Acceptable W=5 Hoop Stress / Axial Flaw Growth Weighted Participant (RMSEWRS)W (RMSED1)W diffavg Time to leakage Hoop WRS D
18 164 47Acceptable Metric Acceptance E
12 169 13Acceptable (RMSEWRS)W
35 G
35 270 63Acceptable (RMSED1)W
275 B
37 269 114Not Acceptable diffavg
0 A
19 178
-37Not Acceptable diffavg
65 C
19 103
-56Not Acceptable F
30 434
-30Not Acceptable
=
1
=1
1
2
=
1 2
=2 1
1
2
Summary Acceptance criteria for WRS validation metrics based on top 3 predictions in Phase 2b round Robin project Investigated alternative metrics
- Second derivative
- Truncated RMSE
- Weighted RMSE No significant improvements could be achieved over metrics proposed in draft NUREG In most cases, diffave alone was sufficient to differentiate between good and bad predictions
- But probably not a good idea to use a single metric: may not work for very different WRS profiles 09/24/2018 WRS Public Meeting 10