ML18270A089
| ML18270A089 | |
| Person / Time | |
|---|---|
| Issue date: | 09/24/2018 |
| From: | Patrick Raynaud NRC/RES/DE/CIB |
| To: | |
| Michael Benson | |
| Shared Package | |
| ML18270A088 | List: |
| References | |
| Download: ML18270A089 (10) | |
Text
WRS Validation Criteria Public Meeting Patrick Raynaud RES/DE/CIB 09/24/2018
WRS NUREG Proposal on Validation Metrics and Acceptance Criteria
- Proposed Quality Metrics
- Root mean square error on WRS: RMSEWRS
- Root mean square error on the 1st derivative of the WRS: RMSED1
- Average difference up to the initial crack depth of interest: diffavg
- Developed based upon flaw growth studies
- Based on average hardening law recommendation 1
= 2
=1 1
1
=
2 2
=2 0.1 1
=
0.1
=1 09/24/2018 WRS Public Meeting 2
WRS Validation Metrics Bases Circumferential Crack Growth (1/2)
- Acceptable circumferential flaw 1 Circumferential Flaw Growth growth: 0.9 0.8 Mean
- Time to leakage prediction 0.7 A 0.6 B
- Very large scatter a/t 0.5 C D
- Proposal: take the top 3 predictions 0.4 E
0.3 (top 50%) 0.2 F
G 0.1 0
0 100 200 300 400 500 600 700 time [months]
Participant RMSEWRS RMSED1 diffavg Crack Growth D 29 409 -10 Acceptable E 32 366 11 Acceptable G 52 512 14 Acceptable B 79 523 92 Not Acceptable A 59 374 -55 Not Acceptable C 28 300 -31 Not Acceptable F 46 624 -20 Not Acceptable 09/24/2018 WRS Public Meeting 3
WRS Validation Metrics Bases Circumferential Crack Growth (2/2)
- Which predictions are acceptable?
- Changes depending on the time considered for crack growth
- Explored 10 and 20 years of crack growth
- Acceptance criteria range need to be widened for 10 or 20 years of growth: 60 years allows for better discrimination between acceptable and unacceptable predictions 1
Circumferential Flaw Growth 1 Circumferential Flaw Growth 0.9 0.9 0.8 Mean 0.8 Mean 0.7 A 0.7 A 0.6 B 0.6 B a/t 0.5 C a/t 0.5 C
?
0.4 D 0.4 D E E 0.3 0.3 F F 0.2 0.2 G G 0.1 0.1 0 0 0 100 200 300 400 500 600 700 0 20 40 60 80 100 120 time [months] time [months]
09/24/2018 WRS Public Meeting 4
WRS Validation Metrics Bases Axial Crack Growth (1/2)
- Acceptable axial flaw growth: 1 Axial Flaw Growth
- Time to leakage prediction 0.9 0.8 Mean
- Very large scatter 0.7 A 0.6 B
- Proposal: take the top 3 predictions a/t 0.5 C (top 50%) 0.4 D E
0.3 F
0.2 G
0.1 0
0 100 200 300 400 500 600 700 time [months]
Participant RMSEWRS RMSED1 diffavg Crack Growth D 49 417 47 Acceptable E 26 330 13 Acceptable G 68 525 63 Acceptable B 74 717 114 Not Acceptable A 51 524 -37 Not Acceptable C 34 331 -56 Not Acceptable F 50 591 -30 Not Acceptable 09/24/2018 WRS Public Meeting 5
WRS Validation Metrics Bases Axial Crack Growth (2/2)
- Which predictions are acceptable?
- Changes depending on the time considered for crack growth
- Explored 10 and 20 years of crack growth
- Acceptance criteria range need to be widened for 10 or 20 years of growth: 60 years allows for better discrimination between acceptable and unacceptable predictions 1
Axial Flaw Growth 1 Axial Flaw Growth 0.9 0.9 0.8 Mean 0.8 Mean 0.7 A 0.7 A 0.6 B 0.6 B
?
a/t 0.5 C a/t 0.5 C 0.4 D 0.4 D E E 0.3 0.3 F F 0.2 0.2 G G 0.1 0.1 0 0 0 100 200 300 400 500 600 700 0 20 40 60 80 100 120 time [months] time [months]
09/24/2018 WRS Public Meeting 6
Other Metrics Investigated RMSE on Second Derivative
- Looking at second derivative as a test for concavity/convexity
- Did not improve differentiation between good and bad predictions 2
1 1
= 2 = 2 +2 2 + 2 4 4
=3 D2 Axial Stress / Circ Flaw Growth Participant RMSEWRS RMSED1 RMSED2 diffavg Crack Growth Axial WRS D 29 409 8130 -10 Acceptable Metric Acceptance E 32 366 6458 11 Acceptable RMSEWRS 55 G 52 512 10025 14 Acceptable RMSED1 520 B 79 523 9207 92 Not Acceptable RMSED2 10050 A 59 374 5882 -55 Not Acceptable diffavg 15 C 28 300 6414 -31 Not Acceptable diffavg -15 F 46 624 12475 -20 Not Acceptable D2 Hoop Stress / Axial Flaw Growth Participant RMSEWRS RMSED1 RMSED2 diffavg Time to leakage Hoop WRS D 49 417 8569 47 Acceptable Metric Acceptance E 26 330 5583 13 Acceptable RMSEWRS 70 G 68 525 10951 63 Acceptable RMSED1 550 B 74 717 14028 114 Not Acceptable RMSED2 11000 A 51 524 15332 -37 Not Acceptable diffavg 0 C 34 331 7421 -56 Not Acceptable diffavg 65 F 50 591 13245 -30 Not Acceptable 09/24/2018 WRS Public Meeting 7
Other Metrics Investigated Truncated RMSE
- Calculate RMSE up to x/t = T
- Calculation performed for T=0.1, 0.2, , 0.9
- Generally no improvement observed 1 1
= 2
=
2 1
=1 =2 T=0.6 Axial Stress / Circ Flaw Growth Truncated Participant (RMSEWRS)T (RMSED1)T diffavg Crack Growth Axial WRS D 32 407 -10 Acceptable Metric Acceptance E 35 444 11 Acceptable (RMSEWRS)T 60 G 57 526 14 Acceptable (RMSED1)T 550 B 67 386 92 Not Acceptable diffavg 15 A 58 403 -55 Not Acceptable diffavg -15 C 27 286 -31 Not Acceptable F 47 748 -20 Not Acceptable T=0.6 Hoop Stress / Axial Flaw Growth Truncated Participant (RMSEWRS)T (RMSED1)T diffavg Time to leakage Hoop WRS D 56 438 47 Acceptable Metric Acceptance E 27 361 13 Acceptable (RMSEWRS)T 75 G 73 518 63 Acceptable (RMSED1)T 525 B 84 661 114 Not Acceptable diffavg 0 A 59 514 -37 Not Acceptable diffavg 65 C 35 385 -56 Not Acceptable F 63 718 -30 Not Acceptable 09/24/2018 WRS Public Meeting 8
Other Metrics Investigated Weighted RMSE
- Apply a weight function on the error to give more importance to low x/t values
- Calculation performed for W=1, 2, 5, 10
- Acceptance criteria range for RMSEWRSW and RMSED1W can be narrowed
- BUT: acceptance criteria on diffavg controls which predictions are acceptable or not
- Not worth the additional computational effort 1
1 1
= 1 2
= 1 2
2
=1 =2 W=5 Axial Stress / Circ Flaw Growth Weighted Participant (RMSEWRS)W (RMSED1)W diffavg Crack Growth Axial WRS D 10 134 -10 Acceptable Metric Acceptance E 16 256 11 Acceptable (RMSEWRS)W 20 G 20 212 14 Acceptable (RMSED1)W 260 B 33 156 92 Not Acceptable diffavg 15 A 25 98 -55 Not Acceptable diffavg -15 C 12 77 -31 Not Acceptable F 24 439 -20 Not Acceptable W=5 Hoop Stress / Axial Flaw Growth Weighted Participant (RMSEWRS)W (RMSED1)W diffavg Time to leakage Hoop WRS D 18 164 47 Acceptable Metric Acceptance E 12 169 13 Acceptable (RMSEWRS)W 35 G 35 270 63 Acceptable (RMSED1)W 275 B 37 269 114 Not Acceptable diffavg 0 A 19 178 -37 Not Acceptable diffavg 65 C 19 103 -56 Not Acceptable F 30 434 -30 Not Acceptable 09/24/2018 WRS Public Meeting 9
Summary
- Acceptance criteria for WRS validation metrics based on top 3 predictions in Phase 2b round Robin project
- Investigated alternative metrics
- Second derivative
- Truncated RMSE
- Weighted RMSE
- No significant improvements could be achieved over metrics proposed in draft NUREG
- In most cases, diffave alone was sufficient to differentiate between good and bad predictions
- But probably not a good idea to use a single metric: may not work for very different WRS profiles 09/24/2018 WRS Public Meeting 10