ML24207A085

Neural Network Investigation on Regression Modeling in Critical Heat Flux Prediction Applications

ML24207A085
Person / Time
Issue date:	07/31/2024
From:	Valiaveedu A NRC/RES/DSA/AAB
To:
Anthony Valiaveedu
Shared Package
ML24213A041	List: ML24207A085
References
Download: ML24207A085 (51)

Text

Anthony R. Valiaveedu Reactor Systems Engineer (Data Scientist)

RES/DSA/AAB Summer 2024 Neural Network Investigation on Regression Modeling in Critical Heat Flux Prediction Applications

Valiaveedu i

This report does not contain or imply legally binding requirements. Nor does this report establish or modify any regulatory guidance or positions of the U.S. Nuclear Regulatory Commission and is not binding on the Commission.

Valiaveedu ii Abstract This paper summarizes the Informal Assistance Request between the Office of Nuclear Reactor Regulation and the Office of Nuclear Regulatory Research on Critical Heat Flux predictions using Machine Learning algorithms.

Recent performance breakthroughs in artificial intelligence (AI) and machine learning (ML),

including advances in deep learning (DL) and the availability of powerful, easy-to-use ML toolboxes, have led to unprecedented interest in AI and ML among nuclear engineers.

Nonetheless, the extensive capabilities of AI and ML remain largely untapped within the realm of scientific computing in nuclear engineering. One formidable hurdle in harnessing their power is the frequent mismatch between existing ML methodologies and the specific demands of nuclear engineering applications and their extensive validation requirements. To enable more trustworthy applications in high-consequence systems like nuclear reactors that are subject to nuclear safety regulations, the ML practitioners have to address several critical issues, including the verification, validation and uncertainty quantification (VVUQ) of AI and ML, data scarcity, scaling-induced uncertainty, and lack of physics in black-box ML models.

The purpose of this exercise is to support development of a VVUQ methodology for ML models by comparing various ML models trained on Critical Heat Flux data. The benefit for the Office of Nuclear Reactor Regulation would be taking advantage of a large-scale study on how different ML models could be developed for CHF, supporting NRC goals in better understanding how AI can be used in nuclear applications. This would support the Chair's tasking on researching uses of AI for agency activities, specifically focusing on the application of ML models for reactor safety analysis.

Acknowledgements This report was developed with significant contributions, expert input, and guidance from Joshau Kaizer and Nazilla Tehrani at the U.S. NRC.

Valiaveedu iii Table of Contents Abstract......................................................................................................................................... ii Table of Contents......................................................................................................................... iii Table of Figures/Tables/Equations............................................................................................... iv 1 Introduction................................................................................................................................ 1 1.1 Purpose............................................................................................................................... 1 1.2 TF-AIML-2........................................................................................................................... 1 1.3 Critical Heat Flux (CHF)...................................................................................................... 1 2 Methodology.............................................................................................................................. 3 2.1 Data..................................................................................................................................... 3 2.2 Neural Network.................................................................................................................... 6 3 Parameters................................................................................................................................ 8 3.1 Overview............................................................................................................................. 8 3.2 Layers.................................................................................................................................. 8 3.3 Neurons............................................................................................................................. 11 3.4 Activation........................................................................................................................... 14 3.5 Optimizers......................................................................................................................... 17 3.6 Learning Rates.................................................................................................................. 20 4 Model....................................................................................................................................... 23 4.1 Ensemble.......................................................................................................................... 23 4.2 Algorithmic vs Neural Network.......................................................................................... 25 4.3 Transfer Learning.............................................................................................................. 28 5 Summary and Conclusion........................................................................................................ 30 6 Bibliography............................................................................................................................. 31

Valiaveedu iv Table of Figures/Tables/Equations Figure 1: OECD Data.................................................................................................................... 3 Figure 2: Clean LUT Data............................................................................................................. 4 Figure 3: OECD Pairplot............................................................................................................... 5 Figure 4: Clean LUT Pairplot........................................................................................................ 6 Figure 5: Layer Loss Plots............................................................................................................ 9 Figure 6: LUT Test Predictions.................................................................................................... 10 Figure 7: OECD Predictions........................................................................................................ 11 Figure 8: Node Loss Plots........................................................................................................... 12 Figure 9: Node LUT Plot............................................................................................................. 13 Figure 10: Node OECD Plot........................................................................................................ 13 Figure 11: Activation Loss Plots.................................................................................................. 15 Figure 12: Activation LUT Plots................................................................................................... 16 Figure 13: Activation OECD Plots............................................................................................... 17 Figure 14: Optimizer Loss Plots.................................................................................................. 18 Figure 15: Optimizers LUT Plots................................................................................................. 19 Figure 16: Optimizer OECD Plots............................................................................................... 19 Figure 17: Learning Rate Loss Plots........................................................................................... 20 Figure 18: LR LUT Plots............................................................................................................. 21 Figure 19: LR OECD Plots.......................................................................................................... 22 Figure 20: 124 Ensemble Overlay Plot....................................................................................... 23 Figure 21: Ensemble Std Heat Plot............................................................................................. 24 Figure 22: Ensemble Std Histogram........................................................................................... 24 Figure 23: CHF Predictions Variations........................................................................................ 25 Figure 24: Model Comparison LUT Data.................................................................................... 26 Figure 25: Model Comparison OECD Data................................................................................. 27 Figure 26: Neural Network OECD Error Map.............................................................................. 27 Figure 27: Polynomial Fit OECD Error Map................................................................................ 28 Figure 28: Same Set................................................................................................................... 28 Figure 29: Transfer Set............................................................................................................... 29 Table 1: Node OECD Error......................................................................................................... 14 Table 2: Activation OECD Error................................................................................................... 17 Table 3: Optimizer OECD Errors................................................................................................. 20 Table 4: LR OECD Errors............................................................................................................ 22 Table 5: Model Comparison OECD Errors.................................................................................. 27 Table 6: Transfer Learning Errors............................................................................................... 29 Equation 1: Error........................................................................................................................... 6

Valiaveedu 1

1 Introduction 1.1 Purpose On October 2023, the Chair issued a tasking order on researching uses of Artificial Intelligence (AI) for the uses of agency activities internally for licensing and oversight processes (Hanson, 2023). This supports and builds off the NRCs AI Strategic Plan and Project Plan to prepare the Agency for evaluating AI proposals (USNRC, 2023). Specifically, under Goal 5 Pursue Use-Cases to Build an AI Foundation, Task 5.4 includes facilitating AI Regulatory Research including the use of program office work requests (USNRC, 2023b). This paper summarizes the activities conducted in one such request from the Office of Nuclear Reactor Regulation (NRR).

Recent performance breakthroughs in AI and machine learning (ML), including advances in deep learning (DL) and the availability of powerful, easy-to-use ML toolboxes, have led to unprecedented interest in AI and ML among nuclear engineers. Nonetheless, the extensive capabilities of AI and ML remain largely untapped within the realm of scientific computing in nuclear engineering. One formidable hurdle in harnessing their power is the frequent mismatch between existing ML methodologies and the specific demands of nuclear engineering applications and their extensive validation requirements. To enable more trustworthy applications in high-consequence systems like nuclear reactors that are subject to nuclear safety regulations, the ML practitioners have to address several critical issues, including the verification, validation and uncertainty quantification (VVUQ) of AI and ML, data scarcity, scaling-induced uncertainty, and lack of physics in black-box ML models.

NRR issued an Informal Assistance Request for a Research Assistance Activity on the use of Machine Learning for complex physics prediction modeling. The purpose of the research is to support development of a VVUQ methodology for ML models by comparing various ML models trained on Critical Heat Flux (CHF) data. The benefit for the Office of Nuclear Reactor Regulation would be taking advantage of a large-scale study on how different ML models could be developed for CHF, supporting NRC goals in better understanding how AI can be used in nuclear applications.

1.2 TF-AIML-2 During the development of the request, the Nuclear Energy Agency (NEA) under the Organisation for Economic Cooperation and Development (OECD) began a task force on benchmarking uncertainty related to predicting CHF called TF-AIML-2 (NEA, 2024). Due to the nature of the research request and its close ties to the exercise conducted by NEA, the research activity was conducted in parallel to the exercise. Data used in this exercise was also used for the purposes of this research activity. The OECD data uses multiple inputs such as: 'Tube Diameter', 'Heated Length', 'Pressure', 'Mass Flux', 'Outlet Quality', 'Inlet Subcooling', and 'Inlet Temperature'. The uses of these variables will be discussed in Section 2.1 Data.

1.3 Critical Heat Flux (CHF)

Critical Heat Flux (CHF) is defined as the point for a fluid when boiling is no longer the most effective method of transferring heat. As such, the value is also known as the Critical Boiling Transition (CBT). Once the heat flux reaches the CHF, unstable vapor patches appear on solid surfaces which would reduce heat transfer and increase temperatures. This phenomenon is important for Light Water Reactors because it indicates a point of potential fuel cladding failure.

Valiaveedu 2

As fuel cladding acts as the first barrier to radioactive releases, its function is of high importance (USNRC, 2010).

There are various methods used to evaluate the CHF for fluids with more recent methods using reactor safety analysis codes. However, due to the large amount of CHF correlations and equations, a lookup table (LUT) was derived to provide accessibility. The specific LUT used for this research activity was the 2006 Groeneveld Lookup Tables that was derived using light water in tubular geometries. It is assumed that when using the LUT, that the tube geometry has a length-to-diameter ratio greater than 50. As such, the parameters used for the LUT are:

Pressure, Mass Flux, and Outlet Quality (USNRC, 2019).

Valiaveedu 3

2 Methodology 2.1 Data The research involves the use of two key data sets: the 2006 Groeneveld LUT and the OECD data table. In order to better provide a comparison, the primary training data used will be the 2006 LUT and then final testing will be conducted using the OECD data to ensure separation when evaluating results. The models used will be trained, validated, and tested on the LUT. The model will then be transferred to evaluate the OECD data.

Due to the LUT use of three input variables for estimating CHF (pressure, mass flux, and quality), the same three input variables were used for the OECD table. The omission of other variables ('Tube Diameter', 'Heated Length', 'Inlet Subcooling', 'Inlet Temperature' ) was determined to not result in a substantial error because the LUTs assumption (i.e., length-to-diameter greater than 50) is only inapplicable for 714 data points. The discrepancy would be only for 3% of the total OECD data table.

When reviewing the 2006 LUT data, there was a large number of 0 CHF values and a right skew in CHF values too. In order to reduce the skew and bias of the CHF values, the zero values were removed and the bottom 97.5 percentile was kept. To better visualize the data, Figure 1: OECD Data and Figure 2: LUT Data illustrates the cleaned datasets of the four variables used (Pressure, Mass Flux, Outlet Quality, and CHF). The total size of the data tables (after cleaning) were 24,579 CHF points for the OECD data and 10,767 CHF points for the LUT.

15% of the LUT data was isolated for testing the model fit before evaluating the model with the OECD data.

Figure 1: OECD Data

Valiaveedu 4

Figure 2: Clean LUT Data Figure 3 and Figure 4 illustrates the data distribution for both data sets. Using the plots, the LUT should encompass most, if not all, the data within the OECD. Understanding the data overlays provide reasonable assurance that the LUT would provide sufficient training to estimate the OECD data.

Valiaveedu 5

Figure 3: OECD Pairplot

Valiaveedu 6

Figure 4: Clean LUT Pairplot 2.2 Neural Network The purpose is to evaluate the use of ML algorithms to predict the output (i.e., CHF) based on the input parameters (Pressure, Mass Flux, and Outlet Quality). To develop the model, Python paired with TensorFlow was used to develop the model. To avoid any inherent biases within the model comparisons, batch sizes were kept the same, SkLearns TrainTestSplit was used to prepare training data with a split of 15% for testing data for shuffling, and the random shuffling was active within each TensorFlow model. Using this method of randomizing the training and validation data will ensure that each model maintains independence from each other and provide an avoidance of overfitting.

Error will be defined in the following manner consistent with how it is commonly defined in CHF.

Equation 1: Error

= /

Valiaveedu 7

This would mean that an ideal state would equal one. Measured values would be values from the data sources themselves, while Predicted values would be the outputs from the trained models.

Loss plots evaluate the error obtained for the training and validation sets during the model training. The trend of the loss plot functions (e.g., Mean Absolute Error, Mean Squared Error, etc.) demonstrates the impact the number of training sets have on the data. The use of loss plots are valuable to evaluate overfitting and underfitting as well. If the Machine Learning validation set error (data not directly used for training) is higher than the training set, overfitting is can be assumed because the model is predicting accurately on the training set but unable to predict accurately outside of the trained values. Therefore, actual value resulting from these plots are not as important compared to how the values are trending.

Valiaveedu 8

3 Parameters 3.1 Overview In general, there has been a lot of research conducted on neural networks and the different parameters behind their development and deployment. This section should reaffirm pre-existing understandings of the relationship of various parameters and their impact to neural network models. More parameters than those listed were evaluated, but the following provided results of interest.

3.2 Layers Neural networks operate via layers that define the relationship between variables. As such as the layers increase, it can result in more complex methodologies to appear. For the purposes of the paper, the outputs of one, three, and five layers will be shown. All the models are controlled in all other aspects and only a new, repetitive layer will be added to the last. Each layer will be 64 Dense using Exponential Linear Unit (ELU) activation and Adam optimizer. It is expected that as more layers are added, the model would increase in accuracy because of the complexity built due to the neurons.

The loss plots between the three models can provide insights into model fit and expectations.

From the outputs of the loss plots, the value of using more epochs (or number of times the model trains on a full set) is drastically reduced. The more significant reduction of epochs before convergence occurs between the use of 1 and 3 layers.

Valiaveedu 9

Figure 5: Layer Loss Plots

Valiaveedu 10 Figure 6: LUT Test Predictions For Figure 6: LUT Test Predictions (and subsequent figures), the red line indicates a perfect model where the prediction (models output) matches the measured values (source values). The test predictions of the three models shows an increasingly accurate model as the layers increase. There is a large change in regression accuracy as the layers increase from 1 to 3.

This correlates to the Figure 5s illustration of the loss plots. The R-squared value for the three models are as follows:

1 Layer: 0.7684896527604796 3 Layer: 0.9942369846748983 5 Layer: 0.99708468881886

Valiaveedu 11 Figure 7: OECD Predictions The three models were then extracted and transferred to predict the OECD data points. Figure 7: OECD Predictions illustrates the models predictions for evaluation. From the illustration, as data is transferred between sources, the error spread grows. However, there is a linear convergence as the number of layers increases, with 3 and 5 layers offering similar results. This correlates with the R-Squared values when testing the model with the LUT data. The R-Squared values for the OECD data are as follows:

1 Layer: 0.40775812932879174 3 Layer: 0.8839859128618166 5 Layer: 0.8945291117226943 3.3 Neurons This section will detail the exploration of neuron density and its impact on models. In simple terms, neurons provide the weight of different parameters to best predict the output value.

Neurons make up each layer.

The base model uses an activation layer twice (i.e., normalize/activation/activation/output) using ELU activation and Adam optimizer. Three densities will be analyzed: 16, 64, and 256. This is to proportionally visualize the changes brought about by increasing neural densities.

Valiaveedu 12 As illustrated in Figure 8: Node Loss Plots, as the density value increases, the training loss and validation loss become increasingly closer. Increasing the number of nodes, in addition, results in a lower number of epochs needed for convergence on the loss plots.

Figure 8: Node Loss Plots

Valiaveedu 13 Figure 9: Node LUT Plot When reviewing the LUT testing data, the results are expected. Unsurprisingly, as the number of nodes increases, the predictions become more accurate. This is seen with the stronger linear correlations between Measured and Predicted values as the nodes increase from 16 to 256 and can be the result of more relationships being considered from the nodal density. The R-Squared values are provided below:

16 Nodes: 0.9022610672858327 64 Nodes: 0.9714557616154169 256 Nodes: 0.9879755271744007 Figure 10: Node OECD Plot Once the models were transferred to predict the OECD data, the results showed that while increasing the nodes increases the accuracy of the predictions. It is interesting to note that the

Valiaveedu 14 neuron density plays a factor to the accuracy of predictions as the model is transferred to a new dataset. The R-Squared values for the models are listed below:

16 Node: 0.5477560322465455 64 Node: 0.8088427463636392 256 Node: 0.8607869222066755 However, given the overlays are very similar to each other, reviewing the error spread associated with each model may provide better insights in the value of each model.

Table 1: Node OECD Error Model Standard Deviation Mean Median 16 7.688231 4.532559 1.30903144 64 8.330266 1.142873 0.96033512 256 8.924479 0.986497 0.91555006 The error values analyzed yields better results for higher density models. This makes sense as more neurons would equate to more complex mapping. The higher neuron layer has a mean and median values close to 1 (ideal), though having a larger standard deviation.

3.4 Activation Though there are multiple activiation functions that could be used for regression modeling, the focus will be on three functions: Rectified Linear Unit (ReLU), Gaussian Error Linear Unit (GELU), and Exponential Linear Unit (ELU). The plots will use two 32 dense layers. These functions were choosen due to their high usages in regression modeling. Comparing these functions can provide insights in evaluating applications using these functions.

The loss plots in Figure 11 show a similar reduction in loss between the three functions as a function of epochs. The only noticeable difference, though slight, is that ELU seems to reach convergence faster than the other functions.

Valiaveedu 15 Figure 11: Activation Loss Plots

Valiaveedu 16 Figure 12: Activation LUT Plots When reviewing the activation functions testing of the LUT data, all of the functions have a relatively strong linear correlation with each other. Overall, the activations generally overpredict with the ELU test data illustrating a lowest spread at lower CHF values compared to the other two functions. However, all the activations underpredict from 12500-18000 kW/m2. These areas of underprediction are non-conservative and are areas of caution for the model. The R-Squared data is as follows:

ELU: 0.9517334878422764 ReLU: 0.9102661960144418 GELU: 0.9405377675590962

Valiaveedu 17 Figure 13: Activation OECD Plots When transferring to the OECD data, the use of ReLU seems to have less error in areas of sparse data (i.e., CHF values greater than 12000). However, the R-Squared and error value analysis can provide further evaluation of the models. Unlike with the training data, there is not a clear area of overprediction or underprediction isolated for any single model and all the models underpredict.

Table 2: Activation OECD Error Model R-Squared Error Standard Deviation Error Mean Error Median ELU 0.751 1.059701 1.392845 1.06797884 ReLU 0.709 1.353496 1.694068 1.22015365 GELU 0.753 1.064457 1.269712 1.08678513 As Table 2: Activation OECD Error tabulates, the error is seen to be reduced when using the ReLU function, with low deviation and error. The testing of GELU and ELU provides similar results in error and accuracy. Both have a deviation close to 1 and R-Squared value close to 0.75. Their errors are also slightly larger than the errors tabulated with ReLU.

3.5 Optimizers Optimizers are used by neural networks to modify the networks connections to better predict outputs. As the majority of Machine Learning systems have been using Adam (an adaptive optimization algorithm using gradient descent) as their optimizer, this section will review the use of Adam, its variant Nadam, and its predecessor RMSprop. As on the TensorFlow documentation, Much like Adam is essentially RMSprop with momentum, Nadam is Adam with Nesterov momentum (TensorFlow, 2024). The default values for the optimizers will be used from TensorFlow with a 2 layer, 32 dense, ELU activation architecture.

Valiaveedu 18 Of the three optimizers used, Nadam and Adam were very similar in loss plot trends which makes sense since only the momentum function changes between them. Of interest is that RMSprop requires more epochs before reaching its elbow than Adam or Nadam.

Figure 14: Optimizer Loss Plots

Valiaveedu 19 Figure 15: Optimizers LUT Plots All the optimizers seemed relatively equivalent when tested using the LUT data RMSprop and Adam seem to be doing equivalently in their testing using the LUT data. Though RMSprop has a slightly higher R-Squared value.

Adam: 0.9507456189344226 RMSprop: 0.9532212151431405 Nadam: 0.9499435230766322 Figure 16: Optimizer OECD Plots

Valiaveedu 20 Once visualizing the OECD data, the three plots show relatively similar predictions. However, there are some underpredictions by RMSprop in values between 5000-8000. The error values are tabulated below.

Table 3: Optimizer OECD Errors Model R-Squared Standard Deviation Mean Median Adam 0.7660851775491511 1.316201 1.417463 1.04284678 RMSprop 0.8109673975410624 0.537296 1.080060 0.95652084 Nadam 0.7533548345973788 1.069933 1.392657 1.0503894 RMSprop had a better overall score with low deviation, high R-Square value, and close to ideal centers of error. Adam and Nadam resulted in similar values, but Nadam had a lower deviation of error values than Adam.

3.6 Learning Rates Learning rates are used to tune models to quicken the speed at which a neural network is able to better predict. Since the default Adam learning rate is 0.001, the three tested for this section are: 0.001 with decay, 0.0013, and 0.0007. The default rate (0.001) will be used as a control point. The decay used is an inverse time decay with a rate of 0.5 with 71000 decay steps to illustrate the impact of learning rate decays. The models will have the same architecture as the Adam architecture used in the Optimizers section.

Figure 17: Learning Rate Loss Plots

Valiaveedu 21 Based on the figures above, the number of epochs prior to convergence decreases as learning rate increases. The decay feature similarly quickens convergence but not as drastically.

Figure 18: LR LUT Plots Testing on the LUT data, the lower rate of 0.0007 has a worse accuracy than the other rates and the default Adam rate. The R-Squared values also shows that the use of larger rates provides a better accuracy in predictions and the use of decay rates appears to have a marginal impact on the accuracy.

Default: 0.9538022610668709 Decay: 0.9506020261334185 0.0013: 0.9689414874888452 0.0007: 0.9101806190183691

Valiaveedu 22 Figure 19: LR OECD Plots Transferring the model provides to the OECD data does not yield clear visual results. The error data for the different rates have been collected and tabulated below.

Table 4: LR OECD Errors Model R-Squared Standard Deviation Mean Median 0.001 (Default) 0.7538680572906653 1.258447 1.470742 1.06111576 0.001 Decay 0.7554256883800715 1.055411 1.323028 1.0231251 0.0013 0.7832646549983692 2.819979 1.213216 0.97121523 0.0007 0.534957143809206 6.041080 3.914359 1.29688783 The data shows that using a decaying learning rate function can be beneficial when evaluating data outside the training data set. The decay model resulted in lower error and smaller deviation than the default model. In addition, the larger learning rate reduces the center of error and increases accuracy of predictions (per the R-Squared value), at the cost of a significantly larger deviation in error values. The lower learning rate yielded no benefit and had largest mean and median error values. The data indicates that values around 0.001 with a decay rate would be ideal for the model.

Valiaveedu 23 4 Model 4.1 Ensemble Neural networks are generally black box systems in which re-testing a model may yield different results. An analysis was conducted to review how much variation is evident when copying the model 124 times using the OECD data (otherwise called an ensemble). To do so, the same architecture was ran from scratch 124 times and the data was collected. The neural network used was a six layer network with ELU activation and a 15% dropout rate.

Figure 20: 124 Ensemble Overlay Plot illustrates the distribution of how the model predicts in different runs by using the data points within the top 95% and bottom 5%. The orange mark indicates the mean value from the 124 points. This proves there is variation in how the model predicts which can result in uncertainty in production level models. Especially, in the context of retesting further investigation should be pursued.

When mapping the standard deviation of the values in Figure 21 and Figure 22, the minimum standard deviation is 50 with larger deviations occurring in areas with larger training data sets.

This could make sense due to more variation in interpretation for the neural network to predict.

The highest standard deviation at a CHF point was noted as 422. In the area of outliers, the standard deviation was notably average or similar to other data points.

Figure 20: 124 Ensemble Overlay Plot

Valiaveedu 24 Figure 21: Ensemble Std Heat Plot Figure 22: Ensemble Std Histogram When reviewing the distribution of the predicted CHF values in Figure 23, the results show no clear distribution between CHF points as illustrated. Values were generally sporadic and no clear center metric would be preferable. Significant outliers were not seen to be common between CHF points, outside of the single point with a 422 standard deviation.

Valiaveedu 25 Figure 23: CHF Predictions Variations 4.2 Algorithmic vs Neural Network Traditional methods of CHF predictions use algorithms that are transparent in operations.

However, algorithms are bound to their governing functions. Neural Networks, however, are opaque in nature and are not subjected to the same limitations as algorithms. This understanding is best illustrated between using a 3rd-degree polynomial compared to a Neural Network. The training data set for the neural network will be used to develop the polynomial fit for the algorithm.

Based on Figure 27, both models provide a strong linear correlation between predicted and true values when using tested with the LUT. When reviewing the R-Squared values both models offer similar results as expected.

Valiaveedu 26 Figure 24: Model Comparison LUT Data Neural Network: 0.9453814142927025 Algorithm (labeled as LR): 0.9423566215834017 When transferring the model to the OECD data, similar results are visually seen in Figure 25, which would be expected with similar R-Squared values for the LUT. However, when reviewing the error data in Table 6: Model Comparison OECD Errors, the benefits of neural networks are clearer. There is a stronger R-Squared value and lower deviation of error, and lower error once the dataset changes. The error values were then heat mapped to be better analyzed in Figure 26 and Figure 27. In those figures, the Neural Network seems to struggle in an isolated region which could indicate a lack of training data. However, the algorithm is able to evaluate that same area with less error but fails to take into account practical conditions. For example, the algorithm predicts negative CHF values. The polynomial fit also results in large spikes in higher error values and is less consistent with its error regions.

Valiaveedu 27 Figure 25: Model Comparison OECD Data Table 5: Model Comparison OECD Errors Model R-Squared Standard Deviation Mean Median Neural Network 0.800628880482503 0.872860 1.222473 0.97553008 Polynomial Fit 0.7138727357417579 1.408887 0.840790 0.74340262 Figure 26: Neural Network OECD Error Map

Valiaveedu 28 Figure 27: Polynomial Fit OECD Error Map 4.3 Transfer Learning Transfer learning is the process of which a model is prepared and trained on a single dataset and then used on a separate dataset. Transfer learning was a fundamental part of this work, as the model was trained on the LUT and transferred to the OECD data. This approach could be common for many deployed, static neural network models and understanding its impact may provide insights for its use.

85% of the LUT data was used to train a model and 15% of the LUT data was set aside for the purposes of evaluation and is plotted in Figure 28: Same Set. The entirety of the OECD data was plotted using the same model in Figure 29: Transfer Set Figure 28: Same Set

Valiaveedu 29 Figure 29: Transfer Set What is evident, based on Figure 28 and Figure 29, is that prediction performance reduces once the model predicts on a different dataset. The exact error values are tabulated in Table 6.

Table 6: Transfer Learning Errors Model R-Squared Standard Deviation Mean Median Local Dataset 0.927 0.375 0.960 0.984 Transfer Dataset 0.786 0.753 1.121 0.947 Based on the error data, transfer learning does increase the error and inaccuracy of the model.

The R-Squared drops, the deviation of error becomes wider, and the central tendency of error becomes larger. This information can be useful when evaluating the transfer of models into operating systems.

Valiaveedu 30 5 Summary and Conclusion This work serves as a starting point for further investigations for VVUQ of neural networks.

Understanding how various parameters interact and impact neural networks would be important for the purposes of developers, deployers, and regulators ensuring safety of end use applications. As neural networks continue to be used in various applications, special attention should be taken into account in how small changes and processes can result in larger impacts to the overall prediction.

A sample code is attached in Appendix A to allow for further testing and evaluation.

Valiaveedu 31 6 Bibliography Hanson, C. T. (2023, October 30). Advancing Use of Artificial Intelligence at the U.S. Nuclear Regulatory Commission.

NEA. (2024). Benchmark on Artificial Intelligence and Machine Learning for Scientific Computing in Nuclear Engineering. Phase 1: Critical Heat Flux Exercise Specifications.

NEA Working Papers.

TensorFlow. (2024, June 7). Nadam. Retrieved from TensorFlow:

https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Nadam USNRC. (2010). Analysis and Computational Predictions of CHF Position and Post-CHF Heat Transfer (NUREG/IA-0236). USNRC.

USNRC. (2019). Critical Heat Flux Data Used to Generate the 2006 Groeneveld Lookup Tables (NUREG/KM-0011). USNRC.

USNRC. (2023). Artificial Intelligence Strategic Plan: Fiscal Years 2023-2027 (NUREG-2261).

USNRC.

USNRC. (2023b). Project Plan for the U.S. Nuclear Regulatory Commission Artifical Intelligence Strategic Plan Fiscal Years 2023-2027, Revision 0. USNRC.

Valiaveedu A-1 APPENDIX A EXAMPLE CODE USED Results may vary.

Initalizing Plot Functions In [27]:

from mpl_toolkits.mplot3d import Axes3D import matplotlib.pyplot as plt import matplotlib.cm as cm from matplotlib.colors import LogNorm from matplotlib import colormaps import numpy as np def chfplt(i, P, J, q, CHF):

fig1 = plt.figure(i) ax = fig1.add_subplot(111, projection = '3d')

color_map = colormaps['jet']

norm = LogNorm(vmin= np.amin(CHF), \\

vmax=np.amax(CHF))

fcolors = color_map(norm(CHF))

heat = cm.ScalarMappable(cmap=color_map, norm=norm) oecd_plt = ax.scatter(P, J, q, c = CHF, norm = norm, cmap = color_map) cbar_oecd = fig1.colorbar(oecd_plt, ax=ax, extend ='max')

cbar_oecd.set_label('CHF (kW/m^2)')

ax.set_xlabel(xlabel ='P (kPa)', fontsize = 'x-large')

ax.set_ylabel(ylabel='J (kg/m^2/s)', fontsize = 'x-large')

ax.set_zlabel(zlabel='q (quality)', fontsize = 'x-large')

ax.set_title('Dataset Heatmap')

plt.show()

def plot_loss(log, t):

plt.figure()

plt.plot(log.history['loss'], label='train_loss')

plt.plot(log.history['val_loss'], label='val_loss')

plt.ylabel('Error [Output]')

plt.xlabel('Epochs')

plt.title(t) plt.legend()

plt.grid(True) plt.show()

def plot_pred(test_labels, zp):

plt.figure()

for i in range(len(zp.T)):

plt.scatter(test_labels, zp.T.iloc[i], marker = 'o', alpha=0.4, label

= f'model {i}')

plt.plot(test_labels,test_labels,color='k', label = 'ideal')

Valiaveedu A-2 plt.grid(True) plt.legend()

plt.xlabel("true")

plt.ylabel("prediction")

Datasets In [10]:

1. first dataset to import is the OECD data, which will be used for evaluating the models import pandas as pd data_oecd = pd.read_csv('./chf_public.csv')

data_oecd = data_oecd.drop([0]) #clean excel file data_oecd = data_oecd.astype(float) data_oecd = pd.DataFrame(data_oecd, columns = ['Pressure', 'Mass Flux', 'Outlet Quality',

'CHF', 'CHF Result'])

1. clean data to match CHF table, all values are in Metric

1. P = kPa, Mass Flux = kg/m^2/s, CHF = kW/m^2 oecd_plt = chfplt(0, data_oecd['Pressure'], data_oecd['Mass Flux'],

data_oecd['Outlet Quality'], data_oecd['CHF'])

1. visualize data, note that Jupter Notebook is not the best with 3d visualizations.

1. map relationships import seaborn as sns g = sns.pairplot(data_oecd(('Pressure','Mass Flux','Outlet Quality','CHF')),diag_kind='kde')

plt.show()

1. second set is the LUT data_tbl = pd.read_pickle('./CHFTable.pkl')

data_tbl['q'] = np.array(data_tbl['q'], dtype=float) #cleans q column to be defined as values data_tbl = data_tbl[data_tbl['CHF'] !=0] #Omits 0 values of CHF for reducing bias top_3 = data_tbl['CHF'].quantile(q=0.975) data_tbl = data_tbl[(data_tbl['CHF']<top_3)]

1. right skewed CHF data, omit higher values due to spare nature of CHF points and further reduce bias

1. P = kPa, Mass Flux = kg/m^2/s, CHF = kW/m^2 tbl_plt = plt.figure(1) tbl_plt = chfplt(1, data_tbl['P'],data_tbl['J'],data_tbl['q'],

data_tbl['CHF'])

1. visualize table

1. map relationships import seaborn as sns

Valiaveedu A-3 g = sns.pairplot(train_data_tbl(('P','J','q','CHF')),diag_kind='kde')

plt.show()

Making the training/testing data In [12]:

1. %%Training and Testing data split import sklearn.model_selection TTS = sklearn.model_selection.train_test_split train_data_tbl, test_data_tbl = TTS(data_tbl, test_size=.15)

1. test size is 15% of LUT

1. features contain only input data (P,J,q)

1. labels are the only outputs train_feature = train_data_tbl.copy()

test_feature = test_data_tbl.copy()

train_labels = train_feature.pop('CHF')

test_labels = test_feature.pop('CHF')

1. additional data information for training sizes batch_size = 128 buffer_size = len(train_data_tbl) steps_p_epoch = buffer_size//batch_size

1. %% Learning rate decay function to reduce impact of outliers import tensorflow as tf from tensorflow.keras import layers

1. normlayer tbl_norm = tf.keras.layers.Normalization(axis=-1) tbl_norm.adapt(np.array(train_feature))

Evaluation Layers Now everything is initialized lets look at the impact of layers.

In [13]:

def get_optimizer():

return tf.keras.optimizers.Adam()

1. 1 Layer x_model = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(1)

])

x_model.compile(

optimizer = get_optimizer(),loss='mse')

log = x_model.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True,

Valiaveedu A-4

)

1. Model 2 x_model_2 = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation='elu'),

layers.Dense(32, activation='elu'),

layers.Dense(1)

])

x_model_2.compile(

optimizer = get_optimizer(),loss='mse')

log2 = x_model_2.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

1. model 3 x_model_3 = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation='elu'),

layers.Dense(32, activation='elu'),

layers.Dense(32, activation='elu'),

layers.Dense(32, activation='elu'),

layers.Dense(1)

])

x_model_3.compile(

optimizer = get_optimizer(),loss='mse')

log3 = x_model_3.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

plot_loss(log, 'Model 1')

plot_loss(log2, 'Model 2')

plot_loss(log3, 'Model 3')

1. evaluate the results with the LUT testing data zp = pd.DataFrame({'L0':x_model.predict(test_feature).flatten(),

'L1':x_model_2.predict(test_feature).flatten(),

'L2':x_model_3.predict(test_feature).flatten(),

})

from sklearn.metrics import r2_score for i in (range(len(zp.T))):

print(r2_score(test_labels, zp.T.iloc[i]))

Valiaveedu A-5 plot_pred(test_labels, zp) plt.show()

oecd_p=pd.DataFrame({'L0':x_model.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L1':x_model_2.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L2':x_model_3.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

})

error = data_oecd(('CHF')).values/oecd_p

1. error is measured as M/P plot_pred(data_oecd['CHF'],oecd_p) plt.show()

for i in (range(len(oecd_p.T))):

print(r2_score(data_oecd['CHF'], oecd_p.T.iloc[i]))

print(np.std(error))

print(np.mean(error))

print(np.median(error, axis=0).T)

Neurons the next evaluation is done with Node density.

In [16]:

1. Model 1 x_model = tf.keras.Sequential([tbl_norm, layers.Dense(16, activation="elu"),

layers.Dense(16, activation="elu"),

layers.Dense(1)

])

x_model.compile(

optimizer = get_optimizer(),loss='mse')

log = x_model.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True,

)

1. Model 2 x_model_2 = tf.keras.Sequential([tbl_norm, layers.Dense(64, activation="elu"),

layers.Dense(64, activation='elu'),

layers.Dense(1)

])

x_model_2.compile(

Valiaveedu A-6 optimizer = get_optimizer(),loss='mse')

log2 = x_model_2.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

1. Model 3 x_model_3 = tf.keras.Sequential([tbl_norm, layers.Dense(256, activation="elu"),

layers.Dense(256, activation='elu'),

layers.Dense(1)

])

x_model_3.compile(

optimizer = get_optimizer(),loss='mse')

log3 = x_model_3.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

plot_loss(log, 'Model 1')

plot_loss(log2, 'Model 2')

plot_loss(log3, 'Model 3')

1. evaluate the results with the LUT testing data zp = pd.DataFrame({'L0':x_model.predict(test_feature).flatten(),

'L1':x_model_2.predict(test_feature).flatten(),

'L2':x_model_3.predict(test_feature).flatten(),

})

from sklearn.metrics import r2_score for i in (range(len(zp.T))):

print(r2_score(test_labels, zp.T.iloc[i]))

plot_pred(test_labels, zp) plt.show()

oecd_p=pd.DataFrame({'L0':x_model.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L1':x_model_2.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L2':x_model_3.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

})

error = data_oecd(('CHF')).values/oecd_p

Valiaveedu A-7

1. error is measured as M/P plot_pred(data_oecd['CHF'],oecd_p) plt.show()

for i in (range(len(oecd_p.T))):

print(r2_score(data_oecd['CHF'], oecd_p.T.iloc[i]))

print(np.std(error))

print(np.mean(error))

print(np.median(error, axis=0).T)

Activations In [18]:

1. Model 1 x_model = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation="elu"),

layers.Dense(1)

])

x_model.compile(

optimizer = get_optimizer(),loss='mse')

log = x_model.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True,

)

1. Model 2 x_model_2 = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="relu"),

layers.Dense(32, activation='relu'),

layers.Dense(1)

])

x_model_2.compile(

optimizer = get_optimizer(),loss='mse')

log2 = x_model_2.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

1. Model 3 x_model_3 = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="gelu"),

layers.Dense(32, activation='gelu'),

layers.Dense(1)

])

Valiaveedu A-8 x_model_3.compile(

optimizer = get_optimizer(),loss='mse')

log3 = x_model_3.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

plot_loss(log, 'Model 1')

plot_loss(log2, 'Model 2')

plot_loss(log3, 'Model 3')

1. evaluate the results with the LUT testing data zp = pd.DataFrame({'L0':x_model.predict(test_feature).flatten(),

'L1':x_model_2.predict(test_feature).flatten(),

'L2':x_model_3.predict(test_feature).flatten(),

})

from sklearn.metrics import r2_score for i in (range(len(zp.T))):

print(r2_score(test_labels, zp.T.iloc[i]))

plot_pred(test_labels, zp) plt.show()

oecd_p=pd.DataFrame({'L0':x_model.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L1':x_model_2.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L2':x_model_3.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

})

error = data_oecd(('CHF')).values/oecd_p

1. error is measured as M/P plot_pred(data_oecd['CHF'],oecd_p) plt.show()

for i in (range(len(oecd_p.T))):

print(r2_score(data_oecd['CHF'], oecd_p.T.iloc[i]))

print(np.std(error))

print(np.mean(error))

print(np.median(error, axis=0).T)

Optimizers In [21]:

Valiaveedu A-9 def get_optimizer():

return tf.keras.optimizers.Adam()

def get_optimizer3():

return tf.keras.optimizers.RMSprop()

def get_optimizer5():

return tf.keras.optimizers.Nadam()

1. Model 1 x_model = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation="elu"),

layers.Dense(1)

])

x_model.compile(

optimizer = get_optimizer(),loss='mse')

log = x_model.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True,

)

1. Model 2 x_model_2 = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation='elu'),

layers.Dense(1)

])

x_model_2.compile(

optimizer = get_optimizer3(),loss='mse')

log2 = x_model_2.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

1. Model 3 x_model_3 = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation='elu'),

layers.Dense(1)

])

x_model_3.compile(

optimizer = get_optimizer5(),loss='mse')

log3 = x_model_3.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0,

Valiaveedu A-10 validation_split=0.15, shuffle = True

)

plot_loss(log, 'Model 1')

plot_loss(log2, 'Model 2')

plot_loss(log3, 'Model 3')

1. evaluate the results with the LUT testing data zp = pd.DataFrame({'L0':x_model.predict(test_feature).flatten(),

'L1':x_model_2.predict(test_feature).flatten(),

'L2':x_model_3.predict(test_feature).flatten(),

})

from sklearn.metrics import r2_score for i in (range(len(zp.T))):

print(r2_score(test_labels, zp.T.iloc[i]))

plot_pred(test_labels, zp) plt.show()

oecd_p=pd.DataFrame({'L0':x_model.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L1':x_model_2.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L2':x_model_3.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

})

error = data_oecd(('CHF')).values/oecd_p

1. error is measured as M/P plot_pred(data_oecd['CHF'],oecd_p) plt.show()

for i in (range(len(oecd_p.T))):

print(r2_score(data_oecd['CHF'], oecd_p.T.iloc[i]))

print(np.std(error))

print(np.mean(error))

print(np.median(error, axis=0).T)

Learning Rates In [23]:

def get_optimizer():

return tf.keras.optimizers.Adam()

lr_schedule = tf.keras.optimizers.schedules.InverseTimeDecay(

0.001, decay_steps=steps_p_epoch*1000,

Valiaveedu A-11 decay_rate=0.5, staircase=False) def get_optimizer3():

return tf.keras.optimizers.Adam(lr_schedule) def get_optimizer5():

return tf.keras.optimizers.Adam(0.0013) def get_optimizer7():

return tf.keras.optimizers.Adam(0.0007)

1. Model 1 x_model = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation="elu"),

layers.Dense(1)

])

x_model.compile(

optimizer = get_optimizer(),loss='mse')

log = x_model.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True,

)

1. Model 2 x_model_2 = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation='elu'),

layers.Dense(1)

])

x_model_2.compile(

optimizer = get_optimizer3(),loss='mse')

log2 = x_model_2.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

1. Model 3 x_model_3 = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation='elu'),

layers.Dense(1)

])

x_model_3.compile(

optimizer = get_optimizer5(),loss='mse')

log3 = x_model_3.fit(

Valiaveedu A-12 train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

1. Model 4 x_model_4 = tf.keras.Sequential([tbl_norm, layers.Dense(32, activation="elu"),

layers.Dense(32, activation='elu'),

layers.Dense(1)

])

x_model_4.compile(

optimizer = get_optimizer7(),loss='mse')

log4 = x_model_4.fit(

train_feature, train_labels, batch_size=batch_size, epochs=200, verbose=0, validation_split=0.15, shuffle = True

)

plot_loss(log, 'Model 1')

plot_loss(log2, 'Model 2')

plot_loss(log3, 'Model 3')

plot_loss(log4, 'Model 4')

1. evaluate the results with the LUT testing data zp = pd.DataFrame({'L0':x_model.predict(test_feature).flatten(),

'L1':x_model_2.predict(test_feature).flatten(),

'L2':x_model_3.predict(test_feature).flatten(),

'L3': x_model_4.predict(test_feature).flatten()

})

from sklearn.metrics import r2_score for i in (range(len(zp.T))):

print(r2_score(test_labels, zp.T.iloc[i]))

plot_pred(test_labels, zp) plt.show()

oecd_p=pd.DataFrame({'L0':x_model.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L1':x_model_2.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L2':x_model_3.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L3':x_model_4.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

})

error = data_oecd(('CHF')).values/oecd_p

1. error is measured as M/P

Valiaveedu A-13 plot_pred(data_oecd['CHF'],oecd_p) plt.show()

for i in (range(len(oecd_p.T))):

print(r2_score(data_oecd['CHF'], oecd_p.T.iloc[i]))

print(np.std(error))

print(np.mean(error))

print(np.median(error, axis=0).T) oecd_p=pd.DataFrame({'L1':x_model.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

'L3':x_model_3.predict(data_oecd.drop(columns=['CHF','CHF Result'])).flatten(),

})

data_oecd['CHF Result'] = oecd_p['L1'].values error = data_oecd(('CHF')).values/oecd_p plot_pred(data_oecd['CHF'],oecd_p) plt.show()

for i in (1,3):

print(r2_score(data_oecd['CHF'], oecd_p['L'+str(i)]))

print(np.std(error))

print(np.mean(error))

print(np.median(error, axis=0).T)

Ensemble In [38]:

def fit_model(train_features, train_labels):

lr_schedule = tf.keras.optimizers.schedules.InverseTimeDecay(

0.001, decay_steps=steps_p_epoch*1000, decay_rate=0.5, staircase=False)

1. Learning rate decay function to reduce impact of outliers

1. norm layer tbl_norm = tf.keras.layers.Normalization(axis=-1) tbl_norm.adapt(np.array(train_features))

x_model = tf.keras.Sequential([tbl_norm, layers.Dense(64, activation="elu"),

layers.Dropout(0.15),

layers.Dense(64, activation='elu'),

layers.Dropout(0.15),

layers.Dense(1)

])

x_model.compile(

optimizer = tf.keras.optimizers.Adam(lr_schedule)\\

Valiaveedu A-14

,loss='mse', metrics=['accuracy'])

log = x_model.fit(train_feature,train_labels, epochs=200, verbose=0, validation_split=0.15, shuffle = True)

1. validation is 15% of training, 200 epochs is estimated plataeu return [x_model, log]

t_members = 124 members = [fit_model(train_feature,train_labels) for _ in range(t_members)]

members = np.asmatrix(members).T def ensmbl_pred (members, test_features):

ens_model = []

for i in range(len(members.T)):

ens_model.append(members[0,i].predict(test_features))

ens_model = np.hstack(ens_model) ens_model = np.average(ens_model,axis=1) return np.array(ens_model.flatten())

1. Ensemble time zp= ensmbl_pred(members, test_feature) from sklearn.metrics import r2_score print(r2_score(test_labels,zp))

def plot_pred(test_labels, zp):

plt.figure()

plt.scatter(test_labels, zp, alpha =0.7, marker = 'x')

plt.plot(test_labels,test_labels,color='r', label = 'ideal')

plt.xlabel("true")

plt.ylabel("prediction")

plot_pred(test_labels, zp) plt.show()

oecd_p=ensmbl_pred(members, data_oecd.drop(columns=['CHF','CHF Result']))

data_oecd['CHF Result'] = oecd_p Results = data_oecd(('CHF')).copy()

Results['CHF Result']=data_oecd(('CHF Result')).copy()

Results['error']=(data_oecd['CHF']/data_oecd['CHF Result'])

plot_pred(data_oecd['CHF'],Results['CHF Result'])

plt.show()

print(r2_score(Results['CHF'], Results['CHF Result']))

data_ens = pd.DataFrame()

for i in range(t_members):

df_name ='Model '+ str(i)

Results[df_name] = members[0,i].predict(data_oecd.drop\\

(columns=['CHF','CHF Result']))

data_ens_m = Results.drop(columns =['CHF', 'CHF Result', 'error'])

Valiaveedu A-15 m_var = []

m_var = data_ens_m.std(axis='columns')

1. obtain variance for each data point plt.figure(2) plt.hist(m_var,bins=100) plt.xlabel('Standard Deviation')

plt.show()

1. plot the final result with the variance associated with the point plt.figure()

mloc=m_var.idxmax()

data_box = data_ens_m.T[mloc]

plt.boxplot(data_box, labels = [mloc])

plt.show()

data_low = []

data_low = data_ens_m.quantile(0.05,axis='columns')

1. obtain lower quantile for each data point data_high = []

data_high=data_ens_m.quantile(0.95,axis='columns')

plt.figure()

plt.scatter(data_oecd["CHF"],data_high, label = 'Upper 95',\\

marker = 'o', color = 'b',alpha=0.3) plt.scatter(data_oecd["CHF"], data_low, label ='Lower 5',\\

marker = '^', color = 'g', alpha=0.3) plt.scatter(data_oecd["CHF"],Results['CHF Result'], label ='Ensemble Model',\\

marker = 's', color = 'tab:orange', alpha=0.3) plt.plot(data_oecd["CHF"],data_oecd["CHF"],color='r', label = 'Ideal')

plt.legend()

plt.grid(True) plt.xlabel("true")

plt.ylabel("prediction")

plt.show()

plt.figure()

plt.grid(True) plt.scatter(data_oecd["CHF"],Results['CHF Result'], c=m_var, cmap='jet')

plt.xlabel("true")

plt.colorbar(label = 'Std')

plt.ylabel("prediction")

plt.show()