ML24026A120
| ML24026A120 | |
| Person / Time | |
|---|---|
| Issue date: | 01/25/2024 |
| From: | Stephen Cumblidge NRC/NRR/DNRL/NPHP |
| To: | |
| References | |
| Download: ML24026A120 (1) | |
Text
NRC Thoughts on AI/ML Data Analysis in NDE Stephen Cumblidge 2024 Industry/NRC NDE Technical Information Exchange Public Meeting
Disclaimer This presentation does not give official NRC positions, but reflects the current NRC technical staff knowledge of this rapidly-evolving subject
Automated Data Analysis is Coming The variety of open-source ML tools has enabled the use of ML algorithms to be developed for many uses Work is ongoing by several groups to use these tools to develop automated data analysis algorithms to analyze NDE data These ML algorithms can potentially greatly increase the reliability and consistency of NDE in the field If used incorrectly, these tools could degrade inspections
Possible Benefits of ADA-Assistance ADA has the potential to improve detection of flaws and improve the human factors of an examination
- In the nuclear industry we have sparse flaws, and computers can maintain vigilance in cases where humans struggle
- Humans and computers make different types of mistakes, and a qualified analyst paired with a good algorithm gives the best of both worlds
- Reduced dose to inspectors
Possible Hazards of ADA ADA adds another layer of complexity on an already complex system Can introduce common-cause failures of inspections across the fleet Licensees may not understand the capabilities and limitations of ADA, which could lead to improper use of ADA ADA assistance may allow people to pass Appendix VIII testing without the skills to recognize unknown degradation in the field
Possible Hazards of ADA (Continued)
ML algorithms can be challenging to train and retrain, possibly making the ML algorithms unreliable The quality of the data and the quality of the labeling is paramount ML algorithms require a new class of experts to performing UT examinations
Expect the Unexpected ADA methods can be very good at handling known problems, but may not work on forms of degradation not in the training set As plants age and new reactors are designed, it is almost certain that new degradation mechanisms will emerge and flaws will appear in places they were not expected
ADA Assisted Procedures One suggested approach by EPRI is for an ADA algorithm to flag areas with flaws, and the algorithm must find all flaws in the qualification set The algorithm can produce more false calls than allowed in the given supplement It will be up to the inspector to determine which of the areas flagged by the algorithm are actually flaws, and ultimately the inspector is responsible for the results
Initial Qualification of ADA-Assisted Examinations The method and criteria for discrimination of an indication is an essential variable in a procedure The ADA algorithm is an essential variable A procedure using ADA-assistance may only be qualified to Appendix VIII when a Level II implements the procedure
Differences in Testing a Person vs an Algorithm An Appendix VIII test requires a UT level II to be overseen by staff and their results graded by the testing authority, and can take several days Testing an algorithm against a large amount of recorded data can be done in hours
NRC Initial Thoughts on Retraining an ML Algorithm If an ML algorithm is retrained, the algorithm has been altered and must be requalified Currently, consistent with the requirement to requalify, if an essential variable is changed, the requalification of an ML algorithm must be done via a successful personnel qualification The NRC understands the potential benefits of writing rules for field-friendly implantation of ML (e.g. requalifying a retrained ML algorithm on-site)
Paths to the Future Skilled Inspectors Engaged AI Experts Optimized Procedures Best Outcome Unskilled Inspectors No AI Experts Unoptimized Procedures Worst Outcome Near Future on Current Trajectory In About Ten years if We are Not Careful New inspectors become overly-dependent on AI tools AI experts move on to new tasks Lack of AI experts prevents effective development and retraining of algorithms Current Inspections Improved New procedures can be developed New degradation can be found Current Inspections may be OK New degradation may not be found New Procedures may be challenging
Possible Paths Forward Additions to Appendix VIII covering the essential variables for ML algorithms Create rules for requalifying an algorithm after modification that does not require a person to pass a personnel test e.g. Finds all flaws in qualification data without too many additional false calls Requirements for personnel to use ADA-assisted procedures to assure that they have appropriate skills e.g. Pass an Appendix VIII tests for the same procedure or the same supplement without ADA assistance Some sort of AI/ML Certification for procedure development and retraining?