ML25273A293
| ML25273A293 | |
| Person / Time | |
|---|---|
| Issue date: | 09/24/2025 |
| From: | Matthew Homiack NRC/RES/DSA/AAB |
| To: | |
| References | |
| Download: ML25273A293 (1) | |
Text
ADVANCING NRC EVENT ANALYSIS Machine Learning Approaches for Classification of Human Performance-Related Events in Licensee Event Reports Matthew Homiack Reactor Systems Engineer (Data Scientist)
U.S. Nuclear Regulatory Commission Office of Nuclear Regulatory Research Division of Systems Analysis Accident Analysis Branch Matthew.Homiack@nrc.gov
Explored many key technical considerations for use of AI/ML in regulatory decision-making through development of an LER classifier.
Explainability Robustness Risk Analysis Model Maintenance Field Performance Degradation Trustworthiness Ethics Test, Evaluation, Verification, and Validation Domain Adaptation Life Cycle Management Bias Security Assurance Processes Data Drift Data Quality, Quantity, Applicability, and Uncertainty Data Collection and Curation Preprocessing Feature Extraction Model Development Model Evaluation Machine Learning Pipeline Technical Considerations
How data is collected and curated shapes its quality, quantity, and applicability, and carries ethical implications.
LER Texts Labeled Dataset Human Factors Information System Search LERs Website 14,533 Records from 1993-2017
,\nOn the evening of December 2,\n1994, an operator was in the process of \nhanging a Protective Tagging Record (PTR) for work on Emergency Service\ni\nWater System B pump in support of scheduled refuel outage activities.\nThis\ntag required the operator to open 600V electrical breaker 12610 at load\n\ncanter 71L-26 located in the East Electrical Bay.\nThe operator located the\ncorrect breaker, but dropped the PTR tag.\nAfter picking up the tag and
\nupon returning his attention to the breaker, but without reverifying\ncorrect breaker 12610, the operator inadvertently pushed the \TRIP\ button\nfor breaker 12606.\
Preprocessing can improve data quality and applicability, but may introduce bias.
Textual information removed during preprocessing shown in orange.
Feature extraction and model development influence the explainability of AI decisions and address bias.
Naive Bayes Model Class Imbalance 32%
Has Human Factors 68%
No Human Factors
Crew Workers Knowledge Token-level contributions per class illustrate the internal logic of the model and enhance explainability.
Briefing Human LERs with Human Factors LERs with no Human Factors Weld Corrosion Capacitor Lightning Manufacturing
Trustworthiness is supported by test and evaluation strategies that align system behavior with intended use.
False Negatives True Positives + False Negatives Performance metrics for high-recall, human-in-the-loop system design False Positive True Negative False Negative True Positive Negative Positive Negative Positive Predicted Actual
=
False Negative Rate True Positives True Positives + False Negatives
=
Recall False Negatives True Positives + False Negatives
=
Precision
Putting the ML model into use would save 17% in classification effort per year.
21 True Negatives (properly classified LERs that are not of interest)
Illustration based on average workload of 137 LER reviews/year 2
False Negatives (acceptably low proportion of mis-classified LERs that are of interest) 23 LER reviews/year Savings 43 True Positives (properly classified LERs that are of interest) 71 False Positives (mis-classified LERs that are not of interest) 114 LER reviews/year Remaining Workload
Lessons and insights from this project will be used to shape future NRC research efforts.
AI/ML success for regulatory decision-making depends on carefully considering all applicable technical considerations in each stage of the development pipeline.
Naive Bayes demonstrates strong performance for human factors classification in LER texts.
Collaboration between data scientists and domain experts is essential for ensuring alignment with real-world needs.