School of CSE Seminar Series: Eric C. Cyr

Speaker: Eric C. Cyr, distinguished scientist at Sandia National Laboratories
Date and Time: October 16, 2:00-3:00 p.m. 
Location: Coda Building, Room 114
Host: Qi Tang

Title: Applying Scientific Computing Perspectives to Neural Network Training

Abstract: Machine learning techniques and their AI decedents are increasingly important part of modern science and engineering. They have proven to be powerful in areas such as surrogate model development, hypothesis generation, and data analysis. Often, these approaches rely on a parameterized function approximation known as the neural network. These approximators are renowned for their flexibility and relative ease of use (if not accuracy). With that ease of use comes a large computational expense. For instance, recent work has demonstrated that pre-training a LLM on the Frontier supercomputer would require two years with ideal parallelization [1]. This talk seeks to examine the neural network with a scientific computing lens to enhance our ability to understand details of the approximation and improve training. Underpinning these ideas is an aspirational hypothesis that insights will lead to multilevel (or hierarchical) methodologies to accelerate training using dramatically fewer resources. 

This talk proceeds in two parts. In the first part, we discuss an adaptive basis perspective that has proved fruitful in Scientific Machine Learning (SciML) [2]. The central idea is that the output of the penultimate layer can be seen as providing a basis for the approximation space. With this perspective we develop efficient “operator-split” training algorithms, and new initialization strategies motivated by stability concerns. The second part extends the adaptive basis perspective by considering all layers of the neural network to be basis functions in a learned approximation space. We discuss the challenges associated with the typical neural network architectures that use globally supported basis functions. We then describe Kolmogorov-Arnold networks (KANs) as an alternate architecture built on locally supported splines basis functions [3]. A one-to-one relationship between KANs and a multi-channel ReLU network is shown and used to argue a relaxation property reminiscent of properties of matrix-split iterative linear solvers. Finally, using a natural refinement algorithm combined with the relaxation property, we develop a multilevel training algorithm for KANs demonstrating it on problems in SciML.

1. Dash, et al., Optimizing distributed training on frontier for large language models. In ISC High Performance 2024 Research Paper Proceedings (39th Intl. Conference), 2024.
2. Cyr, et al., Robust training and initialization of deep neural networks: An adaptive basis viewpoint, Mathematical and Scientific Machine Learning, 2020. 
3. Southworth, et al. Multilevel Training for Kolmogorov Arnold Networks, arXiv:2603.04827, 2026 (submitted to SISC)

Bio: Eric C. Cyr has been at Sandia Albuquerque since 2009 and is a Distinguished Member of the Technical staff. Prior to that he received a BS in Computer Science from Clemson University in 2002, and a PhD in Computer Science from University of Illinois at Urbana-Champaign in 2008. His formal training is in numerical methods, and scientific computing. As a Sandian he has worked in a range of diverse areas including high-performance computing, preconditioners for multi-physics, discretizations for computational plasma simulation, sensitivity analysis for PDEs, numerical optimization, and software development for science applications. In 2018, Eric received the DOE Early Career Award to fund a focus on developing layer-parallel methods for training deep neural networks. This work has broadened his exposure to machine learning. To this end, Eric is working to extend the applicability of HPC to deep learning, understand the behavior and approximation properties of neural networks, and develop methodologies where machine learning can be used to enhance the impact of computation on science and engineering disciplines.