Deep Learning Methodology for Multivariate Time-Series Forecasting

Abstract

Recent advances in artificial intelligence and machine learning have influenced many research domains, including economics and traffic, with enormous methods and theorems proposed to solve domain-specific problems. However, most of them except pure machine learning focuses on problem-solving through adaptation or incremental changes of novel deep learning architectures, frequently without sufficient discussion. This discussion includes how the novel architectures work, what their theoretical basis and assumptions are, and how the architectures affect to certain domain. In addition to the black-box nature and complex architecture of the recent deep learning models, this insufficiency results in less trustworthiness, widening the gap between research and real-world applications. While a little work has been proposed to infuse their domain knowledge and machine learning theorem, such design requires a wide range of expertise, including machine learning, domain knowledge, data analysis, and mathematical backgrounds. In the pure machine learning and computer vision domain, equation-level investigation of the models with respect to optimization and self-introduced biases is one of the crucial research themes, which could be a basis for trustworthy machine learning research. In this dissertation research, I focus on providing design rationale and methodology to improve the models and make them trustworthy with domain knowledge, data-oriented insights, and mathematical backgrounds. This dissertation introduces three-fold novel contributions to the time series forecasting domain by introducing domain-specific theorems and knowledge into the machine learning architectures--(1) I have analyzed the machine learning architectures in the current literature to investigate the architecture's impact and how researchers leverage it; (2) I have infused domain-specific mathematical concepts into the deep learning model to validate the hypothesis that domain knowledge combined with a theoretical understanding of the model is advantageous for both performance and interpretability; and (3) I have provided multiple technical solutions by developing various machine learning methods with domain knowledge, such as input and output representation learning, inductive bias control, and suggestion of knowledge-infused design rationale for the objective function. Regarding the first contribution, I conduct an extensive literature review and then define, categorize, and experiment with existing machine learning techniques and suggest their characteristics. Regarding the second goal, I implement a visual analytics tool to investigate data characteristics and inject mathematically formed domain knowledge into pre-trained models to measure its impact. Last but not least, I introduce three novel machine learning methodologies built upon theoretical considerations and domain knowledge-based model design. The rationality and effectiveness of this work have been investigated with extensive quantitative and qualitative results. This dissertation contributes to the machine learning domain by revisiting the traditional mathematical methods with the theoretical background, invoking trustworthy and explainable machine learning development.