Beyond communication: on-modem inference and training of DNNs for self-learning smart home appliances

Abstract

We introduce the concept of on-modem learning, a novel approach that empowers communication modems embedded in smart home appliances to conduct inference and training of compact-sized DNNs (deep neural networks) using real-time user data. This enables the delivery of personalized intelligent services within the home environment. Recognizing that a significant portion (80-90%) of commercial home appliances lack network connectivity, resulting in idle computing resources within the modem, our proposed on-modem learning transforms a data-oblivious and non-learning modem into an inde- pendent self-learning module. This is achieved by leveraging the unused or under-utilized hardware of the modem for learning from user data. To achieve this objective, we introduce an on-modem DNN inference framework, comprising a Data Payload Parser and On-Modem DNN engine. The former iden- tifies unknown raw data, converting it into a series of recognizable data fields, while the latter provides runtime support for DNN execution on the modem. Additionally, we propose an on-modem DNN train- ing framework that facilitates resource-efficient DNN training on the modem through Greedy and Di- verse Gradient Accumulation (GDGA), Dynamic Programming-based Back-Propagation (DPBP), and On-Modem Reinforcement Learning (OMRL). These collectively enhance the user experience in home appliances by enabling real-time learning from customer data and home appliance dynamics, utilizing the limited computing resources (i.e., processor and memory) of the modem and unlabeled user data. To the best of our knowledge, our on-modem learning approach is the first to execute DNN inference and training directly on the modems of commercial home appliances. To validate the effectiveness of on-modem learning, we implement two on-modem learning applications on an LG Electronics washing machine, i.e., remaining wash time prediction and UE (unbalance error) forecasting. Moreover, we col- lect over six million user data samples from actual LG home appliance customers and apply them to the two application scenarios. The experimental results demonstrate the successful execution of DNN infer- ence and training on modems integrated into commercial home appliances, providing tailored intelligent services to users.