Multi-modal learning-based algae phyla identification using image and particle modalities

Abstract

Algal blooms in freshwater, which are exacerbated by urbanization and climate change, pose significant challenges in the water treatment process. These blooms affect water quality and treatment efficiency. Effective identification of algal proliferation based on the dominant species is important to ensure safe drinking water and a clean water supply. Traditional algae identification techniques, such as microscopy and molecular techniques, are time-consuming and depend on the expertise of the practitioner. This study introduced an artificial intelligence (AI)-based multi-modal approach, which is a recent advancement in techniques for improving algal identification by integrating algal images and particle properties. We employed multi-modal learning to integrate multiple data modalities, including algal images and particle properties acquired using Flow Cam, to provide robustness and reliability for classifying algal phyla, such as Anabaena, Aphanizomenon, Microcystis, Oscillatoria, and Synedra. This study involved acquiring images and particle modalities, which were conducted to integrate the dataset using early, late, and hybrid fusion methods. In addition, explainable AI approaches, including SHapley Additive exPlanations (SHAP) and gradient-weighted class activation mapping (Grad-CAM), were used to investigate the contributions of the algal image and particle modalities to the proposed multi-modal algorithm. The multi-modal algae identifier with late fusion achieved an average F1 score of 0.91 and 0.88 for training and tests related to identifying algal phyla, respectively. Furthermore, compared with other modalities, the image and particle modalities showed significant potential as complementary and reliable components of deep-learning algorithms for algal identification in the water treatment process. These findings can contribute to a safe and clean water supply by effectively identifying the dominant algal species in the water treatment process.