Vispark: GPU-accelerated distributed visual computing using spark

Abstract

With the growing need of big data processing in diverse application domains, MapReduce (e.g., Hadoop) becomes one of the standard computing paradigms for large-scale computing on a cluster system. Despite of its popularity, the current MapReduce framework suffers from inflexibility and inefficiency inherent from its programming model and system architecture. In order to address these problems, we propose Vispark, a novel extension of Spark for GPU-accelerated MapReduce processing on array-based scientific computing and image processing tasks. Vispark provides an easy-to-use, Python-like high-level language syntax and a novel data abstraction for MapReduce programming on a GPU cluster system. Vispark introduces a programming abstraction for accessing neighbor data in the mapper function, which greatly simplifies many image processing tasks using MapReduce by reducing memory footprints and bypassing the reduce stage. We demonstrate the performance of our prototype system on several visual computing tasks, such as image processing, and K-means clustering.