Robust video stitching using adaptive pixel transfer

Abstract

Video stitching is a challenging problem when multiple videos are captured with severely different camera parameters. In this paper, we propose a robust video stitching algorithm which transfers the pixels in the background plane and the foreground objects separately. We first compute the fundamental matrix and the homography on a common ground plane between two videos using the activity based features. The ground plane pixels can be warped by using the homography. For each foreground object, we project the off-plane pixels onto the ground plane by analyzing the dominant direction of object region. The projected ground plane pixel is transferred to another image via homography, and then elevated to find the true corresponding pixel using the epipolar constraint. We also locally adjust the correspondence matching positions based on energy minimization framework. Experimental results demonstrate that the proposed algorithm correctly warps both of the ground plane and the foreground objects between video sequences captured with distinct camera configurations, compared with the state-of-the-art stitching algorithms.