Deep learning based single image and video deblurring with exploiting time-resolved dataset

Abstract

Blind non-uniform deblurring is a highly ill-posed inverse problem that aims to recover the latent sharp image from severe blurs induced by large motion. There are a number of factors to non-uniformly blur videos or image such as camera shake, object motion, and depth variation that particularly make this inverse problem quite challenging. Most non-deep learning works investigated how to estimate unknown non-uniform blur kernels and/or latent frame(s). Recently, deep-learning based approaches have been proposed to tackle deblurring problem with excellent quantitative results and fast computation time.

For single image deblurring with deep learning, multi-scale (MS) approach has been widely used for deblurring that sequentially recovers the downsampled original image in low spatial scale first and then further restores in high spatial scale using the result(s) from lower spatial scale(s). Here, we investigate a novel alternative approach to MS, called multi-temporal (MT), for non-uniform single image deblurring by exploiting time-resolved deblurring dataset from high-speed cameras. MT approach models severe blurs as a series of small blurs so that it deblurs small amount of blurs in the original spatial scale progressively instead of restoring the images in different spatial scales. To realize MT approach, we propose progressive deblurring over iterations and incremental temporal training with temporally augmented training data. Our MT approach, that can be seen as a form of curriculum learning in a wide sense, allows a number of state-of-the-art MS based deblurring methods to yield improved performances without using MS approach. We also proposed a MT recurrent neural network with recurrent feature maps that outperformed state-of-the-art deblurring methods with the smallest number of parameters.

Unlike single image deblurring, one of the key components for video deblurring is how to exploit neighboring frames. Recent state-of-the-art methods either used aligned adjacent frames to the center frame or propagated the information on past frames to the current frame recurrently. Here, we propose multi-blur-to-deblur (MB2D), a novel concept to exploit neighboring frames and time-resolved deblurring dataset for efficient video deblurring. Firstly, inspired by unsharp masking, we argue that using more blurred images with long exposures as additional inputs significantly improves performance. Secondly, we propose multi-blurring recurrent neural network (MBRNN) that can synthesize more blurred images from neighboring frames, yielding substantially improved performance with existing video deblurring methods. Lastly, we propose multi-scale deblurring with connecting recurrent feature map from MBRNN (MSDR) to achieve state-of-the-art performance on the popular GoPro and Su datasets in fast and memory efficient ways.