Adaptive Edge Synthesis for Image Segmentation Using Color and Depth Images

Abstract

Not only the conventional camera, which captures the color of real world scene, currently in wide public use, but the depth camera, which captures the distance from the device to the objects, is being increasingly used. Keeping pace with this trend, many studies in the field of computer vision have been actively undertaken to leverage depth information. One of the uses of depth information is image segmentation. Conventional image segmentation uses only color information to extract an object from the image. However, when the object and the background of the image have similar color statistics, the object cannot be properly extracted from the background. Therefore, by adding depth information to color statistics as a key feature, we can reliably separate the object from the background even though the two sets of color statistics are similar. There are also problems, however, with the application of depth information. The boundary of objects is not clear due to sensor noise or errors, and therefore, when matching the color image and the depth image, the boundaries of the two images are imprecisely matched. In this thesis, we propose an adaptive edge synthesis algorithm to solve the boundary mismatch problem between the color and depth images. We first extract the edges from the color image and the depth image, respectively. Then we find the optimal matching point of a depth edge pixel to a color edge pixel, by maximizing the similarity cost of the normalized cross correlation. We refine the positions of depth edge pixels to those of the matched color edge pixels using the graph cut optimization technique. Finally, we synthesize the final edges by selecting the refined depth edge pixels and the original color edge pixels adaptively, which are directly used for object segmentation. The experimental results demonstrate that the proposed algorithm effectively improves the accuracy of the extracted depth boundaries, and as a consequence, extracts the objects more reliably than the conventional image segmentation algorithms.