Reflection Removal

• Reflection removal for 2D images

The images taken through glass often capture a target transmitted scene as well as undesired reflected scenes. In this paper, we propose an optimization problem to remove reflection automatically from multiple glass images taken at slightly different camera locations. We first warp the multiple glass images to a reference image, where the gradients are consistent in the transmission images while the gradients are varying across the reflection images. Based on this observation, we compute a gradient reliability such that the pixels belonging to the salient edges of the transmission image are assigned high reliability, but that of the reflection images are assigned low values. Then we suppress the gradients of the reflection images and recover the gradients of the transmission images only, by solving the proposed optimization problem in gradient domain. We reconstruct an original transmission image using the resulting optimal gradient map. Experimental results show that the proposed algorithm removes the reflection artifacts from glass images faithfully and outperforms the existing algorithms.

 

• Reflection removal using deep learning

An image captured through a glass plane usually contains both of a target transmitted scene behind the glass plane and a reflected scene in front of the glass plane. We propose a semantic context based network to remove reflection artifacts from a single glass image. We first investigate a non-linear intensity mapping relationship for glass images to synthesize more realistic training sets. Then we devise an efficient reflection removal network using multi-scale generators and an interpreter, where the semantic context of the transmission image is adopted as a high-level cue for the interpreter to guide the generators. We also provide a new test data set of real glass images including the ground truth transmission and reflection images. Experiments are performed on four test data sets and we show that the proposed algorithm decomposes an input glass image into a transmission image and a reflection image more faithfully compared with the four existing state-of-the-art methods.

 

• Reflection removal for 360-degree images

The existing methods for reflection removal mainly focus on removing blurry and weak reflection artifacts and thus often fail to work with severe and strong reflection artifacts. However, in many cases, real reflection artifacts are sharp and intensive enough such that even humans cannot completely distinguish between the transmitted and reflected scenes. We attempt to remove such challenging reflection artifacts using 360-degree images. We adopt the zero-shot learning scheme
to avoid the burden of collecting paired data for supervised learning and the domain gap between different datasets. We first search for the reference image of the reflected scene in a 360-degree image based on the reflection geometry, which is then used to guide the network to restore the faithful colors of the reflection image. We collect 30 test 360-degree images exhibiting challenging reflection artifacts and demonstrate that the proposed method outperforms the existing state-of-the-art methods on 360-degree images.

In 360-degree images, the glass image region and the reference image region capturing the reflected scene exist together, and the glass image is often restored by using the information of reference image. In this paper, we first propose a fully-automatic end-to-end RR framework for 360-degree images which automatically detects the glass and reference image regions and removes the reflection artifacts in the glass image by using the reference information simultaneously. We devise a transformer based U-Net architecture with horizontal windowing scheme to capture the long-range dependencies between the glass and reference images via the self-attention mechanism and suppress the reflection artifacts by using the reference information. We also construct a training dataset of 360-degree images by synthesizing realistic reflection artifacts considering diverse geometric relation and photometric variation between the glass and reference images. The experimental results show that the proposed method detects the glass and reference image regions reliably without user-annotation and achieves better performance of RR compared with the state-of-the-art methods.

 

• Publications