Light Field De-blurring for Robotics Applications Aaron Snoswell and Surya Singh The Robotics Design Lab The University of Queensland, Australia [email protected] and [email protected]

Abstract Light field cameras record light intensity and angle of light incidence. This extra data presents a bevy of new possibilities for robotics; however, the usefulness of light field cameras may be hampered by motion blur when used on moving platforms. Traditional deconvolution de-blurring methods use a single point spread function and perform poorly in the presence of parallax. We derive a 3D point spread function for linear camera motion blur and demonstrate that this can be applied to light field images from a commercial Lytro camera. An experiment testing this approach using a moving camera is described for static scenes with objects distributed throughout the focal range. On average our method gives a doubling of the Q-factor sharpness without excessive artefacts.

1

Introduction

Light field or plenoptic cameras are a new imaging modality device that provide four-dimensional light capture (i.e. measuring light intensity and direction). Like traditional cameras, light field cameras can suffer from motion blur due to finite exposure times. Existing deconvolution de-blurring methods can be applied to 2D slices of a light field image, however have the significant drawback that they only work for scenes with constant depth. While light-field deconvolution has received some attention for light field microscopy [Levoy et al., 2006], to our knowledge it has not yet been applied to de-blurring a light field. Furthermore, the use of light field imaging for robotics is relatively new [Dansereau, 2013b; Dong et al., 2013]. Light field images are of interest generally and for robotics due to the many novel applications they enable. For example, light field cameras are presently popular for the ability to refocus images in software. Potential robotic applications include the ability to reduce the effects of fog, snow and other partial occluders [Dansereau, 2013b], excellent low-light performance [Ng et al., 2005] and the ability to recover depth information from a compact sensor [Adelson and Wang, 1992].

Figure 1: De-blurring light field camera motion. In many robotics applications, motion may result in significant image blurring. The original light-field derived image (left) is processed using our method to give the noticeably sharper result across various scene depths (note the sharp text on the ground and on the can).

There are many robotic vision algorithms that rely on highfrequency image features (for example, dense pixel matching techniques or edge, line or shape detection). Motion blur obscures this high-frequency content and has the potential to limit the performance of these methods. In many robotic scenarios, deconvolution de-blurring is an ineffective solution to this problem due the presence of large depth variation (parallax) within the robot operating environment - e.g. ground or underwater vehicles that operate close to obstacles. We propose a method for deconvolution de-blurring of light field images that shows promising results (e.g. see Figure 1), even for scenes with significant depth variation.

2

Plenoptic De-blurring

The light field is a mathematical description of the way light propagates. Another name for this is the plenoptic function, from the Latin plenus, meaning full, and optic, meaning light. The plenoptic function describes every possible configuration a light ray could ever be in, encompassing the three spatial dimensions, two rotational components, temporal changes, as well as frequency (colour), phase and polarisation changes. In most computer vision literature, the phase and polarisation elements are ignored, leading to a 7-dimensional expression of the plenoptic function;

Figure 2: A visualisation of a 3D linear motion blur PSF, showing multiple 2D PSFs at various scene depths.

f vc te (4) Z where f is the camera focal length and vc and te are the camera velocity and exposure time respectively. This equation was derived using homogeneous projective geometry and assumes the world frame is aligned with the camera frame, and that the camera has square pixels, no sensor skew and an aligned optical axis. Equation 4 can be viewed as describing a 3D PSF - that is, if the focal length, velocity, exposure time and depth of objects in a scene are known, the blur at that point in the image can be computed ( Figure 2 shows this visually). Because of this, an optimal 3D PSF can be used for deconvolution of the different depth planes in the image. After deconvolution of each depth plane, these sharpened images can be composited together to produce a final image sharp at each scene depth. We implemented this technique using a Lytro light field camera. One drawback encountered was that while the Lytro software can be used to recover depth data from light fields, this data is in arbitrary units. In order to perform depth-aware deconvolution, the depths had to be calibrated first. For our purposes we empirically determined that for indoor scenes with controlled lighting, the Lytro depth response followed a linear mapping, however in a real-world application of this technique a distance sensor could be combined with a light field camera for real-time calibration. Figure 3 shows a pictorial overview of the method outlined here. w=

P = P (Vx , Vy , Vz , θ, φ, t, λ)

(1)

where Vx , Vy and Vz are the spatial dimensions, θ and φ are the angular dimensions, t is the temporal dimension and λ describes the colour of the ray. The Lytro camera, a modern light field camera and the camera used in our research, samples three colours and four of the five position/angular dimensions from the light field. Importantly, for space free of occluding obstacles, this is equivalent to sampling all position/angular dimensions - a consequence of the fact that a light ray’s luminance will not change along it’s path unless blocked [Levoy and Hanrahan, 1996]. A key concept within light field research is that of ‘computational photography’ - that is, new, 2D images can be computationally extracted from the light field data. Our goal was to find a way to ensure that these computed 2D images were free of motion blur for the simplified case of a Lytro camera undergoing linear motion. Deconvolution is a thoroughly documented means for removing motion blur from images. In Deconvolution deblurring, the camera output i(x) is modelled as the true image of the world f (x) convolved with some blur kernel g(x) (also known as the Point Spread Function or PSF) plus additive noise n(x). In the frequency domain, this can be written I(f ) = F (f ) × G(f ) + N (f )

(2)

Deconvolution de-blurring methods then are some variant of the expression F (f ) =

I(f ) G(f )

(3)

where various methods are used to approximate g(x), handle the zeros it will contain in the frequency domain, and account for the unknown image noise. These methods have the drawback that they assume the entire scene has been blurred with a constant blur kernel. In scenes with significant depth variation, objects closer to the camera will be blurred more than those farther away. Specifically, for the case of linear motion blur a point source at depth Z within the scene will produce a ‘blur trail’ of width

3

Experimental Method

An experiment was designed to test depth-aware deconvolution for the simplified case of controlled, known, linear camera motion blur with short (< 0.5m) range scenes. Our plan was to carry a commercial DSLR camera and a Lytro on a motion platform at a controlled velocity and capture blurred images of several scenes. The LF images would then be de-blurred, with the sharp DSLR images acting as ground truths by which we could verify our results. Unfortunately, the lens and DSLR combination used (a Nikon D5100 with a 18-55mm Nikon lens) could not adequately bring all scene depths into focus. For this reason alternate methods were used to quantify the quality of the de-blurred images, however the DSLR was left in-place on the motion platform to maintain experimental consistency.

Depth  Map  

Blurred   Light  Field   Data  

Blurred   2D  Image   Slice  

Depth   Plane   Slice  

Depth   Plane   Slice  

Depth   Plane   Slice  

Depth   Plane   Slice  

Depth   Plane   Slice  

Depth   Plane   Slice  

De-­‐blurred   Image  

Figure 3: Schematic diagram showing the flow of data through our implementation of the depth-aware deconvolution technique. Stars indicated de-blurred images.

3.1

Motion platform

A robotic linear motion platform was developed using Lego Mindstorms parts. The platform was designed to carry a Lytro camera and a Nikon D5100 DSLR camera side-by-side. Support for both cameras was made out of Lego parts. The motion platform consisted of a wheeled based that carried the cameras attached by thread to a drum, driven by two Mindstorms motors. The drum was manually held in place during operation, and the Matlab RWHT NXT Toolbox was used to interface with the motors. The angular velocity of the drum was empirically calibrated allowing a desired velocity and linear distance to set in software. Figure 4 shows the design of the linear motion platform.

3.2

Scene Design

Several photographic scenes were arranged using household items. The scenes were arranged indoors in a room with no external facing windows to ensure the scene brightness could be controlled, with all items close (