Introduc)on to light field processing

Laboratoire de Communica0ons Audio-Visuelles (LCAV) Ecole Polytechnique Fédérale de Lausanne (EPFL)

Loïc Baboulaz 2015

“The most perfect photograph currently shows only one aspect of reality; it reduces to a unique image fixed on a plane, as a drawing or a pain0ng would be traced by hand”

Jonas Ferdinand Gabriel Lippmann (16 August 1845 – 13 July 1921)

Jonas Ferdinand Gabriel Lippmann (16 August 1845 – 13 July 1921)

•  Part 1: Light, op/cs & light-field concepts •  Part 2: Tradi/onal camera technology •  Part 3: Light field camera technology

Jonas Ferdinand Gabriel Lippmann (16 August 1845 – 13 July 1921)

“Can we ask Photography to render all the richness that the direct view of an object offers?”

•  Part 4: Sampling & processing the light field

Light

•  Part 1: Light, op/cs & light-field concepts

Nature of light

ElectromagneGc radiaGon (wave vs. parGcle)

Main light properGes

Intensity PropagaGon direcGon Wavelength or color PolarizaGon

Light propagaGon theories

Quantum theory (wave packets) Wave theory (Maxwell’s equaGons) Ray-tracing (geometric opGcs)

•  Part 2: Tradi/onal camera technology •  Part 3: Light field camera technology •  Part 4: Sampling & processing the light field

Ray-tracing

Light Phenomena

Principle

Complicated nature of light is reduced to a large set of narrow beams, or rays. Approximate soluGon of Maxwell’s equaGons

AssumpGon

Objects much bigger than wavelength of light

Pros

ComputaGonally efficient Geometric opGcs Well-suited for opGcal design

Cons

Cannot model diffracGon

ReflecGon RefracGon

DiffracGon

Light field

Light field parametric representa/on

•  For every point in space, the light field L(r) describes the amount of light that is travelling at any instant of 0me

•  For every point in space, the light field L(r) describes the amount of light that is travelling at any instant of 0me •  A 2D light ray can be parameterized as: - Two-line intersec0ons: - Line intersec0on/slope:



, or as:

L(x, u) = L(x, x + d ⋅ p)

Light field parametric representa/on

Light field parametric representa/on

•  For every point in space, the light field L(r) describes the amount of light that is travelling at any instant of 0me •  A 2D light ray can be parameterized as:

•  For every point in space, the light field L(r) describes the amount of light that is travelling at any instant of 0me •  A 2D light ray can be parameterized as: , or as:

- Two-line intersec0ons: - Line intersec0on/slope:



, or as:

•  A 3D light ray is described by 4 dimensions: , or as:

•  Conversion between 2 representa0ons is straighYorward:

L(x, u) = L(x, x + d ⋅ p)

Light field parametric representa/on

Light field parametric representa/on

•  Cartesian ray-space diagram is used to visualize rays, beams and complete light field:

•  Cartesian ray-space diagram is used to visualize rays, beams and complete light field:

?

Light field parametric representa/on

Light field parametric representa/on

•  Cartesian ray-space diagram is used to visualize rays, beams and complete light field:

•  Cartesian ray-space diagram is used to visualize rays, beams and complete light field:

?

Light field parametric representa/on

Light field parametric representa/on

•  Cartesian ray-space diagram is used to visualize rays, beams and complete light field:

•  Cartesian ray-space diagram is used to visualize rays, beams and complete light field:

“Light beams share same directions at plane x ”

“Light beams overlap at plane u ”

Simple op/cal components

Simple op/cal components

•  Propaga0on at flat interface:

•  Propaga0on at flat interface:



•  Converging lens vs. diverging lens optical axis

Simple op/cal components

optical axis

Lens parameters

•  Propaga0on at flat interface: •  Converging lens vs. diverging lens

Front focal point

optical axis

optical axis

•  Naming conven0on for lenses: optical axis

biconvex

biconcave

positive meniscus

Back

negative meniscus

planoconvex

planoconcave

optical axis

Lens parameters

Thin Lens approxima/on



Front

Back focal point

Front

optical axis

•  Lensmaker’s equa0on:

Back focal point

optical axis

•  Thin lens equa0on:

•  Convexity/concavity ó Sign of focal length ó Sign of lens radii. •  P: lens power in dioptres; f: focal length in meters.

Thin Lens approxima/on

Thin Lens imaging Front

Front

Back focal point

•  Thin lens equa0on:

optical axis

Back

optical axis

•  Thin lens formula:

Example of propaga/on in thin lens

Example of propaga/on in thin lens

?

Example of propaga/on in thin lens

?

Example of propaga/on in thin lens

“Shearing of the light field”

Example of propaga/on in thin lens

Example of propaga/on in free space T

? “Shearing of the light field”

“Ray transfer matrix”

Example of propaga/on in free space

Example of propaga/on in free space T

T

?

Example of propaga/on in free space

Example of propaga/on in free space T

T

“Shearing of light field”

“Shearing of light field”

Ray transfer matrix

Ray transfer matrix

•  Ray transfer matrices can be used to propagate light field forward and backward

•  Ray transfer matrices can be used to propagate light field forward and backward

Propaga0on in free space with no scabering

Propaga0on in free space with no scabering

Refrac0on at flat interface

Refrac0on at flat interface

Reflec0on at flat mirror

Reflec0on at flat mirror

Iden0ty matrix

Refrac0on at thin lens

Refrac0on at thin lens

S h e a r matrix

Refrac0on at thick lens

Refrac0on at thick lens

S h e a r matrix Scaling matrix

Ray transfer matrix: example with concatenated op/cal components

Front

Ray transfer matrix: example with concatenated op/cal components

Back

Front

optical axis

Ray transfer matrices

Back

optical axis

Ray transfer matrices

Operations on light field

Operations on light field

Ray transfer matrix: example with concatenated op/cal components

Ray transfer matrix: example with concatenated op/cal components

Front

Back

Front

optical axis

Ray transfer matrices

Operations on light field

Back

optical axis

Ray transfer matrices

Operations on light field

Ray transfer matrix: example with concatenated op/cal components

Front

Ray transfer matrix: example with concatenated op/cal components

Back

Front

Back

optical axis

Ray transfer matrices

optical axis

Ray transfer matrices

Operations on light field

Operations on light field

Surface reflectances

Surface reflectances •  The BRDF describes how light is reflected. It is material-specific. Source

Viewer

Light source

θs

θr

ϕr Scene Observer 1

Observer 2

ϕs

Surface reflectances

Surface reflectance: illustra/on

•  The BRDF describes how light is reflected. It is material-specific. Source

Viewer

Rendering equa0on: Source

θs

θr

Viewer

θs

ϕr

ϕr

ϕs

ϕs

Surface reflectance: illustra/on

Source

Surface reflectance: illustra/on

Viewer

Source -π/2

θs

θr

0

+π/2

Diffuse (Lamber/an)

θr

Viewer -π/2

θs

ϕs

+π/2

Diffuse (Lamber/an)

θr

ϕr

0

ϕr ϕs

-π/2

0

+π/2

Diffuse + specular

Camera technology: the pinhole camera

•  Part 1: Light, op/cs & light-field concepts •  Part 2: Tradi/onal camera technology Anthony Browell, Rosy at Elkington Park 475

•  Part 3: Light field camera technology •  Part 4: Sampling & processing the light field

Pinhole camera

Pinhole camera

Image sensor

Aperture

Scene OpGcal axis

Scene

Image sensor O

OpGcal axis

Pinhole camera

Pinhole camera

Image sensor

Aperture

Image sensor

Scene

Aperture

Scene

OpGcal axis

x

OpGcal axis

x

u

•  Ray-space parameterizaGon:

u

u

•  Ray-space parameterizaGon: x

u x

Pinhole camera

Pinhole camera

Image sensor

Aperture

Image sensor

Scene

Aperture

Scene

OpGcal axis

x

OpGcal axis

u

•  Ray-space parameterizaGon:

Depth u x

•  Each light ray gives one image point

•  Change of depth in the scene

Pinhole camera

Image sensor

Pinhole camera

Aperture

Image sensor

Scene

Aperture

Scene

OpGcal axis

OpGcal axis

Depth

Depth

•  Change of depth in the scene

•  Change of depth in the scene •  All-in-focus image (~ infinite depth of field)

Pinhole camera

Pinhole camera

Image sensor

Aperture

Image sensor

Scene

I(x,y)

Aperture

Scene

I(x,y) OpGcal axis

OpGcal axis

P(X,Y,Z) f - “focal length”

•  Change of depth in the scene •  All-in-focus image (~ infinite depth of field) •  Mapping from 3D to 2D:

Depth

P(X,Y,Z) f - “focal length”

Depth

•  Change of depth in the scene •  All-in-focus image (~ infinite depth of field) •  Mapping from 3D to 2D:

Problem: the amount of light hiNng the sensor is very low, i.e. long exposure /me

Towards single lens camera

Image sensor

Towards single lens camera

Aperture

Image sensor

Scene

Aperture

Scene

OpGcal axis

OpGcal axis

“Open the aperture To allow more light in”

“Open the aperture To allow more light in”

Towards single lens camera

Image sensor

Towards single lens camera

Aperture

Image sensor

Scene

Aperture

OpGcal axis

Problem: blurred image points

Scene OpGcal axis

Problem: blurred image points à Solu/on: add collima/ng lens

Single lens camera

Single lens camera

Image sensor

Aperture

Image sensor

Scene

Aperture

Scene

OpGcal axis

OpGcal axis

Lens

Lens

Single lens camera

Single lens camera

Image sensor

Aperture

Image sensor

Scene

Aperture

OpGcal axis

sharp

Lens

sharp

•  Each image point receives more light à lower exposure Gme •  Image sensor integrates the incoming light cone

Scene OpGcal axis

Lens

Single lens camera

Single lens camera

Image sensor

Aperture

Image sensor

Scene

Aperture

Scene

OpGcal axis

OpGcal axis

Lens

blurry

Lens

sharp

Single lens camera

Single lens camera

Image sensor

Aperture

Image sensor

Scene

Aperture

OpGcal axis

blurry

Lens Focal plane

sharp

à Focal plane: points in object space that are focused by lens on image sensor

Scene OpGcal axis

blurry

Lens

sharp

How to change the focal plane?

Focal plane

Single lens camera

Single lens camera

Image sensor

Aperture

Image sensor

Scene

Aperture

Scene

OpGcal axis

sharp

blurry

blurry

Lens

OpGcal axis

sharp Focal plane

sharp

blurry

How to change the focal plane? By translaGng the posiGon of the image sensor

blurry

Lens Focal plane

sharp

How to change the focal plane? By translaGng the posiGon of the image sensor In other words: •  When refocusing, the light field propagates differently in the camera before acquisiGon, •  The image sensor captures the same light field, sheared differently for each focal plane.

Single lens camera

Single lens camera

Image sensor

Aperture

Image sensor

Scene

Aperture

Scene

OpGcal axis

sharp

OpGcal axis

Lens Focal plane

blurry

How to change the focal plane? By changing the lens configuraGon, i.e. refocusing.

CoC

blurry

Lens Focal plane

sharp

à Circle of Confusion (CoC): amount of acceptable blur for a point to be focused . CoC

CoC

Out-of-focus Focused Out-of-focus

Single lens camera

Single lens camera

Image sensor

Aperture

Image sensor

Scene

Aperture

Scene

OpGcal axis

OpGcal axis

Depth-of-field CoC

blurry

Depth-of-field

Lens

Lens Focal plane

Focal plane

sharp

à Depth-of-field: part of the scene that is focused well enough on image sensor. CoC

How to change the depth-of-field?

CoC

Out-of-focus Focused Out-of-focus

Single lens camera

Single lens camera

Image sensor

Aperture

Image sensor

Scene

Aperture

OpGcal axis

Scene OpGcal axis

Depth-of-field sharp

Lens Focal plane

CoC

How to change the depth-of-field? By closing the aperture of the lens.

Depth-of-field sharp

Lens

CoC

•  Closing the aperture: 1.  Extends the depth of field, 2.  Reduces the amount of light, 3.  Increases the exposure Gme.

Focal plane

Single lens camera: ray-space analysis

Image sensor

Single lens camera: ray-space analysis

Aperture

Scene Main lens

Image sensor

Aperture

Scene Main lens

OpGcal axis

OpGcal axis

x

F

u

Focal plane

Single lens camera: ray-space analysis

Image sensor

Focal plane

Single lens camera: ray-space analysis

Aperture

Scene Main lens

Image sensor

Aperture

OpGcal axis

x

F

u

u x

Ray-space diagram

OpGcal axis

x Focal plane

Scene Main lens

F

u Focal plane

Single lens camera: ray-space analysis

Image sensor

Single lens camera: ray-space analysis

Aperture

Scene

Image sensor

Main lens

Aperture

OpGcal axis

x

F

u

u x

Main lens OpGcal axis

x Focal plane

Scene

F

u Focal plane

u x

Ray-space diagram

Ray-space diagram

Single lens camera: ray-space analysis

Image sensor

Integral Projec/on Operator

Aperture

•  Let be the canonical projec0on operator that reduces an N-dimensional func0on down to M-dimensions by integra0ng the last N-M dimensions [Ng06]:

Scene Main lens OpGcal axis

x

F

u Focal plane

u x

x Integral-projection

Ray-space diagram

Image samples

•  The photographic imaging operator of single lens camera can thus be wriben as:

Single lens camera: ray-space analysis

Image sensor

Single lens camera: ray-space analysis

Aperture

Scene Main lens

Image sensor

Aperture

Scene Main lens

OpGcal axis

x

F

u

OpGcal axis

x Focal plane

F

u Focal plane

u x

Ray-space diagram

Single lens camera: ray-space analysis

Image sensor

Single lens camera: ray-space analysis

Aperture

Scene Main lens

Image sensor

Aperture

OpGcal axis

x

F

u

u x

Ray-space diagram

OpGcal axis

x Focal plane

Scene Main lens

F

u Focal plane

u x

Ray-space diagram

Single lens camera: ray-space analysis

Image sensor

Single lens camera: ray-space analysis

Aperture

Scene

Wide aperture

Image sensor

Main lens

Scene Main lens

OpGcal axis

x

F

u

OpGcal axis

x Focal plane

u x

x

x Integral-projection

Image samples

Ray-space diagram

Single lens camera: ray-space analysis

Narrow aperture

Image sensor

Scene

Narrow aperture

Image sensor

Main lens

F

u

u

Depth-of-field Focal plane

x

x Integral-projection

Scene Main lens

OpGcal axis

Ray-space diagram

Image samples

Ray-space diagram

Single lens camera: ray-space analysis

x

Depth-of-field Focal plane

u

u

x Integral-projection

F

OpGcal axis

x

F

u

u

Depth-of-field Focal plane

x

x Integral-projection

Image samples

Image samples

Ray-space diagram Can we actually capture the light field itself?

Beyond single lens camera

u

•  Part 1: Light, op/cs & light-field concepts •  Part 2: Tradi/onal camera technology

x

•  Part 3: Light field camera technology •  Part 4: Sampling & processing the light field

Beyond single lens camera

u

x Integral-projection

Light field

Beyond single lens camera

?

u

x

Light field

•  How to acquire directly the light-field?

? x

?

Light field

•  How to acquire directly the light-field? •  What is an efficient image generaGon process?

Image samples

Beyond single lens camera

u

Beyond single lens camera

?

?

x

u

?

Light field cameras

x

x

Light field

Image samples

?

x

Light field

Image samples

•  How to acquire directly the light-field? •  What is an efficient image generaGon process? •  What advantages can be gained in terms of rendered images?

•  How to acquire directly the light-field? •  What is an efficient image generaGon process? •  What advantages can be gained in terms of rendered images?

Beyond single lens camera

Beyond single lens camera

u Light field cameras

x

Fourier Slice Photograph Theorem

Light field

•  How to acquire directly the light-field? •  What is an efficient image generaGon process? •  What advantages can be gained in terms of rendered images?

u

?

Light field cameras

x

x

Image samples

?

Fourier Slice Photograph Theorem

Light field

•  How to acquire directly the light-field? •  What is an efficient image generaGon process? •  What advantages can be gained in terms of rendered images?

•  •  •  •  • 

SyntheGc refocusing, SyntheGc aperture Depth map, Parallax images, … x

Image samples

Single lens camera vs. Light field camera

Image sensor

Aperture

Light field camera

Scene Main lens OpGcal axis

Same user-seNngs as in single lens cameras: •  Aperture •  Focus •  Zoom (if available) •  Exposure Gme •  Sensor sensiGvity (ISO)

Focal plane Image Micro-lens sensor array (mla)

Aperture

Scene Main lens

Image Micro-lens sensor array (mla)

Aperture

Scene Main lens

OpGcal axis

OpGcal axis

Focal plane

Micro-lens array and light field camera

Focal plane

Light field camera

Image sensor

Micro-lens array (mla)

Scene

Aperture Main lens

OpGcal axis

Microlens array (Ng06)

Focal plane

Lytro

Raytrix

Light field camera

Image sensor

Light field camera

Micro-lens array (mla)

Scene

Aperture

Image sensor

Main lens

Micro-lens array (mla)

Scene

Aperture Main lens

OpGcal axis

OpGcal axis

Focal plane

Focal plane

x

f

u' pixel

Light field camera

Light field camera ANGULAR RESOLUTION

Image sensor

Micro-lens array (mla)

Scene

Aperture Main lens

OpGcal axis

Focal plane

x

f

u' pixel

du’

Image sensor

Micro-lens array (mla)

Scene

Aperture Main lens

OpGcal axis

Focal plane

Light field camera

Image sensor

Light field camera

Micro-lens array (mla)

Scene

Aperture

Image sensor

Main lens

Micro-lens array (mla)

Scene

Aperture Main lens

OpGcal axis

OpGcal axis

Focal plane

Focal plane

x

f

u' pixel

Light field camera

Light field camera: ray-space analysis SPATIAL RESOLUTION

Image sensor

Micro-lens array (mla)

Scene

Aperture

Image sensor

Micro-lens array (mla)

Scene

Aperture Main lens

Main lens

OpGcal axis

OpGcal axis

x

u

u

Focal plane

Focal plane

dx

f pixel

x u'

dx’

x

x u'

Light field camera: ray-space analysis

Image sensor

Micro-lens array (mla)

Light field camera: ray-space analysis

Scene

Aperture Main lens

Image sensor

Micro-lens array (mla)

Scene

Aperture Main lens

OpGcal axis

x

OpGcal axis

x

u

u

Focal plane

x

u

u

Focal plane

x

x

x

u'

Light field camera: ray-space analysis

Image sensor

Micro-lens array (mla)

u'

Light field camera: ray-space analysis

Scene

Aperture Main lens

Image sensor

Micro-lens array (mla)

Scene

Aperture Main lens

OpGcal axis

x

x

u

u

Focal plane

x

OpGcal axis

x u'

u

u

Focal plane

x

x u'

Light field camera: ray-space analysis

Image sensor

Micro-lens array (mla)

Light field camera: ray-space analysis

Scene

Aperture Main lens

Image sensor

Micro-lens array (mla)

Scene

Aperture Main lens

OpGcal axis

x

OpGcal axis

x

u

u

u

Focal plane

u

dx

Focal plane

du x

x

x

x

u'

Light-field camera: ray-space analysis

Image sensor

Micro-lens array (mla)

u'

Light-field camera: ray-space analysis

Scene

Aperture Main lens

Image sensor

Micro-lens array (mla)

Main lens

OpGcal axis

x u

Focal plane

x

OpGcal axis

x

u

Scene

Aperture

u

u Focal plane

x

Light-field camera: ray-space analysis

Image sensor

Micro-lens array (mla)

Light-field camera: ray-space analysis

Scene

Aperture Main lens

Image sensor

Micro-lens array (mla)

Main lens

OpGcal axis

x u

OpGcal axis

x

u Focal plane

u

x

u Focal plane

x

Light-field camera: ray-space analysis

Image sensor

Micro-lens array (mla)

Scene

Aperture Main lens

•  Part 1: Light, op/cs & light-field concepts OpGcal axis

x u

u Focal plane

•  Part 2: Tradi/onal camera technology •  Part 3: Light field camera technology •  Part 4: Sampling & processing the light field

x

Scene

Aperture

Digital Photographic Imaging

Digital Photographic Imaging

u

u

F

F Image focused on “green” plane

Image focused on “green” plane

?

x

Light field

x Image samples

Light field

Digital Photographic Imaging

Digital Photographic Imaging

u

u

αF, α1 Image focused on “blue” plane

x

x

x Image samples

Light field

Shear + Integral projection

Image focused on “blue” plane x Image samples

Light field

Imaging Operator of a light field camera

Imaging Operator of a light field camera

•  Let be the shear operator that propagates the light-field at :

•  Let be the shear operator that propagates the light-field at :

•  The photographic imaging operator of a light field camera can thus be wriben as:

Imaging Operator of a light field camera

Fourier and Slicing Operators

•  Let be the shear operator that propagates the light-field at :

•  Let be the N-dimensional Fourier transform operator of a func0on and let be its inverse:

•  The photographic imaging operator of a light field camera can thus be wriben as:

•  How about in the Fourier domain?

Fourier and Slicing Operators

Classical Fourier Slice Theorem: illustra/on

u

u

u •  Let be the N-dimensional Fourier transform operator of a func0on and let be its inverse:

•  Let be the slicing operator that reduces an N-dimensional func0on down to an Mdimensional one by sejng to zero the last N-M dimensions:

x

x

Fourier Transform

Fourier Transform

ku

ku

kx

Classical Fourier Slice Theorem: illustra/on 2D signal

x

Fourier Transform ku

kx

kx

Classical Fourier Slice Theorem: illustra/on 1D signal

u

x

2D signal

1D signal

u

x

x

x

kx

kx'

ku

Classical Fourier Slice Theorem: illustra/on 2D signal

Classical Fourier Slice Theorem: illustra/on 1D signal

u

ku

1D signal

u

x

x

x

x

kx

kx'

ku

kx

kx'

Classical Fourier Slice Theorem: illustra/on 2D signal

2D signal

The Generalized Fourier Slice Theorem 1D signal

u

x

x

kx

kx'

ku

What about 4D signals? And shearing operator?

Let f be an N-dimensional func0on. Then: - changing the basis of f, - integral-projec0ng it down to M dimensions and - applying the Fourier transform is equivalent to: - applying the Fourier transform to f, - changing basis with the normalized inversed transpose of the original basis and - slicing it down to M dimensions.

The Generalized Fourier Slice Theorem

The Fourier Slice Photograph Theorem [Ng06]

Let f be an N-dimensional func0on. Then: - changing the basis of f, - integral-projec0ng it down to M dimensions and - applying the Fourier transform is equivalent to: - applying the Fourier transform to f, - changing basis with the normalized inversed transpose of the original basis and - slicing it down to M dimensions.

•  Let recall the photographic operator of a light field camera:

The Fourier Slice Photograph Theorem [Ng06]

The Fourier Slice Photograph Theorem [Ng06]

•  Let recall the photographic operator of a light field camera:

•  Let recall the photographic operator of a light field camera:

The Fourier Slice Photograph Theorem [Ng06]

The Fourier Slice Photograph Theorem [Ng06]

•  Let recall the photographic operator of a light field camera:

A Photograph is the inverse 2D Fourier transform of a dilated 2D slice in the 4D Fourier transform of the light field

The Fourier Slice Photograph Theorem

The Fourier Slice Photograph Theorem

4D signal

2D signal

u

x

4D signal

2D signal

u

x

x

x

kx

kx'

ku

The Fourier Slice Photograph Theorem 4D signal

The Fourier Slice Photograph Theorem 2D signal

u

4D signal

2D signal

u

x

x

ku

x

x

kx

kx'

ku

kx

kx'

Light-field Cameras

Light-field Cameras

•  Focused Plenop/c Camera (Adobe): Image mla sensor

•  Focused Plenop/c Camera (Adobe):

Aperture

Scene

Virtual image Image mla plane sensor

Main lens

Aperture

OpGcal axis

x



Scene Main lens OpGcal axis

u

x



u

Light-field Cameras

Light-field Cameras

•  Focused Plenop/c Camera (Adobe): Virtual image Image mla plane sensor

•  Extended Depth-of-field Plenop/c Camera (Raytrix):

Aperture

Scene

Virtual image Image mla planes sensor

Main lens

Aperture

OpGcal axis

x

x

Virtual image plane

Main lens OpGcal axis

u



Scene

u

Virtual image plane

a

a x

b

x

b

u'

u'

ANGULAR RESOLUTION

SPATIAL RESOLUTION

Light-field Cameras

Light-field Cameras

•  Extended Depth-of-field Plenop/c Camera (Raytrix): Virtual image Image mla planes sensor

Aperture

•  Light-field technology comparison (from Raytrix website 2011): Scene

Main lens

?

OpGcal axis

x

u yes Virtual image planes

Type 1

Type 2

Type 3

Type 1

Type 2

Type 3

Type 1

x u'



yes

Light field image: Lytro example

Raw 4D light->ield

Light field image: Lytro example

Raw 4D light->ield

Light field image: Lytro example

Raw 4D light->ield

Light field image: Lytro example

Raw 4D light->ield

Light field image: Lytro example

Light field image: digital refocusing

u

x

v

y

pixel

Light field image: digital refocusing

Light field image: perspec/ve shi_

Let view All-in-Focus

Right view

Light field image: perspec/ve shi_

Custom light-field camera

Light field image: depth map

Custom light-field camera

Nikon D800 + Micro-Nikkor 105mm f/2.8G

XY-slide

XY-slide Schneider Fine Art XXL Coppal #3 + Sinar lens board Focal length

X-slide

X-slide Focal plane

PainGng OpGcal axis

Acquiring the 4D light field of a painGng

Custom light-field camera Focal length Focal plane

Camera posiGons

OpGcal axis

Sub-aperture images

Region of interest

Custom light-field camera

Custom light-field camera Focal length

Camera posiGons

Focal length Focal plane

Camera posiGons

OpGcal axis

Light rays from ith pixel from each sub-aperture image

Focal plane

OpGcal axis

Light rays from ith pixel from each sub-aperture image Light rays from jth pixel from each sub-aperture image

Custom light-field camera

Custom light-field camera Focal length

Camera posiGons

Focal plane

OpGcal axis Focal plane

PainGng

Focal plane àConjugate pinhole cameras PainGng

Acquiring the 4D light field of a painGng

Sub-aperture image view of acquired light field (major: UV ; minor: XY) 32x32x512x512

Acquiring the 4D light field of a painGng (resoluGon: 512x512x32x32)

Microlens view of acquired light field (major: XY ; minor: UV) 512x512x32x32

Sub-aperture image view of acquired light field (major: UV ; minor: XY) 32x32x512x512

Microlens view of acquired light field (major: XY ; minor: UV) 512x512x32x32

Depth map from acquired light field

What to remember

•  Light field processing is at the convergence of opGcs and image processing. •  Ray tracing offers a convenient framework to manipulate light field. •  OpGcal elements such as thin lens and free space shear the light field. •  The light field is 4D signal and an image is one 2D integral-projecGon of it. •  Light field cameras acquire both spaGal and angular informaGon of light rays. •  From a light field, images with different focus can be generated ater acquisiGon. •  Working in the Fourier domain is computaGonally more efficient. •  The Fourier Slice Theorem can be applied to the light field for image rendering.