Computational Photography

Computational Photography Digital photography Photography has been rapidly changing in recent years, with the transition from analog to digital Tam...
Author: Ernest Scott
7 downloads 0 Views 4MB Size
Computational Photography

Digital photography Photography has been rapidly changing in recent years, with the transition from analog to digital

Tampere 8.2012

Digital photography Computing can now be involved in every step of the imaging process. This is having a huge impact on the art and practice of photography. We don’t just take a picture – we sample the light field

Tampere 8.2012

The light field

Tampere 8.2012

The light field • The light field describes the amount of light (radiance) traveling in every direction through every point in space – Possibly also as a function of wavelength and time – The plenoptic function

• One goal of image based rendering is to construct the light field from a set of images, so that new images can be synthesized

Tampere 8.2012

Computational photography • Traditional (analog and digital) cameras have limitations, e.g.: – Cannot capture scenes with significant differences between bright and dark areas – Limited field of view – Can only focus at a single plane

• Computational photography is an emerging research field that attempts to extend or enhance the capabilities of digital photography by adding computational elements to the imaging process – Optics + sensors + illumination + processing + interaction – Convergence of computer vision, digital imaging, and graphics

Tampere 8.2012

Computational photography • Typical process: take several images, combine them (perhaps with user interaction), create “better” image(s) • May modify – Field of view, depth of field/focus, dynamic range, spatial resolution, wavelength resolution, temporal resolution – Aperture, focus, sensor design, exposures, illumination

• Computational photography can provide – – – – – –

Improved dynamic range Variable focus, resolution, and depth of field Aesthetic image framing Content-based image editing Hints about shape, reflectance, and lighting New interactive forms of photography

Tampere 8.2012

Traditional Photography

Sensor array

Lens Pixels

Image

Tampere 8.2012

Computational Photography

Generalized sensor

Generalized Optics Computation

Samples

Image

Tampere 8.2012

E.g., a computational lens Varioptic Liquid Lens: Electrowetting Applied voltage modifies the curvature of the liquidliquid interface, leading to a variable focal length lens

Tampere 8.2012

The Plenoptic Function (a.k.a. the light field)

Q: What is the set of all things that we can ever see? A: The Plenoptic Function Let’s start with a stationary person and try to parameterize everything that he can see… Tampere 8.2012

A grayscale image

…is intensity of light Seen from a single view point At a single time Averaged over the wavelengths of the visible spectrum (can also do P(x,y), but spherical coordinate are nicer)

P(q,f) Tampere 8.2012

A color image

…is intensity of light Seen from a single view point At a single time As a function of wavelength

P(q,f,l) Tampere 8.2012

A color movie

…is intensity of light Seen from a single view point Over time As a function of wavelength

P(q,f,l,t) Tampere 8.2012

A holographic movie

…is intensity of light Seen from ANY viewpoint Over time As a function of wavelength

P(q,f,l,t,VX,VY,VZ) Tampere 8.2012

The Plenoptic Function

P(q,f,l,t,VX,VY,VZ)

Can reconstruct every possible view, at every moment, from every position, at every wavelength Contains every photograph, every movie, everything that anyone has ever seen! It completely captures our visual reality. Tampere 8.2012

Sampling the Plenoptic Function (top view)

Tampere 8.2012

Synthesizing novel views

Tampere 8.2012

Image based rendering We don’t need a (3D) model of the object Instead, model the object by storing its light field

Stuff

Tampere 8.2012

Some examples of computational photography Research and products

Flash/no-flash image pairs

Eisemann and Durand SIGGRAPH04 Petschnigg et al. SIGGRAPH04

No-flash

Flash

Result Tampere 8.2012

Agrawal et al. SIGGRAPH05

Flash/no-flash image pairs Ambient

Flash

Result

Reflection Layer

Tampere 8.2012

Raskar et al.

High dynamic range imaging

Ambient

Flash

Tampere 8.2012

HDR imaging: varying exposure

Tampere 8.2012

Flash Brightness

Exposure Time Tampere 8.2012

? ? ?

Result Tampere 8.2012

Image retargeting

Setlur et al. SIGGRAPH04

Rearranging the image content to optimize the information of interest, given the constraints (e.g., reduced aspect ratio)

Tampere 8.2012

The Lytro camera

Link

Tampere 8.2012

A practical light field camera

uv-plane

st-plane

Tampere 8.2012

The prototype camera

Adaptive Optics microlens array

125μ square-sided microlenses

4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Tampere 8.2012

Digital refocusing

Σ

Σ • refocusing = summing windows extracted from several microlenses Tampere 8.2012

Example of digital refocusing

Tampere 8.2012

Cinemagraph (www.cinemagraph.com) • Still photographs in which a minor (and repeated) movement occurs – Typically in Animated GIF format

• “A Cinemagraph is an image that contains within itself a living moment that allows a glimpse of time to be experienced and preserved endlessly.”

–34

Tampere 8.2012

Coded aperture photography

Levin et al. SIGGRAPH 2007

• Regular camera with “patterned aperture” (made from cardboard!) in place of regular aperture • From this, can get scene depth and do refocusing • Uses deconvolution with the known patterns

Tampere 8.2012

Output #1: Depth map

Single input image:

Output #2: All-focused image

Lens and defocus Image of a point light source

Lens’ aperture

Lens

Camera sensor

Point spread function

Focal plane

Lens and defocus Image of a defocused point light source

Lens’ aperture

Object

Lens

Camera sensor

Point spread function

Focal plane

Lens and defocus Image of a defocused point light source

Lens’ aperture

Object

Lens

Camera sensor

Point spread function

Focal plane

Lens and defocus Image of a defocused point light source

Lens’ aperture

Object

Lens

Camera sensor

Point spread function

Focal plane

Lens and defocus Image of a defocused point light source

Lens’ aperture

Object

Lens

Camera sensor

Point spread function

Focal plane

Idea 2: Coded Aperture • Mask (code) in aperture plane - make defocus patterns different from natural images and easier to discriminate

Conventional aperture

Our coded aperture

Solution: lens with occluder

Object

Lens

Camera sensor

Point spread function

Focal plane

Solution: lens with occluder Image of a defocused point light source

Aperture pattern

Object

Lens with coded aperture

Camera sensor

Point spread function

Focal plane

Solution: lens with occluder Image of a defocused point light source

Aperture pattern

Object

Lens with coded aperture

Camera sensor

Point spread function

Focal plane

Solution: lens with occluder Image of a defocused point light source

Aperture pattern

Object

Lens with coded aperture

Camera sensor

Point spread function

Focal plane

Solution: lens with occluder Image of a defocused point light source

Aperture pattern

Object

Lens with coded aperture

Camera sensor

Point spread function

Focal plane

Solution: lens with occluder Image of a defocused point light source

Aperture pattern

Object

Lens with coded aperture

Camera sensor

Point spread function

Focal plane

Regularizing depth estimation Try deblurring with 10 different aperture scales

x  arg min | f  x  y |2 Convolution error



_

 l i  (xi ) Derivatives prior

2

+

Keep minimal error scale in each local window + regularization

Input

Local depth estimation

Regularized depth

Regularizing depth estimation

Local depth estimation

Input

Regularized depth

Input

All-focused (deconvolved)

Close-up

Original image

All-focus image

Input

All-focused (deconvolved)

Application: Digital refocusing from a single image

Application: Digital refocusing from a single image

Application: Digital refocusing from a single image

Application: Digital refocusing from a single image

Application: Digital refocusing from a single image

Application: Digital refocusing from a single image

Application: Digital refocusing from a single image

Coded aperture: pros and cons AND depth at a single shot + Image + No loss of image resolution + Simple modification to lens - Depth is coarse

unable to get depth at untextured areas, might need manual corrections.

+ But depth is a pure bonus - Loss some light + But deconvolution increases depth of field

Some projects in computational photography

Composition Context Photography w/Daniel Vaquero

Composition context photography

How can the act of framing an image help to produce better images? A wider array of images to choose from? – Higher resolution, higher SNR, wider FOV, optimal subject pose, etc. Tampere 8.2012

Composition context photography The composition context of a picture: the viewfinder frames, their capture parameters, and inertial sensor data collected while the user is framing the photograph

Tampere 8.2012

Tampere 8.2012

Stanford, Ulm, Nokia RC, UCSB, Disney Research (SIGGRAPH 2010)

The Frankencamera • An open source experimental camera architecture and API for computational photography • Fully programmable: permits control and synchronization of the sensor and image processing pipeline at the microsecond time scale, as well as the ability to incorporate and synchronize external hardware like lenses and flashes.

Tampere 8.2012

Tampere 8.2012

Generalized Autofocus w/Daniel Vaquero

Generalized autofocus

• Choices have to be made in focusing images • Large depth-of-field  small aperture  noisy image • Can we provide alternatives? New focusing options? – Large depth-of-field with wide aperture? – Selective focus? Tampere 8.2012

All-in-focus imaging

Images focused at different distances (focal stack)

Tampere 8.2012

What is our work about? • The capture process for focal stacks – For fusion, we use existing techniques, but we do it on-camera

• Our scenario, typical in camera phones and low-end consumer cameras – Fixed lens aperture – Handheld camera – Limited memory and processing power

Tampere 8.2012

Contrast-based passive autofocus

Sharp But low SNR

Tampere 8.2012

Our method: Generalized autofocus • Contribution: autofocus for focal stacks

• Find the minimal set of required images to generate an allin-focus composite – The choice of this set depends on the scene

depth

• Why is minimizing the number of images important? – Faster capture – Less sensitive to motion – Requires less memory and processing power Tampere 8.2012

Method overview Nokia N900 depth

1. Capture a focal stack of low-resolution images while continuously changing the focal distance (“lens sweep”) 2. Analyze sharpness of low-resolution focal stack 3. Determine the minimal set of required images 4. Recapture the images in the minimal set in high-resolution 5. Perform fusion of the images on-camera Desired: Good user interface to specify what should be in focus

Tampere 8.2012

Tampere 8.2012

Standard autofocus

Tampere 8.2012

All-in-focus result (24 image in stack, 3 used)

Tampere 8.2012

Stack images (4)

Tampere 8.2012

All-in-focus result (24 image in stack, 4 used)

Tampere 8.2012

Generalized Autofocus

Tampere 8.2012

Ongoing work • Improve image fusion • User selectable focus – Location and range – Between x and y – In focus here and here and here…

• Provide focusing opportunities not possible with lens-only solutions

Tampere 8.2012