Tutorial
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
2021
Computational photography with plenoptic camera and light field capture: tutorial EDMUND Y. LAM Department of Electrical and Electronic Engineering, The University of Hong Kong, Pokfulam, Hong Kong, China (e-mail:
[email protected]) Received 14 July 2015; revised 7 September 2015; accepted 10 September 2015; posted 10 September 2015 (Doc. ID 245927); published 14 October 2015
Photography is a cornerstone of imaging. Ever since cameras became consumer products more than a century ago, we have witnessed great technological progress in optics and recording mediums, with digital sensors replacing photographic films in most instances. The latest revolution is computational photography, which seeks to make image reconstruction computation an integral part of the image formation process; in this way, there can be new capabilities or better performance in the overall imaging system. A leading effort in this area is called the plenoptic camera, which aims at capturing the light field of an object; proper reconstruction algorithms can then adjust the focus after the image capture. In this tutorial paper, we first illustrate the concept of plenoptic function and light field from the perspective of geometric optics. This is followed by a discussion on early attempts and recent advances in the construction of the plenoptic camera. We will then describe the imaging model and computational algorithms that can reconstruct images at different focus points, using mathematical tools from ray optics and Fourier optics. Last, but not least, we will consider the trade-off in spatial resolution and highlight some research work to increase the spatial resolution of the resulting images. © 2015 Optical Society of America OCIS codes: (110.1758) Computational imaging; (110.2990) Image formation theory; (110.5200) Photography; (110.3010) Image reconstruction techniques; (100.3020) Image reconstruction-restoration; (100.6640) Superresolution. http://dx.doi.org/10.1364/JOSAA.32.002021
1. INTRODUCTION A. Computational Optical Imaging
The world is rich in information. The goal of imaging is to record, process, and display such information. Before the advent of digital sensors, what we strove to record on film was often a faithful 2D projection of a scene or an object; except perhaps for some darkroom techniques, such as burning and dodging, to adjust the exposure, the photographic print mirrored what was captured on the film. The popularization of digital sensors, including both CCD and CMOS, together with the processing power in the accompanying electronics, has brought about a paradigm shift in imaging. On one hand, it is possible to consider the digital sensor merely a replacement of the analog detector (film), keeping essentially intact the other components of the imaging pipeline. This was the conventional wisdom when digital sensors began to enter the mainstream in the 1990s. On the other hand, a much more powerful question to ask is the following [1]: What kind of information should we record on the digital sensor so that, with suitable processing, we can reconstruct an image that may have better quality or there may be new functionality in the imaging system? Accordingly, this extra flexibility has three dimensions: the imaging hardware—primarily the optics—to decide what to 1084-7529/15/112021-12$15/0$15.00 © 2015 Optical Society of America
capture, the computational algorithm to process the data, and the application of such a computational imaging system. It is critical to note that images only emerge from computing on the information recorded; as such, the captured data do not necessarily have to be the 2D projection of the scene or the object, or anything visually identifiable at all. In fact, for different applications, there is a lot of room to investigate what to capture and to devise efficient algorithms for the image reconstruction [2–4]. B. Computational Photography
Photography is likely the most common form of optical imaging. From pinhole image formation, which was first described in writing during the Spring and Autumn Period in China (roughly from the eighth century B.C. to the fifth century B.C.) [1], to the first permanent photograph by the French inventor Joseph Nicéphore Niépce (the nineteenth century), photography has come a long way as an established technology and art form. With computation, how can we experience photography differently? One answer, which is the focus of this tutorial, is light field photography [5]. Conventionally, whether with films or digital sensors, what is recorded is the amount of light, irrespective of its direction. Light field photography attempts to use a new
2022
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
imaging system design, called a plenoptic camera [6], that can indeed discriminate and capture light from a few directions. Together with proper computational algorithms, capturing the light field gives us at least three advantages over the traditional camera [7]: to reconstruct images with small viewpoint changes, to compute a depth map using depth from focus, and to reconstruct images as if they have been focused at different planes, thus achieving refocus after data capture. There has been significant interest in and development of light field photography over the past decade, including the deployment of some commercial products, and it is now an opportune time to review the technology behind them. Here, we will do so primarily for the benefit of one with a background in geometric and Fourier optics.
Tutorial
design of a portable light field camera system. In explaining this design, we will go in depth with a spatial and a Fourier analysis of the image formation process. Up to this point, the effect of diffraction is ignored, so we can use the simpler representation of light rays. Section 4 goes beyond this and relates light field to the Wigner–Ville distribution under the scalar diffraction theory. The image formation process is also explained using Fourier optics. Whether using geometric or Fourier optics, we will see there is a trade-off between angular and spatial information, and providing a higher-resolution 2D image from the light field is an important issue. Section 5 is devoted to highlighting recent advances in light field photography, particularly in optical and digital superresolution.
C. Other Computational Imaging Systems
2. PLENOPTIC FUNCTION AND LIGHT FIELD
Before we delve into computational photography for the rest of this paper, it is worth pointing out that computation has also been explored for many forms of imaging, notably for machine vision inspection and biomedical applications. For instance, structured illumination with phase-shifting algorithms allows for high-resolution height measurement of the object under inspection [8–10]. Phase diversity is a powerful technique that estimates wavefront aberrations based on multiple captured images, which are mainly used to reconstruct a clear 2D object, but this principle can also be applied to 3D imaging [11]. In computed tomography, the captured data are Fourier transformed to form slices of the object’s spectrum; a reconstruction algorithm, whether it is filtered backprojection or a more advanced iterative method, delivers detailed images of the internal structures of a person [12,13]. A similar approach is also taken with modalities such as diffuse optical tomography (DOT) [14,15]. Meanwhile, optical coherence tomography (OCT), which is a popular bio-optical imaging modality based on Michelson interferometry, also benefits substantially from computation. A form known as spectral-domain OCT collects the data uniformly in wavelength and ordinarily requires resampling and Fourier transform to reconstruct the images [16,17]. Digital holographic microscopy, by its very nature, requires computational processing to deliver a visible image. The phase of the wavefront is converted as a modulation in the magnitude during the recording step; thus, a computational process is needed to reconstruct, for example, quantitative phase information or sectional images of the object [18,19]. These are representative of the emerging subfield of computational biophotonics [20].
A. Theory
D. Organization
This paper is organized as follows. In Section 2, we will explain the theory behind plenoptic function and light field, from the perspective of geometric optics. A ray-space representation will then be introduced, which helps us discuss the image formation process. The main purpose is to show the desirability of being able to capture the light field. Section 3 is then mostly concerned with camera implementation, which has the ability to discriminate light rays from several directions and therefore provides a sampling of the incoming light field. We motivate this from the historical perspective, leading to the first successful
Let us assume that we are dealing with incoherent light traveling around objects that are much larger than the wavelength of light. This allows us to use “rays”—parameterized chiefly by their locations and directions—to describe the image formation process. This is known as ray optics or, more commonly, geometrical optics because light propagation obeys a set of geometrical rules [21]. Specifically, in a homogeneous medium (such as free space), light rays propagate in straight lines. For the purpose of quantifying them and using them for computations, we need a mathematical function to describe each ray. In an attempt to understand the fundamental substances of human and machine vision, Adelson and Bergen in the early 1990s devised the plenoptic function to capture the information carried by the light rays [22]. The result is a 7D function, Pθ; ϕ; λ; t; x; y; z; where θ; ϕ are angular coordinates encoding the direction of the ray, λ is the wavelength, t is time, and x; y; z are the spatial coordinates of a point this ray passes through. This point is taken as the origin of the ray. The word “plenoptic” comes from “plenus” in Latin, meaning complete or full [22]. Thus, does the plenoptic function indeed contain complete information about an object? On one hand, we should remember that we are in the realm of geometrical optics; therefore, information regarding initial phase is absent in the plenoptic function. On the other hand, for our human visual system, what we observe are normally samples and projections of the plenoptic function. Arguably, we do not “see” an object directly; rather, our retina responds to the light rays emanated or reflected from such an object; therefore, the plenoptic function encapsulates the full information communicated to the observer. In fact, in the study of image formation, we often can reduce the dimensionality of the plenoptic function by arguing the following: 1. The function does not vary with time (so t can be omitted). 2. The light is monochromatic (so λ can be omitted). 3. The radiance along a ray is a constant (so only two parameters are needed for the location, effectively omitting z).
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
Tutorial
Furthermore, it is common to replace the angular coordinates θ; ϕ with Cartesian coordinates u; v. As depicted in Fig. 1, we assume that all light rays travel from left to right and pass by two planes along the way. A specific light ray intersects the first plane at coordinates u; v, which govern the location of the ray; it continues to travel and intersects the second plane at coordinates x; y, effectively specifying the propagation direction. Instead of calling this the plenoptic function, it is more common, thanks to the work of Levoy and Hanrahan [23], to refer to this 4D simplification as the light field, denoted
2023
represented by a point in the ray-space diagram. We can visualize a light field as a collection of such points. Figure 2 shows four illustrative situations, each corresponding to a discrete set of light rays:
Lu; v; x; y; which is also called the “lumigraph” in the computer graphics community [24]. Note that the term “light field” was first used in a translated work on illumination engineering in the 1930s [23,25], but this is not in the sense of an electromagnetic field, which underlines the principles of optics [26]. The value of Lu; v; x; y is the amount of light—known as the radiance— of the monochromatic light ray. When considering color image formation, we can assume that the light field is a threecomponent vector with radiance information on the red, green, and blue components or any corresponding wavelengths. In a typical camera, the two-plane parameterization may be applied where the first plane is the position of the main lens and the second plane lies in the electronic sensor. This, unfortunately, would not allow us to capture the light field; as the photodetectors respond to light from all directions, the angular information of the rays is lost. In Section 3, we will discuss a setup known as the plenoptic camera, with a somewhat more complicated and interesting geometry, that can achieve this. Before that, however, let us now turn our attention to a diagrammatic representation of the light field.
(a)
(b)
B. Ray-Space Representation
An exemplary way to plot the light field is to use the so-called ray-space diagram [5,27,28], which allows us to gain insight about different collections of light rays in a compact manner. For the purpose of illustration, we retain only one coordinate in each of the planes in Fig. 1. The light field is then represented as a 2D diagram, which is a plot of x, the spatial axis, versus u, the directional axis. Furthermore, consider the light ray Lu; x: it emanates from a specific u to a specific x and therefore is
(c)
(d)
Fig. 1. Two-plane parameterization of the light field [23].
Fig. 2. Illustrative plots of the ray-space diagram. (a) A regular array of light rays, from a set of points in the u plane to a set of points in the x plane. (b) A set of light rays arriving at the same x position. (c) A set of light rays approaching a location behind the x 0 plane. (d) A set of light rays diverging after converging at a location before the x 0 0 plane.
2024
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
1. As shown in Fig. 2(a), the light rays emanate from a set of regularly spaced u to another set of regularly spaced x. Accordingly, the ray-space diagram consists of a regular matrix of points. 2. As shown in Fig. 2(b), a collection of rays at different points on the u plane focusing onto the same point in the x plane become, in the ray-space, a vertical line. 3. As shown in Fig. 2(c), a collection of rays at different points on the u plane converge to a point at the x plane; thus, on the x 0 plane, which is nearer, these rays are getting closer, but their spatial locations are still different. In the ray-space diagram, these become a line tilted to the right. Another way to describe the phenomenon is that the ray-space has undergone a shearing. This observation is fundamental to plenoptic camera, and we will return to it below. 4. As shown in Fig. 2(d), a collection of rays at different points on the u plane converge to a point at the x plane; thus, on the x 0 0 plane, which is farther, these rays are again diverging to different spatial locations. Similar to the discussion above, in the ray-space diagram, they form a line tilted to the left; equivalently, the ray-space is also sheared but to a different direction.
Tutorial
(a)
C. From Light Field to Image
Suppose now we put a photodetector in the x; y plane. It captures the irradiance of the light, which is a summation of the incident radiance [29,30]; mathematically, this means we integrate the light field, i.e., ZZ ∞ 1 Lu; v; x; yT u; vdudv; (1) Ex; y 2 Z −∞ where Ex; y is the resulting image intensity and Z is the separation between the u; v and x; y planes. The term T u; v represents the aperture, which for a circular lens with radius R can be modeled as a circ function [31,32]: 1 if u2 v2 < R 2 T u; v circu; v (2) 0 otherwise: More precisely, there should be a cos4 θ term inside the integral, where θ is the angle between the light ray and the normal to the u; v and x; y planes; this is sometimes known as the cosine fourth law, representing the physical effect that rays from more oblique directions contribute less energy to the total irradiance [33]. This term, however, can be conveniently absorbed into the definition of the light field [5]. Equation (1) thus provides the linkage between the 4D light field and the 2D image capture in a conventional camera: the image is a projection of the light field onto the x; y axes. Let us now return to the cases depicted in Figs. 2(b)–2(d). Assume that we have placed a lens in the u plane; we can converge the light rays to a point in the x plane, as depicted in Fig. 2(b), if it is placed in the focal plane of the lens, with the use of the paraxial approximation. Now suppose we bring the x plane closer to the u plane. From Fig. 2(c), we know this will result in a tilted line in the ray-space; in addition, we have the necessary mathematics to quantify the angle of the tilt, as follows. Consider the geometry depicted in Fig. 3(a), where we have shrunk the distance between the two planes from Z to αZ . The ray intersects the new plane at x 0 , where x 0 − u αx − u, and therefore,
(b) Fig. 3. Bringing the x plane closer to the u plane results in a tilted line in the ray-space at an angle ψ. (a) Moving the second plane closer to the first, by a factor of α. (b) Corresponding shearing in ray-space, with ψ tan−1 1 − α.
x0 − u : α Thus, this ray can be expressed as x0 − u y0 − v ;v : Lu; v; x; y L u; v; u α α x u
(3)
(4)
We want to depict this in the ray-space diagram with x 0 and u. From Eq. (3), we can also express x 0 in terms of x such that x 0 αx 1 − αu:
(5)
In the ray-space diagram with x 0 and u, this is a straight line with slope 1∕1 − α. We normally define the shear angle ψ as the angle with the vertical line, which is π 1 −1 tan−1 1 − α: (6) ψ − tan 2 1−α This is depicted in Fig. 3(b). Because we are concerned with light rays only, matrix optics offers an alternative to computing the tilting angle [34]. Let us refer back to Fig. 3(a). Now consider two light rays, one starting at position u1 , emanating at an angle θ1 , passing through x 10 , before arriving at x; similarly, the second ray starts at u2 with an angle θ2 , passing through x 20 , before arriving at the same x. With the ray-transfer matrices [21], which make use of the paraxial approximation, we can write 0 1 Z x 1 αZ x1 u1 ; (7) 0 1 θ1 0 1 θ1 θ1
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
Tutorial
u2 θ2
1 Z 0 1
x θ2
1 0
αZ 1
x 20 : θ2
(8)
Therefore, u1 x Z θ1 x 10 αZ θ1 ;
(9)
u2 x Z θ2 x 20 αZ θ2 :
(10)
Δx 0
x 20
x 10 ,
− then, taking the If we define Δu u2 − u1 and difference between the above two equations, we have u2 − u1 Z θ2 − θ1 x 20 − x 10 αZ θ2 − θ1 ;
(11)
which means Δu Δx 0 αΔu:
(12)
Referring back to Fig. 3(b), the tilting angle is controlled by Δu and Δx 0 , where 0 Δx tan−1 1 − α; ψ tan−1 (13) Δu arriving at the same result as in Eq. (6). We can now summarize the process of forming a 2D image from a light field. • Capturing an image is essentially integrating the light field over the angular variables u; v, as Eq. (1) suggests; in the rayspace representation, this is projected onto the horizontal axis. • If we adjust the placement of the electronic sensor, the light field will undergo a shear; bringing the sensor closer to the lens will result in a shear to the right by an angle ψ given by Eq. (6); putting the sensor farther away will have the opposite effect. • If x; y represents the ideal focusing plane with respect to an object, then moving the sensor to another plane x 0 ; y 0 means we first shear the light field and then still project to the horizontal axis. We know by experience that this gives a blurred image. Consequently, if we are able to capture the light field itself instead of its projection, we can reconstruct what it would look like at any plane by shearing it to a different extent and projecting it onto the horizontal axis or, alternatively, simply projecting the light field at various angles [35]. But how would we capture the 4D light field using a 2D sensor? It turns out that answering this question has a history that dates back to over a century ago. 3. PLENOPTIC CAMERA SYSTEMS A. Early Developments
The first attempt to capture the light field is traced to the work of Gabriel Lippmann, the 1908 Nobel Laureate in physics for his invention of color photography based on interference. In that same year of his award, he also published a paper describing a method known as “photographies intégrales”—i.e., integral photography [36]. His chief concern was the development of a stereoscopic display, but he needed a way to record an object for this purpose. He made use of an array of small lenses, with images forming on the film behind the lenses, each having a slightly different perspective. Later on, with backlit diffused light, one can view these subimages together with a stereoscopic
2025
view. We now understand that, taken as a whole, the recorded images in Lippmann’s scheme preserve a sampling of the light field of the object, but the image formation process was not studied in this way during his days. Unfortunately, Lippmann’s idea did not take off for many years, primarily because of the technical difficulties in creating the lens array. (These days, though, with the widespread availability of high-resolution cameras and spatial light modulators, integral imaging has become quite possible and is an active research area [37,38].) In the 1930s, Herbert Ives, a charter member and past president of OSA, improved on Lippmann’s design in two aspects. First, he added a main lens with a large diameter [39]. Second, he analyzed the original approach and found it to be pseudoscopic rather than stereoscopic. He then proposed a secondary exposure to invert the depth [40]. It was not until a few decades later that Chutjian and Collier developed a one-step imaging solution and made possible the first computer-generated integral imagery [41]. The next significant phase of the development emerged primarily from the computer vision and computer graphics communities. Soon after defining the plenoptic function [22], Adelson and Wang outlined an image capture system to collect such data and named it the plenoptic camera [42]. Their objective was to infer the object depths, achieving what is known as a single lens stereo. The “single lens” here refers to the single main lens; in addition, there is a lenticular array (alternatively, pinhole array) in front of the sensor, so that the data recorded on the photodetectors correspond to images from slightly different perspectives. With these subimages, the authors were able to produce depth maps and reconstruct images from different viewing positions [42]. Later, after the “light field” was defined as a simplification of the plenoptic function, it was demonstrated that a camera array could record the light field of an object [43]. Similar to the above, one use of the data was to reproduce images from various viewpoints, a method known as image-based rendering [44]. Another use was to simulate defocus blur, thus synthesizing (or reconstructing) images as if they were captured with a large aperture; this technique is called synthetic aperture photography [45]. The camera array, however, was very large [46,47] and therefore not portable, thus limiting its use to a more controlled environment. A very different idea from those noted above is the development of specialized sensors. In particular, recent advances in complementary metal oxide semiconductor (CMOS) processes have made angle-sensitive photodetectors possible [48]; these photodetectors use pixel structures based on the Talbot effect to permit the capturing of angular light radiance [49,50]. One additional line of technological developments should also be mentioned here, although the purpose of these innovations was not to capture the light field. Instead, the researchers were motivated to learn from nature and, thus, explored the compound eye—commonly found in many insects—as an alternative form of imaging systems [51]. Such an artificial compound eye was realized using a microlens array in front of the digital sensor [52,53], and developments in the design and fabrication of these microlenses would prove to be useful for a portable light field camera later on. Representative work in
2026
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
compound-eye imaging includes Tanida et al. [54,55], who built the TOMBO system, short for thin observation module by bound optics (the name TOMBO also means “butterfly” in Japanese); Chan et al. [56–58], who introduced diversity in the captured subimages and developed a superresolution algorithm for the image reconstruction; and Plemmons et al. [59], who built the complete optical hardware and software systems. They also studied the theoretical and practical performance of the resulting prototype called PERIODIC (practical enhancedresolution-integrated optical-digital imaging camera) [60]. A major advantage was that one could achieve a very thin imaging system; unfortunately, the image quality was often not comparable with that of a traditional camera system with a main lens due to diffraction effects [5]. B. Portable Light Field Camera 1. Placement of the Lenses and Photodetectors
The first design of a portable light field camera can be thought of as a far-field version of integral imaging [61], a miniaturization of the camera array system [43], or a hybridization of simple and compound eyes. The optical system of such a camera is shown in Fig. 4. There are four planes of interest here: 1. The object plane, labeled ζ; η. 2. The aperture plane, which is also the location of the exit pupil. We label it the u; v plane. 3. The imaging plane, with a regular matrix of microlenses. The size of the individual microlens determines the spatial sampling resolution [5]. We label it the x; y plane. 4. The photodetector array plane, labeled σ; τ. The key to an effective light field capture is the precise positioning of these planes [6,62]. First, let us consider the red lines in Fig. 4, which trace the light rays all the way from the object to the photodetector. The object and the main lens are separated by a distance z 0 , and the main lens and the lenslet array are separated by a distance z 1 . As the figure illustrates, the lenslet array is placed where the object is brought to focus; in other words, for a conventional camera, in lieu of the microlenses, we would have the photodetectors there in order to obtain a sharp image. Thus, if the main lens has a focal length f 1 , then
Tutorial 1 1 1 ; z0 z1 f 1
(14)
thus obeying the well-known lens equation. In this plenoptic camera, the photodetector array is placed a certain distance z 2 behind the lenslet array. Accordingly, the light rays that converge at a microlens will again diverge and impinge on the photodetectors at different locations. The recorded image then provides directional resolution of the object. To maximize this, we should ensure the sharpest microlens images, which is achieved by focusing the microlenses on the principal plane of the main lens [6]. This is illustrated by the blue lines in Fig. 4. Because the microlenses are very small compared with the main lens, z 1 is effectively at optical infinity. Consequently, if the microlenses have a focal length f 2 , then z2 f 2:
(15)
Furthermore, we must take care to ensure that the light rays passing through one microlens should sufficiently cover the photodetectors behind it, so we do not waste the pixels. On the other hand, to avoid cross talk, the light rays also should not extend to the photodetectors associated with the adjacent microlenses. This can be achieved if the image side f -number of the main lens (its diameter, d 1 , divided by z 1 ) and the f -number of the microlenses (their diameters, d 2 , divided by z 2 ) are equal [5], i.e., d1 d2 : (16) z1 z2 These specific optical configurations would now allow us to capture a filtered and sampled light field of an object with the photodetector array: the sampling is because there are only a finite number of pixels and microlenses, and the filtering is due to their finite dimensions. There are two complementary perspectives to see why this is the case. First, associate the region on the photodetectors behind each microlens as a subarray image and treat each microlens as a virtual pixel; the subarray image then captures the directional information of the light rays corresponding to that pixel. Mathematically, the subarray image is a collection of the light field Lu; v; x; y with different u; v but the same x; y. Second, consider the two sets of blue lines in Fig. 4. They refer to two pixels on the photodetector array that correspond to the same location in the aperture. We can refer to the collection of such corresponding pixels as a subaperture image. Mathematically, the subarray image is a collection of the light field Lu; v; x; y with the same u; v but different x; y. Therefore, taken together, the photodetectors are sampling the light field from the object. (Readers interested in experimenting with light field data and visualizing these different image representations can download publicly available light field data sets, such as from the Stanford Light Field Archive, or create their own data using plenoptic cameras that are now commercially available.) 2. Spatial Analysis of Image Formation
Fig. 4. Light field camera system. Light rays marked in red show how the microlenses separate them, so the photodetector array can capture a sampling of the light field. Light rays marked in blue show that the photodetector array can also be thought of as recording the images of the exit pupil.
Because the light field of an object is captured in the photodetectors in Fig. 4, we can make use of the discussions in Section 2.C to compute a 2D image from such data. The simplest way is to sum up the values from the photodetectors in each subarray image, which gives the intensity of the virtual
Tutorial
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
pixel at each microlens position; taking these virtual pixel values together would form an image of the object, albeit often at a low resolution. This corresponds to the case depicted in Fig. 2(b). However, a major advantage of recording the light field, as opposed to the intensity image, is the ability to support postcapture refocus of the object [5,62]. Consider the situation shown in Fig. 2(c), where we are recording the light field on the x 0 plane. We know from the earlier discussion that this light field Lu; x 0 is essentially the same as the light field Lu; x after undergoing a shearing operation. Thus, if we first compute the angle ψ using Eq. (6) and then project Lu; x 0 at this angle, as illustrated in Fig. 5, we can reconstruct an image as if the photodetectors were placed at the (x) plane. Overlaying this analysis on Fig. 4, we can see that this tilted summation of the captured light field can lead to an image as if it were taken directly with a sensor a certain distance behind the lenslet array plane. This increases the image distance and, consequently, shortens the object distance, as a result of the lens equation. Similarly, the situation depicted in Fig. 2(d) shows that we can likewise reconstruct an image as if it were taken directly with a sensor some distance in front of the lenslet array plane, corresponding to focusing at the part of the object that is farther from the main lens. Figure 6 shows an example of post-capture refocus from a light field captured directly in a commercial plenoptic camera [6,63]. A set of tools has been made available to decode the light field data and compute the image reconstruction with different parameters [64,65]. A byproduct of such post-capture refocusing is the ability to produce an extended depth-of-field (EDOF) image, which is desirable in many applications [66–68]. In principle, each of the subaperture images already has a higher depth of field compared with an ordinary photograph using the same main lens. This is because the subaperture image, as the name suggests, uses only a smaller part of the aperture, which leads to a larger effective f -number. However, the signal-to-noise ratio (SNR) is usually quite poor because the majority of the captured data is discarded. A better alternative is to refocus the image on a pixel basis, selecting the focus point to be on the depth of the closest object. This can be achieved by reconstructing a stack of images corresponding to different sensor locations and making use of a digital photomontage algorithm to combine them to form a single EDOF image [5,69].
Fig. 5. Tilted projection of the light field.
2027
Fig. 6. Example of post-capture refocus from custom light field data. (a) Image reconstruction with focus at the front. (b) Image reconstruction with focus at the back.
3. Fourier Analysis of Image Formation
The above discussion suggests that image reconstruction is achieved by projecting the 4D light field onto a 2D space at an angle that determines the focus distance of the resulting image. This projection can be computed using numerical integration, by identifying which photodetectors lie along the “line of projection” and summing them up accordingly. However, this process often involves computing the appropriate weights of the photodetector values and may even include resampling of the data, which can be substantial with a large light field. In this section, we explain how this projection of the light field can alternatively be achieved in the Fourier domain, which can lead to asymptotically lower complexity algorithm and further insight into the imaging process. The method is called Fourier slice photography [70]. The mathematical basis is the projection-slice theorem by Bracewell [31,71]. In two dimensions, this well-known theorem states that a projection, i.e., line integration, of a 2D function along an angle ϕ is related to a slice, again at angle ϕ, of the 2D Fourier transform of the function through a 1D Fourier transform. This is represented diagrammatically in Fig. 7. The generalization to higher dimensions is rather straightforward, and we will be especially concerned with projecting 4D
2028
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
Tutorial
reasonable to ask, then, if we can arrive at a similar conclusion if we consider light as electromagnetic waves. Specifically, what may be the effect of diffraction? This question is particularly pertinent if we would like to scale down the dimension of the plenoptic camera, such as in the construction of the light field microscope [76–78]. It has been pointed out that the light field can be identified as the phase-space of geometric optics [79], and the Wigner–Ville distribution can then be used to generalize the phase-space formalism. More specifically, we will show how to relate the light field to the Wigner–Ville distribution, and we can also develop Fourier optics expressions for the plenoptic camera system. Fig. 7. Illustration of the projection-slice theorem. Projection at angle ψ of a 2D function in the x − u plane, which becomes a 1D function in ρ, is related to its 2D Fourier transform in the f x − f u plane sliced at angle ψ by a 1D Fourier transform.
functions to 2D [72]. Remember that our interest is in a tilted projection of the light field; the main result from Fourier slice photography is that such image refocusing, in the Fourier domain, is simply obtained by slicing the 4D Fourier transform of the light field at different angles [70,73,74]. There is, however, an additional mathematical detail regarding the shearing operation. It involves a basis transform that is not orthonormal [75]. Observe from Eq. (5) that 2u3 2 1 0 0 0 32 u 3 6v7 6 0 4 05 4 x 1−α y0 0
1 0 1−α
0 α 0
and therefore the basis transform matrix 2 1 0 0 1 0 6 0 B4 1−α 0 α 0 1−α 0
0 76 v 7 54 5; x 0 y α 03
(18)
Thus, we need to generalize the projection-slice theorem to include such basis transform. Specifically, Ng showed that the Fourier domain would need a change of basis involving the normalized inverse transpose of B, which is [70] 21 0 1 − 1 0 3 B −T 60 α2 4 0 jB −T j 0
1 0 0
0 1 α
0
α
1 − α1 7 5: 0
We make use of the scalar diffraction theory, which is accurate as long as the diffracting structures are large when compared with the wavelength of light [32]. For simplicity, we also limit the discussion to monochromatic waves, where the extension to polychromatic light with different degrees of coherence is possible, though slightly more involved [80,81]. Let us now consider a scalar field at a plane s; t and denote it, in phasor notation, as Us; t. The 4D Wigner–Ville distribution (often simply called the Wigner distribution) is given by [82,83] ZZ W U s; t; ξ; ν
(17)
is
07 5: 0 α
A. Wigner–Ville Distribution
(19)
1 α
With this, the projection of the light field is computed in the Fourier domain by first applying the full 4D Fourier transform to the light field data. Then, the basis transform using Eq. (19) is computed on the light field spectrum. We then extract a 2D slice of the signal, which corresponds to setting the other two dimensions to zero. Finally, we calculate an inverse 2D Fourier transform on the slice. The result, by the projection-slice theorem, is equivalent to the direct projection of the light field data. 4. WAVE OPTICS Thus far, we have analyzed the plenoptic imaging system entirely with geometric optics by tracing the light rays. It is
∞
−∞
×e
α β α β U s ; t U s − ; t − 2 2 2 2
−j2παξβν
dαdβ;
(20)
where ξ; ν are spatial frequency coordinates. Our goal is to relate the light field representation with this Wigner–Ville distribution. Unfortunately, this cannot be done directly because light must propagate isotropically with an infinitesimal pinhole [80]. Thus, while a light field can specify a precise direction, with Wigner–Ville distribution, we must take into account a range of spatial frequencies. The trick is to insert a finite aperture; moving it across the plane results in a generalization known as the observable light field [80]. (The “augmented light field” is another possible generalization of the light field and is used in computational photography analysis as well; for details, we refer readers to [84].) This methodology is depicted in Fig. 8. Assume an aperture function T s; t is translated to be centered at u; v, so the scalar field after the aperture is ˜ t Us; tT s − u; t − v: Us;
(21)
Its angular spectrum, which is the Fourier transform of the scalar field [32], is given by ZZ ∞ y x y x Us;tT s − u;t − ve −j2πλsλt dsdt: A ; λ λ −∞
(22)
The observable light field [from the point u; v at direction x; y, denoted by Lobs u; v; x; y] is given by the magnitude squared of the angular spectrum [80]; thus,
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
Tutorial
2029
Next, we consider what happens after passing through the lenslet array. Assume there are N × N identical microlenses. As each has diameter d 2 and focal length f 2 , the propagations after the lenses are given by [32] ˜ y Ux; y Ux;
Z Z Z Z
π
× e −jλf 2 x−md 2 y−nd 2 ; (23)
Us1 ;t 1 U s2 ;t 2 T s1 −u;t 1 −v −∞ y x ×T s 2 −u;t 2 −ve −j2π λs1 −s2 λt 1 −t 2
×ds1 dt 1 ds 2 dt 2 (24) x y x y W U u; v; ; W T −u; −v; ; ; (25) λ λ λ λ where we arrive at the last line by rewriting the integration variables into averages and differences, as given in [80]. We can therefore see that the observable light field is the smoothed Wigner–Ville distribution, where the blur is given by a Wigner–Ville distribution of the aperture. This trades off resolution in position with direction: the smaller the aperture, the more precise we can be about the position, but the less definite we are with the spatial frequency. Furthermore, when a wavelength becomes negligibly small (bringing us back to the domain of geometric optics), the Wigner–Ville distribution of the aperture is a delta function, and the observable light field is then equivalent to the Wigner–Ville distribution of the wave function. B. Image Formation
We can also derive wave optics expressions for the plenoptic camera image formation process. For simplicity of discussion, we consider the setup as given in Fig. 4 (expressions with more general distance values can be found in [85]). Let the scalar field at the object plane be denoted as Uζ; η. The propagation from the object plane to the lenslet array follows that of a standard imaging setup, with the latter positioned at the normal “image plane.” Thus, we can readily know that the impulse response of this part of the imaging system is given by the Fraunhofer diffraction pattern of the main lens pupil, P 1 u; v, i.e., [32] ZZ ∞ 2π P 1 u; ve −jλz 1 x−M ζuy−M ηv dudv; h1 x; y; ζ; η K −∞
(26) where K is a constant and M is the magnification of the system given by M −z 1 ∕z 0. The scalar field in front of the lenslet array is then ZZ ∞ h1 x; y; ζ; ηUζ; ηdζdη (27) Ux; y −∞
2
2
(28)
where P 2 x; y is the pupil function of the microlens. Finally, the propagation from the lenslet array to the photodetector array is given by a Fresnel propagation with impulse response 2π
∞
because of linearity of wave propagation.
P 2 x − md 2 ; y − nd 2
m1 n1
Fig. 8. Observable light field.
2 x y Lobs u;v;x;y A ; λ λ
N X N X
e j λ z 2 λzjπ σ2 τ2 e 2 ; h2 σ; τ jλz 2 and the scalar field at the photodetector array is ZZ ∞ ˜ ydxdy; h2 σ − x; τ − yUx; Uσ; τ
(29)
(30)
−∞
again because of superposition [32]. To reconstruct an image, a possible method is to backpropagate the field from the photodetector array to an arbitrary object plane. Reference [86] describes how this can be done with coherent and incoherent approaches. 5. LIGHT FIELD SUPERRESOLUTION Whether we analyze the plenoptic camera system using rays or waves, it is evident that the imaging system trades off spatial resolution with angular information in the light field [87]. This is because it takes a collection of captured pixel values together to reconstruct the intensity at a specific location, due to the projection operation described in Section 3.B.2. For example, one of the first consumer light field cameras is equipped with a sensor having 11 million photodetectors, capturing “11 mega-rays,” but typically can only reconstruct an image under 1 megapixel [63]. To tackle this problem, various methods have been developed, which are aimed to achieve better resolution from a plenoptic camera. We can, for example, capture more light field data by multiplexing many views, an idea explored in the programmable aperture camera [88]. This technique, however, requires a long acquisition time, and the object needs to be static. Combining a light field camera with a traditional camera is also a possibility [7]. Alternatively, without increasing the capture data volume, there are still broadly two major approaches: with changes in the way light field data are captured and with more powerful image superresolution algorithms. Below, we highlight the representative methods. A. Optical Methods
The major limitation to spatial resolution is the physical dimensions of the microlens array. In addition, the microlenses require precise alignment and still may introduce errors such as spherical aberration and coma as well as chromatic aberration. A very different method of capturing the light field is to replace the microlens array with an attenuation mask placed between the main lens and the sensor [74]. This design, known as the dappled photography camera system, is shown in Fig. 9.
2030
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
The attenuation mask, which consists of a high-frequency sinusoidal pattern, serves the purpose of heterodyne encoding of the light rays onto the photodetector. In other words, instead of recording each light ray with a pixel, dappled photography captures a linearly independent weighted sum of light rays, combined in a coded fashion, which can be decoded in the image reconstruction process [74]. A major advantage of this design, compared with the plenoptic camera system in Fig. 4, is that it can reconstruct a high-resolution image at the plane in focus. This comes at a price, however; the signal-to-noise ratio (SNR) is further reduced because of the light attenuation at the mask [89]. An alternative plenoptic camera system also exists with the use of a lenslet array but with different placement of the optical elements. Figure 10 shows such a design known as the focused plenoptic camera [90] (the authors also call it the plenoptic 2.0 camera [91]). Recall that, in the plenoptic 1.0 camera (i.e., Fig. 4), the object is imaged at the microlens array plane, and the microlenses focus at the main lens, which is effectively at optical infinity. In contrast, with the focused plenoptic camera, the object is imaged at a plane in front of the lenslet array, and the microlenses then focus on this image, producing a relay system to reimage it onto the sensor. This means that the photodetector has been moved farther from the main lens. Remember that f 2 is the focal length of the microlenses; then 1 1 1 ; z 1B z 2 f 2
(31)
where z 1B is the distance between the main lens image plane and the lenslet array and z 2 is the distance between the latter and the photodetector, as shown in Fig. 10. Consequently, the angular information for any spatial location is sampled by different microlenses. Furthermore, the spatioangular sampling is determined by z 1B and z 2 , as long as Eq. (31) is satisfied; this alleviates the constraint on packing more microlenses, as their density no longer determines the spatial resolution. Instead, we can use larger microlenses to produce higher-quality images, with fewer boundary pixels resulting in a more efficient use of the sensor [90]. This flexibility in spatioangular sampling also results in higher-resolution reconstructed images [92,93]. B. Computational Methods
Following a long tradition in image reconstruction and restoration algorithms [94], we can similarly make use of prior
Tutorial
Fig. 10. Focused plenoptic camera system. Object is focused at a plane in front of the lenslet array, and the microlenses then act as a relay system with the main camera lens.
information about the light field and the image in processing the raw data from the photodetectors [95,96]. For instance, one can model the light field patches using a Gaussian mixture model to achieve light field superresolution [97] or, alternatively, assume a Lambertian scene object in the image formation model and use a Markov random field prior in the reconstruction [89]. One can also make use of the spatial relationship in the subarray images [98,99] and make use of disparity maps to generate superresolved views of an object, which correspond to increasing the sampling rate of the light field in spatial and angular directions [100]. Many computational approaches, in fact, are coupled with changes in the optical system in the data capture. A representative method is the placement of a second attenuation mask at the main lens aperture in Fig. 9. In this design, the two masks together modulate and encode the light field spectrum, and the computational reconstruction takes advantage of the coherence inherent in the angular dimensions of the light field by optimizing with a total variation (TV) norm regularizer [73,101]. Along a similar vein, it has been argued that the natural light field is sparse. Hence, making use of compressed sensing and dictionary learning [102–104], it is possible to construct a compressive light field camera that captures optimized 2D light field projections and design robust sparse reconstruction to recover a higher-resolution light field [105]. 6. CONCLUSIONS The development of light field cameras, as well as the associated image reconstruction and superresolution algorithms, has significantly enriched our arsenal of imaging tools, particularly for computational photography. However, as pointed out by Goodman, light field photography can also shed light on other imaging modalities, such as digital holography, and vice versa [106]. Looking into the future, it would not be surprising to see more applications emerging because of our ability to capture light fields, and biophotonics imaging is certainly an area that has a lot of potential to benefit from this new imaging paradigm. REFERENCES
Fig. 9. Dappled photography with heterodyned light field capture using an attenuation mask. The mask is typically placed close to the detector, i.e., with a small z 2 [74].
1. E. Y. Lam, “Computational photography: advances and challenges,” Proc. SPIE 8122, 81220O (2011). 2. D. R. Gerwe, A. Harvey, and M. E. Gehm, “Computational optical sensing and imaging: Introduction to feature issue,” Appl. Opt. 52, COSI1–COSI2 (2013).
Tutorial
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
3. F. H. Imai, D. C. Linne von Berg, T. Skauli, S. Tominaga, and Z. Zalevsky, “Imaging systems and applications: Introduction to the feature,” Appl. Opt. 53, ISA1–ISA2 (2014). 4. E. Y. Lam, G. Bennett, C. Fernandez-Cull, D. Gerwe, M. Kriss, and Z. Zalevsky, “Imaging systems and signal recovery: introduction to feature issue,” Appl. Opt. 54, IS1–IS2 (2015). 5. R. Ng, “Digital light field photography,” Ph.D. thesis (Stanford University, 2006). 6. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. CSTR 2005-02 (Stanford University Computer Science Department, 2005). 7. V. Boominathan, K. Mitra, and A. Veeraraghavan, “Improving resolution and depth-of-field of light field cameras using a hybrid imaging system,” in IEEE International Conference on Computational Photography (IEEE, 2014), pp. 1–10. 8. F. Deng, W. F. Sze, J. Deng, K. S. Fung, W. Leung, and E. Y. Lam, “Regularized multiframe phase-shifting algorithm for threedimensional profilometry,” Appl. Opt. 51, 33–42 (2012). 9. F. Deng, C. Liu, W. Sze, J. Deng, K. S. Fung, W. Leung, and E. Y. Lam, “Illumination-invariant phase-shifting algorithm for three-dimensional profilometry of a moving object,” Opt. Eng. 51, 097001 (2012). 10. F. Deng, C. Liu, W. Sze, J. Deng, K. S. Fung, and E. Y. Lam, “An INSPECT measurement system for moving objects,” IEEE Trans. Instrum. Meas. 64, 63–74 (2015). 11. P. Kner, “Phase diversity for three-dimensional imaging,” J. Opt. Soc. Am. A 30, 1980–1987 (2013). 12. R. N. Bracewell, Two-Dimensional Imaging (Prentice-Hall, 1995). 13. J. L. Prince and J. M. Links, Medical Imaging Signals and Systems, 2nd ed. (Prentice-Hall, 2014). 14. D. Boas, D. Brooks, E. Miller, C. DiMarzio, M. Kilmer, R. Gaudette, and Q. Zhang, “Imaging the body with diffuse optical tomography,” IEEE Signal Process. Mag. 18, 57–75 (2001). 15. P. Bonfert-Taylor, F. Leblond, R. W. Holt, K. Tichauer, B. W. Pogue, and E. C. Taylor, “Information loss and reconstruction in diffuse fluorescence tomography,” J. Opt. Soc. Am. A 29, 321–330 (2012). 16. J. F. de Boer, B. Cense, B. H. Park, M. C. Pierce, G. J. Tearney, and B. E. Bouma, “Improved signal-to-noise ratio in spectral-domain compared with time-domain optical coherence tomography,” Opt. Lett. 28, 2067–2069 (2003). 17. J. Ke and E. Y. Lam, “Image reconstruction from nonuniformly spaced samples in spectral-domain optical coherence tomography,” Biomed. Opt. Exp. 3, 741–752 (2012). 18. M. K. Kim, Digital Holographic Microscopy: Principles, Techniques, and Applications (Springer, 2011). 19. E. Y. Lam, X. Zhang, H. Vo, T.-C. Poon, and G. Indebetouw, “Threedimensional microscopy and sectional image reconstruction using optical scanning holography,” Appl. Opt. 48, H113–H119 (2009). 20. L. V. Wang and H.-I. Wu, Biomedical Optics: Principles and Imaging (Wiley, 2007). 21. B. E. Saleh and M. C. Teich, Fundamentals of Photonics, 2nd ed. (Wiley, 2007). 22. E. H. Adelson and J. R. Bergen, “The plenoptic function and the elements of early vision,” in Computational Models of Visual Processing (MIT, 1991), pp. 3–20. 23. M. Levoy and P. Hanrahan, “Light field rendering,” in Proceedings of SIGGRAPH (ACM, 1996), pp. 31–42. 24. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” in Proceedings of SIGGRAPH (ACM, 1996), pp. 43–54. 25. A. Gershun, “The light field,” J. Math. Phys. 18, 51–151 (1939). Originally published in Moscow in 1936. 26. M. Born and E. Wolf, Principles of Optics, 7th ed. (Cambridge, 1999). 27. T. Fujii, T. Kimoto, and M. Tanimoto, “Ray space coding for 3D visual communication,” in Picture Coding Symposium (1996), pp. 447–451. 28. T. Fujii and M. Tanimoto, “Free viewpoint TV system based on rayspace representation,” Proc. SPIE 4864, 175–189 (2002). 29. R. Ramamoorthi and P. Hanrahan, “On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object,” J. Opt. Soc. Am. A 18, 2448–2459 (2001).
2031
30. H. H. Barrett and K. J. Myers, Foundations of Image Science (Wiley, 2004). 31. R. N. Bracewell, The Fourier Transform and Its Applications, 3rd ed. (McGraw-Hill, 2000). 32. J. W. Goodman, Introduction to Fourier Optics, 3rd ed. (Roberts and Company, 2004). 33. B. K. Horn, Robot Vision (MIT, 1986). 34. K. Halbach, “Matrix representation of Gaussian optics,” Am. J. Phys. 32, 90–108 (1964). 35. A. Isaksen, L. McMillan, and S. J. Gortler, “Dynamically reparameterized light fields,” in Proceedings of SIGGRAPH (ACM, 2000), pp. 297–306. 36. M. Lippmann, “Photographie—Épreuves réversibles. photographies intégrales,” Académie des Sci. 146, 446–451 (1908). 37. J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. 48, H77–H94 (2009). 38. X. Xiao, B. Javidi, M. Martínez-Corral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications,” Appl. Opt. 52, 546–560 (2013). 39. H. E. Ives, “Parallax panoramagrams made with a large diameter lens,” J. Opt. Soc. Am. 20, 332–340 (1930). 40. H. E. Ives, “Optical properties of a Lippman lenticulated sheet,” J. Opt. Soc. Am. 21, 171–176 (1931). 41. A. Chutjian and R. J. Collier, “Recording and reconstructing threedimensional images of computer-generated subjects by Lippmann integral photography,” Appl. Opt. 7, 99–103 (1968). 42. E. H. Adelson and J. Y. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 99–106 (1992). 43. V. Vaish, B. Wilburn, N. Joshi, and M. Levoy, “Using plane + parallax for calibrating dense camera arrays,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2004), Vol. I, pp. 2–9. 44. R. Szeliski, Computer Vision: Algorithms and Applications (Springer, 2011). 45. M. Levoy, B. Chen, V. Vaish, M. Horowitz, I. McDowall, and M. Bolas, “Synthetic aperture confocal imaging,” in Proceedings of SIGGRAPH (2004), Vol. 23, pp. 825–834. 46. B. Wilburn, N. Joshi, V. Vaish, M. Levoy, and M. Horowitz, “Highspeed videography using a dense camera array,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2004), Vol. II, pp. 294–301. 47. B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy, “High performance imaging using large camera arrays,” ACM Trans. Graph. 24, 765–776 (2005). 48. M. Hirsch, S. Sivaramakrishnan, S. Jayasuriya, A. Wang, A. Molnar, R. Raskar, and G. Wetzstein, “A switchable light field camera architecture with angle sensitive pixels and dictionary-based sparse coding,” in IEEE International Conference on Computational Photography (IEEE, 2014), pp. 1–10. 49. A. Wang, P. Gill, and A. Molnar, “Light field image sensors based on the Talbot effect,” Appl. Opt. 48, 5897–5905 (2009). 50. A. Wang and A. Molnar, “A light-field image sensor in 180 nm CMOS,” IEEE J. Solid-State Circuits 47, 257–271 (2012). 51. M. Land, “The optics of animal eyes,” Contemp. Phys. 29, 435–455 (1988). 52. S. Ogata, J. Ishida, and T. Sasano, “Optical sensor array in an artificial compound eye,” Opt. Eng. 33, 3649–3655 (1994). 53. J. S. Sanders and C. E. Halford, “Design and analysis of apposition compound eye optical sensors,” Opt. Eng. 34, 222–235 (1995). 54. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, “Thin observation module by bound optics (TOMBO): concept and experimental verification,” Appl. Opt. 40, 1806–1813 (2001). 55. J. Tanida, R. Shogenji, Y. Kitamura, K. Yamada, M. Miyamoto, and S. Miyatake, “Color imaging with an integrated compound imaging system,” Opt. Express 11, 2109–2117 (2003). 56. W.-S. Chan, E. Y. Lam, and M. K. Ng, “Investigation of computational compound-eye imaging system with super-resolution reconstruction,” in International Conference on Acoustics, Speech, and Signal Processing (IEEE, 2006), pp. 1177–1180.
2032
Vol. 32, No. 11 / November 2015 / Journal of the Optical Society of America A
57. W.-S. Chan, E. Y. Lam, and M. K. Ng, “Extending the depth of field in a compound-eye imaging system with super-resolution reconstruction,” in International Conference on Pattern Recognition (IEEE, 2006), pp. 623–626. 58. W.-S. Chan, E. Y. Lam, M. K. Ng, and G. Y. Mak, “Super-resolution reconstruction in a computational compound-eye imaging system,” Multidimension. Syst. Signal Process. 18, 83–101 (2007). 59. R. J. Plemmons, S. Prasad, S. Matthews, M. Mirotznik, R. Barnard, B. Gray, V. P. Pauca, T. C. Torgersen, J. van der Gracht, and G. Behrmann, “PERIODIC: Integrated computational array imaging technology,” in OSA Topical Meeting in Computational Optical Sensing and Imaging (Optical Society of America, 2007), paper CMA1. 60. S. Prasad, “Digital superresolution and the generalized sampling theorem,” J. Opt. Soc. Am. A 24, 311–325 (2007). 61. H. Navarro, J. Barreiro, G. Saavedra, M. Martínez-Corral, and B. Javidi, “High-resolution far-field integral-imaging camera by double snapshot,” Opt. Express 20, 890–895 (2012). 62. Y.-R. Ng, P. M. Hanrahan, M. S. Levoy, and M. A. Horowitz, “Imaging arrangements and methods therefor,” U.S. patent US8395696 B2 (March 12, 2013). 63. T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, “Lytro camera technology: theory, algorithms, performance analysis,” Proc. SPIE 8667, 86671J (2013). 64. D. G. Dansereau, O. Pizarro, and S. B. Williams, “Decoding, calibration and rectification for lenselet-based plenoptic cameras,” in IEEE Conf. Comput. Vis. Pattern Recogn., Portland, Oregon, 2013, pp. 1027–1034. 65. D. G. Dansereau, O. Pizarro, and S. B. Williams, “Linear volumetric focus for light field cameras,” ACM Trans. Graph. 34, 1–20 (2015). 66. E. R. Dowski and W. T. Cathey, “Extended depth of field through wave-front coding,” Appl. Opt. 34, 1859–1866 (1995). 67. E. Ben-Eliezer, E. Marom, N. Konforti, and Z. Zalevsky, “Experimental realization of an imaging system with an extended depth of field,” Appl. Opt. 44, 2792–2798 (2005). 68. P. Zammit, A. R. Harvey, and G. Carles, “Extended depth-of-field imaging and ranging in a snapshot,” Optica 1, 209–216 (2014). 69. A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B. Curless, D. Salesin, and M. Cohen, “Interactive digital photomontage,” ACM Trans. Graph. 23, 292–300 (2004). 70. R. Ng, “Fourier slice photography,” ACM Trans. Graph. 24, 735–744 (2005). 71. R. N. Bracewell, “Strip integration in radio astronomy,” Aust. J. Phys. 9, 198–217 (1956). 72. D. H. Garces, W. T. Rhodes, and N. M. Peña, “Projection-slice theorem: a compact notation,” J. Opt. Soc. Am. A 28, 766–769 (2011). 73. Z. Xu, J. Ke, and E. Y. Lam, “High-resolution lightfield photography using two masks,” Opt. Express 20, 10971–10983 (2012). 74. A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin, “Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing,” in Proceedings of SIGGRAPH (2007), Vol. 26, pp. 1–12. 75. R. N. Bracewell, K.-Y. Chang, A. Jha, and Y.-H. Wang, “Affine theorem for two-dimensional Fourier transform,” Electron. Lett. 29, 304 (1993). 76. M. Broxton, L. Grosenick, S. Yang, N. Cohen, A. Andalman, K. Deisseroth, and M. Levoy, “Wave optics theory and 3-D deconvolution for the light field microscope,” Opt. Express 21, 25418–25439 (2013). 77. N. Cohen, S. Yang, A. Andalman, M. Broxton, L. Grosenick, K. Deisseroth, M. Horowitz, and M. Levoy, “Enhancing the performance of the light field microscope using wavefront coding,” Opt. Express 22, 24817–24839 (2014). 78. L. Tian and L. Waller, “3D intensity and phase imaging from light field measurements in an LED array microscope,” Optica 2, 104–111 (2015). 79. M. E. Testorf and M. A. Fiddy, “Lightfield photography and phasespace tomography: A paradigm for computational imaging,” in OSA Topical Meeting in Computational Optical Sensing and Imaging (Optical Society of America, 2009), paper CTuB2. 80. Z. Zhang and M. Levoy, “Wigner distributions and how they relate to the light field,” in IEEE International Conference on Computational Photography (IEEE, 2009), pp. 1–10. 81. T. Cuypers, R. Horstmeyer, S. B. Oh, P. Bekaert, and R. Raskar, “Validity of Wigner distribution function for ray-based imaging,” in
82. 83.
84.
85.
86.
87.
88.
89.
90. 91.
92. 93.
94. 95.
96.
97.
98.
99. 100.
101. 102. 103. 104. 105.
106.
Tutorial
IEEE International Conference on Computational Photography (IEEE, 2011), pp. 1–9. M. J. Bastiaans, “Wigner distribution function and its application to first-order optics,” J. Opt. Soc. Am. 69, 1710–1716 (1979). H. M. Ozaktas, Z. Zalevsky, and M. A. Kutay, The Fractional Fourier Transform: With Applications in Optics and Signal Processing (Wiley, 2001). S. B. Oh, S. Kashyap, R. Garg, S. Chandran, and R. Raskar, “Rendering wave effects with augmented light field,” Comput. Graph. Forum 29, 507–516 (2010). S. A. Shroff and K. Berkner, “Image formation analysis and high resolution image reconstruction for plenoptic imaging systems,” Appl. Opt. 52, D22–D31 (2013). A. Junker, T. Stenau, and K.-H. Brenner, “Scalar wave-optical reconstruction of plenoptic camera images,” Appl. Opt. 53, 5784– 5790 (2014). C. Hahne, A. Aggoun, S. Haxha, V. Velisavljevic, and J. C. J. Fernández, “Light field geometry of a standard plenoptic camera,” Opt. Express 22, 26659–26673 (2014). C.-K. Liang, T.-H. Lin, B.-Y. Wong, C. Liu, and H. H. Chen, “Programmable aperture photography: multiplexed light field acquisition,” in Proceedings of SIGGRAPH (2008), Vol. 27, pp. 1–10. T. E. Bishop and P. Favaro, “The light field camera: extended depth of field, aliasing, and superresolution,” IEEE Trans. Pattern Anal. Mach. Intell. 34, 972–986 (2012). T. Georgiev and A. Lumsdaine, “Focused plenoptic camera and rendering,” J. Electron. Imaging 19, 021106 (2000). T. G. Georgiev, A. Lumsdaine, and S. Goma, “High dynamic range image capture with plenoptic 2.0 camera,” in OSA Topical Meeting in Signal Recovery and Synthesis (Optical Society of America, 2009), paper SWA7P. T. Georgiev, G. Chunev, and A. Lumsdaine, “Superresolution with the focused plenoptic camera,” Proc. SPIE 7873, 78730X (2011). T. Georgiev, “Plenoptic camera resolution,” in OSA Imaging and Applied Optics Congress (Optical Society of America, 2015), paper JTh4A.2. E. Y. Lam, “Image restoration in digital photography,” IEEE Trans. Consum. Electron. 49, 269–274 (2003). A. Levin, W. T. Freeman, and F. Durand,“Understanding camera tradeoffs through a Bayesian analysis of light field projections,” in European Conference on Computer Vision (Springer, 2008), pp. 88–101. Z. Xu and E. Y. Lam, “A spatial projection analysis of light field capture,” in OSA Frontiers in Optics (Optical Society of America, 2010), paper FWH2. K. Mitra and A. Veeraraghavan, “Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (IEEE, 2012), pp. 22–28. Z. Xu and E. Y. Lam, “Light field superresolution reconstruction in computational photography,” in OSA Topical Meeting in Signal Recovery and Synthesis (Optical Society of America, 2011), paper SMB3. J. Xu, Z. Xu, and E. Y. Lam, “Method and apparatus for processing light-field image,” U.S. patent 14,575,091 (June 18, 2015). S. Wanner and B. Goldluecke, “Variational light field analysis for disparity estimation and super-resolution,” IEEE Trans. Pattern Anal. Mach. Intell. 36, 606–619 (2014). Z. Xu and E. Y. Lam, “A high-resolution lightfield camera with dualmask design,” Proc. SPIE 8500, 85000U (2012). D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52, 1289–1306 (2006). E. J. Candès and M. B. Wakin, “An introduction to compressive sampling,” IEEE Signal Process. Mag. 25, 21–30 (2008). J. Ke and E. Y. Lam, “Object reconstruction in block-based compressive imaging,” Opt. Express 20, 22102–22117 (2012). K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar, “Compressive light field photography using overcomplete dictionaries and optimized projections,” ACM Trans. Graph. 32, 1–12 (2013). J. W. Goodman, “Assessing a new imaging modality,” in OSA Imaging and Applied Optics Congress (Optical Society of America, 2012), paper JM1A.1.
