Image Processing on GPU

Image Processing • Define new images from existing image for different purposes • Used for texture preprocessing for 3D graphics and visualization • Simple processing: – Transform each pixel independently • Pixel to pixel operations • Example: RGB image to grey scale image

– Move pixels inside without modifying intensities

• Complex processing – Many advanced technologies for different applications

Noise Reduction • An important application for image processing • Important in many areas • Sources: – Digital camera devices – Scanned documents – Satellite images – Medical imaging –…

Example Noise Types • Salt and pepper: random white and black pixels • Impulse: random white pixels • Gaussian noise: variation in intensity from Gaussian distribution • … • We need to “SMOOTH” the image to reduce noises

Linear Filters • Construct a new but same-size image from original one • Fill new pixel with a weighted sum of old pixels -- filtering • Using the same weights for each pixel • Linear filtering – Output for the sum of tow images is the same as the sum of the outputs obtained for the images separately

Convolution • Filter kernel: the pattern of weights • Convolution: the process of applying the filter on an image • Mathematically:

Smoothing • Reduce noise • Replace each pixel with a weighted average of its neighbors • Local blur effects to remove noise • Low frequency filter to remove high frequency noise

Averaging Smooth • (2k+1)*(2k+1) region

1 Rij = (2k + 1) 2 • Poor blurring • Too simple

i+k

j+k

∑ ∑F

u =i − k v = j − k

uv

Gaussian Smoothing • Symmetric Gaussian kernel – Weights distributed with Gaussian function – Sigma: standard deviation

(x2 + y 2 ) Gσ ( x, y ) = exp(− ) 2 2 2πσ 2σ 1

– In applications, discrete kernel used: ((i − k − 1) 2 + ( j − k − 1) 2 ) H ij = exp(− ) 2 2 2πσ 2σ 1

Gaussian Lookup Table – A lookup table can be pre-generated to accelerate the computation, such as

Gaussian Kernel • If standard deviation is very small (smaller than one pixel), the smoothing have very little effect • For a larger one, the average will be strongly biased toward a consensus of the neighbors. Noise will largely disappear, at the cost of some blurring • Finally, if it is very large, much of detail disappear along with noise

Why use Gaussian • Not the only smoothing filter • Convenient to use: – Efficiency – Separable

Gσ 1 ∗ Gσ 2 = G

σ 12 +σ 2 2

– Performed separately for x, y, z dimensions

Boundary Effects • In practical system, image has boundaries • When computing convolution, need to handle boundaries where the neighbors do not exist – Ignore these locations-image may shrink – Pad the image with constant values – Pad with other methods: periodical …

Example

•

Robert J. Woodham, UBC

Edge Detection • Edges are important information on images – Object detection … – Large gradient – Intensity derivatives

• Edge detection: estimate derivative and an image represented by a discrete set of pixels

What is an edge

• Easy for human • Difficult for computer

Estimate Derivative • Finite difference ∂h ≈ hi +1, j − hi −1, j ∂x • The similar convolution kernel is ⎧0 0 0 ⎫ ⎪ ⎪ G = ⎨1 0 − 1⎬ ⎪0 0 0 ⎪ ⎩ ⎭

• Gradient vector

⎛ ∂f ∂f ⎞ ∇f ( x, y ) = ⎜⎜ , ⎟⎟ ⎝ ∂x ∂y ⎠

Sobel Kernel • A popular gradient magnitude computation

Second Derivatives • Sobel operator can produce thick edges • Ideally, we want thin boundaries • Alternative approach: – Looking for local extrema in first derivative – Where the change in gradient is highest

2D Laplacian • An equivalent measure of second derivative

Edge detection cg code • Laplacian edge detection static const char *edgeFragSource = { "uniform sampler2D texUnit;" "void main(void)" "{" " const float offset = 1.0 / 512.0;" " vec2 texCoord = gl_TexCoord[0].xy;" " vec4 c = texture2D(texUnit, texCoord);" " vec4 bl = texture2D(texUnit, texCoord + vec2(-offset, -offset));" " vec4 l = texture2D(texUnit, texCoord + vec2(-offset, 0.0));" " vec4 tl = texture2D(texUnit, texCoord + vec2(-offset, offset));" " vec4 t = texture2D(texUnit, texCoord + vec2( 0.0, offset));" " vec4 ur = texture2D(texUnit, texCoord + vec2( offset, offset));" " vec4 r = texture2D(texUnit, texCoord + vec2( offset, 0.0));" " vec4 br = texture2D(texUnit, texCoord + vec2( offset, offset));" " vec4 b = texture2D(texUnit, texCoord + vec2( 0.0, -offset));" " gl_FragColor = 8.0 * (c + -0.125 * (bl + l + tl + t + ur + r + br + b));" "}" };

Results not satisfied • Noise are emphasized

How to improve? • Add smoothing filter • Smoothing a differentiated image tends to get good result we want • Smooth the edge detected due to noises

Improved Results

Derivative of Gaussian Filter • The improvement is implemented as: Gaussian smoothing first and then use differential filter

∂ (Gσ ∗ I ) ∂x • This can be replaced by convolution with the derivative of Gaussian ∂ (Gσ ∗ I ) ∂Gσ = ∗I ∂x ∂x

Steps in edge detection • Typical steps: – Filtering: cut down on noise – Enhancement: amplify the difference between edges and non-edges – Detection: use threshold – More: estimate geometry and other operations

Image Processing on GPU • Processing an image is fairly simple and straightforward. • The filter renders a screen-aligned quad into an invisible pixel buffer. – The screen-aligned quad has the input image bound as a texture. – For each rendered pixel, a Cg fragment program is executed, which does the actual image processing in a local neighborhood. – Because the quad is rendered using the input image as a texture, the Cg program can access the pixels in the input image though simple texture reads.

Convolution on GPU

Typical GPU Framework • Load Sources • Apply filters as cg programs • Render image on screen • Must also provide operators for saving images

An Example: Scotopic Filter • Vision under reduced illumination,

– lose most of our ability to see color – Scenes appear blurred and shaded in blue

• • • •

Blur Æ use Gaussian filter Blue shaded Æ use night-vision filter ThenÆ edge detection This procedure mimic the human night vision system

• GPU implementation enable immediate processing, useful for many real applications, such as military

Example

More Techniques and Examples

Image Glow

Add Glow on GPU • (a) The scene is rendered normally. • (b) A rendering of glow sources is blurred to create • (c) a glow texture, which is added to the ordinary scene to produce • (d) the final glow effect.

Glow on GPU • The alpha channel of an object's texture can be used to specify glow sources and the glow source brightness.

Color Control • Color correction is part of almost all print and film imaging applications. – move color images from one color space to another (say, from Adobe RGB to sRGB); – to stylize images (creating illusions such as faded film, cross-processing, or other stylistic variations); – to combine art elements from different sources (such as matching color palettes between different video sources, or models made by different artists); – to give broad coherence and mood to entire parts of a game or scene without changing the underlying models and textures.

General Idea • Change color of individual pixels

– Per channel: alter red, green, and blue components individually – color-mixing: each output channel may be an operation based on the red, green, and blue components simultaneously

• The mathematics of color corrections can be compactly and easily described in a GPU shader. • Just as important, they can be controlled effectively with common tools used widely by computer artists and programmers. – E.g.Relying on Adobe Photoshop to let artists create control resources, which can then be applied in real time via pixel shaders.

Gamma Adjustment • Adjusting Image Gamma and Overall Dynamic Range in photoshop • Cg program implement: •

outPixel = (pow(((inPixel * 255.0) - inBlack) / (inWhite - inBlack), inGamma) * (outWhite - outBlack) + outBlack) / 255.0;

Curved Level Adjustment • Curves tool in Photoshop provides arbitrary remapping of the color channels

Curved Level Adjustment • A 1x256 texture map should be defined, although smaller maps can often be used effectively • Before creating the map, the artist must first define the color transformation, typically by applying adjustment layers to existing still images using the Curves tool in Photoshop. • Create a ramp texture from an original white-black ramp according to the color transformation

Curved Level Adjustment • Cg code • • • • •

float3 InColor = tex2D(inSampler, IN.UV).xyz; float3 OutColor; OutColor.r = tex1D(ColorCorrMap, InColor.r).r; OutColor.g = tex1D(ColorCorrMap, InColor.g).g; OutColor.b = tex1D(ColorCorrMap, InColor.b).b;

• Channel-by-Channel Results of the Red, Green, and Blue Remappings

PDE and Image Processing • We introduce simple local pixel filtering for image processing • Some more complex image processing techniques involves PDEs • Thus, GPU-based PDE solver is very important – Image diffusion – Image editing –…

Repeated Smoothing: Diffusion • Usually we need to repeat smoothing operation • Such operation actually is a solution to the diffusion equation ∂Φ ∂ 2 Φ ∂ 2 Φ = 2 + 2 = ∇ 2Φ ∂t ∂x ∂y

Anisotropic Diffusion • Edge preserving smoothing • Use a function g to control diffusion • Smoothing noise while conserve edges

∂Φ = ∇ ⋅ (c( x, y, σ )∇Φ ) ∂t

Diffusion Function • c() is the diffusion function • A monotonically decreasing function of the image gradient magnitude • Generally g() can be implemented

Gaussian Diffusion Function • g() can be implemented by the function of derivative of Gaussian distribution • Smoothing noise while preserving edges

Demos • Copyright by Joachim Weickert

How to solve the equation? • Typical Partial Differential equation • Basic finite difference operations to discretize the equation • Thus, a linear system is achieved – Show how?

• How to use GPU solving linear system? – Linear algebra on GPU