Vision as Optimal Inference • The problem of visual processing can be thought of as computing a belief distribution • Conscious perception better thought as a decision based on both beliefs and the utility of the choice.

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Hierarchical Organization of Visual Processing

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Visual Areas

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Circuit Diagram of Visual Cortex

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Motion Perception as Optimal Estimation

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Local Translations OpticFlow: (Gibson,1950)

Assigns local image velocities v(x,y,t) Time ~100msec Space ~1-10deg

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Measuring Local Image Velocity Reasons for Measurement ●

Optic Flow useful: ●

●

Heading direction and speed, structure from motion,etc.

Efficient: ●

Efficient code for visual input due to self motion (Eckert & Watson, 1993)

How to measure? ●

Look at the characteristics of the signal

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

X-T Slice of Translating Camera

t

y x

x

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

X-T Slice of Translating Camera

t

y x

x Local translation

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Early Visual Neurons (V1) Ringach et al (1997) y

x

y

t

x x PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

What is Motion? As Visual Input: ● Change in the spatial distribution of light on the sensors. Minimally, dI(x,y,t)/dt ≠ 0

As Perception: ● Inference about causes of intensity change, e.g. I(x,y,t) vOBJ(x,y,z,t) PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Motion Field: Movement of Projected points

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Basic Idea • • • •

1) Estimate point motions 2) use point motions to estimate camera/object motion Problem: Motion of projected points not directly measurable. -Movement of projected points creates displacements of image patches -- Infer point motion from image patch motion – Matching across frames – Differential approach – Fourier/filtering methods

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Problem: Images contain many edges-Aperture problem

Normal flow: Motion component in the direction of the edge

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Aperture Problem (Motion/Form Ambiguity) Result: Early visual measurements are ambiguous w.r.t. motion.

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Aperture Problem (Motion/Form Ambiguity) However, both the motion and the form of the pattern are implicitly encoded across the population of V1 neurons.

Actual motion

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Plaids Rigid motion

This pattern was created by superimposing two drifting gratings, one moving downwards and the other moving leftwards.

Here are the two components displayed side-by-side.

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Find Least squares solution for multiple patches. PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Motion processing as optimal inference •

Slow & smooth: A Bayesian theory for the combination of local motion signals in human vision, Weiss & Adelson (1998)

Figure from: Weiss & Adelson, 1998

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Modeling motion estimation

Local likelihood: Global likelihood:

w(r )(I xvx + I yvy + It ) 2 / 2σ 2 ∑ L(v) ∝ e r −

Lr (v) → p(I | θ ) ∝ ∏ Lr (θ ) r

(Dv )t (r)( Dv)(r )/ 2 ∑ P(V ) ∝ e r −

Prior:

P(V ) → P(θ ) Posterior:

P(θ | I) ∝ P(I | θ) P(θ )

From: Weiss & Adelson, 1998 PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Figures from: Weiss & Adelson, 1998 PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Figure from: Weiss & Adelson, 1998 PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Figure from: Weiss & Adelson, 1998 PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Figure from: Weiss & Adelson, 1998 PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Lightness perception as optimal inference

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Lig ht dir .L

Illuminant

L( λ )

Surface normal N

I(x,y)

€

surface reflectances

Simple rendering model r r xy I(x, y) = S ( λ ) L( λ ) ⋅ N (x, y)

[

S

]

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

xy

(λ )

REFLECTANCE

ILLUMINANT

SIGNAL

X

400

500

600

700

=

400

500

600

700

400

500

600

700

CONES

?

400

500

( PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

600

L , M, S

700

)

Land & McCann’s lightness illusion

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Lateral inhibition

.

Threshold small values Integrate

Differentiate (twice) by convolving image with "Mexican hat filter"

Perceived Lightness

Luminance

Neural network filter explanation

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Apparent surface shape affects lightness perception

•

Knill & Kersten (1991)

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

Inverse graphics solution

Image Luminance x

different "paint"

What model of material reflectances, shape, and lighting fit the image data?

same "paint"

Reflectance

Reflectance

Shape

Shape

point

point

ambient

Illumination

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

ambient

Illumination

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004

PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004