NBIO 401

Fall 2008

Vision – Extra Striate Visual Cortex Monday, November 24, 2008 Pasupathy

Objectives In this lecture we will: • Define and describe the extrastriate visual cortex, its extent in the primate brain and its broad role • Think about and appreciate the complexity of the visual perception problem and why such a large fraction of cortex might be dedicated to solving it • Describe the dorsal and ventral pathways for processing of visual information – the areas and their broad roles • Discuss strategy and experimental tools used to discover how neurons in these areas work • Discuss the organizational principles in the various areas and some examples of properties that have been discovered Definition, extent and function of the extrastriate visual cortex

The existence of multiple visual areas has been a major discovery of the past quarter century in the field of sensory neuroscience. The extrastriate visual cortex includes cortical areas -1-

NBIO 401

Fall 2008

beyond the striate cortex in the occipital, temporal and parietal lobes that are exclusively (or primarily) involved in the processing of visual information. By this definition, the extrastriate areas comprise roughly 30-50% of the cortex (depending on the primate species). Extrastriate cortex is composed of 32 subdivisions that can be distinguished by their physiology, cytoarchitecture, histochemistry and/or connections with other areas (more on this later). Figure above shows all areas in the primate (rhesus monkey) brain dedicated primarily to the processing of visual information. A and B are lateral and medial views of the right hemisphere. C is a flattened representation of the brain. The different colors represent different brain areas. The white regions represent brain areas involved in the processing of other sensory and motor information. Compare the fraction of other sensory and motor systems: Somatosensory 11.5% Motor 7.9% Auditory 3.4% Gustatory 0.5% Prefrontal 9.4% Cingulate 5.3% Perirhinal 1.6% Unspecified 5.3% In previous lectures you learned about how V1 neurons represent visual scenes in terms of orientation, color, spatial frequency, disparity and motion information in small localized image regions called receptive fields. The V1 representation is a high resolution localized representation of image features and is quite different from the way we perceive visual scenes. Our percept of visual scenes is in terms of its component objects, surfaces, and backgrounds. How is the percept synthesized from the localized image feature map in V1? The visual information from V1 is passed on to the extrastriate cortex for further processing and our percept is the result of this further processing. So the role of extrastriate cortices is to synthesize the information it receives from V1 and derive/infer the visual scene that created the feature map in V1 image. So the visual system must solve this difficult inverse problem. Why is such a large area dedicated to vision? Compared to other sensory modalities or the motor system, a much larger cortical area is dedicated to vision. This is because humans are visual animals and most information about the external world is obtained through vision. Vision is important for all of human behavior – when we interact with other people, when we walk/drive down a street or when we are working/reading at our desk. We rely heavily on vision because of its effectiveness as a rapid and accurate source of information about the world around us. The visual system is accurate and reliable in a wide-variety of operating conditions – under different illumination conditions, at different viewing angles and distances. This is the biggest strength of the visual system. Imagine how much less useful your visual system would be if you were able to recognize objects only when i) they are at a certain distance from you ii) when there are no neighboring or overlapping objects and iii) under very specific lighting conditions. These extraneous variables (lighting, pose, viewing distance, clutter) strongly influence the retinal image and the responses of V1 neurons but the visual system, specifically the extrastriate cortex, has figured out ways to disentangle the intrinsic properties of the object (shape, color, motion -2-

NBIO 401

Fall 2008

direction, etc) from these extraneous variables. Let’s look at some example situations to illustrate the point.

Which monster is larger? This is a popular size perception illusion. Both monsters at left are identical in shape and size. However, the one to the right of the image is perceived as the larger of the two. Why is this? This is because of the context provided by the background (if you remove the surrounding background this illusion goes away). The brick pattern of the background provides depth ordering information for this scene – the monster at left is perceived as being closer than the monster at right. But we know that objects farther away from the observer cast a smaller image on the retina than objects nearby. Since both of these monsters cast the exact same retinal image, the monster farther away must be larger and that is how we perceive the scene. Which square is brighter? Squares A and B are the exact same luminance but B appears brighter than A. This again is because of the surrounding background which is consistent with a shadow being cast on some of the squares of the checkerboard pattern. We know that a cast shadow dims a surface and so, if the same amount of light is reflected by two surfaces (A and B), the one in bright light (A) is darker than the one in the shadow (B). So our visual system interprets B as being of a lighter shade of gray than A. Both the examples above illustrate the success rather than the failure of our visual system. Our visual system is not very good at being a camera or a light meter, but that is not its purpose. The important task is to break down the image information into meaningful components, and thereby perceive the nature of the objects and surfaces in view. A large area of the brain is dedicated to solving the visual perception problem because it is a computationally challenging problem. To perceive an object, the visual system has to encode its boundaries (these maybe continuous or not – see the boundary of the giraffe that is partially occluded by another giraffe), its texture, color, brightness, 3D properties of depth, orientation and curvature, transparency, shadowing , velocity and velocity gradient. The visual system is an amazing -3-

NBIO 401

Fall 2008

engineering feat. No modern computer is able to achieve the accuracy/speed or versatility of the human visual system. For example, a small change in facial expression, or a slight tilt of the face would trip up most automated recognition systems today. This is not because computer scientists have not tried to build an effective robust system. It is because the problem being solved by the visual system is difficult and very high dimensional. Strategies and experimental tools used to discover how the extrastriate cortex solves the visual perception problem So we have a big swath of cortex consisting of 32 different areas solving a tremendously difficult problem that we can’t design computers to solve. How is this achieved? To figure out how the brain does it, we need to: i) figure out the specific roles of the 32 different areas, ii) figure out how these areas are inter-connected to discover the direction and nature of information flow, iii) discover how each area performs its function, i.e. how do neurons represent the various stimuli iv) discover how activity in the various areas underlies behavior v) discover how different areas work together to solve the perception problem Over the past several decades researchers have used several techniques including localized lesions, anatomical tracing, single-cell neurophysiology and microstimulation experiments to address the goals outlined above. In lesion experiments, a portion of the cortex is removed and the resulting loss in function is assessed. Lesion experiments tell you what the coarse function of an area is but not how it is achieved. In anatomical tracing studies, the connections between different areas are mapped. This provides clues about direction of information flow but no direct evidence about function. In single cell neurophysiology experiments, the responses of single neurons in different regions of the brain are studied to ascertain the role of the neurons. This technique provides very good spatial resolution (activity of single neurons) but the results are correlational (not causation like in lesion studies) and one is looking at one neuron at a time rather than the network as a whole. Finally, microstimulation experiments provide answers about causation. Here, a small amount of current is injected into a localized brain area (causing excitation of those neurons) and the results assess how this current injection affects behavioral performance. Thus, each of these studies answers different questions and a combination of all techniques is necessary to discover how the visual system works. Using the above techniques researchers have worked hard trying to figure out the function of the various areas in the extrastriate cortex. But progress has been slow and we still know very little about how visual objects are perceived. Work conducted in the past several decades have provided a broad understanding of the organizing principles in the extrastriate cortical areas. These principles are discussed next.

-4-

NBIO 401

Fall 2008

“What” and “Where” visual pathways

Based on anatomical physiological and histochemical criteria, the visual areas are subdivided into 32 distinct areas (300+ connections between the 32 areas). Many boundaries between the various areas can also be identified based on sulcal landmarks. One of these areas is V1, the striate cortex. The remaining 31 areas are the extrastriate cortical areas and they occupy the occipital, temporal and parietal lobes. These 32 visual areas have been arranged broadly into two parallel hierarchical pathways based on lesion and anatomical tracing studies.

-5-

NBIO 401

Fall 2008

The ventral “what” pathway: From striate cortex, one route proceeds ventrally into posterior temporal cortex passing through areas V2, V4 and the sub-regions of the inferior temporal cortex. Based on the fact that lesions in the posterior temporal area result in loss of pattern discrimination and that lesions in the inferotemporal cortex in particular result in failure to recognize previously presented objects, this pathway was assigned the task of analyzing "the physical properties of a visual object (such as its size, color, texture and shape)". This pathway is often known as the “what” pathway since the responses of neurons more closely reflect the attributes of the visual stimulus (rather than its location or motion direction). Results from physiology experiments (discussed next) confirm this hypothesis. The dorsal “where” pathway: The other route proceeds from striate cortex dorsally into the posterior parietal cortex via areas MT, MST and regions of the posterior parietal cortex. Lesions in the posterior parietal cortex in monkeys result in failure to be able to select a response location on the basis of a visual landmark, suggesting that this pathway figures in "perception of spatial relations among objects, and not in their intrinsic qualities". The dorsal pathway is also known as the “where” pathway since neural responses carry information about spatial location of stimuli and motion direction. While we are still pretty far from understanding how each of these areas work and how they contribute to visual perception, organizing principles have been demonstrated in the “what” and “where” pathways. In both pathways, the RF sizes of cells in the successive hierarchical -6-

NBIO 401

Fall 2008

stages become larger. In V1, receptive fields are quite small at the center of fixation, less that 0.5 deg. and it gets larger as you move towards the periphery. RF sizes are roughly 10-16 times the size of V1 RFs in V4 and MT. In MST and IT cortex, receptive fields are even larger, anywhere from 10-40 degrees. The second organizing principle is that response properties are increasingly complex (nonlinear) functions of the stimulus. Examples of this increasing complexity are shown next. Ventral pathway – V1, V4, IT progression V1 neurons carry information about stimulus orientation and direction.

While some V4 neurons are also selective for orientation of a bar, many cells are selective for more complex stimuli. Below is an example neuron that responds to sharp angles that point to the upper left but not to oriented bars or lines. This neuron from area V4 was tested with a variety of angles, curves, straight lines and edges. The background of each icon represents the response strength of the cell to the stimulus shown inside. Darker background represents stronger responses (as per grayscale). This neuron responds strongly to angles and curves pointing to the upper left but not to any of the lines. Thus, the responses of this neuron encode angles and curves, a higher level image feature than oriented lines and edges.

-7-

NBIO 401

Fall 2008

In IT cortex, the last stage in the what pathway, cells are selective for even more complex stimuli. An example is shown in the figure below. This neuron responds to a monkey face or a cartoon face. The neuron responds only when the outline of the face, eyes and mouth are all included. It does not respond to oriented bars, or colored patches.

-8-

NBIO 401

Fall 2008

Dorsal pathway --- V1/MT/MST progression As we saw earlier, some V1 neurons respond selectively to the direction of motion of an oriented bar. In MT, the middle temporal area, located in the superior temporal sulcus, cells respond to motion of a variety of stimuli including random dots. A popular stimulus to study MT neurons is the motion coherence stimulus. This is just a set of moving dots, a fraction of which move coherently in a specific direction. The motion percept is stronger for higher coherence stimuli.

Example MT neuron showing direction selectivity for the coherent motion of a random dot pattern.

-9-

NBIO 401

Fall 2008

At a later stage along the dorsal pathway in area MST (medial superior temporal), cells respond to even more complex motion patterns. In one experiment, researchers studied the responses of MST neurons to planar, circular and radial patterns of random dot motion. Figure below shows examples of MST neurons that respond preferentially to leftward (A), Counter-clockwise (B) and inward motion. Stimulus set is shown below.

- 10 -

NBIO 401

Fall 2008

Example neurons A-C respond preferentially to leftward motion (A), counter-clockwise motion (B) or inward motion (C).

So we know the broad roles of the various areas and how some stimulus attributes are represented. But we still have a lot of work to do before we figure out how the responses of neurons in these different areas underlie our perception of a complex visual scene. Microstimulation All of the experiments are correlational studies. They do not tell us whether the documented activity is important for perception/recognition. To determine causal relationships between neural activity and behavior one has to perturb the neural activity and determine how that alters animal behavior. There have been very few studies that have achieved this and an excellent example is a study where microstimulation in area MT was used to bias animal behavior on a direction discrimination task.

- 11 -

NBIO 401

Fall 2008

In this study, the authors first dropped electrodes in area MT, identified the preferred direction of motion of the neurons at the tip of the electrode and then conducted the following experiment. Stimulus presentation details are shown in the figure above. A drifting random dot pattern was presented in the stimulus aperture and the animal had to report (by making an eye movement to the appropriate LED) whether the dots were moving in the preferred or anti-preferred direction. The temporal details of the visual display are shown in B.

To vary the difficulty of the task, the stimulus had all dots moving in one direction (100% correlation, easy), some dots moving in one direction and other at random (x% correlation, where x depends on what fraction of dots move coherently in one direction; hard) or no dots moving coherently, 0% correlation, most difficult). As the animal performed this task the authors obtained a psychometric function which plots the proportion of trials the animals saccaded to report that the stimuli were moving in the preferred direction.

- 12 -

NBIO 401

Fall 2008

The solid line above indicates the psychometric function obtained from the animal based on no stimulation trials. When the authors injected a very small amount of current as the animal performed the task, the behavioral curve shifted to the left, i.e. the animal picked the preferred direction more often. This result is a convincing demonstration that MT neurons do influence animal behavior. A similar study in the inferotemporal cortex has demonstrated that microstimulation of faceselective clusters in the inferotemporal cortex influences the recognition of faces while identical stimulation in non-face selective regions does not.

The stimulus design

- 13 -

NBIO 401

Fall 2008

Effect of microstimulation

Comparison of results between faceselective clusters (a) and non-face selective clusters.

- 14 -