Neuropsychologia 41 (2003) 1365–1386

Effect of parietal lobe lesions on saccade targeting and spatial memory in a naturalistic visual search task Steven S. Shimozaki a,∗ , Mary M. Hayhoe a , Gregory J. Zelinsky b , Amy Weinstein c , William H. Merigan a , Dana H. Ballard a a Center for Visual Science, University of Rochester, Rochester, NY, USA Department of Psychology, State University of New York, Stony Brook, NY, USA Department of Neurology, Strong Memorial Hospital, University of Rochester, Rochester, NY, USA b

c

Received 8 November 2001; received in revised form 7 January 2003; accepted 7 January 2003

Abstract The eye movements of two patients with parietal lobe lesions and four normal observers were measured while they performed a visual search task with naturalistic objects. Patients were slower to perform the task than the normal observers, and the patients had more fixations per trial, longer latencies for the first saccade during the visual search, and less accurate first and second saccades to the target locations during the visual search. The increases in response times for the patients compared to the normal observers were best predicted by increases in the number of fixations. In order to investigate the effects of spatial memory on search performance, in some trials observers saw a preview of the search display. The patients appeared to have difficulty using previously viewed information, unlike normal observers who benefit from the preview. This suggests a spatial memory deficit. The patients’ deficits are consistent with the hypothesis that the parietal cortex has a role in the selection of targets for saccades, in memory for target location. © 2003 Elsevier Science Ltd. All rights reserved. Keywords: Eye movements; Visual attention

1. Introduction Lesions of the parietal cortex have long been known to cause impairments of visuospatial function. For example, parietal lesions often lead to the syndrome of hemineglect. Occurring predominately with right parietal lesions, hemineglect patients tend to ignore the contralesional (left) side of the environment. Because this syndrome can be distinguished from a loss of visual acuity (hemianopia), hemineglect is considered to be a spatial attention deficit (review Rafal, 1994). It has also been shown that parietal lesions can cause constructional apraxia (a visuospatial deficit in copying and reproducing two- and three-dimensional shapes) (Benowitz, Moya, & Levine, 1990; Benton, 1967; Ruessmann, Sondag, & Beneike, 1988), and deficits in short-term spatial memory (De Renzi, Faglioni, & Previdi, 1977; De Renzi, Faglioni, & Scotti, 1969; De Renzi & Nichelli, 1975). ∗ Corresponding author. Present address: Department of Psychology, University of California, Santa Barbara, CA 93106, USA. Tel.: +1-805-893-3853; fax: +1-805-893-4303. E-mail address: [email protected] (S.S. Shimozaki).

Neurophysiological studies in primates strongly suggest that another aspect of visuospatial processing involving the parietal cortex is saccadic target selection. The parietal cortex projects to areas primarily responsible for eye movement control, such as the superior colliculus (SC) (Andersen, Asanuma, Essick, & Siegel, 1990; Fries, 1984; Lynch, Graybiel, & Lobeck, 1985) and the frontal eye fields (FEFs) (Andersen et al., 1990; Cavada & Goldman-Rakic, 1989; Schall, Morel, King, & Bullier, 1995). Also, areas in the posterior parietal cortex, such as 7a and lateral intra parietal (LIP), respond both to the anticipation and the execution of a saccade (e.g. Andersen, Essick, & Siegel, 1987; Colby, Duhamel, & Goldberg, 1995; Colby, Duhamel, & Goldberg, 1996; Gnadt & Andersen, 1988; Mountcastle, Lynch, Georgopoulos, Sakata, & Acuna, 1975). Andersen (1995) suggests that the LIP activity preceding a saccade encodes the intention to make a saccade, and Duhamel, Colby, and Goldberg (1992) also found that parietal receptive fields shift in preparation of an intended saccade. Gottlieb, Kusunoki, and Goldberg (1998) found that saccade-related activity in LIP depends on behavioral relevance. They suggest that LIP contains a visual ‘salience’ map used in the selection of saccadic targets, as proposed by a number of

0028-3932/03/$ – see front matter © 2003 Elsevier Science Ltd. All rights reserved. doi:10.1016/S0028-3932(03)00042-3

1366

S.S. Shimozaki et al. / Neuropsychologia 41 (2003) 1365–1386

computational models of saccades (Findlay & Walker, 1999; Rao, Zelinsky, Hayhoe, & Ballard, 2002; Wolfe, 1994). Although the attentional deficits caused by parietal lesions in humans have been extensively studied, relatively less is known about the impact of parietal lesions on eye movements. Several previous studies on the effect of parietal lesions on eye movements have focused on hemineglect (e.g. Barton, Behrmann, & Black, 1998; Behrmann, Watt, Black, & Barton, 1997; Chedru, Leblanc, & Lhermitte, 1973; Ishiai, Sugishita, Mitani, & Ishizawa, 1992; Karnath, 1994; Walker & Findlay, 1996). Consistent with the attentional effect of hemineglect, these studies find that saccades of hemineglect patients tend to be biased away from the neglected field (both saccade direction and number of fixations). Studies that have focused more on the eye movements per se, have found that parietal patients have difficulty in a “double-step” saccade task (Duhamel, Goldberg, Fitzgibbon, Sirigu, & Grafman, 1992; Heide, Blankenburg, Zimmerman, & Kompf, 1995; Heide & Kompf, 1998). In this task, the observer must make two successive saccades to two targets flashed briefly and sequentially in the dark before the first saccade can begin. Thus, to make an accurate second saccade, the observer must take into account the spatial displacement of the eye caused by the first saccade. Parietal lesions affected this ability, so that inaccurate second saccades were made despite the presence of accurate first saccades, suggesting an inability to take account of the eye displacement caused by the first saccade. Parietal lesions also caused increased saccade latencies during visually guided reflexive saccades (saccades to sudden onset targets in previously unknown locations) (Heide & Kompf, 1998; Pierrot-Deseilligny, Rivuad, Gaymard, & Agid, 1991a; Pierrot-Deseilligny, Rivuad, Penet, & Rigolet, 1987) and affected saccade latencies and accuracies for memory guided saccades (Pierrot-Deseilligny, Rivuad, Gaymard, & Agid, 1991b). Similar effects were found in normal observers when their posterior parietal cortex was temporarily inactivated by transcranial magnetic stimulation (TMS) (Muri, Vermersch, Rivaud, Gaymard, & Pierrot-Deseilligny, 1996). In addition, several fMRI studies have found activation in the parietal areas during periods of visually-guided saccades, compared to periods of fixation (Corbetta et al., 1998; Darby et al., 1996; Luna et al., 1998; Petit, Clark, Ingeholm, & Haxby, 1997). Some recent studies suggest a specific role of the parietal cortex in the use of spatial memory in saccade targeting. In an fMRI study, Heide et al. (2001), studied a ‘triple-step’ saccade task, which is analogous to the double-step task described above, except that observers had three successive locations to fixate. They found more activation in the right intraparietal area during the triple-step saccade task, compared to various types of single-step (visually- and memory-guided) saccades that had less demands on spatial memory. Also, some recent studies of parietal hemineglect found a greater number of refixations on the ipsilesional side in a dot-counting task (Zihl & Hebel, 1997), as well as in visual search and cancellation tasks of letters, circles,

and line drawings of objects (Husain et al., 2001). These refixations appear to reflect a deficit in the spatial memory in the ipsilesional field. Of particular interest is the study Husain et al. (2001), in which the neglect patients and the normal observers explicitly stated whether each object had been fixated previously. The hemineglect patients not only refixated objects more often, they also did not remember previous fixations. This study further explores the effect of posterior parietal lesions on saccades by examining the saccades of four normal observers and two patients with parietal lesions during a visual search task. Unlike what has been more commonly studied, the two parietal patients did not have observable signs of neglect. In a typical visual search trial, the observer is given a target, views a search display that has or does not have the target, and then must indicate the presence of the target in the search display. Also, experiments in visual search often require the observers to hold their gaze, or have briefly presented stimuli, disallowing the use of eye movements (e.g. Palmer, 1994, 1995; Treisman, 1991). If eye movements are allowed, however, observers tend to make several saccades during a visual search task (Findlay & Gilchrist, 1998; Zelinsky & Sheinberg, 1997). Zelinsky and Sheinberg (1997) and Scialfa, Thomas, and Joffe (1994), for example, found that the number of saccades was an excellent predictor of the response times. Another study by Zelinsky, Rao, Hayhoe, and Ballard (1997) found that, during visual search, the accuracy of the saccades to the target location reliably improved from the first to the last saccade of each trial, showing that saccades reflect the dynamic evolution of the search process. Thus, under natural viewing conditions, the process of saccadic target selection is an integral part of visual search. Given the likely role of the posterior parietal cortex in saccadic target selection, the question we wish to address is how parietal lesions affect the saccadic target selection process during visual search. Another aspect in most visual search studies is that the targets are presented in novel locations on each trial. Thus, the search must be accomplished solely on the basis of the target’s appearance. In the natural world, however, saccadic targets can be selected either on the basis of the target’s appearance (for example, where is the red sweater?) or on the basis of previously acquired information about the target’s location (e.g. the keys are on the table). In these cases, search can be based on the memory of the target’s location as well. A study by Epelboim et al. (1995), for example, showed that spatial memory can affect saccadic performance in a search task. They found that the time taken to tap a specified sequence of colored lights arrayed on a table rapidly decreased as the task was repeated, suggesting that repeated fixations of the locations facilitated the tapping movements. Thus, we were interested to explore saccadic eye movements in both kinds of search process, namely, both in appearance- and spatial memory-based search. A last feature of the experiments was the use of naturalistic displays. Experiments on visual search typically involve

S.S. Shimozaki et al. / Neuropsychologia 41 (2003) 1365–1386

the use of simple stimuli that share common features and are easily segmented from the background (e.g. oriented red and green bars on a white background). In normal scenes, however, individual objects are visually complex and typically have a distinctive set of properties. Additionally, they are usually embedded in a complex background (e.g. a packet of cereal on a breakfast table). Since confusability with other stimuli, and ease of segregation from the background are both likely to affect the search process, we chose search displays that were close to normal scenes. This study therefore adds to the previous work on eye movements following parietal lesions by (a) examining saccadic target selection in the visual search of naturalistic displays, (b) investigating the use of spatial memory in target selection, and (c) using patients with no manifestation of neglect. 2. Method 2.1. Description of observers We identified two patients with lesions localized to a single hemisphere of the parietal cortex. The first patient (RW) was a 42-year-old right-handed male with a resection of a left posterior lateral parietal tumor (glioma) approximately 4 years prior to his participation in the studies (see Fig. 1, drawing of T1-weighted MRI by W. Merigan. A drawing is presented due to the difficulty in observing the lesion in the original MRI). The second patient (WS) was a 62-year-old right-handed male with a right parietal lobe lesion (anterior ventromedial and posterior dorsolateral) caused by an thrombotic stroke approximately 1 year prior to his participation in the studies (see Fig. 2, T1-weighted MRI, 3 days after incident). Both patients were tested on a battery of standard neuropsychological tests, including general intelligence, memory, verbal and language skills, motor skills, and visual perception. Both patients had high average premorbid intelligence, as assessed by the Wechsler Adult Intelligence Scale—Revised (WAIS-R) (Wechsler, 1981), and as suggested by their personal histories (WS was a retired schoolteacher, and RW was a computer programmer on disability). Generally the neuropsychological tests for both patients were within or above normal limits, corresponding with clinical observations that both patients remained highly functional, showing little overall behavioral effect from their brain injury. There were, however, some specific focal changes in cognitive functioning for both patients. Patient WS scored in the low average range for the 7/24 Spatial Recall Test (Rao, Hammeke, McQuillen, Kharti, & Lloyd, 1984). Given his above average scores on the other tests, this suggests a possible visuospatial memory deficit. His scores for Block Design and Digit Span (parts of the WAIS-R) were also in the low average range, suggesting possible deficits in spatial construction and short-term memory, respectively. Patient RW exhibited a mild to moderate decline in the immediate recall of unrelated verbal informa-

1367

tion within a story, and in the immediate recall of complex nonverbal information (Wechsler Memory Scale—Revised (Wechsler, 1987)). Also, RW’s results for the generation of words for specific letters (Controlled Oral Word Association (Benton & Hamsher, 1989)) and three-dimensional construction (Benton, Hamsher, Varney, & Spreen, 1983) were in the low average range. Goldmann visual field perimetry tests indicated no signs of hemianopia in either patient. Both patients did not show any clinical signs of hemineglect the day after their brain injury, and later neuropsychological testing also failed to reveal evidence of hemineglect. For RW, this included more extensive testing using the detection of near-threshold stimuli in an extinction paradigm. Neither patient showed signs of an oculomotor deficit based on clinical observation, or had difficulty with the calibration procedure for the eye tracking equipment. The procedure involved fixating a series of 25 points for about 1 s each, and was a relatively demanding oculomotor task. Both patients had mild peripheral neurological effects from their brain injury. WS experienced numbness and mild pain in his left arm, and RW experienced numbness and occasional tremors in his right arm and right side of the face. Neuropsychological tests corroborated these self-reports. Both patients had normal color vision, and normal or corrected-to-normal visual acuity. The normal observers were four males with ages ranging from 25 to 31 years. All had normal color vision, and normal or corrected-to-normal visual acuity, and no known physical or mental disability. One the authors (GJZ) participated as an observer, and is left-handed. The other three normal observers were right-handed graduate students initially naive to the purpose of the experiment. 2.2. Procedure Both patients and four normal observers participated in the visual search task while their eye positions were monitored. Observers had to indicate the absence or presence of the target in a search display of naturalistic items. The number of items in the search display was one, three or five. In the first condition (No Preview condition, see Fig. 3), observers first saw a display of the target for 1 s, followed by a fixation screen for at least 1 s. (This display was not removed until the observer’s eye position was within 1◦ of visual angle of the cross, guaranteeing eye position near the bottom center at the initiation of search.) Then the search display was shown until the observer responded, indicating the presence or absence of the target in the search display. In this condition the observer could only use information about the target’s appearance to locate the target in the display. In the second condition (Preview condition, see Fig. 4), two additional displays preceding the sequence in the No Preview trials, a 2 s preview display, followed by 1 s fixation screen. The preview display contained all the possible targets in their correct locations in the search display. In other words, if the target appeared in the search display, it

1368

S.S. Shimozaki et al. / Neuropsychologia 41 (2003) 1365–1386

Fig. 1. Drawing of the anatomical MRI (T1-weighted, horizontal) of patient RW describing his glial tumor in the left parietal lobe prior to surgery. The drawings are reversed according to radiological convention, such that the right half of one image represents the left hemisphere, and vice versa.

would only appear at the same location as in the preview. For example, in Fig. 4, once the observer learns the target is the paint scraper, she or he should know to look only in the position indicated by the preview (the far left). Thus, the observer could use both information about the target’s ap-

pearance and the prior information about target location to aid search. The observer’s right eye position was monitored with an SRI Generation V dual-Purkinje eye tracker operating at 500 Hz with an accuracy of approximately 15 visual angle.

S.S. Shimozaki et al. / Neuropsychologia 41 (2003) 1365–1386

1369

Fig. 2. Anatomical MRI (T1-weighted, horizontal) of patient WS 3 days after his thrombotic stroke in the right parietal lobe. The drawings are reversed according to radiological convention, such that the right half of one image represents the left hemisphere, and vice versa.

The observer’s head position was fixed using a dental compression (bite) bar, and an eye patch was worn on the left eye. A calibration procedure was performed every 90 trials for the patients, and every 120 or 180 trials for the normal

observers. The procedure involved fixating 25 points on the display in random order, and fitting the raw tracker data output to the coordinates of the screen by linear interpolation. Offline analyses of the eye movements were performed by

1370

S.S. Shimozaki et al. / Neuropsychologia 41 (2003) 1365–1386

Fig. 3. No Preview trials. The sequence of displays begins at the top left. The search display for a Target Present trial appears at the bottom left, and for a Target Absent trial at the bottom right. Observers indicated their choice for target presence in the search display. Observers fixated the cross in the second dark display before the search display appeared, ensuring the same eye position at the initiation of search.

first converting raw eye position data into eye velocity using a 17-point velocity filter. A saccade then was identified if the filtered (smoothed) eye velocity exceeded 35◦ /s, and if the resulting analyses met the criterion of a minimum fixation duration of 30 ms. Observers viewed images on an AppleColor 14 in. RGB color display at 80 cm from the observer, and driven by an Apple Macintosh IIfx computer. The targets and distractors were digitized naturalistic color objects presented on a background (12◦ × 16◦ ) with an appropriate context. There were three contextual settings; toys in a crib, tools on a workbench, and food on a dinner table. The objects for each trial were chosen randomly from a set of 10 for each context. The object locations were randomized across six locations in the search display, 7◦ from the initial fixation point at the bottom center, and spaced at equal radial angles (22.5◦ ) from the fixation point. Half the search displays contained the target (Target Present), and the other half did not (Target Absent). There

were a total of 12 conditions, 2 (Preview/No Preview) × 2 (Target Present/Target Absent) × 3 (number of items, 1/3/5). Each normal observer participated in 60 trials of each condition, for 720 total trials. WS participated in 45 trials of each condition, for 540 total trials. RW participated in 30 trials of each Preview condition, and 45 trials of each No Preview condition, for 450 total trials. Trials were blocked by Preview or No Preview trials (90 trials per block for the patients, and 120 or 180 trials per block for the normal observers), with target presence and number of items randomized within these blocks. Only the trials with correct responses were analyzed. Analyses of variance were performed at an α level of 0.05 using the statistical package GANOVA (Woodward, Bonett, & Brecht, 1990). Analyses were performed first with respect to the hemifield (left or right) of the target. Results for all observers showed no consistent effect of hemifield, and subsequent results were collapsed over hemifield.

S.S. Shimozaki et al. / Neuropsychologia 41 (2003) 1365–1386

1371

Fig. 4. Preview trials. The same sequence as Fig. 3, except for the preceding Preview and fixation display. The Preview display contained all the possible targets in their correct locations. For example, in this trial, the target paint scraper can only appear in the far left position.

1372

S.S. Shimozaki et al. / Neuropsychologia 41 (2003) 1365–1386

Table 1 Response times, statistical results Normals

RW

WS

1618 (