Integrating audiovisual information for the control of overt attention.

Abstract:

:In everyday life, our brains decide about the relevance of huge amounts of sensory input. Further complicating this situation, this input is distributed over different modalities. This raises the question of how different sources of information interact for the control of overt attention during free exploration of the environment under natural conditions. Different modalities may work independently or interact to determine the consequent overt behavior. To answer this question, we presented natural images and lateralized natural sounds in a variety of conditions and we measured the eye movements of human subjects. We show that, in multimodal conditions, fixation probabilities increase on the side of the image where the sound originates showing that, at a coarser scale, lateralized auditory stimulation topographically increases the salience of the visual field. However, this shift of attention is specific because the probability of fixation of a given location on the side of the sound scales with the saliency of the visual stimulus, meaning that the selection of fixation points during multimodal conditions is dependent on the saliencies of both auditory and visual stimuli. Further analysis shows that a linear combination of both unimodal saliencies provides a good model for this integration process, which is optimal according to information-theoretical criteria. Our results support a functional joint saliency map, which integrates different unimodal saliencies before any decision is taken about the subsequent fixation point. These results provide guidelines for the performance and architecture of any model of overt attention that deals with more than one modality.

journal_name

J Vis

journal_title

Journal of vision

authors

Onat S,Libertus K,König P

doi

10.1167/7.10.11

subject

Has Abstract

pub_date

2007-07-25 00:00:00

pages

11.1-16

issue

10

issn

1534-7362

pii

7/10/11

journal_volume

7

pub_type

杂志文章
  • Gaze patterns in navigation: encoding information in large-scale environments.

    abstract::We investigated the role of gaze in encoding of object landmarks in navigation. Gaze behavior was measured while participants learnt to navigate in a virtual large-scale environment in order to understand the sampling strategies subjects use to select visual information during navigation. The results showed a consiste...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/10.12.28

    authors: Hamid SN,Stankiewicz B,Hayhoe M

    更新日期:2010-10-22 00:00:00

  • Simulating prosthetic vision: Optimizing the information content of a limited visual display.

    abstract::Visual prostheses for the restoration of functional vision are currently under development. To guide prosthesis research and allow for an accurate prognosis of functional gain, simulating the experience of a retinal prosthesis in healthy individuals is desirable. Current simulation paradigms lack crucial aspects of th...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/10.14.32

    authors: van Rheede JJ,Kennard C,Hicks SL

    更新日期:2010-12-29 00:00:00

  • Feature-based attention modulates surround suppression.

    abstract::Stimuli appearing in the surround of the classical receptive field (CRF) can reduce neuronal firing and perceived contrast of a preferred stimulus in the CRF, a phenomenon referred to as surround suppression. Suppression is greatest when the surrounding stimulus has the same orientation and spatial frequency (SF) as t...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/15.1.29

    authors: Flevaris AV,Murray SO

    更新日期:2015-01-28 00:00:00

  • Visual search: the role of peripheral information measured using gaze-contingent displays.

    abstract::Two of the factors limiting progress in understanding the mechanisms of visual search are the difficulty of controlling and manipulating the retinal stimulus when the eyes are free to move and the lack of an ideal observer theory for fixation selection during search. Recently, we developed a method to precisely contro...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/6.9.1

    authors: Geisler WS,Perry JS,Najemnik J

    更新日期:2006-08-03 00:00:00

  • Perceptual insensitivity to higher-order statistical moments of coherent random dot motion.

    abstract::When the visual system analyzes distributed patterns of sensory inputs, what features of those distributions does it use? It has been previously demonstrated that higher-order statistical moments of luminance distributions influence perception of static surfaces and textures. Here, we tested whether the brain also rep...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/18.6.9

    authors: Waskom ML,Asfour J,Kiani R

    更新日期:2018-06-01 00:00:00

  • Prior depth information can bias motion perception.

    abstract::Previous studies of the motion aperture problem have shown that the direction of grating motion can be biased by using binocular disparity to designate borders of the aperture as intrinsic (belonging to the grating) or extrinsic (resulting from occlusion of the grating). Observers report motion in the direction of the...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/4.6.2

    authors: Graf EW,Adams WJ,Lages M

    更新日期:2004-05-19 00:00:00

  • Mach bands and multiscale models of spatial vision: the role of first, second, and third derivative operators in encoding bars and edges.

    abstract::Ernst Mach observed that light or dark bands could be seen at abrupt changes of luminance gradient in the absence of peaks or troughs in luminance. Many models of feature detection share the idea that bars, lines, and Mach bands are found at peaks and troughs in the output of even-symmetric spatial filters. Our experi...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/12.13.18

    authors: Wallis SA,Georgeson MA

    更新日期:2012-12-21 00:00:00

  • Foveal contour interactions and crowding effects at the resolution limit of the visual system.

    abstract::We describe several experiments on contour interactions and crowding effects at the resolution limit of the visual system. As test stimuli we used characters that are often employed in optometric practice for testing visual acuity: Landolt C's, Snellen E's, and rectangular gratings. We tested several hypotheses that h...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/7.2.25

    authors: Danilova MV,Bondarko VM

    更新日期:2007-11-27 00:00:00

  • A horizontal bias in human visual processing of orientation and its correspondence to the structural components of natural scenes.

    abstract::Many encoding mechanisms and processing strategies in the visual system appear to have evolved to better process the prevalent content in the visual world. Here we examine the relationship between the prevalence of natural scene content at different orientations and visual ability for detecting oriented natural scene ...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/4.12.5

    authors: Hansen BC,Essock EA

    更新日期:2004-12-10 00:00:00

  • Redundancy masking: The loss of repeated items in crowded peripheral vision.

    abstract::Crowding is the deterioration of target identification in the presence of neighboring objects. Recent studies using appearance-based methods showed that the perceived number of target elements is often diminished in crowding. Here we introduce a related type of diminishment in repeating patterns (sets of parallel line...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/jov.20.4.14

    authors: Yildirim FZ,Coates DR,Sayim B

    更新日期:2020-04-09 00:00:00

  • Some observations on contrast detection in noise.

    abstract::The standard psychophysical model of our early visual system consists of a linear filter stage, followed by a nonlinearity and an internal noise source. If a rectification mechanism is introduced at the output of the linear filter stage, as has been suggested on some occasions, this model actually predicts that human ...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/8.9.4

    authors: Goris RL,Zaenen P,Wagemans J

    更新日期:2008-07-10 00:00:00

  • Eye movements in repeated multiple object tracking.

    abstract::Contrary to other tasks (free viewing, recognition, visual search), participants often fail to recognize repetition of trials in multiple object tracking (MOT). This study examines the intra- and interindividual variability of eye movements in repeated MOT trials along with the adherence of eye movements to the previo...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/13.7.9

    authors: Lukavský J

    更新日期:2013-06-13 00:00:00

  • Distinguishing influences of overt and covert attention in anticipatory attentional target tracking.

    abstract::This study investigated the relative contributions of overt and covert attention on the apparent anticipatory nature of attention in two experiments, using two different object tracking tasks, both combined with a probe detection task. In Experiment 1, we investigated the distribution of attention for overtly and cove...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/17.4.3

    authors: Frielink-Loing AF,Koning A,van Lier R

    更新日期:2017-04-01 00:00:00

  • Attention enhances contrast appearance via increased input baseline of neural responses.

    abstract::Covert spatial attention increases the perceived contrast of stimuli at attended locations, presumably via enhancement of visual neural responses. However, the relation between perceived contrast and the underlying neural responses has not been characterized. In this study, we systematically varied stimulus contrast, ...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/14.14.16

    authors: Cutrone EK,Heeger DJ,Carrasco M

    更新日期:2014-12-30 00:00:00

  • Distinct effects of attention on the neural responses to form and motion processing: a SSVEP source-imaging study.

    abstract::We measured neural responses to local and global aspects of form and motion stimuli using frequency-tagged, steady-state visual evoked potentials (SSVEPs) combined with magnetic resonance imaging (MRI) data. Random dot stimuli were used to portray either dynamic Glass patterns (Glass, 1969) or coherent motion displays...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/12.10.15

    authors: Palomares M,Ales JM,Wade AR,Cottereau BR,Norcia AM

    更新日期:2012-09-26 00:00:00

  • Probing intermediate stages of shape processing.

    abstract::The visual system provides a representation of what and where objects are. This entails parsing the visual scene into distinct objects. Initially, the visual system encodes information locally. While interactions between adjacent cells can explain how local fragments of an object's contour are extracted from a scene, ...

    journal_title:Journal of vision

    pub_type: 杂志文章,评审

    doi:10.1167/15.7.1

    authors: Loffler G

    更新日期:2015-01-01 00:00:00

  • A functional role for trans-saccadic luminance differences.

    abstract::In typical natural environments, the visual system receives different inputs in quick succession as gaze moves around. We examined whether local trans-saccadic differences in luminance, contrast, and orientation influenced perception and target selection in the eye movement system. Observers initially fixated a periph...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/12.13.14

    authors: Ludwig CJ,Davies JR,Gegenfurtner KR

    更新日期:2012-01-01 00:00:00

  • Associating peripheral and foveal visual input across saccades: a default mode of the human visual system?

    abstract::Spatial processing resolution of a particular object in the visual field can differ considerably due to eye movements. The same object will be represented with high acuity in the fovea but only coarsely in periphery. Herwig and Schneider (in press) proposed that the visual system counteracts such resolution difference...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/14.11.7

    authors: Weiß K,Schneider WX,Herwig A

    更新日期:2014-09-09 00:00:00

  • Planning multiple movements within a fixed time limit: the cost of constrained time allocation in a visuo-motor task.

    abstract::S.-W. Wu, M. F. Dal Martello, and L. T. Maloney (2009) evaluated subjects' performance in a visuo-motor task where subjects were asked to hit two targets in sequence within a fixed time limit. Hitting targets earned rewards and Wu et al. varied rewards associated with targets. They found that subjects failed to maximi...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/10.6.1

    authors: Zhang H,Wu SW,Maloney LT

    更新日期:2010-06-01 00:00:00

  • Spatial relationships between contours impact rapid scene classification.

    abstract::Photographs and line drawings of natural scenes are easily classified even when the image is only briefly visible to the observer. Contour junctions and points of high curvature have been shown to be important for perceptual organization (Attneave, 1954; Biederman, 1987) and have been proposed to be influential in rap...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/18.8.1

    authors: Wilder J,Dickinson S,Jepson A,Walther DB

    更新日期:2018-08-01 00:00:00

  • Surround facilitation for rapid motion perception.

    abstract::Because we live in a dynamic environment with moving eyes and body, our retinas are often stimulated by new scenes that appear suddenly and are only briefly available. How the visual system successfully extracts information from such challenging stimulation is not yet understood. For some stimuli, like photos of natur...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/12.10.3

    authors: Linares D,Motoyoshi I,Nishida S

    更新日期:2012-09-06 00:00:00

  • Lightness perception in simple images: testing the anchoring rules.

    abstract::One approach toward understanding how vision computes surface lightness is to first determine what principles govern lightness in simple stimuli and then test whether these hold for more complex stimuli. Gilchrist (2006) proposed that in the simplest images that produce the experience of a surface (two surfaces differ...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/14.13.25

    authors: Radonjić A,Gilchrist AL

    更新日期:2014-11-25 00:00:00

  • Luminance noise as a novel approach for measuring contrast sensitivity within the magnocellular and parvocellular pathways.

    abstract::This study evaluated the extent to which different types of luminance noise can be used to target selectively the inferred magnocellular (MC) and parvocellular (PC) visual pathways. Letter contrast sensitivity (CS) was measured for three visually normal subjects for letters of different size (0.8°-5.3°) under establis...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/17.8.5

    authors: Hall CM,McAnany JJ

    更新日期:2017-07-01 00:00:00

  • The effects of fixational tremor on the retinal image.

    abstract::The study of fixational eye motion has implications for the neural and computational underpinnings of vision. One component of fixational eye motion is tremor, a high-frequency oscillatory jitter reported to be anywhere from ∼11-60 arcseconds in amplitude. In order to isolate the effects of tremor on the retinal image...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/19.11.8

    authors: Bowers NR,Boehm AE,Roorda A

    更新日期:2019-09-03 00:00:00

  • Detection of between-eye differences in color: Interactions with luminance.

    abstract::Between-eye differences in color or luminance result in the appearance of luster, which provides a cue for detecting between-eye differences. We measured thresholds for detecting between-eye differences in both hue and chromatic contrast (saturation) in dichoptically superimposed color patches. Sensitivity was found t...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/16.3.23

    authors: Jennings BJ,Kingdom FA

    更新日期:2016-01-01 00:00:00

  • How attention and contrast gain control interact to regulate lightness contrast and assimilation: a computational neural model.

    abstract::Recent theories of lightness perception assume that lightness (perceived reflectance) is computed by a process that contrasts the target's luminance with that of one or more regions in its spatial surround. A challenge for any such theory is the phenomenon of lightness assimilation, which occurs when increasing the lu...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/10.14.40

    authors: Rudd ME

    更新日期:2010-12-31 00:00:00

  • The crowding factor method applied to parafoveal vision.

    abstract::Crowding increases with eccentricity and is most readily observed in the periphery. During natural, active vision, however, central vision plays an important role. Measures of critical distance to estimate crowding are difficult in central vision, as these distances are small. Any overlap of flankers with the target m...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/16.11.30

    authors: Ghahghaei S,Walker L

    更新日期:2016-09-01 00:00:00

  • Adaptation to interocular differences in blur.

    abstract::Adaptation to a blurred image causes a physically focused image to appear too sharp, and shifts the point of subjective focus toward the adapting blur, consistent with a renormalization of perceived focus. We examined whether and how this adaptation normalizes to differences in blur between the two eyes, which can rou...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/13.6.19

    authors: Kompaniez E,Sawides L,Marcos S,Webster MA

    更新日期:2013-05-31 00:00:00

  • Daily mixed visual experience that prevents amblyopia in cats does not always allow the development of good binocular depth perception.

    abstract::Kittens reared with mixed daily visual input that consists of episodes of normal (binocular) exposure followed by abnormal (monocular) exposure can develop normal visual acuity in both eyes if the length of the former exposure exceeds a critical amount. However, later studies of the tuning of cells in primary visual c...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/9.5.22

    authors: Mitchell DE,Kennie J,Schwarzkopf DS,Sengpiel F

    更新日期:2009-05-21 00:00:00

  • Combining local and global limitations of visual search.

    abstract::There are different opinions about the roles of local interactions and central processing capacity in visual search. This study attempts to clarify the problem using a new version of relevant set cueing. A central precue indicates two symmetrical segments (that may contain a target object) within a circular array of o...

    journal_title:Journal of vision

    pub_type: 杂志文章

    doi:10.1167/17.4.10

    authors: Põder E

    更新日期:2017-04-01 00:00:00