skip to Main Content

The Multimodal Brain


Cultural inventions often require multimodal mapping processes, for instance when reading language or music arbitrary visual script characters must be mapped onto the corresponding units of spoken language or music. Learning such mappings impacts on our multimodal brains. We investigate the consequences of cultural inventions on brain and cognition as a tool to examine underlying processing dynamics from sensorimotor systems to high-level integration systems. One possibility is that we use the brain’s basic sensory and motor systems for language comprehension: a process often called simulation. We explore the idea that language directly engages these modality-specific systems rather than duplicating or ‘re-enacting’ events in the ‘outside world’. For example, some of our group’s experimental data suggest that spoken words can rapidly activate low-level category-specific visual representations that affect the mere detection of a stimulus, i.e. what we see. We have also observed that visual noise interferes more with concrete spoken word processing than abstract spoken word processing, but only when the task required visual information to be accessed.



Former subproject