Show Less
Restricted access

Building Bridges for Multimodal Research

International Perspectives on Theories and Practices of Multimodal Analysis

Series:

Edited By Janina Wildfeuer

While multimodality is one of the most influential semiotic theories for analysing media artefacts, the concepts of this theory are heterogeneous and widespread. The book takes the differences between approaches in Germany and those in international contexts as a starting point, offering new insights into the analysis of multimodal documents. It features contributions by researchers from more than 15 nations and various disciplines, including theoretical reflections on multimodality, thoughts about methodological, empirical, and experimental approaches as well as analyses of various multimodal artefacts.
Show Summary Details
Restricted access

Seeing the Unforeseen: Eye-Tracking Reading Paths in Multimodal Webpages

Extract



1 Introduction

In an elucidating overview of research conducted on the concept of reading paths, Hiippala (2012: 318) deplores that “the development of a theory of reading paths that would integrate perceptual psychology […] has not been followed up in subsequent multimodal research […] [but] became swamped by semiotically oriented, interpretative multimodal frameworks”. This overwhelmingly interpretative tradition in multimodal research has been criticised several times (cf., e.g., Kaltenbacher 2004: 202; Bateman 2008: 11; Bateman 2014: 239) and constitutes one of the gaps in multimodal research that this book is trying to close. The gap identified here concerns a general flaw in multimodal research. On the one hand, we have a seemingly endless number of studies on multimodal meaning-making grounded in the standard theory of social semiotics originally developed by Kress/van Leeuwen (2006 [1996]). An overwhelming majority of these studies is based on heuristic and interpretative approaches to the question how intersemiotic meaning is created by writers and how it is decoded by readers of multimodal text. Such approaches involve methods like introspection, participant observation, interviews, and verbal protocols. On the other hand, authentic reader behaviour often differs from what interpretative methods suggest and make us believe. Because of this discrepancy, considerable efforts have been undertaken during the past ten years to put multimodal research onto more solid empirical ground. Noteworthy, in this respect, are the development of multimodal corpora and annotation schemes (cf., e.g., Baldry/Thibault 2005), the creation of interactive software to analyse multimodal text (cf., e.g., O’Halloran...

You are not authenticated to view the full text of this chapter or article.

This site requires a subscription or purchase to access the full text of books or journals.

Do you have any questions? Contact us.

Or login to access all content.