Olika perspektiv pa språk, bild och deras samspel. Metodologiska reflexioner.

Författare

Jana Holsanova

Redaktör

Inger Haskå
Carin Sandqvist

Summary, in English

We encounter multimodality every day: in the newspapers, journals, ads, posters, story books, text books, leaflets, encyclopedias, instruction books, computer interfaces and in our interactions. Multimodality has been studied within many different disciplines: semiotics/sociosemiotics, textlinguistics, interface design, and human-computer-interaction. Yet, the approaches in the research of multimodality differ a lot.

The paper gives an overview over different ways of analysing multimodality. Seven different perspectives have been extracted from the different works: Perspectives that focus on (1) Differences between modalities, (2) Similarities between modalities, (3) Interplay between modalities, (4) Formal aspects of the layout, (5) Perception of the different modalities and their interplay, (6) Reproduction of modalities in conversation, and on (7) Multimodal interaction.

The researchers within the first perspective stress the fact that modalities have different inherent strengths and weaknesses and are therefore suitable to express different types of information (Bernsen 1994, Mulet & Sano 1995, Teleman 1994). In the second approach, in contrast, similarities between modalities are focused. For instance, when analysing language and pictures, the same analytic categories are used. The view is based on the assumption that pictures and other visual elements are, like language, signs that form a grammar (Kress & van Leeuwen 1990, 1994, Ledin 1997). The third perspective stresses the interplay between modalities. The questions are: Does one of the modalities dominate the others or do they complement each other? Does one of the modalities strengthten the information given in the others, does it reduce it or does it give additional information? (Danes 1995, Melin 1990, Sandqvist 1995). The analysis of the formal aspects of the layout is characteristic for the fourth perspective (Kress & van Leeuwen, Pettersson 1997, Sandqvist 1995).

Whereas researchers tend to study multimodality of ready-made products in perspectives (1)–(4) mentioned above, the perspectives (5)–(7) are closely connected with the usage of these products. The fifth perspective concerns perception of different modalities. Here, it is interesting to see how the users perceive the interplay between the modalities. Within this area, some research has been done using eye-trackers and verbal protocols (Holsanova 1998, 1999, 2001, Rayner et al. 2001). The sixth perspective stresses the fact that the usage of multimodal products does not end with their perception. The information gained from an multimodal product is often reconstructed and discussed in another communicative situation. It is interesting to see if the information packaging is still the same after the transformation or if it changes (Holsanova & Nekvapil 1995, Scollon 1998). Finally, there is a broad field of multimodal interaction, covering the situations when we as individuals use speech, gestures, body language, facial expressions, drawing and gaze when interacting with other humans or with multimodal interactive systems. We can ask: Where and when do we integrate those streams? Are they complementary or is their content redundant? Are they overlapping or sequential? (Holsanova 2001, Oviatt 1996, 1999). Perception of multimodality, reproduction of multimodality and multimodal interaction deserve to be focused on more extensively in the future research.

Avdelning/ar