The following blog post approaches the topic Multimodality and serves as an introduction to the podcast ‘Knowledge to Go’, produced for the Design Thinking course. For further information on the topic, you can check my group mates’ blog posts too, which talk about the History of Multimodality, Multimodality in the Academic World, and Multimodal Literacy. In this blog post, I will first give an introduction of the subject, defining what is multimodality and mode. Following, I will expand the theory of multimodality to digital writing. Finally, there will be a personal reflection on the topic and on the process of producing a podcast about it.
Multimodality: a brief introduction
The term multimodality started appearing in publications around the 1990s but its use dates from much earlier than that. As said by Frank Serafini (2014) in his book ‘Reading the Visual’, humans have always communicated using more than one mode. With the advance of technology, especially from the 20th century on, it became even easier for that to happen. We create and consume multimodal content almost all the time: by watching a movie, reading comics, sharing a message on What’s App with emojis or stickers, looking through our Instagram feeds, playing video games, among other ways. Thus, it is no surprise that the term has increasingly gained popularity in recent years, with academics researching the phenomenon across different fields, such as semiotics, education, sociology, linguistics, media studies, just to name a few. (Jewitt et al, 2014)
What better way to define multimodality than by adding another mode? Gunther Kress was a linguist and semiotician that contributed enormously to the field. In the following video he explains the concept:
In summary, Multimodality refers to the many ways one can communicate a message. To illustrate that, a good example to better understand the concept is a street sign, as mentioned by the researcher Kress in his book Multimodality: A Social Semiotic Approach to Contemporary Communication (2009): the object has an image, writing text, and color. All these semiotic resources work together to communicate meaning and would probably not be as effective separately. Kress (2009, p.1) further explains that “each mode does a specific thing: image shows what takes too long to read, and writing names what would be difficult to show. Colour is used to highlight specific aspects of the overall message. Without that division of semiotic labor, the sign, quite simply, would not work. Writing names and images shows, while color frames and highlights; each to maximum effect and benefit” Therefore, each element on the street sign is assisting the meaning being conveyed. It is important to add that multimodality should not be confused with multimedia, as media is different from mode: the first one serves as a vehicle for the second to be disseminated. (Serafini 2014)
Furthermore, it is relevant to define what constitutes a mode. There are different perspectives on the term across the existent publications. According to Kress (2009) definition, a mode is “a socially and culturally shaped resource for making meaning” (p.79). Moreover, each mode has its unique potential to help in the communication of a message, and will be selected or not based on its affordances and limitations. Some modes used for communication are “image, writing, layout, gesture, speech, moving image, soundtrack” (p.79) As mentioned before, there is no consensus on this definition and certain elements might be considered a mode by one academic and not by another.
How should one know which mode to use and when? The preference for a mode over another changes according to the instance of communication, time, and culture. As pointed out by Kress (2009), “societies have modal preferences” (p.83), a mode might be used for a specific purpose of communication whereas another mode will be used for another one. Furthermore, he has also observed that in the last few years there has been a shift: the screen is becoming more dominant in comparison to printed publications and image is gradually replacing writing central role in our life. (Kress, 2003) Moreover, in today’s constantly evolving media landscape, it became the norm to engage with a combination of numerous modes. (Bateman et al 2017) To construct a meaning, modes work together and digital technologies have allowed that to happen like never before.
Finally, in their book ‘Introducing Multimodality’ Jewitt et al (2014) establish three key premises to understand multimodality which I find relevant to add here: 1) each mode used to convey meaning has its affordances and limitations; 2) “meaning-making involves the production of multimodal wholes” (p.16), and 3) each mode is deserved of attention when trying to comprehend the meaning. Now that the notion of making meaning through modes is made clear, we can move on to the next section which will discuss Digital Writing.
Digital Writing and the connection with multimodality
As mentioned earlier in this post, Kress (2003) considers that books and writing are being replaced by screen and image, respectively, in the ‘new media age’, allowing new affordances to come into play. The internet page will be much more flexible regarding modalities than a page of a book: it can include moving images, audio, and hyperlinks, for instance. Moreover, a page in a book will never be read in the same way as a page in a blog and two people will likely not read the blog page in the same way. Content in a screen is read non-linearly most of the time. Kress (2003) further mentions that these changes are causing a revolution in the way we communicate and interact with each other, as “the world told is a different world to the world shown” (p.1). He also adds that “the combined effects of the changes in the media and in the uses of modes reach further still; they are not confined to the screen but affect all media and all modes. (p.15) This can be seen recently in books, magazines, and newspapers that incorporate features of digital layouts.
Digital technologies are not only changing the way we consume information but also influencing how we create it. You might have not heard of the term ‘Digital Writing’ before, but you certainly practice quite often. In fact, there is a good example right in front of you: this blog. DeVoss et al (2010) define digital writing as “compositions created with, and oftentimes for reading or viewing on, a computer or other device that is connected to the Internet” (p.7) Examples include Twitter, Facebook, emails, Instagram, blogs, between other platforms: “at one level, web writing is about writing on the web: the flexibility as a multimodal piece, the ability to nimbly circulate, and the capacity to create a network of texts. At another level, the practice is about writing for the web and situating ourselves as readers and writers within its evolving architecture” (Rajchel 2014). This genre tends to be underestimated by some scholars, but it offers a lot of potentials to be explored and should deserve more attention in my opinion, especially in the context of multimodality.
With digital writing, the meaning is often constructed through the use of many modes which might complement each other or even replace it. For instance, on the platform Twitter, one can post a short text, share a link to another website, post videos or pictures, and tag other users. There is also the use of hashtags, which offers a whole new world of possibilities associated with digital writing. Moreover, the platform’s unique affordances give the user skills in fast reading and synthesis power, pedagogy could make use of that to empower students.
Digital writing is not restricted to the informality of social media. For the Master of Digital Culture at Maastricht University, all students had to keep an academic blog updated regularly with assignments and reflections. Through this practice in digital writing, I learned valuable skills that complemented my academic trajectory and made me reflect on how to make meaning through the use of multimodality. It made me realize how much potential other modes than writing could offer to express the complex ideas I learned throughout the course. Moreover, this experience developed my sense of audience, which is a key aspect of digital writing.
To summarise, digital writing is constantly evolving, adapting to the newest technologies and creating new possibilities of meaning making. This genre has provoked many changes in writing, as pointed out by Merchant (2007). Some of them include: texts are not fixed, they can be revised and edited; there is participation from the reader in the author’s text, which can comment and reply on the work; non-linearity in writing and reading; no barriers in the communicative space and increase in multimodality use. Moreover, he explains that “although the ability to combine and access media is likely to become much easier (and much faster) there is little to indicate major changes that will threaten the centrality of written communication.” (Merchant 2007, p.126) Thus, the role of digital writing in society should be looked into more deeply and stimulated between students. There is a lot be learned from that and by engaging with digital writing, they will be better writers in general.
Personal reflection
For our ‘Design Thinking and Maker Culture’ course we had to produce a podcast about a topic of our choice, which was Multimodality. Throughout this process, I learned a lot about the subject and can say that it has a lot of potentials to be explored, especially in the teaching and academic field. Even though we are surrounded by multimodality, when we are doing research we tend to focus too much on written text and neglect other modes, that have unique characteristics and allow a better understanding of the message we are trying to convey. Based on the principle that we live in a multimodal world; it does not make sense to insist on monomodal content. It is expected from us socially, professionally and educationally, that we can make sense of images, sounds, gestures, and texts daily. Therefore, we should be equipped with multimodal skills from our early childhood. Moreover, there are many kinds of intelligence, multimodal communication can respect the differences between people and be more inclusive. With the ubiquity of digital writing in our lives, schools and universities should focus on integrating them into the students learning. Digital literacy is not a differential anymore, but something expected from us in our careers.
Furthermore, as mentioned by Kress (2003), communication is changing, screens and images are dominating more and more. This is often viewed as pessimistic, but digital platforms have allowances that engage all of the reader’s senses and produce meaning in ways that were never possible without the internet. In this blog post, one can gain information through image, writing, video, and audio, for instance. Therefore, I believe we should explore more often the affordances digital writing provides us and get others to do the same as a way of producing a world that is more multimodal and inclusive.
You can listen to our podcast Multimodality Knowledge to Go and find out more about the topic, like the history of it, its use in the academic field, and its importance to literacy.
References
Bateman, J., Wildfeuer, J., & Hiippala, T. (2017). Multimodality: Foundations, research, and analysis–A problem-oriented introduction. Walter de Gruyter GmbH & Co KG.
Baer, Andrea. “Keeping Up With… Digital Writing in the College Classroom.” 2013. Web. 21 Oct. 2015. http://www.ala.org/acrl/publications/keeping_up_with/digital_writing.
DeVoss, D. N., & Eidman-Aadahl, E. (2010). Because digital writing matters: Improving student writing in online and multimedia environments. John Wiley & Sons.
Hendriksen, B. & Kress, G. (2012, Mar 15). “What is Multimodality?”
Kress, G. R. (2003). Literacy in the new media age. Psychology Press.
Kress, G. (2009) Multimodality: a Social Semiotic Approach to Contemporary Communication, London: Routledge.
Jewitt, C., Bezemer, J., O’Halloran, K. (2014). Introducing Multimodality. Routledge.
Merchant, G. (2007). Writing the future in the digital age. Literacy, 41(3), 118-128.
Rajchel, J., (2014) “Consider the Audience,” in Web Writing: Why and How for Liberal Arts Teaching and Learning, ed. Jack Dougherty and Tennyson O’Donnell (University of Michigan Press/Trinity College ePress edition), http://epress.trincoll.edu/webwriting/chapter/rajchel.
Serafini, F. (2014). Reading the Visual: An Introduction to Teaching Multimodal Literacy. Chapter 2. New York: Teachers College Press.
