Laila Boulakhrif

Multimodality in the Academic world

Multimodality and its usability and practicality in the world of academics are thriving more and more. Even though the term was created in the 1990s the method was used long before (Jewitt, Bezemer, & O’Halloran, 2016; Bezemer & Jewitt, 2018). Because of its increase in popularity in the academic world, it is understandable to relate its popularity to the methods successfulness. That is why my interest lies in how multimodality is used in different research fields and branches and not only what it entails. Thus, how does multimodality improve research and what researchers miss if they do not use it? Moreover, I find it important to not only gather the information through academic papers but also to ask people that use this method in their research or line of work about its usability. Therefore, at the end of the blog, a brief interview is offered to see multimodality used in the field of digital culture.

In this blog post, the focus is more on the usability of multimodality in the different research fields and not necessarily the detailed explanation of what multimodality entails. However, to be able to discuss a topic such as multimodality a brief explanation is offered to be able to comprehend the rest of the blog. Thus, multimodality according to Serafini (2014) “refers to texts that utilize a variety of modes to communicate or represent concepts and information” (p. 12). What this means is that an object or medium exists out of different modes such as writing, gesture, speaking, etc. to communicate meaning (Serafini, 2014; Jewitt, Bezemer, & O’Halloran, 2016). To be clear multimodality is not only about new technologies or the different modes. Multimodality is an interdisciplinary method that can show the social changes and cultural patterns (Bezemer, 2009; O’Halloran, & Smith, 2014). Kress (2010) gives the example of a road sign that exists out of three modes: writing, image, and colour (p. 1). He mentions that if the road sign would only exist out of writing that it would not have worked and would not convey what it wanted to. Some modes work better to convey a meaning or information, that is why multimodality is important.

Now the question arises why is it important to know about multimodality or even its usability in the academic world. Research in different fields such as semiotics, musicology, visual sociology, linguistics, etc. have used multimodality in their fields and have started slowly using multimodality interdisciplinary. Thus, analysing “image with writing, speech with gestures, math symbolism with writing, and so forth” (Bezemer & Jewitt, 2018, p. 2). In the case of being interdisciplinary in research, which multimodality encourages, it should be a norm given that as Serafini (2014) so nicely describes it, we are living in a multimodal world (p. 2) which means to understand the meaning behind something which is already multimodal it is understandable that also in research multimodality should be used without boundaries and interdisciplinary (Serafini, 2014; Bezemer & Jewitt, 2018). Thus, as the example mentioned above about the road sign, when researching an object and the relation between different modes it is important to not only know the importance of a mode on its own it is also important to know the relation to the modes together and what every mode can convey. Of course, objects or technology does not stand on its own, there are external factors to take into consideration. That is why multimodality is involved it incorporates different disciplines to get to know an already existing multimodal world.

Since the rise of the internet, people have been programmed differently and are used to the multimodality of technology. Teachers and schools have also contributed to getting students used to multimodality, thus not only using one mode such as text. With the introduction of new technologies, researchers examine what multimodality can offer in the field of pedagogy and education (Jewitt, 2006). According to Jewitt (2006), the reason for schools wanting to get students used to new technologies and the multimodality of it is because of governments pushing this onto schools. This for the reason that governments around the world, since the 1990s, relate new technology to better learning. The problem, however, is that if a new technology has failed to be implemented in classrooms the research has mainly been on the spoken aspect or just one mode even though technologies in themselves exist out of multiple modes. The interwovenness of communication modes is ignored.

Moreover, Jewitt’s research (2006) uses multimodality to “examine how the design and use of image, sound, writing and movement on-screen contributes to the construction of school knowledge and potentials for learning” (p. 4). Thus, what the multiple modes from computers can offer in relation to learning. Jewitt (2006) connects “curriculum knowledge, learning, literacy and pedagogy” (p. 7) to students using technology to learn such as the computer. In regards to curriculum knowledge and learning the modes focussed on by schools, thus what they mostly grade on or find more important, is speech and writing. The problem with only focussing on those two modes is the negligence of the other modes or seeing the other modes as less. Jewitt (2006) explains that because more modes are used, the learning process should also be different. Thus, it is clear that the government pushing new technologies onto schools is not the problem, the problem here is the multimodality that these technologies bring with them and the governments not going along with the changes because it is, of course, the governments that decide what to grade the students on: speech and writing. In the field of education and pedagogy, with multimodality, researchers have shown not only the effects of new technology but also social changes and political external factors.

In the field of linguistics, multimodality is not a new topic. Bezemer & Jewitt (2018) explain how in the field of linguistics language (mainly speech and writing) throughout the years is seen as the most important way of communication especially compared to non-verbal communication modes. With multimodality introduced into the field, this discriminatory way of thinking turned for the better. Multimodality has shown that modes should not be graded in which is more important but what the different modes can offer in regards to making meaning. Because every mode has its advantages and restrictions and not one mode should be seen as the main mode. Moreover, it is important to know that with multimodality in the different fields that not every researcher relates it to having meaning or what Bezemer & Jewitt (2018) call ‘making meaning’. The focus can also be just on the relation between modes and what they offer separately and in relation to one another.

As mentioned earlier, multimodality is used in different fields and is used interdisciplinary. The disciplines mentioned by Bezemer & Jewitt (2018), that use multimodality and the aspect of making meaning, are of course linguistics, sociology, semiotics, but also psychology and anthropology (p. 3). Bezemer & Jewitt (2018) state that with multimodality and its integrating way of doing that even theories and methods used in research have been affected by the method. The theories that integrated multimodality are “Discourse Analysis, Conversation Analysis, Systemic Functional Linguistics and Cognitive Linguistics” (p. 3, 4). In the case of methods, quite of bit of researches have used aspects of methods in relation to methods of other disciplines (p. 4). An example can be how concepts and methods from the discipline of semiotics are integrated with ethnography (p. 4).

Now focussing on what multimodality offered to the theory of Discourse Analysis; and as to understand the growth of how multimodality is used in different fields. Discourse Analysis around the 1960s was only related to the field of linguistics (spoken and written language).  Thus, the relationship between texts would be analysed, the function of a text, etc. (Bhatia, Flowerdew, & Jones, 2008; Cameron & Panović, 2014). With the introduction of multimodality, the focus of analysing discourse shifted from not only considering language to also considering it in relation to image, sound, gesture, movement, etc. This means that other disciplines were involved such as psychology, semiotics, sociology, marketing, business, information technology, law, and so forth (Bhatia, Flowerdew, & Jones, 2008, p. 1). In the case of multimodal Discourse Analysis, the importance was how the relationship between the different modes create symbolic meaning. Because of multimodality, the realization that all the modes are on equal footing came to be. Especially in the era of media where different modes were used and the understanding of linguists that communication is also multimodal (Bhatia, Flowerdew, & Jones, 2008; Hyland, 2011).

                The interview with Claartje Rasterhoff, who is a historian working in the field of digital culture, shows an insight of a real person in the academic world that comes into contact with multimodality:

1.Have you ever used multimodality in your line of work or during researches?

Yes, in my first research project I used maps and other data visualizations, but never saw this as multimodal. In my previous research project (Amsterdam) Time Machine and other digital humanities projects, I’ve also used mapping and deep mapping (text, geodata combined with for instances images). In another project, I’ve collaborated with artists who and designers who used more visual modes of learning and presenting, and with art historians who study images.

2. How has multimodality (if used) changed your way of doing research?

As a historian trained only in reading and producing texts, working more multimodal has really opened up a world of other perspectives. What I understand to be sources has broadened immensely. My view on how research is presented has also changed (more public-facing, combining modes), as did my take on how to produce knowledge (more collaborative, including multiple perspectives).

a. Has it been for the better or not?

Yes, but with broadening comes the feeling of losing depth of specialisation.

3. Have you ever come across researches that in your opinion needed a multimodal approach to improve said research?

A lot of historical studies of material culture and knowledge production were until relatively recently focused mostly on text. Which limits one’s perspective of culture and knowledge. This is changing though.

4. Have you used other modes than the mode (written) text for projects?

I’ve made a short video clip in order to experiment with storytelling for broader audiences. I’ve developed digital platforms that use maps and visual models as a starting point in order to combine different types of sources in a single resource.

5. Is there a text with one mode only or is everything multimodal?

Depends on who you ask. Most people will see it’s one mode, but if you’re into fonts and typesetting you can discern visual aspects in texts.

6. Is there a mode or modes that you see more important compared to other modes?

No, this depends on context. But the written word is obviously still very dominant, also in my thinking and communication and it’s how I express myself best (so far).

                To conclude, multimodality in the academic world, especially in the discipline of linguistics, has helped disciplines branch out to other disciplines and to think outside their line of work. As it should be given that the world and everything in it is multimodal. Multimodality also shows researchers that particular modes are not necessarily above other modes and realize their contribution to making meaning on their own and together with other modes. Moreover, People around the world that are also encouraged by surrounding factors to live in a multimodal technological society understand through interdisciplinary research their way of living much better.

Please feel free to check out my podcast that introduces multimodality from different perspectives:

Reference

Bezemer, Jeff & Jewitt, Carey. (2018). Multimodality: A Guide for Linguists. In L. Litosseliti (ed), Research Methods in Linguistics (2nd edition). London: Continuum.

Bhatia, V. K., Flowerdew, J., & Jones, R. H. (2008). Approaches to discourse analysis. Advances in discourse studies, 1-17.

Cameron, D. & Panović, I. (2014). Multimodal discourse analysis. In Cameron, D., & Panović, I. Working with written discourse (pp. 97-111). 55 City Road, London: SAGE Publications

Hyland, K. (2011). Continuum Companion to Discourse Analysis. Continuum Companions. London: Continuum.

Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach. Psychology Press.

Jewitt, C. (ed.) (2009). The Routledge Handbook of Multimodal Analysis. London: Routledge.

Jewitt, C., Bezemer, J., & O’Halloran, K. (2016). Introducing multimodality. Routledge.

Kress, G. R. (2010). Multimodality: A social semiotic approach to contemporary communication. Taylor & Francis.

O’Halloran, K. L., & Smith, B. A. (2014). Multimodal Studies: An Emerging Research Field. In Multimodal studies: Exploring issues and domains. New York: Routledge.

Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. Teachers College Press.

Leave a Reply

Your email address will not be published. Required fields are marked *