Multimodality is used in many domains, including academic. In academic research, the use of multimodality provides a flexible way for research, because it contains the use of different modes from different perspectives to contribute to a study or a project. With the development of the method used in designing a plan in academics, text-only research is no longer as effective as multimodal one. Multimodal texts are another kind of “text”, which combines at least two modes, such as spoken language, written language, visual (moving and still images), spatial, gestural and audio (Gilje, 2010). For example, the textbooks that combine words and images as well as the ppt with video and audio, which are all the multimodal text and can be applied in academic research. Another application of multimodal text is storyboarding, it is usually be segmented into several panels and provides a visual representation, like a film storyboard. It shows a way by working with different modes rather than focusing only on writing modes (Forceville, 2011). Now, it can also be used in designing a project or research.
This blogpost will first introduce and define the multimodal text. Then, storyboarding will be exemplified and analyzed as an application of multimodal texts in academic research.
What is a multimodal text?
Hallid (1989) explains that multimodal texts as a functional language that is either written or spoken or presented in a different medium of expression. He believes that texts should not be limited to either written or spoken language. Serafini (2014) explains multimodal texts in academic research as a narrow feature that can be distributed in different ways, such as a website, photographic essay or picture book. The latter emphasizes more on the function of distribution and communication and points out the visualization elements used in the text to convey more information. In another word, a multimodal text has more than one mode combining in a text, especially some visual images, such as photographs, graphs, and diagrams ( Serafini, 2014).
Besides, Kress (2015) indicates that multimodality is not a theory, rather, it is a domain for social semiotic work that promotes unity and coherence in the overall text. He noted that only using one mode like language or writing is insufficient to access a discipline question. Therefore, using different modes is necessary. All modes constitute the integrated resources that have significant characteristics use in multimodality (Kress, 2015).
To get more useful information on the use of multimodal text, in the next section, I will introduce an application of multimodal texts, storyboards, and analyze its use.
The use of multimodal texts —Storyboards
A storyboard is a sequence of some cells (Brien, 2013) that using different modes such as texts, images and symbols to comprise different illustrations to displayed pre-visualized ideas, scripts or research plans. Storyboards are used for planning purposes; they are created before the development of the final product, and they are used in showing the progression of an issue through different scenes. All of the elements combined together to balance information and communicate ideas clearly (storyboarding); thus, it is a typical application of multimodal texts. Storyboard design is usually used in preparing for film production and telling the story through visualization (Gilje, 2010); it is primarily used by television commercial advertising clients, film directors and cinematographers visualize the different scenes, create continuation, and identify any problems before their occurrence (Gilje, 2010). However, in academic research, we do not need to create a story with our imagination and change it to texts and images on paper, rather, we can come out our plan of research and integrate the information we coming from different modes in a visual way — by creating storyboards (Bezemer & Kress, 2008).
Figure 1 below is a storyboard made by our group in class that visualizes our ideas of making a podcast about multimodality. (The podcast introduces multimodality and its use in different aspects, the link is at the end of the text.)
It is a simple storyboard comprising five pictures and a few words; however, it can demonstrate the whole process of the design of a podcast. The first picture shows the introductory part of the podcast: a conversation. The second to fourth pictures show the image of different examples used in the podcast. For example, the drawing of social media icons ( Facebook, Instagram, and YouTube) means introducing the use of multimodality via social media. The last picture is a waving hand, which sums up the process and means the podcast is ending. Under each picture, there is a short sentence that explains the main purpose of the picture, which also contributes to the storyboard content. It is also worth noticing that some numbers are at the top of the images; these are the timelines of the podcast that can help build a storyline of the study and clarify the plan.
In this case, it is obvious that different modes express different meanings in different ways (Sefafini, 2014). In the storyboard, texts and images are the two main modes included in the storyboard. The texts show the main point of each part in building the frame of the entire project. Meanwhile, the images visualize the form of the podcast and provide more details to make the readers more clear about the whole podcast. All modes in the entire storyboard are coherent and united, and they all have meaningful characteristics, which are in line with Kress’s (2015) point of view.
Furthermore, according to the example of storyboarding above, if you want to make use of the storyboard, it is important to know that the storyboard is the framework of a project or a study because storyboarding only can display a plan visually rather than design an idea from the beginning (Jewitt, 2011). Second, all the modes that are used in the storyboard should be integrated and make sense. Then dividing the plan into several main parts and draw them like comics, meanwhile, adding some notes with different modes around the images to generate a completed storyline.
However, it is worth noticing that storyboards also have some limitations. The storyboard can only display some visual modes, such as texts, symbols and pictures; it is unable to show other kinds of non-visual and dynamic modes, such as audio and moving images. It means that they are inappropriate to apply in certain disciplines that need to be shown by audio and video. Besides, the storyboard usually displays a formed idea, so the people who want to create a storyboard should be familiar with their research field and already have some ideas because it is not an ideal way of brainstorming.
Now Multimodal texts are common used in many domains and disciplines. Storyboarding is one application of multimodal text which uses a sequence of image cells and some notes to display a convey information in the mind and communicate ideas clearly (Gilje, 2010). It also conforms to the theoretical assumption (Gress, 2015) of a coherent multimodal project. the storyboard of creating a podcast is a concise example to explain the instruction of it. Though the storyboard has some limitations, the advantages of using images and multiple visual modes to convey information and plans cannot be ignored.
Disclaimer: This blog post is part of a series of different elements in regards to the concept of multimodality. This contains multiple blog posts discussing different aspects of this concept as well as a podcast tying all of this together in verbal form. Here are the links to access our Podcast as well as other blogposts:
Podcast: Discovering Multimodality
Blogposts: Introduction to the project, Intro to Multimodality and Everyday Examples, The History of Multimodality, Multimodality in University Websites, Multimodality in Instagram Reels, Art & Multimodality, and Teaching & Multimodality
Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written communication, 25(2), 166-195.
Brien, A. (2013). Storyboarding. Creating multimodal texts. https://creatingmultimodaltexts.com/production- process/pre-production/storyboarding
Forceville, C. J. (2011). Review of: G. Kress (2010) Multimodality: a social semiotic approach to contemporary communication]. Journal of Pragmatics, 43(14), 3624-3626. https://doi.org/10.1016/j.pragma.2011.06.013
Gilje, Ø. (2010). Multimodal Redesign in Filmmaking Practices: An Inquiry of Young Filmmakers’ Deployment of Semiotic Tools in Their Filmmaking Practice. Written Communication, 27(4), 494–522. https://doi.org/10.1177/0741088310377874
Jewitt, C. E. (2011). The Routledge handbook of multimodal analysis. Routledge/Taylor & Francis Group.
Kress, G. (2015). Semiotic Work. Applied Linguistics and a Social Semiotic Account of Multimodality. AILA Review, 28, 49–71. https://doi.org/10.1075/aila.28.03kreInt
Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. Teachers College Press.