This podcast is part of a series of different elements, which all focus on discovering the concept of multimodality. This contains multiple blog posts discussing different aspects of it as well as this podcast tying all of this together in verbal form. If you want to discover more about multimodality, just press play or browse one of our blogs!
Clivaz, C. & Sankar, M. (2016). Multimodal Literacies. DARIAH Teach. [Training module].
Gilje, Ø. (2010). Multimodal Redesign in Filmmaking Practices: An Inquiry of Young Filmmakers’ Deployment of Semiotic Tools in Their Filmmaking Practice. Written Communication, 27(4), 494–522. doi.org/10.1177/0741088310377874
Jewitt, C. (2008). Multimodality and Literacy in School Classrooms. Review of Research in Education, 32(1), 241-267.
Multimodality is used in many domains, including academic. In academic research, the use of multimodality provides a flexible way for research, because it contains the use of different modes from different perspectives to contribute to a study or a project. With the development of the method used in designing a plan in academics, text-only research is no longer as effective as multimodal one. Multimodal texts are another kind of “text”, which combines at least two modes, such as spoken language, written language, visual (moving and still images), spatial, gestural and audio (Gilje, 2010). For example, the textbooks that combine words and images as well as the ppt with video and audio, which are all the multimodal text and can be applied in academic research. Another application of multimodal text is storyboarding, it is usually be segmented into several panels and provides a visual representation, like a film storyboard. It shows a way by working with different modes rather than focusing only on writing modes (Forceville, 2011). Now, it can also be used in designing a project or research.
This blogpost will first introduce and define the multimodal text. Then, storyboarding will be exemplified and analyzed as an application of multimodal texts in academic research.
VIsual Literacy. D. Hattwig, K. Bussert, & A. Medaille.(2013) PORTAL: LIBRARIES AND THE ACADEMY, Volume 13, Issue 1, p. 75.
What is a multimodal text?
Hallid (1989) explains that multimodal texts as a functional language that is either written or spoken or presented in a different medium of expression. He believes that texts should not be limited to either written or spoken language. Serafini (2014) explains multimodal texts in academic research as a narrow feature that can be distributed in different ways, such as a website, photographic essay or picture book. The latter emphasizes more on the function of distribution and communication and points out the visualization elements used in the text to convey more information. In another word, a multimodal text has more than one mode combining in a text, especially some visual images, such as photographs, graphs, and diagrams ( Serafini, 2014).
Besides, Kress (2015) indicates that multimodality is not a theory, rather, it is a domain for social semiotic work that promotes unity and coherence in the overall text. He noted that only using one mode like language or writing is insufficient to access a discipline question. Therefore, using different modes is necessary. All modes constitute the integrated resources that have significant characteristics use in multimodality (Kress, 2015).
To get more useful information on the use of multimodal text, in the next section, I will introduce an application of multimodal texts, storyboards, and analyze its use.
The use of multimodal texts —Storyboards
A storyboard is a sequence of some cells (Brien, 2013) that using different modes such as texts, images and symbols to comprise different illustrations to displayed pre-visualized ideas, scripts or research plans. Storyboards are used for planning purposes; they are created before the development of the final product, and they are used in showing the progression of an issue through different scenes. All of the elements combined together to balance information and communicate ideas clearly (storyboarding); thus, it is a typical application of multimodal texts. Storyboard design is usually used in preparing for film production and telling the story through visualization (Gilje, 2010); it is primarily used by television commercial advertising clients, film directors and cinematographers visualize the different scenes, create continuation, and identify any problems before their occurrence (Gilje, 2010). However, in academic research, we do not need to create a story with our imagination and change it to texts and images on paper, rather, we can come out our plan of research and integrate the information we coming from different modes in a visual way — by creating storyboards (Bezemer & Kress, 2008).
Figure 1 below is a storyboard made by our group in class that visualizes our ideas of making a podcast about multimodality. (The podcast introduces multimodality and its use in different aspects, the link is at the end of the text.)
Figure 1. A photo of a Podcast Storyboard (Source: Author)
It is a simple storyboard comprising five pictures and a few words; however, it can demonstrate the whole process of the design of a podcast. The first picture shows the introductory part of the podcast: a conversation. The second to fourth pictures show the image of different examples used in the podcast. For example, the drawing of social media icons ( Facebook, Instagram, and YouTube) means introducing the use of multimodality via social media. The last picture is a waving hand, which sums up the process and means the podcast is ending. Under each picture, there is a short sentence that explains the main purpose of the picture, which also contributes to the storyboard content. It is also worth noticing that some numbers are at the top of the images; these are the timelines of the podcast that can help build a storyline of the study and clarify the plan.
In this case, it is obvious that different modes express different meanings in different ways (Sefafini, 2014). In the storyboard, texts and images are the two main modes included in the storyboard. The texts show the main point of each part in building the frame of the entire project. Meanwhile, the images visualize the form of the podcast and provide more details to make the readers more clear about the whole podcast. All modes in the entire storyboard are coherent and united, and they all have meaningful characteristics, which are in line with Kress’s (2015) point of view.
Furthermore, according to the example of storyboarding above, if you want to make use of the storyboard, it is important to know that the storyboard is the framework of a project or a study because storyboarding only can display a plan visually rather than design an idea from the beginning (Jewitt, 2011). Second, all the modes that are used in the storyboard should be integrated and make sense. Then dividing the plan into several main parts and draw them like comics, meanwhile, adding some notes with different modes around the images to generate a completed storyline.
However, it is worth noticing that storyboards also have some limitations. The storyboard can only display some visual modes, such as texts, symbols and pictures; it is unable to show other kinds of non-visual and dynamic modes, such as audio and moving images. It means that they are inappropriate to apply in certain disciplines that need to be shown by audio and video. Besides, the storyboard usually displays a formed idea, so the people who want to create a storyboard should be familiar with their research field and already have some ideas because it is not an ideal way of brainstorming.
Now Multimodal texts are common used in many domains and disciplines. Storyboarding is one application of multimodal text which uses a sequence of image cells and some notes to display a convey information in the mind and communicate ideas clearly (Gilje, 2010). It also conforms to the theoretical assumption (Gress, 2015) of a coherent multimodal project. the storyboard of creating a podcast is a concise example to explain the instruction of it. Though the storyboard has some limitations, the advantages of using images and multiple visual modes to convey information and plans cannot be ignored.
Disclaimer: This blog post is part of a series of different elements in regards to the concept of multimodality. This contains multiple blog posts discussing different aspects of this concept as well as a podcast tying all of this together in verbal form. Here are the links to access our Podcast as well as other blogposts:
Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written communication, 25(2), 166-195.
Brien, A. (2013). Storyboarding. Creating multimodal texts. https://creatingmultimodaltexts.com/production- process/pre-production/storyboarding
Forceville, C. J. (2011). Review of: G. Kress (2010) Multimodality: a social semiotic approach to contemporary communication]. Journal of Pragmatics, 43(14), 3624-3626. https://doi.org/10.1016/j.pragma.2011.06.013
Gilje, Ø. (2010). Multimodal Redesign in Filmmaking Practices: An Inquiry of Young Filmmakers’ Deployment of Semiotic Tools in Their Filmmaking Practice. Written Communication, 27(4), 494–522. https://doi.org/10.1177/0741088310377874
Jewitt, C. E. (2011). The Routledge handbook of multimodal analysis. Routledge/Taylor & Francis Group.
Kress, G. (2015). Semiotic Work. Applied Linguistics and a Social Semiotic Account of Multimodality. AILA Review, 28, 49–71. https://doi.org/10.1075/aila.28.03kreInt
Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. Teachers College Press.
Thinking about our lives, multimodality is used almost everywhere, the way we interact amongst ourselves, the popularity of TV shows, the internet, our class, etc. It is fair to say that where information is transmitted, its multimodality can be found and applied. Beyond that, with the advent of Web 2.0, social media has become popular because of its relative ease of use, quick interconnection with the rest of the world, and we cannot overlook that multimodality is commonly used in social media platforms such as Instagram.
Instagram is an app for sharing photos and videos, as well as being a social network. It is similar to Twitter with followers, but relying more on visual images and less on text ( Jewitt, 2009). Many people may do not enjoy reading, but visual things can surely grab their interests (Socialnomics Trends, 2020). The real-time updates can connect the users with the world. So it is popular among people who want to be welcomed especially the younger generation and contributes to at least 1 billion active users monthly (Tankovska, 2021). Recently, Instagram has introduced reels, a novel way to create a short video that less than 15 seconds with creative tools to alter audio and visual effects and share with their followers or anyone on the platform. It provides a new way for people to use different models to share their lives and convey information.
The blogpost will first introduce briefly the function of Instagram reels and then analyze the use of multimodality in Instagram reels based on a Youtube video Here’s How to Use Instagram Reels (Later, 2020), and then summarize the main point. This blogpost is also a part of our multimodality project, if you want to see other aspects of multimodality, you can find the link at the end of the text.
Instagram
Instagram reels — communication and interaction
Instagram reels is a recently added feature on the Instagram platform on August 5, 2020. Users can slide up one reel to the next using their fingers while watching which invites users to create entertaining videos the user has 15 seconds to post a video that is available to the world.
It is worth mentioning that, the Instagram reel not only contains a video shoot by creators, it also can add other modes, such as extra audio, AR effects, some texts, memes as well as hashtags to make the reel video more attractive and shows to the target audience. Some users use Instagram to show details of their life, allowing other people to see their homepage to gain more followers, which also means that the personal domain turns to the public domain under the influence of multimodal features and the social media environment.
The youtube video Here’s How to Use Instagram Reels (2020) shows the use of Instagram reels, the multimodality use can be analyzed based on it.
Later: #1 Marketing Platform for Instagram.(August 25, 2020). Here’s How to Use Instagram Reels
In this video, the author shows some steps to create a reel and explains how to watch and leave a comment under the reel video. The whole process can be analyzed in two aspects.
For the reel creators, firstly they have to shoot a video in less than 15 seconds as the main content of the reel, then select background music as you like and add it to the video, which can make the video content richer. After that, the creator could incorporate stickers or text some words on the screen relating to the video to explain more details of the video or add some of their individual expressions. This process can be seen as the creation of an ensemble, which is the combination of different modes (Sefafini, 2014). In this video, the reel creator uses several modes such as texts, images, sounds, and combines them to complete the narrative, which enables creators to use these modes to convey their content to communicate with their target audiences.
For the audiences, reels can be found on the Instagram search page with some popular Instagram reels posted by some ‘Instagrammers’. Reels have social functions that give people an opportunity to communicate and interact with the reel creators and other audiences while watching the video. The audiences can use some actions to express their opinions and attitudes about the reel by moving their fingers. When the viewers double tap on the video, it means that they “like” it, and long press the screen it means that they are disinterested in it. In the comments, the audiences have the option of typing some words or using memes to express their feeling about the video, which allows the reel creator and other audience to see their position and react to them. From a Social Semiotic perspective, all the modes constitute one domain of semiotic resources (Kress, 2015), which have some social use in communication. The modes in reels attract audiences to use their senses to see the video but also prompt them to have some social interactions with others.
By analyzing the fundamental functions Instagram reels, here are some points to summarize the multimodality use in reels. One is the mode ensemble is created in the video, as different modes can convey different meanings (Sefafini, 2014). Firstly, the dynamic visual mode video is richer than the linguistic modes of speech and writing to some extent ( Kress, 2015), so it can excellently present concrete details. Together with other modes such as audio, text and image, which can convey more information and arouse emotions, all of the force in “modal ensembles” to convey information (Kress, 2010). These modes coordinate with each other and build the content of the reel richer.
On the other hand, the multimodality use in Instagram Reels increases engagement from the target followers. As Kress (2010) says, multimodality is a theory that looks at communication, the Instagram Reel shows a process of using multimodality to build interaction and communication between creators and audiences. When videos are uploaded, the Instagram algorithm is able to promote the content to a larger audience, and this causes heightened engagement. For the creators, what they can do is creating their own content using multimodality to attract audiences and evoking a sense of participation, for example making a topic to evoke discussion. Therefore, the communication and interaction in Instagram reels can be seen as the result of using multimodality.
Overall，Instagram Reels shows the use of multimodality in social media and close to our daily life. In the process of making reels, first, modal ensembles are used while creating content, dynamic visual modes are combined with other modes to fulfill the content. Then interests and attention of the audience are evoked and generate interaction and communication.
Disclaimer: This blog post is part of a series of different elements in regards to the concept of multimodality. This contains multiple blog posts discussing different aspects of this concept as well as a podcast tying all of this together in verbal form. Here are the links to access our Podcast as well as other blogposts:
Jewitt, C. (ed.) (2009). The Routledge Handbook of multimodal analysis. London: Routledge.
Kress, G. R. (2010). Multimodality: A social semiotic approach to contemporary communication. Taylor & Francis.
Kress, G. (2015). Semiotic work: Applied linguistics and a social semiotic account of multimodality. Aila Review, 28(1), 49-71. https://doi.org/10.1075/aila.28.03kre
Tankovska, H. (Feb 10, 2021). Instagram: distribution of global audiences 2021, by age group. Statista. https://www.statista.com/statistics/325587/instagram-global-age-group/
