#Multimodality!

Have you ever heard of the term Multimodality? Now, if you catch yourself thinking “Multimodality? What is that exactly?” — don’t worry, I’ve got you! My friends Camilla, Eleni and I have created a podcast episode that will help you to get a better understanding of multimodality: we will explain what it means by providing you with a brief introduction to the concept and, on top of that, we will also discuss a couple of day-to-day examples. So if you would like to get a better understanding of multimodality, I would suggest to click on this link and listen to our podcast!

Now that you know what multimodality is, I will guide you through an analysis of how Instagram could be used to express multimodal ensembles for various purposes and what different ways there are to interact in a multimodal manner with other users of the app.

---

Multimodality as an Adjective

First of all, let us see how what the connection is between multimodality and digital technologies. Serafini (2014) refers to multimodality as “an adjective that can be used to describe a specific type of text or ensemble (p. 12). He mentions that multimodal entities include “a variety of modes to communicate or represent concepts and information” (p.12). For print-based texts there are three specific modes that we might encounter:
(1) Textual modes, i.e. written languages; (2) Visual modes, such as photographs, drawings and paintings; (3) Design modes, such as typography (Serafini, 2014, p. 13). However, he also mentions that multimodal products do not have to be print-based — they can occur in digital environments too. Due to the rise (and continuous developments) of digital technologies, there are many new modes that can be added to the list, e.g. “sound effects, moving images, and other digitally rendered elements (Serafini, 2014, p. 13). 

One example of a digital technology that allows utilisation of these different modes, is the very popular app Instagram. Over the past decade, Instagram the number of people using this app has significantly been increasing (Chen, 2020). In October 2020, it was calculated that there are over 1 billion active users of the app (Clement, 2020). Originally, Instagram was designed to be a social photography application for smartphones, on which users could share their own photographs and visual works with their ‘followers,’ i.e. people from their own social network and/or other users of the app  (Mirsarraf, et. al., 2017, p. 1). Due to the communicative nature of this platform, Instagram can be seen as a semiotic, social, cultural and technological resource that people can use to create meaning and communicate this meaning to other people (Poulsen, 2018, p. 124; Koowuttayakorn, 2018, p. 140). Considering the origin of the app, the visual mode could be regarded as the salient mode of this platform. However, Instagram offers many other semiotic modes besides images, which allow people to create a multimodal ensemble with an impactful or meaningful message.

---

A Multimodal Post 

So how exactly can Instagram be used as a multimodal tool? We will begin with one of the most basic utilisation of the app: creating a post. Considering that Instagram started out as a photography app, it is hardly surprising that the platform provides a variety of photography-based editorial tools which allow you to edit your image to preference — for example, by adjusting the brightness, contrast, or saturation of the photo. However, if one is looking for a quicker and easier way to enhance an image, there is a possibility of using one of the various pre-edited filters that the app offers.

After editing the photo, it is important to add another mode to this visual image in order to create a true multimodal ensemble. To do this, the edited photo (which represents the visual mode) can be combined with a written mode by adding a caption, i.e. a brief, textual message that will be attached underneath your posted image. The words used in this caption will hugely influence the type of message that will be presented: the same photograph, with a different caption, might completely change the meaning of the multimodal ensemble. 

• 

• 

In order to make the post even more multimodal, one can also insert hashtags and/or tags. Both of these features could be considered keyword metadata that can be used for cross-referencing content or referring to a specific person, subject or theme (Hashtag, n.d., para. 1). In a way, when using hashtags and tags, one is making a direct link between their post and a specific concept or user of Instagram, which will further elaborate on the meaning of your multimodal ensemble. After combining the visual image together with all the textual information and linking the ensemble to some specific themes and/or people, the post is ready to be shared with your social connections and followers.

Thus, as can be seen in the examples above, the action of ‘posting on Instagram’ requires people to utilise a combination of visual and textual features. Multimodality is, therefore, needed for the meaning-making process of the ensemble, as it allows people to create the very specific message that they want to convey. Creating a post on Instagram, therefore, is not as simple as merely uploading a photograph or image. The users of the app are encouraged to use different features such as filters, hashtags and textual captions, all of which carry a “meaning potential” that one can utilise in order to modify their images and illustrate their emotions thought their photos (Poulsen, 2018, p. 132).

• 

• 

Although the app started out as a platform in which photographs were the “centrepiece of social interaction,” with features that were mainly limited to posting foto’s on a personal account, the app has now developed into a platform in which videos seem to become very prominent as well (Poulsen, 2018, p. 137). Over the years, the app elaborated and expanded the number of features and tools that the users can utilise. Consequently, when posting a multimodal ensemble, the users of Instagram participate in a continuous process of meaning-making by selecting specific features from a wide range of multimodal resources provided by Instagram (Poulsen & Kvåle, 2018, p. 700).

---

Multimodal Interaction  

Now that we understand how to share a specific with other people on Instagram, we will be going a step further and see how we can use this app to interact with other people. After all, the app is supposed to be a social application that allows communication between the users and their followers. Although the app might be considered a modernised version or simulation of older photographic media (with some additional editing features, of course) the app is often not merely used for the practices of making, editing and showcasing photos (Poulsen & Kvåle, 2018, p. 712). The ultimate goal of posting a multimodal ensemble on Instagram is the initiation of interaction with their followers. This type of communication could be referred to as “user-to-user interaction,” i.e. interaction between the users of a specific app — which can take place in a variety of different ways (Jovanic & van Leeuwen, 2018, p. 685).

First of all, it is important to note that the user-to-user interaction on Instagram does not necessarily have to be multimodal; it could be entirely text-based and, therefore, portraying only one mode — the textual mode. For example, people who see a specific post on Instagram can reply to this ensemble by means of “verbally realised responses” — also known as ‘comments’ — in the designated comment section underneath the caption (Jovanic & van Leeuwen, 2018, p. 687). However, many people often use a more multimodal dynamic when interacting with other users. In 2015, for instance, Instagram reported that almost half of the texts on this platform contained emojis (Novak et al., 2015, p. 1). Visual features such as emojis can show a specific feeling, emotion or mood that can clarify or reinforce a written text message in a non-verbal manner. Over the years, Instagram has extended the amount of available visual symbols and, apart from emoji’s, one can now also use animated images such as gifs and stickers.

Over the course of time, Instagram kept developing and refining as a platform which allowed users to have more social interaction compared to the early versions of the app (Poulsen, 2018, p. 137). In 2016, for example, Instagram launched a specific feature that allowed an even more multimodal way of communicating with each other: Instagram stories (Poulsen, 2018, p. 131). With this feature, people can upload day-to-day moments on their Instagram account by means of photos, images and videos that will disappear after twenty-four hours (Poulsen, 2018, p. 131). There are many different features in which a variety of modes can be used in Instagram stories, a couple of which are listed here below:

• Quick Reactions: with this feature, followers can quickly send a visual comment to someone’s story via a collection of pre-selected emoji’s);
• Polls, Quizzes or Questions: with this feature, your followers can directly respond to your questions; 
• Music: with this feature, someone can select a specific song and add it as background music to their story. 
• Donation: with this feature one can ask their followers to donate some money to a non-profit organisation
• Location: this feature will show the location of where one has created their story. 
• Instagram Live: with this feature, people can do a live broadcast — either alone, or together with another user of Instagram. The people who are watching this live broadcast can immediately communicate with the people who are live broadcasting (which can be done just textually or visually, but also by means of a combination of both these modes).

---

Multimodality & Intent

Although many people use Instagram as a tool for personal communication and social networking, the app is also often used for professional purposes. Designers, businesses, models, and other professionals can utilise Instagram both for branding as well as sales purposes (Bearne, 2015). Apart from marketing purposes, it also allows companies to build and maintain a trustworthy relationship with their customers (Smith, 2020). Instagram, thus, seems to be a very useful platform for a variety of different purposes. In fact, an analysis by Kruk et al. (2019) showed that there where eight specific intents to be found on Instagram (p. 3):

1. Advocative: one could create content in order to advocate a specific political or sociological view.
2. Promotive: for sales purposes, one could use Instagram to create advertisements. When selling a physical object, the type of content is could be accompanied by the #Ad tag. However, one could also use Instagram to promote a digital product such as an app. Duolingo is an example of this, in which they use Instagram to promote their free language learning app.
3. Exhibitionist: creating content that shows a self-image for the follower by means of selfies, pictures of goods or belongings like clothes, make-up or pets, etc. The ‘self’ is usually the most prominent aspect of the post (Kruk et al., p. 5). 
4. Expressive: content that expresses certain emotions or praises a specific person (or group of people). These post could, for example, be accompanied by hashtags that show a specific form of “affective intent” (Kruk et al., p. 5).
5. Informative: content that provides factual information about a specific subject, topic or event. News websites or journalistic accounts often provide informative content. 
6. Entertainment: content which contains humorous or artistic intentions. Think about comedy sketches, art photography or daily memes. Although these types of content are very different, they are all a form of entertainment.
7. Provocative/discrimination: content that involves discriminating and directly attacking a specific individual or group of people.
8. Provocative/controversial: providing content that is deemed to be shocking or addressing a specific controversial topic. This type of content is usually meant to inspire or provoke people.

Do you want to know how multimodality can be used in digital journalism platforms such as, for example, Vox News? Check out Camila’s blogpost! 

Or are you interested in how multimodality could be used for educational purposes such as, for example, in language learning apps like Duolingo? Check out Eleni’s blogpost!

---

References

Bearne, S. (2015). Snappy Happy: How to use Instagram. The Guardian. Retrieved on 8 November 2020 from https://www.theguardian.com/small-business-network/2015/jan/13/how-use- instagram-promote-business.  

Chen. J. (2020). Important Instagram stats you need to know for 2020. Sproutsocial. Retrieved from: https://sproutsocial.com/insights/instagram-stats/

Clement. J. (2020). Distribution of Instagram users worldwide as of October 2020, by age group. Statista. Retrieved from: https://www.statista.com/statistics/325587/instagram-global-age-group/

Hashtag. (n.d.) In Wikipedia. Retrieved from: https://en.wikipedia.org/wiki/Hashtag

Jovanovic, D. & van Leeuwen, T. (2018) Multimodal dialogue on social media. Social Semiotics, 28(5), pp. 683-699.
DOI: 10.1080/10350330.2018.1504732

Koowuttayakorn, S. (2018). An Investigation of Instagram’s Metonymy: A Multimodal Social Semiotic Approach. In LEARN Journal : Language Education and Acquisition Research Network Journal, 1(11).

Kruk, J., Lubin, J., Sikka, K. Lin, X., Jurafsky, D. And Divakaran, A. (2019). Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts. Association for Computational Linguistics.
DOI: 10.18653/v1/D19-1469

Mirsarraf, M., Shairi, H. & Ahmadpanah A. (2017). Social Semiotic Aspects of Instagram Social Network. IEEE International Conference on INnovations in Intelligent SysTems and Applications (INISTA).
DOI: 10.1109/INISTA.2017.8001204

Novak, P., Smailović, J., Sluban, B. & Mozetič, I. (2015). Sentiment of Emojis. PLoS ONE, 10(12).
DOI: 10.1371/journal.pone.0144296 

Poulsen, S. (2018). Becoming a semiotic technology – a historical study of Instagram’s tools for making and sharing photos and videos. Internet Histories, 2(1), pp. 121-139.
DOI: 0.1080/24701475.2018.1459350

Poulsen, S. Kvåle, G. (2018). Studying social media as semiotic technology: a social semiotic multimodal framework. In Social Semiotics, 29(5), pp. 700-717.
DOI: 10.1080/10350330.2018.1505689 

Serafini, F. (2014). Reading the Visual: An Introduction to Teaching Multimodal Literacy (New York: Teachers College press).

Smith, M. (2020). Brands, fans and influencers: why social media for small business is about more than sales. The Guardian. Retrieved on November 8, 2020 from: https://www.the guardian.com/go-make-it/2020/jun/12/brands-fans-and-influencers-why-social-media-for- small-business-is-about-more-than-sales