Mengxin's blog

Multimodality in Academic Area— The Multimodal Text

Introduction

Multimodality is used in many domains, including academic. In academic research, the use of multimodality provides a flexible way for research, because it contains the use of different modes from different perspectives to contribute to a study or a project. With the development of the method used in designing a plan in academics, text-only research is no longer as effective as multimodal one. Multimodal texts are another kind of “text”, which combines at least two modes, such as spoken language, written language, visual (moving and still images), spatial, gestural and audio (Gilje, 2010). For example, the textbooks that combine words and images as well as the ppt with video and audio, which are all the multimodal text and can be applied in academic research. Another application of multimodal text is storyboarding, it is usually be segmented into several panels and provides a visual representation, like a film storyboard. It shows a way by working with different modes rather than focusing only on writing modes (Forceville, 2011). Now, it can also be used in designing a project or research. 

This blogpost will first introduce and define the multimodal text. Then, storyboarding will be exemplified and analyzed as an application of multimodal texts in academic research.

VIsual Literacy. D. Hattwig, K. Bussert, & A. Medaille.(2013)
 PORTAL: LIBRARIES AND THE ACADEMY, Volume 13, Issue 1, p. 75. 

What is a multimodal text?

Hallid (1989) explains that multimodal texts as a functional language that is either written or spoken or presented in a different medium of expression. He believes that texts should not be limited to either written or spoken language. Serafini (2014) explains multimodal texts in academic research as a narrow feature that can be distributed in different ways, such as a website, photographic essay or picture book. The latter emphasizes more on the function of distribution and communication and points out the visualization elements used in the text to convey more information. In another word, a multimodal text has more than one mode combining in a text, especially some visual images, such as photographs, graphs, and diagrams ( Serafini, 2014).

Besides, Kress (2015) indicates that multimodality is not a theory, rather, it is a domain for social semiotic work that promotes unity and coherence in the overall text. He noted that only using one mode like language or writing is insufficient to access a discipline question. Therefore, using different modes is necessary. All modes constitute the integrated resources that have significant characteristics use in multimodality (Kress, 2015). 

To get more useful information on the use of multimodal text, in the next section, I will introduce an application of multimodal texts, storyboards, and analyze its use.

The use of multimodal texts —Storyboards

 

A storyboard is a sequence of some cells (Brien, 2013) that using different modes such as texts, images and symbols to comprise different illustrations to displayed pre-visualized ideas, scripts or research plans. Storyboards are used for planning purposes; they are created before the development of the final product, and they are used in showing the progression of an issue through different scenes. All of the elements combined together to balance information and communicate ideas clearly (storyboarding); thus, it is a typical application of multimodal texts. Storyboard design is usually used in preparing for film production and telling the story through visualization (Gilje, 2010); it is primarily used by television commercial advertising clients, film directors and cinematographers visualize the different scenes, create continuation, and identify any problems before their occurrence (Gilje, 2010). However, in academic research, we do not need to create a story with our imagination and change it to texts and images on paper, rather, we can come out our plan of research and integrate the information we coming from different modes in a visual way — by creating storyboards (Bezemer & Kress, 2008). 

Figure 1 below is a storyboard made by our group in class that visualizes our ideas of making a podcast about multimodality. (The podcast introduces multimodality and its use in different aspects, the link is at the end of the text.)

Figure 1. A photo of a Podcast Storyboard (Source: Author)

It is a simple storyboard comprising five pictures and a few words; however, it can demonstrate the whole process of the design of a podcast. The first picture shows the introductory part of the podcast: a conversation. The second to fourth pictures show the image of different examples used in the podcast. For example, the drawing of social media icons ( Facebook, Instagram, and YouTube) means introducing the use of multimodality via social media. The last picture is a waving hand, which sums up the process and means the podcast is ending. Under each picture, there is a short sentence that explains the main purpose of the picture, which also contributes to the storyboard content. It is also worth noticing that some numbers are at the top of the images; these are the timelines of the podcast that can help build a storyline of the study and clarify the plan.

In this case, it is obvious that different modes express different meanings in different ways (Sefafini, 2014). In the storyboard, texts and images are the two main modes included in the storyboard. The texts show the main point of each part in building the frame of the entire project. Meanwhile, the images visualize the form of the podcast and provide more details to make the readers more clear about the whole podcast. All modes in the entire storyboard are coherent and united, and they all have meaningful characteristics, which are in line with Kress’s (2015) point of view.

Furthermore, according to the example of storyboarding above, if you want to make use of the storyboard, it is important to know that the storyboard is the framework of a project or a study because storyboarding only can display a plan visually rather than design an idea from the beginning (Jewitt, 2011). Second, all the modes that are used in the storyboard should be integrated and make sense. Then dividing the plan into several main parts and draw them like comics, meanwhile, adding some notes with different modes around the images to generate a completed storyline.

However, it is worth noticing that storyboards also have some limitations. The storyboard can only display some visual modes, such as texts, symbols and pictures; it is unable to show other kinds of non-visual and dynamic modes, such as audio and moving images. It means that they are inappropriate to apply in certain disciplines that need to be shown by audio and video. Besides, the storyboard usually displays a formed idea, so the people who want to create a storyboard should be familiar with their research field and already have some ideas because it is not an ideal way of brainstorming.


Now Multimodal texts are common used in many domains and disciplines. Storyboarding is one application of multimodal text which uses a sequence of image cells and some notes to display a convey information in the mind and communicate ideas clearly (Gilje, 2010). It also conforms to the theoretical assumption (Gress, 2015) of a coherent multimodal project. the storyboard of creating a podcast is a concise example to explain the instruction of it. Though the storyboard has some limitations, the advantages of using images and multiple visual modes to convey information and plans cannot be ignored.


Disclaimer: This blog post is part of a series of different elements in regards to the concept of multimodality. This contains multiple blog posts discussing different aspects of this concept as well as a podcast tying all of this together in verbal form. Here are the links to access our Podcast as well as other blogposts:

Podcast: Discovering Multimodality

Blogposts: Introduction to the project, Intro to Multimodality and Everyday Examples, The History of Multimodality, Multimodality in University Websites, Multimodality in Instagram Reels, Art & Multimodality, and Teaching & Multimodality

Reference List

Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written communication25(2), 166-195.

Brien, A. (2013). Storyboarding. Creating multimodal texts. https://creatingmultimodaltexts.com/production- process/pre-production/storyboarding

Forceville, C. J. (2011). Review of: G. Kress (2010) Multimodality: a social semiotic approach to contemporary communication]. Journal of Pragmatics, 43(14), 3624-3626. https://doi.org/10.1016/j.pragma.2011.06.013

Gilje, Ø. (2010). Multimodal Redesign in Filmmaking Practices: An Inquiry of Young Filmmakers’ Deployment of Semiotic Tools in Their Filmmaking Practice. Written Communication, 27(4), 494–522. https://doi.org/10.1177/0741088310377874

Jewitt, C. E. (2011). The Routledge handbook of multimodal analysis. Routledge/Taylor & Francis Group.

Kress, G. (2015). Semiotic Work. Applied Linguistics and a Social Semiotic Account of Multimodality. AILA Review, 28, 49–71. https://doi.org/10.1075/aila.28.03kreInt

Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. Teachers College Press.

Multimodality in Instagram reels

Introduction

Thinking about our lives, multimodality is used almost everywhere, the way we interact amongst ourselves, the popularity of TV shows, the internet, our class, etc. It is fair to say that where information is transmitted, its multimodality can be found and applied. Beyond that, with the advent of Web 2.0, social media has become popular because of its relative ease of use, quick interconnection with the rest of the world, and we cannot overlook that multimodality is commonly used in social media platforms such as Instagram.

Instagram is an app for sharing photos and videos, as well as being a social network. It is similar to Twitter with followers, but relying more on visual images and less on text ( Jewitt, 2009). Many people may do not enjoy reading, but visual things can surely grab their interests (Socialnomics Trends, 2020). The real-time updates can connect the users with the world. So it is popular among people who want to be welcomed especially the younger generation and contributes to at least 1 billion active users monthly (Tankovska, 2021). Recently, Instagram has introduced reels, a novel way to create a short video that less than 15 seconds with creative tools to alter audio and visual effects and share with their followers or anyone on the platform. It provides a new way for people to use different models to share their lives and convey information.

The blogpost will first introduce briefly the function of Instagram reels and then analyze the use of multimodality in Instagram reels based on a Youtube video Here’s How to Use Instagram Reels (Later, 2020), and then summarize the main point. This blogpost is also a part of our multimodality project, if you want to see other aspects of multimodality, you can find the link at the end of the text.

Instagram

Instagram reels — communication and interaction

Instagram reels is a recently added feature on the Instagram platform on August 5, 2020.  Users can slide up one reel to the next using their fingers while watching which invites users to create entertaining videos the user has 15 seconds to post a video that is available to the world. 

It is worth mentioning that, the Instagram reel not only contains a video shoot by creators, it also can add other modes, such as extra audio, AR effects, some texts, memes as well as hashtags to make the reel video more attractive and shows to the target audience. Some users use Instagram to show details of their life, allowing other people to see their homepage to gain more followers, which also means that the personal domain turns to the public domain under the influence of multimodal features and the social media environment.

The youtube video Here’s How to Use Instagram Reels (2020) shows the use of Instagram reels, the multimodality use can be analyzed based on it.

Later: #1 Marketing Platform for Instagram.(August 25, 2020). Here’s How to Use Instagram Reels

In this video, the author shows some steps to create a reel and explains how to watch and leave a comment under the reel video. The whole process can be analyzed in two aspects.

For the reel creators, firstly they have to shoot a video in less than 15 seconds as the main content of the reel, then select background music as you like and add it to the video, which can make the video content richer. After that, the creator could incorporate stickers or text some words on the screen relating to the video to explain more details of the video or add some of their individual expressions. This process can be seen as the creation of an ensemble, which is the combination of different modes (Sefafini, 2014). In this video, the reel creator uses several modes such as texts, images, sounds, and combines them to complete the narrative, which enables creators to use these modes to convey their content to communicate with their target audiences.

For the audiences, reels can be found on the Instagram search page with some popular Instagram reels posted by some ‘Instagrammers’. Reels have social functions that give people an opportunity to communicate and interact with the reel creators and other audiences while watching the video. The audiences can use some actions to express their opinions and attitudes about the reel by moving their fingers. When the viewers double tap on the video, it means that they “like” it, and long press the screen it means that they are disinterested in it. In the comments, the audiences have the option of typing some words or using memes to express their feeling about the video, which allows the reel creator and other audience to see their position and react to them. From a Social Semiotic perspective, all the modes constitute one domain of semiotic resources (Kress, 2015), which have some social use in communication. The modes in reels attract audiences to use their senses to see the video but also prompt them to have some social interactions with others.

By analyzing the fundamental functions Instagram reels, here are some points to summarize the multimodality use in reels. One is the mode ensemble is created in the video, as different modes can convey different meanings (Sefafini, 2014). Firstly, the dynamic visual mode video is richer than the linguistic modes of speech and writing to some extent ( Kress, 2015), so it can excellently present concrete details. Together with other modes such as audio, text and image, which can convey more information and arouse emotions, all of the force in “modal ensembles” to convey information (Kress, 2010). These modes coordinate with each other and build the content of the reel richer.

 On the other hand, the multimodality use in Instagram Reels increases engagement from the target followers. As Kress (2010) says, multimodality is a theory that looks at communication, the Instagram Reel shows a process of using multimodality to build interaction and communication between creators and audiences. When videos are uploaded, the Instagram algorithm is able to promote the content to a larger audience, and this causes heightened engagement. For the creators, what they can do is creating their own content using multimodality to attract audiences and evoking a sense of participation, for example making a topic to evoke discussion. Therefore, the communication and interaction in Instagram reels can be seen as the result of using multimodality.


Overall,Instagram Reels shows the use of multimodality in social media and close to our daily life. In the process of making reels, first, modal ensembles are used while creating content, dynamic visual modes are combined with other modes to fulfill the content. Then interests and attention of the audience are evoked and generate interaction and communication. 


Disclaimer: This blog post is part of a series of different elements in regards to the concept of multimodality. This contains multiple blog posts discussing different aspects of this concept as well as a podcast tying all of this together in verbal form. Here are the links to access our Podcast as well as other blogposts:

Podcast: Discovering Multimodality

Blogposts: Introduction to the project, Intro to Multimodality and Everyday Examples, The History of Multimodality, Multimodality in University Websites, Multimodality in Academic Area— The Multimodal Text, Art & Multimodality, and Teaching & Multimodality

Reference List

Jewitt, C. (ed.) (2009). The Routledge Handbook of multimodal analysis. London: Routledge.

Kress, G. R. (2010). Multimodality: A social semiotic approach to contemporary communication. Taylor & Francis.

Kress, G. (2015). Semiotic work: Applied linguistics and a social semiotic account of multimodality. Aila Review, 28(1), 49-71. https://doi.org/10.1075/aila.28.03kre

Later: #1 Marketing Platform for Instagram (August 25, 2020). Here’s How to Use Instagram Reels. Youtube. https://www.youtube.com/watch?v=mn2l4gKjSb8&t=312s

Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. Teachers College Press. 

Social Trends. (March 18, 2020). Why do people like Instagram?. Socialnomics. https://socialnomics.net/2020/03/18/why-do-people-like-instagram/

Tankovska, H. (Feb 10, 2021). Instagram: distribution of global audiences 2021, by age group. Statista. https://www.statista.com/statistics/325587/instagram-global-age-group/

Reflection-period 2

Time goes so fast! Period two will end in one week. This blogpost is a reflection on this period.

In this period, we have two courses — Machines of Knowledge and Design Thinking and Maker Culture. The two courses are both interesting and give me a new perspective about media and some new ways of thinking.

In the Machines of Knowledge course, we talked a lot about the Internet using different theories, and what interested me most is the public sphere. Since we all participate in online society with social media give us more opportunities to express our opinions, the public sphere is also affected by the new social environment. What are the changes and what happened in this new online public sphere? I think I can put them in my further research. Besides, we learned a new method in this course, text analysis. Though it is totally new and a little bit hard for me to use proficiently, I have to say it is really a helpful method to deal with a huge number of data and analyze it objectively, such as the date of social media.

The other course, Design Thinking and Maker Culture is different from other courses we learned before. It doesn’t give us too many specific examples or case studies, rather it performs like a tool and brings us some ideas that we can use in a real project. The topic I am most interested in is multimodality, also it is the theme of our group project. Everybody uses different modes not only in daily life but also in academic research even does not realize them. As for me, I want to know more about the use of multimodality in our life, which can improve the efficiency of study and work. Otherwise, I think the podcast that we are asked to make is also interesting, except for creating the content of it, I learned some skills about recording and editing the sound, which is very useful.

Brief talk: #Kony 2012

What is Kony 2012?

In March 2012, the American non-profit organisation Invisible Children published an online video named Kony 2012. The purpose of this 30-minute video was to make Joseph Kony, the leader of the Lord’s Resistance Army (LRA) in Uganda known globally in order to have him arrested by the end of 2012 when the campaign expired. Within six days of the video release, it had garnered 100 million clicks and had consequently become a hot topic of discussion in the media. Now the clicks are more than one hundred million.

Joseph Kony

The documentary is based on the personal account of Jason Russell, the co-founder of Invisible Children and director of Kony 2012, in Northern Uganda in 2006 and his meeting with Jacob Acaye, a victim of LRA atrocities. In this video, Jason sets himself the task of explaining the Kony 2012 campaign to his four-year-old son. He begins by explaining the story of their “African friend” Jacob, whom Jason helped, and the unjust suffering facing the children of Uganda: tens of thousands of children are kidnapped and some of the girls are brutalised as sex slaves, and to kill and maim tens of thousands of people. There are no human rights under the control of LRA.

From a post-colonialist perspective, this video describes Africa people as victims and serves a colonial discourse by using images that portray the people of Uganda or Africa as primitive and inferior. But the campaign is framed with an approach of politics of pity and emphasising the relation between the fortunate American youth and the unfortunate Black African. Furthermore, it reinforces colonial stereotypes of Africans as helpless and incapable of dealing with atrocities on their own. Furthermore, the campaign is framed with an approach of politics of pity and emphasising the relation between the fortunate American youth and the unfortunate Black African. It reinforces colonial stereotypes of Africans as helpless and incapable of dealing with atrocities on their own. As a result, the way the campaign aims to engage youth through humanitarian empathy or compassion may turn out to be an inadvertent form of imperialism.

Reference

Review- Kony 2012 https://www.e-ir.info/2013/01/30/review-kony-2012/

Reflection

We took two courses in period 1, Transformation in Digital Cultures and Real Virtuality.This reflection is about how do I feel about the two courses and the topic I am interested in both of them.

The course, Transformation and Digital Cultures, talks about some digital technologies that are worth mentioning, as well as how they affect cultures and our society. First, I have to say that the whole course is well designed and I really enjoy the different topics though some of them are difficult to understand. 

The topic I am really interested in is technomoral change since this topic is no longer strange and can not be ignored in our digital life. As we knew, there are not only hard impacts but also soft impacts of technology, it can be said that each technology has soft impacts that change the ethics, core value and morality of individuals and society, for example, moral and ethical issues brought by robots. I also mentioned soft impacts in my academic paper about self-tracking. According to Swierstra (2015), impacts like these are qualitative rather than quantitative, the core values at stake are unclear or contested; and the results are co-produced by the user. So technomoral change, especially its soft impacts, is a wide and flexible topic which is worthy to have further research on. In my further study, I will still focus on the technomoral change brought by new technologies and gain a deeper understanding of the soft impacts, especially on individuals. 

What I am most interested in Real Virtuality is a special method, phenomenology, which is brand new for me. Not only does it studies how we live in the world, but also studies our experience in the midst of being engaged with the world ( Kamphof, 2020).  It is also an important method to work on all of the philosophical concepts and theories in this course. For or our own research, we need to figure out the research question according to our observation and experience and write them in anecdotes, which is a very interesting thing. In my further study, I will continue to use phenomenology as my research method on media studies. 

Reference

Kamphof I., (2020),phenomenology reader, Maastricht university press. Educational material.

Swierstra, T. (2015). Identifying the normative challenges posed by technology’s ‘soft’impacts. Etikk i praksis-Nordic Journal of Applied Ethics, (1), 5-20.

Academic Paper: WeChat Movement: How do self-tracking technologies affect our lives?

Introduction

WeChat is a widely used chat app which has more than one billion users around the world, most of them are from China. WeChat Movement is one of WeChat’s functions, which also has a huge number of users, as long as you are a WeChat user and follow WeChat Movement official account, then allow it to read your location and movement data from your smart phone, it begins to work: record your daily steps, generate steps ranking list. Because it is a self-tracking application based on social media, it has both self- tracking and social functions. So how could it affect users’ lives?

It is worth noting that self-tracking has been a widely studied topic around the world, and there are also many Chinese scholars have studied the influence of WeChat Movement on communication. According to them, self-tracking plays different roles when people use it (Lyall and Robards, 2017). Not only does it record data as a tool, it also generats a range of affective forces by participants (Lupton, 2019) and causes an impact on users’ social and interpersonal relationships (Wan, 2017). It means that self-tracking technology is not a simple technology to track people’s physical condition, it has a certain social impact. Though different scholars have different research focuses, they all point to the “soft impact (Swierstra, 2015)” of technology, which can not be ignored in technomoral change. Researches on self-tracking deserve further consideration since it it first affects people’s life as a technology, and then brings deeper social impacts.

Ranking

In this paper, I will take a specific self-tracking application: WeChat Movement as my research subject, to discuss what changes WeChat movement has brought to users’ lives and our society. Firstly, I will introduce ethnography and qualitative interviews which are used as main methods. Secondly, I will present a literature review about self-tracking and technomoral change as theoretical sources. Then the data of my qualitative interview will be analyzed, in order to have a discussion with the literature to get my own opinions. At last, I will draw conclusions based on the theories and findings to answer my research questions: to what extent does this self-tracking application, WeChat movement, affect personal life and society?

The idea of blog design

I just want my blog being simple, clear and easy to explore, of course, has a sense of beauty.

First of all, let us focus on the main color of my blog– light blue, and the main picture on the home page– waves, which I took this when I traveled to the seaside. They look harmonious and can make viewers calm down.

the home page👍

The main body of my blog is divided into two parts, the left are some widgets: search bar, courses, archives, and tags, and the right are some recent posts. Of course, you can share the content to your social media using the plugin buttons.

In implement stage, I would also add some photos linked to other pages and also add some videos if need. Then I will post new posts both about my study and life.

Some small tips:

  • There is a translator in the lower left corner of the page
  • You can turn on the night mode in the lower right corner

Hope you like it! 🙂

Anecdotes: Having class online

Here are two anecdotes about my online class experience.

  1. The first day of class

It is the first day of our class, I am sitting on a bench in a garden feeling a little bit of nervous. It is a snap decision to have class outside, I think, but I do not have other choices. I wish I am at home now, and the scene of my bedroom come to my mind, a gleam of bright light comes through the window and light up the whole room, quiet and comfortable. But now, there are just some trees around me, and a woman is making a phone call on my left side. “Good morning, everyone!” The voice of our teacher comes from my earphone. I get back to my senses and stare at the mobile screen. I see everyone is sitting in front of the desk except me, sitting outside with a tall tree over my head. No time to focus on others! Thinking about how to introduce yourself, it is your turn! I say to myself in mind. I become so nervous that even I can not hear what my classmates saying clearly. Besides, the woman on my left side must hear what I am saying when I start my self-introduction. How strange a Chinese girl is sitting on a bench speaking English to the mobile! Oh, it is my turn, I hear the teacher saying my name and then everyone stops taking, waiting for me to begin. 

2. Uniqueness of online class

In a tutorial class, the teacher divides us into small breakout groups to discuss what online experience is different from offline. I am in a four-person group, everyone is looking at the screen through the camera that we can see all of us. Someone says though we can talk to each other, there still a long distance between us, it is the Internet that builds our relationship in class. Suddenly a sentence comes to my screen in the middle says:” your Internet connection is unstable.” Then I see all videos are freezing, everyone stops their movements. After that, voices from the meeting disappear, leave me in silence. So I have to stop talking with my classmates and fix my Internet connection as soon as possible. It takes me half a minute, and then I enter the group discussion again, I see the other three group members are looking at me and laugh, a girl jokes:” Mengxin has just experienced the uniqueness of the online class.” I say sorry about my dropping and laugh too. That is really a difference for online classes that offline classes do not have.

Hello world!

Welcome to MA in Media Studies: Digital Cultures at Maastricht University. This is your first post. Edit or delete it, then start writing!

Translate »
Follow by Email
Instagram