Author: Mengxin

Materiality In the Digital Collection: A Case Study of “ MASTERPIECES UP CLOSE” in the Rijksmuseum

The development of technologies and the popularity of the Internet have let collections be presented in digital ways. Many museums create online sites and transform their collections into digital. Especially during the COVID-19 lockdown, museums are not able to open, so a lot of museums transfer their collections into digital and let the public visit them without leaving home. However, there are many differences between traditional collections and digital collections, and materiality is one. In traditional collections, every collection is physical and material that you can see, feel or touch, but in digital collections, the materiality has to be performed in digital ways. Even though the materiality of collections can not be fully presented with the digital tools, the digital collections still find ways to enhance materiality, such as reproducing material features and adding metadata to evoke the sensory response and enhance the physical experience even letting the audiences participate and interact.

In this blog post, I will focus on the materiality in collections and discuss the materiality in digital collections about how the materiality gain and losses in the digital collection. To discuss the materiality more specifically, I will use an interactive virtual digital collection: “Rijksmuseum Masterpieces Up Close” as an example and analyze the materiality performance in this digital collection.

The Role of Materiality in Collections

What is materiality? In brief, materiality is the continuation of matter and material (Browaeys, 2019). Among these, the matter is the stuff that has shape, body, and can be touched (p.3), the matter is “what can be felt and touched” (p.4). Materiality closely connects with them but continues in emotion and way of thinking (p.4). Therefore, materiality not only focuses on the physical and material characteristics of objects but also evokes sensory and intellect response in a more profound way. 

In collections, materiality plays an essential role. It can evoke intellectual meaning, sensory reaction, and emotion-based on tangible characteristics of the object (Lester, 2018). The physical form of an object includes its characteristics such as size, weight, color, and texture. All the features let the audiences generate sensory responses and emotional experiences by looking, feeling, or smelling (Lester, 2018, p.9). However, an object can become a collection not simply rely on its looks or material, it also has to show the culture or history in a particular context. By reflecting the background or history of the object, the materiality of an object lets the audiences know the value and evoke an intellectual response to understand the object deeper. Additionally, the content of a collection, such as the text, can provide both intellectual response and emotional experience (p.9). For example, reading a sad old letter in a disaster lets you know the content and feel the sadness from the text. In this case, your intellect and emotion entwined together.

How can we receive the information and react to it? Our bodies can interact with the environment or an object and focus on it to have a particular cognitive and sensory response without consciousness, which is called “body schema” (p.11). In the process of reacting to a collection, our body feels its materiality of a collection and responds to it in a “close to automatic way” (p.12). Besides this, the reaction of our body also affects the way we understand the object. Therefore, the materiality of collections plays a role in evoking the various responses of our body, and our body shapes how to perceive it. The body response, information and material also shape each other.

The materiality of a collection not only evokes the intellectual, sensory, and emotional response from the audiences but can also be used to convey the thoughts and values of its creators (Lester, 2018). For example, the creator of the object creates it with a specific material and uses text to present his/her wishes and purpose. But not all information can be transmitted by the creators and makes sense for the audience, and a large amount of data can lead to the audiences can not understand them all, only relevant and essential information can make sense for the audiences (Browaeys, 2019). Therefore, the materiality can show the purpose of the creator and affect the information transmission to the audiences. 

Materiality in Digital Collections

With the development of technologies and the transformation of the collections, the digital collection is nothing new. Some physical collections turn into digital and even born-digital collections without physical forms occur. However, unlike traditional collections, digital collections are based on the Internet and technological devices that make the collections presenting on screen and can not be touched or felt directly. In this case, materiality has been changed in the digital collection but presented in other ways.

When digitalizing collections, it seems that the materiality has been diminished, as the digital environment only provides pure information and diminishes the physical experience (Tebeau, 2016). For example, some literacy critics argue the original texts cannot fully translate into other forms. If the original transcript is printed or transmitted digitally, it will lose its meaning because the original one provides “forms of evidence”, but the reproduction does not (Manoff, 2016, p.314). However, physical experience is not the only way to comprehend the world. Although digital materiality is virtual, it can present things in new ways (Browaeys, 2019). 

Materiality in digital collections can perform and enhance in several ways. Different from static physical objects, digital objects can be rearranged, combined, and generate new modes (Manoff, 2016, p.313). Thus, the objects in the digital collections can be shown in different ways and orders to show their materiality better. Additionally, digital technologies can also enhance materiality and connect audiences with them. 

Using our sensory response to enhance the physical experience can also enhance the materiality in the digital era. In the way of translating the physical features into digital space, there is a term called “synesthetic visualization”, which refers to “coupling visual representations with cues for other sensory modalities (Forlini & Hinriches, 2017, p.2)”. Focus on the visual, adding other metadata in audio and screen-based interaction to combine the visual expression of the digital object with other forms to evoke the sensory response. More specifically, there are two forms of synthetic visualization to enhance materiality: represent the material features by getting metadata about the physical object and showing them on screen (p.2); enhance physicalization by combining metadata with the content that related to the materiality (p. 3). 

With the help of digital tools, not only can materiality be enhanced by generating sensory responses, but it can also be enhanced by adding data and context to let the audience participate and build a new space. For example, the sound-based data create the physical experience of listening that can expand the context for a collection (Tebeau, 2016, p.481). Otherwise, the digital tools emphasize the materiality and build the new context of the collection, such as adding metadata and organizing online activities, and enables the audiences to participate and interact with the collection to enhance the experience (Tebeau, 2016). In this way, a new space is built for the audiences to experience the digital materiality and engage in the collection, which the traditional collection does not have.

Therefore, even though the materiality of objects diminish a few in transmitting to digital, the digital technologies can evoke the sensory experience to enhance the materiality and build new context for the audiences. In the next section, there is a case of “Masterpieces Up Close” in Rijksmuseum lets us see how the materiality is enhanced in this digital collection.

A case study: “Rijksmuseum Masterpieces Up Close”  

Website of ‘Rijksmuseum Masterpieces Up Close” (Screenshot)

The Rijksmuseum is the national museum of the Netherlands that collects the art objects in 800 years of history. “Rijksmuseum Masterpieces Up Close” is a digital collection of the Rijksmuseum created in 2020 to present the collections of the Gallery of Honor. Because of the COVID-19 lockdown last year, museums in the Netherlands are not able to open to the public, many museums convert their collections online, and creates digital collections such as “Rijksmuseum Masterpieces Up Close” that enable the public to visit the collection at home. This collection is not a born-digital collection. Instead, it reproduces The Gallery of Honor of the Rijksmuseum online with a 360 degree view as a “virtual tour”. The Gallery of honor is an extended corridor with masterpieces of famous artists in the 17th century on both sides of the alcoves (Gallery of Honour, Rijksmuseum). By visiting this digital collection, you can see the whole gallery and paintings without leaving home.

“Gallery of Honor” Full View (Screenshot)

When entering the “Masterpieces Up Close” website, the visitors can move their mouses and click the arrow mark on the floor to proceed around the Gallery of Honor in the 360 degree view and all the paintings displayed on both sides, same as the offline gallery. It builds a space that shows the complete view of the gallery and gives people a sense of walking around it. In addition, there is a map on the bottom left corner of the website. By clicking the map marks, visitors can go to any block of the gallery without following the original route, enabling the visitor to “design” their trip based on their preference. By watching the full view and choosing the route, “Masterpieces Up Close” enhances the physical experience and builds a new space for visitors to participate and interact with the collection (Tebeau, 2016).

Besides, the digital gallery makes a lot of effort to show the details of the artworks. When clicking the mark on the painting, the painting will zoom in and occupy the whole screen. Unlike only visitors can only watch the masterpieces at a certain distance in traditional collections, in this digital collection, visitors can zoom in and zoom out to see both the full view and the details of the painting only by moving the painting mouses. All the paintings are shown in high-resolution and keep the original appearance, so the material, color even the cracks can be seen clearly. The color and the cracks provide the details that show the history of the painting.

By doing so, the collection evokes a strong visual response through the high-resolution image, and details represent the material features of the artworks to enhance the materiality by evoking sensory response (Forlini & Hinrichs, 2017). 

This digital collection also provides the text and oral story to introduce the masterpieces. By clicking the “play” button, the oral story of the painting will be told in tens of seconds. At the same time, the text of the oral story will also be shown next to the painting. In this case, the audio and text transfer the knowledge to the viewers, and this knowledge can evoke the intellect response of the viewers and understand the painting deeper. Therefore, the context of the artwork is extended by adding audio and evoke a listening response (Tebeau, 2016).

 The Milk Maid (Screenshot)

Take the famous “The Milkmaid” as an example. When playing the sound material, firstly, you will hear the sound of pouring the milk, following by the oral introduction of the painting that includes the year, the theme of the painting, details such as the stream of milk, nails on the wall, and the scattered bread, together with the purpose of the artist. The image also zooms in, following the audio to show the details. In this case, viewers can feel the material of the painting and get knowledge of it as well as generate the sensory response.

Listen to the Audio (Recording from

In general, the “Masterpieces Up Close” collection presents the material features and enhances the physicalization by driving a virtual, audio representation based on screen and showing the content (Forlini & Hinriches, 2017). Even though the painting itself does not generate the audio, the oral storytelling and sound can provide more information and enhance the sensory experience of listening more than just visualization (Tebeau, 2016). By doing so, the visualization and sound-based materials evoke the sensory response of the viewers. Using text and sound to add the content and data gives visitors more knowledge of the paintings and evokes their intellectual response. However, the collection still lacks some metadata, as the text and sound cannot provide the complete information of the painting, such as the size and weight. The lacking metadata leads to the incomplete context of the collection to a certain extent.

Another interesting experience of the “Masterpieces Up Close” collection is the “Key Challenge” that builds a new context of the collection. The “Key Challenge” lets the visitors find keys in the digital Gallery of Honor with the help of hints presented in few words and small icons. This challenge gives the visitors a sense of engagement and has fun in the digital collection. In the process of searching keys, the hints can emphasize the elements of the paintings that make the participants more focus on the content of the artworks and further experience the materiality of those digital works. Participation and interactivity also enhance the experience of the digital collection (Tebeau, 2016).  

Key Challenge (Screenshot)

Overall, the digital collection “Masterpieces Up Close” in the Rijksmuseum enhances the materiality in many digital ways. Even though the online collections cannot provide the materiality the same as the offline visiting as they are limited in the screen, which cannot give the visitor a full sensory experience, such as touching and feeling. However, it produces a 360 degree environment to give people a sense of engagement and adds the sensory experience of the artwork by providing more details of the image as well as showing more information and content by adding text and sound. In addition to that, the “Key Challeng” also adds more fun for the viewers to participate in and further experience the materiality of the artworks.

In sum, materiality plays an important role in collections, as it evokes the response of intellectual, sensory, and emotion of the audiences and also transmits the thoughts and purpose of the creators. Although digital collections have diminished some materiality, they still find ways to enhance the materiality in represent the material feature and enhance the physical experience. The case of “Rijksmuseum Masterpieces Up Close” confirms this view. By adding 360 degree audio, introduction text, audio, and interactive challenge, “Masterpieces Up Close” provides a lot of evidence showing the materiality of the collections and builds a new context for the viewers to participate in and have fun.

Reference List 

Browaeys. C. (2019). Materiality in the digital age. Human beings connected to matter. Presses universitaires de Grenoble.

Forlini, S., & Hinrichs, U. (2017, June). Synesthetic visualization: balancing sensate experience and sense making in digitized print collections. In Proceedings of the conference on Digital Preservation for Social Sciences and Humanities.

Gallery of Honour. Rijksmuseum.

Lester, P. (2018). Of mind and matter: the archive as object. Archives and Records, 39(1), 73-8

Manoff, M. (2006). The materiality of digital collections: Theoretical and historical perspectives. portal: Libraries and the Academy, 6(3), 311-325.

Masterpieces Up Close. Rijksmuseum.

Tebeau, M. (2016). Engaging the Materiality of the Archive in the Digital Age. Collections, 12(4), 475-487.

Reflection- Use Metashape to create a Jigsaw Puzzle 3D Model

The jigsaw puzzle (left) and its 3D model (right)

My object is a jigsaw puzzle, which composed of 1000 small pieces. In the process of making a 3D puzzle model on Metashape, I have faced many challenges and try to solve them.

The most difficult thing to create a 3D puzzle model is to take pictures of good quality. The puzzle pieces are separate and can not be moved together, so for the first time, I put the puzzle in a box and take pictures of them on the table. When putting the pictures in Metashape to align them and build a dense cloud, there are several big holes on the puzzle box, because it is difficult to take the adjacent sides of the box’s right angels clearly, some parts of the box picture are missing so the software can not build a complete model. Then I throw away the box and only put a pile of puzzles on the table and took pictures of the puzzle as well as a part of the table. 

Photo hemisphere

The jigsaw puzzle has so many details and reflective texture, I took pictures from four angles from bottom to top and totally took 64 pictures which build a hemisphere around the puzzle. In order to shoot clear images, I focus the camera on the puzzle overlap part. To avoid the reflection and guarantee enough light, I turn on the light above the puzzle and control the light by opening and closing the curtain.

3D jigsaw puzzle model on Metashape

In the process of building the 3D model in Metashape, things get easier. All of my pictures are aligned successfully. Due to the 3D model contains both puzzle and the table, I do not create a mask on them. I deleted some useless points after alignment and build the dense cloud. I choose the medium quality because of my computer limitation. After building the dense cloud, I delete the points outside the object by rotating the model. However, the 3D puzzle model does not have a bottom because it is put on a table without shooting one. To deal with this, I use a 100% close hole tool to create a bottom for the model. Then I build the texture. The 3D model looks not bad.

The 3D model is uploaded on Sketchfab. This video shows the finished Jigsaw puzzle 3D model published on Sketchfab:

Finished 3D model on Sketchfab

Click HERE to the 3D model in Sketchfab.

Podcast – A Living Room Talk: Discovering Multimodality

This podcast is part of a series of different elements, which all focus on discovering the concept of multimodality. This contains multiple blog posts discussing different aspects of it as well as this podcast tying all of this together in verbal form. If you want to discover more about multimodality, just press play or browse one of our blogs!


Bezemer, J. (2012, March 15). “What is Multimodality” [Video]. Youtube.

Brien, A. (2013). Storyboarding.…oryboarding

Clivaz, C. & Sankar, M. (2016). Multimodal Literacies. DARIAH Teach. [Training module].

Gilje, Ø. (2010). Multimodal Redesign in Filmmaking Practices: An Inquiry of Young Filmmakers’ Deployment of Semiotic Tools in Their Filmmaking Practice. Written Communication, 27(4), 494–522.

Jewitt, C. (2008). Multimodality and Literacy in School Classrooms. Review of Research in Education, 32(1), 241-267.

Multimodality in Academic Area— The Multimodal Text


Multimodality is used in many domains, including academic. In academic research, the use of multimodality provides a flexible way for research, because it contains the use of different modes from different perspectives to contribute to a study or a project. With the development of the method used in designing a plan in academics, text-only research is no longer as effective as multimodal one. Multimodal texts are another kind of “text”, which combines at least two modes, such as spoken language, written language, visual (moving and still images), spatial, gestural and audio (Gilje, 2010). For example, the textbooks that combine words and images as well as the ppt with video and audio, which are all the multimodal text and can be applied in academic research. Another application of multimodal text is storyboarding, it is usually be segmented into several panels and provides a visual representation, like a film storyboard. It shows a way by working with different modes rather than focusing only on writing modes (Forceville, 2011). Now, it can also be used in designing a project or research. 

This blogpost will first introduce and define the multimodal text. Then, storyboarding will be exemplified and analyzed as an application of multimodal texts in academic research.

VIsual Literacy. D. Hattwig, K. Bussert, & A. Medaille.(2013)
 PORTAL: LIBRARIES AND THE ACADEMY, Volume 13, Issue 1, p. 75. 

What is a multimodal text?

Hallid (1989) explains that multimodal texts as a functional language that is either written or spoken or presented in a different medium of expression. He believes that texts should not be limited to either written or spoken language. Serafini (2014) explains multimodal texts in academic research as a narrow feature that can be distributed in different ways, such as a website, photographic essay or picture book. The latter emphasizes more on the function of distribution and communication and points out the visualization elements used in the text to convey more information. In another word, a multimodal text has more than one mode combining in a text, especially some visual images, such as photographs, graphs, and diagrams ( Serafini, 2014).

Besides, Kress (2015) indicates that multimodality is not a theory, rather, it is a domain for social semiotic work that promotes unity and coherence in the overall text. He noted that only using one mode like language or writing is insufficient to access a discipline question. Therefore, using different modes is necessary. All modes constitute the integrated resources that have significant characteristics use in multimodality (Kress, 2015). 

To get more useful information on the use of multimodal text, in the next section, I will introduce an application of multimodal texts, storyboards, and analyze its use.

The use of multimodal texts —Storyboards


A storyboard is a sequence of some cells (Brien, 2013) that using different modes such as texts, images and symbols to comprise different illustrations to displayed pre-visualized ideas, scripts or research plans. Storyboards are used for planning purposes; they are created before the development of the final product, and they are used in showing the progression of an issue through different scenes. All of the elements combined together to balance information and communicate ideas clearly (storyboarding); thus, it is a typical application of multimodal texts. Storyboard design is usually used in preparing for film production and telling the story through visualization (Gilje, 2010); it is primarily used by television commercial advertising clients, film directors and cinematographers visualize the different scenes, create continuation, and identify any problems before their occurrence (Gilje, 2010). However, in academic research, we do not need to create a story with our imagination and change it to texts and images on paper, rather, we can come out our plan of research and integrate the information we coming from different modes in a visual way — by creating storyboards (Bezemer & Kress, 2008). 

Figure 1 below is a storyboard made by our group in class that visualizes our ideas of making a podcast about multimodality. (The podcast introduces multimodality and its use in different aspects, the link is at the end of the text.)

Figure 1. A photo of a Podcast Storyboard (Source: Author)

It is a simple storyboard comprising five pictures and a few words; however, it can demonstrate the whole process of the design of a podcast. The first picture shows the introductory part of the podcast: a conversation. The second to fourth pictures show the image of different examples used in the podcast. For example, the drawing of social media icons ( Facebook, Instagram, and YouTube) means introducing the use of multimodality via social media. The last picture is a waving hand, which sums up the process and means the podcast is ending. Under each picture, there is a short sentence that explains the main purpose of the picture, which also contributes to the storyboard content. It is also worth noticing that some numbers are at the top of the images; these are the timelines of the podcast that can help build a storyline of the study and clarify the plan.

In this case, it is obvious that different modes express different meanings in different ways (Sefafini, 2014). In the storyboard, texts and images are the two main modes included in the storyboard. The texts show the main point of each part in building the frame of the entire project. Meanwhile, the images visualize the form of the podcast and provide more details to make the readers more clear about the whole podcast. All modes in the entire storyboard are coherent and united, and they all have meaningful characteristics, which are in line with Kress’s (2015) point of view.

Furthermore, according to the example of storyboarding above, if you want to make use of the storyboard, it is important to know that the storyboard is the framework of a project or a study because storyboarding only can display a plan visually rather than design an idea from the beginning (Jewitt, 2011). Second, all the modes that are used in the storyboard should be integrated and make sense. Then dividing the plan into several main parts and draw them like comics, meanwhile, adding some notes with different modes around the images to generate a completed storyline.

However, it is worth noticing that storyboards also have some limitations. The storyboard can only display some visual modes, such as texts, symbols and pictures; it is unable to show other kinds of non-visual and dynamic modes, such as audio and moving images. It means that they are inappropriate to apply in certain disciplines that need to be shown by audio and video. Besides, the storyboard usually displays a formed idea, so the people who want to create a storyboard should be familiar with their research field and already have some ideas because it is not an ideal way of brainstorming.

Now Multimodal texts are common used in many domains and disciplines. Storyboarding is one application of multimodal text which uses a sequence of image cells and some notes to display a convey information in the mind and communicate ideas clearly (Gilje, 2010). It also conforms to the theoretical assumption (Gress, 2015) of a coherent multimodal project. the storyboard of creating a podcast is a concise example to explain the instruction of it. Though the storyboard has some limitations, the advantages of using images and multiple visual modes to convey information and plans cannot be ignored.

Disclaimer: This blog post is part of a series of different elements in regards to the concept of multimodality. This contains multiple blog posts discussing different aspects of this concept as well as a podcast tying all of this together in verbal form. Here are the links to access our Podcast as well as other blogposts:

Podcast: Discovering Multimodality

Blogposts: Introduction to the project, Intro to Multimodality and Everyday Examples, The History of Multimodality, Multimodality in University Websites, Multimodality in Instagram Reels, Art & Multimodality, and Teaching & Multimodality

Reference List

Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written communication25(2), 166-195.

Brien, A. (2013). Storyboarding. Creating multimodal texts. process/pre-production/storyboarding

Forceville, C. J. (2011). Review of: G. Kress (2010) Multimodality: a social semiotic approach to contemporary communication]. Journal of Pragmatics, 43(14), 3624-3626.

Gilje, Ø. (2010). Multimodal Redesign in Filmmaking Practices: An Inquiry of Young Filmmakers’ Deployment of Semiotic Tools in Their Filmmaking Practice. Written Communication, 27(4), 494–522.

Jewitt, C. E. (2011). The Routledge handbook of multimodal analysis. Routledge/Taylor & Francis Group.

Kress, G. (2015). Semiotic Work. Applied Linguistics and a Social Semiotic Account of Multimodality. AILA Review, 28, 49–71.

Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. Teachers College Press.

Multimodality in Instagram reels


Thinking about our lives, multimodality is used almost everywhere, the way we interact amongst ourselves, the popularity of TV shows, the internet, our class, etc. It is fair to say that where information is transmitted, its multimodality can be found and applied. Beyond that, with the advent of Web 2.0, social media has become popular because of its relative ease of use, quick interconnection with the rest of the world, and we cannot overlook that multimodality is commonly used in social media platforms such as Instagram.

Instagram is an app for sharing photos and videos, as well as being a social network. It is similar to Twitter with followers, but relying more on visual images and less on text ( Jewitt, 2009). Many people may do not enjoy reading, but visual things can surely grab their interests (Socialnomics Trends, 2020). The real-time updates can connect the users with the world. So it is popular among people who want to be welcomed especially the younger generation and contributes to at least 1 billion active users monthly (Tankovska, 2021). Recently, Instagram has introduced reels, a novel way to create a short video that less than 15 seconds with creative tools to alter audio and visual effects and share with their followers or anyone on the platform. It provides a new way for people to use different models to share their lives and convey information.

The blogpost will first introduce briefly the function of Instagram reels and then analyze the use of multimodality in Instagram reels based on a Youtube video Here’s How to Use Instagram Reels (Later, 2020), and then summarize the main point. This blogpost is also a part of our multimodality project, if you want to see other aspects of multimodality, you can find the link at the end of the text.


Instagram reels — communication and interaction

Instagram reels is a recently added feature on the Instagram platform on August 5, 2020.  Users can slide up one reel to the next using their fingers while watching which invites users to create entertaining videos the user has 15 seconds to post a video that is available to the world. 

It is worth mentioning that, the Instagram reel not only contains a video shoot by creators, it also can add other modes, such as extra audio, AR effects, some texts, memes as well as hashtags to make the reel video more attractive and shows to the target audience. Some users use Instagram to show details of their life, allowing other people to see their homepage to gain more followers, which also means that the personal domain turns to the public domain under the influence of multimodal features and the social media environment.

The youtube video Here’s How to Use Instagram Reels (2020) shows the use of Instagram reels, the multimodality use can be analyzed based on it.

Later: #1 Marketing Platform for Instagram.(August 25, 2020). Here’s How to Use Instagram Reels

In this video, the author shows some steps to create a reel and explains how to watch and leave a comment under the reel video. The whole process can be analyzed in two aspects.

For the reel creators, firstly they have to shoot a video in less than 15 seconds as the main content of the reel, then select background music as you like and add it to the video, which can make the video content richer. After that, the creator could incorporate stickers or text some words on the screen relating to the video to explain more details of the video or add some of their individual expressions. This process can be seen as the creation of an ensemble, which is the combination of different modes (Sefafini, 2014). In this video, the reel creator uses several modes such as texts, images, sounds, and combines them to complete the narrative, which enables creators to use these modes to convey their content to communicate with their target audiences.

For the audiences, reels can be found on the Instagram search page with some popular Instagram reels posted by some ‘Instagrammers’. Reels have social functions that give people an opportunity to communicate and interact with the reel creators and other audiences while watching the video. The audiences can use some actions to express their opinions and attitudes about the reel by moving their fingers. When the viewers double tap on the video, it means that they “like” it, and long press the screen it means that they are disinterested in it. In the comments, the audiences have the option of typing some words or using memes to express their feeling about the video, which allows the reel creator and other audience to see their position and react to them. From a Social Semiotic perspective, all the modes constitute one domain of semiotic resources (Kress, 2015), which have some social use in communication. The modes in reels attract audiences to use their senses to see the video but also prompt them to have some social interactions with others.

By analyzing the fundamental functions Instagram reels, here are some points to summarize the multimodality use in reels. One is the mode ensemble is created in the video, as different modes can convey different meanings (Sefafini, 2014). Firstly, the dynamic visual mode video is richer than the linguistic modes of speech and writing to some extent ( Kress, 2015), so it can excellently present concrete details. Together with other modes such as audio, text and image, which can convey more information and arouse emotions, all of the force in “modal ensembles” to convey information (Kress, 2010). These modes coordinate with each other and build the content of the reel richer.

 On the other hand, the multimodality use in Instagram Reels increases engagement from the target followers. As Kress (2010) says, multimodality is a theory that looks at communication, the Instagram Reel shows a process of using multimodality to build interaction and communication between creators and audiences. When videos are uploaded, the Instagram algorithm is able to promote the content to a larger audience, and this causes heightened engagement. For the creators, what they can do is creating their own content using multimodality to attract audiences and evoking a sense of participation, for example making a topic to evoke discussion. Therefore, the communication and interaction in Instagram reels can be seen as the result of using multimodality.

Overall,Instagram Reels shows the use of multimodality in social media and close to our daily life. In the process of making reels, first, modal ensembles are used while creating content, dynamic visual modes are combined with other modes to fulfill the content. Then interests and attention of the audience are evoked and generate interaction and communication. 

Disclaimer: This blog post is part of a series of different elements in regards to the concept of multimodality. This contains multiple blog posts discussing different aspects of this concept as well as a podcast tying all of this together in verbal form. Here are the links to access our Podcast as well as other blogposts:

Podcast: Discovering Multimodality

Blogposts: Introduction to the project, Intro to Multimodality and Everyday Examples, The History of Multimodality, Multimodality in University Websites, Multimodality in Academic Area— The Multimodal Text, Art & Multimodality, and Teaching & Multimodality

Reference List

Jewitt, C. (ed.) (2009). The Routledge Handbook of multimodal analysis. London: Routledge.

Kress, G. R. (2010). Multimodality: A social semiotic approach to contemporary communication. Taylor & Francis.

Kress, G. (2015). Semiotic work: Applied linguistics and a social semiotic account of multimodality. Aila Review, 28(1), 49-71.

Later: #1 Marketing Platform for Instagram (August 25, 2020). Here’s How to Use Instagram Reels. Youtube.

Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. Teachers College Press. 

Social Trends. (March 18, 2020). Why do people like Instagram?. Socialnomics.

Tankovska, H. (Feb 10, 2021). Instagram: distribution of global audiences 2021, by age group. Statista.

Reflection-period 2

Time goes so fast! Period two will end in one week. This blogpost is a reflection on this period.

In this period, we have two courses — Machines of Knowledge and Design Thinking and Maker Culture. The two courses are both interesting and give me a new perspective about media and some new ways of thinking.

In the Machines of Knowledge course, we talked a lot about the Internet using different theories, and what interested me most is the public sphere. Since we all participate in online society with social media give us more opportunities to express our opinions, the public sphere is also affected by the new social environment. What are the changes and what happened in this new online public sphere? I think I can put them in my further research. Besides, we learned a new method in this course, text analysis. Though it is totally new and a little bit hard for me to use proficiently, I have to say it is really a helpful method to deal with a huge number of data and analyze it objectively, such as the date of social media.

The other course, Design Thinking and Maker Culture is different from other courses we learned before. It doesn’t give us too many specific examples or case studies, rather it performs like a tool and brings us some ideas that we can use in a real project. The topic I am most interested in is multimodality, also it is the theme of our group project. Everybody uses different modes not only in daily life but also in academic research even does not realize them. As for me, I want to know more about the use of multimodality in our life, which can improve the efficiency of study and work. Otherwise, I think the podcast that we are asked to make is also interesting, except for creating the content of it, I learned some skills about recording and editing the sound, which is very useful.

Brief talk: #Kony 2012

What is Kony 2012?

In March 2012, the American non-profit organisation Invisible Children published an online video named Kony 2012. The purpose of this 30-minute video was to make Joseph Kony, the leader of the Lord’s Resistance Army (LRA) in Uganda known globally in order to have him arrested by the end of 2012 when the campaign expired. Within six days of the video release, it had garnered 100 million clicks and had consequently become a hot topic of discussion in the media. Now the clicks are more than one hundred million.

Joseph Kony

The documentary is based on the personal account of Jason Russell, the co-founder of Invisible Children and director of Kony 2012, in Northern Uganda in 2006 and his meeting with Jacob Acaye, a victim of LRA atrocities. In this video, Jason sets himself the task of explaining the Kony 2012 campaign to his four-year-old son. He begins by explaining the story of their “African friend” Jacob, whom Jason helped, and the unjust suffering facing the children of Uganda: tens of thousands of children are kidnapped and some of the girls are brutalised as sex slaves, and to kill and maim tens of thousands of people. There are no human rights under the control of LRA.

From a post-colonialist perspective, this video describes Africa people as victims and serves a colonial discourse by using images that portray the people of Uganda or Africa as primitive and inferior. But the campaign is framed with an approach of politics of pity and emphasising the relation between the fortunate American youth and the unfortunate Black African. Furthermore, it reinforces colonial stereotypes of Africans as helpless and incapable of dealing with atrocities on their own. Furthermore, the campaign is framed with an approach of politics of pity and emphasising the relation between the fortunate American youth and the unfortunate Black African. It reinforces colonial stereotypes of Africans as helpless and incapable of dealing with atrocities on their own. As a result, the way the campaign aims to engage youth through humanitarian empathy or compassion may turn out to be an inadvertent form of imperialism.


Review- Kony 2012


We took two courses in period 1, Transformation in Digital Cultures and Real Virtuality.This reflection is about how do I feel about the two courses and the topic I am interested in both of them.

The course, Transformation and Digital Cultures, talks about some digital technologies that are worth mentioning, as well as how they affect cultures and our society. First, I have to say that the whole course is well designed and I really enjoy the different topics though some of them are difficult to understand. 

The topic I am really interested in is technomoral change since this topic is no longer strange and can not be ignored in our digital life. As we knew, there are not only hard impacts but also soft impacts of technology, it can be said that each technology has soft impacts that change the ethics, core value and morality of individuals and society, for example, moral and ethical issues brought by robots. I also mentioned soft impacts in my academic paper about self-tracking. According to Swierstra (2015), impacts like these are qualitative rather than quantitative, the core values at stake are unclear or contested; and the results are co-produced by the user. So technomoral change, especially its soft impacts, is a wide and flexible topic which is worthy to have further research on. In my further study, I will still focus on the technomoral change brought by new technologies and gain a deeper understanding of the soft impacts, especially on individuals. 

What I am most interested in Real Virtuality is a special method, phenomenology, which is brand new for me. Not only does it studies how we live in the world, but also studies our experience in the midst of being engaged with the world ( Kamphof, 2020).  It is also an important method to work on all of the philosophical concepts and theories in this course. For or our own research, we need to figure out the research question according to our observation and experience and write them in anecdotes, which is a very interesting thing. In my further study, I will continue to use phenomenology as my research method on media studies. 


Kamphof I., (2020),phenomenology reader, Maastricht university press. Educational material.

Swierstra, T. (2015). Identifying the normative challenges posed by technology’s ‘soft’impacts. Etikk i praksis-Nordic Journal of Applied Ethics, (1), 5-20.

Academic Paper: WeChat Movement: How do self-tracking technologies affect our lives?


WeChat is a widely used chat app which has more than one billion users around the world, most of them are from China. WeChat Movement is one of WeChat’s functions, which also has a huge number of users, as long as you are a WeChat user and follow WeChat Movement official account, then allow it to read your location and movement data from your smart phone, it begins to work: record your daily steps, generate steps ranking list. Because it is a self-tracking application based on social media, it has both self- tracking and social functions. So how could it affect users’ lives?

It is worth noting that self-tracking has been a widely studied topic around the world, and there are also many Chinese scholars have studied the influence of WeChat Movement on communication. According to them, self-tracking plays different roles when people use it (Lyall and Robards, 2017). Not only does it record data as a tool, it also generats a range of affective forces by participants (Lupton, 2019) and causes an impact on users’ social and interpersonal relationships (Wan, 2017). It means that self-tracking technology is not a simple technology to track people’s physical condition, it has a certain social impact. Though different scholars have different research focuses, they all point to the “soft impact (Swierstra, 2015)” of technology, which can not be ignored in technomoral change. Researches on self-tracking deserve further consideration since it it first affects people’s life as a technology, and then brings deeper social impacts.


In this paper, I will take a specific self-tracking application: WeChat Movement as my research subject, to discuss what changes WeChat movement has brought to users’ lives and our society. Firstly, I will introduce ethnography and qualitative interviews which are used as main methods. Secondly, I will present a literature review about self-tracking and technomoral change as theoretical sources. Then the data of my qualitative interview will be analyzed, in order to have a discussion with the literature to get my own opinions. At last, I will draw conclusions based on the theories and findings to answer my research questions: to what extent does this self-tracking application, WeChat movement, affect personal life and society?

Translate »
Follow by Email