Multimodality, Knowledge to go: The History of Multimodality

‘Multimodality’ is a term that is now widely used in the academic world，which was first coined in the mid-1990s. It is used, for instance, by Charles Goodwin, in a seminal article that he submitted to the Journal of Pragmatics in 1998 (Goodwin, 2000)Since then, a myriad of conferences, monographs, edited volumes and other academic discussion forums have been produced that are dedicated to multimodality(Jewitt et al , 2016). So there is no ‘multimodality ‘ in the world before 1990s? The answer is no!

In this blog, I will briefly introduce the origin and historical development of multimodality in a multimodal way. Since the term multimodality was introduced in the 1990s, I will use the 1990s as the first time node to introduce the history of multimodality. After the arrival of the digital age, great changes have taken place in the use of multimodality, so this is my second time node.

 If you are interested, please keep reading!

What is multimodality

Multimodality refers to the diversity of communicative symbols in oral or written communication（Dai , 2013). If you want to understand this word in an easier way, you can  break it apart, then you  have multi and modes, so it means the use of many modes. Mode is a channel that expresses a message. It can include written language, spoken language, and patterns of meaning that are visual, audio, gestural, etc. When we use more than one mode to express meaning, we have multimodality. For example, when you use gestures while talking, then this is a multimodal way.

Now you know what is multimodality, next we come to the origin of multimodality.

Origins of multimodality

Numerous accounts of language are make use of spontaneous interaction and evolve to multimodal form of interaction. For example, research reveals that people from all forms of cultural and linguistic backgrounds use both verbal and visual signs when communicating with one another every day. Even two year olds understand multimodal forms of communication. Therefore, multimodality has always existed with human communication. However, according to Jewitt (2008), multimodality’s academic roots can be traced to linguistic theories like the Halliday’s social semiotic theory of communication developed in 1978. The Halliday’s social semiotic theory of communication is a theory that theory that wants people to approach communication differentiation from a social point of view. This means that different modes of communication exist not because of the rules and regulations set in place but because of what the communication accomplishes socially. For example, in a bank, the employees will speak calmly and with minimal hand gestures because they want their audience to remain calm to reduce chaos in the delicate working environment. therefore, even before the term multimodality was coined, other linguistic social semiotic theories had started exploring areas of multimodality. Nevertheless, it is important to examine how multimodality was used by people before the theories that gave forth to the concept of multimodality.

So let’s back to the question I mentioned at first paragraph, there is no ‘multimodality ‘ in the world before 1990s? The answer is obviously  no ! The next part I will tell you something about how did people use multimodal ways to  communicate in different ages.

The  historical development of multimodality

Humans have created and shared texts including visual images, graphic designs and written language for hundreds of years, even if they  did not call this behavior as  multimodality. As the examples Serafini(2014) gave to us that Egyptians used visual images and hieroglyphs to adorn their temples and burial sites. Renaissance scholars, in particular Leonardo da Vinci, used drawings to enhance their written observations and scientific notebooks.

It is evident that multimodality existed log before the term was coined because since man was able to develop word, they had to use gestures, facial features and other physical abilities to understand communication from its most basic form. However, through various advancements including the oral language era, the literacy era, and the printing age, people have continued to understand each other in a better way.

Multimodality during the oral language age

It is very easy to divide human history based on the sensibilities of the media that they had around them. Media is crucial to multimodality because it allows its users to use it along language for more effective communication. Notably, there have been several distinct eras that can be used to understand the progression of multimodality. The oral language era was the first (Boston University, 2017). It is in this era that the use of sound can be found to be in association with light, and simple symbols as well as graphics to pass information. Some of these graphics include smoke and fire as signals in communication between people in long distances. This can be considered as the budding stage of the word. Use of word started developing for more effective communication.

Besides the developing word that was used during this period, other graphics might have been used including gesture and gaze to assist these budding words (Jewitt, et al., 2016). This is because research reveals that gaze and gestures have a long linguistic history and might have been the foundation of image and writing when studying semiotics. It is also during this same age that words grew into language in an attempt to make meaning. The attempt of language at this point began with language attempting to make general meanings because it did not have a base point from other languages or other linguistic styles. However, with these general meanings, gaze and gestures provided more specific explanations. Therefore, multimodality allowed for more specific understanding of commands between human beings who did not have a language or rules of engagement when attempting to communicate. It gave meaning to human interaction. It is at this point that literacy begins to develop.  

Multimodality during the literacy age

The second era was the literary age. In this age, people started developing text through the various rules of language that they had developed over the centuries. The first attempt at text was through pictograms. Pictograms create the foundation of writing where pictures are used to convey meaning (Boston University, 2017). A modern example today can be derived from early education when children are asked to represent for example number 6 using pictures. These children might draw photos of objects that they are familiar with like balls, or even simple circles. However, since in the beginning of the literacy age language was yet to be developed, people used a sequence of standardized pictures to construct language. Evidence of pictograms as a mode of communication can be found in the Neolithic period and later in Mesoamerican writings. The progression of the pictograms gave rise to phonetic alphabet. The early alphabet was developed from pictographs that were pressed on soft clay and left to dry. Unfortunately, it was a skill that was only known by a few because it required a lot of training. However, through trade, this form of writing spread through regions like Egypt, Syria, Israel, and Lebanon. When traders learnt the first 22 symbols, they made the phonetic language more accessible to ordinary people. The final stage was typography which allowed the printing press to be develop. Typography is the art of arranging letters to form not only words but also a copy that is visually appealing to the reader.

Arguably, this era was quite important to people because they could now get an understanding of information because text supplemented words during communication. Serafini (2014) even states that during this period, the artists that were responsible for the visual images and the graphics could add designs to emphasize their work. Additionally, precious metals like silver and gold was also used to encourage enhanced observation. Therefore, people now could not only understand information, but even store it and refer to it as needed. This led to a greater understanding of humanity and their activities making people farm better and even trade better. It led to an immense elevation of how people lived.

Multimodality during the printing age

Notably, the third era was the printing age. Before the printing age, no one new how easy it would be reproduce text, paintings, photographs, and other visual aids at such a cheap price.  even at the begining of the print age, printing pictures on text was time-consuming, labor-intensive, and expensive. So people use wooden boards to carve pictures and print them on paper.However, it was now possible.

The development of typography allowed people to develop paper, ink, and woodblock printing that lead to wide spread printing (Boston University, 2017). The mechanical movable printing technology spread in Europe very fast because of the immense need to spread information through reproducing large quantities of written information in the region. Constant development and modification of the available printing technology finally lead to the creation of the first printing press that was made in Germany. Even though paper was first developed in Asia and specifically China, the demand in the European market made Europe the center location of the printing age. It is through these developments that journalism was born because people were desperate for information. The trade boom in the region as well as the social instability mad people anxious of the future which gave rise to journalism that mass produced newspaper to feed the population. In addition, this period witnessed population growth due to the rapid economic expansion of the industrial revolution. All these people had various interest. To satisfy all these people, the newspapers were divided into categories that met the needs of various audiences.

The result of printing age is that newspaper became one of the oldest medium of mass communication. People on one end of the country could receive the same information with the people at the opposite end of the country without it being distorted. Ideas could be published and used to influence masses of people. Additionally, culture and a desirable lifestyle emerged from the words printed in the newspaper and were soon adopted and replicated n various households within the country. A great example is women moving from only being home caretakers to joining the workforce during the industrial revolution. After mass publishing, more women joined the workforce in the factories which was a great leap for women everywhere to date. Therefore, newspapers as a form of text used in conjunction with language during the typing era led to communication in a much wider scale. An agenda could be more widely spread and influence more widely felt. It also inspired more people to self-publish contributing to a wider pool of knowledge.

Multimodality during the digital age

It is argued that the printing age led to many revolutions that changed the world to what it is that we recognize today. However, one of the most dramatic eras in multimodality is the digital age . It was characterized by people engaging various forms of media like pictures, videos, audios, and text to supplement the spoken language (Boston University, 2017). People started exchanging and sharing information on social networks at a fast and unprecedented way that could not have been anticipated in the printing age. Nowadays, people only need to present visual images, sound effects, video clips and written text in the digital environment through the same basic code (digital bytes), which allows them to be combined in various multimodal configurations relatively simple(Serafini 2014).

Suddenly, people had more power to publish and air their grievances than ever before because now the government and rich conglomerates did not control news distribution agencies like newspapers. People now had more choice on the content they consumed allowing them to have more access to the kind of information that they want. As a result, creating and disseminating information is now easier than ever before. Visual images also become more important and are now part of written and spoken language. A great example is the rampant use of PowerPoint Presentation. Another difference noted is that now we interpret visual images more differently than written word. For example, some pictures are worth a thousand words but a thousand words are not worth a picture (Serafini, 2014).

These changes show how important visual images became after the 1990s. It will be interesting to see the kind of developments that will take place in the future and the kind of supplementary graphics that will take root in communication and how they influence culture. However, for now, visuals are now the most important forms of graphics in multimodality and may even be replacing the importance of spoken language as people continuously consume content like YouTube and Netflix increasingly alone instead of with other people.

This is an introduction to the history of multimodality. If you are still interested in the topic of multimodality, you can find the podcast I did at Maastricht University with my three companions—Leah, Laila and Nina here.