How Powerful Is the Combination of Sound and Visuals? The Modality Principle for Optimal Learning Video

What Is the Modality Principle?

Amid the surge of learning videos in the corporate world, more and more companies are realizing that how material is presented shouldn't stop at visual aesthetics alone. How the human brain actually processes information needs to be factored in too. One important principle that helps us understand this is the Modality Principle, as described by Richard E. Mayer in his book Multimedia Learning (2009).

Put simply, the Modality Principle states that people learn better from images paired with narration than from images paired with written text. Why? Because humans have two main channels for processing information: a visual channel (for images and text we see) and an auditory channel (for sounds or words we hear).

When a learning video displays images and written text at the same time, both elements compete for the same visual channel. As a result, the brain has to work harder and struggles to absorb information optimally.

By contrast, if that text is converted into spoken narration, the cognitive load gets split: images are processed through the visual channel, while sound is processed through the auditory channel. This process is called modality off-loading. The result is deeper understanding and stronger recall for learners.

Mayer's research shows that in 17 out of 17 experiments, participants who learned through a combination of animation and narration achieved far better comprehension and problem-solving results than those who learned through a combination of animation and written text. The effect was even substantial, with an effect size of d = 1.02 — a highly significant figure in learning research.

This finding shows that when we create training videos containing charts, data, or animation, adding long blocks of on-screen text can actually reduce learning effectiveness. Complex material with limited time to spare is best delivered through narration, helping employees process information more efficiently.

How Do You Apply It?

At Monkey Melody, we often come across a common mistake when converting training material into video: long explanatory text simply gets pasted onto the screen alongside images or animation. This approach actually causes learners to lose focus and struggle to process the main message.

To avoid this, our team applies the Modality Principle by carefully separating the function of visuals and audio. We analyze each part of the material to determine what can be represented visually, and what's better delivered through narration. Visuals are designed to convey core concepts, while detailed explanations come through a presenter's voice or a professional voice-over talent.

For example, in a Nutrifood learning video about distribution development, we paired a lot of visual assets with a presenter's voice. But in certain sections dealing with key points, we used text alone, with no accompanying image. At times, we added simple icons to complement the text in appropriate portions. All of these considerations are, of course, planned out from the storyboard stage.

In some PLN videos, we went even further and used only a voice-over talent's voice, with no on-screen presenter at all. That's how important verbal information is in rounding out a learning video. This approach means viewers don't have to read long blocks of on-screen text, and can instead focus on understanding the relationships between concepts through the combination of image and sound.

Of course, the Modality Principle isn't something that must always be applied. It works best for material that's complex and fast-paced, where learners don't have enough time to read and look at visuals at the same time. But for slower-paced material, or where participants can control their own learning pace (interactive e-learning, for instance), written text can still be used without creating excessive load.

There are also certain conditions where on-screen text is actually needed — namely when the material contains technical terms or symbols that are hard to pronounce, and when participants are non-native speakers or have a hearing impairment. The key is matching the presentation format to the context and characteristics of the audience.

Having worked on more than 1,000 learning multimedia projects, Monkey Melody understands that every company faces different challenges and goals. That's why we offer a tailored approach to ensure every video we produce achieves its intended training objective. With the latest technology and a creative approach, we're ready to help your company navigate the training challenges ahead.