The McGurk effect: when we hear with our eyes

Let's see what the McGurk effect is, a curious phenomenon of human perception.

Visual and auditory information are key to understanding speech. When we talk to someone, we don't just hear what they say, we also look at how they say it by moving their lips.

The ability of human speech is based on the integration of visual and auditory information, which is evidenced by the fact that we can experience a curious illusory phenomenon: the McGurk effect.

It can be said that this peculiar phenomenon occurs when we hear with our eyes, causing what we hear to change according to what we see. Let's discover what this interesting visuo-auditory effect consists of.

What is the McGurk effect?

We tend to believe that our senses work independently: when we hear, we only hear; and when we see, we only see.. Based on this belief, it would be reasonable to think that a visual stimulus cannot distort the way we perceive sound. However, the reality is that it can, since our perceptual experiences are the product of a complex process of mixing information, the same mixture that gives rise to a peculiar phenomenon: the McGurk effect.

Surely on more than one occasion you have had a conversation in an extremely noisy environment. It may have been in a nightclub, on the terrace of a bar on a busy street or in a school classroom. When there is loud background noise, we find it difficult to understand what the person in front of us is saying and, in order to understand something, we use the old and instinctive trick of watching their mouth as they speak.

In these cases, visual and auditory information are not analyzed separately, but are combined.. The human brain has a region called superior temporal sulcus, which is specialized in combining both types of information, in the examples we have given, it would be in charge of combining the phonemes that our interlocutor pronounces with the movement of his lips.

Because of this ability to combine multimodal information, the superior temporal sulcus is the neurological scenario where the illusory McGurk effect occurswhich would be nothing more than the result of an error in the decoding of the message when two different sensory modalities interact, making what we see not match what we hear.

If we do a quick search on Youtube, we can find more than one video where this phenomenon is exposed in a practical way. This link directs us to a good example of this phenomenon:

In this particular case, the person in the video pronounces /ba/ all the time, however, depending on how he/she moves his/her lips, you can hear either /ba/ or /pa/..

This effect can also occur with other syllable combinations. For example, it can be achieved with the combination /ka/ (visual) and /pa/ (auditory), which gives rise to the perception of /ta/. Another example would be to see someone making lip movements that correspond to the syllable /ga/ but while emitting the syllable /ba/ it will be perceived as /da/.

The way we hear the same sound varies drastically depending on whether we look at the way the person speaking to us moves their lips. This not only affects the perception of simple sets of sounds such as syllables, but has also been found to work with complete sentences, although you have probably witnessed this yourself in some of the situations mentioned above.

One of the first findings related to the McGurk effect and the interaction between sensory modalities is that being able to see how the lips move and how they interact with each other. the possibility of seeing how our interlocutor's lips move considerably improves the volume of what we hear..

It has been found that we have the sensation of hearing phonemes up to 15 decibels louder when we have the speaker within our field of vision. This is true even when the acoustic conditions are not adverse, such as being in a soundless room or in a quiet place.

History of its discovery

This phenomenon was first described in 1976 in an article by British cognitive psychologist Harry McGurk with his colleague John MacDonald entitled "Hearing lips and seeing voices". Their original study was to investigate the imitation patterns of a group of children who were developing speech ability, and the experiment involved presenting them with several videos of people pronouncing different syllables.

However, an error occurred in the playback. The technician in charge of editing the video made a mistake and caused the image and sound to be out of sync, resulting in a recording of a person pronouncing something that did not correspond to the sound heard.

When the video was played back, McGurk and MacDonald heard a third phoneme instead of the one that was articulated with the lips and the one that was emitted. It was serendipity that led these two researchers to discover this peculiar auditory illusion.

Its importance in the study of human speech

The discovery of this effect is considered evidence that the visual and auditory systems have evolved together to allow, among other things, better speech processing.. Our visual system helps us to discriminate sounds that are difficult to differentiate, an advantage that is realized by deaf people when lip-reading.

Being able to see how our interlocutor moves his or her lips increases confidence in the message perceived through the auditory system. That is, if two independent systems point to the same solution, in this case the same message, we have more confidence in that message than if we only receive it through one channel.

It is worth mentioning that The McGurk effect does not occur automatically.. For it to occur, we need to pay attention to our interlocutor and, when distracting stimuli, both visual and auditory, are incorporated, this illusion is attenuated.

In fact, this proves that the effect is not due to a poor reception of visual or auditory information, but to an error in the integration of these two sensory modalities.

Another fact that gives strength to the idea that the visual system supports the auditory system is that, when we see a person talking to us but cannot hear what he or she is saying at all, not only our visual cortex is activated, but also our auditory cortex, even though we are not hearing anything.even though we are not hearing anything.

The McGurk effect and brain dysfunctions.

It has been shown that brain lesions and dysfunctions in reading abilitiesThe McGurk effect, in addition to manifesting mental disorders, influences the likelihood of the McGurk effect occurring.

People who have had a callosotomy manifest the McGurk effect more slowly. It appears that children with specific language disorder have a weaker McGurk effect compared to children who do not have language acquisition disorders or literacy difficulties.

Laterality also plays a roleIt is seen that right-handed people are more prone to experience this effect.

The McGurk effect in different languages

Regardless of which language is spoken, its speakers depend, to some degree, on visual information during speech perception. However, it has been found that the intensity of the McGurk it has been shown that the intensity of the McGurk effect varies from language to language.It was found that in languages such as Spanish, Turkish, Italian, Dutch, English and German speakers experience this effect more strongly than speakers of Chinese and Japanese.

That speakers of Asian languages exhibit the McGurk effect less frequently may be due to the cultural practice of avoiding eye contact.. In addition, Chinese and Japanese in particular are two languages with very syllabic linguistic structures, usually of the consonant+vowel and consonant+vowel+consonant types, which makes them particularly skilled at detecting syllables regardless of how their interlocutor moves his or her lips.