In plain words
Give the AI a photo or short video of a face, plus an audio track. It produces a new video where the face says those words, with lips, jaw and expressions moving to match. That matching is called lip sync.
Companies use it to make training videos without filming each update, to dub a lecture into Hindi or Tamil so the speaker's mouth matches the new language, or to give a customer-help video a presenter. Some avatars are invented faces. Others are digital copies of a real person, made with their permission.
The same technology can make a real person appear to say things they never said. When that is done without consent to mislead, it is a deepfake. That is why honest uses rest on two rules: the person agrees, and viewers are told it is AI.
Everyday example
A dubbing artist for a film
When a Telugu film is dubbed into Hindi, a dubbing artist records new lines and tries to match the actor's lip movements. Viewers know it is dubbed. An AI lip-sync tool goes the other way: it keeps the voice and reshapes the actor's lips on screen to fit the new words.
Where the example breaks: Dubbing is open and agreed, and everyone knows it happened. An AI avatar can be made of someone who never agreed, and it can look real enough that viewers cannot tell. The technology does not check consent; people and rules have to.
How it actually works
Lip sync is a narrow, well-studied problem: given audio, predict how the mouth region should look in each frame.
Read the audio
The speech is turned into a sequence of short sound features, many per second. Different sounds need different mouth shapes: lips closed for "m" and "b", rounded for "oo", wide for "ee".
Generate the mouth, frame by frame
A neural network takes the video frames and the matching slices of audio, and redraws the mouth and jaw in each frame to fit the sound. The rest of the face and the background are kept.
Train with a lip-sync judge
Wav2Lip, published in 2020 by researchers at IIIT Hyderabad and the University of Bath, trained its generator with feedback from a separate expert network that only judges whether lips and sound are in sync. This made it work on faces it had never seen, in ordinary videos.
From one photo to a whole talking head
Newer research animates the whole head from a single still photo and a voice clip. Microsoft's VASA-1 (April 2024) made 512 by 512 pixel video at up to 40 frames a second. Microsoft said it had no plans to release it as a product or API until it was confident it would be used responsibly.
- Lip sync for any face
- Wav2Lip, August 2020, by researchers at IIIT Hyderabad and the University of Bath
- Talking head from one photo
- VASA-1, April 2024, Microsoft Research; not released to the public
- EU rule, from 2 August 2026
- The AI Act requires those who use AI to publish a deepfake, meaning content resembling real people, places or events that could falsely appear authentic, to disclose it is artificially generated or manipulated
- India rule, from 20 February 2026
- Amended IT Rules require AI-generated content on platforms to be prominently labelled
Voice is usually made separately, with text-to-speech or a voice clone. Both halves raise the same question of consent.
Questions people ask
Is it legal to make an AI avatar of someone?
Making one of yourself is generally allowed, within the service's terms. Of someone else, you need their clear permission, and laws vary by country. Using a person's face or voice to deceive, defame or defraud is illegal in many places, including under India's IT Act. Labels are increasingly required too.
What is the difference between an AI avatar and a deepfake?
The technology overlaps. An AI avatar is usually made openly, with the person's consent, and shown as AI, such as a company's virtual presenter. A deepfake usually means a real person is made to appear to say or do something without consent, often to mislead.
How can you spot an AI lip-sync video?
Watch the mouth: do lip movements match the sound, especially "p", "b" and "m"? MIT's Detect Fakes project also lists skin that looks too smooth and odd blinking among the cues. Good fakes can pass these checks, so checking the source and looking for an AI label matter most.
What should you do if someone makes an AI video of you?
Save the links and screenshots, report it to the platform, and in India report it at cybercrime.gov.in or call the 1930 helpline. India's amended IT Rules also shortened the time platforms have to act on some takedown orders. Our deepfake page has more steps.
Further reading
- A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild (Prajwal et al.)arXiv, 2020
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (Xu et al.)arXiv, 2024
- Article 50: Transparency Obligations for Providers and Deployers of Certain AI SystemsEU Artificial Intelligence Act, 2024
- Govt tightens rules on AI-generated content, mandates labels and faster takedownsBusiness Today, 2026
- Microsoft teases easy video deepfake tool, declines to release itBiometric Update, 2024