Researchers from Kuaishou Technology, the University of Science and Technology of China, and Fudan University have developed a groundbreaking AI model called LivePortrait that significantly improves the quality and controllability of portrait animation.
The model introduces features such as stitching and retargeting modules, allowing for more realistic animated portraits. This live model was built on Gradio, a part of the Hugging Face family.
Read the full paper here – https://arxiv.org/pdf/2407.03168
LivePortrait can efficiently animate static portrait images with precise control over stitching and facial features. The model, described in a recent paper, aims to make portrait animation more realistic, expressive and computationally efficient. This live portrait model is built on Gradio, a part of the hugging face family.
Read the full paper here – https://arxiv.org/pdf/2407.03168
LivePortrait employs an implicit keypoint-based framework to animate portraits, advancing previous methods significantly. It can seamlessly stitch animated faces back into original images, allowing the animation of full-body portraits and multiple faces within a single image.
Additionally, it provides fine-grained control over eye and lip movements through dedicated retargeting modules. LivePortrait achieves high-quality results while being much faster than prior diffusion-based methods, running at 12.8ms per frame on an RTX 4090 GPU.
The researchers enhanced an existing implicit keypoint model by scaling up training data to 69 million high-quality images, using a mixed image-video training strategy, upgrading network architecture, and introducing new optimisation techniques. They also designed small MLP networks that act as implicit blend shapes to enable precise control over facial features.
In experiments, LivePortrait outperformed both diffusion-based and non-diffusion methods on standard benchmarks for portrait animation quality and motion accuracy. The stitching and retargeting modules allowed the seamless integration of animated faces into original images and fine control over eye and mouth movements.
The developers hope LivePortrait will enable more controllable animations for applications like video conferencing, social media, and entertainment. However, they note potential ethical concerns around deep fake misuse and suggest visual artefacts in current results could aid detection.
While some limitations remain, such as handling large pose variations, the researchers believe LivePortrait represents an important advance in efficient, high-quality portrait animation with enhanced creative control.
A few months back, Stability AI also developed something similar, like Stable Video 3D, which generates 3D videos from single images. However, they did not focus on portrait generation.
