（CVPR 2023）SadTalker：Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

                 

Wenxuan Zhang ^*,1,2   Xiaodong Cun ^*,2  Xuan Wang ³  Yong Zhang ²  Xi Shen ² 
Yu Guo¹   Ying Shan ²   Fei Wang ¹  

¹ Xi'an Jiaotong University   ² Tencent AI Lab   ³ Ant Group  

CVPR 2023

TL;DR:       single portrait image 🙎‍♂️      +       audio 🎤       =       talking head video 🎞.

🔥 Highlight

• 🔥 The extension of the stable-diffusion-webui is online. Just install it in extensions -> install from URL -> https://github.com/Winfredy/SadTalker, checkout more details here.

• 🔥 full image mode is online! checkout here for more details.

still+enhancer in v0.0.1	still + enhancer in v0.0.2	input image @bagbag1815

• 🔥 Several new mode, eg, still mode, reference mode, resize mode are online for better and custom applications.

• 🔥 Happy to see our method is used in various talking or singing avatar, checkout these wonderful demos at bilibili and twitter #sadtalker.

📋 Changelog (Previous changelog can be founded here)

• [2023.04.08]: ❗️❗️❗️ In v0.0.2, we add a logo watermark to the generated video to prevent abusing since it is very realistic.

• [2023.04.08]: v0.0.2, full image animation, adding baidu driver for download checkpoints. Optimizing the logic about enhancer.

🚧 TODO

• training code of each componments.
• Audio-driven Anime Avatar.
• interpolate ChatGPT for a conversation demo 🤔
• integrade with stable-diffusion-web-ui. (stay tunning!)

⚙️ Installation (中文windows教程|日本語コース )

Installing Sadtalker on Linux:

git clone https://github.com/Winfredy/SadTalker.git

cd SadTalker 

conda create -n sadtalker python=3.8

conda activate sadtalker

pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113

conda install ffmpeg

pip install -r requirements.txt

### tts is optional for gradio demo. 
### pip install TTS

More tips about installnation on Windows and the Docker file can be founded here

Sd-Webui-Extension:

CLICK ME

Installing the lastest version of stable-diffusion-webui and install the sadtalker via extension.

Then, restarting the stable-diffusion-webui(The models will be downloaded automatically in the right place if you have a good speed network). Or (Important!) you need pre-download sadtalker checkpoints to SADTALKTER_CHECKPOINTS in webui_user.sh(linux) or webui_user.bat(windows) by:

# windows (webui_user.bat)
set COMMANDLINE_ARGS=--no-gradio-queue  --disable-safe-unpickle
set SADTALKER_CHECKPOINTS=D:\SadTalker\checkpoints

# linux (webui_user.sh)
export COMMANDLINE_ARGS=--no-gradio-queue  --disable-safe-unpickle
export SADTALKER_CHECKPOINTS=/path/to/SadTalker/checkpoints

After installation, the SadTalker can be used in stable-diffusion-webui directly. (Some important discussion if you are unable to use full mode).

Download Trained Models

bash scripts/download_models.sh

🔮 Quick Start (Best Practice)

Animating Portrait Image from default config.

python inference.py --driven_audio <audio.wav> --source_image <video.mp4 or picture.png> --enhancer gfpgan 

The results will be saved in results/$SOME_TIMESTAMP/*.mp4.

More examples and configuration and tips can be founded in the >>> best practice documents <<<.

Full body/image Generation

Using --still to generate a natural full body video. You can add enhancer to improve the quality of the generated video.

python inference.py --driven_audio <audio.wav> \
                    --source_image <video.mp4 or picture.png> \
                    --result_dir <a file to store results> \
                    --still \
                    --preprocess full \
                    --enhancer gfpgan 

Local Gradio demo.

A local gradio demo similar to our hugging-face demo can be run by:

## you need manually install TTS(https://github.com/coqui-ai/TTS) via `pip install tts` in advanced.

python app.py

🛎 Citation

If you find our work useful in your research, please consider citing:

@article{zhang2022sadtalker,
  title={SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation},
  author={Zhang, Wenxuan and Cun, Xiaodong and Wang, Xuan and Zhang, Yong and Shen, Xi and Guo, Yu and Shan, Ying and Wang, Fei},
  journal={arXiv preprint arXiv:2211.12194},
  year={2022}
}

💗 Acknowledgements

Facerender code borrows heavily from zhanglonghao's reproduction of face-vid2vid and PIRender. We thank the authors for sharing their wonderful code. In training process, We also use the model from Deep3DFaceReconstruction and Wav2lip. We thank for their wonderful work.

🥂 Related Works

• StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN (ECCV 2022)
• CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior (CVPR 2023)
• VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild (SIGGRAPH Asia 2022)
• DPE: Disentanglement of Pose and Expression for General Video Portrait Editing (CVPR 2023)
• 3D GAN Inversion with Facial Symmetry Prior (CVPR 2023)
• T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations (CVPR 2023)

📢 Disclaimer

This is not an official product of Tencent. This repository can only be used for personal/research/non-commercial purposes.

LOGO: color and font suggestion: ChatGPT, logo font：Montserrat Alternates .

All the copyright of the demo images and audio are from communities users or the geneartion from stable diffusion. Free free to contact us if you feel uncomfortable.