T2I-Adapter

Official implementation of T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models.

Paper

We propose T2I-Adapter, a simple and small (~70M parameters, ~300M storage space) network that can provide extra guidance to pre-trained text-to-image models while freezing the original large text-to-image models.

T2I-Adapter aligns internal knowledge in T2I models with external control signals. We can train various adapters according to different conditions, and achieve rich control and editing effects.

⏬ Download Models

Put the downloaded models in the T2I-Adapter/models folder.

1. The T2I-Adapters can be download from https://huggingface.co/TencentARC/T2I-Adapter.
2. The pretrained Stable Diffusion v1.4 models can be download from https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/tree/main. You need to download the sd-v1-4.ckpt file.
3. [Optional] If you want to use Anything v4.0 models, you can download the pretrained models from https://huggingface.co/andite/anything-v4.0/tree/main. You need to download the anything-v4.0-pruned.ckpt file.
4. The pretrained clip-vit-large-patch14 folder can be download from https://huggingface.co/openai/clip-vit-large-patch14/tree/main. Remember to download the whole folder!
5. The pretrained keypose detection models include FasterRCNN (human detection) from https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth and HRNet (pose detection) from https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth.

After downloading, the folder structure should be like this:

🔧 Dependencies and Installation

• Python >= 3.6 (Recommend to use Anaconda or Miniconda)
• PyTorch >= 1.4

pip install -r requirements.txt

• If you want to use the full function of keypose-guided generation, you need to install MMPose. For details please refer to https://github.com/open-mmlab/mmpose.

💻 How to Test

• The results are in the experiments folder.
• If you want to use the Anything v4.0, please add --ckpt models/anything-v4.0-pruned.ckpt in the following commands.

Sketch Adapter

• Sketch to Image Generation

python test_sketch.py --plms --auto_resume --prompt "A car with flying wings" --path_cond examples/sketch/car.png --ckpt models/sd-v1-4.ckpt --type_in sketch

• Image to Image Generation

python test_sketch.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/anything_sketch/human.png --ckpt models/sd-v1-4.ckpt --type_in image

• Generation with Anything setting

python test_sketch.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/anything_sketch/human.png --ckpt models/anything-v4.0-pruned.ckpt --type_in image

Gradio Demo

python gradio_sketch.py

Keypose Adapter

• Keypose to Image Generation

python test_keypose.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/keypose/iron.png --type_in pose

• Image to Image Generation

python test_keypose.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/sketch/human.png --type_in image

• Generation with Anything setting

python test_keypose.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/sketch/human.png --ckpt models/anything-v4.0-pruned.ckpt --type_in image

Segmentation Adapter

python test_seg.py --plms --auto_resume --prompt "A black Honda motorcycle parked in front of a garage" --path_cond examples/seg/motor.png

Two adapters: Segmentation and Sketch Adapters

python test_seg_sketch.py --plms --auto_resume --prompt "An all white kitchen with an electric stovetop" --path_cond examples/seg_sketch/mask.png --path_cond2 examples/seg_sketch/edge.png

Local editing with adapters

python test_sketch_edit.py --plms --auto_resume --prompt "A white cat" --path_cond examples/edit_cat/edge_2.png --path_x0 examples/edit_cat/im.png --path_mask examples/edit_cat/mask.png

Stable Diffusion + T2I-Adapters (only ~70M parameters, ~300M storage space)

The following is the detailed structure of a Stable Diffusion model with the T2I-Adapter.

🚀 Interesting Applications

Stable Diffusion results guided with the sketch T2I-Adapter

The corresponding edge maps are predicted by PiDiNet. The sketch T2I-Adapter can well generalize to other similar sketch types, for example, sketches from the Internet and user scribbles.

Stable Diffusion results guided with the keypose T2I-Adapter

The keypose results predicted by the MMPose. With the keypose guidance, the keypose T2I-Adapter can also help to generate animals with the same keypose, for example, pandas and tigers.

T2I-Adapter with Anything-v4.0

Once the T2I-Adapter is trained, it can act as a plug-and-play module and can be seamlessly integrated into the finetuned diffusion models without re-training, for example, Anything-4.0.

✨ Anything results with the plug-and-play sketch T2I-Adapter (no extra training)

Anything results with the plug-and-play keypose T2I-Adapter (no extra training)

Local editing with the sketch adapter

When combined with the inpaiting mode of Stable Diffusion, we can realize local editing with user specific guidance.

✨ Change the head direction of the cat

✨ Add rabbit ears on the head of the Iron Man.

Combine different concepts with adapter

Adapter can be used to enhance the SD ability to combine different concepts.

✨ A car with flying wings. / A doll in the shape of letter ‘A’.

Sequential editing with the sketch adapter

We can realize the sequential editing with the adapter guidance.

Composable Guidance with multiple adapters

Stable Diffusion results guided with the segmentation and sketch adapters together.

Logo materials: adapter, lightbulb