README.md
6.4 KB · 153 lines · markdown Raw
1 ---
2 license: openrail++
3 tags:
4 - text-to-image
5 - stable-diffusion
6 library_name: diffusers
7 inference: false
8 ---
9
10 # SDXL-Lightning
11
12 ![Intro Image](sdxl_lightning_samples.jpg)
13
14 SDXL-Lightning is a lightning-fast text-to-image generation model. It can generate high-quality 1024px images in a few steps. For more information, please refer to our research paper: [SDXL-Lightning: Progressive Adversarial Diffusion Distillation](https://arxiv.org/abs/2402.13929). We open-source the model as part of the research.
15
16 Our models are distilled from [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0). This repository contains checkpoints for 1-step, 2-step, 4-step, and 8-step distilled models. The generation quality of our 2-step, 4-step, and 8-step model is amazing. Our 1-step model is more experimental.
17
18 We provide both full UNet and LoRA checkpoints. The full UNet models have the best quality while the LoRA models can be applied to other base models.
19
20 ## Demos
21
22 * Generate with all configurations, best quality: [Demo](https://huggingface.co/spaces/ByteDance/SDXL-Lightning)
23
24 ## Checkpoints
25
26 * `sdxl_lightning_Nstep.safetensors`: All-in-one checkpoint, for ComfyUI.
27 * `sdxl_lightning_Nstep_unet.safetensors`: UNet checkpoint only, for Diffusers.
28 * `sdxl_lightning_Nstep_lora.safetensors`: LoRA checkpoint, for Diffusers and ComfyUI.
29
30 ## Diffusers Usage
31
32 Please always use the correct checkpoint for the corresponding inference steps.
33
34 ### 2-Step, 4-Step, 8-Step UNet
35
36 ```python
37 import torch
38 from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel, EulerDiscreteScheduler
39 from huggingface_hub import hf_hub_download
40 from safetensors.torch import load_file
41
42 base = "stabilityai/stable-diffusion-xl-base-1.0"
43 repo = "ByteDance/SDXL-Lightning"
44 ckpt = "sdxl_lightning_4step_unet.safetensors" # Use the correct ckpt for your step setting!
45
46 # Load model.
47 unet = UNet2DConditionModel.from_config(base, subfolder="unet").to("cuda", torch.float16)
48 unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="cuda"))
49 pipe = StableDiffusionXLPipeline.from_pretrained(base, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda")
50
51 # Ensure sampler uses "trailing" timesteps.
52 pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")
53
54 # Ensure using the same inference steps as the loaded model and CFG set to 0.
55 pipe("A girl smiling", num_inference_steps=4, guidance_scale=0).images[0].save("output.png")
56 ```
57
58 ### 2-Step, 4-Step, 8-Step LoRA
59
60 Use LoRA only if you are using non-SDXL base models. Otherwise use our UNet checkpoint for better quality.
61 ```python
62 import torch
63 from diffusers import StableDiffusionXLPipeline, EulerDiscreteScheduler
64 from huggingface_hub import hf_hub_download
65
66 base = "stabilityai/stable-diffusion-xl-base-1.0"
67 repo = "ByteDance/SDXL-Lightning"
68 ckpt = "sdxl_lightning_4step_lora.safetensors" # Use the correct ckpt for your step setting!
69
70 # Load model.
71 pipe = StableDiffusionXLPipeline.from_pretrained(base, torch_dtype=torch.float16, variant="fp16").to("cuda")
72 pipe.load_lora_weights(hf_hub_download(repo, ckpt))
73 pipe.fuse_lora()
74
75 # Ensure sampler uses "trailing" timesteps.
76 pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")
77
78 # Ensure using the same inference steps as the loaded model and CFG set to 0.
79 pipe("A girl smiling", num_inference_steps=4, guidance_scale=0).images[0].save("output.png")
80 ```
81
82 ### 1-Step UNet
83 The 1-step model is only experimental and the quality is much less stable. Consider using the 2-step model for much better quality.
84
85 The 1-step model uses "sample" prediction instead of "epsilon" prediction! The scheduler needs to be configured correctly.
86
87 ```python
88 import torch
89 from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel, EulerDiscreteScheduler
90 from huggingface_hub import hf_hub_download
91 from safetensors.torch import load_file
92
93 base = "stabilityai/stable-diffusion-xl-base-1.0"
94 repo = "ByteDance/SDXL-Lightning"
95 ckpt = "sdxl_lightning_1step_unet_x0.safetensors" # Use the correct ckpt for your step setting!
96
97 # Load model.
98 unet = UNet2DConditionModel.from_config(base, subfolder="unet").to("cuda", torch.float16)
99 unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="cuda"))
100 pipe = StableDiffusionXLPipeline.from_pretrained(base, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda")
101
102 # Ensure sampler uses "trailing" timesteps and "sample" prediction type.
103 pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing", prediction_type="sample")
104
105 # Ensure using the same inference steps as the loaded model and CFG set to 0.
106 pipe("A girl smiling", num_inference_steps=1, guidance_scale=0).images[0].save("output.png")
107 ```
108
109
110 ## ComfyUI Usage
111
112 Please always use the correct checkpoint for the corresponding inference steps.
113 Please use Euler sampler with sgm_uniform scheduler.
114
115 ### 2-Step, 4-Step, 8-Step Full
116
117 1. Download the full checkpoint (`sdxl_lightning_Nstep.safetensors`) to `/ComfyUI/models/checkpoints`.
118 1. Download our [ComfyUI full workflow](comfyui/sdxl_lightning_workflow_full.json).
119
120 ![SDXL-Lightning ComfyUI Full Workflow](comfyui/sdxl_lightning_workflow_full.jpg)
121
122 ### 2-Step, 4-Step, 8-Step LoRA
123
124 Use LoRA only if you are using non-SDXL base models. Otherwise use our full checkpoint for better quality.
125
126 1. Prepare your own base model.
127 1. Download the LoRA checkpoint (`sdxl_lightning_Nstep_lora.safetensors`) to `/ComfyUI/models/loras`
128 1. Download our [ComfyUI LoRA workflow](comfyui/sdxl_lightning_workflow_lora.json).
129
130 ![SDXL-Lightning ComfyUI LoRA Workflow](comfyui/sdxl_lightning_workflow_lora.jpg)
131
132 ### 1-Step
133
134 The 1-step model is only experimental and the quality is much less stable. Consider using the 2-step model for much better quality.
135
136 1. Update your ComfyUI to the latest version.
137 1. Download the full checkpoint (`sdxl_lightning_1step_x0.safetensors`) to `/ComfyUI/models/checkpoints`.
138 1. Download our [ComfyUI full 1-step workflow](comfyui/sdxl_lightning_workflow_full_1step.json).
139
140 ![SDXL-Lightning ComfyUI Full 1-Step Workflow](comfyui/sdxl_lightning_workflow_full_1step.jpg)
141
142
143 ## Cite Our Work
144 ```
145 @misc{lin2024sdxllightning,
146 title={SDXL-Lightning: Progressive Adversarial Diffusion Distillation},
147 author={Shanchuan Lin and Anran Wang and Xiao Yang},
148 year={2024},
149 eprint={2402.13929},
150 archivePrefix={arXiv},
151 primaryClass={cs.CV}
152 }
153 ```