README.md
13.4 KB · 196 lines · markdown Raw
1 ---
2 license: other
3 license_name: qwen-research
4 license_link: https://huggingface.co/Qwen/Qwen-Image-2.1/blob/main/LICENSE
5 library_name: diffusers
6 pipeline_tag: text-to-image
7 base_model: Qwen/Qwen-Image-2.1
8 base_model_relation: finetune
9 tags:
10 - lightx2v
11 - qwen-image
12 - image-generation
13 - image-editing
14 - distillation
15 - reinforcement-learning
16 - diffusers
17 ---
18
19 <div align="center">
20 <a href="https://github.com/ModelTC/LightX2V">
21 <img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/img_lightx2v.png" alt="LightX2V" width="600">
22 </a>
23 </div>
24
25 <p align="center">
26 🤗 <a href="https://huggingface.co/lightx2v">Hugging Face</a>&nbsp;&nbsp;|
27 &nbsp;&nbsp;💻 <a href="https://github.com/ModelTC/LightX2V">GitHub</a>&nbsp;&nbsp;|
28 &nbsp;&nbsp;🌐 <a href="https://x2v.light-ai.top/">LightX2V</a>&nbsp;&nbsp;|
29 &nbsp;&nbsp;💬 <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/wechat.jpg">WeChat</a>
30 </p>
31
32 ## Introduction
33
34 **Four-step image generation and editing. Built with Qwen.**
35
36 This model is the reinforcement-learning-refined version of LightX2V Lightning v2, distilled from [Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1). It supports text-to-image generation, single-reference image editing, and multi-reference image generation with **4 denoising steps** and **CFG = 1.0**.
37
38 RL training was performed with [LightX2V RL](https://github.com/ModelTC/LightX2V/tree/main/lightx2v_rl).
39
40 ## Model files
41
42 The release provides the **full transformer weights** in a single `diffusion_pytorch_model.safetensors` file, together with `config.json`. The weights are stored in FP32 and can be loaded in BF16 for inference.
43
44 `inference_config.json` records the base model, four sampling sigmas, guidance scale, KV cache setting, and scheduler parameters. It is a release-specific configuration file, explicitly read by the example below; Diffusers does not load it automatically.
45
46 The inference files are:
47
48 ```text
49 config.json
50 diffusion_pytorch_model.safetensors
51 inference_config.json
52 README.md
53 assets/img_lightx2v.png
54 assets/wechat.jpg
55 assets/comparison/
56 ```
57
58 `merge_manifest.json` is a local export audit record, is not used for inference, and can be excluded when uploading.
59
60 Load the processor, text encoder, and VAE from the original Qwen-Image-2.1 pipeline, then replace its transformer as shown below. This checkpoint is not a LoRA adapter or a standalone pipeline bundle.
61
62 ## Quick start
63
64 Install a recent Diffusers version with `QwenImage21Pipeline` support:
65
66 ```bash
67 pip install git+https://github.com/huggingface/diffusers.git
68 pip install "transformers>=5.17.0" accelerate safetensors pillow
69 ```
70
71 Run the following code from the downloaded model directory, or change `MODEL_DIR` to its local path. The base pipeline components will be downloaded from Hugging Face when needed.
72
73 ### Load the model
74
75 ```python
76 import json
77 from pathlib import Path
78
79 import torch
80 from PIL import Image
81 from diffusers import (
82 FlowMatchEulerDiscreteScheduler,
83 QwenImage21Pipeline,
84 QwenImage21Transformer2DModel,
85 )
86
87 MODEL_DIR = "."
88 with (Path(MODEL_DIR) / "inference_config.json").open(encoding="utf-8") as f:
89 inference_config = json.load(f)
90 SIGMAS = inference_config["sample_sigmas"]
91
92 transformer = QwenImage21Transformer2DModel.from_pretrained(
93 MODEL_DIR,
94 torch_dtype=torch.bfloat16,
95 )
96 pipe = QwenImage21Pipeline.from_pretrained(
97 inference_config["base_model"],
98 transformer=transformer,
99 torch_dtype=torch.bfloat16,
100 )
101 # The four sigmas are already shifted. Do not apply dynamic shifting again.
102 pipe.scheduler = FlowMatchEulerDiscreteScheduler(
103 **inference_config["scheduler"],
104 )
105 pipe.to("cuda")
106
107
108 def generate(prompt, image=None, seed=42):
109 return pipe(
110 prompt=prompt,
111 image=image,
112 width=1024,
113 height=1024,
114 sigmas=SIGMAS,
115 num_inference_steps=len(SIGMAS),
116 true_cfg_scale=inference_config["true_cfg_scale"],
117 use_kv_cache=inference_config["use_kv_cache"],
118 generator=torch.Generator("cpu").manual_seed(seed),
119 ).images[0]
120 ```
121
122 Use this exact sigma schedule for all three tasks. The terminal sigma of zero is appended by the scheduler. Setting only `num_inference_steps=4` does not reproduce this schedule.
123
124 ### Text-to-image
125
126 ```python
127 image = generate(
128 "A ceramic teapot beside a small bowl of oranges on a wooden table, "
129 "soft morning window light, detailed glaze, natural colors."
130 )
131 image.save("t2i.png")
132 ```
133
134 ### Single-reference editing
135
136 ```python
137 reference = Image.open("reference.png").convert("RGB")
138 image = generate(
139 "Change the jacket to dark green. Preserve the person's face, pose, "
140 "and the rest of the scene.",
141 image=[reference],
142 )
143 image.save("edit.png")
144 ```
145
146 ### Multi-reference generation
147
148 Pass reference images in the same order used by the prompt:
149
150 ```python
151 references = [
152 Image.open("person.png").convert("RGB"),
153 Image.open("jacket.png").convert("RGB"),
154 ]
155 image = generate(
156 "Show the person from image 1 wearing the jacket from image 2. "
157 "Preserve the person's identity and the jacket's design, "
158 "with natural lighting and a simple studio background.",
159 image=references,
160 )
161 image.save("multi_reference.png")
162 ```
163
164 The examples use 1024 × 1024 output. For other aspect ratios, set both dimensions to multiples of 32. Generation quality can vary with the prompt and seed; fine text, anatomy, object counts, and precise reference preservation may still require checking.
165
166 ## Comparison with Qwen-Image-2.1 (40 steps)
167
168 Selected qualitative examples from our validation runs: **Lightning v2 RL uses 4 steps**, while the baseline uses the original [Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1) weights with 40-step Euler sampling. Both use CFG = 1.0, the same full enhanced prompts, the same input references where applicable, and 1024 × 1024 output.
169
170 ### Text-to-image
171
172 | Example / prompt | Qwen-Image-2.1 · 40 steps | Lightning v2 RL · 4 steps |
173 | --- | --- | --- |
174 | [Dolphin at a Burning Shoreline](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/12/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/12/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/t2i/12/base_40step.png" alt="Dolphin at a Burning Shoreline, original model, 40 steps" width="360"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/12/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/t2i/12/rl_4step.png" alt="Dolphin at a Burning Shoreline, RL model, 4 steps" width="360"></a> |
175 | [Weathered Bicycle in the Rain](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/13/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/13/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/t2i/13/base_40step.png" alt="Weathered Bicycle in the Rain, original model, 40 steps" width="360"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/13/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/t2i/13/rl_4step.png" alt="Weathered Bicycle in the Rain, RL model, 4 steps" width="360"></a> |
176
177 ### Single-reference editing
178
179 | Example / prompt | Reference images | Qwen-Image-2.1 · 40 steps | Lightning v2 RL · 4 steps |
180 | --- | --- | --- | --- |
181 | [Flying Dragon above Waterfall Gardens](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/20/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/20/input_1.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/20/input_1.png" alt="Reference 1" width="125"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/20/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/20/base_40step.png" alt="Flying Dragon above Waterfall Gardens, original model, 40 steps" width="300"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/20/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/20/rl_4step.png" alt="Flying Dragon above Waterfall Gardens, RL model, 4 steps" width="300"></a> |
182 | [Guitarist and Cat in a Busy Marketplace](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/22/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/22/input_1.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/22/input_1.png" alt="Reference 1" width="125"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/22/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/22/base_40step.png" alt="Guitarist and Cat in a Busy Marketplace, original model, 40 steps" width="300"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/22/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/22/rl_4step.png" alt="Guitarist and Cat in a Busy Marketplace, RL model, 4 steps" width="300"></a> |
183
184 ### Multi-reference generation
185
186 | Example / prompt | Reference images | Qwen-Image-2.1 · 40 steps | Lightning v2 RL · 4 steps |
187 | --- | --- | --- | --- |
188 | [Food Transfer into a Stylized Scene](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/input_1.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/27/input_1.png" alt="Reference 1" width="125"></a><br><a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/input_2.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/27/input_2.png" alt="Reference 2" width="125"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/27/base_40step.png" alt="Food Transfer into a Stylized Scene, original model, 40 steps" width="300"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/27/rl_4step.png" alt="Food Transfer into a Stylized Scene, RL model, 4 steps" width="300"></a> |
189 | [Floral Porcelain Teapot in a Portrait Scene](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/input_1.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/29/input_1.png" alt="Reference 1" width="125"></a><br><a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/input_2.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/29/input_2.png" alt="Reference 2" width="125"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/29/base_40step.png" alt="Floral Porcelain Teapot in a Portrait Scene, original model, 40 steps" width="300"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/29/rl_4step.png" alt="Floral Porcelain Teapot in a Portrait Scene, RL model, 4 steps" width="300"></a> |
190
191 ## License and acknowledgements
192
193 This model is derived from Qwen-Image-2.1 and is subject to the [Qwen Research License](https://huggingface.co/Qwen/Qwen-Image-2.1/blob/main/LICENSE), including its non-commercial terms.
194
195 Thanks to the Qwen team and the Diffusers community. See [LightX2V](https://github.com/ModelTC/LightX2V), the earlier [Qwen-Image-2.1-Lightning](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning) release, and [Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) for related work.
196