README.md
| 1 | --- |
| 2 | license: other |
| 3 | license_name: qwen-research |
| 4 | license_link: https://huggingface.co/Qwen/Qwen-Image-2.1/blob/main/LICENSE |
| 5 | library_name: diffusers |
| 6 | pipeline_tag: text-to-image |
| 7 | base_model: Qwen/Qwen-Image-2.1 |
| 8 | base_model_relation: finetune |
| 9 | tags: |
| 10 | - lightx2v |
| 11 | - qwen-image |
| 12 | - image-generation |
| 13 | - image-editing |
| 14 | - distillation |
| 15 | - reinforcement-learning |
| 16 | - diffusers |
| 17 | --- |
| 18 | |
| 19 | <div align="center"> |
| 20 | <a href="https://github.com/ModelTC/LightX2V"> |
| 21 | <img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/img_lightx2v.png" alt="LightX2V" width="600"> |
| 22 | </a> |
| 23 | </div> |
| 24 | |
| 25 | <p align="center"> |
| 26 | 🤗 <a href="https://huggingface.co/lightx2v">Hugging Face</a> | |
| 27 | 💻 <a href="https://github.com/ModelTC/LightX2V">GitHub</a> | |
| 28 | 🌐 <a href="https://x2v.light-ai.top/">LightX2V</a> | |
| 29 | 💬 <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/wechat.jpg">WeChat</a> |
| 30 | </p> |
| 31 | |
| 32 | ## Introduction |
| 33 | |
| 34 | **Four-step image generation and editing. Built with Qwen.** |
| 35 | |
| 36 | This model is the reinforcement-learning-refined version of LightX2V Lightning v2, distilled from [Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1). It supports text-to-image generation, single-reference image editing, and multi-reference image generation with **4 denoising steps** and **CFG = 1.0**. |
| 37 | |
| 38 | RL training was performed with [LightX2V RL](https://github.com/ModelTC/LightX2V/tree/main/lightx2v_rl). |
| 39 | |
| 40 | ## Model files |
| 41 | |
| 42 | The release provides the **full transformer weights** in a single `diffusion_pytorch_model.safetensors` file, together with `config.json`. The weights are stored in FP32 and can be loaded in BF16 for inference. |
| 43 | |
| 44 | `inference_config.json` records the base model, four sampling sigmas, guidance scale, KV cache setting, and scheduler parameters. It is a release-specific configuration file, explicitly read by the example below; Diffusers does not load it automatically. |
| 45 | |
| 46 | The inference files are: |
| 47 | |
| 48 | ```text |
| 49 | config.json |
| 50 | diffusion_pytorch_model.safetensors |
| 51 | inference_config.json |
| 52 | README.md |
| 53 | assets/img_lightx2v.png |
| 54 | assets/wechat.jpg |
| 55 | assets/comparison/ |
| 56 | ``` |
| 57 | |
| 58 | `merge_manifest.json` is a local export audit record, is not used for inference, and can be excluded when uploading. |
| 59 | |
| 60 | Load the processor, text encoder, and VAE from the original Qwen-Image-2.1 pipeline, then replace its transformer as shown below. This checkpoint is not a LoRA adapter or a standalone pipeline bundle. |
| 61 | |
| 62 | ## Quick start |
| 63 | |
| 64 | Install a recent Diffusers version with `QwenImage21Pipeline` support: |
| 65 | |
| 66 | ```bash |
| 67 | pip install git+https://github.com/huggingface/diffusers.git |
| 68 | pip install "transformers>=5.17.0" accelerate safetensors pillow |
| 69 | ``` |
| 70 | |
| 71 | Run the following code from the downloaded model directory, or change `MODEL_DIR` to its local path. The base pipeline components will be downloaded from Hugging Face when needed. |
| 72 | |
| 73 | ### Load the model |
| 74 | |
| 75 | ```python |
| 76 | import json |
| 77 | from pathlib import Path |
| 78 | |
| 79 | import torch |
| 80 | from PIL import Image |
| 81 | from diffusers import ( |
| 82 | FlowMatchEulerDiscreteScheduler, |
| 83 | QwenImage21Pipeline, |
| 84 | QwenImage21Transformer2DModel, |
| 85 | ) |
| 86 | |
| 87 | MODEL_DIR = "." |
| 88 | with (Path(MODEL_DIR) / "inference_config.json").open(encoding="utf-8") as f: |
| 89 | inference_config = json.load(f) |
| 90 | SIGMAS = inference_config["sample_sigmas"] |
| 91 | |
| 92 | transformer = QwenImage21Transformer2DModel.from_pretrained( |
| 93 | MODEL_DIR, |
| 94 | torch_dtype=torch.bfloat16, |
| 95 | ) |
| 96 | pipe = QwenImage21Pipeline.from_pretrained( |
| 97 | inference_config["base_model"], |
| 98 | transformer=transformer, |
| 99 | torch_dtype=torch.bfloat16, |
| 100 | ) |
| 101 | # The four sigmas are already shifted. Do not apply dynamic shifting again. |
| 102 | pipe.scheduler = FlowMatchEulerDiscreteScheduler( |
| 103 | **inference_config["scheduler"], |
| 104 | ) |
| 105 | pipe.to("cuda") |
| 106 | |
| 107 | |
| 108 | def generate(prompt, image=None, seed=42): |
| 109 | return pipe( |
| 110 | prompt=prompt, |
| 111 | image=image, |
| 112 | width=1024, |
| 113 | height=1024, |
| 114 | sigmas=SIGMAS, |
| 115 | num_inference_steps=len(SIGMAS), |
| 116 | true_cfg_scale=inference_config["true_cfg_scale"], |
| 117 | use_kv_cache=inference_config["use_kv_cache"], |
| 118 | generator=torch.Generator("cpu").manual_seed(seed), |
| 119 | ).images[0] |
| 120 | ``` |
| 121 | |
| 122 | Use this exact sigma schedule for all three tasks. The terminal sigma of zero is appended by the scheduler. Setting only `num_inference_steps=4` does not reproduce this schedule. |
| 123 | |
| 124 | ### Text-to-image |
| 125 | |
| 126 | ```python |
| 127 | image = generate( |
| 128 | "A ceramic teapot beside a small bowl of oranges on a wooden table, " |
| 129 | "soft morning window light, detailed glaze, natural colors." |
| 130 | ) |
| 131 | image.save("t2i.png") |
| 132 | ``` |
| 133 | |
| 134 | ### Single-reference editing |
| 135 | |
| 136 | ```python |
| 137 | reference = Image.open("reference.png").convert("RGB") |
| 138 | image = generate( |
| 139 | "Change the jacket to dark green. Preserve the person's face, pose, " |
| 140 | "and the rest of the scene.", |
| 141 | image=[reference], |
| 142 | ) |
| 143 | image.save("edit.png") |
| 144 | ``` |
| 145 | |
| 146 | ### Multi-reference generation |
| 147 | |
| 148 | Pass reference images in the same order used by the prompt: |
| 149 | |
| 150 | ```python |
| 151 | references = [ |
| 152 | Image.open("person.png").convert("RGB"), |
| 153 | Image.open("jacket.png").convert("RGB"), |
| 154 | ] |
| 155 | image = generate( |
| 156 | "Show the person from image 1 wearing the jacket from image 2. " |
| 157 | "Preserve the person's identity and the jacket's design, " |
| 158 | "with natural lighting and a simple studio background.", |
| 159 | image=references, |
| 160 | ) |
| 161 | image.save("multi_reference.png") |
| 162 | ``` |
| 163 | |
| 164 | The examples use 1024 × 1024 output. For other aspect ratios, set both dimensions to multiples of 32. Generation quality can vary with the prompt and seed; fine text, anatomy, object counts, and precise reference preservation may still require checking. |
| 165 | |
| 166 | ## Comparison with Qwen-Image-2.1 (40 steps) |
| 167 | |
| 168 | Selected qualitative examples from our validation runs: **Lightning v2 RL uses 4 steps**, while the baseline uses the original [Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1) weights with 40-step Euler sampling. Both use CFG = 1.0, the same full enhanced prompts, the same input references where applicable, and 1024 × 1024 output. |
| 169 | |
| 170 | ### Text-to-image |
| 171 | |
| 172 | | Example / prompt | Qwen-Image-2.1 · 40 steps | Lightning v2 RL · 4 steps | |
| 173 | | --- | --- | --- | |
| 174 | | [Dolphin at a Burning Shoreline](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/12/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/12/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/t2i/12/base_40step.png" alt="Dolphin at a Burning Shoreline, original model, 40 steps" width="360"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/12/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/t2i/12/rl_4step.png" alt="Dolphin at a Burning Shoreline, RL model, 4 steps" width="360"></a> | |
| 175 | | [Weathered Bicycle in the Rain](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/13/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/13/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/t2i/13/base_40step.png" alt="Weathered Bicycle in the Rain, original model, 40 steps" width="360"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/t2i/13/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/t2i/13/rl_4step.png" alt="Weathered Bicycle in the Rain, RL model, 4 steps" width="360"></a> | |
| 176 | |
| 177 | ### Single-reference editing |
| 178 | |
| 179 | | Example / prompt | Reference images | Qwen-Image-2.1 · 40 steps | Lightning v2 RL · 4 steps | |
| 180 | | --- | --- | --- | --- | |
| 181 | | [Flying Dragon above Waterfall Gardens](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/20/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/20/input_1.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/20/input_1.png" alt="Reference 1" width="125"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/20/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/20/base_40step.png" alt="Flying Dragon above Waterfall Gardens, original model, 40 steps" width="300"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/20/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/20/rl_4step.png" alt="Flying Dragon above Waterfall Gardens, RL model, 4 steps" width="300"></a> | |
| 182 | | [Guitarist and Cat in a Busy Marketplace](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/22/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/22/input_1.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/22/input_1.png" alt="Reference 1" width="125"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/22/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/22/base_40step.png" alt="Guitarist and Cat in a Busy Marketplace, original model, 40 steps" width="300"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_single/22/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_single/22/rl_4step.png" alt="Guitarist and Cat in a Busy Marketplace, RL model, 4 steps" width="300"></a> | |
| 183 | |
| 184 | ### Multi-reference generation |
| 185 | |
| 186 | | Example / prompt | Reference images | Qwen-Image-2.1 · 40 steps | Lightning v2 RL · 4 steps | |
| 187 | | --- | --- | --- | --- | |
| 188 | | [Food Transfer into a Stylized Scene](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/input_1.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/27/input_1.png" alt="Reference 1" width="125"></a><br><a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/input_2.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/27/input_2.png" alt="Reference 2" width="125"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/27/base_40step.png" alt="Food Transfer into a Stylized Scene, original model, 40 steps" width="300"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/27/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/27/rl_4step.png" alt="Food Transfer into a Stylized Scene, RL model, 4 steps" width="300"></a> | |
| 189 | | [Floral Porcelain Teapot in a Portrait Scene](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/prompt.txt) | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/input_1.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/29/input_1.png" alt="Reference 1" width="125"></a><br><a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/input_2.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/29/input_2.png" alt="Reference 2" width="125"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/base_40step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/29/base_40step.png" alt="Floral Porcelain Teapot in a Portrait Scene, original model, 40 steps" width="300"></a> | <a href="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/blob/main/assets/comparison/i2i_multi/29/rl_4step.png"><img src="https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning-v2-RL/resolve/main/assets/comparison/i2i_multi/29/rl_4step.png" alt="Floral Porcelain Teapot in a Portrait Scene, RL model, 4 steps" width="300"></a> | |
| 190 | |
| 191 | ## License and acknowledgements |
| 192 | |
| 193 | This model is derived from Qwen-Image-2.1 and is subject to the [Qwen Research License](https://huggingface.co/Qwen/Qwen-Image-2.1/blob/main/LICENSE), including its non-commercial terms. |
| 194 | |
| 195 | Thanks to the Qwen team and the Diffusers community. See [LightX2V](https://github.com/ModelTC/LightX2V), the earlier [Qwen-Image-2.1-Lightning](https://huggingface.co/lightx2v/Qwen-Image-2.1-Lightning) release, and [Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) for related work. |
| 196 | |