README.md · RynnBrain1.1-2B

1

---

2

language:

3

- en

4

- zh

5

library_name: transformers

6

license: apache-2.0

7

tags:

8

- robotics

9

- embodied-ai

10

- egocentric

11

- spatiotemporal

12

- vision-language-model

13

- video-understanding

14

- grounding

15

- planning

16

- navigation

17

- ocr

18

- image-text-to-text

19

- video-text-to-text

20

- custom_code

21

base_model:

22

- Qwen/Qwen3.5-2B

23

pipeline_tag: image-text-to-text

24

---

25

26

# RynnBrain 1.1

27

28

<img src="https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/assets/images/logo.png?raw=ture" style="width: 60%; height: auto;">

29

30

31

Towards More Capable and Generalizable Embodied Foundation Models

32

33

34

       💫 <a href="https://alibaba-damo-academy.github.io/RynnBrain/"><b>Project Page</b></a> &nbsp;&nbsp;|&nbsp;&nbsp; 🤗 <a href="https://huggingface.co/collections/Alibaba-DAMO-Academy/rynnbrain-11"><b>Hugging Face</b></a> &nbsp;&nbsp;|&nbsp;&nbsp; 🤖 <a href="https://modelscope.cn/collections/DAMO_Academy/RynnBrain-11"><b>ModelScope</b></a> &nbsp;&nbsp;|&nbsp;&nbsp; 📚 <a href="https://github.com/alibaba-damo-academy/RynnBrain/tree/main/cookbooks"><b>Cookbooks</b></a> &nbsp;&nbsp;|&nbsp;&nbsp; 📄 <a href="https://arxiv.org/abs/2602.14979"><b>arXiv</b></a>

35

36

37

38

## News

39

* **[2026.07.16]**  ✨✨ Release **RynnBrain 1.1** checkpoints (**2B / 9B / 122B-A10B**) on <a href="https://huggingface.co/collections/Alibaba-DAMO-Academy/rynnbrain-11">Hugging Face</a> & <a href="https://modelscope.cn/collections/DAMO_Academy/RynnBrain-11">ModelScope</a>!

40

* **[2026.07.16]**  ✨✨ Release the **RynnBrain 1.1** Technical Report on <a href="https://arxiv.org/abs/2602.14979v1">arXiv</a>!

41

42

43

44

45

## Introduction

46

  We present **RynnBrain 1.1**, a systematic upgrade of RynnBrain for embodied intelligence. RynnBrain 1.1 is released in three scales: **2B**, **9B**, and **122B-A10B**, extending the model family from compact dense models to its first 122B-level sparse-MoE model.

47

48

## What's New in 1.1 🚀

49

* **Unified Embodied Scaling to 122B**:

50

Establishes the first embodied brain model at the 122B scale under a unified training recipe shared across 2B, 9B, and 122B-A10B, enabling a systematic study of how embodied cognition, spatial reasoning, grounding, and planning evolve with scale.

51

* **Real-robot VLA transfer**:

52

Bridges perception and action through RynnBrain-VLA, translating embodied understanding into real-robot control and demonstrating strong cross-platform generalization on Unitree G1, Astribot, and Tianji-Wuji across humanoid, bimanual, and dexterous-hand tasks.

53

* **Native 3D and contact point grounding**:

54

Introduces explicit 3D-grounded training and a new contact point prediction task, extending RynnBrain from image-plane localization to metric 3D understanding and action-relevant interaction grounding.

55

56

## Performance

57

58

- General Embodied Understanding

59

60

RynnBrain 1.1-2B vs. other 2B-scale models

61

62

<img src="https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/assets/images/RynnBrain1.1-2B.png?raw=ture" style="width: 80%; height: auto;">

63

64

65

- Real-Robot VLA Evaluation

66

67

Real-robot VLA success rates vs. baselines

68

69

<img src="https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/assets/images/VLA_results.png?raw=ture" style="width: 80%; height: auto;">

70

71

72

- 3D Grounding

73

74

3D grounding accuracy vs. baselines

75

76

<img src="https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/assets/images/3D_chart.png?raw=ture" style="width: 80%; height: auto;">

77

78

79

## Model Zoo

80

81

82

| :--------------- | :------------------- | :---------: | :--------: |

83

| **RynnBrain1.1-2B (This checkpoint)**  | **Qwen3.5-2B** | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnBrain1.1-2B)    | [Link](https://modelscope.cn/models/DAMO_Academy/RynnBrain1.1-2B)   |

84

| RynnBrain1.1-9B  | Qwen3.5-9B | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnBrain1.1-9B)    | [Link](https://modelscope.cn/models/DAMO_Academy/RynnBrain1.1-9B)   |

85

| RynnBrain1.1-122B-A10B  | Qwen3.5-122B-A10B | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnBrain1.1-122B-A10B)    | [Link](https://modelscope.cn/models/DAMO_Academy/RynnBrain1.1-122B-A10B)   |

86

87

88

89

## Quick Start

90

91

### Inference with 🤗 transformers

92

93

**Minimal dependencies**

94

```shell

95

pip install transformers==5.2.0

96

```

97

**Run text generation**

98

```python

99

import torch

100

from transformers import AutoModelForImageTextToText, AutoProcessor

101

102

conversation = [

103

{

104

"role": "user",

105

"content": [

106

{"type": "image", "image": "cookbooks/assets/object_location/images/000000086408.jpg"},

107

{

108

"type": "text",

109

                "text": "What appliance can be used to heat food quickly.\nGenerate coordinates for one object bounding box. Constraints: x1,y1,x2,y2 in [0,1000]. Response must be in the format: <object> (x1, y1), (x2, y2) </object>",

110

},

111

],

112

}

113

]

114

115

model_path = "Alibaba-DAMO-Academy/RynnBrain1.1-2B"

116

processor = AutoProcessor.from_pretrained(model_path)

117

118

model = AutoModelForImageTextToText.from_pretrained(

119

model_path,

120

dtype=torch.bfloat16,

121

)

122

model.to("cuda")

123

124

model_inputs = processor.apply_chat_template(

125

conversation,

126

add_generation_prompt=True,

127

enable_thinking=False,

128

tokenize=True,

129

return_dict=True,

130

return_tensors="pt",

131

)

132

model_inputs = model_inputs.to("cuda")

133

134

output_ids = model.generate(

135

**model_inputs,

136

max_new_tokens=256,

137

do_sample=False,

138

)

139

output_ids = output_ids[:, model_inputs["input_ids"].size(1) :]

140

response = processor.decode(output_ids[0], skip_special_tokens=True)

141

print(response)

142

```

143

144

145

146

### Inference with SGLang

147

148

For installation and advanced usages, please refer to the official [documentation](https://docs.sglang.io).

149

150

**OpenAI-Compatible Serving**

151

```shell

152

# launch server

153

python3 -m sglang.launch_server --model-path Alibaba-DAMO-Academy/RynnBrain1.1-2B --host 0.0.0.0 --port 8000

154

```

155

156

```python

157

# inference using openai api

158

import base64

159

import io

160

161

from openai import OpenAI

162

from PIL import Image

163

164

def pil_to_url(image: Image.Image):

165

image_format = image.format if image.format else 'PNG'

166

buffered = io.BytesIO()

167

image.save(buffered, format=image_format)

168

img_str = base64.b64encode(buffered.getvalue()).decode('utf-8')

169

return f'data:image/{image_format.lower()};base64,{img_str}'

170

171

messages = [

172

{

173

'role': 'user',

174

'content': [

175

            {'type': 'image_url', 'image_url': {'url': pil_to_url(Image.open('cookbooks/assets/object_location/images/000000086408.jpg'))}},

176

            {'type': 'text', 'text': 'What appliance can be used to heat food quickly.\nGenerate coordinates for one object bounding box. Constraints: x1,y1,x2,y2 ∈ [0,1000]. Response must be in the format: <object> (x1, y1), (x2, y2) </object>'},

177

],

178

}

179

]

180

181

client = OpenAI(api_key="", base_url="http://localhost:8000/v1")

182

response = client.chat.completions.create(

183

model="default",

184

messages=messages,

185

stream=False,

186

).choices[0].message.content

187

print(response)

188

```

189

190

**Offline Engine**

191

```python

192

import sglang as sgl

193

from transformers import AutoProcessor

194

195

def main():

196

conversation = [

197

{

198

'role': 'user',

199

'content': [

200

{'type': 'image'},

201

                {'type': 'text', 'text': 'What appliance can be used to heat food quickly.\nGenerate coordinates for one object bounding box. Constraints: x1,y1,x2,y2 ∈ [0,1000]. Response must be in the format: <object> (x1, y1), (x2, y2) </object>'},

202

],

203

}

204

]

205

206

model_path = 'Alibaba-DAMO-Academy/RynnBrain1.1-2B'

207

llm = sgl.Engine(model_path=model_path)

208

processor = AutoProcessor.from_pretrained(model_path)

209

210

prompt = processor.apply_chat_template(

211

conversation,

212

add_generation_prompt=True,

213

tokenize=False,

214

)

215

216

output = llm.generate(

217

prompt=prompt,

218

image_data='cookbooks/assets/object_location/images/000000086408.jpg',

219

sampling_params={"temperature": 0.8, "top_p": 0.95},

220

)

221

print(f"Prompt: {prompt}\nGenerated text: {output['text']}")

222

223

if __name__ == '__main__':

224

main()

225

```

226

227

228

229

## Cookbooks

230

Check out the [cookbooks](https://github.com/alibaba-damo-academy/RynnBrain/tree/main/cookbooks) that showcase RynnBrain's capabilities in cognition, localization, reasoning, and planning.

231

232

233

| Category             | Cookbook name                                                                                   | Description |

234

|----------------------|--------------------------------------------------------------------------------------------------|-------------|

235

| Spatial Understanding | [1_spatial_understanding.ipynb](https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/1_spatial_understanding.ipynb)                     | Shows the model's ability for spatial understanding in the video scene. |

236

| Object Understanding | [2_object_understanding.ipynb](https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/2_object_understanding.ipynb)                       | Shows how the model understands object categories, attributes, and relations and counting ability. |

237

| Object Grounding     | [3_object_grounding.ipynb](https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/3_object_grounding.ipynb)                               | Locates specific objects with bounding boxes in an image or video based on instructions. |

238

| Area Location        | [4_area_location.ipynb](https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/4_area_location.ipynb)                                     | Identifies and marks specified regions by points in an image or video. |

239

| Affordance Location  | [5_affordance_location.ipynb](https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/5_affordance_location.ipynb)                         | Finds areas or objects with specific affordances in an image or video. |

240

| Trajectory Location  | [6_trajectory_location.ipynb](https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/6_trajectory_location.ipynb)                         | Infers and annotates trajectories or motion paths in an image or video. |

241

| 🆕 **Contact Point Prediction** | [7_contact_point_prediction.ipynb](https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/7_contact_point_prediction.ipynb)               | **Predicts an instruction-conditioned contact point and in-plane orientation from an image.** |

242

| 🆕 **3D Grounding**         | [8_3d_grounding.ipynb](https://github.com/alibaba-damo-academy/RynnBrain/blob/main/cookbooks/8_3d_grounding.ipynb)                                       | **Predicts 3D bounding boxes (position, dimensions, orientation) from a single RGB image with camera intrinsics.** |

243

244

245

246

## Training

247

248

**Pretraining & Evaluation**

249

250

Please refer to [RynnScale](https://github.com/alibaba-damo-academy/RynnScale/tree/main/projects/rynn_brain) for details of pretraining and evaluation.

251

252

253

## 📑 Citation

254

255

If you find RynnBrain useful for your research and applications, please cite using this BibTeX:

256

257

```bibtex

258

@article{damo2026rynnbrain,

259

title={RynnBrain: Open Embodied Foundation Models},

260

  author={Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng, Kehan Li, Xin Li, Jiangpin Liu, Yunxuan Mao, Zhikai Wang, Yuqian Yuan, Minghao Zhu, Xiao Lin, Yang Bai, Qian Jiang, Yaxi Zhao, Minghua Zeng, Junlong Gao, Yuming Jiang, Jun Cen, Siteng Huang, Liuyi Wang, Wenqiao Zhang, Chengju Liu, Jianfei Yang, Shijian Lu, Deli Zhao},

261

journal={arXiv preprint arXiv:2602.14979v1},

262

year={2026},

263

url = {https://arxiv.org/abs/2602.14979v1}

264

}

265

266

@article{damo2026rynnbrain11,

267

title={RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model},

268

  author={Kehan Li, Bohan Hou, Minghao Zhu, Tianyi Zhang, Zesen Cheng, Zhikai Wang, Sicong Leng, Xin Li, Xiao Lin, Biying Yao, Minghua Zeng, Jiangpin Liu, Ronghao Dang, Jiayan Guo, Siteng Huang, Haoyu Zhao, Heng Ping, Yaxi Zhao, Kexiang Wang, Tong Lu, Shengke Xue, Jiahao Tang, Yulei Wang, Zejing Wang, Jianwei Gao, Shijian Lu, Chengju Liu, Jianfei Yang, Mingxiu Chen, Deli Zhao},

269

journal={arXiv preprint arXiv:2607.17977},

270

year={2026},

271

url = {https://arxiv.org/abs/2607.17977}

272

}

273

```

274

275