README.md · FineWeb

1

---

2

license: odc-by

3

task_categories:

4

- text-generation

5

language:

6

- en

7

pretty_name: FineWeb

8

size_categories:

9

- n>1T

10

configs:

11

- config_name: default

12

data_files:

13

- split: train

14

path: data/*/*

15

- config_name: sample-10BT

16

data_files:

17

- split: train

18

path: sample/10BT/*

19

- config_name: sample-100BT

20

data_files:

21

- split: train

22

path: sample/100BT/*

23

- config_name: sample-350BT

24

data_files:

25

- split: train

26

path: sample/350BT/*

27

- config_name: CC-MAIN-2025-05

28

data_files:

29

- split: train

30

path: data/CC-MAIN-2025-05/*

31

- config_name: CC-MAIN-2025-08

32

data_files:

33

- split: train

34

path: data/CC-MAIN-2025-08/*

35

- config_name: CC-MAIN-2025-13

36

data_files:

37

- split: train

38

path: data/CC-MAIN-2025-13/*

39

- config_name: CC-MAIN-2025-18

40

data_files:

41

- split: train

42

path: data/CC-MAIN-2025-18/*

43

- config_name: CC-MAIN-2025-21

44

data_files:

45

- split: train

46

path: data/CC-MAIN-2025-21/*

47

- config_name: CC-MAIN-2025-26

48

data_files:

49

- split: train

50

path: data/CC-MAIN-2025-26/*

51

- config_name: CC-MAIN-2024-51

52

data_files:

53

- split: train

54

path: data/CC-MAIN-2024-51/*

55

- config_name: CC-MAIN-2024-46

56

data_files:

57

- split: train

58

path: data/CC-MAIN-2024-46/*

59

- config_name: CC-MAIN-2024-42

60

data_files:

61

- split: train

62

path: data/CC-MAIN-2024-42/*

63

- config_name: CC-MAIN-2024-38

64

data_files:

65

- split: train

66

path: data/CC-MAIN-2024-38/*

67

- config_name: CC-MAIN-2024-33

68

data_files:

69

- split: train

70

path: data/CC-MAIN-2024-33/*

71

- config_name: CC-MAIN-2024-30

72

data_files:

73

- split: train

74

path: data/CC-MAIN-2024-30/*

75

- config_name: CC-MAIN-2024-26

76

data_files:

77

- split: train

78

path: data/CC-MAIN-2024-26/*

79

- config_name: CC-MAIN-2024-22

80

data_files:

81

- split: train

82

path: data/CC-MAIN-2024-22/*

83

- config_name: CC-MAIN-2024-18

84

data_files:

85

- split: train

86

path: data/CC-MAIN-2024-18/*

87

- config_name: CC-MAIN-2024-10

88

data_files:

89

- split: train

90

path: data/CC-MAIN-2024-10/*

91

- config_name: CC-MAIN-2023-50

92

data_files:

93

- split: train

94

path: data/CC-MAIN-2023-50/*

95

- config_name: CC-MAIN-2023-40

96

data_files:

97

- split: train

98

path: data/CC-MAIN-2023-40/*

99

- config_name: CC-MAIN-2023-23

100

data_files:

101

- split: train

102

path: data/CC-MAIN-2023-23/*

103

- config_name: CC-MAIN-2023-14

104

data_files:

105

- split: train

106

path: data/CC-MAIN-2023-14/*

107

- config_name: CC-MAIN-2023-06

108

data_files:

109

- split: train

110

path: data/CC-MAIN-2023-06/*

111

- config_name: CC-MAIN-2022-49

112

data_files:

113

- split: train

114

path: data/CC-MAIN-2022-49/*

115

- config_name: CC-MAIN-2022-40

116

data_files:

117

- split: train

118

path: data/CC-MAIN-2022-40/*

119

- config_name: CC-MAIN-2022-33

120

data_files:

121

- split: train

122

path: data/CC-MAIN-2022-33/*

123

- config_name: CC-MAIN-2022-27

124

data_files:

125

- split: train

126

path: data/CC-MAIN-2022-27/*

127

- config_name: CC-MAIN-2022-21

128

data_files:

129

- split: train

130

path: data/CC-MAIN-2022-21/*

131

- config_name: CC-MAIN-2022-05

132

data_files:

133

- split: train

134

path: data/CC-MAIN-2022-05/*

135

- config_name: CC-MAIN-2021-49

136

data_files:

137

- split: train

138

path: data/CC-MAIN-2021-49/*

139

- config_name: CC-MAIN-2021-43

140

data_files:

141

- split: train

142

path: data/CC-MAIN-2021-43/*

143

- config_name: CC-MAIN-2021-39

144

data_files:

145

- split: train

146

path: data/CC-MAIN-2021-39/*

147

- config_name: CC-MAIN-2021-31

148

data_files:

149

- split: train

150

path: data/CC-MAIN-2021-31/*

151

- config_name: CC-MAIN-2021-25

152

data_files:

153

- split: train

154

path: data/CC-MAIN-2021-25/*

155

- config_name: CC-MAIN-2021-21

156

data_files:

157

- split: train

158

path: data/CC-MAIN-2021-21/*

159

- config_name: CC-MAIN-2021-17

160

data_files:

161

- split: train

162

path: data/CC-MAIN-2021-17/*

163

- config_name: CC-MAIN-2021-10

164

data_files:

165

- split: train

166

path: data/CC-MAIN-2021-10/*

167

- config_name: CC-MAIN-2021-04

168

data_files:

169

- split: train

170

path: data/CC-MAIN-2021-04/*

171

- config_name: CC-MAIN-2020-50

172

data_files:

173

- split: train

174

path: data/CC-MAIN-2020-50/*

175

- config_name: CC-MAIN-2020-45

176

data_files:

177

- split: train

178

path: data/CC-MAIN-2020-45/*

179

- config_name: CC-MAIN-2020-40

180

data_files:

181

- split: train

182

path: data/CC-MAIN-2020-40/*

183

- config_name: CC-MAIN-2020-34

184

data_files:

185

- split: train

186

path: data/CC-MAIN-2020-34/*

187

- config_name: CC-MAIN-2020-29

188

data_files:

189

- split: train

190

path: data/CC-MAIN-2020-29/*

191

- config_name: CC-MAIN-2020-24

192

data_files:

193

- split: train

194

path: data/CC-MAIN-2020-24/*

195

- config_name: CC-MAIN-2020-16

196

data_files:

197

- split: train

198

path: data/CC-MAIN-2020-16/*

199

- config_name: CC-MAIN-2020-10

200

data_files:

201

- split: train

202

path: data/CC-MAIN-2020-10/*

203

- config_name: CC-MAIN-2020-05

204

data_files:

205

- split: train

206

path: data/CC-MAIN-2020-05/*

207

- config_name: CC-MAIN-2019-51

208

data_files:

209

- split: train

210

path: data/CC-MAIN-2019-51/*

211

- config_name: CC-MAIN-2019-47

212

data_files:

213

- split: train

214

path: data/CC-MAIN-2019-47/*

215

- config_name: CC-MAIN-2019-43

216

data_files:

217

- split: train

218

path: data/CC-MAIN-2019-43/*

219

- config_name: CC-MAIN-2019-39

220

data_files:

221

- split: train

222

path: data/CC-MAIN-2019-39/*

223

- config_name: CC-MAIN-2019-35

224

data_files:

225

- split: train

226

path: data/CC-MAIN-2019-35/*

227

- config_name: CC-MAIN-2019-30

228

data_files:

229

- split: train

230

path: data/CC-MAIN-2019-30/*

231

- config_name: CC-MAIN-2019-26

232

data_files:

233

- split: train

234

path: data/CC-MAIN-2019-26/*

235

- config_name: CC-MAIN-2019-22

236

data_files:

237

- split: train

238

path: data/CC-MAIN-2019-22/*

239

- config_name: CC-MAIN-2019-18

240

data_files:

241

- split: train

242

path: data/CC-MAIN-2019-18/*

243

- config_name: CC-MAIN-2019-13

244

data_files:

245

- split: train

246

path: data/CC-MAIN-2019-13/*

247

- config_name: CC-MAIN-2019-09

248

data_files:

249

- split: train

250

path: data/CC-MAIN-2019-09/*

251

- config_name: CC-MAIN-2019-04

252

data_files:

253

- split: train

254

path: data/CC-MAIN-2019-04/*

255

- config_name: CC-MAIN-2018-51

256

data_files:

257

- split: train

258

path: data/CC-MAIN-2018-51/*

259

- config_name: CC-MAIN-2018-47

260

data_files:

261

- split: train

262

path: data/CC-MAIN-2018-47/*

263

- config_name: CC-MAIN-2018-43

264

data_files:

265

- split: train

266

path: data/CC-MAIN-2018-43/*

267

- config_name: CC-MAIN-2018-39

268

data_files:

269

- split: train

270

path: data/CC-MAIN-2018-39/*

271

- config_name: CC-MAIN-2018-34

272

data_files:

273

- split: train

274

path: data/CC-MAIN-2018-34/*

275

- config_name: CC-MAIN-2018-30

276

data_files:

277

- split: train

278

path: data/CC-MAIN-2018-30/*

279

- config_name: CC-MAIN-2018-26

280

data_files:

281

- split: train

282

path: data/CC-MAIN-2018-26/*

283

- config_name: CC-MAIN-2018-22

284

data_files:

285

- split: train

286

path: data/CC-MAIN-2018-22/*

287

- config_name: CC-MAIN-2018-17

288

data_files:

289

- split: train

290

path: data/CC-MAIN-2018-17/*

291

- config_name: CC-MAIN-2018-13

292

data_files:

293

- split: train

294

path: data/CC-MAIN-2018-13/*

295

- config_name: CC-MAIN-2018-09

296

data_files:

297

- split: train

298

path: data/CC-MAIN-2018-09/*

299

- config_name: CC-MAIN-2018-05

300

data_files:

301

- split: train

302

path: data/CC-MAIN-2018-05/*

303

- config_name: CC-MAIN-2017-51

304

data_files:

305

- split: train

306

path: data/CC-MAIN-2017-51/*

307

- config_name: CC-MAIN-2017-47

308

data_files:

309

- split: train

310

path: data/CC-MAIN-2017-47/*

311

- config_name: CC-MAIN-2017-43

312

data_files:

313

- split: train

314

path: data/CC-MAIN-2017-43/*

315

- config_name: CC-MAIN-2017-39

316

data_files:

317

- split: train

318

path: data/CC-MAIN-2017-39/*

319

- config_name: CC-MAIN-2017-34

320

data_files:

321

- split: train

322

path: data/CC-MAIN-2017-34/*

323

- config_name: CC-MAIN-2017-30

324

data_files:

325

- split: train

326

path: data/CC-MAIN-2017-30/*

327

- config_name: CC-MAIN-2017-26

328

data_files:

329

- split: train

330

path: data/CC-MAIN-2017-26/*

331

- config_name: CC-MAIN-2017-22

332

data_files:

333

- split: train

334

path: data/CC-MAIN-2017-22/*

335

- config_name: CC-MAIN-2017-17

336

data_files:

337

- split: train

338

path: data/CC-MAIN-2017-17/*

339

- config_name: CC-MAIN-2017-13

340

data_files:

341

- split: train

342

path: data/CC-MAIN-2017-13/*

343

- config_name: CC-MAIN-2017-09

344

data_files:

345

- split: train

346

path: data/CC-MAIN-2017-09/*

347

- config_name: CC-MAIN-2017-04

348

data_files:

349

- split: train

350

path: data/CC-MAIN-2017-04/*

351

- config_name: CC-MAIN-2016-50

352

data_files:

353

- split: train

354

path: data/CC-MAIN-2016-50/*

355

- config_name: CC-MAIN-2016-44

356

data_files:

357

- split: train

358

path: data/CC-MAIN-2016-44/*

359

- config_name: CC-MAIN-2016-40

360

data_files:

361

- split: train

362

path: data/CC-MAIN-2016-40/*

363

- config_name: CC-MAIN-2016-36

364

data_files:

365

- split: train

366

path: data/CC-MAIN-2016-36/*

367

- config_name: CC-MAIN-2016-30

368

data_files:

369

- split: train

370

path: data/CC-MAIN-2016-30/*

371

- config_name: CC-MAIN-2016-26

372

data_files:

373

- split: train

374

path: data/CC-MAIN-2016-26/*

375

- config_name: CC-MAIN-2016-22

376

data_files:

377

- split: train

378

path: data/CC-MAIN-2016-22/*

379

- config_name: CC-MAIN-2016-18

380

data_files:

381

- split: train

382

path: data/CC-MAIN-2016-18/*

383

- config_name: CC-MAIN-2016-07

384

data_files:

385

- split: train

386

path: data/CC-MAIN-2016-07/*

387

- config_name: CC-MAIN-2015-48

388

data_files:

389

- split: train

390

path: data/CC-MAIN-2015-48/*

391

- config_name: CC-MAIN-2015-40

392

data_files:

393

- split: train

394

path: data/CC-MAIN-2015-40/*

395

- config_name: CC-MAIN-2015-35

396

data_files:

397

- split: train

398

path: data/CC-MAIN-2015-35/*

399

- config_name: CC-MAIN-2015-32

400

data_files:

401

- split: train

402

path: data/CC-MAIN-2015-32/*

403

- config_name: CC-MAIN-2015-27

404

data_files:

405

- split: train

406

path: data/CC-MAIN-2015-27/*

407

- config_name: CC-MAIN-2015-22

408

data_files:

409

- split: train

410

path: data/CC-MAIN-2015-22/*

411

- config_name: CC-MAIN-2015-18

412

data_files:

413

- split: train

414

path: data/CC-MAIN-2015-18/*

415

- config_name: CC-MAIN-2015-14

416

data_files:

417

- split: train

418

path: data/CC-MAIN-2015-14/*

419

- config_name: CC-MAIN-2015-11

420

data_files:

421

- split: train

422

path: data/CC-MAIN-2015-11/*

423

- config_name: CC-MAIN-2015-06

424

data_files:

425

- split: train

426

path: data/CC-MAIN-2015-06/*

427

- config_name: CC-MAIN-2014-52

428

data_files:

429

- split: train

430

path: data/CC-MAIN-2014-52/*

431

- config_name: CC-MAIN-2014-49

432

data_files:

433

- split: train

434

path: data/CC-MAIN-2014-49/*

435

- config_name: CC-MAIN-2014-42

436

data_files:

437

- split: train

438

path: data/CC-MAIN-2014-42/*

439

- config_name: CC-MAIN-2014-41

440

data_files:

441

- split: train

442

path: data/CC-MAIN-2014-41/*

443

- config_name: CC-MAIN-2014-35

444

data_files:

445

- split: train

446

path: data/CC-MAIN-2014-35/*

447

- config_name: CC-MAIN-2014-23

448

data_files:

449

- split: train

450

path: data/CC-MAIN-2014-23/*

451

- config_name: CC-MAIN-2014-15

452

data_files:

453

- split: train

454

path: data/CC-MAIN-2014-15/*

455

- config_name: CC-MAIN-2014-10

456

data_files:

457

- split: train

458

path: data/CC-MAIN-2014-10/*

459

- config_name: CC-MAIN-2013-48

460

data_files:

461

- split: train

462

path: data/CC-MAIN-2013-48/*

463

- config_name: CC-MAIN-2013-20

464

data_files:

465

- split: train

466

path: data/CC-MAIN-2013-20/*

467

---

468

# 🍷 FineWeb

469

<center>

470

    <img src="https://huggingface.co/datasets/HuggingFaceFW/admin/resolve/main/fineweb-logo.png" alt="FineWeb: The finest collection of data the web has to offer">

471

</center>

472

473

> 15 trillion tokens of the finest data the 🌐 web has to offer

474

475

# Table of Contents

476

- [🍷 FineWeb](#-fineweb)

477

* [What is it?](#what-is-it)

478

* [What is being released?](#what-is-being-released)

479

* [Changelog](#changelog)

480

* [How to download and use 🍷 FineWeb](#how-to-download-and-use-🍷-fineweb)

481

+ [Using 🏭 `datatrove`](#using-datatrove)

482

+ [Using `huggingface_hub`](#using-huggingface_hub)

483

+ [Using `datasets`](#using-datasets)

484

* [Breakdown by dump/crawl](#breakdown-by-dumpcrawl)

485

* [Dataset performance evaluation and ablations](#dataset-performance-evaluation-and-ablations)

486

+ [Hyper-parameters for ablation models](#hyper-parameters-for-ablation-models)

487

+ [Ablation evaluation benchmarks](#ablation-evaluation-benchmarks)

488

+ [Comparison with other datasets](#comparison-with-other-datasets)

489

- [Dataset card for 🍷 FineWeb](#dataset-card-for-🍷-fineweb)

490

* [Dataset Summary](#dataset-summary)

491

* [Dataset Structure](#dataset-structure)

492

+ [Data Instances](#data-instances)

493

+ [Data Fields](#data-fields)

494

+ [Data Splits](#data-splits)

495

* [Dataset Creation](#dataset-creation)

496

+ [Curation Rationale](#curation-rationale)

497

+ [Source Data](#source-data)

498

+ [Data processing steps](#data-processing-steps)

499

+ [Annotations](#annotations)

500

+ [Personal and Sensitive Information](#personal-and-sensitive-information)

501

* [Considerations for Using the Data](#considerations-for-using-the-data)

502

+ [Social Impact of Dataset](#social-impact-of-dataset)

503

+ [Discussion of Biases](#discussion-of-biases)

504

+ [Other Known Limitations](#other-known-limitations)

505

* [Additional Information](#additional-information)

506

+ [Licensing Information](#licensing-information)

507

+ [Future work](#future-work)

508

+ [Citation Information](#citation-information)

509

510

## What is it?

511

512

The 🍷 FineWeb dataset consists of more than **18.5T tokens** (originally 15T tokens) of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 [`datatrove`](https://github.com/huggingface/datatrove/) library, our large scale data processing library.

513

514

🍷 FineWeb was originally meant to be a fully open replication of 🦅 [RefinedWeb](https://huggingface.co/papers/2306.01116), with a release of the **full dataset** under the **ODC-By 1.0 license**. However, by carefully adding additional filtering steps, we managed to push the performance of 🍷 FineWeb well above that of the original 🦅 RefinedWeb, and models trained on our dataset also outperform models trained on other commonly used high quality web datasets (like C4, Dolma-v1.6, The Pile, SlimPajama, RedPajam2) on our aggregate group of [benchmark tasks](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py).

515

516

That said, we think there is still room for additional filtering and improvement and intend to continue exploring how to improve the dataset quality in coming versions of 🍷 FineWeb.

517

518

## What is being released?

519

520

Along with the dataset, which includes all CommonCrawl dumps since 2013, we also share all the code needed to fully reproduce our processing setup using the 🏭 [`datatrove`](https://github.com/huggingface/datatrove/) library [here](https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py). To enable full replication of our results, we have also published the small ablation models we have trained using [`nanotron`](https://github.com/huggingface/nanotron/) to validate the dataset and compare it with other reference datasets. You will find them [here](https://huggingface.co/collections/HuggingFaceFW/ablation-models-662457b0d213e8c14fe47f32), with checkpoints every 1000 steps. We have also published our evaluation results [here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/eval_results.csv). Our evaluation setup is available [here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py).

521

522

You will find details on the different processing decisions we took and some interesting explorations of deduplication methods on our [blogpost](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).

523

524

## Changelog

525

_Previous versions remain available in the branch `version name`._

526

527

- **v1.4.0 (11-07-2025):** Added 6 new snapshots: `CC-MAIN-2025-05`, `CC-MAIN-2025-08`, `CC-MAIN-2025-13`, `CC-MAIN-2025-18`, `CC-MAIN-2025-21`, and `CC-MAIN-2025-26` (January to June 2025)

528

- **v1.3.0 (31-01-2025):** Fixed an issue with some dumps where some documents hadn't been processed: `CC-MAIN-2024-10`, `CC-MAIN-2024-18`, `CC-MAIN-2024-22`, `CC-MAIN-2024-26`, `CC-MAIN-2024-30`, `CC-MAIN-2024-33`, `CC-MAIN-2024-38`, `CC-MAIN-2024-42`, `CC-MAIN-2024-46` -- they now contain more data (~400B additional tokens). We also removed specific domains in response to a [C&D notice](https://huggingface.co/datasets/huggingface-legal/takedown-notices/blob/main/2025/2025-01-22-Torstar.md).

529

- **v1.2.0 (03-01-2025):** Added 8 new snapshots: `CC-MAIN-2024-22`, `CC-MAIN-2024-26`, `CC-MAIN-2024-30`, `CC-MAIN-2024-33`, `CC-MAIN-2024-38`, `CC-MAIN-2024-42`, `CC-MAIN-2024-46`, `CC-MAIN-2024-51`, covering May to December 2024.

530

- **v1.1.0 (31-05-2024):** We reprocessed and reuploaded 11 dumps, `CC-MAIN-2021-49` to `CC-MAIN-2023-40`, as we found a bug on their deduplication. We also added the most recent dump: `CC-MAIN-2024-18`, crawled over April 2024. Expect a small perf improvement

531

- **v1.0.0 (21-04-2024):** Initial version

532

533

## How to download and use 🍷 FineWeb

534

535

You can load the full dataset or a specific crawl/dump (see table below). Dumps have the format `CC-MAIN-(year)-(week number)`.

536

537

### (Smaller) sample versions

538

Along with config `default` (all the data), and the configs for each individual dump, you can also download the following configs:

539

- `sample-350BT`: a subset randomly sampled from the whole dataset of around 350B gpt2 tokens (388GB)

540

- `sample-100BT`: a subset randomly sampled from the whole dataset of around 100B gpt2 tokens (277.4GB)

541

- `sample-10BT`: a subset randomly sampled from the whole dataset of around 10B gpt2 tokens (27.6GB)

542

543

`sample-10B` was sampled from `sample-100B` which in turn was sampled from `sample-350BT`.

544

545

### Using 🏭 [`datatrove`](https://github.com/huggingface/datatrove/)

546

547

```python

548

from datatrove.pipeline.readers import ParquetReader

549

550

# limit determines how many documents will be streamed (remove for all)

551

# to fetch a specific dump: hf://datasets/HuggingFaceFW/fineweb/data/CC-MAIN-2024-10

552

# replace "data" with "sample/100BT" to use the 100BT sample

553

data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb/data", limit=1000)

554

for document in data_reader():

555

# do something with document

556

print(document)

557

558

###############################

559

# OR for a processing pipeline:

560

###############################

561

562

from datatrove.executor import LocalPipelineExecutor

563

from datatrove.pipeline.readers import ParquetReader

564

from datatrove.pipeline.filters import LambdaFilter

565

from datatrove.pipeline.writers import JsonlWriter

566

567

pipeline_exec = LocalPipelineExecutor(

568

pipeline=[

569

# replace "data/CC-MAIN-2024-10" with "sample/100BT" to use the 100BT sample

570

ParquetReader("hf://datasets/HuggingFaceFW/fineweb/data/CC-MAIN-2024-10", limit=1000),

571

LambdaFilter(lambda doc: "hugging" in doc.text),

572

JsonlWriter("some-output-path")

573

],

574

tasks=10

575

)

576

pipeline_exec.run()

577

```

578

579

### Using `huggingface_hub`

580

581

```python

582

from huggingface_hub import snapshot_download

583

folder = snapshot_download(

584

"HuggingFaceFW/fineweb",

585

repo_type="dataset",

586

local_dir="./fineweb/",

587

# replace "data/CC-MAIN-2023-50/*" with "sample/100BT/*" to use the 100BT sample

588

allow_patterns="data/CC-MAIN-2023-50/*")

589

```

590

591

For faster downloads, make sure to install `pip install huggingface_hub[hf_transfer]` and set the environment variable `HF_HUB_ENABLE_HF_TRANSFER=1`.

592

593

### Using `datasets`

594

595

```python

596

from datasets import load_dataset

597

# use name="sample-10BT" to use the 10BT sample

598

fw = load_dataset("HuggingFaceFW/fineweb", name="CC-MAIN-2024-10", split="train", streaming=True)

599

```

600

601

## Breakdown by dump/crawl

602

603

604

| --- | --- |----------------|------------------------|

605

| CC-MAIN-2025-26 | June 2025 | 419.6 | 152.4 |

606

| CC-MAIN-2025-21 | May 2025 | 462.8 | 168.1 |

607

| CC-MAIN-2025-18 | April 2025 | 506.8 | 184.2 |

608

| CC-MAIN-2025-13 | March 2025 | 491.1 | 178.5 |

609

| CC-MAIN-2025-08 | February 2025 | 472.0 | 171.6 |

610

| CC-MAIN-2025-05 | January 2025 | 558.8 | 203.5 |

611

| CC-MAIN-2024-51 | December 2024 | 362.6 | 131.2 |

612

| CC-MAIN-2024-46 | November 2024 | 474.6 | 172.9 |

613

| CC-MAIN-2024-42 | October 2024 | 434.0 | 158.1 |

614

| CC-MAIN-2024-38 | September 2024 | 506.2 | 184.6 |

615

| CC-MAIN-2024-33 | August 2024 | 400.6 | 145.9 |

616

| CC-MAIN-2024-30 | July 2024 | 451.3 | 164.6 |

617

| CC-MAIN-2024-26 | June 2024 | 496.5 | 181.2 |

618

| CC-MAIN-2024-22 | May 2024 | 499.7 | 182.5 |

619

| CC-MAIN-2024-18 | April 2024 | 520.6 | 190.3 |

620

| CC-MAIN-2024-10 | February/March 2024 | 581.3 | 212.6 |

621

| CC-MAIN-2023-50 | November/December 2023 | 650.0 | 239.7 |

622

| CC-MAIN-2023-40 | September/October 2023 | 668.7 | 252.0 |

623

| CC-MAIN-2023-23 | May/June 2023 | 654.4 | 249.2 |

624

| CC-MAIN-2023-14 | March/April 2023 | 621.3 | 236.5 |

625

| CC-MAIN-2023-06 | January/February 2023 | 621.9 | 233.9 |

626

| CC-MAIN-2022-49 | November/December 2022 | 631.2 | 237.5 |

627

| CC-MAIN-2022-40 | September/October 2022 | 606.4 | 228.7 |

628

| CC-MAIN-2022-33 | August 2022 | 434.6 | 163.5 |

629

| CC-MAIN-2022-27 | June/July 2022 | 574.9 | 216.1 |

630

| CC-MAIN-2022-21 | May 2022 | 646.4 | 242.7 |

631

| CC-MAIN-2022-05 | January 2022 | 520.1 | 195.4 |

632

| CC-MAIN-2021-49 | November/December 2021 | 413.7 | 155.5 |

633

| CC-MAIN-2021-43 | October 2021 | 601.5 | 221.0 |

634

| CC-MAIN-2021-43 | October 2021 | 601.5 | 221.0 |

635

| CC-MAIN-2021-39 | September 2021 | 518.9 | 190.6 |

636

| CC-MAIN-2021-31 | July/August 2021 | 593.9 | 217.7 |

637

| CC-MAIN-2021-25 | June 2021 | 424.4 | 155.7 |

638

| CC-MAIN-2021-21 | May 2021 | 455.9 | 167.4 |

639

| CC-MAIN-2021-17 | April 2021 | 556.0 | 204.1 |

640

| CC-MAIN-2021-10 | February/March 2021 | 463.2 | 169.6 |

641

| CC-MAIN-2021-04 | January 2021 | 562.4 | 205.4 |

642

| CC-MAIN-2020-50 | November/December 2020 | 422.8 | 154.3 |

643

| CC-MAIN-2020-45 | October 2020 | 426.9 | 155.8 |

644

| CC-MAIN-2020-40 | September 2020 | 555.5 | 202.4 |

645

| CC-MAIN-2020-34 | August 2020 | 379.6 | 138.7 |

646

| CC-MAIN-2020-29 | July 2020 | 489.6 | 178.7 |

647

| CC-MAIN-2020-24 | May/June 2020 | 398.7 | 145.1 |

648

| CC-MAIN-2020-16 | March/April 2020 | 454.0 | 165.6 |

649

| CC-MAIN-2020-10 | February 2020 | 369.6 | 134.7 |

650

| CC-MAIN-2020-05 | January 2020 | 483.3 | 176.4 |

651

| CC-MAIN-2019-51 | December 2019 | 359.3 | 130.9 |

652

| CC-MAIN-2019-47 | November 2019 | 395.4 | 144.0 |

653

| CC-MAIN-2019-43 | October 2019 | 422.3 | 153.9 |

654

| CC-MAIN-2019-39 | September 2019 | 394.4 | 143.7 |

655

| CC-MAIN-2019-35 | August 2019 | 454.2 | 165.4 |

656

| CC-MAIN-2019-30 | July 2019 | 416.6 | 151.5 |

657

| CC-MAIN-2019-26 | June 2019 | 412.9 | 150.1 |

658

| CC-MAIN-2019-22 | May 2019 | 432.8 | 157.4 |

659

| CC-MAIN-2019-18 | April 2019 | 426.7 | 155.3 |

660

| CC-MAIN-2019-13 | March 2019 | 417.8 | 152.1 |

661

| CC-MAIN-2019-09 | February 2019 | 467.2 | 169.9 |

662

| CC-MAIN-2019-04 | January 2019 | 438.1 | 158.7 |

663

| CC-MAIN-2018-51 | December 2018 | 498.6 | 180.8 |

664

| CC-MAIN-2018-47 | November 2018 | 437.7 | 158.9 |

665

| CC-MAIN-2018-43 | October 2018 | 468.8 | 169.9 |

666

| CC-MAIN-2018-39 | September 2018 | 429.2 | 155.2 |

667

| CC-MAIN-2018-34 | August 2018 | 408.2 | 148.0 |

668

| CC-MAIN-2018-30 | July 2018 | 501.5 | 181.4 |

669

| CC-MAIN-2018-26 | June 2018 | 467.5 | 170.0 |

670

| CC-MAIN-2018-22 | May 2018 | 398.6 | 144.2 |

671

| CC-MAIN-2018-17 | April 2018 | 435.1 | 158.1 |

672

| CC-MAIN-2018-13 | March 2018 | 471.5 | 171.5 |

673

| CC-MAIN-2018-09 | February 2018 | 490.2 | 178.0 |

674

| CC-MAIN-2018-05 | January 2018 | 493.5 | 180.7 |

675

| CC-MAIN-2017-51 | December 2017 | 442.6 | 161.5 |

676

| CC-MAIN-2017-47 | November 2017 | 457.9 | 167.1 |

677

| CC-MAIN-2017-43 | October 2017 | 535.6 | 194.9 |

678

| CC-MAIN-2017-39 | September 2017 | 444.5 | 162.3 |

679

| CC-MAIN-2017-34 | August 2017 | 503.2 | 183.4 |

680

| CC-MAIN-2017-30 | July 2017 | 439.2 | 161.2 |

681

| CC-MAIN-2017-26 | June 2017 | 491.5 | 179.8 |

682

| CC-MAIN-2017-22 | May 2017 | 441.0 | 161.5 |

683

| CC-MAIN-2017-17 | April 2017 | 596.8 | 218.6 |

684

| CC-MAIN-2017-13 | March 2017 | 579.8 | 212.1 |

685

| CC-MAIN-2017-09 | February 2017 | 492.2 | 180.2 |

686

| CC-MAIN-2017-04 | January 2017 | 474.3 | 174.4 |

687

| CC-MAIN-2016-50 | December 2016 | 448.9 | 165.4 |

688

| CC-MAIN-2016-44 | October 2016 | 467.8 | 172.0 |

689

| CC-MAIN-2016-40 | September 2016 | 386.1 | 142.8 |

690

| CC-MAIN-2016-36 | August 2016 | 339.6 | 126.3 |

691

| CC-MAIN-2016-30 | July 2016 | 346.0 | 128.4 |

692

| CC-MAIN-2016-26 | June 2016 | 256.5 | 95.5 |

693

| CC-MAIN-2016-22 | May 2016 | 310.9 | 115.4 |

694

| CC-MAIN-2016-18 | April 2016 | 298.1 | 110.8 |

695

| CC-MAIN-2016-07 | February 2016 | 342.7 | 127.2 |

696

| CC-MAIN-2015-48 | November 2015 | 353.9 | 131.3 |

697

| CC-MAIN-2015-40 | September 2015 | 284.0 | 105.5 |

698

| CC-MAIN-2015-35 | August 2015 | 359.4 | 133.2 |

699

| CC-MAIN-2015-32 | July 2015 | 352.4 | 130.1 |

700

| CC-MAIN-2015-27 | June 2015 | 335.5 | 124.0 |

701

| CC-MAIN-2015-22 | May 2015 | 380.2 | 140.4 |

702

| CC-MAIN-2015-18 | April 2015 | 389.0 | 143.8 |

703

| CC-MAIN-2015-14 | March 2015 | 337.5 | 124.5 |

704

| CC-MAIN-2015-11 | February 2015 | 361.4 | 133.3 |

705

| CC-MAIN-2015-06 | January 2015 | 356.1 | 131.3 |

706

| CC-MAIN-2014-52 | December 2014 | 388.5 | 143.3 |

707

| CC-MAIN-2014-49 | November 2014 | 319.9 | 117.7 |

708

| CC-MAIN-2014-42 | October 2014 | 371.1 | 136.4 |

709

| CC-MAIN-2014-41 | September 2014 | 408.1 | 150.2 |

710

| CC-MAIN-2014-35 | August 2014 | 395.7 | 145.6 |

711

| CC-MAIN-2014-23 | July 2014 | 425.0 | 156.5 |

712

| CC-MAIN-2014-15 | April 2014 | 369.1 | 135.7 |

713

| CC-MAIN-2014-10 | March 2014 | 396.2 | 146.2 |

714

| CC-MAIN-2013-48 | Winter 2013 | 396.8 | 145.9 |

715

| CC-MAIN-2013-20 | Summer 2013 | 393.9 | 144.5 |

716

| Total | | 50,446.9 | 18,527.0 |

717

718

## Dataset performance evaluation and ablations

719

720

We conducted our dataset performance ablations and evaluations by training a series of 1.8B parameters models on 27 billion tokens. To compare 🍷 FineWeb with other datasets, we also trained one of these 1.8B models per target dataset, on 350 billion tokens sampled from it (or the entire dataset when its size was < 350 billion tokens).

721

722

### Hyper-parameters for ablation models

723

724

The detailed configurations for training the 1.8B parameters ablation model can be found here (link will be added soon).

725

726

### Ablation evaluation benchmarks

727

728

To conduct the ablations for each of our dataset filtering choices, we selected a set of benchmarks which we identified as “high-signal” benchmarks. These benchmarks were selected according to the following criteria:

729

730

- small variance between runs trained on different samplings of the same dataset

731

- performance increasing monotically during training (or close)

732

- separation between runs on datasets of known quality (C4, The Pile, RedPajama) higher than the variance between runs with various modeling/data seeds

733

734

We used the following list of benchmark for our ablation runs:

735

736

- commonsense_qa (acc/acc_norm)

737

- hellaswag (acc/acc_norm)

738

- openbookqa (acc/acc_norm)

739

- piqa (acc/acc_norm)

740

- siqa (acc/acc_norm)

741

- winogrande (acc/acc_norm)

742

- arc (acc/acc_norm)

743

- mmlu (acc/acc_norm)

744

745

To compare runs we consider an aggregate score, the average of the scores for these tasks.

746

747

The prompts for all these benchmarks are formatted in order to compute and compare the log-likelihood of the full answers for each multiple choice question. All the implementation details for the benchmarks are available in `lighteval` [here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py).

748

749

### Comparison with other datasets

750

751

We compared 🍷 FineWeb with the following datasets:

752

753

- [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)

754

- [C4](https://huggingface.co/datasets/allenai/c4)

755

- [Dolma v1.6](https://huggingface.co/datasets/allenai/dolma) (the CommonCrawl part)

756

- [The Pile](https://huggingface.co/datasets/EleutherAI/pile)

757

- [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B)

758

- [RedPajama2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2) (deduplicated)

759

760

You will find these models on [this collection](https://huggingface.co/collections/HuggingFaceFW/ablation-models-662457b0d213e8c14fe47f32). We have uploaded checkpoints at every 1000 training steps. You will also find our full [evaluation results here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/eval_results.csv).

761

762

<center>

763

<img src="https://huggingface.co/datasets/HuggingFaceFW/admin/resolve/main/fineweb-ablations.png" alt="ablations">

764

</center>

765

766

_Note:_ The plot is smoothed by averaging 5k steps in a rolling window.

767

768

# Dataset card for 🍷 FineWeb

769

770

## Dataset Description

771

772

- **Homepage and Repository:** [https://huggingface.co/datasets/HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb)

773

- **Point of Contact:** please create a discussion on the Community tab

774

- **License:** Open Data Commons Attribution License (ODC-By) v1.0

775

776

### Dataset Summary

777

778

This dataset was created by processing 96 [CommonCrawl](https://commoncrawl.org/) dumps comprising web data crawled from the summer of 2013 to April of 2024. 🍷 FineWeb includes a variety of domains and topics in English and is primarily intended to be used as a research artifact on public data in the context of pretraining dataset for large language models. The CommonCrawl data was carefully processed, filtered and deduplicated with the 🏭 [`datatrove`](https://github.com/huggingface/datatrove/) library, resulting in the largest publicly available clean LLM pretraining dataset, counting around 15 trillion tokens (gpt2 tokenizer).

779

780

## Dataset Structure

781

782

### Data Instances

783

784

The following is an example sample from the dataset. It is part of the `CC-MAIN-2021-43` and was crawled on `2021-10-15T21:20:12Z`.

785

786

```json

787

{

788

   "text": "This is basically a peanut flavoured cream thickened with egg yolks and then set into a ramekin on top of some jam. Tony, one of the Wedgwood chefs, suggested sprinkling on some toasted crushed peanuts at the end to create extra crunch, which I thought was a great idea. The result is excellent.",

789

"id": "<urn:uuid:e5a3e79a-13d4-4147-a26e-167536fcac5d>",

790

"dump": "CC-MAIN-2021-43",

791

   "url": "<http://allrecipes.co.uk/recipe/24758/peanut-butter-and-jam-creme-brulee.aspx?o_is=SimilarRecipes&o_ln=SimRecipes_Photo_7>",

792

"date": "2021-10-15T21:20:12Z",

793

   "file_path": "s3://commoncrawl/crawl-data/CC-MAIN-2021-43/segments/1634323583083.92/warc/CC-MAIN-20211015192439-20211015222439-00600.warc.gz",

794

"language": "en",

795

"language_score": 0.948729,

796

"token_count": 69

797

}

798

```

799

800

### Data Fields

801

802

- `text` (string): the main text content

803

- `id` (string): original unique identifier for this sample from CommonCrawl

804

- `dump` (string): the CommonCrawl dump this sample was a part of

805

- `url` (string): url to the original page where `text` was present

806

- `date` (string): crawl date (from CommonCrawl)

807

- `file_path` (string): s3 path for the individual CommonCrawl warc file containing this sample

808

- `language` (string): `en` for all the samples in this dataset

809

- `language_score` (float): language prediction score (`0.01.0`) as reported by the [fastText language classifier](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/filters/language_filter.py)

810

- `token_count` (int): number of tokens when applying the `gpt2` tokenizer to this sample

811

812

### Data Splits

813

814

The `default` subset includes the entire dataset. If you would like to only use the data from a particular [CommonCrawl dump](https://commoncrawl.org/overview), you can use the dump name as a subset. You will find the full list of available dumps on the table above.

815

From experiments we have run, not all dumps give the same performance. For relatively small trainings (<550 billion tokens) we recommend using the recent `CC-MAIN-2023-50`, `CC-MAIN-2024-10` and `CC-MAIN-2024-18`.

816

817

## Dataset Creation

818

819

### Curation Rationale

820

821

While multiple open-weights models have regularly been released in recent months, these releases often do not include the model's training data. With 🍷 FineWeb we aim to provide the open source community with a very large clean pretraining dataset that can be used to push the envelope on truly open source models (open source models where data is also released).

822

823

### Source Data

824

825

The source data consists of webpages crawled by the CommonCrawl foundation over the 2013-2024 time period.

826

827

We then extracted the main page text from the html of each webpage, carefully filtered each sample and deduplicated each individual CommonCrawl dump/crawl.

828

829

While we originally intended to deduplicate the dataset as a whole, our ablations showed that training on a sampling of individually deduplicated dumps/crawls outperformed training on a sampling of all the dumps/crawls deduplicated together. You will find more details on our [blogpost](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).

830

831

### Data processing steps

832

833

We used the 🏭 `datatrove` library to process the data.

834

You can find a **working script** that launches the [entire processing pipeline here](https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py).

835

836

The data processing pipeline consists of:

837

838

1. [Url Filtering](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/url_filter.py), removing documents originating from Malicious and NSFW websites, using both block-list as well as subwords detection

839

2. [Trafilatura](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/extractors/trafilatura.py) text extraction on the raw HTML from CommonCrawl’s warc files

840

3. [FastText LanguageFilter](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/language_filter.py), removing any document with `en` language score lower than **0.65**

841

4. Quality filtering

842

    1. [Gopher Repetition /](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/gopher_repetition_filter.py) [Quality](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/gopher_quality_filter.py)

843

    2. [C4 Quality filters](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/c4_quality_filter.py) except `terminal_punct` rule

844

    3. [FineWeb custom filters](https://github.com/huggingface/datatrove/blob/05194d3960741e7d5c0bd0d6dd69d44514622549/src/datatrove/pipeline/filters/fineweb_quality_filter.py), consisting of heuristics for removing list-like documents, documents with repeated lines and documents with likely wrong line formatting.

845

5. [MinHash deduplication](https://github.com/huggingface/datatrove/blob/6daa5e879e06b21e6886b37e2b1be4ae58a658b6/src/datatrove/pipeline/dedup/minhash.py) with each crawl deduplicated individually (5-grams, 14x8 hash functions)

846

6. [PII Formatting](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/formatters/pii.py) to anonymize email and public IP addresses

847

848

### Annotations

849

850

We augment the original samples with the `language`, `language_score` and `token_count` annotations. The language related annotations are automatically generated by our [language filter](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/filters/language_filter.py). `token_count` is generated by [applying the gpt2 tokenizer](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/tokens/counter.py) to the `text` column.

851

852

### Personal and Sensitive Information

853

854

We anonymize email addresses and public IP addresses.

855

856

For emails, we apply a regex pattern and replace any occurrence of an email address with either `email@example.com` or `firstname.lastname@example.org`. For IP addresses, we also employ a regex pattern and then further filter to only anonymize IP addresses [allocated for public networks](https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml). Matched IP addresses are then replaced with one of the following randomly generated IP addresses, which at the time of dataset creation were not responding to ping requests: `22.214.171.124`, `126.96.36.199`, `188.8.131.52`, `184.108.40.206`, `220.127.116.11`, and `18.104.22.168`. We decided against applying regex patterns for phone numbers due to the high false positive rate.

857

858

Despite our efforts, given that 🍷 FineWeb is sourced from the internet at large, it is very likely that some personable identifiable information (PII) will be present. If you find your own PII in 🍷 FineWeb and would like it removed, please fill out our [PII removal form](https://forms.gle/VyNT3ZAUPZjPuWp39).

859

860

## Considerations for Using the Data

861

862

### Social Impact of Dataset

863

864

With the release of this dataset we aim to make model training more accessible to the machine learning community at large.

865

866

While multiple open-weights models with strong performance have been publicly released in the past, more often than not these releases are not accompanied by the corresponding training dataset. This is unfortunate as the dataset specificities and characteristics have been demonstrated to have a very large impact and role in the performances of the models. As the creation of a high quality training dataset is a fundamental requirement to training an LLM capable of excelling at downstream tasks, with 🍷 FineWeb we (a) not only make the dataset creation process more transparent, by sharing our entire processing setup including the codebase used, we also (b) help alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community.

867

868

### Discussion of Biases

869

870

Efforts were made to minimize the amount of NSFW and toxic content present in the dataset by employing filtering on the URL level. However, there are still a significant number of documents present in the final dataset that could be considered toxic or contain harmful content. As 🍷 FineWeb was sourced from the web as a whole, any harmful biases typically present in it may be reproduced on our dataset.

871

872

We deliberately avoided using machine learning filtering methods that define text quality based on the similarity to a “gold” source such as wikipedia or toxicity classifiers as these methods have been known to [disproportionately remove content in specific dialects](https://aclanthology.org/D16-1120/) and [overclassify as toxic text related to specific social identities](https://arxiv.org/pdf/2109.07445.pdf), respectively.

873

874

### Other Known Limitations

875

876

As a consequence of some of the filtering steps applied, it is likely that code content is not prevalent in our dataset. If you are training a model that should also perform code tasks, we recommend you use 🍷 FineWeb with a code dataset, such as [The Stack v2](https://huggingface.co/datasets/bigcode/the-stack-v2). You should also probably consider complementing 🍷 FineWeb with specialized curated sources (such as Wikipedia, for example) as they will likely have better formatting than the wikipedia content included in 🍷 FineWeb (we did not tailor the processing to individual websites).

877

878

## Additional Information

879

880

### Licensing Information

881

882

The dataset is released under the **Open Data Commons Attribution License (ODC-By) v1.0** [license](https://opendatacommons.org/licenses/by/1-0/). The use of this dataset is also subject to [CommonCrawl's Terms of Use](https://commoncrawl.org/terms-of-use).

883

884

### Future work

885

886

We plan to not only continue but also expand our efforts to create open-source high quality training datasets and to improve 🍷 FineWeb itself in future iterations.

887

888

## Citation Information

889

Paper on [arXiv](https://arxiv.org/abs/2406.17557)

890

```

891

@inproceedings{

892

penedo2024the,

893

title={The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale},

894

  author={Guilherme Penedo and Hynek Kydl{\'\i}{\v{c}}ek and Loubna Ben allal and Anton Lozhkov and Margaret Mitchell and Colin Raffel and Leandro Von Werra and Thomas Wolf},

895

booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},

896

year={2024},

897

url={https://openreview.net/forum?id=n6SCkn2QaG}

898

}

899

```