The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

Overview

FineWeb is a 15-trillion token pretraining dataset derived from 96 Common Crawl snapshots. It is designed to produce better-performing large language models than other open datasets. The dataset and its curation methodology are fully documented and ablated to advance understanding of high-quality data curation.

Best for

Researchers and engineers building or benchmarking open LLMs with high-quality pretraining data.

Use cases

• Pretraining large language models from scratch
• Ablation studies on data curation techniques
• Benchmarking open-source dataset quality for LLM training

Notes

Pros

• Proven to improve LLM performance over other open datasets
• Fully documented and ablated curation process
• Large scale with 15 trillion tokens from diverse web sources

Cons

• Requires significant compute resources to process and use
• Derived only from Common Crawl, limiting domain coverage
• Not a ready-to-use tool; requires integration into training pipelines

Pairs with

Other entries in the index that connect to this one.