For years, foundation models in robotics have primarily used vision-language pretraining as the stepping stone towards scaling robotics, allowing us to transfer¹ the benefits of semantic generalization from existing large multimodal models. But what's been missing is how to effectively scale large multimodal model training in the domain of robotics itself—to establish scaling laws that corroborate the consistent (and predictable) improvement of robot intelligence with more compute & data, as has underpinned progress in other domains e.g. LLMs.² This requires an architecture, training procedure, and data engine that pushes new sensorimotor capabilities, provides behavioral generalization, and grows with the vast and ever-expanding experience generated by interacting with the real physical world.

To this end, we’re introducing GEN-0, a new class of embodied foundation models built for multimodal training directly on high-fidelity raw physical interaction. Its architecture builds on the strengths of vision and language models while also going beyond them—natively designed to capture human-level reflexes and physical commonsense. One core feature is Harmonic Reasoning, in which the models are trained to simultaneously think and act seamlessly. We’ve shared a glimpse of the capabilities of early precursors in our prior videos,³ and today we are sharing that not only does GEN-0 have breakthrough fundamental capabilities, but these capabilities are scaling:

• Surpassing the Intelligence Threshold – in an unprecedented high-data regime for robotics, we observe a phase transition at 7B where smaller models exhibit ossification,⁴ while larger ones continue to improve. We’ve since scaled GEN-0 to 10B+ model sizes, and observe fast adaptation to new tasks with increasingly less post-training.
• Scaling Laws – GEN-0 models exhibit strong scaling laws, in which more pretraining data and compute consistently (and predictably) improve downstream post-training performance of the model across many tasks.
• Harmonic Reasoning - Although for language chatbots it is straightforward to spend more time thinking before responding,⁵ the same is not as simple for physical systems acting in the real world – physics doesn't stop. To address this problem, Harmonic Reasoning involves a fundamentally new approach to training models, and creates a "harmonic" interplay between asynchronous, continuous-time streams of sensing and acting tokens. This allows us to scale to very large model sizes without depending on System1-System2 architectures⁶ or inference-time guidance.⁷
• Cross-Embodiment – GEN-0 architecture works on different robots by design. We have tested our models on 6DoF, 7DoF, and 16+DoF semi-humanoid robots.
• No Longer Limited By Data – GEN-0 is pretrained on our in-house robotics dataset, which includes over 270,000 hours of real-world diverse manipulation data, growing at a rate of 10,000 hours a week and accelerating.
• The Science of Pretraining – different mixtures of pretraining data (from various sources e.g. data foundries) yield GEN-0 models with different characteristics. We share some early notes from our empirical observations in this high-data regime, and how that traces back to specific data collection operations.

We believe that GEN-0 marks the beginning of a new era: embodied foundation models whose capabilities predictably scale with physical interaction data – not just from text, images, or simulation – but the real world. Here are videos of GEN-0 in action on new tasks:

Surpassing the Intelligence Threshold

Our scaling experiments show that GEN-0 models must be large enough to absorb vast amounts of physical interaction data. We observe that smaller models exhibit a phenomenon similar to ossification⁴ under data overload, while larger ones continue to improve—demonstrating a surprising “phase transition” in the intelligence capacity of our models:

• 1B models struggle to absorb complex and diverse sensorimotor data during pretraining – model weights become unable to absorb new information over time.
• 6B models begin to benefit from pretraining and show strong multi-task capabilities.
• 7B+ models are able to internalize large-scale robotic pretraining data that transfers to downstream tasks with only a few thousand steps of post-training.

To our knowledge, this is the first time that model ossification⁸ has been observed in robotics. This might have eluded past research due to (a) the lack of a high data regime in robotics until now, and (b) large enough model sizes in this regime. Ossification has previously been observed in LLM literature^4,9 in the high data regime but with much smaller models, on the order of O(10M) parameters rather than O(1B). The observation that this phase transition occurs in robotics but with much larger model sizes echoes Moravec’s Paradox:¹⁰ what humans find effortless—perception and dexterity—demands far more computational complexity than abstract reasoning. Our experiments suggest that intelligence in the physical world (i.e. physical commonsense) may have a higher activation threshold in terms of compute, and we’re only beginning to explore what lies beyond.

Scaling Laws for Robotics

Scaling laws are commonly measured during pretraining, as shown in Figure 1, which shows the relationship of model size and compute on a downstream zero-shot task during pretraining. Another type of scaling law relates to the benefits of pretraining that persist into finetuning.⁴ At sufficient model scale, we also observe a strong power-law relationship (Figure 3) between pretraining data scale and downstream post-training performance. This applies to all of our tasks we've measured, including partner and customer-inspired applications and their workflows across a wide range of industrial sectors – including apparel, manufacturing, logistics, automotive, and electronics.

More specifically, we take a variety of model checkpoints (Figure 2) that have been pretrained using our training procedure on different subsets of our pretraining dataset, and then post-train these checkpoints on multi-task language-conditioned data i.e. supervised fine-tuning simultaneously on 16 different task sets. We find that more pretraining improves downstream model performance across all tasks (Figure 2).




Model performance is predictable with a power-law relationship (Figure 3), with which we can answer questions like “how much pretraining data do we need to reach a specific next-action prediction error?” or “how much post-training data (for a specific task) can we buy with more pretraining data?” Given a fixed data and finetuning budget on a downstream task, and given a pretraining dataset of varying size \(D\), the validation error \(L(\cdot)\) on the downstream task can be predicted via a power-law of the form: $$ L(D) = (D_c / D)^{\alpha_{D}} \ . $$ For example, in the case of Clothes Handling (which involves sorting, unscrambling, buttoning, and hanging clothes in a real workplace), we can predict model performance given 1 billion action trajectories. These estimates guide conversations on partner-related tasks and can provide estimates on how much more data is needed to reach specific levels of performance.

Robotics is No Longer Limited By Data

Our Foundation models are trained on an unprecedented corpus of 270,000 hours of real-world manipulation trajectories collected across diverse activities in 1,000s of homes, warehouses, and workplaces worldwide. Today, our robot data operations provide over 10,000 new hours per week and are accelerating. This is all powered by a global network of hardware and 1,000s of data collection devices and robots.

Mapping the Universe of Manipulation

To scale GEN-0 capabilities, we are constructing the largest and most diverse real-world manipulation dataset ever built, including every manipulation task humans can think of – from peeling potatoes, to threading bolts – spanning homes, bakeries, laundromats, warehouses, factories, and more. Here is an example internal search tool we have built to explore this universe:

Infrastructure for Internet-Scale Robot Data

Building the operations and ML infrastructure to support this is no easy feat. For robot models and data at this scale, we built custom hardware, dataloaders, and network infrastructure (including laying new dedicated Internet lines) to support the uplink bandwidth from a diverse set of data collection sites all around the world. We’ve negotiated multi-cloud contracts, built custom upload machines, scaled to O(10K) cores for continual multimodal data processing, compressed dozens of Petabytes of data, using dataloading techniques behind frontier video foundation models, capable of absorbing 6.85 years of real-world manipulation experience per day of training.

Science of Pretraining

From large-scale ablations, we find that data quality and diversity matters more than sheer volume, and that carefully constructed data mixtures can lead to different pretrained model characteristics. For example, Table 1 shows the performance metrics of different models trained on 8 different pretraining datasets, and their downstream impact when finetuned on 10 long-horizon task sets, organized into 3 groups that evaluate different dimensions: dexterity, real-world applications, and generalization.

Performance is measured in terms of validation prediction M.S.E. \(\text{MSE}_{\text{val}} = ||\mathbf{a}^{\star} - \hat{\mathbf{a}}||_2^2 \) and reverse Kullback–Leibler divergence¹¹ (reverse KL), which better measures mode-seeking behavior.^12,13 To estimate reverse KL, we use a Monte-Carlo estimator where the policy induces an empirical density \(q =\frac{1}{M}\sum_{m=1}^{M}\mathcal{N}\left(\mathbf{a}; \hat{\mathbf{a}}_{m},\mathbf{I}\right)\) via a unit-variance mixture of Gaussians centered at \(M\) policy samples \(\{\hat{\mathbf{a}}_m\}_{m=1}^{M}\), and the data/ground-truth induces a unit-variance Gaussian \(p(\mathbf{a})=\mathcal{N}\left(\mathbf{a}; \mathbf{a}^\star,\mathbf{I}\right) \) centered at \(\mathbf{a}^\star\). We approximate the expectation with policy samples: $$ \widehat{D}_{\mathrm{KL}}(q||p) \approx \frac{1}{M}\sum_{m=1}^{M}\Big[\log q(\hat{\mathbf{a}}_{m})-\log p(\hat{\mathbf{a}}_{m})\Big] \ . $$ Experiments show that models with both low prediction errors and low reverse KL tend to perform better with supervised finetuning (SFT) for postraining, while models with high prediction errors and low reverse KL tend to be more distributionally multimodal, which can help post-training reinforcement learning. Having multiple data collection strategies at scale allows us to continually A/B test which data improves pretraining the most.



More on these learnings in future posts.

Citation

Please cite this work as Or use the BibTeX citation: