Generalist Team

GEN-1.5 Embodied Foundation Models are One-Shot Learners

Humans have a remarkable ability to perform new physical skills from only one or a few examples. Our latest robot foundation model, GEN-1.5, exhibits the beginnings of that same ability: it can learn a new task in seconds, from a single example, without gradient updates or fine-tuning. It displays broad capabilities across one-shot and few-shot learning from demonstration, as well as zero-shot physical generalization. Although the tasks are simple and short-horizon, this is the first model we know for which one-shot and few-shot learning of physical skills have emerged at scale. We view these results as a significant step towards our mission of building general intelligence for the physical world.

The promise of a robot foundation model is simple to state: walk up to a robot, and get it to do any task almost immediately. Whether zero-shot, one-shot, or few-shot, what matters from a capability standpoint is immediacy and generality: can the model learn a new task quickly and generalize to new situations? For physical tasks, this level of intelligence demands both broad abilities in comprehending task intent, as well as adapting in real time, closed-loop, to the unexpected variation of the real world.

For language models, the ability to quickly learn new tasks from just one or a few examples arrived as a hallmark capability of GPT-3.¹ Across a broad suite of language tasks, it achieved roughly 45% average accuracy with one-shot in-context prompting without training, and up to ~65% with few-shot (~100 examples).¹ Models prior to GPT-3 had shown flashes of zero-shot ability and even initial few-shot results², but GPT-3 achieved significantly broader few-shot performance, paired with what at the time was an immense step in generalization capabilities.

In robotics, the analogous pursuit of systems that could generalize a task from one or a few demonstrations, has persisted for decades — tracing back at least to the teach-by-guiding of the 1954 Unimate patent³ and MIT’s 1970 Copy Demo.⁴ A large number of prior works, including our own,⁵ have shown various forms of in-context learning but over a limited set of task variations, or under restrictions to particular objects, task types, or sensing modalities.^6,7,8,9,10 The ability to learn closed-loop physical skills from just one or a few demonstrations, and to do so across a broad range of tasks without such restrictions, has predominantly been considered out of reach. Such an ability may also likely be underpinned by a foundation that enables other broad generalization capabilities.

Introducing GEN-1.5

We’ve created GEN-1.5, our latest robot foundation model that exhibits broad one-shot and few-shot learning from demonstration capabilities, as well as zero-shot generalization, e.g. improvisation and novel tool use (e.g. brush, dustpan, etc.). GEN-1.5 is a large multimodal model that processes video input (30 seconds of memory, alongside other sensor, language, and proprioceptive inputs) and produces 100 Hz action trajectories. Its capabilities include:

• One-shot learning via in-context prompting. The model learns new tasks in seconds when prompted with 3 to 12 seconds of a single demonstration, no training required. We refer to the use of sensorimotor examples in the context window as “physical prompting.”
  • Compositional generalization. Given two different physical prompts in context, the model chains them into a single longer-horizon behavior.
  • Zero-shot sim-to-real transfer. A demonstration recorded in simulation works as a physical prompt for a real-world task, even though pretraining contains no simulation data.
  • Human-to-robot imitation. In some cases a person can demonstrate a task with their own hands, in view of the robot’s cameras, and the model reproduces it with the robot’s hands.
• Few-shot adaptation via gradient descent. The model can be fine-tuned to a new task in 1–10 gradient steps on 1–5 minutes of data (~10–50 demonstrations).
• Improvising new strategies and tool use. The model generalizes at the level of behavioral strategies: forming entirely new trajectories to reach a goal, using unseen tools (e.g. brush, dustpan, etc.) to create new solutions to tasks demonstrated with other tools, and working ambidextrously even when prompted or fine-tuned to perform the task with a specific hand.

These capabilities appear to emerge directly from pretraining on large amounts of physical interaction data. We did not explicitly train for any of them: no architectural changes to promote in-context learning, no inner or outer meta-learning loop¹¹ pressuring the model to adapt from minimal data, no auxiliary objectives¹² encouraging improvisation. To our surprise, GEN-1.5 does this across a broad range of physical tasks out of the box.

Experiments across 10 diverse tasks show 59% (±10% std. dev.) average success with one-shot in-context prompting, straight from the pretrained model. With few-shot learning, performance rises to 83% (±9% std. dev.) via 10 gradient steps on 5 minutes of data per task (~50 demonstrations). In some cases, in-context learning a new task exceeds the performance of 1–5 gradient steps on the same demonstration data. Although the tasks are simple and short-horizon, and the success rates are modest, this is the first model we know of that has demonstrated the general ability to learn a wide range of dexterous closed-loop physical tasks from just one-shot or few-shot demonstrations.

Scaling Pretraining for Robotics

Over the past two years, we’ve been focused on building a pretraining engine for scaling embodied foundation models trained from the ground up on physical experience, alongside algorithmic improvements that have compounded the rate of progress. As we announced nine months ago, leading up to GEN-0¹³ we started to see predictable scaling laws.¹⁴ Five months later, we announced GEN-1,¹⁵ which demonstrated the ability to be post-trained to task mastery at 99%+ success rates and showed initial signs of improvisational intelligence.

GEN-1.5’s initial pretraining began in parallel — it has now been training continuously for over eight months. We left it running because every metric we tracked kept improving with the engine: absorbing more data, scaling more efficiently with compute, and achieving step-change gains with successive surgical architectural and algorithmic changes. It was clear that the model was getting better, and the trend was consistent: new tasks were becoming more data-efficient, more compute-efficient, and more general.

As the model continued to train, we began experimenting with how few finetuning steps we could use to adapt to new tasks, finding the model could learn new tasks from 100s, then 10s, then eventually, 1 gradient step on just one minute of data. As far as we know, the ability to learn skills with such few gradient steps had not been observed before. We then asked, can this model learn new tasks without training, purely in-context and with zero gradient steps? That this works at all changes how we think about how these models can be used, about their potential impact, and the road ahead for building general physical intelligence.

One-Shot Learning In-Context

GEN-1.5 can be prompted with a single demonstration inserted into its 30-second context window, and the remainder holds rolling observations. Physical prompts are sensorimotor examples (i.e. sensor data plus action trajectories), recorded either as human data (with a pair of handheld grippers) or as rollouts from the robot itself. Once the prompt is in context, the model performs the task immediately, with no training steps. The performance of one-shot learning in-context is modest (59% average success across diverse tasks including handling zippers, opening jars, grabbing money out of wallets, etc.), but the fact that inserting a single demonstration in the context buffer, without ever training for it, yields any measurable competence at all was unexpected. This drastically accelerates reaching a base level of performance that can be subsequently refined towards mastery.¹⁵ Skills learned in-context are currently more brittle than finetuned models, but can generalize to some perturbations, improvise, and recover from mistakes.

We did not explicitly train GEN-1.5 for in-context learning, and the tasks we tested were not engineered into the pretraining data beforehand. This is a general model which we are prompting without regard to the pretraining data distribution.

Why this capability emerges from pretraining is difficult to pinpoint. One hypothesis, by analogy to language, is that the distribution of physical observations and actions may exhibit “burstiness” and Zipfian structure of the kind that has been linked to in-context learning in language models.¹⁶ It is also possible that physical work contains naturally repetitive cycles, and the model may have learned to detect and extend such patterns, as language models do with general sequences.¹⁷ The model was pretrained on randomly sampled continuous spans from our data engine (activities captured in homes, warehouses, factories, and elsewhere) with no bespoke infrastructure for packing examples into context — physical prompts introduce discontinuous jumps in time that the model never saw in training.

Robotics is inherently multimodal; and as in human learning, there are many ways to teach a robot something new — the two options of either (a) demonstrations or (b) language instructions are perhaps the most natural for having humans specify tasks.¹⁸ While language suffices for some task specifications, many physical actions are difficult to precisely describe in language¹⁹ (e.g. it is far easier to show exactly how to seat two Lego bricks than to say it). Prompting a task in native observations and actions is also a more comprehensive test of sensorimotor understanding: the model must infer the goal from the demonstration, repurpose existing knowledge, and improvise under new initial conditions.

Compositional Generalization with Physical Prompt Engineering

Physical prompts can also be composed. For example, if we place demonstrations of two different tasks in context (each recorded independently, with no transition between them), GEN-1.5 can chain them into one continuous behavior, performing one task and then flowing into the next. The model bridges the two on its own, producing intermediate motions (repositioning, regrasping, error recovery) that appear in neither demonstration.

In practice, this opens up “physical prompt engineering”: rather than collecting a demonstration of a full compound task, one can assemble it from a small library of short, reusable physical prompts. As models improve, composing skills in context may become a practical way to program longer-horizon behaviors; the physical analogue of chaining instructions in a language prompt.

Zero-Shot Sim-to-Real Transfer with In-Context Learning

In-context learning also crosses the sim-to-real gap. A prompt can be formed entirely from simulated experience (e.g., from a scripted policy, an RL agent, or a human teleoperating a simulated robot) and be used to prompt the real robot. To clarify, “zero-shot sim2real transfer” typically refers to training a policy in a simulator on a particular task, then running that policy in the real world without real-world data for that task. In the case we show here, however, the model was not trained on the task in either the simulator or the real world.

GEN-1.5 pretraining contains no simulation data, neither rendered video nor simulated dynamics, yet the model can be effectively prompted by rollouts from the simulator. The prompted behaviors then generalize as other physically prompted behaviors do: to different hands, and to new object positions and sizes in the real scene. For a subset of tasks, this means demonstrations no longer need to be collected physically — they can instead be gathered by whichever means inside a simulator.

Human-to-Robot In-Context Learning

In some cases, in-context learning transfers across the embodiment gap entirely: a human demonstrates a task with their own hands, observable through the robot’s cameras, and the robot can reproduce it immediately afterward.

Few Gradient Step Adaptation

GEN-1.5 can also adapt to new physical tasks in extremely few gradient steps, as few as 1 to 10. Typically, training previous robot models on a new task can take tens of thousands of gradient steps (sometimes orders of magnitude more), but a hallmark capability of foundation models²⁰ is that they can be rapidly adapted to new tasks with a small amount of fine-tuning. While this mechanism can be built in explicitly e.g. with second order gradients encouraging fast adaptation,¹¹ we find that the pretrained base of GEN-1.5 already adapts in very few steps without any such machinery.

Importantly, such an extremely small amount of training steps requires orders of magnitude less task-specific compute, and opens up a much more flexible view of task adaptation than heavy finetuning. It may be more apt to describe this as test-time training²¹ in an extremely low-data regime. Test-time training commonly uses tens of gradient steps; GEN-1.5 learns a new physical task in 1–10 steps on 5 minutes of data. Ten steps change the model weights on held out tasks by less than 0.15%, suggesting that fine-tuning slightly reconfigures knowledge already present rather than building new representations.

In our experiments for 10-step adaptation, we sample sequences from 5 minutes of data and train with gradient descent using hyperparameters similar to pretraining. In the extreme one-step regime, sampling from one minute of data, success on a held-out task is 66.5%, and performance improves with larger batch sizes and higher learning rates. We did not tune this procedure or sweep adaptation-specific hyperparameters; these results come largely out of the box.

Physical Generalization

For every task above, fine-tuned models generalize well beyond their demonstrations — not only to new embodiments, object instances, and environments, but also to fundamentally different manipulation strategies for the same goal: alternative grasps and motions, clearing obstacles, and using tools absent from the fine-tuning data.

The ability to improvise under unexpected situations appears to be central to how the model masters new tasks: it can correct its own mistakes, sometimes before they occur. We first observed traces of this behavior in GEN-1.¹⁵ In GEN-1.5 it is both more frequent and more sophisticated, and it strengthens as the number of fine-tuning gradient steps decreases, presumably because lightly adapted models stay closer to their pretrained priors and can draw on a broader repertoire of behaviors when the situation departs from the demonstrations.

More examples of intriguing emergent improvisation behaviors:

Looking Ahead

When we started this journey, we did not set out to specifically build a one-shot learner. Much of our history as a team has been spent building the fundamental machinery required to iterate on the science of pretraining in robotics from first principles, beginning with a data engine that could fuel the model science with high-quality physical experience at scale.

GEN-1.5 is a milestone we believe to be profound scientifically, not because of higher success rates, but because it represents a new frontier of generality — one that challenges our own understanding of how these models behave when pretrained at a scale of physical interaction data few thought possible without shortcuts. GEN-0 and GEN-1 each gave us increasing confidence that more (and better) pretraining would make adaptation to new tasks more data-efficient. Every trend we measured pointed in the same direction: more pretraining makes adaptation faster, cheaper, and more general. We do not yet see where that curve asymptotes.

What is clear now, and perhaps obvious in hindsight, is that past a certain threshold of pretraining, the cost of adaptation becomes negligible. Emergent in-context learning from a few seconds of data, or one gradient step on one minute of demonstrations, is no longer task-specific training in the conventional sense. It is closer to reminding the model of something it nearly knows, with a tiny amount of compute. That this works at all, changes how we think about how these models can be used, about their potential impact, and the road ahead for building general physical intelligence.

For decades, robots have been marketed as “general-purpose” machines that could in principle do anything — a contrast to the single-purpose factory automation of the past. But that promise was always conditioned on an expert programming them, which took months of effort and specialized knowledge. If interacting with a robot reduces to simply showing it what to do, then two things change fundamentally: how quickly a robot becomes useful (seconds, not months), and who can work with one (anyone).

We are in the early days of our mission to build physical AGI and make it useful to everyone. If you are interested in joining us on this journey, reach out at generalistai.com/careers.

Citation

Please cite this work as

Or use the BibTeX citation: