Learning Physical Principles
from Interaction

Self-Evolving Planning via Test-Time Memory

Haoyang Li · Yang You · Hao Su · Leonidas Guibas

Stanford University · UC San Diego

Paper Code Video

Abstract

Reliable object manipulation requires understanding physical properties that vary across objects and environments. Vision-language model (VLM) planners can reason about friction and stability in general terms; however, they often cannot predict how a specific ball will roll on a particular surface or which stone will provide a stable foundation without direct experience. We present PhysMem, a memory framework that enables VLM robot planners to learn physical principles from interaction at test time, without updating model parameters. The system records experiences, generates candidate hypotheses, and verifies them through targeted interaction before promoting validated knowledge to guide future decisions. A central design choice is verification before application: the system tests hypotheses against new observations rather than applying retrieved experience directly.

76%
Success Rate
3.3×
vs. Baseline
0
Parameter Updates

Presentation

Watch the Overview

Overview of PhysMem

Method

How PhysMem Works

Inspired by the scientific method, PhysMem transforms raw interaction experiences into verified physical principles through a four-stage loop.

System overview of PhysMem

System overview. Left: A three-tier memory system stores raw experiences, clusters them into testable hypotheses, and promotes verified knowledge as principles. Right: The VLM agent retrieves principles and hypotheses from memory to guide planning.

Collect

Gather interaction experiences: visual observations, actions taken, and their outcomes.

Hypothesize

Generate candidate physical principles from patterns observed in collected experiences.

Verify

Validate hypotheses through action-level outcome attribution across multiple trials.

Promote

Verified principles join persistent memory, guiding all future planning decisions.

Demonstrations

Tasks & Environments

PhysMem is evaluated on three physically challenging manipulation tasks requiring understanding of contact physics, momentum, and stability.

Parts Organization

Arrange irregularly shaped objects into a target configuration, requiring understanding of geometry and contact physics.

Ball Navigation

Guide a ball to a target location by pushing, requiring grasp of momentum and friction dynamics.

Balanced Stacking

Stack objects to achieve balance, demanding knowledge of center-of-mass and stability principles.

Results

76% Average Success Rate

3.3× improvement over the strongest VLM baseline

Improvement Over Time

Performance improves steadily as PhysMem accumulates verified principles from each interaction trial.

Cross-VLM Transfer

Principles discovered by one VLM transfer to others — physical knowledge is model-agnostic.

Zero Parameter Updates

All learning happens in context — no fine-tuning, no gradient updates, no retraining required.

OOD Generalization

Learned principles generalize to out-of-distribution scenarios with novel materials, masses, and environments.

Comparison

Before vs. After Memory

Side-by-side comparison showing how PhysMem's learned principles transform task performance.

Ball Navigation
Without Memory
With PhysMem
Parts Organization
Without Memory
With PhysMem

Generalization

Out-of-Distribution Transfer

PhysMem's principles generalize to unseen material properties, object masses, and even lunar gravity.

OOD material variations
No Prior, No Update
No Prior, With Update
With Prior, No Update
With Prior + Update
No Prior, No Update
No Prior, With Update
With Prior, No Update
With Prior + Update
No Prior, No Update
No Prior, With Update
With Prior, No Update
With Prior + Update
No Prior, No Update
No Prior, With Update
With Prior, No Update
With Prior + Update

Citation

BibTeX

@article{li2025physmem,
  title   = {Learning Physical Principles from Interaction:
             Self-Evolving Planning via Test-Time Memory},
  author  = {Li, Haoyang and You, Yang and Su, Hao and Guibas, Leonidas},
  journal = {arXiv preprint arXiv:2602.20323},
  year    = {2025}
}