PhysMem

Scaling Test-time Physical Memory for Robot Manipulation

Haoyang Li · Yang You · Hao Su · Leonidas Guibas

Stanford University · UC San Diego

Stanford University UC San Diego
PDF arXiv Video Code Thread
TL;DR

World Interaction → Experience → Embodied Memory → Deliberation → Action

Abstract

Reliable object manipulation requires understanding physical properties that vary across objects and environments. Vision-language model (VLM) planners can reason about friction and stability in general terms; however, they often cannot predict how a specific ball will roll on a particular surface or which stone will provide a stable foundation without direct experience. We present PhysMem, a memory framework that enables VLM robot planners to learn physical principles from interaction at test time, without updating model parameters. The system records experiences, generates candidate hypotheses, and verifies them through targeted interaction before promoting validated knowledge to guide future decisions. A central design choice is verification before application: the system tests hypotheses against new observations rather than applying retrieved experience directly.

76%
Success Rate
3.3×
vs. Baseline
0
Parameter Updates

Presentation

Watch the Overview

Overview of PhysMem

Demonstrations

Tasks & Environments

PhysMem is evaluated on three physically challenging manipulation tasks requiring understanding of contact physics, momentum, and stability.

Parts Organization

Arrange irregularly shaped objects into a target configuration, requiring understanding of geometry and contact physics.

Ball Navigation

Guide a ball to a target location by pushing, requiring grasp of momentum and friction dynamics.

Balanced Stacking

Stack objects to achieve balance, demanding knowledge of center-of-mass and stability principles.

Comparison

Before vs. After Memory

Side-by-side comparison showing how PhysMem's learned principles transform task performance.

Ball Navigation
Without Memory
With PhysMem
Parts Organization
Without Memory
With PhysMem

Analysis

Test-Time Learning Curve

How does PhysMem improve through interaction, and how much experience is needed? We evaluate performance at different experience utilization levels (0%, 25%, 50%, 100%) across three runs per condition.

Test-time learning curves across tasks

Test-time learning curves. Without memory (0%), performance stays flat. With full memory, Parts Organization improves from −1 to 9.7 and Ball Navigation shows an even larger gain (14.7 vs. 0.7). Task complexity determines how much experience is needed: Ball Navigation benefits from full experience, while Balanced Stacking shows diminishing returns where 50% nearly matches 100%.

Generalization

Out-of-Distribution Transfer

PhysMem's principles generalize to unseen material properties, object masses, and even lunar gravity.

OOD material variations
Ball
No Prior, No Update
No Prior, With Update
With Prior, No Update
With Prior + Update
Ball (Moon)
No Prior, No Update
No Prior, With Update
With Prior, No Update
With Prior + Update
Parts
No Prior, No Update
No Prior, With Update
With Prior, No Update
With Prior + Update
Stack
No Prior, No Update
No Prior, With Update
With Prior, No Update
With Prior + Update

Method

How PhysMem Works

Inspired by the scientific method, PhysMem transforms raw interaction experiences into verified physical principles through a four-stage loop.

System overview of PhysMem

System overview. Left: A three-tier memory system stores raw experiences, clusters them into testable hypotheses, and promotes verified knowledge as principles. Right: The VLM agent retrieves principles and hypotheses from memory to guide planning.

Collect

Gather interaction experiences: visual observations, actions taken, and their outcomes.

Hypothesize

Generate candidate physical principles from patterns observed in collected experiences.

Verify

Validate hypotheses through action-level outcome attribution across multiple trials.

Promote

Verified principles join persistent memory, guiding all future planning decisions.

Citation

BibTeX

@article{li2025physmem,
  title   = {PhysMem: Scaling Test-time Physical Memory for Robot Manipulation},
  author  = {Li, Haoyang and You, Yang and Su, Hao and Guibas, Leonidas},
  journal = {arXiv preprint arXiv:2602.20323},
  year    = {2025}
}