Object-level 3D Gaussian representation learning

Gaussian-JEPA

Joint-Embedding Predictive Learning for 3D Gaussian Splats

Bin Ren¹, Qi Ma², Yue Li³, Zongyan Han¹, Yidi Li⁴, Yuqian Fu⁵, Rao Muhammad Anwer¹, Theo Gevers³, Fahad Shahbaz Khan¹, Salman Khan¹

¹MBZUAI ²ETH Zürich ³University of Amsterdam ⁴Taiyuan University of Technology ⁵KAUST

TL;DR: Gaussian-JEPA learns reusable features directly from 3D Gaussian assets. It predicts latent representations of held-out Gaussian token blocks from visible context, without reconstructing raw Gaussian attributes.

Paper soon Code

1K Gaussians 64 local tokens 4 target scales 14 input attributes

Motivation

One object admits different Gaussian observations.

Dense Gaussian assets exceed a practical encoder budget. Independent fixed-budget sampling therefore changes the primitive realization, although the underlying object is unchanged.

Reconstruction

Recover Gaussian attributes

Coordinates, opacity, scale, rotation, and appearance are decoded in input space.

Gaussian-JEPA

Predict hidden representations

Visible context predicts stop-gradient features of spatial target blocks at multiple scales.

Method

Multi-scale targets, shared predictive dynamics.

A 1K-Gaussian observation is grouped into 64 local tokens. Four non-overlapping targets cover 32 tokens with heterogeneous spatial support; the exact complement forms the shared context.

Gaussian-JEPA grouping, target construction, context encoder, EMA target encoder, predictor, and latent objectives — The online encoder observes the shared context. For each block, a shared EMA encoder supplies targets, while complementary projections and feature-space grounding define the latent objective.

Gaussian tokenization

All geometry and appearance attributes are encoded; xyz defines local neighborhoods.

Heterogeneous targets

Blocks of 11, 9, 7, and 5 groups provide local-to-coarse predictive supervision.

Feature-space objective

Prediction and grounding operate on learned targets rather than raw Gaussian attributes.

Gaussian-specific evidence

Evaluate what reconstruction accuracy does not show.

Frozen representations are tested under independent resampling and spatially partial observations. A shared completion decoder then measures whether partial features support complete Gaussian prediction.

Mean embedding drift−29.3%

Resampling consistency

Lower drift across independent 1K samples while retaining instance retrieval.

CDF of embedding drift for Gaussian-MAE and Gaussian-JEPA

R@1 AUC+10.92

Partial observations

Stronger partial-to-complete retrieval as spatial evidence is removed.

Retrieval under increasing missing Gaussian groups

Renderable completion

Frozen features support more coherent missing geometry.

Given the same partial 512-Gaussian input, identical decoders predict complete 1K-Gaussian representations. Gaussian-JEPA improves Chamfer distance, F-score, and render-space metrics.

Chamfer ↓: 0.0678
F_1% ↑: 7.42
SSIM ↑: 0.7469

Qualitative Gaussian shape completion comparison

Semantic transfer

Competitive features on standard downstream tasks.

Gaussian methods use a matched 1K-Gaussian pretraining and transfer budget. Frozen probing isolates the quality of the pretrained encoder.

Evaluation	Gaussian-MAE	Gaussian-JEPA
MN10 · Full	94.16	94.94
MN40 · Full	92.54	92.63
MN10 · Linear	93.50	93.72
MN40 · Linear	88.97	90.47

ShapeNet-Part 84.5 class mIoU 86.1 instance mIoU

Part segmentation representation visualizations — Qualitative structure learned by point- and Gaussian-based encoders.

Author affiliations

Citation

Gaussian-JEPA

Please cite this work as below. Archival metadata will be updated with the public manuscript.

@misc{gaussian_jepa,
  title   = {Gaussian-JEPA: Joint-Embedding Predictive Learning
             for 3D Gaussian Splats},
  author  = {Ren, Bin and Ma, Qi and Li, Yue and Han, Zongyan
             and Li, Yidi and Fu, Yuqian and Anwer, Rao Muhammad
             and Gevers, Theo and Khan, Fahad Shahbaz
             and Khan, Salman},
  year    = {2026},
  note    = {Project page and code release}
}