Given a single 360° panorama and a textual trajectory description, OmniCam autoregressively generates an SE(3) camera pose sequence that respects scene geometry — no manual keyframing, no geometry-unaware templates.
Camera trajectory is a fundamental control signal for world models, video generation, navigation, and robotic manipulation, yet current methods rely on manually defined paths or geometry-unaware templates that ignore scene structure. Three representational limitations hinder progress: the lack of omnidirectional spatial perception; pose tokenization discontinuity arising from quaternion sign ambiguity and scene-scale inconsistencies; and entanglement between geometric and semantic features in single-branch encoders. All three are exacerbated by insufficient geometric perception throughout the pipeline, which motivates geometry-grounded pose token learning — a principle that integrates explicit 3D reasoning into trajectory generation. This principle is realized in OmniCam, an autoregressive model that generates SE(3) camera pose sequences from a single panorama and textual trajectory descriptions. To enable training at scale, we construct the OmniCaT dataset containing 267K trajectories across four camera behaviors. OmniCam achieves 28–47% trajectory error reduction and 66% collision reduction on held-out scenes, and generalizes zero-shot to DataDoP with 35% lower ATE, 69% fewer collisions, and 75% higher visibility.
Each addresses one representational limitation of prior camera-trajectory models.
Lifts the input 360° panorama into an explicit 3D point cloud, giving the generator omnidirectional spatial perception instead of a flat, geometry-unaware image feature.
Absolute rotation + relative translation with a canonical sign convention and scene-aware scaling, removing quaternion sign ambiguity and scene-scale discontinuities before uniform binning.
Separate geometric and semantic branches provide complementary conditioning, with 3D target grounding injecting the intended object’s centroid into the autoregressive decoder.
A single instruction tag selects the intent; OmniCam plans a geometry-consistent path.
Push toward and frame a named object, keeping it centered and visible along the path.
Orbit a subject at a stable radius, revealing it from multiple viewpoints.
Sweep coverage-maximizing paths ideal for downstream 3D reconstruction.
Explore free space with smooth, collision-free forward motion.
Each case shows the planned clip (point-cloud render along the generated path) and the geometry-completed result clip. All previews autoplay.
The video opens on a modern house with a clean, minimalist design; the camera pans toward the dark green double doors.
A grand, stately home at dusk; a coverage sweep moves forward along the paved walkway toward the arched entrance.
A tranquil courtyard at dusk; the camera orbits left around the sculpted tree and glowing facade.
A cozy cabin under a starry sky; the camera pans right to frame the glowing decorated tree in the yard.
A snow-covered clearing framed by a low fence; the camera drifts right across the misty, serene landscape.
A grand dining hall; the camera tilts upward to reveal the ceiling moldings and crystal chandeliers.
An opulent dining hall; the camera glides forward along the table, drawing the viewer deeper into the room.
Walls swirl in blue and gold; the camera pans right toward the sunlit wooden-framed window.
A warmly lit swirling room; the camera moves forward and right, sweeping coverage toward the sunflower table.
A fully automated pipeline turns raw panoramas into 267K geometry-grounded, language-annotated trajectories.
Panoramic depth estimation, point-cloud filtering, open-vocabulary semantic targeting, and NavMesh construction produce a navigable, labeled 3D scene.
Heuristics on the navigation graph generate Target, Surround, Reconstruct and Wander paths that stay collision-free and geometry-aware.
Point-based rendering plus kinematic-primitive extraction and VLM captioning yield paired (panorama, instruction, SE(3) trajectory) samples.
Feeding OmniCam poses to a video model yields sharper, temporally consistent camera control.
Coverage-maximizing Reconstruct paths supply well-distributed views for scene reconstruction.
Geometry-aware viewpoints improve downstream manipulation and exploration policies.
@inproceedings{omnicam2026,
title = {OmniCam: Omni-Camera Trajectory Generation via
Geometry-Grounded Pose Token Learning},
author = {Anonymous Authors},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}