Research note · pose & depth estimation

Can 1,000 Open Scenes Make a State-of-the-Art 3D Model Better?

We fine-tuned R3, a state-of-the-art streaming 3D reconstruction model, on roughly one thousand of our COLMAP scenes. The short answer is yes — measurably, reproducibly, and on the checkpoint people actually deploy. The longer answer is the work around it: what breaks when you push a foundation model onto new data (depth, external benchmarks), how we walked every regression back, and where we ended up with a model that beats the public checkpoint on its own turf.

R3 — relative-pose regression on a DA3 / DINOv2-L backbone [1,2,3] · both released variants studied: «short» (4–32 views) and «long» (32–100 views, the deployment default)
~1000 of our COLMAP scenes, metric ARKit poses [4,5] · replay anchor: ARKitScenes [6]
Eval: Sim3-aligned ATE · 50 held-out scenes in-domain · TUM-RGBD [7] · an independent 800-frame FPV video vs a COLMAP reference

01The question

If we own data a model has never seen, what is it worth?

This is a practical question before it is an academic one. We own about a thousand outdoor and indoor scenes reconstructed with COLMAP, with metric camera poses from phone AR tracking. R3 is a relative-pose regression model built on a Depth Anything 3 / DINOv2 backbone, trained on a large mix of public datasets that does not include ours. We wanted to know whether further training on our scenes makes it better, and at what cost — because the same playbook applies to any foundation model you want to specialize on proprietary data.

Three numbers hold the story together; the rest of the note reconstructs them one at a time.

−25%

in-domain trajectory error vs the deployed public checkpoint (r3_long, 50 held-out scenes, 100 views)

2.48×

external-benchmark degradation with a naive fine-tune — the risk, measured

out-of-domain regressions at our recommended operating point (α=0.3): it matches or beats the public checkpoint on every external test we ran

02The gain

On our domain, pose accuracy improves — visibly

Our first study used the «short» variant (windows of up to 32 views). We evaluate on 50 scenes held out of training entirely, measuring trajectory error (ATE) normalized by window extent. The base model starts at 37.7%; after fine-tuning we land around 32%, with rotation error dropping from 5.0° to 3.7°. Three independent runs converge to the same numbers, so this is not one lucky checkpoint.

The most direct way to see it is to overlay the camera trajectory the model reconstructs onto ground truth. Where the base model drifts, the adapted one stays on the path.

Predicted vs ground-truth trajectories on held-out scenes — Reconstructed camera trajectory, top view, on three held-out scenes: ground truth (black), base model (red), fine-tuned (green), all Sim3-aligned. The per-scene ATE drop is printed above each panel. Scene names are descriptive; one location (Málaga) is confirmed by its marketplace listing, linked in the interactive view below.

The same thing, explorable in 3D. What R3 outputs — and what fine-tuning improves — are camera poses: below you can inspect them directly. The reconstructed trajectory (ground truth, base, fine-tuned, Sim3-aligned) with a sparse context cloud. Drag to rotate, scroll to zoom; the buttons toggle each trajectory.

drag = orbit · scroll = zoom

Per-frame and cumulative position error along the sequence: base vs fine-tuned. The green area is the gain.

One clarification on method: we show poses, not a dense fused surface. R3's per-frame depth is good, but its scale is not perfectly consistent across frames, and fusing all depths into a single surface fragments on our outdoor scenes. Pose is what we measure, what fine-tuning improves, and exactly what this view shows.

Online reconstruction, frame by frame

In the style of R3's own demo: on the left the input stream and the rendered flythrough — the point cloud accumulating frame by frame, filmed by an external camera travelling above the reconstruction (rendered offline, as R3's project page does; predicted depth as an inset top-right, trajectory colored blue→red over time); on the right the reference point cloud, the full COLMAP reconstruction of the area — static, mouse-explorable, with the trajectory and current position. We compare our fine-tuned model against the R3 base; below, cumulative position error grows in real time.

Input stream

Rendered flythrough

Point cloud

drag to rotate · scroll to zoom · right-click to pan

Third-person flythrough over the growing reconstruction (predicted depth inset top-right, trajectory blue→red over time); on the right, the static COLMAP reference cloud with the trajectory and a marker at the current position — rotatable with the mouse. Bottom: cumulative position error, base vs fine-tuned. The short variant works in 32-view windows: on the 200-frame video, inference runs over overlapping windows, each Sim3-aligned to the COLMAP reference; the error shown is per-window, and the flythrough shows the local reconstruction of the most recent frames.