Full search page →

DLCV 11A · Depth Estimation and Optical Flow

Deep Learning for Computer VisionAdvanced38 minUpdated Nov 6, 2026

Needs: DLCV 11 · 3D Reconstruction: From Multi-View Geometry to NeRF and Gaussian Splatting

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 4

  1. Robust matching costs: ZNCC and census
  2. Semi-global matching step by step
  3. Sub-pixel disparity and the soft argmin
  4. Monocular depth foundation models: data and the camera

Paper guides · 3

  1. RAFT (Teed and Deng, 2020)
  2. MiDaS (Ranftl et al., 2022)
  3. Guide to the Modern view surveys

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →
Explore this chapter 1 interactive experiment

Run block matching on a synthetic stereo pair with known disparity, and see small windows turn noisy while large windows fatten the foreground.

What you’ll learn

  • Distinguish stereo correspondence, temporal flow and monocular depth inference.
  • Relate disparity to calibrated camera geometry and depth error.
  • Compare matching costs, cost volumes and iterative refinement.
  • Separate relative depth, metric depth and view-synthesis supervision.
  • Evaluate with explicit alignment, visibility masks, units and dataset splits.

The big picture

A depth map gives, for every pixel, how far away the surface is. An optical flow field gives, for every pixel, where that point moves in the next frame. Robots, cars, phones and video editors need both: depth to know where things are, flow to know how they move. Both are answers to the same question: which pixel in the other image shows the same point?

Why it matters. Stereo depth lets robots and cars measure distance without a laser scanner; single-image depth blurs portrait backgrounds on phones and places objects into video; optical flow drives video super-resolution, frame interpolation and stabilization.

How this tutorial relates to others. DLCV 11 covers camera geometry, triangulation and multi-view stereo; DLCV 09A.3 introduces brightness constancy and warping; long-range point tracking is in DLCV 07 and sparse keypoint matching in IP 12. We link to them instead of repeating them. Our focus is the dense case: one answer per pixel.

Figure 1 shows the four model families of this tutorial: three match two images, the fourth relies on what it has learned.

Four rows of four boxes joined by arrows. Classical stereo: matching cost, aggregation, winner-take-all, refinement. Cost-volume network: shared CNN features, cost volume, 3D CNN regularization, soft argmin. Iterative refinement: features and context, all-pairs correlation, look up around current estimate, GRU update with a loop arrow. Monocular network: one image, encoder, dense decoder, relative or metric depth.
Figure 1 — Four families of dense correspondence and depth models. Classical stereo (grey) and cost-volume networks (blue) follow the same four stages; iterative models (green) repeat a lookup-and-update step; monocular networks (orange) predict depth from a single image.

Three dense correspondence problems

Plain version. Rectified stereo matches pixels along the same row. Optical flow matches across frames; monocular depth estimates geometry without a second image.

Precise version. Let I1,I2I_1, I_2 be two images of H×WH \times W pixels and p=(u,v)\mathbf{p} = (u, v) a pixel, with column uu and row vv.

  • Optical flow is a vector field F(p)=(Δu,Δv)\mathbf{F}(\mathbf{p}) = (\Delta u, \Delta v) such that the point seen at p\mathbf{p} in I1I_1 is seen at p+F(p)\mathbf{p} + \mathbf{F}(\mathbf{p}) in I2I_2. Two unknowns per pixel; the cameras and the scene may both move.
  • Stereo disparity is the special case of two cameras that see the scene at the same instant and have been rectified (next section), so that every match lies on the same row: F(p)=(−d(p),0)\mathbf{F}(\mathbf{p}) = (-d(\mathbf{p}), 0). One unknown per pixel, the disparity d≥0d \ge 0, which converts to depth.
  • Monocular depth asks for a depth map Z(p)Z(\mathbf{p}) from I1I_1 alone. No correspondence is available, so the answer comes from learned regularities, and it is only defined up to an unknown scale unless the network has learned metric cues.
  • Scene flow asks for the 3D motion of the point seen at each pixel; it combines depth at two times with optical flow (see Scene flow below).

The two-image problems share one structure: a cost that scores a candidate match, a search over candidates, and a way to resolve pixels where the cost is ambiguous.

Stereo geometry: rectification and disparity

Plain version. In a calibrated rectified pair, nearby points have larger disparity. Focal length and camera baseline convert that disparity into depth.

Rectification

For a general pair of cameras, the match of a pixel lies on its epipolar line in the other image (DLCV 11, epipolar geometry). Rectification warps both images with a homography each so that all epipolar lines become horizontal and corresponding lines have the same row index [1]. After rectification the two virtual cameras have the same intrinsics, parallel optical axes and a purely horizontal baseline. Stereo rigs are calibrated once, and every frame is rectified before matching; all the stereo benchmarks below distribute rectified pairs. In the rest of this tutorial the pair is always rectified.

Disparity and depth

Place the left camera centre at the origin and the right camera at distance BB along the xx axis (Figure 2a). A point at depth ZZ and horizontal position XX projects to column xL=fX/Zx_L = fX/Z in the left image and xR=f(X−B)/Zx_R = f(X - B)/Z in the right image, both measured from the principal point. Subtracting gives

d=xL−xR=fBZ,Z=fBd,d = x_L - x_R = \frac{fB}{Z}, \qquad Z = \frac{fB}{d},

where ff is the focal length in pixels, BB is the baseline (distance between the camera centres, in metres), ZZ is the depth along the optical axis and dd is the disparity in pixels. Depth and disparity are inversely proportional: disparity is an inverse depth, scaled by fBfB. Large disparities belong to near objects, and a disparity of zero means a point at infinity.

Left: a top-view diagram with two camera centres joined by a green baseline B, an image line at distance f in front of each, a red star P at depth Z, blue rays from both cameras to P and orange dots where the rays cross the image lines, labelled x_L and x_R; text above reads disparity d = x_L minus x_R = fB/Z. Right: a log-scale plot of depth error against depth from 1 to 60 m; an orange curve for B = 0.12 m rises from a few millimetres to about 10 m, and a blue curve for B = 0.5 m stays about four times lower.
Figure 2 — (a) A rectified stereo pair seen from above: a point shifts by the disparity d between the two images. (b) Depth error for a disparity error of 0.25 px with a focal length of 700 px; the error grows with the square of the distance and shrinks with a wider baseline.

Because Z=fB/dZ = fB/d, a small disparity error δd\delta d becomes a depth error of about δZ≈Z2δd/(fB)\delta Z \approx Z^2 \delta d / (fB), the same relation that DLCV 11, Triangulation derives. The snippet puts numbers on it for a hypothetical camera with f=700f = 700 px.

Code · Python · 9 lines
import numpy as np

f = 700.0                                    # focal length in pixels (a hypothetical rig)
for B in [0.12, 0.5]:                        # baselines in metres
    print(f"baseline {B} m: depth at a disparity of 1 px = {f * B:.0f} m")
    for Z in [2, 10, 30, 60]:
        d = f * B / Z                        # disparity in pixels
        dZ = Z ** 2 * 0.25 / (f * B)         # depth error for a 0.25 px disparity error
        print(f"   Z = {Z:2d} m  d = {d:6.2f} px  depth error for 0.25 px: {dZ:6.3f} m ({100 * dZ / Z:4.1f}%)")

Output

baseline 0.12 m: depth at a disparity of 1 px = 84 m
   Z =  2 m  d =  42.00 px  depth error for 0.25 px:  0.012 m ( 0.6%)
   Z = 10 m  d =   8.40 px  depth error for 0.25 px:  0.298 m ( 3.0%)
   Z = 30 m  d =   2.80 px  depth error for 0.25 px:  2.679 m ( 8.9%)
   Z = 60 m  d =   1.40 px  depth error for 0.25 px: 10.714 m (17.9%)
baseline 0.5 m: depth at a disparity of 1 px = 350 m
   Z =  2 m  d = 175.00 px  depth error for 0.25 px:  0.003 m ( 0.1%)
   Z = 10 m  d =  35.00 px  depth error for 0.25 px:  0.071 m ( 0.7%)
   Z = 30 m  d =  11.67 px  depth error for 0.25 px:  0.643 m ( 2.1%)
   Z = 60 m  d =   5.83 px  depth error for 0.25 px:  2.571 m ( 4.3%)

At 10 m, a quarter-pixel disparity error costs 30 cm with a 12 cm baseline and 7 cm with a 50 cm baseline. At 60 m, the short baseline sees a disparity of only 1.4 px and its depth is uncertain by 18%. This is why stereo cameras for driving use wide baselines and high resolution, and why sub-pixel accuracy matters: the last fraction of a pixel decides the far range.

Classical stereo matching

Plain version. Classical block matching searches candidate disparities, compares local patches and refines unreliable matches.

Scharstein and Szeliski organized dense two-frame stereo algorithms into four steps that most methods perform in some form [2]:

  1. Matching cost computation: a per-pixel, per-disparity dissimilarity.
  2. Cost (support) aggregation: combine costs over a neighbourhood to reduce noise.
  3. Disparity computation or optimization: choose a disparity per pixel, either locally or by optimizing an energy over the whole image.
  4. Disparity refinement: sub-pixel estimates, consistency checks, filling and smoothing.

The Middlebury stereo website, which accompanies that paper, still hosts an online evaluation [2], [3]. The plane sweep in DLCV 11 is exactly steps 1–3 with windowed SSD; here we look at each step more closely and add the refinement.

Matching costs and the cost volume

The simplest cost compares intensities in a small window. The sum of absolute differences (SAD) at pixel (u,v)(u, v) and disparity dd is

C(u,v,d)=∑(i,j)∈W∣IL(u+i,v+j)−IR(u+i−d,v+j)∣,C(u, v, d) = \sum_{(i, j) \in \mathcal{W}} \bigl| I_L(u + i, v + j) - I_R(u + i - d, v + j) \bigr|,

where IL,IRI_L, I_R are the left and right images, W\mathcal{W} is the set of offsets of a w×ww \times w window centred on the pixel, and dd ranges over integer hypotheses 0,1,…,D−10, 1, \dots, D - 1. Squared differences give SSD. Computing CC for every pixel and every dd fills a cost volume of size D×H×WD \times H \times W. The winner-take-all (WTA) rule picks d^(u,v)=arg⁡min⁡dC(u,v,d)\hat d(u, v) = \arg\min_d C(u, v, d).

SAD assumes both cameras record the same brightness for the same point. Real cameras differ in gain and exposure, so practical systems use zero-mean normalized cross-correlation (ZNCC) or the census transform, which encodes which neighbours are darker than the pixel as a bit string and compares bit strings [4].

Optional: Robust matching costs: ZNCC and census (in the Supplementary notes). The definitions, and a test in which a brightness change between the views breaks SAD but not the other two.

Figure 3a shows one row of a cost volume: dark valleys trace the surfaces. Figure 3b shows four typical cost curves. A textured pixel has one sharp minimum; a textureless pixel a flat curve; a repetitive pattern several equal minima, one per period; and an occluded pixel, hidden in the right image, no correct minimum at all.

Left: a colour map with column x on the horizontal axis and disparity hypothesis on the vertical axis; dark horizontal valleys at disparities near 20, 9, 16, 13 and 8 follow a white dashed true-disparity line, and red dots of the lowest-cost choice sit on the valleys except near the left edge and object boundaries. Right: four normalized cost curves against disparity; a blue textured-wall curve with one sharp dip near 9, a nearly flat grey textureless curve, an orange stripes curve with several deep dips and a red occluded curve with no clear minimum.
Figure 3 — (a) One row of the SAD cost volume of the synthetic scene below; low cost (dark) follows the true disparity (dashed). (b) Cost curves of four pixels; dotted lines mark their true disparities.

Code: block matching with a left–right check

The snippet ray-casts a rectified pair of a synthetic scene with known depth: a slanted textured wall, a box, a disc and a thin pole in front of it, plus a textureless patch and a patch of fine stripes on the wall. It builds the SAD cost volume, takes the WTA disparity, and runs the left–right consistency check: match again with the right image as reference, and reject pixels where the two answers disagree by more than one pixel. Occluded pixels usually fail this check, because their “match” in the right image belongs to a different surface whose own match points elsewhere. For comparison it also runs OpenCV’s semi-global matcher (next subsection). Errors are reported as bad-1, the percentage of pixels whose disparity is off by more than 1 px, separately for each kind of region.

Code · Python · 68 lines
import numpy as np
import cv2
from scipy import ndimage as ndi

rng = np.random.default_rng(0)
H, W, f, B = 180, 280, 320.0, 0.25                    # image size, focal length (px), baseline
tex = ndi.gaussian_filter(rng.random((700, 700)), 1.2)
tex = (tex - tex.min()) / np.ptp(tex)                 # random texture painted on every surface

def render(cx):
    """Ray-cast a rectified camera at (cx, 0, 0) looking down +Z: image, depth Z, region labels."""
    v, u = np.mgrid[0:H, 0:W].astype(float)
    dx, dy = (u - W / 2) / f, (v - H / 2) / f            # ray direction (dx, dy, 1)
    Z = (9 + 0.15 * cx) / (1 - 0.15 * dx)                # slanted wall Z = 9 + 0.15 X
    lab = np.zeros((H, W), int)                          # 0 wall, 1 box, 2 disc, 3 thin pole
    for k, z, inside in [(2, 6.0, lambda X, Y: (X - 1.2) ** 2 + (Y - 0.3) ** 2 < 0.36),
                         (3, 5.0, lambda X, Y: np.abs(X - 0.2) < 0.04),
                         (1, 4.0, lambda X, Y: (np.abs(X + 1.0) < 0.6) & (np.abs(Y) < 0.6))]:
        hit = inside(cx + z * dx, z * dy) & (z < Z)
        Z, lab = np.where(hit, z, Z), np.where(hit, k, lab)
    X, Y = cx + Z * dx, Z * dy
    img = ndi.map_coordinates(tex, [350 + 40 * Y + 60 * lab, 350 + 40 * X], order=1)
    flat = (lab == 0) & (X > 2.2) & (X < 3.6) & (Y > -2.0) & (Y < -0.9)
    rep = (lab == 0) & (X > -3.6) & (X < -2.2) & (Y > 0.9) & (Y < 2.0)
    img[flat] = 0.5                                                  # textureless patch: label 4
    img[rep] = 0.5 + 0.4 * np.sin(2 * np.pi * X[rep] / 0.15)        # repetitive stripes: label 5
    lab[flat], lab[rep] = 4, 5
    return img + rng.normal(0, 0.01, img.shape), Z, lab

left, Z_left, lab = render(0.0)
right, Z_right, _ = render(B)
d_true = f * B / Z_left                                # ground-truth disparity of the left view
rows, cols = np.mgrid[0:H, 0:W]
xr = np.round(cols - d_true).astype(int)               # where each left pixel lands in the right view
occluded = (xr < 0) | (np.abs(Z_right[rows, np.clip(xr, 0, W - 1)] - Z_left) > 0.05)

def sad_volume(ref, other, D, win, sign):
    """cost[d](y, x) = sum over a win x win window of |ref(y, x) - other(y, x - sign*d)|."""
    cost = np.full((D, H, W), np.inf)
    for d in range(D):
        c = ndi.uniform_filter(np.abs(ref - np.roll(other, sign * d, axis=1)), win) * win ** 2
        if sign > 0: c[:, :d] = np.inf                 # no partner pixel exists there
        else:        c[:, W - d:] = np.inf
        cost[d] = c
    return cost

D, win = 32, 7
d_left = sad_volume(left, right, D, win, +1).argmin(0)     # winner-take-all, left image as reference
d_right = sad_volume(right, left, D, win, -1).argmin(0)    # winner-take-all, right image as reference
lr_ok = np.abs(d_left - d_right[rows, np.clip(cols - d_left, 0, W - 1)]) <= 1   # left-right check

sgbm = cv2.StereoSGBM_create(minDisparity=0, numDisparities=D, blockSize=5,
                             P1=8 * 25, P2=32 * 25, uniquenessRatio=0, disp12MaxDiff=-1)
u8 = lambda a: np.clip(a * 255, 0, 255).astype(np.uint8)
d_sgbm = sgbm.compute(u8(left), u8(right)).astype(float) / 16    # OpenCV stores 16 x disparity

valid = ~occluded & (cols >= D)                        # visible in both views, away from the left strip
regions = {"textured": valid & np.isin(lab, [0, 1, 2]), "textureless": valid & (lab == 4),
           "stripes": valid & (lab == 5), "thin pole": valid & (lab == 3)}
bad = lambda d, m: 100 * np.mean(np.abs(d - d_true)[m] > 1)      # % of pixels off by > 1 px
print(f"disparity {d_true.min():.1f}-{d_true.max():.1f} px; occluded: {100 * occluded.mean():.1f}% of pixels")
print("bad-1 (%)          " + "".join(f"{k:>12s}" for k in regions))
for name, d in [("SAD + WTA", d_left), ("OpenCV SGBM", d_sgbm)]:
    print(f"{name:19s}" + "".join(f"{bad(d, m):12.1f}" for m in regions.values()))
print(f"left-right check: keeps {100 * lr_ok[valid].mean():.1f}% of visible pixels, "
      f"rejects {100 * (~lr_ok[occluded]).mean():.1f}% of occluded ones")
kept = valid & lr_ok
print(f"SAD + WTA bad-1 on all visible pixels: {bad(d_left, valid):.1f}% -> after the check {bad(d_left, kept):.1f}%")

Output

disparity 8.3-20.0 px; occluded: 7.7% of pixels
bad-1 (%)              textured textureless     stripes   thin pole
SAD + WTA                   1.8        67.7        89.6        17.4
OpenCV SGBM                 1.7        18.2        53.3        67.2
left-right check: keeps 96.6% of visible pixels, rejects 93.0% of occluded ones
SAD + WTA bad-1 on all visible pixels: 6.1% -> after the check 3.6%

These numbers come from this synthetic scene only. On textured surfaces, SAD block matching is right on 98% of the pixels; in the textureless patch it is wrong on two thirds of them and on the stripes on nine out of ten, as the cost curves predicted. The left–right check rejects 93% of the occluded pixels while keeping 96.6% of the visible ones, and bad-1 on the kept pixels drops from 6.1% to 3.6%. Rejected pixels are holes, not errors: later stages know not to trust them and can fill them from neighbours.

Six panels. (a) A grey random-texture left image with outlines around a flat grey rectangle at the top right, a striped rectangle at the bottom left and a thin vertical pole. (b) The true disparity map: a bright square, a pink disc and a thin pink line on a dark slanted background. (c) The SAD result: mostly correct but noisy in the flat rectangle, wrong in the stripes, and with fattened borders around the square, disc and pole. (d) The same after the left-right check, with rejected pixels in green along object borders and in the flat patch. (e) The OpenCV SGBM result: the flat patch is filled smoothly, but the thin pole has mostly vanished. (f) A bar chart of bad-1 per region for both methods.
Figure 4 — Block matching on the synthetic rectified pair. (c) SAD 7 × 7 with winner-take-all; (d) after the left–right check (rejected pixels in green); (e) OpenCV's semi-global matcher; (f) bad-1 per region from the snippet. The left strip, which has no partner pixels, is not evaluated.

Cost aggregation and semi-global matching

Plain version. Window matching decides each pixel alone. A better rule lets neighbouring pixels “vote”: a pixel in a blank region should take the disparity of its textured neighbours, unless there is good evidence for a depth jump.

The classical way to say this is an energy over the whole disparity map DD,

E(D)=∑pC(p,D(p))+∑(p,q)P1 [ ∣D(p)−D(q)∣=1 ]+P2 [ ∣D(p)−D(q)∣>1 ],E(D) = \sum_{\mathbf{p}} C\bigl(\mathbf{p}, D(\mathbf{p})\bigr) + \sum_{(\mathbf{p}, \mathbf{q})} P_1 \, [\,|D(\mathbf{p}) - D(\mathbf{q})| = 1\,] + P_2 \, [\,|D(\mathbf{p}) - D(\mathbf{q})| \gt 1\,],

where the second sum runs over pairs of neighbouring pixels p,q\mathbf{p}, \mathbf{q}, [⋅][\cdot] is 1 when the condition holds and 0 otherwise, P1P_1 is a small penalty for a change of one disparity level (slanted surfaces) and P2>P1P_2 \gt P_1 is a larger penalty for a jump (depth edges) [5], [6]. Minimizing this energy exactly over a 2D image is intractable in general.

Semi-global matching (SGM) by Hirschmüller [5] approximates it by minimizing the energy along many straight 1D paths through each pixel, each of which can be solved exactly by dynamic programming, and summing the results. Along a path with direction r\mathbf{r} the aggregated cost is

Lr(p,d)=C(p,d)+min⁡(Lr(p−r,d),  Lr(p−r,d±1)+P1,  min⁡kLr(p−r,k)+P2)−min⁡kLr(p−r,k),L_{\mathbf{r}}(\mathbf{p}, d) = C(\mathbf{p}, d) + \min\Bigl( L_{\mathbf{r}}(\mathbf{p} - \mathbf{r}, d),\; L_{\mathbf{r}}(\mathbf{p} - \mathbf{r}, d \pm 1) + P_1,\; \min_k L_{\mathbf{r}}(\mathbf{p} - \mathbf{r}, k) + P_2 \Bigr) - \min_k L_{\mathbf{r}}(\mathbf{p} - \mathbf{r}, k),

where p−r\mathbf{p} - \mathbf{r} is the previous pixel on the path and the last term only keeps the numbers bounded [6]. The costs of all directions are summed and WTA picks the disparity, so each pixel receives evidence from far away in many directions at a cost linear in the size of the cost volume. OpenCV’s StereoSGBM is a modified version: by default 5 directions instead of the full algorithm’s 8, small blocks instead of single pixels, and the Birchfield–Tomasi cost instead of mutual information [7].

The snippet shows SGM’s trade-off. In the textureless patch, bad pixels drop from 67.7% to 18.2%, because the smoothness term carries the wall’s disparity into it. But the thin pole, about five pixels wide, is mostly erased (17.4% → 67.2%): it costs two P2P_2 jumps for little evidence (Figure 4e). Neither method resolves the stripes, a genuine ambiguity of the data. Smoothness priors bet that the world is made of large surfaces, and that bet loses on thin structures.

Optional: Semi-global matching step by step (in the Supplementary notes). The recursion along one scanline in code, why P1<P2P_1 \lt P_2, and how two opposite paths fill a textureless gap without blurring a depth step.

Refinement: sub-pixel disparity and post-processing

WTA returns integer disparities, but Figure 2b showed that fractions of a pixel matter. The usual fix fits a parabola through the cost at the best integer disparity and its two neighbours and takes the parabola’s minimum. Further post-processing steps include the left–right check, a median filter and filling of rejected pixels from their neighbours; MC-CNN’s pipeline, for example, combines cross-based aggregation, SGM, a left–right check, sub-pixel enhancement, a median filter and a bilateral filter [6].

Optional: Sub-pixel disparity and the soft argmin (in the Supplementary notes). The parabola formula, and why a learned soft argmin can land between two surfaces at a depth edge.

Learned stereo matching

Plain version. Every step of the classical pipeline has a hand-made rule. Deep stereo replaces them, one at a time, by learned ones: first the cost, then the aggregation, and finally the whole pipeline trained end to end on large synthetic datasets.

The surveys of Laga et al. [8] and Tosi et al. [9] follow the same order as this section: learned parts of the pipeline, then end-to-end networks, then the architectures of the 2020s.

Learning the matching cost

MC-CNN by Žbontar and LeCun [6] trains a convolutional network to decide whether two small patches show the same point, using pairs built from ground-truth disparities as a binary classification set. The network’s output replaces SAD as the matching cost, and the rest of the classical pipeline (cross-based aggregation, SGM, left–right check, sub-pixel enhancement, filtering) follows. The lesson: a learned similarity is far more discriminative than raw intensities, but the hand-made aggregation remained.

End-to-end networks with cost volumes

End-to-end training needs dense ground truth for many images, which real scans rarely give. Mayer et al. [10] rendered three datasets (FlyingThings3D, Monkaa, Driving); FlyingThings3D alone has about 25,000 stereo frames of random objects flying along random 3D paths, and with it they trained DispNet, an encoder–decoder that regresses disparity [10]. Together these “Scene Flow” datasets became the usual pre-training data for stereo networks.

GC-Net by Kendall et al. [11] made the cost volume explicit and learnable. A shared 2D CNN extracts features from both images. For every disparity level the left feature is concatenated with the right feature shifted by that level, which gives a 4D volume of size height × width × (max disparity + 1) × feature size [11]. 3D convolutions over this volume learn the aggregation, using context in space and in disparity. Finally, a soft argmin turns the regularized costs cdc_d into a sub-pixel disparity:

d^=∑d=0Dmax⁡d⋅σ(−cd),σ(−cd)=e−cd∑d′e−cd′,\hat d = \sum_{d=0}^{D_{\max}} d \cdot \sigma(-c_d), \qquad \sigma(-c_d) = \frac{e^{-c_d}}{\sum_{d'} e^{-c_{d'}}},

where σ\sigma is the softmax over the disparity dimension and Dmax⁡D_{\max} the largest disparity [11]. Unlike the argmin it is differentiable, so the network trains end to end with a regression loss on d^\hat d. PSMNet [12] added a spatial pyramid pooling module that gathers context at several scales before the cost volume and a stacked-hourglass 3D CNN with intermediate supervision to regularize it, trained with the smooth L1 loss [12]. Both target ill-posed regions such as occlusions, repeated patterns, textureless and reflective surfaces, where local patches carry no answer [12].

The price of 3D convolutions is memory and computation proportional to D×H×WD \times H \times W, so these networks work at reduced resolution and a fixed maximum disparity.

Iterative refinement and zero-shot stereo

RAFT-Stereo [13] adapts the RAFT optical flow network (described in the optical flow section below) to rectified stereo: it replaces RAFT’s 4D all-pairs volume by a 3D volume that only compares pixels on the same row, and refines a disparity field step by step with recurrent units that look up the volume around the current estimate, adding multi-level GRUs that pass information across the image more efficiently [13]. There is no 3D convolution and no fixed disparity range to regularize. Tosi et al. call this design “a game changer”, noting that it brought much higher robustness to domain shift and that most recent frameworks follow it [9]. IGEV-Stereo [14] combines the two ideas: a geometry encoding volume that encodes geometry and context gives the recurrent updates a good starting point and is indexed by them at every iteration [14].

The open problem these surveys emphasized was generalization: networks fine-tuned on one benchmark often fail on another [9], [15]. FoundationStereo [16] targets zero-shot use: it is trained on a large synthetic dataset of about one million stereo pairs with automatic removal of ambiguous samples, and it adapts features from a monocular vision foundation model through a side-tuning backbone to narrow the gap between synthetic and real images [16].

Monocular depth estimation

Plain version. With one image there is nothing to match. A network has to recognize depth the way people do with one eye closed: from perspective, the known sizes of things, occlusion, shading, haze and texture that gets finer with distance. These cues tell you which things are nearer, but they cannot tell a toy car from a real one.

Why one image is not enough: scale ambiguity

Scale the whole scene by a factor ss and the image does not change, because projection divides by depth (Figure 5a; see also DLCV 11 and, for 3D detection, DLCV 03B). Eigen et al. called the global scale “a fundamental ambiguity in depth prediction” and showed that much of their error came from the mean depth alone [17].

A network can still learn metric cues, such as typical object sizes or the camera height of a driving dataset, but they fail when the camera or scene type changes. So two kinds of output are used:

  • Relative (affine-invariant) depth. The prediction is correct up to an unknown scale and shift. Many such models, MiDaS and Depth Anything among them, predict inverse depth (disparity-like values) [18], [19], so the output y^\hat y satisfies 1/Z≈sy^+t1/Z \approx s \hat y + t for unknown s>0s \gt 0 and tt. This is enough for ordering, segmentation, editing and many robot tasks when combined with a few metric measurements.
  • Metric depth. The prediction is in metres. It requires either a known camera or a model that estimates the camera from the image.

Losses that ignore scale and shift

To train on datasets whose depth is known only up to scale, or in different units, the loss must ignore the unknown factors. Eigen et al. introduced a scale-invariant log error [17]:

D(y,y∗)=1n∑i=1n(log⁡yi−log⁡yi∗+α(y,y∗))2,α(y,y∗)=1n∑i=1n(log⁡yi∗−log⁡yi),D(y, y^*) = \frac{1}{n} \sum_{i=1}^{n} \bigl( \log y_i - \log y_i^* + \alpha(y, y^*) \bigr)^2, \qquad \alpha(y, y^*) = \frac{1}{n} \sum_{i=1}^{n} \bigl( \log y_i^* - \log y_i \bigr),

where yiy_i and yi∗y_i^* are the predicted and true depths at pixel ii, nn is the number of pixels with ground truth, and α\alpha is the log-scale offset that best aligns the two maps; multiplying yy by any constant leaves DD unchanged [17]. For training they used a blend of this error and the ordinary log-space L2 error, weighted by λ=0.5\lambda = 0.5, which they found gave good absolute-scale predictions [17].

MiDaS by Ranftl et al. [18] went one step further: it predicts inverse depth (disparity) up to scale and shift, and aligns each prediction to the ground truth before computing the loss. With d∈RM\mathbf{d} \in \mathbb{R}^M the predicted disparities and d∗\mathbf{d}^* the ground truth at the MM valid pixels, the alignment solves

(s,t)=arg⁡min⁡s,t∑i=1M(sdi+t−di∗)2,d^i=sdi+t,(s, t) = \arg\min_{s, t} \sum_{i=1}^{M} \bigl( s d_i + t - d_i^* \bigr)^2, \qquad \hat d_i = s d_i + t,

which is an ordinary least-squares fit of a line with a closed-form solution [18]. The scale-and-shift-invariant loss is then Lssi=12M∑iρ(d^i−di∗)\mathcal{L}_{\text{ssi}} = \frac{1}{2M} \sum_i \rho(\hat d_i - d_i^*) with a per-pixel loss ρ\rho, here the squared error [18]. MiDaS also has a robust variant (median-based alignment, trimmed absolute loss), a gradient-matching term that keeps depth edges sharp, and mixes five training datasets, including one built from 3D films [18]. It is evaluated by zero-shot cross-dataset transfer, on datasets never seen in training [18].

Optional: MiDaS (Ranftl et al., 2022) (in the Paper guides). The problem the paper poses, how its tables are aligned, and what to check before comparing its numbers with others.

The snippet builds a toy depth map (a ground plane, a far wall and a box), simulates a relative model whose output is inverse depth with an unknown scale and shift, and compares three ways of reading it.

Code · Python · 28 lines
import numpy as np

rng = np.random.default_rng(1)
H, W, f, cam_h = 120, 160, 150.0, 1.5
v, u = np.mgrid[0:H, 0:W].astype(float)
Z = np.full((H, W), 30.0)                                      # far wall
below = v > H / 2 + 1
Z[below] = np.minimum(30.0, f * cam_h / (v[below] - H / 2))    # ground plane seen by a camera 1.5 m high
box = (np.abs(u - 50) < 15) & (v > 45) & (v < 80)
Z[box] = 5.0                                                   # an object 5 m away

# A "relative" network outputs inverse depth up to an unknown scale and shift, plus some noise.
pred = 0.37 / Z + 0.05 + rng.normal(0, 0.001, Z.shape)

def metrics(Zp, Zt):
    absrel = np.mean(np.abs(Zp - Zt) / Zt)
    rmse = np.sqrt(np.mean((Zp - Zt) ** 2))
    delta1 = np.mean(np.maximum(Zp / Zt, Zt / Zp) < 1.25)
    return f"AbsRel {absrel:6.3f}   RMSE {rmse:6.2f} m   delta<1.25 {100 * delta1:5.1f}%"

Z_raw = 1 / pred                                               # read the output as if it were metric
Z_med = Z_raw * np.median(Z) / np.median(Z_raw)                # median scaling (scale only)
A = np.c_[pred.ravel(), np.ones(pred.size)]                    # least squares for (s, t) in inverse depth
s, t = np.linalg.lstsq(A, (1 / Z).ravel(), rcond=None)[0]
Z_ssi = 1 / (s * pred + t)
print("raw output        ", metrics(Z_raw, Z))
print("median scaling    ", metrics(Z_med, Z))
print("scale+shift (LSQ) ", metrics(Z_ssi, Z), f"  s = {s:.3f}, t = {t:.4f}")

Output

raw output         AbsRel  0.460   RMSE  10.46 m   delta<1.25  10.2%
median scaling     AbsRel  0.785   RMSE   7.24 m   delta<1.25  54.8%
scale+shift (LSQ)  AbsRel  0.043   RMSE   1.84 m   delta<1.25  99.5%   s = 2.699, t = -0.1348

These are toy numbers. Read as metric depth, the output is useless (δ < 1.25 on 10% of the pixels). Median scaling multiplies the depth by one factor; it helps δ but makes AbsRel worse, because no single factor can undo a shift in inverse depth. The least-squares fit finds s=2.699s = 2.699, t=−0.1348t = -0.1348, almost exactly the simulated 1/Z=2.703 y^−0.1351/Z = 2.703\,\hat y - 0.135, and the map becomes nearly perfect. The same prediction can look poor or excellent depending on the alignment, so always check which one a paper used.

Three panels. (a) A camera at the bottom with an image line; dashed rays from the camera pass the edges of a small blue box nearby and continue to the edges of an orange box 2.5 times larger and 2.5 times farther away. (b) Photometric L1 error against a global depth scale from 0.5 to 2: a blue curve for a stereo pair with known baseline has a sharp minimum at 1, and an orange dashed line for video with learned translation is perfectly flat. (c) Predicted inverse depth against true inverse depth: grey raw points lie on a straight line with the wrong slope and offset; green points after least-squares scale and shift lie on the dotted diagonal.
Figure 5 — Scale ambiguity. (a) A box 2.5 times larger and farther gives the same image. (b) A self-supervised photometric loss pins the scale only when the baseline is known; if the camera translation is also learned, every scale fits equally well. (c) An affine-invariant prediction becomes accurate after a least-squares scale and shift in inverse depth (data from the snippet).

From relative to metric foundation models

Modern monocular models use standard dense-prediction networks; the progress came from training data and from how scale is handled. On the relative side, DPT [20] brought a Vision Transformer encoder to dense prediction, Depth Anything [19], [21] scaled training with tens of millions of pseudo-labelled images, and Marigold [22] turned an image diffusion model into a depth estimator. On the metric side, ZoeDepth [23] adds metric heads to a relative model, while Metric3D [24], [25], UniDepth [26] and Depth Pro [27] handle the camera explicitly, by mapping images to a canonical focal length or by estimating the camera from the image. The one idea to remember: relative depth is now robust across many kinds of images, and metric depth becomes usable once the model knows, or can infer, the camera. Dense pointmap models such as DUSt3R and VGGT, which predict geometry and cameras jointly, are covered in DLCV 11, Feed-forward reconstruction.

Optional: Monocular depth foundation models: data and the camera (in the Supplementary notes). What each of these seven models changes, model by model.

Self-supervised depth from stereo pairs and video

Plain version. Instead of ground-truth depth, use a second view as the teacher. If the network’s depth were right, you could warp the second view onto the first and get the same picture. The difference between the warped and the real image is the training signal.

View synthesis as supervision

Garg et al. [28] trained a monocular network on stereo pairs with a known, small camera displacement: the predicted depth of the left image, together with the known baseline, warps the right image into the left view, and the photometric error of this reconstruction is the loss, with no depth labels at all [28]. Godard et al. [29] added a left–right consistency loss that makes the disparities predicted for the two views agree, which improved quality [29]. Zhou et al. [30] removed the second camera: from unlabelled monocular video they trained a depth network and a pose network together, with view synthesis between neighbouring frames as the only supervision [30].

For a target frame ItI_t and a source frame It′I_{t'}, each target pixel p\mathbf{p} (in homogeneous coordinates) is mapped to the source by

p′∼K Tt→t′ Dt(p) K−1p,\mathbf{p}' \sim K \, T_{t \to t'} \, D_t(\mathbf{p}) \, K^{-1} \mathbf{p},

where KK is the camera intrinsic matrix, Dt(p)D_t(\mathbf{p}) is the predicted depth of the pixel, Tt→t′T_{t \to t'} is the predicted relative camera pose (rotation and translation) and ∼\sim means equality up to the homogeneous scale. Sampling It′I_{t'} at p′\mathbf{p}' with a bilinear sampler (DLCV 09A.3) gives the synthesized image It′→tI_{t' \to t}; gradients flow through it into both networks.

Monodepth2

Monodepth2 by Godard et al. [31] showed that a simple model with careful design choices beats more complex ones. Its photometric error between two images combines SSIM (DLCV 09, full-reference metrics) and L1:

pe(Ia,Ib)=α2(1−SSIM(Ia,Ib))+(1−α)∥Ia−Ib∥1,α=0.85,\mathrm{pe}(I_a, I_b) = \frac{\alpha}{2} \bigl(1 - \mathrm{SSIM}(I_a, I_b)\bigr) + (1 - \alpha) \lVert I_a - I_b \rVert_1, \qquad \alpha = 0.85,

where Ia,IbI_a, I_b are the target image and a synthesized image, compared per pixel [31], [32]. Three ideas make it robust [31]:

  1. Per-pixel minimum reprojection. With several source frames (previous and next), take the minimum error over sources instead of the average, Lp=min⁡t′pe(It,It′→t)\mathcal{L}_p = \min_{t'} \mathrm{pe}(I_t, I_{t' \to t}), because a pixel occluded in one source is usually visible in the other.
  2. Auto-masking. Ignore pixels whose error without warping is already lower than with warping. This removes a static camera, and objects that move with the camera, which would otherwise teach the network “infinite depth” for cars driving ahead.
  3. Full-resolution multi-scale loss. Upsample lower-resolution depth predictions to the input size before computing the error, which reduces artefacts.

An edge-aware smoothness term, weighted by λ=0.001\lambda = 0.001, discourages depth changes where the image itself is smooth [31].

Scale, again

Trained from monocular video, a depth network never sees a metric quantity: depth and translation can be scaled together without changing p′\mathbf{p}'. Figure 5b shows it on the synthetic stereo scene: with the baseline fixed, the photometric loss has a sharp minimum at the true scale; if the translation scales with the depth, the loss is flat. This is why Monodepth2 reports monocular-trained results after per-image median ground-truth scaling, following SfMLearner, but not for stereo-trained ones, “as scale can be inferred from the known camera baseline” [31]. The other limits are those of matching itself: textureless and reflective surfaces, moving objects and occlusions.

Optical flow as a learning problem

Plain version. Flow is stereo without the rulebook: the match can move in any direction, by a fraction of a pixel or by a hundred pixels, and objects move independently. Classical methods solved an energy for every image pair; deep methods learn to match and refine from synthetic videos.

From energies to networks

The classical methods start from brightness constancy, I1(p)=I2(p+F(p))I_1(\mathbf{p}) = I_2(\mathbf{p} + \mathbf{F}(\mathbf{p})), which gives one equation for two unknowns per pixel, and add an assumption to resolve it: global smoothness of the flow field (Horn and Schunck [33]), constant flow in a small window solved by least squares (Lucas and Kanade [34]), or a local polynomial model of the image (Farnebäck [35]). DLCV 04B introduces these ideas and the aperture problem.

Optional: Brightness constancy and the classical flow solvers (in DLCV 04B’s Supplementary notes). The linearized flow equation, the Horn–Schunck energy and the polynomial expansion behind Farnebäck’s method.

Hur and Roth [36] describe the move to deep learning in two families: CNNs used as feature extractors inside classical energy formulations, and end-to-end networks that regress the flow directly. The second family won. Four milestones lead to RAFT:

  • FlowNet [37] regressed flow end to end with a CNN. Lacking real training data, its authors built Flying Chairs, 22,872 image pairs of rendered 3D chair models moved by random affine transformations over photographs, and found that networks trained on it still generalized to Sintel and KITTI [37]. Its FlowNetC variant added a correlation layer that compares the two images’ features at a range of displacements.
  • FlowNet 2.0 [38] stacked several FlowNets, each warping the second image with the current estimate, and showed that the order of the training datasets matters a lot [38].
  • SPyNet [39] used a classical image pyramid: at each level one image is warped by the current flow and a small network predicts an update of less than a pixel [39].
  • PWC-Net [40] combined pyramid, warping and cost volume on learned features, and is 17 times smaller than FlowNet2 [40].

RAFT: all-pairs correlation and iterative updates

Plain version. RAFT compares every pixel of the first image with every pixel of the second, once. Then it starts from zero motion and, like a person adjusting a sketch, repeatedly looks at the match scores near its current guess and nudges the guess, the same small update applied again and again.

RAFT by Teed and Deng [41] has three parts [41]:

  1. Features. A CNN encoder maps both images to features at 1/8 resolution with D=256D = 256 channels; a separate context network encodes the first image only.
  2. All-pairs correlation. The correlation volume holds the dot product of every feature pair,
Cijkl=∑hg(I1)ijh g(I2)klh,C_{ijkl} = \sum_{h} g(I_1)_{ijh} \, g(I_2)_{klh},

where g(I)ijhg(I)_{ijh} is channel hh of the feature at row ii, column jj, and C∈RH×W×H×WC \in \mathbb{R}^{H \times W \times H \times W} at feature resolution. Average pooling of the last two dimensions with kernels 1, 2, 4 and 8 gives a four-level correlation pyramid, which keeps full resolution for the first image and covers large displacements in the coarse levels. 3. Iterative updates. Starting from f0=0\mathbf{f}_0 = \mathbf{0}, a convolutional GRU (gated recurrent unit) repeatedly reads the correlation values in a small neighbourhood (radius r=4r = 4) around the position each pixel currently points to, at all pyramid levels, together with the context features and the current flow, and outputs an update: fk+1=fk+Δf\mathbf{f}_{k+1} = \mathbf{f}_k + \Delta \mathbf{f}. The weights are shared across iterations. A learned convex upsampling turns the 1/8-resolution flow into full resolution.

The loss supervises every iterate with exponentially increasing weight,

L=∑i=1NγN−i ∥fgt−fi∥1,γ=0.8,\mathcal{L} = \sum_{i=1}^{N} \gamma^{N - i} \, \lVert \mathbf{f}_{\text{gt}} - \mathbf{f}_i \rVert_1, \qquad \gamma = 0.8,

where fi\mathbf{f}_i is the upsampled flow after update ii, NN is the number of updates and fgt\mathbf{f}_{\text{gt}} the ground truth [41]. RAFT unrolls 12 updates in training and can run more at test time; the paper evaluates after 32 updates on Sintel and 24 on KITTI [41]. Training follows the standard schedule: Flying Chairs, then FlyingThings3D, then fine-tuning on the target benchmark [41].

Why did this design win? The all-pairs volume avoids a weakness of coarse-to-fine schemes, which can lose a small fast object at a coarse level for good [41]; the shared update behaves like a learned optimizer; and the recipe transferred to stereo [13] and scene flow [42].

Optional: RAFT (Teed and Deng, 2020) (in the Paper guides). How to read RAFT’s ablations and its benchmark tables.

After RAFT

Later models change one part of the recipe. GMFlow [43] and Unimatch [44] replace iterative lookups by global matching with Transformer features: every pixel is compared with all pixels of the other image and the flow is the softmax-weighted average position; Unimatch uses one model for flow, stereo and depth from posed images [44]. FlowFormer [45] encodes RAFT’s cost volume with transformer layers. SEA-RAFT [46] keeps RAFT but adds a mixture-of-Laplace loss, a directly regressed initial flow and rigid-motion pre-training, for speed and better generalization [46].

Code: measuring flow errors

The snippet builds a pair of frames with known flow: the background (the camera test image) rotates by 2° and zooms by 3% about the centre, and a textured patch moves by (14, 6) pixels on top of it. It runs OpenCV’s Farnebäck flow with one and with four pyramid levels and reports the end-point error (EPE, the length of the error vector) and the Fl outlier rate used by KITTI (defined in Evaluation) on different kinds of pixels.

Code · Python · 49 lines
import numpy as np
import cv2
from scipy import ndimage as ndi
from skimage import color, data, transform

I0 = transform.resize(data.camera() / 255.0, (240, 320), anti_aliasing=True)
H, W = I0.shape
rows, cols = np.mgrid[0:H, 0:W].astype(float)
cy, cx = H / 2, W / 2

# Ground-truth motion from frame 0 to frame 1: the background rotates by 2 degrees and zooms by 3 %
# about the image centre; a 50 x 50 textured patch moves by (dx, dy) = (14, 6) pixels.
a, s = np.deg2rad(2.0), 1.03
bx = cx + s * (np.cos(a) * (cols - cx) - np.sin(a) * (rows - cy))     # where each background pixel goes
by = cy + s * (np.sin(a) * (cols - cx) + np.cos(a) * (rows - cy))
flow = np.stack([bx - cols, by - rows], -1)
patch = transform.resize(color.rgb2gray(data.astronaut())[30:230, 150:350], (50, 50), anti_aliasing=True)
obj0 = (rows >= 90) & (rows < 140) & (cols >= 120) & (cols < 170)
flow[obj0] = (14.0, 6.0)

# Render frame 1 by backward mapping (inverse rotation and zoom), then paste the moved patch.
xs = cx + (np.cos(a) * (cols - cx) + np.sin(a) * (rows - cy)) / s
ys = cy + (-np.sin(a) * (cols - cx) + np.cos(a) * (rows - cy)) / s
I1 = ndi.map_coordinates(I0, [ys, xs], order=3, mode="reflect")
I0 = I0.copy(); I0[90:140, 120:170] = patch
I1[96:146, 134:184] = patch
# occluded: background points of frame 0 that land under the patch in frame 1
occ = ~obj0 & (by >= 96) & (by < 146) & (bx >= 134) & (bx < 184)

def evaluate(est, mask):
    epe = np.linalg.norm(est - flow, axis=-1)[mask]
    mag = np.linalg.norm(flow, axis=-1)[mask]
    fl = np.mean((epe > 3) & (epe > 0.05 * mag))       # KITTI outlier: > 3 px and > 5 % of the true motion
    return f"EPE {epe.mean():5.2f} px  Fl {100 * fl:5.1f}%"

u8 = lambda im: (np.clip(im, 0, 1) * 255).round().astype(np.uint8)
inner = np.zeros((H, W), bool); inner[10:-10, 10:-10] = True
grad = ndi.gaussian_filter(np.hypot(ndi.sobel(I0, 0), ndi.sobel(I0, 1)), 4)
bg = inner & ~obj0 & ~occ
flat = grad < 0.05                                     # almost no texture (mostly the sky)
for levels in [1, 4]:
    est = cv2.calcOpticalFlowFarneback(u8(I0), u8(I1), None, pyr_scale=0.5, levels=levels, winsize=15,
                                       iterations=5, poly_n=7, poly_sigma=1.5, flags=0)
    print(f"Farneback, {levels} pyramid level(s):")
    for name, m in [("background, textured", bg & ~flat), ("background, flat", bg & flat),
                    ("moving patch", obj0), ("occluded", occ), ("all (Fl-all)", inner)]:
        print(f"   {name:21s} {evaluate(est, m)}")
print(f"true motion: background up to {np.linalg.norm(flow[~obj0], axis=-1).max():.1f} px, patch "
      f"{np.hypot(14, 6):.1f} px; occluded pixels: {occ.sum()}")

Output

Farneback, 1 pyramid level(s):
   background, textured  EPE  0.87 px  Fl   9.3%
   background, flat      EPE  5.51 px  Fl  90.3%
   moving patch          EPE 18.24 px  Fl 100.0%
   occluded              EPE  5.07 px  Fl  68.0%
   all (Fl-all)          EPE  2.86 px  Fl  35.6%
Farneback, 4 pyramid level(s):
   background, textured  EPE  1.18 px  Fl  11.9%
   background, flat      EPE  5.52 px  Fl  90.5%
   moving patch          EPE  2.30 px  Fl  30.4%
   occluded              EPE 12.20 px  Fl  99.8%
   all (Fl-all)          EPE  2.55 px  Fl  35.2%
true motion: background up to 9.3 px, patch 15.2 px; occluded pixels: 871

These numbers are from this toy pair only. With one level, the patch’s 15 px motion is out of reach and every patch pixel is an outlier; with four levels the patch is found (EPE 2.30 px), the coarse-to-fine idea of SPyNet and PWC-Net in classical form. Two failures remain. In flat regions (mostly the sky) 90% of the pixels are outliers: there is nothing to match. On occluded pixels, the strip the patch covers in frame 1, the pyramid makes things worse (EPE 12.2 px), because the coarse levels drag them along with the patch (Figure 6). Learned methods are trained to handle both cases from data, but neither disappears.

Six panels. (a) Frame 0: the cameraman photo with a pasted face patch. (b) Frame 1: the same scene slightly rotated and zoomed, with the patch moved down and right. (c) The true flow as a smooth colour field with a uniform blue square for the patch, and a colour-wheel key in the corner. (d) The Farneback estimate: similar colours on textured areas but washed-out white in the sky and on the dark coat, and a blob around the patch. (e) The end-point error map: dark on textured background and on the patch, bright in the sky, at the image corners and in a ring around the patch; a cyan outline marks the occluded strip. (f) A red and grey map of outliers covering the sky, the corners and the ring around the patch; the title says 35 percent.
Figure 6 — Farnebäck flow (4 pyramid levels) on the toy pair. Flow is drawn with hue for direction and saturation for length. Errors concentrate in textureless regions, around the moving patch and in the occluded strip (cyan). A 10 px border is not evaluated.

Scene flow in brief

Plain version. Optical flow says how a point moves in the image. Scene flow says how it moves in 3D: closer, farther, left or right, in metres.

Scene flow is the dense 3D motion field of the scene, a term that goes back to Vedula et al. [47]; optical flow is its projection onto the image. RAFT-3D states the task as: given a pair of stereo or RGB-D video frames, estimate the 3D motion of every pixel [42]. With a stereo camera it can be estimated from the disparity at two times plus the optical flow between them. The KITTI 2015 benchmark by Menze and Geiger [48] evaluates exactly these parts: D1 (disparity in the first frame), D2 (disparity in the second frame), Fl (optical flow) and SF, which counts a pixel as wrong if any of the three is wrong [49]. Menze and Geiger modelled the scene as a small number of rigidly moving objects, with each superpixel a 3D plane assigned to one of them [48]; RAFT-3D [42] instead estimates a dense field of rigid 3D motions with RAFT-style iterations and groups pixels softly into rigid objects with learned embeddings [42]. Dynamic reconstruction (DLCV 11) and point tracking (DLCV 07) are its long-range relatives.

Evaluation

Plain version. All three tasks are scored per pixel against ground truth, with an average error and a “percentage of bad pixels”. The details, especially how monocular predictions are aligned and which pixels count, change the numbers more than you would expect.

Depth metrics

The standard set goes back to Eigen et al. [17]. With yiy_i the predicted and yi∗y_i^* the true depth at the nn evaluated pixels,

AbsRel=1n∑i∣yi−yi∗∣yi∗,RMSE=1n∑i(yi−yi∗)2,SqRel=1n∑i(yi−yi∗)2yi∗,δk=1n∣{i:max⁡(yiyi∗,yi∗yi)<1.25k}∣,\begin{aligned} \text{AbsRel} &= \frac{1}{n} \sum_i \frac{|y_i - y_i^*|}{y_i^*}, & \text{RMSE} &= \sqrt{\frac{1}{n} \sum_i (y_i - y_i^*)^2}, \\ \text{SqRel} &= \frac{1}{n} \sum_i \frac{(y_i - y_i^*)^2}{y_i^*}, & \delta_k &= \frac{1}{n} \Bigl| \Bigl\{ i : \max\Bigl(\tfrac{y_i}{y_i^*}, \tfrac{y_i^*}{y_i}\Bigr) \lt 1.25^k \Bigr\} \Bigr|, \end{aligned}

plus RMSE of log⁡y\log y [17]. AbsRel, SqRel and RMSE are errors (lower is better); δ1,δ2,δ3\delta_1, \delta_2, \delta_3 with k=1,2,3k = 1, 2, 3 are accuracies (higher is better). AbsRel weights near and far pixels by their relative error; RMSE is dominated by far pixels; δ1\delta_1 counts pixels within 25% of the truth.

State three conventions with every result: the alignment (none for metric models, per-image median scaling for self-supervised monocular ones [31], least-squares scale and shift for affine-invariant ones [18]); the depth cap (80 m is standard on KITTI [31]); and the split (often that of Eigen et al. on KITTI [17], [31]). The alignment snippet showed how much the first one alone changes.

Stereo metrics

  • EPE (mean absolute disparity error, in pixels).
  • bad-τ: percentage of pixels with a disparity error larger than τ\tau pixels. Middlebury’s Version 3 evaluation reports bad 2.0 by default, on all or only non-occluded pixels [3]; KITTI 2012 reports the percentage of erroneous pixels at thresholds from 2 to 5 pixels, in non-occluded areas (Out-Noc) and in total (Out-All) [50].
  • D1 (KITTI 2015): a pixel’s disparity is correct if its error is below 3 px or below 5% of the true value; D1 is the percentage of outliers, reported over background, foreground or all pixels (D1-bg, D1-fg, D1-all) [49]. The relative clause keeps near objects, with large disparities, from being judged too harshly.

Flow metrics

  • EPE: the mean Euclidean distance between estimated and true flow vectors, 1n∑i∥Fi−Fi∗∥2\frac{1}{n} \sum_i \lVert \mathbf{F}_i - \mathbf{F}^*_i \rVert_2, where Fi\mathbf{F}_i and Fi∗\mathbf{F}^*_i are the estimated and true flow at pixel ii; FlowNet calls it the standard error measure for optical flow and also trains with it [37]. Sintel results are reported as average EPE (often written AEPE).
  • Fl-all (KITTI 2015): the percentage of pixels whose end-point error is at least 3 px and at least 5% of the true flow length, over all pixels with ground truth [49]. The snippet’s evaluate function computes exactly this.

Datasets and benchmarks

DatasetTaskWhat it containsGround truth
KITTI 2012 [51], [50]stereo, flow194 training and 195 test pairs of real road scenessparse, from a laser scanner
KITTI 2015 [48], [49]stereo, flow, scene flow200 training and 200 test scenes with moving carssparse, semi-automatic, with 3D CAD models fitted to moving vehicles
Middlebury 2014 [52]stereo33 high-resolution indoor scenes; 20 used in the Version 3 evaluation (10 training, 10 test)dense, subpixel-accurate
ETH3D [53]stereo, multi-view stereoindoor and outdoor scenes with DSLR images and low-resolution stereo videos; includes a low-resolution two-view benchmarkhigh-precision laser scans
MPI Sintel [54]flow (also stereo, depth)frames of an open-source animated film; “clean” and “final” passes, the final one with motion blur and atmospheric effectsdense, rendered
Flying Chairs [37]flow (training)22,872 synthetic image pairsdense, rendered
FlyingThings3D [10]stereo, flow, scene flow (training)about 25,000 synthetic stereo frames at 960 × 540dense, rendered
NYU Depth v2 [55]monocular depth (indoor)Kinect video of 464 indoor scenes; 1,449 densely labelled RGB-D pairsdense, from the Kinect depth camera, missing values filled in

Sizes and ground-truth types come from the cited pages and papers; the description of Sintel’s two passes is from [37], and KITTI 2015’s semi-automatic ground truth from [49], [48]. The synthetic sets are used for pre-training, the real ones for fine-tuning and testing.

Failure modes

Plain version. Every method in this tutorial relies on one of two things: something to match, or a prior learned from training data. Each failure below breaks one of them.

  • Textureless regions (white walls, sky, road): flat matching costs (Figure 3b), filled by smoothness or learned context, both possibly wrong.
  • Repetitive patterns (fences, tiles): several disparities fit equally well, as the stripes showed.
  • Reflective and transparent surfaces: the matched content is a reflection or the scene behind. Tosi et al. count transparent and reflective objects among the four main open challenges of deep stereo, with domain shift, over-smoothing and asymmetric stereo pairs [9].
  • Occlusions: no correct match exists; they need detection (left–right check) and filling, and in flow they are dragged along with neighbouring motion.
  • Thin structures and depth edges: windows fatten foreground objects, smoothness erases poles and wires, and a soft argmin over a two-peaked cost lands between surfaces.
  • Large motion of small objects: coarse-to-fine schemes can lose them; RAFT’s all-pairs volume targets this [41].
  • Domain gap: models pre-trained on synthetic data and fine-tuned on one benchmark degrade on new cameras, weather and scenes [9], [36].
  • Scale and the camera (monocular): relative depth has no scale, and metric depth fails when the focal length is unknown or the image has been cropped unexpectedly.
  • Moving objects in self-supervision break the static-scene assumption; auto-masking handles only the simplest cases [31]. Rajapaksha et al. add thin, small and distant objects to the open gaps of monocular depth [56].

Modern view

What the surveys say

Five reviews cover the ground of this tutorial. Laga et al. [8] survey deep stereo (two-view and multi-view) up to about 2019, split into methods that learn parts of the classical pipeline and end-to-end pipelines, with datasets and supervision modes; monocular depth is outside their scope. Poggi et al. [15] stress that learning and stereo help each other: learning improves stereo, and stereo supervises monocular networks; they also cover confidence estimation and domain shift. Tosi et al. [9] review the 2020s: architectures grouped into cost volume aggregation, neural architecture search, iterative optimization, Vision Transformers and Markov random fields, and four open challenges (domain shift, over-smoothing, transparent and reflective objects, asymmetric stereo); in 2024 they noted that a stereo foundation model was still missing. Rajapaksha et al. [56] organize monocular depth by input and output, architecture and degree of supervision. Hur and Roth [36] trace optical flow from energy minimization to CNNs and name domain overfitting and complex training schedules as the main limits.

Optional: Guide to the Modern view surveys (in the Paper guides). Which survey to read for which question, and what each one does not cover.

The state of the art, 2023–2026

  • Foundation models for monocular depth. Large-scale data (Depth Anything [19], [21]), generative priors (Marigold [22]) and explicit camera handling (Metric3D [24], [25], UniDepth [26], Depth Pro [27]) made zero-shot depth from arbitrary photos practical, first relative and now increasingly metric.
  • Iterative matching for stereo and flow. The RAFT recipe of a correlation volume plus recurrent updates dominates both tasks [9]; recent variants add geometry volumes (IGEV-Stereo [14]), global matching with attention (GMFlow, Unimatch [43], [44]) or simpler training (SEA-RAFT [46]).
  • Zero-shot stereo. FoundationStereo [16] answered the missing-foundation-model point of Tosi et al. by combining a million synthetic pairs with monocular priors.
  • Convergence with 3D reconstruction. Feed-forward models that predict pointmaps and cameras (DLCV 11) blur the line between depth estimation, stereo and structure from motion.

Open problems

  1. Metric depth in the wild: unknown, cropped or unusual cameras, and a reliable uncertainty for each prediction.
  2. Non-Lambertian scenes: glass, mirrors and water for both stereo and flow [9].
  3. Sharp boundaries and thin structures without over-smoothing or “flying pixels” between surfaces.
  4. Temporal consistency: per-frame depth flickers in video; flow and depth over long sequences should agree with each other.
  5. Evaluation: sparse or semi-automatic ground truth, differing alignment conventions and benchmark saturation make small differences hard to interpret.

Key takeaways

  • Rectified stereo constrains matches to image rows; flow searches across time; monocular depth relies on learned or geometric assumptions.
  • Depth sensitivity increases as disparity shrinks; calibration, baseline and sub-pixel precision limit usable range.
  • Cost aggregation and regularization fill uncertain regions but can erase thin structures.
  • Distinguish relative from metric depth and state any scale or shift alignment used in evaluation.
  • Photometric supervision depends on visibility and scene assumptions; moving or occluded content needs special handling.
  • Report error units, evaluated pixels, depth caps and splits; comparable model names do not imply comparable protocols.

Exercises

  1. Choose a baseline. A stereo camera with f=1000f = 1000 px must measure depth at 40 m with an error below 1 m, assuming a disparity error of 0.3 px. What is the smallest baseline? What disparity does a point at 40 m then have, and what happens at 80 m?
Hint

From δZ≈Z2δd/(fB)<1\delta Z \approx Z^2 \delta d / (fB) \lt 1: B>1600⋅0.3/1000=0.48B \gt 1600 \cdot 0.3 / 1000 = 0.48 m. The disparity at 40 m is fB/Z=12fB/Z = 12 px. At 80 m the error is four times larger (about 4 m) and the disparity is 6 px.

  1. Window size and fattening. In the block-matching snippet, change win to 3, 7 and 15 and compare bad-1 on the four regions. Why does a large window help in one place and hurt in another?
Hint

In our run, textureless bad-1 fell from 84.4% (3 × 3) to 67.7% (7 × 7) and 39.9% (15 × 15), while the thin pole rose from 15.3% to 17.4% and 74.6%. A large window gathers texture from outside the flat patch, but near depth edges it mixes two surfaces and spreads the foreground disparity into the background (“foreground fattening”), which swallows the pole.

  1. Stripes and the period. The stripes on the wall have a period of 0.15 world units, and the wall there is about 8.6 units away. Estimate their period in pixels (use f=320f = 320 px), then explain why block matching gives wrong disparities there. Which change to the scene would make the stripes matchable, and which change to the algorithm might help?
Hint

The period is about 320⋅0.15/8.6≈5.6320 \cdot 0.15 / 8.6 \approx 5.6 px, smaller than the true disparity there (about 9.3 px), so shifts that differ by one period fit equally well. A wider search does not help. Non-periodic texture would; a smoothness term that prefers the surrounding wall’s disparity reduces the errors (in our run SGBM’s bad-1 on the stripes was 53.3% against 89.6% for block matching) but does not remove them.

  1. Scale-invariant losses by hand. Show that Eigen’s error D(y,y∗)D(y, y^*) does not change when yy is multiplied by a constant c>0c \gt 0. Then show that MiDaS’s least-squares (s,t)(s, t) makes Lssi\mathcal{L}_{\text{ssi}} unchanged under d↦ad+bd \mapsto a d + b for any a≠0a \ne 0 and bb.
Hint

log⁡(cyi)=log⁡c+log⁡yi\log(c y_i) = \log c + \log y_i, and α\alpha absorbs −log⁡c-\log c, so every residual is unchanged. For MiDaS, if (s,t)(s, t) is optimal for dd, then (s/a,t−sb/a)(s/a, t - sb/a) gives the same aligned values for ad+bad + b.

  1. Pyramids and large motion. In the flow snippet, move the patch by (24, 10) instead of (14, 6) and run with 1, 4 and 6 pyramid levels; then move it by only (2, 1). Update the pasted position in frame 1 and the occlusion mask too. When does the pyramid help?
Hint

In our run, at (24, 10) the patch was lost with every setting (EPE about 26 px), and 6 levels gave exactly the same result as 4, so the extra levels added nothing here. At (2, 1) all three settings gave a patch EPE of 0.27 px. A pyramid helps only for motions that its coarse levels can bring within reach of the local model; beyond that, a different search is needed, such as RAFT’s all-pairs correlation.

Supplementary notes

4 supplementary notes, each on a page of its own; the main text does not depend on them.

Open the notes →

Paper guides

3 paper guides, each on a page of its own.

Open the paper guides →

References

  1. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. doi
  2. D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” International Journal of Computer Vision, vol. 47, no. 1–3, pp. 7–42, 2002. doi
  3. D. Scharstein, R. Szeliski and H. Hirschmüller, “Middlebury Stereo Evaluation — Version 3,” official benchmark page (bad 2.0 metric, non-occluded and all masks). link · stereo home page
  4. R. Zabih and J. Woodfill, “Non-parametric local transforms for computing visual correspondence,” in Computer Vision — ECCV ‘94 (Lecture Notes in Computer Science), pp. 151–158, Springer, 1994. doi
  5. H. Hirschmüller, “Stereo processing by semiglobal matching and mutual information,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 328–341, 2008. doi
  6. J. Žbontar and Y. LeCun, “Stereo matching by training a convolutional neural network to compare image patches,” Journal of Machine Learning Research, vol. 17, no. 65, pp. 1–32, 2016. JMLR · arXiv
  7. OpenCV contributors, “cv::StereoSGBM class documentation,” OpenCV source, modules/calib3d/include/opencv2/calib3d.hpp (4.x branch). source
  8. H. Laga, L. V. Jospin, F. Boussaid and M. Bennamoun, “A survey on deep learning techniques for stereo-based depth estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 4, pp. 1738–1764, 2022. doi · arXiv
  9. F. Tosi, L. Bartolomei and M. Poggi, “A survey on deep stereo matching in the twenties,” International Journal of Computer Vision, vol. 133, no. 7, pp. 4245–4276, 2025. doi · arXiv
  10. N. Mayer, E. Ilg, P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy and T. Brox, “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in Proc. IEEE CVPR, pp. 4040–4048, 2016. CVF · doi
  11. A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach and A. Bry, “End-to-end learning of geometry and context for deep stereo regression,” in Proc. IEEE ICCV, 2017. CVF
  12. J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in Proc. IEEE/CVF CVPR, 2018. CVF
  13. L. Lipson, Z. Teed and J. Deng, “RAFT-Stereo: Multilevel recurrent field transforms for stereo matching,” in Proc. International Conference on 3D Vision (3DV), pp. 218–227, 2021. doi · arXiv
  14. G. Xu, X. Wang, X. Ding and X. Yang, “Iterative geometry encoding volume for stereo matching,” in Proc. IEEE/CVF CVPR, 2023. CVF
  15. M. Poggi, F. Tosi, K. Batsos, P. Mordohai and S. Mattoccia, “On the synergies between machine learning and binocular stereo for depth estimation from images: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi · arXiv
  16. B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo and S. Birchfield, “FoundationStereo: Zero-shot stereo matching,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
  17. D. Eigen, C. Puhrsch and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in Proc. NeurIPS, 2014. proceedings · arXiv
  18. R. Ranftl, K. Lasinger, D. Hafner, K. Schindler and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 3, pp. 1623–1637, 2022. doi · arXiv
  19. L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng and H. Zhao, “Depth Anything: Unleashing the power of large-scale unlabeled data,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  20. R. Ranftl, A. Bochkovskiy and V. Koltun, “Vision transformers for dense prediction,” in Proc. IEEE/CVF ICCV, 2021. CVF
  21. L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng and H. Zhao, “Depth Anything V2,” in Proc. NeurIPS, 2024. proceedings · arXiv
  22. B. Ke, A. Obukhov, S. Huang, N. Metzger, R. C. Daudt and K. Schindler, “Repurposing diffusion-based image generators for monocular depth estimation,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  23. S. F. Bhat, R. Birkl, D. Wofk, P. Wonka and M. Müller, “ZoeDepth: Zero-shot transfer by combining relative and metric depth,” arXiv:2302.12288, 2023. arXiv
  24. W. Yin, C. Zhang, H. Chen, Z. Cai, G. Yu, K. Wang, X. Chen and C. Shen, “Metric3D: Towards zero-shot metric 3D prediction from a single image,” in Proc. IEEE/CVF ICCV, 2023. CVF · arXiv
  25. M. Hu, W. Yin, C. Zhang, Z. Cai, X. Long, K. Wang, H. Chen, G. Yu, C. Shen and S. Shen, “Metric3D v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 10579–10596, 2024. doi · arXiv
  26. L. Piccinelli, Y.-H. Yang, C. Sakaridis, M. Segu, S. Li, L. Van Gool and F. Yu, “UniDepth: Universal monocular metric depth estimation,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  27. A. Bochkovskii, A. Delaunoy, H. Germain, M. Santos, Y. Zhou, S. R. Richter and V. Koltun, “Depth Pro: Sharp monocular metric depth in less than a second,” in Proc. ICLR, 2025. OpenReview · arXiv
  28. R. Garg, V. Kumar B.G., G. Carneiro and I. Reid, “Unsupervised CNN for single view depth estimation: Geometry to the rescue,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 740–756, 2016. doi · arXiv
  29. C. Godard, O. Mac Aodha and G. J. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” in Proc. IEEE CVPR, 2017. CVF
  30. T. Zhou, M. Brown, N. Snavely and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” in Proc. IEEE CVPR, 2017. CVF
  31. C. Godard, O. Mac Aodha, M. Firman and G. J. Brostow, “Digging into self-supervised monocular depth estimation,” in Proc. IEEE/CVF ICCV, 2019. CVF · arXiv
  32. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  33. B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artificial Intelligence, vol. 17, no. 1–3, pp. 185–203, 1981. doi
  34. B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 1981. pdf
  35. G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in Image Analysis (Lecture Notes in Computer Science), pp. 363–370, Springer, 2003. doi
  36. J. Hur and S. Roth, “Optical flow estimation in the deep learning age,” in Modelling Human Motion, N. Noceti, A. Sciutti and F. Rea, Eds., pp. 119–140, Springer, 2020. doi · arXiv
  37. A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. van der Smagt, D. Cremers and T. Brox, “FlowNet: Learning optical flow with convolutional networks,” in Proc. IEEE ICCV, 2015. CVF · arXiv
  38. E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy and T. Brox, “FlowNet 2.0: Evolution of optical flow estimation with deep networks,” in Proc. IEEE CVPR, 2017. CVF
  39. A. Ranjan and M. J. Black, “Optical flow estimation using a spatial pyramid network,” in Proc. IEEE CVPR, 2017. CVF
  40. D. Sun, X. Yang, M.-Y. Liu and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in Proc. IEEE/CVF CVPR, 2018. CVF
  41. Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 402–419, 2020. doi · arXiv
  42. Z. Teed and J. Deng, “RAFT-3D: Scene flow using rigid-motion embeddings,” in Proc. IEEE/CVF CVPR, 2021. CVF · arXiv
  43. H. Xu, J. Zhang, J. Cai, H. Rezatofighi and D. Tao, “GMFlow: Learning optical flow via global matching,” in Proc. IEEE/CVF CVPR, 2022. CVF
  44. H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao and A. Geiger, “Unifying flow, stereo and depth estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 13941–13958, 2023. doi · arXiv
  45. Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai and H. Li, “FlowFormer: A transformer architecture for optical flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 668–685, 2022. doi · arXiv
  46. Y. Wang, L. Lipson and J. Deng, “SEA-RAFT: Simple, efficient, accurate RAFT for optical flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 36–54, 2024. doi · arXiv
  47. S. Vedula, S. Baker, P. Rander, R. Collins and T. Kanade, “Three-dimensional scene flow,” in Proc. IEEE ICCV, vol. 2, pp. 722–729, 1999. doi
  48. M. Menze and A. Geiger, “Object scene flow for autonomous vehicles,” in Proc. IEEE CVPR, pp. 3061–3070, 2015. CVF · doi
  49. A. Geiger, P. Lenz, C. Stiller and R. Urtasun, “Scene flow evaluation 2015 (stereo 2015, flow 2015, scene flow 2015),” The KITTI Vision Benchmark Suite, official benchmark page (dataset size, D1, D2, Fl and SF outlier definitions). link
  50. A. Geiger, P. Lenz, C. Stiller and R. Urtasun, “Stereo evaluation 2012,” The KITTI Vision Benchmark Suite, official benchmark page (dataset size, Out-Noc and Out-All at 2–5 px). link
  51. A. Geiger, P. Lenz and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” in Proc. IEEE CVPR, pp. 3354–3361, 2012. doi
  52. D. Scharstein, H. Hirschmüller, Y. Kitajima, G. Krathwohl, N. Nešić, X. Wang and P. Westling, “High-resolution stereo datasets with subpixel-accurate ground truth,” in Pattern Recognition (Proc. GCPR) (Lecture Notes in Computer Science), pp. 31–42, Springer, 2014. doi · datasets
  53. T. Schöps, J. L. Schönberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys and A. Geiger, “A multi-view stereo benchmark with high-resolution images and multi-camera videos,” in Proc. IEEE CVPR, pp. 2538–2547, 2017. CVF · datasets
  54. D. J. Butler, J. Wulff, G. B. Stanley and M. J. Black, “A naturalistic open source movie for optical flow evaluation,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 611–625, 2012. doi · project page
  55. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 746–760, 2012. doi · dataset page
  56. U. Rajapaksha, F. Sohel, H. Laga, D. Diepeveen and M. Bennamoun, “Deep learning-based depth estimation methods from monocular image and videos: A comprehensive survey,” ACM Computing Surveys, vol. 56, no. 12, 2024. doi · arXiv

Check your understanding

This tutorial has 5 exercises. Can you solve them without opening the hints?

Go to the exercises ↑

Next DLCV 12 · Image Generation: From GANs to Text-to-Image Systems · 36 min