DLCV 07 · Why do identities swap when objects meet?
AdvancedDLCV 07 · Object Tracking: From Single-Object to Multi-ObjectImage lab
Description
Track three coloured blocks through 30 synthetic frames with a Kalman filter, a cost matrix and the Hungarian algorithm, and watch two identities swap when only box overlap is used.
Try this: At the default setting (λ = 1, box overlap only) the orange and blue walkers swap tracks at frame 15, right after they meet: 2 identity switches, and 30 of the 90 object-frames carry the wrong id. Lower λ in steps of 0.05: the swap disappears between λ = 0.9 and 0.8, and at λ = 0.5 the mean IoU of the tracks with their own objects rises from 0.582 to 0.850. Now set the jitter to 0: overlap alone still swaps, so the turn and not the noise causes it. Finally set the jitter to 5 pixels: even λ = 0.5 gets 4 identity switches. At frame 13 the colour cost 1 − cos of track #1 is 0.47 for the blue walker and 0.61 for its own orange walker. Why is colour less reliable now?
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Appearance. Both walkers are 30 × 40 pixels, so size carries no information. Overlap with a straight-line guess is the cue that fails: after the turn, the blue track’s predicted box overlaps the orange walker more (1 − IoU = 0.36) than the blue walker (0.50), and at frame 15 the Hungarian algorithm swaps the two tracks (2 identity switches). The colour cost 1 − cos is 0.02 for the blue track with the blue walker and 0.19 with the orange one; mixing it in at λ = 0.5 gives 0 switches.
Lab · DLCV 07 · Why do identities swap when objects meet?
Loading image…
Try this: At the default setting (λ = 1, box overlap only) the orange and blue walkers swap tracks at frame 15, right after they meet: 2 identity switches, and 30 of the 90 object-frames carry the wrong id. Lower λ in steps of 0.05: the swap disappears between λ = 0.9 and 0.8, and at λ = 0.5 the mean IoU of the tracks with their own objects rises from 0.582 to 0.850. Now set the jitter to 0: overlap alone still swaps, so the turn and not the noise causes it. Finally set the jitter to 5 pixels: even λ = 0.5 gets 4 identity switches. At frame 13 the colour cost 1 − cos of track #1 is 0.47 for the blue walker and 0.61 for its own orange walker. Why is colour less reliable now?
Synthetic and tiny on purpose: everything is drawn by code, no photographs. Scene: three 30 × 40 pixel person-shaped blocks in a 160 × 128 frame over 30 frames (orange, blue and green shirts, each with its own trouser colour) plus Gaussian pixel noise of 0.03. The orange and blue walkers approach each other along one line at 4 pixels per frame, stand at the same x at frame 14 (the blue one 8 pixels lower and in front) and turn back; the green one walks along the bottom. The scene uses meet-and-turn rather than a pass-through because a pass-through at constant velocity is kept apart by motion alone and swaps only by chance under noise, whereas after a turn the straight-line guess points at the wrong walker (the case the chapter calls the bounce). The tracker follows the chapter: a Kalman filter on [cx, cy, w, h, vx, vy] (q = 1, r = 4), C = λ(1 − IoU) + (1 − λ)(1 − m·e) where e is the unit-length colour histogram (4 bins per channel) of the pixels inside the detection box and m a track memory updated with α = 0.9, and the Hungarian algorithm. Simplified: every frame has exactly one detection per object (no misses, false alarms, gating, birth or death); detections arrive in a rotating order and carry their true object only for scoring; the jitter is a fixed field of random numbers (seed 16) scaled by the slider, so it never re-rolls. Metrics: an identity switch is an object matched to a different track id than in the previous frame (the chapter’s definition), so two swapped tracks count 2; the mean IoU compares each track’s filtered box with the true box of the object the track started on; MOTA = 1 − (FN + FP + IDSW) / GT with FN = FP = 0 and GT = 90 object-frames, so two switches cost only 0.022 although the ids stay wrong for 15 frames. One jitter draw is one clip: over 200 other draws at σ = 2.5, overlap alone switches in 164 and λ = 0.5 in 12; at σ = 5, in 165 and 90. A NumPy and SciPy re-implementation reproduces every number.
Method and image sources
- R. E. Kalman, A New Approach to Linear Filtering and Prediction Problems, Journal of Basic Engineering, 82(1), 35-45, 1960 (the constant-velocity filter)
- H. W. Kuhn, The Hungarian Method for the Assignment Problem, Naval Research Logistics Quarterly, 2(1-2), 83-97, 1955
- A. Bewley, Z. Ge, L. Ott, F. Ramos and B. Upcroft, Simple Online and Realtime Tracking, IEEE ICIP, 2016 (Kalman prediction, IoU cost, Hungarian matching)
- N. Wojke, A. Bewley and D. Paulus, Simple Online and Realtime Tracking with a Deep Association Metric, IEEE ICIP, 2017 (adding an appearance distance to the motion cost)
- K. Bernardin and R. Stiefelhagen, Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics, EURASIP Journal on Image and Video Processing, 2008 (MOTA and identity switches)
- SciPy documentation, scipy.optimize.linear_sum_assignment (the reference used to test the Hungarian implementation)