完整搜尋頁面 →

MOSSE:在傅立葉域學習的相關濾波器

選讀・DLCV 07・物件追蹤:從單一物件到多物件 的補充資料(第 2/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 更多 Re-ID 模型:強基線、質心與 CLIP-ReID
  2. MOSSE:在傅立葉域學習的相關濾波器
  3. 匈牙利演算法:最佳的一對一指派
  4. 點追蹤模型:TAPIR 與 CoTracker 系列

論文導讀 · 1

  1. 追蹤綜述論文的細節

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

相關濾波器是一張在搜尋視窗上滑動的模板,與影像最吻合的位置得分最高。MOSSE [17] 學出來的模板,會讓得分圖在目標上形成單一尖銳的峰、其餘位置幾乎平坦,而且用 FFT 一次算出所有位置的得分。

令 FiF_i 為第 ii 個預處理後訓練視窗的二維離散傅立葉轉換(DFT),GiG_i 為期望響應 gig_i(目標中心的高斯峰)的 DFT。相關定理把相關運算變成逐元素乘法,所以視窗的響應是 F−1(F⊙H∗)\mathcal{F}^{-1}(F \odot H^*),其中 H∗H^* 是共軛後的濾波器,⊙\odot 是逐元素相乘,F−1\mathcal{F}^{-1} 是反 DFT。正文中的損失函數因此可以逐頻率拆開:對每個頻率索引 (u,v)(u, v),它只是一個複數 H∗(u,v)H^*(u, v) 的函數,

min⁡H∗(u,v)∑i∣Fi(u,v) H∗(u,v)−Gi(u,v)∣2⇒H∗(u,v)=∑iGi(u,v) Fi∗(u,v)∑iFi(u,v) Fi∗(u,v)+ε,\min_{H^*(u,v)} \sum_i \big\lvert F_i(u,v)\,H^*(u,v) - G_i(u,v) \big\rvert^2 \quad\Rightarrow\quad H^*(u,v) = \frac{\sum_i G_i(u,v)\,F_i^*(u,v)}{\sum_i F_i(u,v)\,F_i^*(u,v) + \varepsilon},

其中 ∗^* 是複數共軛,ε>0\varepsilon > 0 是一個小的正則化常數。分子是輸入與期望輸出之間的交叉頻譜,分母是輸入的能量頻譜。若沒有 ε\varepsilon,只要訓練視窗在某個頻率上幾乎沒有能量,除法就會不穩定;論文因此用第一格視窗的多個擾動版本來訓練,並加入 ε\varepsilon [17]。為了跟上外觀變化,保留兩個移動累加量,學習率為 η∈(0,1)\eta \in (0, 1),並令 Ht∗=At/(Bt+ε)H^*_t = A_t / (B_t + \varepsilon):

At=η G⊙Ft∗+(1−η) At−1,Bt=η Ft⊙Ft∗+(1−η) Bt−1,A_t = \eta\, G \odot F_t^* + (1-\eta)\, A_{t-1}, \qquad B_t = \eta\, F_t \odot F_t^* + (1-\eta)\, B_{t-1},

其中 FtF_t 是第 tt 格在新位置裁出的視窗的 DFT。論文報告 η=0.125\eta = 0.125 在實務上效果良好 [17]。峰值旁瓣比(peak-to-sidelobe ratio, PSR)=(gmax⁡−μsl)/σsl= (g_{\max} - \mu_{\text{sl}})/\sigma_{\text{sl}} 衡量峰有多尖:gmax⁡g_{\max} 是響應的最大值,μsl,σsl\mu_{\text{sl}}, \sigma_{\text{sl}} 是峰周圍 11×1111 \times 11 視窗以外的響應的平均與標準差。論文把約 7 的 PSR 視為目標被遮擋或已丟失的訊號 [17],這時就停止更新。

何時失效。 DFT 把視窗當成無限週期影像的一個週期,所以相關是循環的:濾波器等於同時看過視窗的所有循環位移,左右(上下)邊緣的接縫也會被當成真的邊緣。這就是視窗要先預處理的原因(取對數、正規化成零平均與固定尺度,再乘上讓邊界淡出到零的餘弦窗,依照 [17])。濾波器也只估計平移,而且只看一個固定大小的視窗,所以快速的尺度或旋轉變化會讓它失效。KCF [18] 保留相同的循環矩陣想法,再加上核函數與多通道特徵。

下面的程式用 scikit-image 內建測試照片的一個視窗,以隨機旋轉八次的版本訓練,再在一段合成序列中追蹤它:場景每格移動 (2, 4) 像素(列、行),並加上感測器雜訊。

程式碼 · Python · 50 行
import numpy as np
from scipy import ndimage
from skimage import data

scene = data.camera().astype(float)
S = 64                                              # window size (pixels)
yy, xx = np.mgrid[:S, :S]
g = np.exp(-((yy - S // 2) ** 2 + (xx - S // 2) ** 2) / (2 * 2.0 ** 2))   # Gaussian peak, sigma = 2
G = np.fft.fft2(g)
cos_win = np.outer(np.hanning(S), np.hanning(S))

def window(frame, cy, cx):                          # S x S crop centred on (cy, cx)
    return frame[cy - S // 2: cy + S // 2, cx - S // 2: cx + S // 2]

def prep(p):                                        # log, zero mean / unit std, cosine window
    p = np.log(p + 1.0)
    return (p - p.mean()) / (p.std() + 1e-5) * cos_win

def respond(Hs, patch):                             # response = ifft2(F * H*)
    return np.fft.ifft2(np.fft.fft2(prep(patch)) * Hs).real

def psr(resp):                                      # peak-to-sidelobe ratio
    iy, ix = np.unravel_index(resp.argmax(), resp.shape)
    side = np.ones(resp.shape, bool); side[max(iy - 5, 0): iy + 6, max(ix - 5, 0): ix + 6] = False
    return (resp.max() - resp[side].mean()) / resp[side].std()

rng = np.random.default_rng(0)
cy, cx = 160, 250                                   # target centre in frame 0
A = np.zeros((S, S), complex); B = np.zeros((S, S), complex)
for _ in range(8):                                  # train on 8 small random rotations of the first window
    big = scene[cy - S: cy + S, cx - S: cx + S]      # 2S x 2S crop: rotate about the target
    warped = ndimage.rotate(big, rng.uniform(-5, 5), reshape=False, mode="nearest")
    F = np.fft.fft2(prep(window(warped, S, S)))
    A += G * F.conj(); B += F * F.conj()
eps, eta = 0.1, 0.125                               # regularizer and learning rate

est = np.array([cy, cx])
for t in range(1, 9):                               # the scene moves by (2, 4) pixels per frame
    frame = ndimage.shift(scene, (2 * t, 4 * t), order=1, mode="nearest")
    frame = np.clip(frame + rng.normal(0, 8, frame.shape), 0, 255)   # sensor noise
    resp = respond(A / (B + eps), window(frame, *est))
    iy, ix = np.unravel_index(resp.argmax(), resp.shape)
    est = est + np.array([iy - S // 2, ix - S // 2])             # peak offset = displacement
    print(f"frame {t}: estimate {est}  truth {[cy + 2 * t, cx + 4 * t]}  PSR {psr(resp):.1f}")
    F = np.fft.fft2(prep(window(frame, *est)))                   # online update at the new position
    A = eta * G * F.conj() + (1 - eta) * A
    B = eta * F * F.conj() + (1 - eta) * B

other = respond(A / (B + eps), window(scene, 60, 100))           # a window with no target in it
print("PSR on an unrelated window: %.1f" % psr(other))

八格之中估計位置都等於真實位置,目標上的 PSR 介於 36.9 與 39.9 之間(第 1 格 37.2,第 8 格 39.9),而在沒有目標的視窗上只有 2.8。這是個玩具實驗:動作只是一張靜態照片的純平移,所以它只驗證「峰的偏移量等於位移」,以及「PSR 能分辨有目標與沒有目標」,並不代表真實追蹤的準確度;那要看上文的基準資料集。

參考文獻

  1. Shyandram,〈物件追蹤Object Tracking 簡介〉,Medium 文章,2023;2026 年更新。連結
  2. G. Ciaparrone, F. Luque Sánchez, S. Tabik, L. Troiano, R. Tagliaferri and F. Herrera, “Deep learning in video multi-object tracking: A survey,” Neurocomputing, 2020. arXiv:1907.12740
  3. W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu and T.-K. Kim, “Multiple object tracking: A literature review,” Artificial Intelligence, vol. 293, 2021 (arXiv 2014). arXiv:1409.7618
  4. M. Adžemović, “Deep learning-based multi-object tracking: A comprehensive survey from foundations to state-of-the-art,” arXiv:2506.13457, 2025. arXiv
  5. S. M. Marvasti-Zadeh, L. Cheng, H. Ghanei-Yakhdan and S. Kasaei, “Deep learning for visual tracking: A comprehensive survey,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, 2022. arXiv:1912.00535
  6. J. Thangavel (Kugarajeevan), T. Kokul, A. Ramanan and S. Fernando, “Transformers in single object tracking: An experimental survey,” IEEE Access, vol. 11, 2023. arXiv:2302.11867
  7. M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao and S. C. H. Hoi, “Deep learning for person re-identification: A survey and outlook,” IEEE TPAMI, vol. 44, 2022. arXiv:2001.04193
  8. J. Luiten, A. Ošep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé and B. Leibe, “HOTA: A higher order metric for evaluating multi-object tracking,” International Journal of Computer Vision, 2020. arXiv:2009.07736
  9. K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: The CLEAR MOT metrics,” EURASIP Journal on Image and Video Processing, 2008. doi
  10. E. Ristani, F. Solera, R. S. Zou, R. Cucchiara and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” ECCV Workshop on Benchmarking Multi-Target Tracking, 2016. arXiv:1609.01775
  11. K. Peng, A. Mathur and A. Narayanan, “Mitigating dataset harms requires stewardship: Lessons from 1000 papers,” arXiv:2108.02922, 2021. arXiv
  12. L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang and Q. Tian, “Scalable person re-identification: A benchmark,” ICCV, 2015. doi
  13. L. Wei, S. Zhang, W. Gao and Q. Tian, “Person transfer GAN to bridge domain gap for person re-identification,” CVPR, 2018. arXiv:1711.08565
  14. H. Luo, Y. Gu, X. Liao, S. Lai and W. Jiang, “Bag of tricks and a strong baseline for deep person re-identification,” CVPR Workshops, 2019. arXiv:1903.07071
  15. M. Wieczorek, B. Rychalska and J. Dąbrowski, “On the unreasonable effectiveness of centroids in image retrieval,” arXiv:2104.13643, 2021. arXiv
  16. S. Li, L. Sun and Q. Li, “CLIP-ReID: Exploiting vision-language model for image re-identification without concrete text labels,” AAAI, 2023. arXiv:2211.13977
  17. D. S. Bolme, J. R. Beveridge, B. A. Draper and Y. M. Lui, “Visual object tracking using adaptive correlation filters,” CVPR, 2010. doi
  18. J. F. Henriques, R. Caseiro, P. Martins and J. Batista, “High-speed tracking with kernelized correlation filters,” IEEE TPAMI, vol. 37, 2015. arXiv:1404.7584
  19. L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi and P. H. S. Torr, “Fully-convolutional Siamese networks for object tracking,” arXiv:1606.09549, 2016 (ECCV 2016 workshops). arXiv
  20. B. Li, J. Yan, W. Wu, Z. Zhu and X. Hu, “High performance visual tracking with Siamese region proposal network,” CVPR, 2018. doi
  21. B. Ye, H. Chang, B. Ma, S. Shan and X. Chen, “Joint feature learning and relation modeling for tracking: A one-stream framework” (OSTrack), ECCV, 2022. arXiv:2203.11991
  22. X. Chen, H. Peng, D. Wang, H. Lu and H. Hu, “SeqTrack: Sequence to sequence learning for visual object tracking,” CVPR, 2023 (the arXiv entry was later extended as SeqTrackv2). arXiv:2304.14394
  23. N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma et al., “SAM 2: Segment anything in images and videos,” ICLR, 2025. arXiv:2408.00714
  24. C.-Y. Yang, H.-W. Huang, W. Chai, Z. Jiang and J.-N. Hwang, “SAMURAI: Adapting segment anything model for zero-shot visual tracking with motion-aware memory,” arXiv:2411.11922, 2024. arXiv
  25. N. Carion, L. Gustafson, Y.-T. Hu et al., “SAM 3: Segment anything with concepts,” ICLR, 2026. arXiv:2511.16719
  26. Y. Wu, J. Lim and M.-H. Yang, “Object tracking benchmark,” IEEE TPAMI, vol. 37, 2015. doi
  27. H. Fan, L. Lin, F. Yang, P. Chu, G. Deng, S. Yu et al., “LaSOT: A high-quality benchmark for large-scale single object tracking,” CVPR, 2019. arXiv:1809.07845
  28. L. Huang, X. Zhao and K. Huang, “GOT-10k: A large high-diversity benchmark for generic object tracking in the wild,” IEEE TPAMI, vol. 43, 2021. arXiv:1810.11981
  29. M. Müller, A. Bibi, S. Giancola, S. Al-Subaihi and B. Ghanem, “TrackingNet: A large-scale dataset and benchmark for object tracking in the wild,” arXiv:1803.10794, 2018 (ECCV 2018). arXiv
  30. R. E. Kalman, “A new approach to linear filtering and prediction problems,” Journal of Basic Engineering, vol. 82, no. 1, pp. 35–45, 1960. doi
  31. H. W. Kuhn, “The Hungarian method for the assignment problem,” Naval Research Logistics Quarterly, vol. 2, pp. 83–97, 1955. doi
  32. A. Bewley, Z. Ge, L. Ott, F. Ramos and B. Upcroft, “Simple online and realtime tracking,” ICIP, 2016. arXiv:1602.00763
  33. N. Wojke, A. Bewley and D. Paulus, “Simple online and realtime tracking with a deep association metric,” ICIP, 2017. doi
  34. Y. Zhang, P. Sun, Y. Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu and X. Wang, “ByteTrack: Multi-object tracking by associating every detection box,” ECCV, 2022. arXiv:2110.06864
  35. J. Cao, J. Pang, X. Weng, R. Khirodkar and K. Kitani, “Observation-centric SORT: Rethinking SORT for robust multi-object tracking,” CVPR, 2023. arXiv:2203.14360
  36. N. Aharon, R. Orfaig and B.-Z. Bobrovsky, “BoT-SORT: Robust associations multi-pedestrian tracking,” arXiv:2206.14651, 2022. arXiv
  37. Z. Wang, L. Zheng, Y. Liu, Y. Li and S. Wang, “Towards real-time multi-object tracking” (JDE), ECCV, 2020. arXiv:1909.12605
  38. Y. Zhang, C. Wang, X. Wang, W. Zeng and W. Liu, “FairMOT: On the fairness of detection and re-identification in multiple object tracking,” IJCV, 2021. arXiv:2004.01888
  39. P. Bergmann, T. Meinhardt and L. Leal-Taixé, “Tracking without bells and whistles” (Tracktor), ICCV, 2019. arXiv:1903.05625
  40. F. Zeng, B. Dong, Y. Zhang, T. Wang, X. Zhang and Y. Wei, “MOTR: End-to-end multiple-object tracking with transformer,” ECCV, 2022. arXiv:2105.03247
  41. Y. Zhang, T. Wang and X. Zhang, “MOTRv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors,” CVPR, 2023. arXiv:2211.09791
  42. R. Gao, J. Qi and L. Wang, “Multiple object tracking as ID prediction” (MOTIP), CVPR, 2025. arXiv:2403.16848
  43. S. Li, T. Fischer, L. Ke, H. Ding, M. Danelljan and F. Yu, “OVTrack: Open-vocabulary multiple object tracking,” CVPR, 2023. arXiv:2304.08408
  44. A. Milan, L. Leal-Taixé, I. Reid, S. Roth and K. Schindler, “MOT16: A benchmark for multi-object tracking,” arXiv:1603.00831, 2016 (MOT16/MOT17). arXiv
  45. P. Dendorfer, H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler and L. Leal-Taixé, “MOT20: A benchmark for multi object tracking in crowded scenes,” arXiv:2003.09003, 2020. arXiv
  46. P. Sun, J. Cao, Y. Jiang, Z. Yuan, S. Bai, K. Kitani and P. Luo, “DanceTrack: Multi-object tracking in uniform appearance and diverse motion,” CVPR, 2022. arXiv:2111.14690
  47. Y. Cui, C. Zeng, X. Zhao, Y. Yang, G. Wu and L. Wang, “SportsMOT: A large multi-object tracking dataset in multiple sports scenes,” ICCV, 2023. arXiv:2304.05170
  48. F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan and T. Darrell, “BDD100K: A diverse driving dataset for heterogeneous multitask learning,” CVPR, 2020. arXiv:1805.04687
  49. C. Doersch, Y. Yang, M. Vecerik, D. Gokay, A. Gupta, Y. Aytar et al., “TAPIR: Tracking any point with per-frame initialization and temporal refinement,” ICCV, 2023. arXiv:2306.08637
  50. N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi and C. Rupprecht, “CoTracker: It is better to track together,” ECCV, 2024. arXiv:2307.07635
  51. N. Karaev, I. Makarov, J. Wang, N. Neverova, A. Vedaldi and C. Rupprecht, “CoTracker3: Simpler and better point tracking by pseudo-labelling real videos,” arXiv:2410.11831, 2024. arXiv
  52. Z. Tang, S. Wang, D. C. Anastasiu et al., “The 9th AI City Challenge,” ICCV Workshops, 2025. arXiv:2508.13564
  53. VOT Challenge organizers, “The Visual Object Tracking (VOT) challenge series,” website, 2013–present. votchallenge.net
  54. R. Jonker and A. Volgenant, “A shortest augmenting path algorithm for dense and sparse linear assignment problems,” Computing, vol. 38, no. 4, pp. 325–340, 1987. doi
  55. D. F. Crouse, “On implementing 2D rectangular assignment algorithms,” IEEE Transactions on Aerospace and Electronic Systems, vol. 52, no. 4, pp. 1679–1696, 2016. doi