MOSSE:在傅立葉域學習的相關濾波器
選讀・DLCV 07・物件追蹤:從單一物件到多物件 的補充資料(第 2/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
相關濾波器是一張在搜尋視窗上滑動的模板,與影像最吻合的位置得分最高。MOSSE [17] 學出來的模板,會讓得分圖在目標上形成單一尖銳的峰、其餘位置幾乎平坦,而且用 FFT 一次算出所有位置的得分。
令 為第 個預處理後訓練視窗的二維離散傅立葉轉換(DFT), 為期望響應 (目標中心的高斯峰)的 DFT。相關定理把相關運算變成逐元素乘法,所以視窗的響應是 ,其中 是共軛後的濾波器, 是逐元素相乘, 是反 DFT。正文中的損失函數因此可以逐頻率拆開:對每個頻率索引 ,它只是一個複數 的函數,
其中 是複數共軛, 是一個小的正則化常數。分子是輸入與期望輸出之間的交叉頻譜,分母是輸入的能量頻譜。若沒有 ,只要訓練視窗在某個頻率上幾乎沒有能量,除法就會不穩定;論文因此用第一格視窗的多個擾動版本來訓練,並加入 [17]。為了跟上外觀變化,保留兩個移動累加量,學習率為 ,並令 :
其中 是第 格在新位置裁出的視窗的 DFT。論文報告 在實務上效果良好 [17]。峰值旁瓣比(peak-to-sidelobe ratio, PSR) 衡量峰有多尖: 是響應的最大值, 是峰周圍 視窗以外的響應的平均與標準差。論文把約 7 的 PSR 視為目標被遮擋或已丟失的訊號 [17],這時就停止更新。
何時失效。 DFT 把視窗當成無限週期影像的一個週期,所以相關是循環的:濾波器等於同時看過視窗的所有循環位移,左右(上下)邊緣的接縫也會被當成真的邊緣。這就是視窗要先預處理的原因(取對數、正規化成零平均與固定尺度,再乘上讓邊界淡出到零的餘弦窗,依照 [17])。濾波器也只估計平移,而且只看一個固定大小的視窗,所以快速的尺度或旋轉變化會讓它失效。KCF [18] 保留相同的循環矩陣想法,再加上核函數與多通道特徵。
下面的程式用 scikit-image 內建測試照片的一個視窗,以隨機旋轉八次的版本訓練,再在一段合成序列中追蹤它:場景每格移動 (2, 4) 像素(列、行),並加上感測器雜訊。
程式碼 · Python · 50 行
import numpy as np
from scipy import ndimage
from skimage import data
scene = data.camera().astype(float)
S = 64 # window size (pixels)
yy, xx = np.mgrid[:S, :S]
g = np.exp(-((yy - S // 2) ** 2 + (xx - S // 2) ** 2) / (2 * 2.0 ** 2)) # Gaussian peak, sigma = 2
G = np.fft.fft2(g)
cos_win = np.outer(np.hanning(S), np.hanning(S))
def window(frame, cy, cx): # S x S crop centred on (cy, cx)
return frame[cy - S // 2: cy + S // 2, cx - S // 2: cx + S // 2]
def prep(p): # log, zero mean / unit std, cosine window
p = np.log(p + 1.0)
return (p - p.mean()) / (p.std() + 1e-5) * cos_win
def respond(Hs, patch): # response = ifft2(F * H*)
return np.fft.ifft2(np.fft.fft2(prep(patch)) * Hs).real
def psr(resp): # peak-to-sidelobe ratio
iy, ix = np.unravel_index(resp.argmax(), resp.shape)
side = np.ones(resp.shape, bool); side[max(iy - 5, 0): iy + 6, max(ix - 5, 0): ix + 6] = False
return (resp.max() - resp[side].mean()) / resp[side].std()
rng = np.random.default_rng(0)
cy, cx = 160, 250 # target centre in frame 0
A = np.zeros((S, S), complex); B = np.zeros((S, S), complex)
for _ in range(8): # train on 8 small random rotations of the first window
big = scene[cy - S: cy + S, cx - S: cx + S] # 2S x 2S crop: rotate about the target
warped = ndimage.rotate(big, rng.uniform(-5, 5), reshape=False, mode="nearest")
F = np.fft.fft2(prep(window(warped, S, S)))
A += G * F.conj(); B += F * F.conj()
eps, eta = 0.1, 0.125 # regularizer and learning rate
est = np.array([cy, cx])
for t in range(1, 9): # the scene moves by (2, 4) pixels per frame
frame = ndimage.shift(scene, (2 * t, 4 * t), order=1, mode="nearest")
frame = np.clip(frame + rng.normal(0, 8, frame.shape), 0, 255) # sensor noise
resp = respond(A / (B + eps), window(frame, *est))
iy, ix = np.unravel_index(resp.argmax(), resp.shape)
est = est + np.array([iy - S // 2, ix - S // 2]) # peak offset = displacement
print(f"frame {t}: estimate {est} truth {[cy + 2 * t, cx + 4 * t]} PSR {psr(resp):.1f}")
F = np.fft.fft2(prep(window(frame, *est))) # online update at the new position
A = eta * G * F.conj() + (1 - eta) * A
B = eta * F * F.conj() + (1 - eta) * B
other = respond(A / (B + eps), window(scene, 60, 100)) # a window with no target in it
print("PSR on an unrelated window: %.1f" % psr(other))八格之中估計位置都等於真實位置,目標上的 PSR 介於 36.9 與 39.9 之間(第 1 格 37.2,第 8 格 39.9),而在沒有目標的視窗上只有 2.8。這是個玩具實驗:動作只是一張靜態照片的純平移,所以它只驗證「峰的偏移量等於位移」,以及「PSR 能分辨有目標與沒有目標」,並不代表真實追蹤的準確度;那要看上文的基準資料集。
參考文獻
- Shyandram,〈物件追蹤Object Tracking 簡介〉,Medium 文章,2023;2026 年更新。連結
- G. Ciaparrone, F. Luque Sánchez, S. Tabik, L. Troiano, R. Tagliaferri and F. Herrera, “Deep learning in video multi-object tracking: A survey,” Neurocomputing, 2020. arXiv:1907.12740
- W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu and T.-K. Kim, “Multiple object tracking: A literature review,” Artificial Intelligence, vol. 293, 2021 (arXiv 2014). arXiv:1409.7618
- M. Adžemović, “Deep learning-based multi-object tracking: A comprehensive survey from foundations to state-of-the-art,” arXiv:2506.13457, 2025. arXiv
- S. M. Marvasti-Zadeh, L. Cheng, H. Ghanei-Yakhdan and S. Kasaei, “Deep learning for visual tracking: A comprehensive survey,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, 2022. arXiv:1912.00535
- J. Thangavel (Kugarajeevan), T. Kokul, A. Ramanan and S. Fernando, “Transformers in single object tracking: An experimental survey,” IEEE Access, vol. 11, 2023. arXiv:2302.11867
- M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao and S. C. H. Hoi, “Deep learning for person re-identification: A survey and outlook,” IEEE TPAMI, vol. 44, 2022. arXiv:2001.04193
- J. Luiten, A. Ošep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé and B. Leibe, “HOTA: A higher order metric for evaluating multi-object tracking,” International Journal of Computer Vision, 2020. arXiv:2009.07736
- K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: The CLEAR MOT metrics,” EURASIP Journal on Image and Video Processing, 2008. doi
- E. Ristani, F. Solera, R. S. Zou, R. Cucchiara and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” ECCV Workshop on Benchmarking Multi-Target Tracking, 2016. arXiv:1609.01775
- K. Peng, A. Mathur and A. Narayanan, “Mitigating dataset harms requires stewardship: Lessons from 1000 papers,” arXiv:2108.02922, 2021. arXiv
- L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang and Q. Tian, “Scalable person re-identification: A benchmark,” ICCV, 2015. doi
- L. Wei, S. Zhang, W. Gao and Q. Tian, “Person transfer GAN to bridge domain gap for person re-identification,” CVPR, 2018. arXiv:1711.08565
- H. Luo, Y. Gu, X. Liao, S. Lai and W. Jiang, “Bag of tricks and a strong baseline for deep person re-identification,” CVPR Workshops, 2019. arXiv:1903.07071
- M. Wieczorek, B. Rychalska and J. Dąbrowski, “On the unreasonable effectiveness of centroids in image retrieval,” arXiv:2104.13643, 2021. arXiv
- S. Li, L. Sun and Q. Li, “CLIP-ReID: Exploiting vision-language model for image re-identification without concrete text labels,” AAAI, 2023. arXiv:2211.13977
- D. S. Bolme, J. R. Beveridge, B. A. Draper and Y. M. Lui, “Visual object tracking using adaptive correlation filters,” CVPR, 2010. doi
- J. F. Henriques, R. Caseiro, P. Martins and J. Batista, “High-speed tracking with kernelized correlation filters,” IEEE TPAMI, vol. 37, 2015. arXiv:1404.7584
- L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi and P. H. S. Torr, “Fully-convolutional Siamese networks for object tracking,” arXiv:1606.09549, 2016 (ECCV 2016 workshops). arXiv
- B. Li, J. Yan, W. Wu, Z. Zhu and X. Hu, “High performance visual tracking with Siamese region proposal network,” CVPR, 2018. doi
- B. Ye, H. Chang, B. Ma, S. Shan and X. Chen, “Joint feature learning and relation modeling for tracking: A one-stream framework” (OSTrack), ECCV, 2022. arXiv:2203.11991
- X. Chen, H. Peng, D. Wang, H. Lu and H. Hu, “SeqTrack: Sequence to sequence learning for visual object tracking,” CVPR, 2023 (the arXiv entry was later extended as SeqTrackv2). arXiv:2304.14394
- N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma et al., “SAM 2: Segment anything in images and videos,” ICLR, 2025. arXiv:2408.00714
- C.-Y. Yang, H.-W. Huang, W. Chai, Z. Jiang and J.-N. Hwang, “SAMURAI: Adapting segment anything model for zero-shot visual tracking with motion-aware memory,” arXiv:2411.11922, 2024. arXiv
- N. Carion, L. Gustafson, Y.-T. Hu et al., “SAM 3: Segment anything with concepts,” ICLR, 2026. arXiv:2511.16719
- Y. Wu, J. Lim and M.-H. Yang, “Object tracking benchmark,” IEEE TPAMI, vol. 37, 2015. doi
- H. Fan, L. Lin, F. Yang, P. Chu, G. Deng, S. Yu et al., “LaSOT: A high-quality benchmark for large-scale single object tracking,” CVPR, 2019. arXiv:1809.07845
- L. Huang, X. Zhao and K. Huang, “GOT-10k: A large high-diversity benchmark for generic object tracking in the wild,” IEEE TPAMI, vol. 43, 2021. arXiv:1810.11981
- M. Müller, A. Bibi, S. Giancola, S. Al-Subaihi and B. Ghanem, “TrackingNet: A large-scale dataset and benchmark for object tracking in the wild,” arXiv:1803.10794, 2018 (ECCV 2018). arXiv
- R. E. Kalman, “A new approach to linear filtering and prediction problems,” Journal of Basic Engineering, vol. 82, no. 1, pp. 35–45, 1960. doi
- H. W. Kuhn, “The Hungarian method for the assignment problem,” Naval Research Logistics Quarterly, vol. 2, pp. 83–97, 1955. doi
- A. Bewley, Z. Ge, L. Ott, F. Ramos and B. Upcroft, “Simple online and realtime tracking,” ICIP, 2016. arXiv:1602.00763
- N. Wojke, A. Bewley and D. Paulus, “Simple online and realtime tracking with a deep association metric,” ICIP, 2017. doi
- Y. Zhang, P. Sun, Y. Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu and X. Wang, “ByteTrack: Multi-object tracking by associating every detection box,” ECCV, 2022. arXiv:2110.06864
- J. Cao, J. Pang, X. Weng, R. Khirodkar and K. Kitani, “Observation-centric SORT: Rethinking SORT for robust multi-object tracking,” CVPR, 2023. arXiv:2203.14360
- N. Aharon, R. Orfaig and B.-Z. Bobrovsky, “BoT-SORT: Robust associations multi-pedestrian tracking,” arXiv:2206.14651, 2022. arXiv
- Z. Wang, L. Zheng, Y. Liu, Y. Li and S. Wang, “Towards real-time multi-object tracking” (JDE), ECCV, 2020. arXiv:1909.12605
- Y. Zhang, C. Wang, X. Wang, W. Zeng and W. Liu, “FairMOT: On the fairness of detection and re-identification in multiple object tracking,” IJCV, 2021. arXiv:2004.01888
- P. Bergmann, T. Meinhardt and L. Leal-Taixé, “Tracking without bells and whistles” (Tracktor), ICCV, 2019. arXiv:1903.05625
- F. Zeng, B. Dong, Y. Zhang, T. Wang, X. Zhang and Y. Wei, “MOTR: End-to-end multiple-object tracking with transformer,” ECCV, 2022. arXiv:2105.03247
- Y. Zhang, T. Wang and X. Zhang, “MOTRv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors,” CVPR, 2023. arXiv:2211.09791
- R. Gao, J. Qi and L. Wang, “Multiple object tracking as ID prediction” (MOTIP), CVPR, 2025. arXiv:2403.16848
- S. Li, T. Fischer, L. Ke, H. Ding, M. Danelljan and F. Yu, “OVTrack: Open-vocabulary multiple object tracking,” CVPR, 2023. arXiv:2304.08408
- A. Milan, L. Leal-Taixé, I. Reid, S. Roth and K. Schindler, “MOT16: A benchmark for multi-object tracking,” arXiv:1603.00831, 2016 (MOT16/MOT17). arXiv
- P. Dendorfer, H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler and L. Leal-Taixé, “MOT20: A benchmark for multi object tracking in crowded scenes,” arXiv:2003.09003, 2020. arXiv
- P. Sun, J. Cao, Y. Jiang, Z. Yuan, S. Bai, K. Kitani and P. Luo, “DanceTrack: Multi-object tracking in uniform appearance and diverse motion,” CVPR, 2022. arXiv:2111.14690
- Y. Cui, C. Zeng, X. Zhao, Y. Yang, G. Wu and L. Wang, “SportsMOT: A large multi-object tracking dataset in multiple sports scenes,” ICCV, 2023. arXiv:2304.05170
- F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan and T. Darrell, “BDD100K: A diverse driving dataset for heterogeneous multitask learning,” CVPR, 2020. arXiv:1805.04687
- C. Doersch, Y. Yang, M. Vecerik, D. Gokay, A. Gupta, Y. Aytar et al., “TAPIR: Tracking any point with per-frame initialization and temporal refinement,” ICCV, 2023. arXiv:2306.08637
- N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi and C. Rupprecht, “CoTracker: It is better to track together,” ECCV, 2024. arXiv:2307.07635
- N. Karaev, I. Makarov, J. Wang, N. Neverova, A. Vedaldi and C. Rupprecht, “CoTracker3: Simpler and better point tracking by pseudo-labelling real videos,” arXiv:2410.11831, 2024. arXiv
- Z. Tang, S. Wang, D. C. Anastasiu et al., “The 9th AI City Challenge,” ICCV Workshops, 2025. arXiv:2508.13564
- VOT Challenge organizers, “The Visual Object Tracking (VOT) challenge series,” website, 2013–present. votchallenge.net
- R. Jonker and A. Volgenant, “A shortest augmenting path algorithm for dense and sparse linear assignment problems,” Computing, vol. 38, no. 4, pp. 325–340, 1987. doi
- D. F. Crouse, “On implementing 2D rectangular assignment algorithms,” IEEE Transactions on Aerospace and Electronic Systems, vol. 52, no. 4, pp. 1679–1696, 2016. doi