Momentum Contrast for Unsupervised Visual Representation Learning
Abstract
We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.
Cited in this thesis
Frequently Cited Together
- Generalization and parameter estimation in feedforward nets: Some experiments1 chapter
- Fishers' preference for mobile traceability platform: challenges in achieving a 1 chapter
- Backpropagation applied to handwritten zip code recognition1 chapter
- Idiot's Bayes—not so stupid after all?1 chapter
- An analytical and machine learning approach for total mercury and methylmercury 1 chapter
- Scatsimclr: self-supervised contrastive learning with pretext task regularizatio1 chapter
BibTeX
@inproceedings{He2020,
title = {Momentum contrast for unsupervised visual representation learning},
author = {He, Kaiming and Fan, Haoqi and Wu, Yuxin and Xie, Saining and Girshick, Ross},
booktitle = {Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
pages = {9729–9738},
year = {2020},
}