Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Abstract
We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.
Cited in this thesis
Frequently Cited Together
- Generalization and parameter estimation in feedforward nets: Some experiments1 chapter
- Fishers' preference for mobile traceability platform: challenges in achieving a 1 chapter
- Backpropagation applied to handwritten zip code recognition1 chapter
- Idiot's Bayes—not so stupid after all?1 chapter
- An analytical and machine learning approach for total mercury and methylmercury 1 chapter
- Scatsimclr: self-supervised contrastive learning with pretext task regularizatio1 chapter
BibTeX
@article{Grill2020,
title = {Bootstrap your own latent-a new approach to self-supervised learning},
author = {Grill, Jean-Bastien and Strub, Florian and Altche, Florent and Tallec, Corentin and Richemond, Pierre and Buchatskaya, Elena and Doersch, Carl and Avila Pires, Bernardo and Guo, Zhaohan and Gheshlaghi Azar, Mohammad and others},
journal = {Advances in neural information processing systems},
volume = {33},
pages = {21271–21284},
year = {2020},
}