Unifying Synergies between Self-supervised Learning and Dynamic Computation
Tarun Krishna (DCU),* Ayush K. Rai (Dublin City University), Alexandru F Drimbarean (Xperi), Eric Arazo (Insight Centre for Data Analytics (DCU)), Paul Albert (Insight Centre for Data Analytics (DCU)), Alan Smeaton (Insight Centre for Data Analytics, Dublin City University), Kevin McGuinness (DCU), Noel O Connor (Home)
The 34th British Machine Vision Conference
Abstract
Computationally expensive training strategies make self-supervised learning (SSL) impractical for resource constrained industrial settings. Techniques like knowledge distillation (KD), dynamic computation (DC), and pruning are often used to obtain a lightweight model, which usually involves multiple epochs of fine-tuning (or distilling steps) of a large pre-trained model, making it more computationally challenging. In this work we present a novel perspective on the interplay between SSL and DC paradigms. In particular, we show that it is feasible to simultaneously learn a dense and gated sub-network from scratch in a SSL setting without any additional fine-tuning or pruning steps. The co-evolution during pre-training of both dense and gated encoder offers a good accuracy-efficiency trade-off and therefore yields a generic and multi-purpose architecture for application specific industrial settings. Extensive experiments on several image classification benchmarks including CIFAR-10, STL-10, CIFAR-100, and ImageNet-100, demonstrate that the proposed training strategy provides a dense and corresponding gated sub-network that achieves comparable (on-par) performance compared with the vanilla self-supervised setting, but at a significant reduction in computation in terms of FLOPs, under a range of target budgets (td ).Video
Citation
@inproceedings{Krishna_2023_BMVC,author = {Tarun Krishna and Ayush K. Rai and Alexandru F Drimbarean and Eric Arazo and Paul Albert and Alan Smeaton and Kevin McGuinness and Noel O Connor},title = {Unifying Synergies between Self-supervised Learning and Dynamic Computation},booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},publisher = {BMVA},year = {2023},url = {https://papers.bmvc2023.org/0394.pdf}}Copyright © 2023The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised byThe British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).