Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples’ positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example’s loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to almost constant memory usage.
@inproceedings{gao-etal-2021-scaling, title = "Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup", author = "Gao, Luyu and Zhang, Yunyi and Han, Jiawei and Callan, Jamie", editor = "Rogers, Anna and Calixto, Iacer and Vuli{\'c}, Ivan and Saphra, Naomi and Kassner, Nora and Camburu, Oana-Maria and Bansal, Trapit and Shwartz, Vered", booktitle = "Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)", month = aug, year = "2021", address = "Online", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2021.repl4nlp-1.31/", doi = "10.18653/v1/2021.repl4nlp-1.31", pages = "316--321", abstract = "Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples' positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example`s loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to almost constant memory usage."}
<?xml version="1.0" encoding="UTF-8"?><modsCollection xmlns="http://www.loc.gov/mods/v3"><mods ID="gao-etal-2021-scaling"> <titleInfo> <title>Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup</title> </titleInfo> <name type="personal"> <namePart type="given">Luyu</namePart> <namePart type="family">Gao</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Yunyi</namePart> <namePart type="family">Zhang</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Jiawei</namePart> <namePart type="family">Han</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Jamie</namePart> <namePart type="family">Callan</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <originInfo> <dateIssued>2021-08</dateIssued> </originInfo> <typeOfResource>text</typeOfResource> <relatedItem type="host"> <titleInfo> <title>Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)</title> </titleInfo> <name type="personal"> <namePart type="given">Anna</namePart> <namePart type="family">Rogers</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Iacer</namePart> <namePart type="family">Calixto</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Ivan</namePart> <namePart type="family">Vulić</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Naomi</namePart> <namePart type="family">Saphra</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Nora</namePart> <namePart type="family">Kassner</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Oana-Maria</namePart> <namePart type="family">Camburu</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Trapit</namePart> <namePart type="family">Bansal</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Vered</namePart> <namePart type="family">Shwartz</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <originInfo> <publisher>Association for Computational Linguistics</publisher> <place> <placeTerm type="text">Online</placeTerm> </place> </originInfo> <genre authority="marcgt">conference publication</genre> </relatedItem> <abstract>Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples’ positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example‘s loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to almost constant memory usage.</abstract> <identifier type="citekey">gao-etal-2021-scaling</identifier> <identifier type="doi">10.18653/v1/2021.repl4nlp-1.31</identifier> <location> <url>https://aclanthology.org/2021.repl4nlp-1.31/</url> </location> <part> <date>2021-08</date> <extent unit="page"> <start>316</start> <end>321</end> </extent> </part></mods></modsCollection>
%0 Conference Proceedings%T Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup%A Gao, Luyu%A Zhang, Yunyi%A Han, Jiawei%A Callan, Jamie%Y Rogers, Anna%Y Calixto, Iacer%Y Vulić, Ivan%Y Saphra, Naomi%Y Kassner, Nora%Y Camburu, Oana-Maria%Y Bansal, Trapit%Y Shwartz, Vered%S Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)%D 2021%8 August%I Association for Computational Linguistics%C Online%F gao-etal-2021-scaling%X Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples’ positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example‘s loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to almost constant memory usage.%R 10.18653/v1/2021.repl4nlp-1.31%U https://aclanthology.org/2021.repl4nlp-1.31/%U https://doi.org/10.18653/v1/2021.repl4nlp-1.31%P 316-321