Computer Science > Machine Learning
arXiv:2502.18326 (cs)
[Submitted on 17 Feb 2025]
Title:Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks
View a PDF of the paper titled Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks, by Thadd\"aus Wiedemer and 4 other authors
View PDFHTML (experimental)Abstract:We investigate the success conditions for compositional generalization of CLIP models on real-world data through performance prediction. Prior work shows that CLIP requires exponentially more pretraining data for linear performance gains on individual concepts. This sample-inefficient scaling could be mitigated if CLIP systematically understood new inputs as compositions of learned components, allowing rare observation to be mapped to common concepts. To explore CLIP's compositional generalization ability, we filter retrieval corpora for samples with object combinations not present in the pretraining corpus. We show that CLIP's performance on these samples can be accurately predicted from the pretraining frequencies of individual objects. Our findings demonstrate that CLIP learns to disentangle objects observed in its pretraining data and can recompose them straightforwardly. Additionally, we are the first to show how this ability scales with pretraining data. For data curation in practice, our results suggest that balancing object occurrences improves generalization, which should benefit CLIP's efficiency and accuracy without scaling data volume.
| Comments: | NeurIPS 2024 Workshop on Compositional Learning: Perspectives, Methods, and Paths Forward |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2502.18326 [cs.LG] |
| (orarXiv:2502.18326v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2502.18326 arXiv-issued DOI via DataCite |
Full-text links:
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
- Other Formats
View a PDF of the paper titled Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks, by Thadd\"aus Wiedemer and 4 other authors
References & Citations
Bookmark
Bibliographic and Citation Tools
Bibliographic Explorer(What is the Explorer?)
Connected Papers(What is Connected Papers?)
Litmaps(What is Litmaps?)
scite Smart Citations(What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv(What is alphaXiv?)
CatalyzeX Code Finder for Papers(What is CatalyzeX?)
DagsHub(What is DagsHub?)
Gotit.pub(What is GotitPub?)
Hugging Face(What is Huggingface?)
Papers with Code(What is Papers with Code?)
ScienceCast(What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower(What are Influence Flowers?)
CORE Recommender(What is CORE?)
IArxiv Recommender(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community?Learn more about arXivLabs.