1. Anne Hendricks, L., Wang, O., Shechtman, E., Sivic, J., Darrell, T., Russell, B.: Localizing moments in video with natural language. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5803–5812 (2017)

   Google Scholar 

2. Bagad, P., Tapaswi, M., Snoek, C.G.: Test of time: instilling video-language models with a sense of time. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2503–2516 (2023)

   Google Scholar 

3. Bai, J., et al.: Lat: latent translation with cycle-consistency for video-text retrieval. arXiv preprintarXiv:2207.04858 (2022)

4. Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: a joint video and image encoder for end-to-end retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728–1738 (2021)

   Google Scholar 

5. Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: ICML, vol. 2, p. 4 (2021)

   Google Scholar 

6. Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: a large-scale video benchmark for human activity understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961–970 (2015)

   Google Scholar 

7. Cao, M., Yang, T., Weng, J., Zhang, C., Wang, J., Zou, Y.: LocVTP: video-text pre-training for temporal localization. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, pp. 38–56. Springer, Cham (2022).https://doi.org/10.1007/978-3-031-19809-0_3

   Chapter  Google Scholar 

8. Chen, D., Dolan, W.B.: Collecting highly parallel data for paraphrase evaluation. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 190–200 (2011)

   Google Scholar 

9. Chen, Y., et al.: Vilem: visual-language error modeling for image-text retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11018–11027 (2023)

   Google Scholar 

10. Ego4D Consortium: Egocentric live 4D perception (Ego4D) database: a large-scale first-person video database, supporting research in multi-modal machine perception for daily life activity

    Google Scholar 

11. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprintarXiv:1810.04805 (2018)

13. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprintarXiv:2010.11929 (2020)

14. Freitag, M., Al-Onaizan, Y.: Beam search strategies for neural machine translation. arXiv preprintarXiv:1702.01806 (2017)

15. Fu, T.J., et al.: Violet: end-to-end video-language transformers with masked visual-token modeling. arXiv preprintarXiv:2111.12681 (2021)

16. Gabeur, V., Sun, C., Alahari, K., Schmid, C.: Multi-modal transformer for video retrieval. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 214–229. Springer, Cham (2020).https://doi.org/10.1007/978-3-030-58548-8_13

    Chapter  Google Scholar 

17. Ge, Y., et al.: Bridging video-text retrieval with multiple choice questions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16167–16176 (2022)

    Google Scholar 

18. Ge, Y., et al.: MILES: visual BERT pre-training with injected language semantics for video-text retrieval. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13695, pp. 691–708. Springer, Cham (2022).https://doi.org/10.1007/978-3-031-19833-5_40

    Chapter  Google Scholar 

19. Ging, S., Zolfaghari, M., Pirsiavash, H., Brox, T.: Coot: cooperative hierarchical transformer for video-text representation learning. Adv. Neural. Inf. Process. Syst.33, 22605–22618 (2020)

    Google Scholar 

20. Grauman, K., et al.: Ego4d: around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995–19012 (2022)

    Google Scholar 

21. Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y.: The curious case of neural text degeneration. arXiv preprintarXiv:1904.09751 (2019)

22. Huang, J., Li, Y., Feng, J., Wu, X., Sun, X., Ji, R.: Clover: towards a unified video-language alignment and fusion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14856–14866 (2023)

    Google Scholar 

23. Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., Wu, Y.: Exploring the limits of language modeling. arXiv preprintarXiv:1602.02410 (2016)

24. Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: 2011 International Conference on Computer Vision, pp. 2556–2563. IEEE (2011)

    Google Scholar 

25. Lei, J., Berg, T.L., Bansal, M.: Revealing single frame bias for video-and-language learning. arXiv preprintarXiv:2206.03428 (2022)

26. Lei, J., et al.: Less is more: clipbert for video-and-language learning via sparse sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7331–7341 (2021)

    Google Scholar 

27. Li, D., Li, J., Li, H., Niebles, J.C., Hoi, S.C.: Align and prompt: video-and-language pre-training with entity prompts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4953–4963 (2022)

    Google Scholar 

28. Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprintarXiv:2301.12597 (2023)

29. Li, J., Li, D., Xiong, C., Hoi, S.: Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning, pp. 12888–12900. PMLR (2022)

    Google Scholar 

30. Li, L., Chen, Y.C., Cheng, Y., Gan, Z., Yu, L., Liu, J.: Hero: hierarchical encoder for video+ language omni-representation pre-training. arXiv preprintarXiv:2005.00200 (2020)

31. Li, L., et al.: Lavender: unifying video-language understanding as masked language modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23119–23129 (2023)

    Google Scholar 

32. Li, Y., Min, K., Tripathi, S., Vasconcelos, N.: Svitt: temporal learning of sparse video-text transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18919–18929 (2023)

    Google Scholar 

33. Liu, Y., Albanie, S., Nagrani, A., Zisserman, A.: Use what you have: video retrieval using representations from collaborative experts. arXiv preprintarXiv:1907.13487 (2019)

34. Liu, Z., et al.: Video swin transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3202–3211 (2022)

    Google Scholar 

35. Lu, H., Fei, N., Huo, Y., Gao, Y., Lu, Z., Wen, J.R.: Cots: collaborative two-stream vision-language pre-training model for cross-modal retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15692–15701 (2022)

    Google Scholar 

36. Miech, A., Alayrac, J.B., Smaira, L., Laptev, I., Sivic, J., Zisserman, A.: End-to-end learning of visual representations from uncurated instructional videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9879–9889 (2020)

    Google Scholar 

37. Miech, A., Zhukov, D., Alayrac, J.B., Tapaswi, M., Laptev, I., Sivic, J.: Howto100m: learning a text-video embedding by watching hundred million narrated video clips. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2630–2640 (2019)

    Google Scholar 

38. Oord, A.V.D., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprintarXiv:1807.03748 (2018)

39. Patrick, M., et al.: Support-set bottlenecks for video-text representation learning. arXiv preprintarXiv:2010.02824 (2020)

40. Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)

    Google Scholar 

41. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI Blog1(8), 9 (2019)

    Google Scholar 

42. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

    Google Scholar 

43. Rohrbach, A., Rohrbach, M., Tandon, N., Schiele, B.: A dataset for movie description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3202–3212 (2015)

    Google Scholar 

44. Rouditchenko, A., et al.: Avlnet: learning audio-visual language representations from instructional videos. arXiv preprintarXiv:2006.09199 (2020)

45. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprintarXiv:1910.01108 (2019)

46. Shao, D., Xiong, Y., Zhao, Y., Huang, Q., Qiao, Y., Lin, D.: Find and focus: retrieve and localize video events with natural language queries. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

    Google Scholar 

47. Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556–2565 (2018)

    Google Scholar 

48. Shi, Y., et al.: Learning semantics-grounded vocabulary representation for video-text retrieval. In: Proceedings of the 31st ACM International Conference on Multimedia, pp. 4460–4470 (2023)

    Google Scholar 

49. Soomro, K., Zamir, A.R., Shah, M.: UCF101: a dataset of 101 human actions classes from videos in the wild. arXiv preprintarXiv:1212.0402 (2012)

50. Tewel, Y., Shalev, Y., Schwartz, I., Wolf, L.: Zerocap: zero-shot image-to-text generation for visual-semantic arithmetic. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17918–17928 (2022)

    Google Scholar 

51. Wang, J., et al.: Object-aware video-language pre-training for retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3313–3322 (2022)

    Google Scholar 

52. Wang, J., et al.: All in one: exploring unified video-language pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6598–6608 (2023)

    Google Scholar 

53. Wang, P., et al.: OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In: International Conference on Machine Learning, pp. 23318–23340. PMLR (2022)

    Google Scholar 

54. Wu, W., Luo, H., Fang, B., Wang, J., Ouyang, W.: Cap4video: what can auxiliary captions do for text-video retrieval? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10704–10713 (2023)

    Google Scholar 

55. Wu, X., Gao, C., Lin, Z., Wang, Z., Han, J., Hu, S.: Rap: redundancy-aware video-language pre-training for text-video retrieval. arXiv preprintarXiv:2210.06881 (2022)

56. Xu, H., et al.: Videoclip: contrastive pre-training for zero-shot video-text understanding. arXiv preprintarXiv:2109.14084 (2021)

57. Xu, J., Mei, T., Yao, T., Rui, Y.: MSR-VTT: a large video description dataset for bridging video and language. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5288–5296 (2016)

    Google Scholar 

58. Xu, M., et al.: Boundary-sensitive pre-training for temporal localization in videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7220–7230 (2021)

    Google Scholar 

59. Xue, H., et al.: CLIP-ViP: adapting pre-trained image-text model to video-language representation alignment. arXiv preprintarXiv:2209.06430 (2022)

60. Yan, R., et al.: Video-text pre-training with learned regions for retrieval. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 3100–3108 (2023)

    Google Scholar 

61. Yang, J., Bisk, Y., Gao, J.: Taco: token-aware cascade contrastive learning for video-text alignment. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11562–11572 (2021)

    Google Scholar 

62. Yang, X., et al.: Learning trajectory-word alignments for video-language tasks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2504–2514 (2023)

    Google Scholar 

63. Ye, Q., et al.: Hitea: hierarchical temporal-aware video-language pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15405–15416 (2023)

    Google Scholar 

64. Zhang, G., Ren, J., Gu, J., Tresp, V.: Multi-event video-text retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22113–22123 (2023)

    Google Scholar 

65. Zhang, H., Liu, D., Lv, Z., Su, B., Tao, D.: Exploring temporal concurrency for video-language representation learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15568–15578 (2023)

    Google Scholar 

66. Zhao, Y., Misra, I., Krähenbühl, P., Girdhar, R.: Learning video representations from large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6586–6597 (2023)

    Google Scholar 

67. Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: enhancing vision-language understanding with advanced large language models. arXiv preprintarXiv:2304.10592 (2023)

68. Zhu, L., Yang, Y.: Actbert: learning global-local video-text representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8746–8755 (2020)

    Google Scholar 

Download references