Movatterモバイル変換


[0]ホーム

URL:


Skip to main content
Cornell University

Monday, May 5: arXiv will be READ ONLY at 9:00AM EST for approximately 30 minutes. We apologize for any inconvenience.

We gratefully acknowledge support from the Simons Foundation,member institutions, and all contributors.Donate
arxiv logo>cs> arXiv:2312.13139
arXiv logo
Cornell University Logo

Computer Science > Robotics

arXiv:2312.13139 (cs)
[Submitted on 20 Dec 2023 (v1), last revised 21 Dec 2023 (this version, v2)]

Title:Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

View PDFHTML (experimental)
Abstract:Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a straightforward GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data after pre-trained on a large-scale video dataset. We perform extensive experiments on the challenging CALVIN benchmark and a real robot. On CALVIN benchmark, our method outperforms state-of-the-art baseline methods and improves the success rate from 88.9% to 94.9%. In the setting of zero-shot unseen scene generalization, GR-1 improves the success rate from 53.3% to 85.4%. In real robot experiments, GR-1 also outperforms baseline methods and shows strong potentials in generalization to unseen scenes and objects. We provide inaugural evidence that a unified GPT-style transformer, augmented with large-scale video generative pre-training, exhibits remarkable generalization to multi-task visual robot manipulation. Project page:this https URL
Comments:Project page:this https URL
Subjects:Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
Cite as:arXiv:2312.13139 [cs.RO]
 (orarXiv:2312.13139v2 [cs.RO] for this version)
 https://doi.org/10.48550/arXiv.2312.13139
arXiv-issued DOI via DataCite

Submission history

From: Hongtao Wu [view email]
[v1] Wed, 20 Dec 2023 16:00:43 UTC (8,668 KB)
[v2] Thu, 21 Dec 2023 05:34:23 UTC (8,668 KB)
Full-text links:

Access Paper:

Current browse context:
cs.RO
Change to browse by:
export BibTeX citation

Bookmark

BibSonomy logoReddit logo

Bibliographic and Citation Tools

Bibliographic Explorer(What is the Explorer?)
Connected Papers(What is Connected Papers?)
scite Smart Citations(What are Smart Citations?)

Code, Data and Media Associated with this Article

CatalyzeX Code Finder for Papers(What is CatalyzeX?)
Hugging Face(What is Huggingface?)
Papers with Code(What is Papers with Code?)

Demos

Hugging Face Spaces(What is Spaces?)

Recommenders and Search Tools

Influence Flower(What are Influence Flowers?)
CORE Recommender(What is CORE?)

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?Learn more about arXivLabs.

Which authors of this paper are endorsers? |Disable MathJax (What is MathJax?)

[8]ページ先頭

©2009-2025 Movatter.jp