Repository files navigation

We present SoFar, the first 6-DoF system for spatial reasoning and robotic manipulation.

We introduce the concept ofsemantic orientation, representing the object orientation condition on open vocabulary language.

Zekun Qi *,Wenyao Zhang *,Yufei Ding *,Runpei Dong,Xinqiang Yu,Jingwen Li,Lingyun Xu,Baoyu Li,Xialin He,Guofan Fan,Jiazhao Zhang,Jiawei He,Jiayuan Gu,Xin Jin,Kaisheng Ma,Zhizheng Zhang,He Wang andLi Yi.

• Install
• Dataset
• SoFar
  • Demo
  • Evaluation
• PointSO
  • Pretrain
  • Finetune
• OrienText300K
• Open6DOR V2
• 6-DoF SpatialBench
• TODO

Setup environment:

conda create -n sofar python=3.12 -yconda activate sofargit clone https://github.com/qizekun/SoFar.gitcd SoFarpip install -e.pip install -e segmentation/SAM

Download checkpoints:

mkdir checkpoints&cd checkpoints# Florence-2huggingface-cli download microsoft/Florence-2-base# Segment Anythingwget -c https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth# PointSOwget -c https://huggingface.co/qizekun/PointSO/resolve/main/small.pthwget -c https://huggingface.co/qizekun/PointSO/resolve/main/base_finetune.pth

More detailed installation instructions can be found inINSTALL.md.Note that CPU devices inference for SoFar are also supported, such asMacOS, Windows, etc.

Our method is based on mature VLMs such as Qwen, ChatGPT, Gemini, etc., if you have an OpenAI key, you can obtain the service by setting the OpenAI key. Note that gemini-2.0-flash-exp is comparable and even better than the gpt-4o, especially the Open6DOR task.

export OPENAI_API_KEY=your_openai_key

Qwen-VL-2.5 can already handle embodied brain tasks.If you do not have an OpenAI-API Key, you can achieve comparable performance by loadingQwen:

pip install qwen-vl-utils[decord]==0.0.8 tritonpip install flash-attn --no-build-isolationpython scripts/qwen_demo.py

python scripts/open6dor_demo.py

python scripts/manipulation_demo.py

python scripts/vqa_demo.py

We evaluate SoFar's performance on two tracks in SimplerEnv, and SoFar achieved SOTA performance in all cases. Due to the independent configuration of the environment, we provided detailed evaluation code inSimplerEnv-SOFAR.

Download the refined dataset followingDATASET.md.

# Predict on Open6DOR datasetpython open6dor/open6dor_eval_perception.py# Evaluate the metricspython open6dor/eval_open6dor.py

Note that Open6DOR uses theobserver's perspective, which means it is oriented relative to the robotic arm.This implies that the X-axis and Y-axis of the observer coordinate system are opposite to those of the robotic arm's base coordinate system.This is reflected in the system prompt: in the observer coordinate system, the Y-axis extends from left to right, and the X-axis extends from far to near.

Additionally, for the Open6DOR task, we recommend usingsmall_finetune.pth as the orientation model inpointso.py to achieve better performance.

Open6DOR V2 execution environment & evaluation is available atOpen6DOR-Libero, you can see the readme for more instructions.

Download the refined dataset followingDATASET.md.

python spatialbench/eval_spatialbench.py

The pipeline of PointSO is as follows:

1. We generate high-quality, standardized, upright 3D asset datasetsOrientext300K through filtering and automatic annotating, and produce corresponding semantic orientations.

2. Train PointSO by adding random rotation, single-view interference, and Gaussian noise (set inconfig.yaml) to 3D assets.

3. Inference with PointSO, in the real world, most point cloud data are partial and in free orientations.

The released weights is onHuggingface PointSO, and the code is in theorientation folder.

Download the PointMAE as initialization.

wget https://github.com/Pang-Yatian/Point-MAE/releases/download/main/pretrain.pth -P orientation/

Perpare the OrienText300K dataset followingDATASET.md.

cd orientationsh train_ddp.sh

Perpare the Open6DOR finetuning dataset followingDATASET.md.The dataset is generated from isaac sim with different assets from Open6DOR.Finetune PointSO will significantly improve the performance on Open6DOR rotation track & 6-DoF track.We recommend using the finetuned version of PointSO for the Open6DOR V2 evaluation.

cd orientationsh train_ddp_ft.sh

We obtained the OrienText300K dataset by rendering multi-views of Objaverse and annotating with ChatGPT, including the filtering of Objaverse 1.0, 350K orientation-text pairs, and 8M multi-view images.The complete multi-view data will be uploaded.

In addition, if your work requires filtering 3D data, theattributes.zip we use to filter OrienText300K may be helpful for your work.We provide multi-view annotations for each object in Objaverse across multiple dimensions, removing low-quality, meaningless, noise, and 3D assets containing useless data.

OrienText300K samples, containing various objects and natural text for interaction.

Data open source onHuggingface OrienteText300K.

We also provide the code for rendering multi-views with Blender (version: 4.2.0) inrender_views.py, so that you can reproduce or use it on your own dataset.This rendering code has undergone very complex debugging and testing.We would appreciate it if this code is useful to you and cite our paper.

A challenging and comprehensive benchmark for open-instruction 6-DoF object rearrangement tasks.

We remove the erroneous data from Open6DOR V1 and eliminated parts that required manual judgment to facilitate replication.Open6DOR V2 contains ~4500 tasks for 6-DoF object rearrangement & spatial relationship evaluation.

Data open source onHuggingface Open6DOR V2.

Previous spatial perception LLMs mainly focused on operations of positional relationships, such as left-right, near-far, size, and counting, etc.In actual object manipulation, the orientation of the object is also a very important factor.Therefore, we proposed a new 6-DoF spatial perception benchmark dataset for evaluating the model's reasoning capabilities in position, orientation, and position-orientation relationships.We evaluated existing spatial perception models on this benchmark dataset.

Data open source onHuggingface 6-DoF SpatialBench.

• Release the evaluation code for Simpler-Env for Google Robot & Widow-X.
• Release the inference code with Qwen-VL-2.5.
• Add cpu devices inference support, such as MacOS.
• Release the evaluation code for Open6DOR-Libero.
• Release more version of PointSO. (About 2 week)
• Release the improved version of OrienText300K. (About 1 month)
• Release gradio demo for SoFar & PointSO. (About 1 month)
• Release the Objaverse-XL version dataset & PointSO. (Maybe 2 month or more)

If you have any questions related to the code or the paper, feel free to email Zekun (qizekun@gmail.com).

• Segment Anything
• ReCon
• Florence-2
• LLaVA

If you find SoFar, PointSO, OrienText300K, Open6DOR V2 or 6-DoF SpatialBench helpful for your research, please consider citing the following BibTeX entry.

@article{qi2025sofar,author ={Qi, Zekun and Zhang, Wenyao and Ding, Yufei and Dong, Runpei and Yu, Xinqiang and Li, Jingwen and Xu, Lingyun and Li, Baoyu and He, Xialin and Fan, Guofan and Zhang, Jiazhao and He, Jiawei and Gu, Jiayuan and Jin, Xin and Ma, Kaisheng and Zhang, Zhizheng and Wang, He and Yi, Li},title ={SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation},journal ={arXiv preprint arXiv:2502.13143},year ={2025}}