__pycache__/

*.py[cod]

<h3align="center">

<ahref="https://arxiv.org/pdf/2412.15109">Arxiv</a> |

<a>Video</a> |

<ahref="https://nimolty.github.io/Seer/">Webpage</a>

</h3>

Relevant checkpoints are available on the[website](https://drive.google.com/drive/folders/1F3IE95z2THAQ_lt3DKUFdRGc86Thsnc7?usp=sharing).

|Model|Checkpoint|

|:------:|:------:|

|CALVIN ABC-D|[Seer](https://drive.google.com/drive/folders/17Gv9snGCkViuhHmzN3eTWlI0tMfGSGT3?usp=sharing) /[Seer Large](https://drive.google.com/drive/folders/1AFabqfDEi69oMo0FTGhEiH2QSRLYBR9r?usp=drive_link)|

|CALVIN ABC-D|[Seer](https://drive.google.com/drive/folders/17Gv9snGCkViuhHmzN3eTWlI0tMfGSGT3?usp=sharing)(Avg.Len. : 3.98)/[Seer Large](https://drive.google.com/drive/folders/1AFabqfDEi69oMo0FTGhEiH2QSRLYBR9r?usp=drive_link) (Avg.Len. : 4.30)|

|Real-World|[Seer (Droid Pre-trained)](https://drive.google.com/drive/folders/1rT8JKLhJGIo97jfYUm2JiFUrogOq-dgJ?usp=drive_link)|

-[x] Release real-world expriment code.

-[x] Release CALVIN ABC-D experiment code (Seer).

-[] Release CALVIN ABC-D experiment code (Seer-Large).

-[x] Release CALVIN ABC-D experiment code (Seer-Large).

-[ ] Release LIBERO-LONG experiment code.

**(1) Env**

**(1)CondaEnv**

conda create-n seer python=3.10

conda activate seer

pip install-r requirements.txt

pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0--index-url https://download.pytorch.org/whl/cu121

**(5) Create a soft link to CALVIN**

cd${YOUR_PATH_TO_SEER}

ln-s$CALVIN_ROOT calvin

*:exclamation:**pretrain.sh, finetune.sh, scratch, eval.sh:**

Please update the following:

***calvin_dataset_path** to the directory where you have stored the CALVIN ABC-D data.

***checkpoint_path** to the parent directory where your experiment checkpoints are saved.

***save_checkpoint_path** to the parent directory where your experiment checkpoints are saved. Recommend to create a```checkpoints``` folder in the project root directory.

***finetune_from_pretrained_ckpt** to the location of your pre-trained checkpoint.

***resume_from_checkpoint** to the location of your fine-tuned checkpoint.

***vit_ckpt_path** to the location of your ViT checkpoint (downloaded from the[website](https://drive.google.com/file/d/1bSsvRI4mDM3Gg51C6xO0l9CbojYw3OEt/view?usp=sharing)).

***vit_checkpoint_path** to the location of your ViT checkpoint (downloaded from the[website](https://drive.google.com/file/d/1bSsvRI4mDM3Gg51C6xO0l9CbojYw3OEt/view?usp=sharing)). Recommend to be stored in```checkpoints/vit_mae/mae_pretrain_vit_base.pth```.

*:exclamation:**networkx:**

Due to compatibility issues between the networkx library in CALVIN and Python 3.10, we provide a compatible version of networkx.zip on the[website](https://drive.google.com/file/d/1z-d1SaI0rXfBtBicw1zPSsP-wE-26oLq/view?usp=sharing). Download and unzip it, then replace the existing networkx library in the following path:

bash scripts/CALVIN_ABC_D/Seer/pretrain.sh

bash scripts/CALVIN_ABC_D/Seer-Large/pretrain.sh

bash scripts/CALVIN_ABC_D/Seer/finetune.sh

bash scripts/CALVIN_ABC_D/Seer-Large/finetune.sh

bash scripts/CALVIN_ABC_D/Seer/eval.sh

bash scripts/CALVIN_ABC_D/Seer/scratch.sh

bash scripts/CALVIN_ABC_D/Seer-Large/scratch.sh

bash scripts/CALVIN_ABC_D/Seer/scratch.sh

bash scripts/CALVIN_ABC_D/Seer/eval.sh

bash scripts/CALVIN_ABC_D/Seer-Large/eval.sh

We provide code for pre-training on both the DROID and OXE datasets. Users should update thecheckpoint_path to the directory where you want to save the training checkpoints, and modify the root_dir to the location where the preprocessed real data is stored. Additionally, users should configure the SLURM information in the provided scripts.

We provide code for pre-training on both the DROID and OXE datasets. Users should update thesave_checkpoint_path to the directory where you want to save the training checkpoints, and modify the root_dir to the location where the preprocessed real data is stored. Additionally, users should configure the SLURM information in the provided scripts.

Preparation

model=SeerAgent(

finetune_type=args.finetune_type,

clip_device=device_id,

checkpoint_path=args.vit_ckpt_path,

vit_checkpoint_path=args.vit_checkpoint_path,

sequence_length=args.sequence_length,

num_resampler_query=args.num_resampler_query,

num_obs_token_per_image=args.num_obs_token_per_image,

action_pred_steps=args.action_pred_steps,

obs_pred=args.obs_pred,

atten_only_obs=args.atten_only_obs,

attn_robot_proprio_state=args.attn_robot_proprio_state,

atten_goal=args.atten_goal,

atten_goal_state=args.atten_goal_state,

mask_l_obs_ratio=args.mask_l_obs_ratio,

defgenerate_attention_mask(K,num_A,num_B,atten_goal,atten_goal_state,

atten_only_obs,

attn_robot_proprio_state,

mask_l_obs_ratio,

num_obs_token,action_pred_steps):

attention_mask[start_index+num_A+num_obs_token:start_index+num_A+num_obs_token+action_pred_steps]=-float('inf')

attention_mask[start_index+num_A+num_obs_token:start_index+num_A+num_obs_token+action_pred_steps,start_index+2:start_index+num_A]=0.0

attention_mask[start_index+num_A+num_obs_token:start_index+num_A+num_obs_token+action_pred_steps,start_index+num_A:start_index+num_A+num_obs_token]=0.0

ifattn_robot_proprio_state:

attention_mask[start_index+num_A+num_obs_token:start_index+num_A+num_obs_token+action_pred_steps,start_index+1:start_index+2]=0.0

ifmask_l_obs_ratio>0:

count=int(mask_l_obs_ratio* (num_obs_token))

selected_numbers=np.random.choice(range(num_obs_token),size=count,replace=False)

self,

finetune_type,

clip_device,

checkpoint_path,

vit_checkpoint_path,

sequence_length=10,

num_resampler_query=9,

num_obs_token_per_image=10,

obs_pred=False,

atten_only_obs=False,

attn_robot_proprio_state=False,

atten_goal=False,

atten_goal_state=False,

mask_l_obs_ratio=0.0,

self.atten_goal=atten_goal

self.atten_goal_state=atten_goal_state

self.atten_only_obs=atten_only_obs

self.attn_robot_proprio_state=attn_robot_proprio_state

self.mask_l_obs_ratio=mask_l_obs_ratio

self.hidden_dim=hidden_dim

self.phase=phase

assertself.phasein ["pretrain","finetune","evaluate"]

self.gripper_width=gripper_width

self.checkpoint_path=checkpoint_path

self.vit_checkpoint_path=vit_checkpoint_path

self.text_projector=nn.Linear(512,self.hidden_dim)

self.arm_state_encoder=nn.Linear(6,ARM_STATE_FEATURE_DIM)

self.gripper_state_encoder=nn.Linear(2,GRIPPER_STATE_FEATURE_DIM)

self.state_projector=nn.Linear(ARM_STATE_FEATURE_DIM+GRIPPER_STATE_FEATURE_DIM,self.hidden_dim)

atten_goal=self.atten_goal,

atten_goal_state=self.atten_goal_state,

atten_only_obs=self.atten_only_obs,

attn_robot_proprio_state=self.attn_robot_proprio_state,

mask_l_obs_ratio=self.mask_l_obs_ratio,

num_obs_token=this_num_obs_token,

action_pred_steps=self.action_pred_steps),

self.transformer_backbone=GPT2Model(config)

MLP_hidden_dim=self.hidden_dim//2

self.action_decoder=nn.Sequential(

nn.Linear(self.hidden_dim,192),

nn.Linear(self.hidden_dim,MLP_hidden_dim),

nn.ReLU(),

nn.Linear(192,192),

nn.Linear(MLP_hidden_dim,MLP_hidden_dim),

nn.ReLU(),

)

self.arm_action_decoder=nn.Sequential(

nn.Linear(192,6),

nn.Linear(MLP_hidden_dim,6),

torch.nn.Tanh(),

)

self.gripper_action_decoder=nn.Sequential(

nn.Linear(192,1),

nn.Linear(MLP_hidden_dim,1),

torch.nn.Sigmoid(),

)

self.IMAGE_DECODER_hidden_dim=384

self.IMAGE_DECODER_hidden_dim=self.hidden_dim

self.NUM_MASK_TOKEN=int(calvin_input_image_size**2/patch_size/patch_size)# i.e. num_patch

self.PATCH_SIZE=patch_size

self.mask_token=nn.Parameter(torch.zeros(1,1,self.IMAGE_DECODER_hidden_dim))

self.initialize_weights()

checkpoint=torch.load(checkpoint_path,map_location='cpu')

msg=self.vision_encoder.load_state_dict(checkpoint['model'],strict=False)

print(self.vit_checkpoint_path)

vit_checkpoint=torch.load(self.vit_checkpoint_path,map_location='cpu')

self.vision_encoder.load_state_dict(vit_checkpoint['model'],strict=False)

self.clip_model,self.image_processor=clip.load("ViT-B/32",device=clip_device)

ifos.path.exists("checkpoints/clip/ViT-B-32.pt"):

self.clip_model,self.image_processor=clip.load("checkpoints/clip/ViT-B-32.pt",device=clip_device)

else:

self.clip_model,self.image_processor=clip.load("ViT-B/32",device=clip_device)

definitialize_weights(self):

atten_goal=self.atten_goal,

atten_goal_state=self.atten_goal_state,

atten_only_obs=self.atten_only_obs,

attn_robot_proprio_state=self.attn_robot_proprio_state,

mask_l_obs_ratio=self.mask_l_obs_ratio,

num_obs_token=this_num_obs_token,

action_pred_steps=self.action_pred_steps).to(self.device),

self.model=SeerAgent(

finetune_type=self.args.finetune_type,

clip_device=self.device_id,

checkpoint_path=self.args.vit_ckpt_path,

save_checkpoint_path=self.args.vit_checkpoint_path,

sequence_length=self.args.sequence_length,

num_resampler_query=self.args.num_resampler_query,

num_obs_token_per_image=self.args.num_obs_token_per_image,

export GIT_PYTHON_REFRESH=quiet

calvin_dataset_path="calvin/dataset/task_ABC_D"

calvin_conf_path="calvin/calvin_models/conf"

vit_checkpoint_path="checkpoints/vit_mae/mae_pretrain_vit_base.pth"# downloaded from https://drive.google.com/file/d/1bSsvRI4mDM3Gg51C6xO0l9CbojYw3OEt/view?usp=sharing

resume_from_checkpoint="checkpoints/CALVIN_ABC_D/Seer_Large/12.pth"# checkpoint path to be evaluated

IFS='/'read -ra path_parts<<<"$resume_from_checkpoint"

run_name="${path_parts[-2]}"

log_name="${path_parts[-1]}"

log_folder="eval_logs/$run_name"

mkdir -p"$log_folder"

log_file="eval_logs/$run_name/evaluate_$log_name.log"

node=1

node_num=8

torchrun --nnodes=${node} --nproc_per_node=${node_num} --master_port=10211 eval_calvin.py\

--traj_cons \

--rgb_pad 10 \

--gripper_pad 4 \

--gradient_accumulation_steps 1 \

--bf16_module"vision_encoder" \

--vit_checkpoint_path${vit_checkpoint_path} \

--calvin_dataset${calvin_dataset_path} \

--calvin_conf_path${calvin_conf_path} \

--workers 16 \

--lr_scheduler cosine \

--save_every_iter 50000 \

--num_epochs 20 \

--seed 42 \

--batch_size 64 \

--precision fp32 \

--weight_decay 1e-4 \

--num_resampler_query 16 \

--num_obs_token_per_image 16 \

--run_name${run_name} \

--transformer_layers 24 \

--hidden_dim 1024 \

--transformer_heads 16 \

--phase"evaluate" \

--finetune_type"calvin" \

--action_pred_steps 3 \

--sequence_length 10 \

--future_steps 3 \

--window_size 13 \

--obs_pred \

--resume_from_checkpoint${resume_from_checkpoint}| tee${log_file} \

calvin_dataset_path="calvin/dataset/task_ABC_D"

save_checkpoint_path="checkpoints/"

finetune_from_pretrained_ckpt="checkpoints/pretrain_Seer_ptbs512_24layers_16heads_hd1024-Large_calvin_abc_d/9.pth"

vit_checkpoint_path="checkpoints/vit_mae/mae_pretrain_vit_base.pth"# downloaded from https://drive.google.com/file/d/1bSsvRI4mDM3Gg51C6xO0l9CbojYw3OEt/view?usp=sharing

node=8

node_num=8

torchrun --nnodes=${node} --nproc_per_node=${node_num} --master_port=10211 train.py \

--traj_cons \

--rgb_pad 10 \

--gripper_pad 4 \

--gradient_accumulation_steps 1 \

--bf16_module"vision_encoder" \

--vit_checkpoint_path${vit_checkpoint_path} \

--calvin_dataset${calvin_dataset_path} \

--workers 8 \

--lr_scheduler cosine \

--save_every_iter 100000 \

--num_epochs 20 \

--seed 42 \

--batch_size 8 \

--precision fp32 \

--learning_rate 1e-3 \

--warmup_epochs 3 \

--finetune_type"calvin" \

--wandb_project seer \

--weight_decay 1e-4 \

--num_resampler_query 16 \

--num_obs_token_per_image 16 \

--run_name finetune_Seer-Large_calvin_abc_d \

--save_checkpoint_path${save_checkpoint_path} \

--transformer_layers 24 \

--hidden_dim 1024 \

--transformer_heads 16 \

--phase"finetune" \

--action_pred_steps 3 \

--sequence_length 10 \

--future_steps 3 \

--window_size 13 \

--obs_pred \

--loss_image \

--loss_action \

--save_checkpoint \

--report_to_wandb \

--offline \

--finetune_from_pretrained_ckpt${finetune_from_pretrained_ckpt} \