Repository files navigation

Dehao Yuan , Cornelia Fermüller , Tahseen Rabbani , Furong Huang , Yiannis Aloimonos  

ICML2024     [Paper]    [Video Talk]

• [Dec 8, 2024] We did an extension project of VecKM. We apply VecKM to event camera inputs and predict the motion field. Check outVecKM_flow.

VecKM is a generic local point cloud encoder, a.k.a. point tokenizer, patch encoder, etc. The API is easily used by the following codes, after installing the package:

git clone https://github.com/dhyuan99/VecKM.gitcd VecKMconda create -n VecKM python=3.13conda activate VecKMpip install --upgrade pip setuptools wheelpython setup.py sdist bdist_wheelpip install .

Seedemo for the codes and data for running the demo:

cd demopython main.py

We provide two implementations of VecKM:ExactVecKM andFastVecKM, corresponding to Eqn. (3) and Eqn. (2) in the paper.

• ExactVecKM computes an accurate local point cloud encoding, in a slower manner. Suitable for tasks where accurate geometry is needed, e.g. normal estimation.
• FastVecKM computes a noisy local point cloud encoding with large point cloud inputs (e.g. size > 50000), in a faster manner. Suitable for tasks where only coarse geometry is needed, e.g. classification.

Both of them

• receive inputs with shape (n, 3) and output (n, d), operations defined atHighlighted Features.
• scalable to point cloud size > 50000 and neighborhood size > 500 on 16GB memory.

The API call is as simple as followed, as shown in./demo/main.py:

fromVecKM.encoderimportExactVecKM,FastVecKMpts=torch.tensor(pts).float()pts=pts.cuda()# pts has shape (n, 3).vkm=ExactVecKM(pt_dim=3,enc_dim=384,radius=0.1)vkm=vkm.cuda()G=vkm(pts)# G has shape (n, 384).vkm=FastVecKM(pt_dim=3,enc_dim=384,radius=0.1)vkm=vkm.cuda()G=vkm(pts)# G has shape (n, 384).

VecKM.visualize.check_vkm_quality_3d provides a visual check of the encoding quality (Figure 5 in the paper). It will

• visualize the local point cloud aroundpts[which_pt], with radius specified invkm.radius.
• visualize the reconstructed point cloud distribution from the VecKM encoding.

After executing the script

fromVecKM.visualizeimportcheck_vkm_quality_3dcheck_vkm_quality_3d(vkm,pts,which_pt=0)

It will generate a gif showing the 3d visualization. If the distribution aligns with the point cloud, it means the encoding quality is good. Checkdemo for more details.

Local point cloud encoding is usually followed by point-wise regression, classification, regression, etc., as shown below.

If you find it helpful, please consider citing our papers:

@InProceedings{pmlr-v235-yuan24b,  title =  {A Linear Time and Space Local Point Cloud Geometry Encoder via Vectorized Kernel Mixture ({V}ec{KM})},  author =       {Yuan, Dehao and Fermuller, Cornelia and Rabbani, Tahseen and Huang, Furong and Aloimonos, Yiannis},  booktitle =  {Proceedings of the 41st International Conference on Machine Learning},  pages =  {57871--57886},  year =  {2024},  editor =  {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix},  volume =  {235},  series =  {Proceedings of Machine Learning Research},  month =  {21--27 Jul},  publisher =    {PMLR},  pdf =  {https://raw.githubusercontent.com/mlresearch/v235/main/assets/yuan24b/yuan24b.pdf},  url =  {https://proceedings.mlr.press/v235/yuan24b.html},}