DynoSurf

This repository includes the source code of the paper DynoSurf: Neural Deformation-based Temporally Consistent Dynamic Surface Reconstruction (ECCV 2024).

Authors: Yuxin Yao, Siyu Ren, Junhui Hou, Zhi Deng, Juyong Zhang, Wenping Wang.

Project Page | Paper | Data

TODO

• Release the code.
• Release preprocessed data.

Install all dependencies

git clone https://github.com/yaoyx689/DynoSurf.git
cd DynoSurf 
conda create -n dynosurf python=3.9 
conda activate dynosurf

# install pytorch (https://pytorch.org/)
pip install torch==1.13.0+cu116 torchvision==0.14.0+cu116 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu116

# install pytorch3d (https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md)
pip install "git+https://github.com/facebookresearch/pytorch3d.git"

# install kaolin (https://kaolin.readthedocs.io/en/latest/notes/installation.html)
pip install kaolin==0.14.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.13.0_cu116.html

# install other dependencies
pip install -r requirements.txt 

Dependencies (click to expand)

• numpy=1.24.4
• torch=1.13.0+cu116
• openmesh=1.2.1
• pytorch3d=0.7.4
• configargparse=1.7
• point-cloud-utils=0.30.4
• pymcubes=0.1.4
• pykdtree=1.3.11
• pymeshlab=2022.2.post4
• tqdm=4.62.3
• kaolin=0.14.0
• pyvista=0.38.5
• tetgen=0.6.2
• open3d=0.17.0
• matplotlib=3.8.2
• tensorboard
• pymeshfix==0.16.2

Quick start

cd scripts
python run.py 

Generate data for training

1. If the input point cloud sequence does not provide normals, put the folder of them under data_source/raw_data/, compute the normals for them and write them into data_source/[foldername]. If the normals are already provided, skip this step.

   cd process_data 
   python process_raw_data.py [folder_name]
   

2. Write the input data into a format that can be read by the code.

   python gene_json.py [folder_name]
   

3. Execute training. Change the folder_name in scripts/run.sh and run

   cd scripts 
   python run.py 
   

Parameter adjustment

• Resolution: If you want to adjust the resolution of the reconstructed surface, you can adjust the parameter tet_grid_volume (around $10^{-8}$ ~ $10^{-6}$) in confs/base.conf, which controls the resolution of the generated tetrahedron. The smaller the value, the higher the resolution.

• Template Frame: If you want to specify the frame index corresponding to the template surface, please modify the template_idx in confs/base.conf.

Preprocessed data and Evaluation

The data used in the paper to compare with other methods can be downloaded here. It contains 32 sequences from the AMA dataset, 109 sequences from DFAUST dataset and 89 sequences from DT4D dataset. You can find a description of the data format here.

We also show the evaluation tool in evaluation, which you can use eval_geometric.py or eval_geometric_dt4d.py to calculate the chamfer distance, normal consistent and F-score, and use eval_correspondences.py to calculate the correspondences errors.

Citation

If you find our code or paper helps, please consider citing:

@inproceedings{yao2024dynosurf,
  author    = {Yao, Yuxin and Ren, Siyu and Hou, Junhui and Deng, Zhi and Zhang, Juyong and Wang, Wenping},
  title     = {DynoSurf: Neural Deformation-based Temporally Consistent Dynamic Surface Reconstruction},
  booktitle   = {European Conference on Computer Vision},
  year      = {2024},
}

Acknowledgment

Some of the implementary refer to DMTet and TeCH.