Project for the MVA "Object Recognition and Computer Vision" class (https://www.di.ens.fr/willow/teaching/recvis19/).
Clone the repository
git clone https://github.com/ManifoldFR/recvis-projectThe initial HMR repository uses Python 2.7 and TensorFlow 1.3. We patched HMR to be able to use more recent versions of Python and TensorFlow 1.x (tested on TensorFlow 1.15 and Python 3.7).
We reverse-engineered motion-reconstruction to use with AlphaPose (included in the repository) and Python 3 instead of Openpose.
The patched versions of the repos are included in this repository.
DeepMimic requires a specific format for motion capture files. They can be obtained from BVH (BioVision Hierarchy Animation) files. Converting from BVH to DeepMimic requires the BvhToDeepMimic package
pip install bvhtodeepmimicPut the videos in a directory (by default data/vid) and call
python -m run_alphaposeThen run
python -m refine_hmrThe output will be in the refined directory.
Use https://github.com/BartMoyaers/BvhToDeepMimic to convert the BVH files to DeepMimic-formatted files.
We repurposed MoCap conversion files from the PyBullet reimplementation of DeepMimic (credit to Erwin Coumans and Yihang Yin), inverse_kinematics.py and transformation.py. We wrote a wrapper for this code that you can modify and call as
python ik_hmr_deepmimic.py
For information, the HMR output joints are as follow (from a comment inside of the HMR source):
- Right ankle
- Right knee
- Right hip
- Left hip
- Left knee
- Left ankle
- Right wrist
- Right elbow
- Right shoulder
- Left shoulder
- Left elbow
- Left wrist
- Neck
- Head top
- nose
- left_eye
- right_eye
- left_ear
- right_ear
- SFV: Reinforcement Learning of Physical Skills from Videos, Peng, Kanazawa, Malik, Abbeel and Levine. In ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2018). https://xbpeng.github.io/projects/SFV/2018_TOG_SFV.pdf | GitHub project page GitHub code repo
- DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills: https://xbpeng.github.io/projects/DeepMimic/index.html
- Estimating 3D Motion and Forces of Person-Object Interactionsfrom Monocular Video. https://arxiv.org/pdf/1904.02683.pdf | Project page
- End-to-end Recovery of Human Shape and Pose: https://arxiv.org/pdf/1712.06584.pdf | GitHub repo
- RMPE: Regional Multi-person Pose Estimation: https://arxiv.org/abs/1612.00137.pdf | Project page