After seeing the amazing work done by Angjoo Kanazawa, Michael J. Black, David W. Jacobs, Jitendra Malik on the End-to-end Recovery of Human Shape and Pose paper. I wanted to have a look if I could use OpenPose and HMR to implement a one click video to mocap solution. The script takes a video and outputs a separate maya file for each person in the video containing and animated HIK skeleton.
The HMR repository was slightly adjusted to work with python3 in a windows environment and matrices to be exported over multiple frames to support video. The number of joints exported by the HMR solution is insufficient to create a working HIK skeleton, a center hip and spine joint had to be estimated. The OpenPose util was updated to recognize the same person over multiple frames, and omit people if they are under a certain presence threshold.
For this demo the animation curves exported have been simplified to reduce initial jitter. As you can see it will not be a render ready solution but potentially a great tool to get an initial starting point or simply to be used as 3D reference.
In order for this tool to work a couple of 3rd party application will have to be installed. To make this an easy process bat files are located in the 3rdparty directory and can be used to download/extract and install the necessary applications.
- Run 3rdparty/getFFmpeg.bat
- Download and extract FFmpeg ( 20180630-9f0077c ).
- Run 3rdparty/getOpenPose.bat
- Download and extract OpenPose ( 1.3.0 ).
- Download models for OpenPose
- Run 3rdparty/getHMR.bat ( Requires python3 + pip3 accessible in PATH )
- Pull the latest custom fork of the HMR repository.
- Download models for HMR
- Pip install requirements.txt
From the command line:
- –video_path: Path to video
- –output_dir: Directory to output the maya files too
- –keep_temp: Keep temp files for debugging ( False by default )
- –mayapy_exe: Overwrite mayapy.exe ( Default requires accessible in PATH )
- –python2_exe: Overwrite python.exe ( Default requires accessible in PATH )
- –python3_exe: Overwrite python3.exe ( Default requires accessible in PATH )
The exe files can be overwritten in case the python interpreters are not accessible through the PATH variable and a relative path cannot be provided.
- No camera tracking ( static camera advised )
- No partial body ( full body in view each frame advised )
- No ankle/wrist and head rotation
- Limited depth adjustment
As loads of things are running in process it is quite simple for something to go wrong. For this reason a log file is implemented that gets saved into the output_dir. If the desired result is unexpected these logs can be investigated to find out what and where something went wrong.
Keypoint matching example:
- HMR ( custom fork )
- Version v1.3.0.
- Link: https://github.com/CMU-Perceptual-Computing-Lab/openpose/releases
- Version 20180630-9f0077c.
- Link: https://ffmpeg.zeranoe.com/builds/
- Version 18.05.
- Link: https://www.7-zip.org/download.html
- Version wget-1.19.1-win64.
- Link: https://eternallybored.org/misc/wget/
End-to-end Recovery of Human Shape and Pose
Computer Vision and Pattern Regognition (CVPR) 2018
Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
Hand Keypoint Detection in Single Images using Multiview Bootstrapping
Convolutional pose machines