OpenMM is an open-source tool that can perform multimodal feature extraction. In other words, this tool will allow you to easily extract video, audio, and linguistic features all at once. This tool builds upon existing GitHub repos for visual feature extraction OpenFace and audio feature extraction Covarep. I integrate these existing repos with my code for linguistic feature extraction (LingAnalysis). OpenMM provides a simple way for researchers to extract multimodal features. The tool only requires a video as input and outputs a csv of multimodal features, audio conversion and speech-to-text are handled internally. My hope is that this will help promote more interest and research in building multimodal systems.
Please see the Wiki for installation instructions.
If you use any of this code in your work, please cite:
OpenMM: An Open-Source Multimodal Feature Extraction Tool (Michelle Renee Morales, Stefan Scherer, Rivka Levitan), In Proceedings of Interspeech 2017, ISCA, 2017.
@inproceedings{morales_openmm:_2017,
address = {Stockholm, Sweden},
title = {{OpenMM}: {An} {Open}-{Source} {Multimodal} {Feature} {Extraction} {Tool}},
url = {https://www.researchgate.net/publication/319185055_OpenMM_An_Open-Source_Multimodal_Feature_Extraction_Tool},
doi = {10.21437/Interspeech.2017-1382}}
This repo represents code from my dissertation work. I did my best to ensure that the code runs out of the box, but there are always issues. So please understand that this is research code and not a commercial level product. However, if you encounter any problems/bugs/issues please contact me on github or email me at [email protected] for any bug reports/questions/suggestions.