Repository files navigation

multimolang is a package designed to support multimodal data analysis for linguistic research. It serves as the main repository for theMULTIFLOW project and provides tools to process, analyze, and derive insights from multimodal datasets, starting with gesture trajectories extracted from video data.

The first implemented tool,dfMaker, processes raw OpenPose data into structured dataframes, optimized for large-scale, multimodal data science workflows. Future versions ofmultimolang will include additional tools for prosody analysis and multimodal language integration.

If you usemultimolang or any of its core functions (such asdfMaker) in your research, please cite:

Herreño Jiménez, B. (2024).multimolang: Multimodal Language Analysis [R package version 0.1.1]. The R Foundation.https://doi.org/10.32614/cran.package.multimolang

You can also retrieve the BibTeX entry by running in R:

citation("multimolang")

# Install from CRANinstall.packages("multimolang")# Or install using devtoolsdevtools::install_github("daedalusLAB/multimolang")

For detailed installation instructions and usage examples, please refer to the package vignettes.

Currently, the main function in the package isdfMaker, which provides:

• Flexible Input Configuration: Accepts JSON files from OpenPose as input.
• Transformation Capabilities: Scales and normalizes body keypoints for gesture trajectory analysis.
• Big Data Optimization: Designed to handle large-scale multimodal datasets efficiently.

# Define paths to example data included with the packageinput.folder<- system.file("extdata","ex_videos","o1",package="multimolang")output.file<- file.path(tempdir(),"processed_data.csv")output.path<- tempdir()# Run dfMaker with example datadf<- dfMaker(input.folder=input.folder,output.file=output.file,output.path=output.path,no_save=FALSE,fast_scaling=TRUE,transformation_coords= c(1,1,5,5))# View the resulting datahead(df)

Below is the current progress and pending tasks formultimolang:

Note: Future updates will incorporate tools for prosody and voice analysis, as well as co-speech gesture processing.

TheMULTIFLOW project investigates the interplay between gesture, prosody, and language to uncover multimodal signatures in communication. Learn more about our research and goals.

• R functions for creating dataframes from OpenPose raw data.
  • Analyzing gesture trajectories from normalized body keypoint detection.
  • Building Massive Co-Speech Gesture Datasets for Specific Linguistic Patterns.
  • Gestural behavior is systematically attuned to language: Novel data analysis of co-speech gesture and its implications for multimodal interfaces.

We welcome contributions tomultimolang! For bug reports, feature requests, or collaboration inquiries, open an issue onGitHub.

• dfMaker Details

• Updated file naming conventions to comply with CRAN requirements.
• Improved compatibility with CRAN by refining documentation and dependencies.
• Minor fixes todfMaker for enhanced error handling and performance.
• Added package vignettes for detailed usage examples.

For a complete list of changes, refer to the CHANGELOG in the repository.