Repository files navigation

OpenISS is a motion capture data aggregation and delivery framework for VFX thathas library instances abstracting various middleware and cameras for many application types.

Applications include AI+Art, performing arts, interactive film, facial animation,NUI applications using OpenGL, Processing, TensorFlow, Keras, and MARF.

The OpenISS framework provides a uniform abstraction layer over libfreenect,libfreenect2, librealsense, OpenNI2, NiTE2, NuiTrack, Azure SDK, PCL, and ROS and hasREST and SOAP web services. There are different front-end modules aswell based on OpenGL, Processing, and web browsers (JS).

The public GitHub updates are done periodically from private forks wherea lot of integration / contributions happen.

OpenISS is inspired by the development of ISSv1, ISSv2, ISSv3 and MultiCamTk++, but is being builtfrom scratch using C/C++ and various wrappers and uses similar or same open-source libraries, middleware and toolkits forsensors and creative coding. Various wrappers are being developed. See background on the inspirational ISS below.OpenISS API is likewise for the first time made it possble to access Kinect 1 and 2as well as OpenCV as REST and SOAP services for creative near-realtime online broadcasting.OpenISS is poised to be the core replacement for ISSv2's pipeline. It also serves asan educational tool for graduate and undergraduate students in computer vision, computationarts, pattern recognition, AI, machine learning, and game development. It is designedto be portable.

• OpenISS component development began in 2016 in C
• Expanded to C++ and Java since and ongoing

• Project Lead:Serguei A. Mokhov

• Kyle Ross, Unity plugin and related rigging
• Konstantinos (Kosta) Psimoulis: Web Services lead (REST and SOAP), SimpleOpenNIRS/liblrealsense2 support
• Jonathan Llewellyn: actvity/action recognition, VFX pipeline
• Dominique Cartier, point clouds integration and object/surface reconstruction

• Jashanjot Singh: Gesture and Hand Tracking Framework, NiTE2/NuiTrack, and ROS support, SWIG support for Java/Processing
• Haotao (Eric) Lai: Person Re-identifcation, Green Screening, device abstracion (Kinects 1 and 2, RealSense D435), TensorFlow/Keras/Python abstractions; Kinect 2 web service
• Yiran (Bernie) Shen: Facial Recognition and data (landmarks and expressions) provider framework abstraction over OpenFace, OpenCV, including TensortFlow for VFX
• Yuhao Mao: Art and AI, Sound Visualization, Style Transfer
• Chao Wang: Style Transfer, Magenta support, motion-based audio support, finger tracking
• Zihao Song: Singnal Processing for Audio and Image Processing effects and MARF

• Seehere for more documentation.

• See detailed listhere.

• Updates for Linux and C in EL6 (CentOS 6.x), CSI230-101 Fall 2017 course students teams:

  • Calum Phillips, Rosser Martinez, Matthew Roy
  • Alex Rader, Cory Smith, Nicholas Robbins

• Original build automation contributors for Linux in EL6 (CentOS 6.x), CSI230-101 Fall 2016 course students teams:

  • Brian Baron, Colin Brady, Robert Gentile
  • Gabriel Pereyra, Justin Mulkin, Duncan Carrol, Lucas Spiker

• We are working on our Docker imageshere

• Please usecmake
• More documentation to follow

• Please usecmake
• More documentation to follow

EL7 (RHEL, CentOS, Scientific Linux) are as of September 2018 default andpreferred build, so the development effort focuses around this platform,but it is known to run on macOS and Ubuntu.

EL6 is now considered legacy.

Notice, EL6 requires a newer kernel forproper USB3 and NVIDIA support. If you preferto install dependencies manually, you can followthe scripts referenced here (build.sh andel6.sh) and repeat theirrelevant steps one by one.

Install:yum install gitgit clone https://github.com/OpenISS/OpenISS.gitgit submodule update --init --recursiveNavigate to OpenISS/srcRun the command:make depsThis will install all of the 3rd partydependenciesCompile:cd OpenISS/srcmakeClean:Navigate to OpenISS/srcRun the command:make cleanRemoves the installed dependencies

• Seeissues
• Seepull requests

• OS X (mostly there)
• Windows (partially there for some components)

• Java (in progress)
• C++ (mostly there)
• Processing (in progress)
• Max (in progress)
• PureData (in progress)
• Python (started)
• ...

OpenISS’s core goals are to enable achieving something akinto the original ISS below in an open educational setting. Then,to build custom applications based on it, using OpenISS APIas a core.

• Origins are in Dr. Miao Song’s thesis:
  • http://arxiv.org/abs/1212.6250
• SIGGRAPH and SIGGRAPH Asia
  • https://doi.org/10.1145/3355047.3359423
  • https://doi.org/10.1145/3084863.3084864
  • https://doi.org/10.1145/2988458.2988460
  • https://doi.org/10.1145/2992138.2992139
  • https://doi.org/10.1145/2668947.2668953
  • https://doi.org/10.1145/2820926.2820940
• JIDPS
  • https://doi.org/10.3233/jid-2016-0026
• GEM
  • http://dx.doi.org/10.1109/GEM.2015.7377204
  • http://dx.doi.org/10.1109/GEM.2015.7377247
• VSMM
  • http://dx.doi.org/10.1109/VSMM.2014.7136675
• IDEAS
  • http://doi.acm.org/10.1145/3105831.3105862

• https://www.youtube.com/watch?v=yjH0S7Otwj4
• https://vimeo.com/169321384
• https://vimeo.com/169319502
• https://vimeo.com/166999333
• https://vimeo.com/153062959
• https://vimeo.com/152476114
• https://vimeo.com/151683852
• https://vimeo.com/141811579
• https://vimeo.com/141081567
• https://vimeo.com/130122925
• https://vimeo.com/129692753
• https://vimeo.com/121177927
• https://vimeo.com/96355443
• https://vimeo.com/85049604
• https://vimeo.com/68347351
• https://vimeo.com/51329588
• https://vimeo.com/50069419
• https://vimeo.com/49682696
• https://vimeo.com/49399617

• TravisCI EL6 and EL7 builds on GitHub:Derek Weitzel, Building CentOS packages on Travis-CI,https://djw8605.github.io/2016/05/03/building-centos-packages-on-travisci/