ACM Kinect Conductor

Interactive Game for Engineering Open House 2018,,

Kinect Conductor is a interactive music game by ACM SigMusic for Engineering Open House 2018 at UIUC. The game part consists the following:

  • A C++ program based on Freenect and NiTE libraries that fetches body skeleton data of players.
  • A NodeJs server for providing the skeleton data.
  • A Python Flask server that provides a set of pre-composed music patterns.
  • A game developed with Unreal Engine 4. It fetches player skeleton data and music pattern data from the servers, display them on the screen. When players hit the music notes, it sends midi signals to a software synthesizer.
  • A software synthesizer (example project from JUCE that runs the VST plugin Dexed)

I am the main developer and I participated in all tasks above except music composition.

Web app for composing music patterns

This is a web app I built in a day for composers to quickly write music patterns that run on our server. Most functionalities are realized with JointJS and Jquery. It does the following:

  • Create “chord” nodes, which contain the note and instrument information for a measure of music chord.
  • Create “Transitions”, which links chord nodes to indicate chord progressions.
  • Save and load the music graph using Json format.

Project Summary

Kinect Conductor is a project by ACM SigMusic. I designed the graphics, game play, web servers, and communication method between programs.


Kinect Conductor screenshot Kinect Conductor on exhibit
Copyright © 2020, Fanbo Xiang. Website generated with org-mode and Jekyll.
Please feel free to use my designs however you like. Code is available here.