Mediapipe Unity

| Posted on by



Project Description

TFLite models are normally not supported by the ml-agents framework. This causes the problem that quantized models can not be currently run by the Unity Inference Engine whereas performance drawbacks are inevitable during the runtime. Currently, tflite models’ performances are backed by the GPU delegates which have been implemented in the mean time and are being developed further. Especially, with the development of frameworks like MediaPipe, Unity could need more power on the Android side due to the unsupport of the TFLite models. This plugin helps to integrate TFLite inference into the Unity Engine and also to the R&D teams for preparing quick prototypes as the Unity Engine has an easy environment to accomplish this purpose.

Roadmap

Mediapipe Unity
  • MediaPipe offers open source cross-platform, customizable ML solutions for live and streaming media.
  • Using MediaPipe Hand Tracking with Unity 3 We are working on a project for a Mobile (Android) Mixed-Reality (MR) game using only hand gestures for interacting with the virtual objects. We've researched for frameworks and api's that we could use for the project.
  • Unity plugin to run MediaPipe graphs. Contribute to homuler/MediaPipeUnityPlugin development by creating an account on GitHub.
  • My tests with the MediaPipe Hand tracking library (3D version) just released opensource by Google. References: - MediaPipe Hand Tracking repo: https://github.
  1. Simple TFLite interpreter integration(in development).
  2. Reusable plug&play API on the C# side to reduce development time without rewriting the same components again(in development).
  3. Easy integration into the AR Foundations environment to use computer vision neural nets for several tasks for AR scenes(in development).
  4. Performance optimizations.

Initializing unity; mediapipe.

Project Structure

TFARInterface : A Unity interface to use the tflite plugin.

TFLite Android : A plugin to support tflite models on the Android side.TODO: iOS plugin will be added.

Contributions

Contributions are welcomed and encouraged!

Mediapipe Unity

  1. TFLite plugin will be ported to C++ for better performance.
  2. You can add your .tflite models into the /assets folder to share them for reuse(pull request with model details is encouraged!).

Acknowledgments

Google Mediapipe Unity

Thanks to the MediaPipe Team for having published their useful models for essential AR interactions!

Mediapipe Unity Free

Mediapipe Unity

Warnings

Mediapipe Unity Player

Currently the plugin uses the alpha version of the TF Lite framework which means that you should use the plugin accordingly.The productions versions will be integrated later.