https://github.com/google-ai-edge/LiteRT

LiteRT, successor to TensorFlow Lite. is Google's On-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization

Google's on-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization

📖 Get Started | 🤝 Contributing | 📜 License | 🛡 Security Policy | 📄 Documentation

Nightly Builds	Continuous Builds	Other Builds

LiteRT continues the legacy of TensorFlow Lite as the trusted, high-performance runtime for on-device AI.

LiteRT features advanced GPU/NPU acceleration, delivers superior ML & GenAI performance, making on-device ML inference easier than ever.

• 🆕 New LiteRT Compiled Model API: Streamline development with automated accelerator selection, true async execution, and efficient I/O buffer handling.

  • Automated accelerator selection vs explicit delegate creation
  • Async execution for faster overall execution time
  • Easy NPU runtime and model distribution
  • Efficient I/O buffer handling

• 🤖 Unified NPU Acceleration: Offer seamless access to NPUs from major chipset providers with a consistent developer experience. LiteRT NPU, previously under Early access program is available to all users: https://ai.google.dev/edge/litert/next/npu

• ⚡ Best-in-class GPU Performance: Use state-of-the-art GPU acceleration for on-device ML. The new buffer interoperability enables zero-copy and minimizes latency across various GPU buffer types.

• 🧠 Superior Generative AI inference: Enable the simplest integration with the best performance for GenAI models.

LiteRT is designed for cross-platform deployment on a wide range of hardware.

Platform	CPU Support	GPU Support	NPU Support
🤖 Android	✅	✅OpenCL ✅OpenGL	Google Tensor* ✅ Qualcomm ✅ MediaTek S.LSI* Intel*
🍎 iOS	✅	✅ Metal	ANE*
🐧 Linux	✅	✅ WebGPU	N/A
🍎 macOS	✅	✅ WebGPU ✅ Metal	ANE*
💻 Windows	✅	✅ WebGPU	Intel*
🌐 Web	✅	✅ WebGPU	Coming soon
🧩 IoT	✅	✅ WebGPU	Broadcom* Raspberry Pi*

*Coming soon

Coming soon...

For a comprehensive guide to setting up your application with LiteRT, see the Get Started guide.

You can build LiteRT from source:

1. Start a docker daemon.
2. Run build_with_docker.sh under docker_build/

The script automatically creates a Linux Docker image, which allows you to build artifacts for Linux and Android (through cross compilation). See build instructions in CMake build instructions and Bazel build instructions for more information on how to build runtime libraries with the docker container.

For more information about using docker interactive shell or building different targets, please refer to docker_build/README.md.

Every developer's path is different. Here are a few common journeys to help you get started based on your goals:

• Goal: Convert a model from PyTorch to run on LiteRT.
• Path1 (classic models): Use the LiteRT Torch Converter to transform your PyTorch model into the .tflite format, and use AI Edge Quantizer to optimize the model for optimal performance under resource constraints. From there, you can deploy it using the standard LiteRT runtime.
• Path2 (LLMs): Use LiteRT Generative Torch API to reauthor and convert your PyTorch LLMs into Apache format, and deploy it using LiteRT LM.

• Goal: Run a pre-trained model (like image segmentation) in a mobile app for the first time.
• Path1 (Beginner dev): Follow step-by-step instructions via Android Studio to create a Real-time segmentation App for CPU/GPU/NPU inference. Source code link.
• Path2 (Experienced dev): Start with the Get Started guide, find a pre-trained .tflite model on Kaggle Models, and use the standard LiteRT runtime to integrate it into your Android or iOS app.

• Goal: Accelerate an existing model to run faster and more efficiently on-device.
• Path:
  • Explore the LiteRT API to easily leverage hardware acceleration.
  • For working with Generative AI: Dive into LiteRT LM, our specialized solution for running GenAI models.

• Goal: Deploy a large language model (LLM) or diffusion model on a mobile device.
• Path: Dive into LiteRT LM, our specialized solution for running GenAI models. You'll focus on model quantization and optimizations specific to large model architectures.

Our commitment is to make LiteRT the best runtime for any on-device ML deployment. Our product strategies are:

• Expanding Hardware Acceleration: Broadening our support for NPUs and improving performance across all major hardware accelerators.
• Generative AI Optimizations: Introducing new optimizations and features specifically for the next wave of on-device generative AI models.
• Improving Developer Tools: Building better tools for debugging, profiling, and optimizing models.
• Platform Support: Enhancing support for core platforms and exploring new ones.

We welcome contributions to LiteRT. Please see the CONTRIBUTING.md file for more information on how to contribute.

We encourage you to reach out if you need help.

• GitHub Issues: For bug reports and feature requests, please file a new issue on our GitHub Issues page.
• GitHub Discussions: For questions, general discussions, and community support, please visit our GitHub Discussions.

LiteRT is part of a larger ecosystem of tools for on-device machine learning. Check out these other projects from Google:

• LiteRT Samples: A collection of LiteRT sample apps.
• LiteRT Torch Converter: A tool in LiteRT to convert PyTorch models into the LiteRT(.tflite) format for on-device deployment.
• LiteRT Generative Torch API: A library in LiteRT to reauthor LLMs for efficient conversion and on-device inference.
• LiteRT-LM: A library to efficiently run Large Language Models (LLMs) across edge platforms, built on top of LiteRT.
• XNNPACK: A highly optimized library of neural network inference operators for ARM, x86, and WebAssembly architectures that provides high-performance CPU acceleration for LiteRT.
• MediaPipe: A framework for building cross-platform, customizable ML solutions for live and streaming media.

This project is dedicated to fostering an open and welcoming environment. Please read our Code of Conduct to understand the standards of behavior we expect from all participants in our community.

LiteRT is licensed under the Apache-2.0 License.