3D Reconstruction for all

Massive thanks to @GradeEterna for the beautiful scenes

Brush is a 3D reconstruction engine using Gaussian splatting. It works on a wide range of systems: macOS/windows/linux, AMD/Nvidia/Intel cards, Android, and in a browser. To achieve this, it uses WebGPU compatible tech and the Burn machine learning framework.

Machine learning for real time rendering has tons of potential, but most ML tools don't work well with it: Rendering requires realtime interactivity, usually involve dynamic shapes & computations, don't run on most platforms, and it can be cumbersome to ship apps with large CUDA deps. Brush on the other hand produces simple dependency free binaries, runs on nearly all devices, without any setup.

Try the web demo NOTE: Only works on Chrome and Edge. Firefox and Safari are hopefully supported soon)

Brush takes in COLMAP data or datasets in the Nerfstudio format. Training is fully supported natively, on mobile, and in a browser. While training you can interact with the scene and see the training dynamics live, and compare the current rendering to input views as the training progresses.

It also supports masking images:

• Images with transparency. This will force the final splat to match the transparency of the input.
• A folder of images called 'masks'. This ignores parts of the image that are masked out.

Brush also works well as a splat viewer, including on the web. It can load .ply & .compressed.ply files. You can stream in data from a URL (for a web app, simply append ?url=).

Brush also can load .zip of splat files to display them as an animation, or a special ply that includes delta frames (see cat-4D and Cap4D!).

Brush can be used as a CLI. Run brush --help to get an overview. Every CLI command can work with --with-viewer which also opens the UI, for easy debugging.

While training, additional data can be visualized with the excellent rerun. To install rerun on your machine, please follow their instructions. Open the ./brush_blueprint.rbl in the viewer for best results.

First install rust 1.88+. You can run tests with cargo test --all. Brush uses the wonderful rerun for additional visualizations while training, run cargo install rerun-cli if you want to use it.

Simply cargo run or cargo run --release from the workspace root. Brush can also be used as a CLI, run cargo run --release -- --help to use the CLI directly from source. See the notes about the CLI in the features section.

Brush can be compiled to WASM. Run npm run dev to start the demo website using Next.js, see the brush_nextjs directory.

Brush uses wasm-pack to build the WASM bundle. You can also use it without a bundler, see wasm-pack's documentation.

WebGPU is still an upcoming standard, and as such, only Chrome 134+ on Windows and macOS is currently supported.

As a one time setup, make sure you have the Android SDK & NDK installed.

• Check if ANDROID_NDK_HOME and ANDROID_HOME are set
• Add the Android target to rust rustup target add aarch64-linux-android
• Install cargo-ndk to manage building a lib cargo install cargo-ndk

Each time you change the rust code, run

• cargo ndk -t arm64-v8a -o crates/brush-app/app/src/main/jniLibs/ build
• Nb: Nb, for best performance, build in release mode. This is separate from the Android Studio app build configuration.
• cargo ndk -t arm64-v8a -o crates/brush-app/app/src/main/jniLibs/ build --release

You can now either run the project from Android Studio (Android Studio does NOT build the rust code), or run it from the command line:

./gradlew build
./gradlew installDebug
adb shell am start -n com.splats.app/.MainActivity

You can also open this folder as a project in Android Studio and run things from there. Nb: Running in Android Studio does not rebuild the rust code automatically.

Rendering and training are generally faster than gsplat. You can run benchmarks of some of the kernels using cargo bench.

gSplat, for their reference version of the kernels

Peter Hedman, George Kopanas & Bernhard Kerbl, for the many discussions & pointers.

The Burn team, for help & improvements to Burn along the way

Raph Levien, for the original version of the GPU radix sort.

GradeEterna, for feedback and their scenes.

This is not an official Google product. This repository is a forked public version of the google-research repository