3D Gaussian splatting for Three.js

This repository contains a Three.js-based implementation of 3D Gaussian Splatting for Real-Time Radiance Field Rendering, a technique for the real-time visualization of real-world 3D scenes. Their project was CUDA-based and I wanted to build a viewer that was accessible via the web.

When I started, web-based viewers were already available -- A WebGL-based viewer from antimatter15 and a WebGPU viewer from cvlab-epfl -- However no Three.js version existed. I used those versions as a starting point for my initial implementation, but as of now this project contains all my own code.

Highlights:

Organized into ES modules
Rendering is done entirely through Three.js
The sorting algorithm is a C++ counting sort contained in a WASM module.
Rasterization code is documented to describe 2D covariance computations as well as computations of corresponding eigen-values and eigen-vectors
Scene is partitioned via octree that is used to cull non-visible splats prior to sorting
Splat data (position, covariance, color) is stored via textures so that only splat indexes are transferred between host and GPU
Allows a Three.js scene or object group to be rendered along with the splats

Online demo: https://projects.markkellogg.org/threejs/demo_gaussian_splats_3d.php

This is still very much a work in progress! There are several things that still need to be done:

Improve the method by which splat data is stored in textures (currently much texture space is wasted or packed inefficiently)
Properly incorporate spherical harmonics data to achieve view dependent lighting effects
Improve the layout of the SplatBuffer object for better efficiency and reduced file size
Improve splat sorting -- maybe an incremental sort of some kind?
Implement double buffering so that the next splat index array in the main thread can be filled while the current one is sorted in the worker thread

Building

Navigate to the code directory and run

npm install

Followed by

npm run build

To view the demo scenes locally run

npm run demo

The demo will be accessible locally at http://127.0.0.1:8080/index.html. You will need to download the data for the demo scenes and extract them into

<code directory>/build/demo/assets/data

The demo scene data is available here: https://projects.markkellogg.org/downloads/gaussian_splat_data.zip

Usage

To run the built-in viewer:

const viewer = new GaussianSplat3D.Viewer({
  'cameraUp': [0, -1, -0.6],
  'initialCameraPosition': [-1, -4, 6],
  'initialCameraLookAt': [0, 4, -0]
});
viewer.init();
viewer.loadFile('<path to .ply or .splat file>')
.then(() => {
    viewer.start();
});

The loadFile() method will accept the original .ply files as well as my custom .splat files.

To convert a .ply file into the stripped-down .splat format (currently only compatible with this viewer):

const plyLoader = new GaussianSplat3D.PlyLoader();
plyLoader.loadFromFile('<path to .ply file>')
.then((splatBuffer) => {
    new GaussianSplat3D.SplatLoader(splatBuffer).saveToFile('converted_file.splat');
});

This code will prompt your browser to automatically start downloading the converted .splat file.

It is now possible to integrate your own Three.js scene into the viewer (still somewhat experimental). The Viewer class now accepts a scene parameter by which you can pass in any 'normal' Three.js objects you want to be rendered along with the splats. Rendering the splats correctly with external obejcts requires a special sequence of steps so the viewer needs to be aware of them:

const scene = new THREE.Scene();

const boxColor = 0xBBBBBB;
const boxGeometry = new THREE.BoxGeometry(2, 2, 2);
const boxMesh = new THREE.Mesh(boxGeometry, new THREE.MeshBasicMaterial({'color': boxColor}));
scene.add(boxMesh);
boxMesh.position.set(3, 2, 2);

const viewer = new GaussianSplat3D.Viewer({
  'scene': scene,
  'cameraUp': [0, -1, -0.6],
  'initialCameraPosition': [-1, -4, 6],
  'initialCameraLookAt': [0, 4, -0]
});
viewer.init();
viewer.loadFile('<path to .ply or .splat file>')
.then(() => {
    viewer.start();
});

The viewer allows for various levels of customization via constructor parameters. You can control when its update() and render() methods are called by passing false for the selfDrivenMode parameter and then calling those methods whenever/wherever you decide is appropriate. You can tell the viewer to not use its built-in camera controls by passing false for the useBuiltInControls parameter. You can also use your own Three.js renderer and camera by passing those values to the viewer's constructor. The sample below shows all of these options:

const renderWidth = 800;
const renderHeight = 600;

const rootElement = document.createElement('div');
rootElement.style.width = renderWidth + 'px';
rootElement.style.height = renderHeight + 'px';
document.body.appendChild(rootElement);

const renderer = new THREE.WebGLRenderer({
    antialias: false
});
renderer.setSize(renderWidth, renderHeight);
rootElement.appendChild(renderer.domElement);

const camera = new THREE.PerspectiveCamera(65, renderWidth / renderHeight, 0.1, 500);
camera.position.copy(new THREE.Vector3().fromArray([-1, -4, 6]));
camera.lookAt(new THREE.Vector3().fromArray([0, 4, -0]));
camera.up = new THREE.Vector3().fromArray([0, -1, -0.6]).normalize();

const viewer = new GaussianSplat3D.Viewer({
    'selfDrivenMode': false,
    'renderer': renderer,
    'camera': camera,
    'useBuiltInControls': false
});
viewer.init();
viewer.loadFile('<path to .ply or .splat file>')
.then(() => {
    requestAnimationFrame(update);
});

Since selfDrivenMode is false, it is up to the developer to call the update() and render() methods on the Viewer class:

function update() {
    requestAnimationFrame(update);
    viewer.update();
    viewer.render();
}