Skip to content

Three.js Integration Module Architecture

Visionary renders Gaussian splats and classic Three.js content on the same WebGPU device. This document explains how the orchestration layer is wired, how depth and overlays are captured, and how the low-level integration (GaussianSplattingThreeWebGPU) stays reusable outside of the editor.

High-level layout

THREE.WebGPURenderer (owner of GPUDevice/GPUCanvasContext)
│
├─ Scene Graph (meshes, lights, gizmos, GaussianModel instances)
│   └─ GaussianThreeJSRenderer (extends THREE.Mesh)
│        ├─ Hooks into onBeforeRender to run Gaussian preprocess
│        ├─ Captures color+depth via renderThreeScene()
│        ├─ Issues drawSplats() with shared command encoder
│        └─ Composites optional gizmo overlay
│
└─ Low-level helper: GaussianSplattingThreeWebGPU
        ├─ Owns GaussianRenderer (compute + render pipelines)
        ├─ Manages PointCloud / DynamicPointCloud lifetime
        └─ Uses DirectCameraAdapter for camera conversion

Motivations

  1. Single GPU device – No handshake between WebGL and WebGPU contexts, zero texture copies, no canvas stacking logic.
  2. Deterministic depth – Scene meshes and splats read/write the same depth values (captured once per frame).
  3. Extensibility – GaussianThreeJSRenderer lives in the scene graph, so XR, post-processing, editor gizmos, and OrbitControls continue to function normally.

Frame timeline

function animate(nowMs: number) {
  requestAnimationFrame(animate);

  gaussianRenderer.updateDynamicModels(camera, nowMs * 0.001);  // (1) ONNX deformation
  gaussianRenderer.renderOverlayScene(gizmoScene, camera);      // (2) optional helpers
  gaussianRenderer.renderThreeScene(camera);                    // (3) capture scene color+depth
  gaussianRenderer.drawSplats(threeRenderer, scene, camera);    // (4) splat + overlay composite
}
  1. Dynamic update – Camera matrices (converted through CameraAdapter) drive ONNX-based DynamicPointClouds so they can bake parallax-aware shading data.
  2. Overlay render – Gizmos or HUD scenes render into gizmoOverlayRT, using the same Three.js renderer to ensure consistent tone mapping.
  3. Scene capture – The main scene renders to an internal HalfFloat render target with a Float depth texture. A WebGPU fullscreen pass blits the color buffer to the actual canvas, handling the linear→sRGB conversion that Three.js expects.
  4. Gaussian pass – onBeforeRender precomputes transforms and dispatches compute work. drawSplats() submits one render pass that reads the captured depth texture (when auto depth is on) and optionally composites the gizmo overlay on top of the swap chain.

Shared-device command flow

  • GaussianThreeJSRenderer obtains the GPUDevice and GPUCanvasContext from THREE.WebGPURenderer.backend.
  • Every frame it creates a single command encoder per stage (prepareMulti and drawSplats). There is no secondary context or canvas—splats draw directly into context.getCurrentTexture().
  • The renderer relies on GaussianRenderer.prepareMulti(...) / renderMulti(...) so all visible models are batched in one compute/render pair. This keeps queue submissions minimal even with many models.

Camera conversion

Both DirectCameraAdapter (used by GaussianSplattingThreeWebGPU) and CameraAdapter (used elsewhere in the app) follow the same set of rules:

  • Start from camera.matrixWorldInverse (Three’s view matrix) and multiply by R_y(π) to adopt the renderer’s +Z-forward convention without changing handedness.
  • Multiply the projection matrix by the same R_y(π) to keep P * V identical to Three.js output.
  • Negate the second row after preprocess to counter the single Y-flip that happens when packing splats.
  • Derive focal lengths in pixel space from the active viewport so compute shaders receive accurate lens data.

The adapters expose viewMatrix(), projMatrix(), position(), frustumPlanes(), and a projection.focal() shim that matches what the Gaussian renderer expects.

Depth capture & overlays

Auto Depth Mode (default)

  1. renderThreeScene() lazily allocates a THREE.RenderTarget(width, height, HalfFloat) and a matching THREE.DepthTexture(width, height, FloatType).
  2. The Three.js scene renders into that target once per frame.
  3. A WebGPU render pass blits the target color texture to the swap chain while converting from linear to sRGB.
  4. When drawSplats() runs, it fetches the WebGPU texture handle that backs DepthTexture and plugs it into the render pass descriptor as depthStencilAttachment (load/store = load). Splats therefore respect all mesh occluders automatically.
  5. Gizmo overlays are composited afterwards using a simple textured full-screen quad rendered with premultiplied-alpha blending.

Disabling auto depth (setAutoDepthMode(false)) hands control back to the caller. Legacy projects can then call setOccluderMeshes(...), which renders a reduced scene into a temporary depth texture each frame. This path is kept solely for compatibility reasons.

Diagnostics

  • diagnoseDepth() prints whether auto depth is enabled, render-target sizes, and current GaussianRenderer depth settings.
  • disposeDepthResources() clears cached render targets so the next frame recreates them—useful after device loss or canvas resize anomalies.

GaussianSplattingThreeWebGPU internals

GaussianSplattingThreeWebGPU is intentionally minimal:

const gs = new GaussianSplattingThreeWebGPU();
await gs.initialize(renderer.backend.device);
await gs.loadPLY('/models/room.ply');

const encoder = device.createCommandEncoder();
gs.render(encoder, swapChainView, camera, [width, height], depthView);
device.queue.submit([encoder.finish()]);
  • initialize(device) instantiates GaussianRenderer(device, 'bgra8unorm', 3) and ensures the GPU sorter pipelines are ready.
  • loadPLY / loadFile rely on defaultLoader and wrap the result in a PointCloud.
  • render(...) updates the internal DirectCameraAdapter, runs prepareMulti, optionally toggles depth with setDepthEnabled, and emits one render pass.
  • The helper exposes setVisible, numPoints, setDepthEnabled, and dispose for lifecycle management.

Dynamic model handling

  • GaussianModel.update() executes ONNX inference by passing the current transform, view matrix, projection matrix, and an optional animation time to DynamicPointCloud.
  • GaussianThreeJSRenderer.updateDynamicModels() simply iterates over every registered model and awaits model.update(...). This runs before any GPU work so the splat buffers always contain fresh positions/SH data.
  • The renderer also exposes granular setters (setModelGaussianScale, setModelOpacityScale, setModelRenderMode, etc.) plus global variants (setGlobalTimeScale, startAllAnimations, …) that operate on every GaussianModel.

Error handling

  • Missing WebGPU support: callers should guard renderer creation (see initThreeContext) and fall back to WebGL-only scenes.
  • Camera mismatches: keep all conversions in CameraAdapter / DirectCameraAdapter—ad-hoc matrix tweaks quickly lead to mirrored outputs.
  • Device loss: call disposeDepthResources() and diagnoseDepth() after a device.lost event; the next renderThreeScene() invocation will recreate the targets.

Files worth reading

  • src/app/GaussianThreeJSRenderer.ts – orchestration layer, auto depth, overlays, runtime controls.
  • src/three-integration/GaussianSplattingThreeWebGPU.ts – lightweight helper for direct integrations.
  • src/camera/CameraAdapter.ts – reusable camera conversion logic.
  • src/app/GaussianModel.ts – Object3D wrapper with auto-sync and animation helpers.
  • src/renderer/gaussian_renderer.ts – shared compute/render pipelines.