Skip to content

Point Cloud Module Architecture

The point cloud module is the canonical bridge between data ingestion and GPU execution. It receives GenericGaussianPointCloudTS objects (from IO or ONNX paths), instantiates the GPU buffers required by preprocess and render pipelines, tracks per-model uniforms, and exposes bind groups plus dynamic hooks so every downstream stage can consume a consistent contract.

Pipeline Map

┌──────────────┐      ┌───────────────────────────┐      ┌────────────────────┐
│ IO / ONNX    │ ───► │ PointCloud / DynamicPC    │ ───► │ Preprocess Compute │
│ (data source)│      │  • GPU buffers            │      │  • depth sort pass │
└──────────────┘      │  • Uniform + model params │      └─────────┬──────────┘
                      │  • Bind group handles     │                │
                      └───────────────────────────┘                ▼
                                                     ┌────────────────────────┐
                                                     │ Renderer / Indirect MW │
                                                     └────────────────────────┘

Static assets follow the IO path while real-time scenes inject GPU buffers directly through DynamicPointCloud. Both converge on identical bind group layouts, enabling the preprocess, sorting, and renderer modules to remain agnostic to data provenance.

Construction Path

  1. Metadata ingestion – PointCloud reads counts, SH degree, bounding boxes, and optional center/up vectors from the GenericGaussianPointCloudTS implementation. These values seed public fields such as numPoints, shDeg, bbox, and orientation helpers.
  2. Gaussian & SH storage – Unless externalBuffers is supplied, CPU buffers are uploaded into WebGPU buffers labelled gaussians/storage and sh/storage with usage STORAGE | COPY_DST (point_cloud.ts constructor).
  3. Projected splat buffer – A dedicated 2D buffer (splat2DBuffer) is allocated using BUFFER_CONFIG.SPLAT_STRIDE to ensure enough space for projected attributes, sort keys, and indirect draw metadata (GPUBufferUsage.STORAGE | COPY_SRC | COPY_DST | INDIRECT).
  4. Uniform buffers – Two UniformBuffer instances are spawned:
  5. uniforms: 16 bytes for point count and SH degree, consumed by preprocess shaders.
  6. modelParamsUniforms: 128 bytes aligned to 16-byte boundaries, carrying transforms, offsets, shading controls, and precision descriptors.
  7. Bind groups – The module fetches cached layouts from getBindGroupLayout / getRenderBindGroupLayout and immediately creates pointcloud/bg (compute) plus pointcloud/render/bg (render) instances so pipelines can bind them without additional wiring.

Bind Group Contract

  • Compute (@group(0))
  • binding 0: Gaussian storage (read-only-storage).
  • binding 1: SH storage (read-only-storage).
  • binding 2: 2D splat buffer (storage – also used as indirect target).
  • binding 3: Draw uniforms (uniform).
  • Render (@group(0))
  • binding 2: Read-only access to the projected splat buffer for vertex-stage rasterization.

Layouts live inside a WeakMap cache per GPUDevice, guaranteeing deterministic binding order and avoiding layout churn when multiple point clouds coexist.

Model Parameter Uniform

PointCloud keeps the authoritative transform/configuration block in CPU memory and writes it through updateModelParamsBuffer (also called by setTransform and updateModelParamsWithOffset). The structure mirrors the WGSL expectation:

  • Bytes 0–63: Column-major model matrix.
  • Bytes 64–95: Metadata for multi-instance rendering (baseOffset, numPoints) and shading controls (gaussianScaling, maxShDeg, kernelSize, opacityScale, cutoffScale, rendermode).
  • Bytes 96–119: Precision descriptors (gaussDataType, colorDataType, per-channel scales and zero-points) used to decode quantized INT8/UINT8 data in shaders.
  • Bytes 120–127: Reserved padding.

Because the buffer is rebuilt whenever updateModelParamsBuffer runs, downstream systems simply keep a reference to modelParamsUniforms.buffer and write it into whichever bind group is responsible for model uniforms in the renderer.

DynamicPointCloud Flow

DynamicPointCloud inherits every capability from PointCloud but swaps the constructor contract:

  • Accepts pre-existing GPUBuffer handles (gaussianBuffer, shBuffer) plus an optional countBuffer describing the live draw count for indirect pipelines.
  • Computes the SH degree from the provided colorChannels (3, 4, 12, 27, 48) and exposes the colorMode (rgb if 4 channels, otherwise sh).
  • Stores optional precisionInfo (gaussian/color) so quantization metadata can be injected into the model params block with setPrecisionForShader.
  • Manages an embedded TimelineController, exposing helpers such as startAnimation, setTimeScale, setTimeUpdateMode, setAnimationIsLoop, and resetFrameTime so animation managers can drive ONNX models with consistent timing semantics.
  • The asynchronous update method composes the camera matrix with the model transform, computes an adjusted frame time (respecting loop/offline preview modes), and invokes the ONNX generator, which writes directly into the bound GPU buffers—no CPU copies in the steady state.

Precision & Buffer Swapping

  • replaceStorageBuffers (base class) swaps Gaussian/SH buffers and rebuilds the compute bind group while preserving the projected splat buffer and uniforms.
  • applyFP16 (dynamic subclass) is a convenience wrapper that replaces buffers, marks both gaussian/color precisions as float16, and refreshes shader metadata.
  • setPrecisionForShader rewrites bytes 96–119 of the model params buffer using DataView setters so shaders can decode INT8/UINT8 payloads on demand.

These pathways let the preprocess module run conversion shaders (e.g., convert_precision.wgsl) and then rebind the results without re-instantiating PointCloud.

Resource Lifecycle & Diagnostics

  • getSplatBuffer publishes the raw GPU storage plus metadata, which the preprocess module uses to drive compute dispatch sizing.
  • Parameter setters (setGaussianScaling, setMaxShDeg, setKernelSize, setOpacityScale, setCutoffScale, setRenderMode) update cached state and reuse updateModelParamsBuffer to flush changes.
  • DynamicPointCloud.countBuffer() exposes the optional draw-count buffer so the renderer can configure indirect draws.
  • getPerformanceStats surfaces timeline-related telemetry (isPlaying, animation speed, ONNX linkage) for UI overlays or debugging tools.
  • dispose currently clears timeline listeners; GPU buffers remain externally owned to avoid double-freeing shared resources.

Data Flow Summary

[GenericGaussianPointCloudTS] ──► PointCloud ctor ──► {gaussianBuffer, shBuffer, splatBuffer}
                                             │
                                             ├─► UniformBuffer (draw)
                                             └─► UniformBuffer (model params)

[ONNX Generator] ──► DynamicPointCloud.update ──► direct GPU writes ──► same bind groups

By standardising buffer lifetimes, bind group layouts, and precision metadata, the point cloud module ensures that IO loaders, ONNX pipelines, preprocess kernels, sorters, and renderers all agree on a single contract, keeping Visionary’s WebGPU stack predictable and extensible.