Skip to content

Point Cloud Module

The Point Cloud Module converts normalized Gaussian splat data into GPU resources that power Visionary’s preprocess, sorting, and render pipelines. It encapsulates buffer creation, bind-group wiring, runtime parameter tuning, and dynamic ONNX-driven animation so the rest of the system can focus on algorithms instead of plumbing.

Overview

The module delivers:

  • GPU Buffer Orchestration – Uploads CPU data or accepts pre-created GPU buffers, and allocates the projected 2D splat buffer used by preprocessing, sorting, and indirect draws.
  • Uniform & Metadata Streaming – Maintains draw-level uniforms plus a 128-byte model-parameter block that carries transforms, splat knobs, and precision metadata understood by WGSL shaders.
  • Bind Group Management – Provides cached compute and render bind group layouts and auto-instantiated bind groups that match the expectations of preprocess (@group(0)) and render stages.
  • Runtime Controls – Exposes setters for Gaussian scaling, spherical harmonics degree, kernel size, opacity, cutoff, and render mode so tools can tweak appearance without touching shaders.
  • Dynamic Data Path – DynamicPointCloud pairs the buffer infrastructure with ONNX generators, optional draw-count buffers, and a timeline controller to drive real-time content.

Responsibilities

GPU Data Lifecycle

  • Accept GenericGaussianPointCloudTS (alias of GaussianDataSource) objects from the IO module.
  • Upload Gaussian attributes and SH coefficients into dedicated GPUBuffer objects, or re-use externally produced buffers for zero-copy workflows.
  • Allocate the projected splat buffer with the stride defined in BUFFER_CONFIG.SPLAT_STRIDE (32 bytes) so compute shaders can emit screen-space records plus indirect-draw payloads.

Uniform & Bind Group Plumbing

  • Initialize draw uniforms ([numPoints, shDegree, 0, 0]) via UniformBuffer.
  • Populate the model-parameter uniform with transforms, offsets, shading knobs, and precision descriptors, then publish helper methods (updateModelParamsBuffer, setTransform, parameter setters) to keep it in sync with scene state.
  • Provide bindGroup() and renderBindGroup() handles that match the cached layouts from layouts.ts, guaranteeing consistency across pipelines.

Dynamic & Precision-aware Rendering

  • Allow runtime buffer swaps through replaceStorageBuffers, enabling FP32→FP16 or quantized conversions without rebuilding the object.
  • DynamicPointCloud augments the base class with ONNX generator hooks, precision metadata injection (setPrecisionForShader, applyFP16), optional count buffers for indirect draws, and a TimelineController that governs playback, looping, and time scaling.

Core Components

  • PointCloud (point_cloud.ts) – Central class that owns GPU buffers, uniforms, and bind groups for a static dataset.
  • DynamicPointCloud (dynamic_point_cloud.ts) – Extends PointCloud with direct GPU buffer ingestion, ONNX generator integration, and advanced timeline controls.
  • layouts.ts – Exposes getBindGroupLayout, getRenderBindGroupLayout, and BUFFER_CONFIG; all layouts are cached per GPUDevice.
  • splat_buffer.ts – Defines SplatBuffer, GenericGaussianPointCloud, and helper classes mirroring GPU memory layouts.
  • index.ts – Barrel export used by the rest of Visionary.

Buffer & Layout Snapshot

Buffer Source Usage Flags Content
Gaussian storage CPU upload or external GPU buffer STORAGE | COPY_DST Packed positions, opacity, and covariance (10× f16 per splat).
SH storage CPU upload or external GPU buffer STORAGE | COPY_DST Packed spherical harmonics coefficients (degree-dependent).
2D Splat buffer Allocated per point cloud STORAGE | COPY_SRC | COPY_DST | INDIRECT Screen-space splat payload plus sort keys (stride = 32 bytes).
Draw uniforms Allocated per point cloud UNIFORM | COPY_DST [numPoints, shDegree, …] used by preprocess kernels.
Model params Allocated per point cloud UNIFORM | COPY_DST 128-byte block with transform, offsets, shading knobs, and precision metadata.

Model Parameter Layout (128 bytes)

Byte Range Field
0 – 63 4×4 transform matrix (column-major).
64 – 67 baseOffset used when the preprocessor packs multiple point clouds.
68 – 71 numPoints mirror for shaders.
72 – 75 gaussianScaling.
76 – 79 maxShDeg.
80 – 83 kernelSize.
84 – 87 opacityScale.
88 – 91 cutoffScale.
92 – 95 rendermode.
96 – 99 gaussDataType (0=f32, 1=f16, 2=i8, 3=u8).
100 – 103 colorDataType.
104 – 107 gaussScale (for quantized decoding).
108 – 111 gaussZeroPoint.
112 – 115 colorScale.
116 – 119 colorZeroPoint.
120 – 127 Reserved/padding for future metadata.

Usage Pattern

import { PointCloud } from 'src/point_cloud';

const gaussianData = await defaultLoader.loadFile(file); // IO module
const pc = new PointCloud(device, gaussianData);

pc.updateModelParamsBuffer(modelMatrix);
pc.setGaussianScaling(1.2);

computePass.setBindGroup(0, pc.bindGroup());
renderPass.setBindGroup(0, pc.renderBindGroup());

For dynamic sources:

import { DynamicPointCloud } from 'src/point_cloud';

const dpc = new DynamicPointCloud(device, gaussianGPU, shGPU, maxPoints, countBuffer, 48, precisionInfo);
dpc.setOnnxGenerator(onnxGenerator);
dpc.setTimeUpdateMode(TimeUpdateMode.VARIABLE_DELTA);
await dpc.update(cameraMatrix, modelMatrix, performance.now());

Integration Points

  • IO Module – Supplies the GenericGaussianPointCloudTS instances used by the constructor.
  • Preprocess Module – Uses bindGroup() to read Gaussians/SH data and write projected splats.
  • Sort Module – Consumes the shared 2D splat buffer and indirect draw metadata.
  • Renderer Module – Uses renderBindGroup() to sample projected splats.
  • App/Managers – Adjust runtime parameters (scaling, kernel size, render mode) and trigger updateModelParamsBuffer() when transforms change.
  • ONNX/Tensor Path – Feeds GPU buffers plus precision metadata into DynamicPointCloud.

File Structure

src/point_cloud/
├── index.ts                # Barrel exports
├── point_cloud.ts          # PointCloud class
├── dynamic_point_cloud.ts  # DynamicPointCloud subclass
├── layouts.ts              # Bind group layouts + buffer config
└── splat_buffer.ts         # Interfaces mirroring GPU layouts
  • Architecture – Buffer layouts, bind group design, and GPU memory choreography.
  • API Reference – Constructor signatures, update hooks, and dynamic timeline flags.
  • Preprocess Module – Shows how splat data is projected before sorting.
  • Sorting Module – Explains how projected splats are ordered for rendering.
  • Renderer Module – Details how the shared buffers feed final draw calls.