Skip to content

1. Foundations

Visionary operates based on the WebGPU environment and ONNX Runtime (ORT). This chapter introduces how to correctly initialize the environment and load resources to build the fundamental rendering pipeline.

1.1 Environment Initialization

1.1.1 Configure ONNX Runtime (ORT)

📎 Related Modules: 17-config module explains the ORT resource paths and runtime environment configuration; for WebGPU-based ORT scheduling, refer to 13-onnx module.

Visionary uses onnxruntime-web (ORT) to handle the decoding and post-processing of Gaussian Splatting. Since ORT relies on .wasm files (such as ort-wasm.wasm, ort-wasm-simd.wasm) for high-performance computing, the paths to these static resources must be explicitly specified before loading models.

Interface Source: src/config/ort-config

/**
 * Initialize ONNX Runtime environment configuration.
 * @param wasmPaths - (Optional) The directory path where WASM files are stored. Defaults to '/src/ort/'.
 *                    Can be a single string path or an array of paths.
 */
function initOrtEnvironment(wasmPaths?: string | string[]): void

Usage Example: It is recommended to call this at the application entry point (e.g., main.ts or during the component mounted phase):

import { initOrtEnvironment } from 'src/config/ort-config';

// 1. Define the static resource directory (Ensure ort-wasm.wasm etc. exist here after build)
const wasmPath = '/static/ort/'; 

// 2. Initialize the environment
initOrtEnvironment(wasmPath);

Feature Note: initOrtEnvironment is fault-tolerant. If window.ort is not yet loaded when called, it initiates a polling mechanism (checking every 50ms) until ORT is ready, at which point the configuration is automatically applied.

1.1.2 Create WebGPU Rendering Context

📎 Related Modules: The low-level implementation of initThreeContext lives in the WebGPUContext of the 01-app module; see the 12-three-integration module for additional renderer-integration details.

initThreeContext is the core function to start the engine. It is responsible for initializing the THREE.WebGPURenderer and synchronously lifting the underlying WebGPU Device so it can be shared between the rendering engine and the inference engine.

Interface Source: src/app/three-context

⚠️ Critical Prerequisites: The file /models/onnx_dummy.onnx must exist in the project's public directory. This is a minimal ONNX file used to trigger the initialization process of the ORT WebGPU backend.

/**
 * Initialize the WebGPU Renderer
 * @param canvasElement - The Canvas DOM element used for rendering
 * @returns Promise<THREE.WebGPURenderer | null>
 */
async function initThreeContext(canvasElement: HTMLCanvasElement): Promise<THREE.WebGPURenderer | null>

Default Configuration Behavior: * Device Sharing: Enforces sharing of gpu.device and gpu.context, enabling zero-copy interaction between the rendering engine and the inference engine. * Performance Mode: adapterPowerPreference is set to 'high-performance'. * Clear Color: Defaults to gray (#808080); it is recommended to modify this as needed after initialization. * Pixel Ratio: Automatically capped at a maximum of 2.

Usage Example:

import { initThreeContext } from 'src/app/three-context';

const canvas = document.getElementById('canvas') as HTMLCanvasElement;

try {
    const renderer = await initThreeContext(canvas);

    if (!renderer) {
        throw new Error("WebGPU initialization failed. Check browser compatibility or onnx_dummy.onnx path.");
    }

    // Set transparent background to blend with UI
    renderer.setClearColor(0x000000, 0);

    // Start rendering loop...
} catch (e) {
    console.error("Context Error:", e);
}

1.2 Unified Asset Loading

📎 Related Modules: The unified loader is implemented by the FileLoader/GaussianLoader within the 14-managers module and relies on the low-level format parsing provided by the 02-io module.

Visionary encapsulates a smart loader, loadUnifiedModel, which supports automatic identification and loading of Mesh and Gaussian Splatting models.

Interface Source: src/app/unified-model-loader

1.2.1 Core Interface Definition

// Supported model type strings
type ModelType = 'ply' | 'onnx' | 'gltf' | 'glb' | 'obj' | 'fbx' | 'gaussian';

interface UnifiedLoadOptions {
    /** Explicitly specify type. If unspecified, it is detected via extension or header sniffing. */
    type?: ModelType;

    /** Model name */
    name?: string;

    /** Flag for Gaussian models */
    isGaussian?: boolean;

    /** [Gaussian Only] Initial camera view matrix (used for initial particle sorting) */
    cameraMatrix?: Float32Array;

    /** [Gaussian Only] Initial projection matrix */
    projectionMatrix?: Float32Array;

    /** Progress callback (0.0 - 1.0) */
    onProgress?: (progress: number) => void;
}

interface LoadResult {
    /** 
     * List of loaded model objects.
     * - Standard Model: Mesh or Group
     * - Gaussian: Proxy Object3D containing Splat data
     */
    models: THREE.Object3D[];

    /** 
     * [Critical] Gaussian-specific renderer instance.
     * Only exists when the loaded content includes Gaussian Splatting.
     */
    gaussianRenderer?: GaussianThreeJSRenderer;

    info: {
        type: ModelType;
        name: string;
        isGaussian: boolean;
    };
}

1.2.2 Smart Detection Logic

The loader has built-in ambiguity detection for .ply files: 1. Extension Detection: .compressed.ply, .splat, .onnx etc. are treated directly as Gaussian. 2. Content Sniffing: * Local File: Reads the first 4KB to check for feature fields like rot_0, scale_0. * Online URL: Initiates a Range: bytes=0-4095 request to pre-read the file header, determining the type without downloading the large file.

1.2.3 Usage Examples

Scenario A: Loading Mesh 3D Models

The loader parses Mesh 3D models (e.g., .glb, .gltf, .fbx, .obj) into standard THREE.Object3D (usually THREE.Mesh or THREE.Group) and automatically adds them to the scene.

import { loadUnifiedModel } from 'src/app/unified-model-loader';

const result = await loadUnifiedModel(renderer, scene, '/models/your_model.glb');
const mesh = result.models[0]; 
mesh.position.set(0, 0, 0);

Scenario B: Loading Gaussian Splatting

When loading Gaussian models, it is recommended to pass the current camera matrices to ensure correct sorting for the first frame.

import { loadUnifiedModel } from 'src/app/unified-model-loader';

// 1. Prepare matrices
camera.updateMatrixWorld();
const camMat = new Float32Array(camera.matrixWorldInverse.elements);
const projMat = new Float32Array(camera.projectionMatrix.elements);

// 2. Load model
const result = await loadUnifiedModel(renderer, scene, '/models/splat_model.ply', {
    cameraMatrix: camMat,
    projectionMatrix: projMat,
    name: 'MySplat'
});

// 3. Handle renderer reference
if (result.gaussianRenderer) {
    // You MUST save this reference and call its drawSplats method in the animation loop
    this.gaussianThreeJSRenderer = result.gaussianRenderer;

    // If it's the first model, it typically init automatically; 
    // if appending, note the renderer reuse logic.
    console.log('Gaussian Renderer Ready:', this.gaussianThreeJSRenderer.uuid);
}

1.2.4 Gaussian Renderer Management & Reuse

📎 Related Modules: The Gaussian rendering pipeline is defined in the 12-three-integration module, and the GaussianModel data structures plus sync logic are documented in the 16-models module.

Unlike standard Mesh models, Gaussian Splatting models require a special GaussianThreeJSRenderer for sorting and rasterization. This renderer consumes significant GPU resources (Buffers, Worker threads).

The design principle of loadUnifiedModel is stateless—it returns a new gaussianRenderer instance every time it is called.

To avoid performance crashes, do not run multiple Gaussian Renderers in the scene simultaneously. Instead, implement "reuse logic" at the application layer:

  1. First Model: Save the renderer instance returned by the loader.
  2. Subsequent Models: Append the new model to the existing renderer and discard the redundant renderer returned by the loader.
import { loadUnifiedModel } from 'src/app/unified-model-loader';
import { GaussianThreeJSRenderer } from 'src/app/GaussianThreeJSRenderer';

// Define a member variable to hold the unique renderer reference
private activeGaussianRenderer: GaussianThreeJSRenderer | null = null;

async function handleModelLoad(url: string) {
    const result = await loadUnifiedModel(renderer, scene, url);

    // 1. Handle standard Mesh models
    // ...

    // 2. Handle Gaussian models
    if (result.gaussianRenderer && result.models.length > 0) {
        const newSplatModel = result.models[0];

        if (!this.activeGaussianRenderer) {
            // [Scenario A: Initialization] 
            // No active renderer, use the instance returned by the loader
            this.activeGaussianRenderer = result.gaussianRenderer;

        } else {
            // [Scenario B: Appending] 
            // Renderer exists, "merge" the new model into it
            this.activeGaussianRenderer.appendGaussianModel(newSplatModel);

            // CRITICAL: Dispose of the redundant temporary renderer created by the loader
            // to release WebGPU resources.
            if (typeof result.gaussianRenderer.dispose === 'function') {
                result.gaussianRenderer.dispose();
            }
        }
    }
}