Models Architecture

Overview

The models module provides the data layer for all model types in Visionary. It defines unified interfaces that work across Gaussian Splatting formats (PLY, SPZ, KSplat, SPLAT, SOG, Compressed PLY), ONNX dynamic models, and FBX mesh files. The module is interface-driven with clear separation between runtime state and UI-friendly snapshots.

Model Interfaces (ModelEntry, ModelInfo)
│
├── ModelManager (in app/managers) - registry, lookup, visibility
├── FileLoader (in app/managers) - IO pipeline, format detection
├── ONNXManager (in app/managers) - ONNX model lifecycle
└── Wrappers (FBXModelWrapper) - format-specific functionality

Core Interfaces

ModelEntry

Runtime representation of a loaded model. Contains full references to pointCloud/object3D and all metadata.

interface ModelEntry {
  id: string;                                    // Unique identifier (auto-generated)
  name: string;                                   // Display name
  visible: boolean;                               // Render visibility flag
  pointCloud: PointCloud | DynamicPointCloud | FBXModelWrapper;  // Format-specific data
  pointCount: number;                             // Total point/vertex count
  isDynamic: boolean;                             // True for ONNX models
  modelType: ModelType;                           // Format type
  colorMode?: 'sh' | 'rgb';                       // Color encoding mode
  colorChannels?: number;                         // Number of color channels
}

Key Points:

pointCloud can be PointCloud (static Gaussian), DynamicPointCloud (ONNX), or FBXModelWrapper (FBX mesh)
modelType supports: 'ply' | 'spz' | 'ksplat' | 'splat' | 'sog' | 'compressed.ply' | 'onnx' | 'fbx'
isDynamic distinguishes streaming point clouds (ONNX) from static assets
colorMode and colorChannels allow UI to display color metadata consistently

ModelInfo

Lightweight, read-only snapshot for UI layers. Does not include pointCloud reference to avoid holding large object graphs.

interface ModelInfo {
  id: string;
  name: string;
  visible: boolean;
  pointCount: number;
  isDynamic: boolean;
  modelType: ModelType;
  colorMode?: 'sh' | 'rgb';
  colorChannels?: number;
}

Use Case: Safe for React rendering, UI lists, and serialization. ModelManager.getModels() returns ModelInfo[] for UI consumption.

ModelType

Union type for all supported model formats:

type ModelType = 
  | 'ply' | 'spz' | 'ksplat' | 'splat' | 'sog' | 'compressed.ply'  // Gaussian formats
  | 'onnx'                                                          // Dynamic ONNX
  | 'fbx';                                                          // FBX mesh

Integration with Managers

The models module interfaces are used by managers in the app/managers module:

ModelManager Integration

ModelManager (in app/managers) uses ModelEntry and ModelInfo:

Maintains an array of ModelEntry instances
Provides getModels(): ModelInfo[] for UI consumption
Provides getFullModels(): ModelEntry[] for debugging/testing
Handles model registration, removal, and queries
Tracks visibility, capacity limits, and statistics

FileLoader Integration

FileLoader (in app/managers) creates ModelEntry instances:

Loads Gaussian formats (PLY, SPZ, KSplat, SPLAT, SOG, Compressed PLY) → creates ModelEntry with PointCloud
Loads FBX files → creates ModelEntry with FBXModelWrapper
Uses ModelManager.addModel() to register entries

ONNXManager Integration

ONNXManager (in app/managers) creates dynamic model entries:

Loads ONNX models → creates ModelEntry with DynamicPointCloud
Sets isDynamic: true for ONNX entries
Manages ONNX generator lifecycle

FBXModelWrapper

The FBXModelWrapper class wraps Three.js FBX assets and implements ITimelineTarget for timeline integration.

Features

Three.js Integration: Wraps THREE.Group and THREE.AnimationMixer
Animation Support: Full animation control (play, pause, stop, speed, time)
Timeline Integration: Implements ITimelineTarget interface
Transform Support: 4x4 transform matrix with setTransform() method
Visibility Control: setVisible() / getVisible() methods
Vertex Counting: getVertexCount() for statistics

Animation Control

class FBXModelWrapper implements ITimelineTarget {
  // Timeline interface
  setAnimationTime(time: number): void;
  setAnimationSpeed(speed: number): void;
  getAnimationSpeed(): number;
  startAnimation(speed?: number): void;
  pauseAnimation(): void;
  resumeAnimation(): void;
  stopAnimation(): void;
  setTimeScale(scale: number): void;
  setTimeOffset(offset: number): void;
  setTimeUpdateMode(mode: 'fixed_delta' | 'variable_delta'): void;
  getCurrentTime(): number;
  update(deltaTime: number): void;

  // Animation management
  switchToClip(clipIndex: number): boolean;
  getClipInfo(): Array<{name: string, duration: number}>;
  supportsAnimation(): boolean;
}

FBXLoadOptions

interface FBXLoadOptions {
  autoPlay?: boolean;        // Auto-play first animation
  defaultSpeed?: number;      // Default playback speed
  loop?: boolean;             // Loop animation
  blendMode?: 'normal' | 'additive';  // Animation blend mode
}

Wrappers

FBXModelWrapper – loads FBX assets, exposes materials, animations, and hierarchy utilities.
Additional wrappers follow the same pattern for GLTF, OBJ, ONNX-derived models, etc.
Wrappers can expose convenience methods (play animation, reload textures) while keeping base contracts intact.

Data Flow

Loading Flow

File Input → FileLoader.loadFile() or FileLoader.loadSample()
Format Detection → Detects format (Gaussian, ONNX, FBX)
Data Loading → Uses IO module's defaultLoader or FBXLoaderManager
ModelEntry Creation → Creates ModelEntry with appropriate pointCloud/wrapper
Registration → ModelManager.addModel() registers the entry
UI Update → ModelManager.getModels() returns ModelInfo[] for UI

ONNX Flow

ONNX Load → ONNXManager.loadONNXModel()
DynamicPointCloud Creation → Creates DynamicPointCloud with ONNX generator
ModelEntry Creation → Creates ModelEntry with isDynamic: true
Registration → ModelManager.addModel() registers the entry
Animation Updates → AnimationManager updates dynamic models each frame

FBX Flow

FBX Load → FBXLoaderManager.loadFromFile()
FBXModelWrapper Creation → Creates wrapper with animation mixer
ModelEntry Creation → Creates ModelEntry with FBXModelWrapper
Registration → ModelManager.addModel() registers the entry
Animation Control → Timeline or direct animation methods control playback

Extensibility

Adding New Formats

Extend ModelType: Add new format to ModelType union type
Create Loader: Implement loader in app/managers (or use IO module)
Create Wrapper (optional): If format needs special handling, create wrapper class
Update FileLoader: Add format detection and loading logic
Test: Ensure ModelEntry creation works correctly

Example: Adding GLTF Support

// 1. Extend ModelType
type ModelType = ... | 'gltf' | 'glb';

// 2. Create wrapper (if needed)
class GLTFModelWrapper {
  // ... implementation
}

// 3. Update ModelEntry pointCloud type
pointCloud: PointCloud | DynamicPointCloud | FBXModelWrapper | GLTFModelWrapper;

// 4. Update FileLoader to handle .gltf/.glb

Performance & Memory

ModelEntry holds full pointCloud reference (use for runtime operations)
ModelInfo is lightweight (use for UI lists and serialization)
Disposal: Call dispose() on wrappers and point clouds when removing models
Capacity Limits: ModelManager enforces max model count (default 10000)
Statistics: Track point counts and visibility for performance monitoring

Best Practices

Use ModelInfo for UI: Prefer ModelInfo for React components and UI lists
Use ModelEntry for Operations: Use ModelEntry when you need pointCloud reference
Type Guards: Check isDynamic before accessing dynamic-specific features
Format Detection: Use modelType to determine available operations
Memory Management: Always dispose wrappers and point clouds when removing models

This architecture keeps model ingestion predictable while remaining flexible enough for complex pipelines (mixed formats, dynamic ONNX content, FBX animations, and editor tooling).