Deltafin is an open-source research project enabling the execution of large Mixture-of-Experts models on consumer hardware by utilizing efficient streaming, quantization, and native kernel optimizations for local inference.

Key Points

• Supports Apple Silicon macOS and x86-64/aarch64 Linux with automatic hardware acceleration via MPS, CUDA, or CPU.
• Implements a Mixture-of-Experts architecture that requires approximately 1.7 TB for a full local installation or 215 GB for streaming mode.
• Achieves a median inference speed of 14.6 seconds per token on a 64 GB M1 Max by using int8 quantization and expert-caching techniques.
• Provides an OpenAI-compatible API server, allowing integration with standard coding agents and chat interfaces.
• Uses n-gram speculation and layer-by-layer loading to minimize memory overhead and improve throughput on resource-constrained machines.

Why it Matters

This project demonstrates that massive, high-parameter models can run on standard workstations by prioritizing I/O efficiency and intelligent memory management over raw compute power. It serves as a critical existence proof for local AI, offering a blueprint for developers to run complex models without relying on expensive cloud-based infrastructure.