Kog has launched a high-speed inference engine that optimizes GPU software stacks to achieve 3,000 tokens per second for single-request AI agent workloads on standard datacenter hardware.

Key Points

• The Kog Inference Engine achieves high speeds by replacing standard multi-kernel stacks with a "monokernel" that runs as a single, persistent GPU program.
• Performance is bottlenecked by memory bandwidth rather than compute power, allowing 8-GPU nodes to reach speeds previously reserved for dedicated inference hardware.
• The company’s "Laneformer" architecture uses Delayed Tensor Parallelism to overlap cross-device communication with computation, reducing latency in multi-GPU setups.
• A public tech preview is available at playground.kog.ai, demonstrating the engine's capabilities using a 2B-parameter coding model on an 8x AMD MI300X node.
• Kog is a Paris-based startup founded in 2023 that has raised $5 million to focus on low-level GPU engineering for autonomous AI agents.

Why it Matters

As AI agents shift from simple chatbots to complex, iterative systems, the speed of single-request token generation has become the primary bottleneck for productivity. By unlocking the full memory bandwidth of existing datacenter GPUs, this technology enables faster reasoning and tool-use loops without requiring proprietary inference hardware.