In today’s world, real-time inference has become the gold standard in AI interactions. Consumers expect instantaneous responses, whether they’re typing a question into a chatbot or expecting an AI-generated code review. This demand means that infrastructure is tailored for those on the other end, constantly waiting for a reply. However, every millisecond of latency translates to a potential loss, as current serving stacks prioritize cold-start issues over maximizing throughput. It’s a delicate dance of efficiency versus immediacy.
During our journey at Theory, where we developed various internal AI systems, we recognized the power and efficiency of embracing queueing. By deploying ten agents to work on a single task, we found that the productivity gains were astronomical. This approach, termed token-maxxing, allows for the fullest utilization of computational resources, maximizing the output for every dollar spent. Yet, while this method yields impressive results, it also poses the challenge of unsustainable costs if not managed correctly.
Our turning point came with the introduction of Neil Movva and Samir Menon from Sail Research. Neil previously developed one of the fastest LLM inference stacks at Together AI, while Samir specialized in running LLMs within hardware enclaves at Blyss. These two systems engineers understood the intricacies of our needs and began constructing the very system that would revolutionize our approach.
As the inference market diversifies into categories like real-time, near-real-time, and batch processing, asynchronous inference has carved a niche within the batch tier, offering significant cost advantages. The essence of maximizing efficiency lies in astute model selection and routing, ensuring the best model is applied to each task.
Sail’s innovative approach focuses on distributing requests across open models, such as DeepSeek, Qwen, Kimi, and GLM. It dynamically selects the most cost-effective model for each task’s requirements. Remarkably, using GLM-5.1 on Sail can yield savings of up to 6x per token compared to other heavyweights like Anthropic’s Haiku. Thus, choosing to wait two minutes for a code review can be a wise decision when considering the cost savings involved.
When it comes to operational efficiency, Sail optimally uses available spot capacity, failing over to more reliable computation resources when necessary. This fleet-aware orchestration helps in maintaining high utilization while keeping costs low, a vital consideration in today’s competitive landscape.
Real-time inference stacks typically reserve dedicated capacity for each request, which can become inefficient. In contrast, queued stacks make the most of idle capacity, illustrating how different architectural choices lead to varied economic outcomes. With Sail’s architecture, tasks can flow seamlessly without the burden of excessive costs.
The concept of Sailboxes represents a breakthrough in managing computational demand. These cloud computers adjust to the working rhythm of agents, remaining active as long as needed while preserving state across tasks. They can pause during inference waits, resuming almost instantaneously once responses arrive. This means users only pay for active time, eliminating charges for idle periods—a practical advantage in any operation.
To date, Sail has processed trillions of tokens across various applications, including code reviews, deep research, and cybersecurity, showcasing its impressive reach and efficiency. Recognizing the transformative potential of such technology, we proudly announced our Series A investment in Sail, in partnership with industry leaders like Kleiner Perkins, Redpoint, and Sequoia.
As AI agents evolve from simple chat assistants into sophisticated background workers—capable of scanning entire codebases overnight or enriching CRM databases—the future of AI not only resides in real-time interactions but also in efficient background processes. We are excited to collaborate with Neil, Samir, and the talented Sail team to shape that future.
If you’re venturing into agent-based systems, get started here to explore the possibilities.