Rethinking AI Architecture: The Case for Prioritizing Routing Over Model Selection

In the fast-evolving landscape of AI, many teams make the critical mistake of placing their model selection at the forefront of their decision-making process. However, the emphasis should be on routing before settling on the model. A well-thought-out routing mechanism is key to optimizing performance and cost, often resulting in substantial savings and efficiency gains. Let’s delve into how this pivot can transform the way we handle AI-driven applications.

The Role of Routing in AI Spend

Brian Armstrong’s recent insights on Coinbase provide an illustrative example of this paradigm shift. He highlighted how the company reduced its AI expenditure by half, all while witnessing a boom in token usage. The magic formula? Not cumbersome spend alerts or growth-inhibiting friction, but rather enhanced defaults, improved routing, and effective caching. Simply put, engineers should have the autonomy to choose any model they prefer, but the defaults they work with significantly impact outcomes.

The Three Layers of the Routing Problem

At the heart of effective routing are three distinct layers, each designed to tackle a specific aspect of the task at hand. Understanding this layered approach is vital for teams looking to optimize their AI architecture.

1. Skill Classifier: This component serves as the initial gatekeeper, transforming raw user requests into concrete actions. It discerns intent, answering questions like, “What task am I performing?” Whether it’s drafting a reply, summarizing a repository, or executing a migration, the classifier pinpoints the operation needed, setting the stage for efficient processing.

2. Router: Next comes the router, which decides the tier of execution for the classified operation. Importantly, the router does not interpret the user prompt itself; instead, it utilizes the classifier’s output along with a few additional features such as complexity, context size, and historical success rates. This streamlined decision-making process ensures that resources are allocated effectively, and the selected tier is best suited for the job.

3. Model Selector: Finally, we have the model selector, which identifies the most cost-effective model within the designated tier that still meets a required confidence threshold. This allows for the dynamic selection of models while ensuring that operational budgets are maintained.

Distinguishing Classifier from Router

A common misconception is conflating the roles of the classifier and the router. The classifier tackles a language problem—understanding intent—while the router deals with a scheduling problem, determining which resources will be utilized. This distinction is crucial because blending these two functionalities can obscure the model selection process and impede A/B testing efforts across different models for the same operation.

The Value of Local Compute and Asynchronous Processing

One compelling advantage of focusing on routing is the ability to leverage local compute options. Running local models is nearly free, and asynchronous batch reasoning offers an order of magnitude cost reduction compared to real-time inference. This raises an important question: How much of our workload genuinely requires immediate responses?

Surprisingly, the answer often points to a small fraction. When a system is appropriately designed to queue tasks, many operations—such as drafting replies or summarizing reports—do not require instantaneous results. Queueing transforms how we think about processing requests and opens avenues for more economical and efficient operational workflows.

Building an Efficient Router System

Our initial foray into this routing philosophy was integrated into our agent runtime, which evaluated tasks based on complexity, context size, and memory retrieval potential. Two distinct feedback mechanisms further enhance this architecture, operating on varying time scales to continually improve the routing decisions:

• Synchronous Failure-Mode Signals: Before a task execution begins, a predictor annotates it with crucial metadata—missing context, long dependencies, security-sensitive prompts, and potentially risky actions. This preemptive scrutiny helps in catching known challenging tasks beforehand.

• Nightly Closed-Loop Feedback: A batch evaluator examines past traces and refines the router’s decision-making framework overnight. Operating on asynchronous inference helps keep evaluation costs low, making this a sustainable strategy for constant improvement.

Emphasizing Skill Distillation

As more operations are distilled to their essence, it becomes clear that 70-80% of the agent traffic can effectively be handled by local models, particularly for routine tasks. This means that the focus should genuinely shift towards designing systems centered around routing, rather than getting bogged down in early-stage model selection.

By developing a robust routing strategy, teams can better manage workloads, drastically reduce operational costs, and enhance system performance. This transition emphasizes a paradigm where model selection becomes an afterthought—an informed decision rather than a starting point.