How to Slash AI Token Costs by 80%? - by using LiteLLM as an Intelligent Router

As AI features scale in production, engineering teams inevitably hit a financial wall: token expenses. Defaulting to frontier models for every request is the equivalent of commuting across town in a fleet of helicopters. It gets you there, but it is wildly expensive and unnecessary.

A vast majority of everyday production workloads like classifying intent, extracting structured JSON, or formatting text do not require top-tier multi-billion parameter models. By positioning LiteLLM Proxy as an AI Router between your client applications and downstream providers, you can dynamically route prompts to the cheapest, fastest model capable of handling the task.

We can configure LiteLLM as an AI router using static rules, SLM-powered dynamic routing, and custom Python logic, while pairing the right tasks with the right models.

One common strategy is to create a task-to-model mapping matrix like the one below. 

Once you have this mapping, you can use static rules or SLM-powered dynamic routing to implement this. 

Static Routing

Static routing rules let you define deterministic paths based on model aliases, token lengths, user authorization tiers, or specific headers. Think of static routing like a fixed highway detour sign or a strict set of fixed rules. It doesn't read or "understand" the actual meaning of a message; it simply follows pre-set instructions to decide where to send it.

Here is how it works:

• Fixed Map Rules: You manually tell the system, "If a request comes from User A, send it to Model A. If it comes from User B, send it to Model B."
• Fixed Keyword Matching: You write pre-set patterns ahead of time (e.g., "If the text contains SELECT or JOIN, send to the SQL Model")
• Backup Safety Net: It can act as an automatic backup. You can set a rule that says: "Always try Provider A first. If Provider A is broken or busy, automatically send the exact same request to Provider B."

Because these rules are written in advance and never change on their own, they are extremely fast and cheap to run, but they aren't smart enough to adapt to what the user is actually talking about.

Dynamic Routing using SLMs

Static rules based solely on character limits or headers miss nuance that a short prompt can still demand complex mathematical logic. 

To route based on semantic complexity, you can introduce an SLM (Small Language Model) node (e.g., Llama-3-8B or Phi-3) directly before the main router call. The SLM evaluates the prompt and tries to assign an intent score. 

The SLM evaluates difficulty on a scale of 1 to 5 within ~30–50ms.

• Score 1–2 (Routine): Dispatched to Tier 1 (gpt-4o-mini / llama-3-8B).
• Score 3 (Moderate): Dispatched to Tier 2 (gemini-pro).
• Score 4–5 (High Complexity): Dispatched to Tier 3 (frontier models).

Some of the popular SLMs for dynamic routing are given below: 

If an 8B SLM adds too much latency to your application, many high-volume production systems use non-generative models as routers instead:

• BERT-based Classifiers (e.g., RoBERTa): Fine-tuned specifically to predict prompt complexity or score query difficulty. These return decisions in under 10 milliseconds.
• Text Embedding Models (e.g., text-embedding-3-small, BGE-Small): Used for semantic routing. The prompt is converted into a mathematical vector and instantly matched against pre-indexed topic clusters (e.g., matching SQL terms to a database specialized LLM).

Dynamic Routing using custom code

When pre-built routing algorithms do not fit your specific pipeline, LiteLLM allows custom programmatic rules using Python callbacks. You can write custom Python code that inspects the request payloads, detects code blocks via regex, and dynamically reassigns target models.

Thus we can achieve the key benefits of an AI Gateway/Router:

• Massive Cost Savings: Route up to 80% of routine traffic away from expensive flagship models to low-cost alternatives.
• Improved Throughput & Speed: SLMs and lightweight models generate first-token responses in a fraction of the time required by high-parameter reasoning models.
• High Availability & Resilience: Built-in failover capabilities ensure client applications remain operational even during provider outages.

Ruminating on Agent Skills

 The best place to understand agent skills is through the official open standard website - https://agentskills.io/home. This site gives a very clear understanding of the agent skills specification and should be your first go-to place to start learning about skills. 

A Skill is nothing a portable package (aka folder) of instructions, scripts, and resources that gives an AI agent specialized capabilities and domain expertise. Skills also use progressive disclosure - The agent discovers what skills exist, then loads only the ones relevant to the current task, and only reads the parts it needs at that moment. In practice, this means:

• Your agent doesn’t carry all your company’s policies, patterns, and examples in every call.
• It loads just enough context to do the job well, then responds.
• That makes Skills ideal for enterprise and public-sector use, where you want governed, auditable, reusable capability packs rather than giant, monolithic prompts.

Think of Skills as modular, AI-readable playbooks. They help you:

• Standardise behaviour: Every agent that uses the “Azure Functions” skill follows the same patterns and conventions.
• Scale expertise: You write the skill once, then many agents (Copilot, custom agents, internal tools) can use it.
• Keep context lean: The agent loads only the relevant skill(s) for the task, reducing noise and cost.
• Separate concerns: Skills isolate domain knowledge from core agent logic, making both easier to maintain.

A typical Skill folder structure will look like this: 

Skills and MCP serve complementary roles in agentic AI. MCP (Model Context Protocol) is the connectivity layer: it standardises how an AI application discovers and invokes external tools, APIs, data sources, and services. Skills are reusable procedural instructions that explain how an agent should approach a task, including workflows, domain rules, sequencing, and best practices. 

In simple terms, MCP provides the capabilities, while Skills provide the orchestration—for example, an MCP server could expose CRM tools, while a “customer-onboarding” Skill instructs the agent how to validate information, apply policy, call those tools, and handle exceptions

Ruminating on Human in the Loop (HITL) vs Human on the Loop (HOTL)

 As AI systems become more embedded in enterprise workflows, the conversation is no longer just about capability—it’s about control. Two models often come up in this discussion: 

• Human in the Loop (HITL) 
• Human on the Loop (HOTL)

While they sound similar, they represent fundamentally different approaches to how humans interact with AI systems.

Understanding this distinction is critical for designing safe, scalable, and efficient AI-driven processes.

Human in the Loop (HITL): Control Before Action

In the HITL model, humans are directly embedded in the decision-making process. The AI generates outputs, but execution depends on explicit human approval or validation.  

This model is best suited for:

• High-risk decisions (financial transactions, compliance approvals)
• Low-confidence AI outputs
• Regulatory or audit-heavy environments

Think of HITL as a gated workflow: the AI proposes, but the human disposes.

For example, in an ERP system like Oracle Fusion, an AI might recommend vendor payments or flag anomalies—but a finance controller must approve before funds are released. This ensures accountability and reduces the risk of automation errors propagating into real-world impact.

The trade-off is clear: higher reliability and governance, but reduced speed and scalability.

Human on the Loop (HOTL): Control Through Oversight

HOTL shifts the paradigm. Here, AI systems operate autonomously, making decisions and executing actions without requiring prior human approval. Humans remain in a supervisory role and can intervene when necessary. 

This model is ideal for:

• High-volume, repetitive tasks
• Real-time decision environments
• Mature AI systems with proven accuracy

In this setup, the human is not blocking the process—they are monitoring it. A good example is automated fraud detection. An AI system might automatically block suspicious transactions in real time, while human analysts review flagged patterns and adjust thresholds or intervene in edge cases. The system moves fast, but oversight ensures it doesn’t drift into unsafe behavior. 

The trade-off here flips: speed and scalability increase, but it requires strong monitoring, alerting, and fallback mechanisms.

Confusing HITL and HOTL can lead to poorly designed systems. Overusing HITL creates bottlenecks and defeats the purpose of automation. Overusing HOTL without proper guardrails can introduce silent failures at scale.

The real design challenge is deciding:

• When does AI need approval?
• When can it act independently?
• How do we transition from HITL to HOTL as confidence grows?

This is where concepts like confidence thresholds, risk scoring, and progressive autonomy come into play.

The Two-Model Perspective

Another way to interpret this “bi-modal” structure is through a two-model system:

• A decision model that performs the task (e.g., classification, prediction, action)
• A governance model that determines whether human intervention is required

For instance, an AI might assign a confidence score to its output. If the score is below a defined threshold, the system routes the task into a HITL flow. If it exceeds the threshold, it proceeds autonomously under HOTL. This layered approach allows organizations to dynamically balance risk and efficiency, rather than hardcoding one model across all scenarios.

Effective AI governance will increasingly rely on:

• Dynamic switching between HITL and HOTL
• Real-time monitoring and explainability
• Feedback loops that continuously improve both models

Organizations that get this right will not only scale AI faster but also build trust in its decisions. In the end, the question is not whether humans should be involved—it’s how and when.

Ruminating on 'Service as Software'

The evolution of Agentic AI has ushered in a new paradigm called 'Service as Software'. The following article lays down the core concept very succinctly:  https://www.thoughtful.ai/blog/service-as-software

Snippet from the article:

Service as a Software (SaS) is a business model where the value isn’t the software itself, but the complete service it delivers through automation. Instead of selling access to tools, companies sell the outcome those tools create. In SaS, the software operates in the background while AI Agents perform the actual work—handling tasks, making decisions, and producing results without human input. This removes the need for the customer to operate the software, train staff, or manage workflows—the outcome is what’s sold, not the tool.

The biggest advantage of this paradigm shift towards Digital Labour is infinite scalability & real-time agility. Traditional services scale linearly—hire more people for more demand, pray for retention. SaS agents? They ramp up instantly. Seasonal surges in e-commerce support? No frantic recruiting. Project deadlines looming? Agents multiply efforts without overtime. A customer support team, for instance, swaps human shifts for AI deployments that triage queries, resolve 80% on the spot, and escalate the rest seamlessly—slashing costs while boosting satisfaction. Pricing seals the deal: Move beyond flat fees to outcome-based models. Charge per resolved ticket, qualified lead, or audit completed. 

JSON-RPC vs REST and why JSON-RPC is used in MCP?

I was going down the rabbit hole of MCP protocol details and realised that it was using JSON-RPC instead of REST. 

JSON-RPC is a simple protocol that lets a program on one computer run a function on another computer. It uses JSON to send and receive the requests and responses, making it easy to use and understand.

It is transport-agnostic and can work over HTTP, TCP, sockets, or other message-passing environments. A JSON-RPC request typically includes the method to be called, parameters for that method (optional), and an ID to match the response.

Given below is a simple example of a request and response:

Given below are the top 3 differences in JSON-RPC vs REST for API design:

Architecture Style:

• JSON-RPC: RPC-oriented, focusing on invoking specific methods or procedures on the server (e.g., calling a function like getBalance()). It treats interactions as direct commands.
• REST: Resource-oriented, centered on manipulating resources (e.g., /users/{id}) using standard HTTP methods like GET, POST, PUT, and DELETE.

Endpoints and HTTP Methods:

• JSON-RPC: Uses a single endpoint (e.g., /rpc) with POST requests for all method calls, simplifying routing but limiting HTTP verb usage.
• REST: Employs multiple endpoints (e.g., /users, /orders) and leverages various HTTP methods (GET, POST, PUT, DELETE) to represent different actions on resources.

Request/Response Structure:

• JSON-RPC: Requests and responses follow a strict JSON format with fields like "method", "params", "id", and "result" or "error". Supports batching natively.
• REST: Uses flexible request formats (URL paths, query parameters, headers) and responses rely on HTTP status codes (e.g., 200, 404) with custom payloads; batching requires custom implementation.

MCP chose JSON-RPC because its lightweight, single-endpoint design ensures fast and efficient communication for real-time AI tasks. It supports batch requests, allowing multiple operations in one call, which suits MCP’s complex AI workflows.