Apache Arrow vs Apache Iceberg

Many folks get confused between these two similar open standards; as both of them are used for columnar data. 

Apache Arrow defines how data should be arranged inside your computer's RAM while a program is actively running.Because it organizes data by column instead of by row in memory, modern computer processors can use hardware acceleration (like SIMD) to calculate millions of rows simultaneously. Arrow doesn't care about hard drives, cloud buckets, or transactions; its only job is to make sure your CPU or GPU can read and process active data as fast as physically possible.

Apache Iceberg does not define how data looks in RAM, nor does it define the raw file format. Instead, it sits on top of your storage (like Amazon S3 or Google Cloud Storage) and acts as a brilliant organizer for your files.When you save billions of rows of data, they get broken up into thousands of individual files (usually Apache Parquet files). Iceberg maintains a highly efficient "manifest" (a catalog) of exactly which files belong to which table. This allows engines to perform complex database operations like ACID transactions (ensuring data isn't corrupted during writes), time travel (querying what the data looked like last Tuesday), and schema evolution (renaming a column without rewriting the whole dataset).

In a modern data pipeline, Arrow and Iceberg complement each other. 

A typical workflow looks like this:

• Storage: Your massive dataset sits permanently on cloud storage as a collection of Parquet files, tracked and managed by Apache Iceberg.
• Loading: A query engine (like Dremio, DuckDB, or Snowflake) wants to read the data. It asks Iceberg which files it needs.
• Processing: As those files are read from the disk into the computer's RAM, the engine converts the data into the Apache Arrow format.
• Execution: Your Python script or analytical engine manipulates, filters, and analyzes the data instantly at the speed of RAM using Arrow's zero-copy mechanics.

Apache Arrow Zero Copy is a clever way for different software programs to share data instantly without wasting time moving it around. Normally, when one tool wants to send data to another, it has to convert that data into a special stream of code, send it over, and then the receiving tool has to translate it back before using it, which slows everything down. Arrow fixes this by setting up a universal, shared blueprint for how data is arranged inside a computer's memory. Because every program agrees on this exact layout, they don't need to copy, move, or translate anything; instead, one program simply points the other to the exact spot in the computer's RAM where the data is already sitting, allowing the second program to read it immediately at the speed of hardware.

This approach is highly utilized in modern data engineering tools (like Apache Spark, pandas, and Ray) to pass massive datasets between different libraries and languages at the speed of hardware RAM.

Ruminating on Mutuals

If you have ever opened a savings account, taken out a mortgage, or looked into health insurance in the UK, you have likely come across names like Nationwide or The Exeter. While they look like standard banks or insurance providers, they operate on a completely different business model: they are mutuals.

Most large businesses are owned by external shareholders. Their primary goal is to generate profit and pay out dividends to the shareholders. A mutual flips this model on its head. A mutual is an organization owned entirely by its customers (who are called members).

Because a mutual has no external shareholders to satisfy:

• Surplus profits are kept inside the business. 
• Money is funneled back to members through better interest rates, lower fees, expanded coverage, or direct cash payouts. 
• Members hold voting rights on how the organization is run.

"Mutual" is the umbrella term. In the UK, there are 2 prominent types of mutuals - i.e. building societies and friendly societies. 

A building society is a member-owned mutual financial institution that provides everyday banking, savings accounts, and mortgage lending. Its main purpose (in the past) was to help local communities pool their savings so members can borrow money to buy or build homes. Profits are usually reinvested to benefit members rather than paid to outside shareholders. The largest building society in the world is Nationwide, with other well-known examples including Yorkshire Building Society and Coventry Building Society.

A friendly society is a member-owned mutual insurer that provides cover for health, income protection, and long-term care. Long before the NHS or modern state welfare, workers would pay small weekly amounts into a shared fund so that if a member became sick, injured, or died, the society would pay their lost income or funeral costs. Today, examples include The Exeter, which specialises in health and income insurance, as well as other mutual providers such as Royal London and Liverpool Victoria.

How to Slash AI Token Costs by 80%? - by using LiteLLM as an Intelligent Router

As AI features scale in production, engineering teams inevitably hit a financial wall: token expenses. Defaulting to frontier models for every request is the equivalent of commuting across town in a fleet of helicopters. It gets you there, but it is wildly expensive and unnecessary.

A vast majority of everyday production workloads like classifying intent, extracting structured JSON, or formatting text do not require top-tier multi-billion parameter models. By positioning LiteLLM Proxy as an AI Router between your client applications and downstream providers, you can dynamically route prompts to the cheapest, fastest model capable of handling the task.

We can configure LiteLLM as an AI router using static rules, SLM-powered dynamic routing, and custom Python logic, while pairing the right tasks with the right models.

One common strategy is to create a task-to-model mapping matrix like the one below. 

Once you have this mapping, you can use static rules or SLM-powered dynamic routing to implement this. 

Static Routing

Static routing rules let you define deterministic paths based on model aliases, token lengths, user authorization tiers, or specific headers. Think of static routing like a fixed highway detour sign or a strict set of fixed rules. It doesn't read or "understand" the actual meaning of a message; it simply follows pre-set instructions to decide where to send it.

Here is how it works:

• Fixed Map Rules: You manually tell the system, "If a request comes from User A, send it to Model A. If it comes from User B, send it to Model B."
• Fixed Keyword Matching: You write pre-set patterns ahead of time (e.g., "If the text contains SELECT or JOIN, send to the SQL Model")
• Backup Safety Net: It can act as an automatic backup. You can set a rule that says: "Always try Provider A first. If Provider A is broken or busy, automatically send the exact same request to Provider B."

Because these rules are written in advance and never change on their own, they are extremely fast and cheap to run, but they aren't smart enough to adapt to what the user is actually talking about.

Dynamic Routing using SLMs

Static rules based solely on character limits or headers miss nuance that a short prompt can still demand complex mathematical logic. 

To route based on semantic complexity, you can introduce an SLM (Small Language Model) node (e.g., Llama-3-8B or Phi-3) directly before the main router call. The SLM evaluates the prompt and tries to assign an intent score. 

The SLM evaluates difficulty on a scale of 1 to 5 within ~30–50ms.

• Score 1–2 (Routine): Dispatched to Tier 1 (gpt-4o-mini / llama-3-8B).
• Score 3 (Moderate): Dispatched to Tier 2 (gemini-pro).
• Score 4–5 (High Complexity): Dispatched to Tier 3 (frontier models).

Some of the popular SLMs for dynamic routing are given below: 

If an 8B SLM adds too much latency to your application, many high-volume production systems use non-generative models as routers instead:

• BERT-based Classifiers (e.g., RoBERTa): Fine-tuned specifically to predict prompt complexity or score query difficulty. These return decisions in under 10 milliseconds.
• Text Embedding Models (e.g., text-embedding-3-small, BGE-Small): Used for semantic routing. The prompt is converted into a mathematical vector and instantly matched against pre-indexed topic clusters (e.g., matching SQL terms to a database specialized LLM).

Dynamic Routing using custom code

When pre-built routing algorithms do not fit your specific pipeline, LiteLLM allows custom programmatic rules using Python callbacks. You can write custom Python code that inspects the request payloads, detects code blocks via regex, and dynamically reassigns target models.

Thus we can achieve the key benefits of an AI Gateway/Router:

• Massive Cost Savings: Route up to 80% of routine traffic away from expensive flagship models to low-cost alternatives.
• Improved Throughput & Speed: SLMs and lightweight models generate first-token responses in a fraction of the time required by high-parameter reasoning models.
• High Availability & Resilience: Built-in failover capabilities ensure client applications remain operational even during provider outages.

Ruminating on Agent Skills

 The best place to understand agent skills is through the official open standard website - https://agentskills.io/home. This site gives a very clear understanding of the agent skills specification and should be your first go-to place to start learning about skills. 

A Skill is nothing a portable package (aka folder) of instructions, scripts, and resources that gives an AI agent specialized capabilities and domain expertise. Skills also use progressive disclosure - The agent discovers what skills exist, then loads only the ones relevant to the current task, and only reads the parts it needs at that moment. In practice, this means:

• Your agent doesn’t carry all your company’s policies, patterns, and examples in every call.
• It loads just enough context to do the job well, then responds.
• That makes Skills ideal for enterprise and public-sector use, where you want governed, auditable, reusable capability packs rather than giant, monolithic prompts.

Think of Skills as modular, AI-readable playbooks. They help you:

• Standardise behaviour: Every agent that uses the “Azure Functions” skill follows the same patterns and conventions.
• Scale expertise: You write the skill once, then many agents (Copilot, custom agents, internal tools) can use it.
• Keep context lean: The agent loads only the relevant skill(s) for the task, reducing noise and cost.
• Separate concerns: Skills isolate domain knowledge from core agent logic, making both easier to maintain.

A typical Skill folder structure will look like this: 

Skills and MCP serve complementary roles in agentic AI. MCP (Model Context Protocol) is the connectivity layer: it standardises how an AI application discovers and invokes external tools, APIs, data sources, and services. Skills are reusable procedural instructions that explain how an agent should approach a task, including workflows, domain rules, sequencing, and best practices. 

In simple terms, MCP provides the capabilities, while Skills provide the orchestration—for example, an MCP server could expose CRM tools, while a “customer-onboarding” Skill instructs the agent how to validate information, apply policy, call those tools, and handle exceptions

Ruminating on Human in the Loop (HITL) vs Human on the Loop (HOTL)

 As AI systems become more embedded in enterprise workflows, the conversation is no longer just about capability—it’s about control. Two models often come up in this discussion: 

• Human in the Loop (HITL) 
• Human on the Loop (HOTL)

While they sound similar, they represent fundamentally different approaches to how humans interact with AI systems.

Understanding this distinction is critical for designing safe, scalable, and efficient AI-driven processes.

Human in the Loop (HITL): Control Before Action

In the HITL model, humans are directly embedded in the decision-making process. The AI generates outputs, but execution depends on explicit human approval or validation.  

This model is best suited for:

• High-risk decisions (financial transactions, compliance approvals)
• Low-confidence AI outputs
• Regulatory or audit-heavy environments

Think of HITL as a gated workflow: the AI proposes, but the human disposes.

For example, in an ERP system like Oracle Fusion, an AI might recommend vendor payments or flag anomalies—but a finance controller must approve before funds are released. This ensures accountability and reduces the risk of automation errors propagating into real-world impact.

The trade-off is clear: higher reliability and governance, but reduced speed and scalability.

Human on the Loop (HOTL): Control Through Oversight

HOTL shifts the paradigm. Here, AI systems operate autonomously, making decisions and executing actions without requiring prior human approval. Humans remain in a supervisory role and can intervene when necessary. 

This model is ideal for:

• High-volume, repetitive tasks
• Real-time decision environments
• Mature AI systems with proven accuracy

In this setup, the human is not blocking the process—they are monitoring it. A good example is automated fraud detection. An AI system might automatically block suspicious transactions in real time, while human analysts review flagged patterns and adjust thresholds or intervene in edge cases. The system moves fast, but oversight ensures it doesn’t drift into unsafe behavior. 

The trade-off here flips: speed and scalability increase, but it requires strong monitoring, alerting, and fallback mechanisms.

Confusing HITL and HOTL can lead to poorly designed systems. Overusing HITL creates bottlenecks and defeats the purpose of automation. Overusing HOTL without proper guardrails can introduce silent failures at scale.

The real design challenge is deciding:

• When does AI need approval?
• When can it act independently?
• How do we transition from HITL to HOTL as confidence grows?

This is where concepts like confidence thresholds, risk scoring, and progressive autonomy come into play.

The Two-Model Perspective

Another way to interpret this “bi-modal” structure is through a two-model system:

• A decision model that performs the task (e.g., classification, prediction, action)
• A governance model that determines whether human intervention is required

For instance, an AI might assign a confidence score to its output. If the score is below a defined threshold, the system routes the task into a HITL flow. If it exceeds the threshold, it proceeds autonomously under HOTL. This layered approach allows organizations to dynamically balance risk and efficiency, rather than hardcoding one model across all scenarios.

Effective AI governance will increasingly rely on:

• Dynamic switching between HITL and HOTL
• Real-time monitoring and explainability
• Feedback loops that continuously improve both models

Organizations that get this right will not only scale AI faster but also build trust in its decisions. In the end, the question is not whether humans should be involved—it’s how and when.

Split before data pre-processing or after?

In machine learning workflows, the standard practice is to split datasets into training and testing subsets before applying most preprocessing transformations to prevent data leakage. 

However, certain preliminary data cleaning operations may be performed safely on the entire dataset beforehand, as they do not depend on statistical summaries or introduce information from the test set into the training process. 

Given below are examples of preprocessing that can be done before splitting. 

• Removing duplicates. 
• Fixing data types - e.g. date strings
• Remove bad data or impossible values - e.g. age > 150
• Removing whitespace from strings - e.g. trim the text

Thus, as long as you are not using statistics to impute missing values in the dataset, you can do the preprocessing before the split (into training/test). 

Operations involving data-derived statistics—such as imputation with means/medians, standardization, one-hot encoding based on frequencies, or percentile-based outlier removal—must be fitted exclusively on the training set. Hence this kind of data pre-processing should be only done after splitting, or you will end up with something called as 'data leakage'.

So what exactly is data leakage? You can understand it with the following analogy. 

• Imagine you're studying for an exam.
• You’re supposed to practice using your textbook (training data) and then take the exam (test data) to see how well you’ve learned.
• Now imagine someone secretly shows you some of the exam questions while you’re studying.
• When you take the test, you score really high — but not because you truly understood the material. You just recognized the questions. That’s data leakage!!!

In simple terms:

• The training data is what the model learns from.
• The test data is supposed to check how well it learned.
• If information from the test data sneaks into training, the model gets an unfair advantage.
• It looks like it performs very well.
• But when you give it completely new data in the real world, performance drops.
• So data leakage makes the model look smarter than it actually is — and that’s dangerous because it won’t work as well in real-life situations.

Example where imputation is done before splitting. 

• Suppose you are building a model to predict house prices, and the dataset contains missing values in the feature “Lot Size.”
• You calculate the mean lot size using the entire dataset (including both training and test data) and use that value to fill in all missing entries.
• After performing this imputation, you split the data into training and test sets.
• This creates data leakage because the imputed values were influenced by information from the test set.
• As a result, the model’s evaluation may appear more accurate than it truly is, since the training process indirectly incorporated knowledge from unseen data.

 Another example of data leakage is target leakage explained here - https://www.narendranaidu.com/2026/02/ruminating-on-target-leakage-in-ml.html

Ruminating on target leakage in ML models

Target leakage is a type of data leakage where training data includes info directly tied to the outcome (target variable), but that info wouldn't exist at prediction time. Your model "cheats" during training, looks amazing on paper, but fails on new data. This often sneaks in through feature engineering or data collection, leading to overfitting. 

Examples:

• You're building a model to spot who'll get a sinus infection. Your dataset has a feature "took_antibiotics." Sounds useful, right? Wrong—patients take antibiotics after getting sick, so this feature leaks the target. Drop it!
• Predicting if employees will quit. Including "retention_bonus_offered" leaks info because bonuses come after quit signals, not before. The model learns from a reaction to churn, not its causes.
• In credit card fraud prediction, using "chargeback_filed" as a feature is leakage gold. Chargebacks happen post-fraud, so the model peeks at the future.

Golden rule to avoid target leakage: Always ask: "Would this feature exist before the prediction?" If no, remove it.