DEV Community: Ferit

Why I Built Inkspect: How Do I Know the AI Is Really Done?

Ferit — Sun, 01 Nov 2026 11:47:25 +0000

I give an AI agent a task. It changes the code, runs some checks, and tells me the work is done.

Before I give it the next task, I have one more question:

What evidence shows that it is done?

I started asking this because of problems I had faced in my own projects. An agent would finish a task, but more problems would appear when I checked the result.

Sometimes a requirement was missing. Sometimes the checks did not cover the behavior I cared about. I needed to understand what the agent had actually delivered.

This repeated problem is why I built Inkspect.

The gap between two tasks

When I work with coding agents, each task often builds on the task before it.

If I accept unfinished work, the next task starts from that point. Finding the problem later means going back to something I thought was complete.

So the moment between two tasks matters to me.

I want to know what was checked, what problems were found, and what still needs attention. That information helps me decide whether to fix something or continue.

Reading the changed lines is part of this process. Understanding the original task and the surrounding code matters too. I wrote about this idea in A diff is only half the story.

A real example from building Inkspect

One example comes from an earlier Go version of Inkspect’s review engine.

During development, a local Codex source review found a problem with saved review data.

The engine recorded model IDs returned by the AI service. These labels help explain which models were involved in a review.

But this information could be lost when a review ended early.

The engine saved progress at different stages. These saved points, called checkpoints, included usage data but missed the model IDs.

If a run failed, was cancelled, or went through recovery after an interruption, its final report could lose those IDs.

The information had existed during the run. It did not reach the saved report.

This example came from a local development review with Codex. It was not an Inkspect production review.

From a finding to a tested fix

The finding gave us a clear question to test:

After an interrupted run, does the final report still contain the recorded model IDs?

A test checked three cases. The saved test results show the difference:

Test case	Before the fix	After the fix
The run failed	Failed	Passed
The run was cancelled	Failed	Passed
An interrupted run went through recovery	Failed	Passed

The fix was to include the session’s recorded model IDs in each stage checkpoint.

That gave the failure and recovery paths the information they needed to build the final report.

A follow-up source review checked the correction too. The tests ran the cases. The AI reviewer inspected the code.

We had a specific problem, a test that showed it, a correction, and a passing result for the same cases.

That gave us a clear reason to close the finding.

The loop I use now

Today, I use Inkspect as part of my development loop:

I give the coding agent a task.
The agent finishes its work.
Inkspect reviews the result and reports findings with evidence.
When needed, fixes and further checks follow.
The agent gets a handoff when the work is ready for the next task.

Inkspect can also make fixes when needed and tell the agent when it is ready to continue.

This makes review part of the work between tasks. Its findings help decide what happens next.

What I want from a review

I want a finding to answer a few simple questions:

What is the problem?
When does it happen?
What evidence supports it?
What change could fix it?
How can we check the fix?

The reviewer needs to support its own claims too. If something is unclear, I want to see that uncertainty.

The checkpoint bug is a small example, but it explains why I care about this process. A clear finding helped turn a hidden gap into a test and a fix.

That is the kind of workflow I am building Inkspect around.

When a coding agent says “done,” what do you check before giving it the next task?

This article was written with AI assistance, using my experience and saved review and test records from the project.

AI Writes 80% of Your Code. Who Reviews It?

Ferit — Thu, 02 Apr 2026 18:01:43 +0000

AI writes your functions, scaffolds your services, and ships PRs before you finish your coffee. Welcome to 2026, where "vibe coding" isn't a meme anymore: developers describe what they want in plain English, and AI generates the code.

The output is staggering. Teams report 40-70% of their committed code now originates from AI. But here's the number nobody puts on their landing page: AI co-authored code contains 1.7x more major issues than human-written code, including 2.74x more security vulnerabilities.

We got really good at generating code. We didn't get better at reviewing it.

The Review Bottleneck Nobody Planned For

Traditional code review was designed for a world where a senior engineer writes 200 lines a day. Now a junior dev with an AI assistant pushes 2,000. The reviewer's workload didn't scale, it exploded.

Most teams respond in one of two ways. They rubber-stamp PRs to keep velocity up, or they create a review backlog so deep that it kills the speed AI was supposed to deliver. Both options end the same way: bugs in production.

The core problem isn't volume, though. It's context. When a human writes code, they carry the project's architecture in their head. They know the naming conventions, the edge cases from last quarter's outage, the service boundary that shouldn't be crossed. AI doesn't carry any of that. It generates plausible code that compiles, passes lint, and quietly violates three architectural decisions your team made six months ago.

Why Diff-Only Review Tools Fall Short

Here's where most AI review tools break down. They look at the diff: the lines added and removed in a pull request. That's it.

A diff-only reviewer sees a function that looks correct in isolation. It doesn't know that your project already has a utility doing the same thing in a different module. It can't tell that the new database query bypasses the caching layer every other service uses. It won't flag that the error handling pattern contradicts what your team agreed on in an ADR last month.

Studies from early 2026 show that full-codebase-aware review tools catch 40-60% more cross-file issues than diff-only approaches. When AI is writing code that doesn't understand your project, your reviewer needs to understand your project deeply enough for both of them.

Codebase-Aware Review: Closing the Gap

This is the approach we built Octopus Review around. Instead of reviewing diffs in a vacuum, Octopus indexes your entire codebase using RAG (Retrieval-Augmented Generation) with vector search. When a PR comes in, the review has full project context: your patterns, your abstractions, your existing code.

The difference is practical. When AI-generated code introduces a new HTTP client wrapper, a diff-only tool checks if the syntax is correct. Octopus checks if you already have one, whether the new one follows your error handling conventions, and whether it respects the service boundaries defined elsewhere in the repo.

Every review comment comes with a severity level (Critical, Major, Minor, Suggestion, or Tip), so you're not drowning in noise. A mismatched bracket and a security vulnerability don't sit in the same bucket.

Here's what it looks like in practice with the CLI:

octopus pr review 42

 PR #247: Add user notification service

 CRITICAL (1)
 ├─ src/services/notify.ts:42
 │  Unvalidated user input passed directly to email template.
 │  This creates an injection vector. Sanitize with your existing
 │  `sanitizeHtml()` utility in src/utils/sanitize.ts.

 MAJOR (2)
 ├─ src/services/notify.ts:15
 │  New NotificationClient duplicates functionality of existing
 │  MessagingService in src/services/messaging.ts. Consider extending
 │  the existing service instead.
 ├─ src/services/notify.ts:38
 │  Missing retry logic. All other service calls in this project
 │  use the withRetry() wrapper from src/utils/retry.ts.

 SUGGESTION (1)
 ├─ src/services/notify.ts:5
 │  Consider moving notification templates to src/templates/
 │  to match the project's existing template organization.

 4 comments across 1 file | Review time: 12s

That second MAJOR comment is the one that matters most. A diff-only tool would never catch it because the duplication exists in a completely different file. Octopus catches it because it has indexed the entire codebase and knows MessagingService already exists.

Your Standards, Not Generic Rules

The other gap in vibe-coded projects is consistency. AI models are trained on the entire internet. They'll write perfectly valid Go in one function and subtly different Go in the next, mixing community conventions your team never adopted.

Octopus has a Knowledge Base where you feed your team's standards, architectural decision records, and style guides. Reviews then enforce YOUR rules, not generic best practices from Stack Overflow circa 2023.

This matters more in the vibe coding era than ever. When half your code comes from an AI that has no memory of your last sprint, someone needs to be the institutional memory. That someone can be automated.

Open Source, Self-Hostable, No Vendor Lock-in

One more thing that matters when AI is touching every line of your code: where does that code go during review?

Octopus is open source (Modified MIT) and fully self-hostable. Your code is processed in-memory only. Embeddings are persisted for search, but source code is never stored. Bring your own API key for Claude or OpenAI, and run it on your own infrastructure.

git clone https://github.com/octopusreview/octopus.git
docker-compose up -d

That's it. Full AI code review running on your hardware, your keys, your rules.

The Bottom Line

Vibe coding isn't going away. The teams that thrive won't be the ones generating the most code. They'll be the ones who close the gap between generation and understanding.

If your review process can't see beyond the diff, it can't catch what AI gets wrong. And AI gets a lot wrong, quietly, confidently, at scale.

Give Octopus Review a try. Star the GitHub repo. Join the Discord if you want to talk about what codebase-aware review actually looks like in practice.

Octopus Review is an open-source, RAG-powered AI code review tool. It works with GitHub and Bitbucket, reviews PRs with full codebase context, and can be self-hosted with zero vendor lock-in.

From scikit-learn to Production, Deploying ML Models That Actually Work

Ferit — Mon, 02 Mar 2026 14:31:42 +0000

There is a gap between training a model in a Jupyter Notebook and running it in production. Most tutorials stop at model.score() and call it done. This article covers the full pipeline: data preprocessing, model selection, evaluation, serialization, and serving a scikit-learn model behind a FastAPI endpoint.

The Problem

We needed a transaction risk scoring system for a crypto payment gateway. The model receives transaction features and returns a fraud probability between 0 and 1. Requirements:

Latency under 50ms per prediction
Handle 1,000 requests per second
Update the model weekly without downtime
Explainable predictions (regulators want to know why a transaction was flagged)

scikit-learn turned out to be the right tool. Not TensorFlow, not PyTorch. For tabular data with fewer than 100 features, gradient boosted trees in scikit-learn are hard to beat.

Data Preprocessing Pipeline

Raw transaction data is messy. Missing values, mixed types, different scales. scikit-learn pipelines handle this cleanly:

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer

numeric_features = [
    "amount_usd", "gas_price", "tx_count_24h",
    "address_age_days", "avg_tx_value"
]

categorical_features = [
    "chain", "token_type", "sender_country"
]

numeric_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="constant", fill_value="unknown")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])

preprocessor = ColumnTransformer([
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features)
])

The key insight here is that the preprocessor becomes part of the saved model. When you serialize the pipeline, all the preprocessing logic (imputation values, scaling parameters, encoding mappings) travels with the model. No separate preprocessing code needed at inference time.

Model Selection

We evaluated four models on our dataset:

from sklearn.ensemble import (
    GradientBoostingClassifier,
    RandomForestClassifier,
    AdaBoostClassifier
)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

models = {
    "logistic_regression": LogisticRegression(max_iter=1000),
    "random_forest": RandomForestClassifier(n_estimators=200),
    "gradient_boosting": GradientBoostingClassifier(n_estimators=200),
    "adaboost": AdaBoostClassifier(n_estimators=200)
}

results = {}
for name, model in models.items():
    pipeline = Pipeline([
        ("preprocessor", preprocessor),
        ("classifier", model)
    ])

    scores = cross_val_score(
        pipeline, X_train, y_train,
        cv=5, scoring="roc_auc"
    )

    results[name] = {
        "mean_auc": scores.mean(),
        "std": scores.std()
    }
    print(f"{name}: AUC = {scores.mean():.4f} (+/- {scores.std():.4f})")

Results on our dataset:

Model	AUC	Latency (p95)
Logistic Regression	0.87	0.2ms
Random Forest	0.93	3.1ms
Gradient Boosting	0.96	1.8ms
AdaBoost	0.91	2.4ms

Gradient boosting won on accuracy. The 1.8ms latency is well within our 50ms budget.

Hyperparameter Tuning

Grid search with cross-validation finds the best parameters:

from sklearn.model_selection import GridSearchCV

param_grid = {
    "classifier__n_estimators": [100, 200, 300],
    "classifier__max_depth": [3, 5, 7],
    "classifier__learning_rate": [0.01, 0.05, 0.1],
    "classifier__min_samples_leaf": [5, 10, 20]
}

pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", GradientBoostingClassifier())
])

search = GridSearchCV(
    pipeline,
    param_grid,
    cv=5,
    scoring="roc_auc",
    n_jobs=-1,
    verbose=1
)

search.fit(X_train, y_train)

print(f"Best AUC: {search.best_score_:.4f}")
print(f"Best params: {search.best_params_}")

Best configuration for our dataset:

best_model = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", GradientBoostingClassifier(
        n_estimators=200,
        max_depth=5,
        learning_rate=0.05,
        min_samples_leaf=10
    ))
])

Evaluation Beyond Accuracy

For fraud detection, accuracy is misleading because the dataset is imbalanced (99% legitimate, 1% fraud). We care about precision and recall:

from sklearn.metrics import (
    classification_report,
    roc_auc_score,
    precision_recall_curve
)

best_model.fit(X_train, y_train)
y_pred = best_model.predict(X_test)
y_proba = best_model.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred))
print(f"ROC AUC: {roc_auc_score(y_test, y_proba):.4f}")

# Find optimal threshold
precisions, recalls, thresholds = precision_recall_curve(y_test, y_proba)

# We want at least 90% recall (catch 90% of fraud)
for i, recall in enumerate(recalls):
    if recall >= 0.90:
        optimal_threshold = thresholds[i]
        print(f"Threshold for 90% recall: {optimal_threshold:.3f}")
        print(f"Precision at this threshold: {precisions[i]:.3f}")
        break

Feature Importance

Regulators need explainability. scikit-learn makes this straightforward:

import numpy as np

classifier = best_model.named_steps["classifier"]
feature_names = (
    numeric_features +
    list(best_model.named_steps["preprocessor"]
         .named_transformers_["cat"]
         .named_steps["encoder"]
         .get_feature_names_out(categorical_features))
)

importances = classifier.feature_importances_
indices = np.argsort(importances)[::-1]

print("Top 10 features:")
for i in range(min(10, len(indices))):
    idx = indices[i]
    print(f"  {feature_names[idx]}: {importances[idx]:.4f}")

Output:

Top 10 features:
  amount_usd: 0.2341
  tx_count_24h: 0.1876
  address_age_days: 0.1523
  gas_price: 0.0987
  avg_tx_value: 0.0834
  chain_ETH: 0.0612
  chain_TRC20: 0.0445
  sender_country_unknown: 0.0398
  token_type_USDT: 0.0321
  chain_BTC: 0.0287

This tells us: high transaction amounts from new addresses with unusual gas prices are the strongest fraud signals. That is explainable to a regulator.

Model Serialization

Save the entire pipeline (preprocessor + model) as a single artifact:

import joblib
from datetime import datetime

model_version = datetime.now().strftime("%Y%m%d_%H%M%S")
model_path = f"models/fraud_detector_{model_version}.joblib"

joblib.dump(best_model, model_path)

# Verify the saved model works
loaded_model = joblib.load(model_path)
assert np.allclose(
    loaded_model.predict_proba(X_test),
    best_model.predict_proba(X_test)
)

# Save metadata
metadata = {
    "version": model_version,
    "auc": roc_auc_score(y_test, y_proba),
    "threshold": optimal_threshold,
    "features": feature_names,
    "training_samples": len(X_train)
}

import json
with open(f"models/fraud_detector_{model_version}.json", "w") as f:
    json.dump(metadata, f, indent=2)

Serving with FastAPI

The production API loads the model once at startup and serves predictions:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import joblib
import numpy as np

app = FastAPI()

# Load model at startup
model = joblib.load("models/fraud_detector_latest.joblib")
metadata = json.load(open("models/fraud_detector_latest.json"))
threshold = metadata["threshold"]

class TransactionInput(BaseModel):
    amount_usd: float
    gas_price: float
    tx_count_24h: int
    address_age_days: int
    avg_tx_value: float
    chain: str
    token_type: str
    sender_country: str

class PredictionOutput(BaseModel):
    fraud_probability: float
    is_flagged: bool
    model_version: str

@app.post("/predict", response_model=PredictionOutput)
async def predict(tx: TransactionInput):
    features = pd.DataFrame([tx.model_dump()])

    probability = model.predict_proba(features)[0][1]

    return PredictionOutput(
        fraud_probability=round(probability, 4),
        is_flagged=probability >= threshold,
        model_version=metadata["version"]
    )

@app.get("/health")
async def health():
    return {"status": "ok", "model_version": metadata["version"]}

Hot-Swapping Models

To update the model without downtime, we use a simple versioning scheme:

import threading

class ModelManager:
    def __init__(self, model_dir: str):
        self.model_dir = model_dir
        self.current_model = None
        self.current_metadata = None
        self.lock = threading.Lock()
        self.load_latest()

    def load_latest(self):
        model_files = sorted(glob.glob(f"{self.model_dir}/fraud_detector_*.joblib"))
        if not model_files:
            raise FileNotFoundError("No model found")

        latest = model_files[-1]
        new_model = joblib.load(latest)
        meta_path = latest.replace(".joblib", ".json")
        new_metadata = json.load(open(meta_path))

        with self.lock:
            self.current_model = new_model
            self.current_metadata = new_metadata

    def predict(self, features):
        with self.lock:
            return self.current_model.predict_proba(features)

Performance Numbers

After deploying this setup:

Prediction latency: 2ms p50, 8ms p95
Throughput: 2,400 requests/second on a single core
Model size: 12MB (serialized pipeline)
Fraud detection rate: 94% recall at 87% precision
False positive rate: 0.3%

The full implementation is on GitHub: python-machine-learning

If you are deploying ML models to production or working with scikit-learn at scale, I would like to hear about your experience. Find me on GitHub.

Building Multi-Model AI Agents with OpenAI, Ollama, Groq and Gemini

Ferit — Sun, 01 Mar 2026 22:23:42 +0000

Most AI applications today rely on a single LLM provider. That works fine until the API goes down, rate limits hit, or your costs spiral out of control. A better approach is to build agents that can orchestrate multiple models and switch between them based on the task at hand.

In this article, I will walk through how I built an AI agent framework that supports OpenAI GPT-4, Ollama local models, Groq ultra-fast inference, and Google Gemini as interchangeable backends.

Why Multi-Model?

Each provider has different strengths:

OpenAI GPT-4 has the best reasoning and function calling
Ollama runs locally with zero latency and no API costs
Groq delivers sub-200ms inference for real-time applications
Gemini excels at multimodal tasks (vision, audio, code)

By abstracting the provider layer, your agent can pick the right model for each subtask, fall back gracefully when one provider fails, and optimize cost by routing simple tasks to cheaper models.

Architecture Overview

The framework has four main components:

Agent Core -> Planning -> Tool Execution -> Memory
     |            |            |              |
  LLM Router   Task Graph   Registry     Redis/PostgreSQL
     |
  OpenAI | Ollama | Groq | Gemini

The LLM Router is the key piece. It decides which provider handles each request based on configurable rules.

Setting Up the Provider Layer

First, define a common interface that all providers implement:

from abc import ABC, abstractmethod
from dataclasses import dataclass

@dataclass
class LLMResponse:
    content: str
    model: str
    provider: str
    tokens_used: int
    latency_ms: float

class LLMProvider(ABC):
    @abstractmethod
    async def complete(self, messages: list, tools: list = None) -> LLMResponse:
        pass

    @abstractmethod
    async def embed(self, text: str) -> list[float]:
        pass

Then implement each provider:

import openai
import ollama
from groq import Groq
import google.generativeai as genai

class OpenAIProvider(LLMProvider):
    def __init__(self, model="gpt-4"):
        self.client = openai.AsyncOpenAI()
        self.model = model

    async def complete(self, messages, tools=None):
        start = time.monotonic()
        response = await self.client.chat.completions.create(
            model=self.model,
            messages=messages,
            tools=tools
        )
        latency = (time.monotonic() - start) * 1000
        return LLMResponse(
            content=response.choices[0].message.content,
            model=self.model,
            provider="openai",
            tokens_used=response.usage.total_tokens,
            latency_ms=latency
        )

class OllamaProvider(LLMProvider):
    def __init__(self, model="llama3"):
        self.model = model

    async def complete(self, messages, tools=None):
        start = time.monotonic()
        response = await ollama.AsyncClient().chat(
            model=self.model,
            messages=messages
        )
        latency = (time.monotonic() - start) * 1000
        return LLMResponse(
            content=response["message"]["content"],
            model=self.model,
            provider="ollama",
            tokens_used=response.get("eval_count", 0),
            latency_ms=latency
        )

class GroqProvider(LLMProvider):
    def __init__(self, model="llama3-70b-8192"):
        self.client = Groq()
        self.model = model

    async def complete(self, messages, tools=None):
        start = time.monotonic()
        response = self.client.chat.completions.create(
            model=self.model,
            messages=messages
        )
        latency = (time.monotonic() - start) * 1000
        return LLMResponse(
            content=response.choices[0].message.content,
            model=self.model,
            provider="groq",
            tokens_used=response.usage.total_tokens,
            latency_ms=latency
        )

class GeminiProvider(LLMProvider):
    def __init__(self, model="gemini-pro"):
        genai.configure(api_key=os.environ["GEMINI_API_KEY"])
        self.model = genai.GenerativeModel(model)

    async def complete(self, messages, tools=None):
        start = time.monotonic()
        response = await self.model.generate_content_async(
            messages[-1]["content"]
        )
        latency = (time.monotonic() - start) * 1000
        return LLMResponse(
            content=response.text,
            model="gemini-pro",
            provider="gemini",
            tokens_used=0,
            latency_ms=latency
        )

The LLM Router

The router picks the best provider based on task type, latency requirements, and availability:

class LLMRouter:
    def __init__(self, providers: dict[str, LLMProvider]):
        self.providers = providers
        self.fallback_order = ["openai", "groq", "ollama", "gemini"]

    async def route(self, messages, task_type="general", tools=None):
        provider_name = self._select_provider(task_type)

        for name in self._get_fallback_chain(provider_name):
            try:
                provider = self.providers[name]
                return await provider.complete(messages, tools)
            except Exception as e:
                logger.warning(f"{name} failed: {e}, trying next provider")
                continue

        raise RuntimeError("All providers failed")

    def _select_provider(self, task_type):
        routing_rules = {
            "reasoning": "openai",
            "realtime": "groq",
            "local": "ollama",
            "vision": "gemini",
            "general": "openai"
        }
        return routing_rules.get(task_type, "openai")

    def _get_fallback_chain(self, primary):
        chain = [primary]
        for name in self.fallback_order:
            if name != primary:
                chain.append(name)
        return chain

Building the Agent

With the router in place, the agent itself is straightforward:

class Agent:
    def __init__(self, router: LLMRouter, tools: ToolRegistry, memory: Memory):
        self.router = router
        self.tools = tools
        self.memory = memory

    async def execute(self, task: str) -> str:
        context = await self.memory.get_relevant(task)

        messages = [
            {"role": "system", "content": self._build_system_prompt(context)},
            {"role": "user", "content": task}
        ]

        while True:
            response = await self.router.route(
                messages,
                task_type=self._classify_task(task),
                tools=self.tools.get_schemas()
            )

            if not response.has_tool_calls:
                break

            tool_results = await self.tools.execute(response.tool_calls)
            messages.extend(tool_results)

        await self.memory.store(task, response.content)
        return response.content

Tool Registry

Tools give the agent the ability to interact with external systems:

class ToolRegistry:
    def __init__(self):
        self._tools = {}

    def register(self, name: str, func, schema: dict):
        self._tools[name] = {"func": func, "schema": schema}

    async def execute(self, tool_calls):
        results = []
        for call in tool_calls:
            tool = self._tools[call.name]
            result = await tool["func"](**call.arguments)
            results.append({
                "role": "tool",
                "content": str(result),
                "tool_call_id": call.id
            })
        return results

    @classmethod
    def default(cls):
        registry = cls()
        registry.register("web_search", web_search, web_search_schema)
        registry.register("code_execute", code_execute, code_execute_schema)
        registry.register("file_read", file_read, file_read_schema)
        return registry

Putting It All Together

providers = {
    "openai": OpenAIProvider("gpt-4"),
    "ollama": OllamaProvider("llama3"),
    "groq": GroqProvider("llama3-70b-8192"),
    "gemini": GeminiProvider("gemini-pro")
}

router = LLMRouter(providers)
tools = ToolRegistry.default()
memory = RedisMemory(url="redis://localhost:6379")

agent = Agent(router=router, tools=tools, memory=memory)

result = await agent.execute(
    "Analyze the performance bottlenecks in our API and suggest fixes"
)

Cost Optimization

One of the biggest benefits of multi-model routing is cost control. Here is a practical routing strategy:

Task Type	Provider	Cost per 1M tokens
Complex reasoning	OpenAI GPT-4	$30
Simple Q&A	Groq LLaMA 3	$0.59
Code generation	Ollama (local)	$0
Image analysis	Gemini Pro	$0.50

By routing 70% of requests to Groq/Ollama and only using GPT-4 for complex tasks, we reduced our monthly AI costs by 80%.

What I Learned

Building this framework taught me a few things:

Provider abstraction pays off fast. The moment one API has an outage, your system keeps running.
Latency varies wildly. Groq at 200ms vs OpenAI at 1-2s makes a real difference for interactive applications.
Local models are underrated. Ollama with LLaMA 3 handles 80% of tasks without any API calls.
Memory is the hard part. Deciding what to remember and what to forget matters more than which model you use.

The full source code is available on GitHub: ai-agent-framework

If you are building AI agents or working with multiple LLM providers, I would love to hear about your approach. Drop a comment below or connect with me on GitHub.