DEV Community: aiunplugged

Evaluating RAG — retrieval + generation metrics that matter

aiunplugged — Sat, 24 Oct 2026 19:05:54 +0000

Most RAG systems that silently decay in production decay in the retrieval layer. The LLM output still looks fluent, so the surface tests pass. Users just notice that the answers get subtly wrong — citing the wrong clause, mixing up two customers, giving an outdated version of the policy. By the time anyone reports it, the regression has been live for weeks.

Evaluation is the layer that catches those regressions before they ship. This article covers the metrics that actually tell you something, with runnable Python, and the minimum test-set recipe to get started. Prerequisite reading: the RAG pipeline walkthrough ships the four-step loop this article plugs into.

The two levels of RAG evaluation

A RAG system has two independent failure modes, so it needs two independent sets of metrics:

Retrieval quality — given a question, did the retriever return the chunks that actually contain the answer? If not, no amount of LLM cleverness can produce the right answer.
Generation quality — given the retrieved chunks (whatever they are), did the LLM produce a correct, faithful answer? A fluent answer that invents facts the chunks don't support is a generation failure, not a retrieval failure.

Evaluating only one layer misses half the problem. If retrieval is bad but generation masks it with plausible-sounding filler, the user sees "correct-looking wrong answers" and loses trust before anyone notices the metric that would have surfaced it.

Retrieval metrics

Precision@k

Of the top-K chunks the retriever returned for a question, how many are relevant?

def precision_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    top_k = retrieved_ids[:k]
    if not top_k:
        return 0.0
    hits = sum(1 for doc_id in top_k if doc_id in relevant_ids)
    return hits / len(top_k)

Example: top-5 retrieved = [A, B, C, D, E], relevant = {A, C}. Precision@5 = 2/5 = 0.4.

Use it when the context window is limited and the LLM gets confused by irrelevant chunks. Higher precision means less noise in the generation prompt.

Recall@k

Of all the relevant chunks that exist for a question, how many did the retriever surface in the top-K?

def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    top_k = set(retrieved_ids[:k])
    if not relevant_ids:
        return 1.0  # nothing to find — vacuously perfect
    hits = len(top_k & relevant_ids)
    return hits / len(relevant_ids)

Example: top-5 retrieved = [A, B, C, D, E], relevant = {A, C, F}. Recall@5 = 2/3 ≈ 0.67 (missed F).

Use it when the answer requires combining multiple sources. If recall@k is low, the LLM is working with incomplete information and may fabricate the missing piece.

Mean Reciprocal Rank (MRR)

How high up the ranking does the first relevant chunk appear?

def reciprocal_rank(retrieved_ids: list[str], relevant_ids: set[str]) -> float:
    for i, doc_id in enumerate(retrieved_ids):
        if doc_id in relevant_ids:
            return 1.0 / (i + 1)
    return 0.0


def mean_reciprocal_rank(evaluations: list[tuple[list[str], set[str]]]) -> float:
    if not evaluations:
        return 0.0
    return sum(reciprocal_rank(r, rel) for r, rel in evaluations) / len(evaluations)

Example: first relevant chunk at rank 1 → RR = 1.0; at rank 3 → RR = 1/3 ≈ 0.33; not retrieved → RR = 0.

MRR is the right metric when the application only shows the top result to the user (search autocomplete, "best match" chat answers). It's less useful for multi-document synthesis where several top chunks matter.

Normalised Discounted Cumulative Gain (nDCG)

The retrieval metric most forgiving of what order relevant chunks appear in — but rewards higher positions. Useful when some relevant chunks are more relevant than others (graded relevance, not binary).

import math


def dcg(relevances: list[float]) -> float:
    """Discounted Cumulative Gain. relevances[i] = graded relevance of result i."""
    return sum(rel / math.log2(i + 2) for i, rel in enumerate(relevances))


def ndcg_at_k(retrieved_ids: list[str], relevance_grades: dict[str, float], k: int) -> float:
    """Normalised DCG@k. relevance_grades = {doc_id: grade}, grade >= 0."""
    actual = [relevance_grades.get(doc_id, 0.0) for doc_id in retrieved_ids[:k]]
    ideal = sorted(relevance_grades.values(), reverse=True)[:k]
    if not ideal or sum(ideal) == 0:
        return 0.0
    return dcg(actual) / dcg(ideal)

nDCG = 1.0 means the retrieved ranking matches the ideal ranking. nDCG = 0 means nothing relevant was retrieved.

Which metric to pick

A defensible default for most production RAG: track recall@k (did we find all the needed context?) and precision@k (how much noise in that context?). Add MRR if the UI only shows the top result. Add nDCG only if relevance is actually graded, not binary — which is rare in practice.

Generation metrics

Retrieval metrics judge the retriever. Generation metrics judge whether the LLM used the retrieved context correctly.

Faithfulness

Did the generated answer stick to the retrieved context, or did it invent claims the chunks don't support? Faithfulness is the metric that catches hallucination in a RAG system where retrieval was fine.

The honest way to measure faithfulness at scale is LLM-as-judge: a second model inspects the generated answer and the retrieved context, and flags claims in the answer that aren't supported by the context. Lightweight proxy for small test sets:

def claims_in_answer_covered_by_context(answer: str, context: str) -> float:
    """
    Trivial version — split the answer into sentences and check whether each
    sentence has a token-overlap with the context above a threshold.
    Not accurate enough for production; use an LLM judge for anything serious.
    """
    import re
    sentences = re.split(r"(?<=[.!?])\s+", answer.strip())
    if not sentences:
        return 1.0

    context_tokens = set(context.lower().split())
    covered = 0
    for s in sentences:
        s_tokens = set(s.lower().split())
        if not s_tokens:
            continue
        overlap = len(s_tokens & context_tokens) / len(s_tokens)
        if overlap > 0.4:
            covered += 1
    return covered / len(sentences)

Use this proxy to spot the worst drift. For reliable faithfulness in production, use LLM-as-judge with a prompt tuned to your domain.

Answer relevance

Did the answer address the question that was asked, or did it address an adjacent question? A hallucinated answer that happens to match some retrieved context can still be irrelevant.

Measured by prompting a judge model: "Given this question and this answer, rate how directly the answer responds, 1 to 5." Store the average per test query, track across releases.

Context recall + context precision

The two retrieval-adjacent metrics that look at how well the generation step used the context:

Context recall: of all the relevant information in the retrieved context, how much appeared in the answer? Low context recall = the LLM retrieved the answer but didn't surface it.
Context precision: of the retrieved chunks, how many were actually used in generating the answer? Low context precision = the retriever is noisy and the LLM is working harder than it should.

Both are classically measured with LLM-as-judge over a test set, and are the metrics that Ragas and similar libraries report by default.

Building a minimum test set

A test set is a list of questions with the "correct" answer for each, plus (for retrieval metrics) a set of chunk IDs that should have been retrieved. The smallest useful test set has roughly 20-50 questions covering the obvious categories of query the production system will see.

Minimum shape:

from dataclasses import dataclass


@dataclass
class EvalQuery:
    question: str
    expected_answer: str
    relevant_chunk_ids: set[str]  # Chunks that contain the answer
    category: str  # Tag for slicing results — "pricing", "troubleshooting", etc.


TEST_SET = [
    EvalQuery(
        question="What is the refund window after purchase?",
        expected_answer="7 days from the date of purchase.",
        relevant_chunk_ids={"chunk_42"},
        category="policy",
    ),
    EvalQuery(
        question="Which Python version is required?",
        expected_answer="Python 3.10 or later.",
        relevant_chunk_ids={"chunk_17", "chunk_18"},
        category="setup",
    ),
    # ... 18 more
]

Where the questions come from matters more than how many. Three defensible sources:

Real user queries from logs — if the system is already live, the top-50 most common actual questions are the right test set. Resolve "correct answer" by hand.
Support ticket questions — most teams have an archive of "customer asked X" questions with a resolved answer. Those are gold.
Hand-written by whoever knows the corpus best — fastest to start, but biased toward the writer's mental model of what users ask.

The handbook's Chapter 4 covers a five-step human review protocol for assembling the first 50 queries, and the pattern for growing the test set from production logs. For a free-article minimum, 20 hand-written queries covering 3-5 categories is enough to catch obvious regressions.

A simple pytest harness

Wire the metrics into pytest so the eval runs on every PR. Minimum version:

import pytest

from your_rag_app import rag_pipeline  # your retrieve + generate function
from your_eval_set import TEST_SET
from your_metrics import precision_at_k, recall_at_k


@pytest.fixture(scope="session")
def predictions():
    results = []
    for query in TEST_SET:
        retrieved_ids, answer = rag_pipeline(query.question)
        results.append({
            "query": query,
            "retrieved_ids": retrieved_ids,
            "answer": answer,
        })
    return results


def test_retrieval_recall_above_threshold(predictions):
    recalls = [
        recall_at_k(p["retrieved_ids"], p["query"].relevant_chunk_ids, k=5)
        for p in predictions
    ]
    avg = sum(recalls) / len(recalls)
    assert avg > 0.75, f"Average recall@5 = {avg:.3f}, below 0.75 threshold"


def test_retrieval_precision_above_threshold(predictions):
    precisions = [
        precision_at_k(p["retrieved_ids"], p["query"].relevant_chunk_ids, k=5)
        for p in predictions
    ]
    avg = sum(precisions) / len(precisions)
    assert avg > 0.30, f"Average precision@5 = {avg:.3f}, below 0.30 threshold"

Thresholds (0.75 recall, 0.30 precision) are starting points — tune to the system's current baseline, then enforce no regression below it. Run on every pull request so a chunking tweak or embedding model change can't silently break retrieval quality.

Tooling landscape in 2026

Three evaluation libraries dominate for RAG:

Ragas (pip install ragas) — the de facto standard for RAG-specific metrics. Ships faithfulness, answer relevance, context recall, context precision out of the box, using an LLM judge. Lowest-friction way to add generation metrics.
DeepEval (pip install deepeval) — pytest-native. Metrics look like unit tests, which is nice when CI integration is the priority. Covers RAG and non-RAG LLM apps.
Promptfoo (npm install -g promptfoo or Python API) — config-driven YAML eval sets, side-by-side model comparison, caching built in. Good when the main thing being evaluated is prompt + model choice, not retrieval.

Each has trade-offs around LLM-judge cost, metric calibration, and how it integrates with pytest. The handbook's Chapter 4 includes a decision table for picking between them based on team shape and existing CI.

What this article deliberately does NOT cover

Evaluation in production has a longer tail than the metrics above. The parts not covered here, and where to find them:

The four tested LLM-as-judge prompts that calibrate reliably against human review, with the measurement protocol for calibration.
The full pytest + GitHub Actions CI harness that runs on every PR without blowing up LLM API bills.
How to tell real metric movement from statistical noise on a 50-query test set — the Wilcoxon signed-rank procedure that avoids false positives.
The five-step human review protocol for building the first 50-query test set from scratch when there are no production logs yet.
Growing the test set from production logs — the sampling strategy that keeps the test set representative over time.
Ragas vs DeepEval vs Promptfoo — the decision table based on team shape, LLM-judge cost, and existing CI.

Those live in Chapter 4 of The RAG Production Handbook — launching November 9, 2026. First 20 buyers at ₹699 (regular launch ₹999, rising to ₹1,499 on Dec 1).

RAG chunking — 5 strategies with runnable Python

aiunplugged — Fri, 16 Oct 2026 17:51:51 +0000

Originally published on aiunplugged.in — cross-posting for the Dev.to community.

Chunking is the step that most RAG tutorials gloss over — "just split your documents into 500-token chunks" — and most retrieval problems trace back to. The right chunk strategy depends on the shape of the source documents and the shape of the queries, not on a blog post's default. This article walks the five strategies that cover ~95% of production RAG pipelines in 2026, with runnable Python for each.

Prerequisite: the RAG pipeline walkthrough ships the four-step loop in one file. This article drops into step 1 of that loop.

Why chunking matters

A chunk is the unit of retrieval. When a user asks a question, the retriever returns the top-K chunks whose embeddings sit closest to the question's embedding. Three things can go wrong, all driven by chunk shape:

Chunks too large — the embedding averages away specificity. The chunk gets retrieved for too many loosely-related queries, pushing truly relevant chunks out of the top-K.
Chunks too small — the chunk loses context. The retrieved passage reads fine on its own but the LLM can't answer the question because the resolution of the answer sits in neighbouring passages that weren't retrieved.
Chunks split across semantic boundaries — a definition is in one chunk, the example is in another, and only one gets retrieved. The LLM gets half the answer.

The five strategies below address those three failure modes in different ways.

Setup — shared across all strategies

Every strategy below uses this scaffolding. Install once:

pip install langchain-text-splitters sentence-transformers numpy

Shared helper for displaying chunks:

from dataclasses import dataclass
from typing import Callable


@dataclass
class Chunk:
    text: str
    index: int
    strategy: str
    metadata: dict


def show(chunks: list[Chunk]) -> None:
    for c in chunks:
        preview = c.text[:80].replace("\n", " ")
        print(f"[{c.strategy} #{c.index:02d}] ({len(c.text)} chars) {preview}...")

Sample input used in every example below:

SAMPLE = """
# Production RAG — chunking

Chunking is the step that determines retrieval quality more than any other.
A chunk too large averages away specificity. A chunk too small loses context.

## Fixed-size chunking

The simplest strategy. Split the document into pieces of exactly N characters
or N tokens, with M units of overlap between adjacent chunks. Works when the
corpus is uniform prose with no strong structural cues.

## Recursive splitting

Recursive splitting walks a hierarchy of separators — paragraphs, then
sentences, then words — and splits at the highest level that keeps chunks
within the size limit. Preserves structure better than fixed-size.
""".strip()

Strategy 1 — Fixed-size chunking

Split at N characters (or tokens) with M characters of overlap. No awareness of structure, sentences, or semantics. Fast, deterministic, and often the right default when the corpus is uniform.

def fixed_size_chunk(text: str, size: int = 400, overlap: int = 50) -> list[Chunk]:
    chunks = []
    start = 0
    i = 0
    while start < len(text):
        end = min(start + size, len(text))
        chunks.append(Chunk(
            text=text[start:end],
            index=i,
            strategy="fixed",
            metadata={"start": start, "end": end},
        ))
        start = end - overlap
        i += 1
    return chunks


show(fixed_size_chunk(SAMPLE, size=200, overlap=30))

When it wins: uniform prose, logs, chat transcripts, source code where line boundaries don't carry meaning. Also the right default when the corpus is small enough that chunking strategy doesn't move retrieval quality much.

When it loses: structured documents where cutting at an arbitrary character boundary breaks a heading away from its content, or splits a code block mid-function.

Strategy 2 — Recursive character splitting

Walks a priority list of separators — paragraph breaks, then line breaks, then sentences, then words, then characters — and splits at the highest-priority separator that keeps chunks under the size limit. This is LangChain's RecursiveCharacterTextSplitter, and it's the right default for mixed structured content.

from langchain_text_splitters import RecursiveCharacterTextSplitter


def recursive_chunk(text: str, size: int = 400, overlap: int = 50) -> list[Chunk]:
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=size,
        chunk_overlap=overlap,
        # Priority order — try to split at paragraph break first, then line, etc.
        separators=["\n\n", "\n", ". ", " ", ""],
    )
    raw = splitter.split_text(text)
    return [
        Chunk(text=c, index=i, strategy="recursive", metadata={})
        for i, c in enumerate(raw)
    ]


show(recursive_chunk(SAMPLE, size=200, overlap=30))

When it wins: mixed-structure prose — documentation, articles, Notion pages, Markdown with occasional lists and code blocks. The default for most team docs.

When it loses: very long paragraphs without internal breaks (the splitter falls back to character-level and loses the benefit), or when the separators-list doesn't match the document format (for example, source code where the hierarchy is class → method → statement).

Strategy 3 — Sentence-boundary chunking

Split the document into sentences first, then greedily group consecutive sentences into chunks until a size budget is hit. Chunks never cut mid-sentence, which keeps embeddings cleaner.

import re


def sentence_chunk(text: str, size: int = 400) -> list[Chunk]:
    # Split on sentence terminators. Not perfect — abbreviations like "e.g."
    # and "Dr." will trip a naive regex. For production, use spaCy or NLTK.
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())

    chunks: list[Chunk] = []
    current: list[str] = []
    current_len = 0
    i = 0

    for s in sentences:
        if current_len + len(s) > size and current:
            chunks.append(Chunk(
                text=" ".join(current),
                index=i,
                strategy="sentence",
                metadata={"n_sentences": len(current)},
            ))
            i += 1
            current = [s]
            current_len = len(s)
        else:
            current.append(s)
            current_len += len(s) + 1  # +1 for the joining space

    if current:
        chunks.append(Chunk(
            text=" ".join(current),
            index=i,
            strategy="sentence",
            metadata={"n_sentences": len(current)},
        ))

    return chunks


show(sentence_chunk(SAMPLE, size=200))

When it wins: Q&A corpora, FAQ documents, interview transcripts, dialogue. Any corpus where the natural unit of meaning is a sentence or two.

When it loses: code (semicolons terminate statements but the function is the meaningful unit), heavily-abbreviated text that confuses sentence tokenisers, non-English scripts without clear sentence terminators.

Strategy 4 — Document-structure chunking

Use the document's own structural markers — markdown headings, HTML <h1>/<h2>, docstring sections — to define chunk boundaries. Each chunk is a self-contained section. Chunk sizes end up uneven, which is fine as long as no single section blows past the embedding model's context window.

def markdown_section_chunk(text: str, min_size: int = 100) -> list[Chunk]:
    # Split at any level of markdown heading. A chunk starts at a heading and
    # ends at the next heading of equal or higher level.
    lines = text.split("\n")
    chunks: list[Chunk] = []
    buffer: list[str] = []
    current_heading = ""
    i = 0

    def flush():
        nonlocal i
        if not buffer:
            return
        section = "\n".join(buffer).strip()
        if len(section) >= min_size:
            chunks.append(Chunk(
                text=section,
                index=i,
                strategy="structure",
                metadata={"heading": current_heading},
            ))
            i += 1

    for line in lines:
        if re.match(r"^#{1,6}\s", line):
            flush()
            buffer = [line]
            current_heading = line.lstrip("# ").strip()
        else:
            buffer.append(line)
    flush()

    return chunks


show(markdown_section_chunk(SAMPLE))

When it wins: technical documentation, markdown knowledge bases, structured reports, policy documents, textbook chapters. Any corpus where the writer deliberately organised ideas under headings.

When it loses: unstructured prose (novels, email, chat), documents where sections are too large to embed as single chunks, documents where section titles are generic ("Overview", "Introduction") and don't help the embedding carry context.

Strategy 5 — Semantic chunking

Embed every sentence, then look at the cosine similarity between adjacent sentence embeddings. Where the similarity drops below a threshold, that's a topic boundary — split there. Chunks end up grouped by semantic coherence rather than character count.

More expensive than the other four (one embedding call per sentence at index time), but produces the cleanest chunks when the corpus is dense and topics shift smoothly within sections.

import numpy as np
from sentence_transformers import SentenceTransformer


_model = None


def _embedder():
    global _model
    if _model is None:
        _model = SentenceTransformer("all-MiniLM-L6-v2")
    return _model


def semantic_chunk(
    text: str,
    similarity_threshold: float = 0.5,
) -> list[Chunk]:
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
    if len(sentences) < 2:
        return [Chunk(
            text=text,
            index=0,
            strategy="semantic",
            metadata={"n_sentences": len(sentences)},
        )]

    embeddings = _embedder().encode(sentences)
    # Cosine similarity between each pair of adjacent sentences
    sims = []
    for i in range(len(sentences) - 1):
        a, b = embeddings[i], embeddings[i + 1]
        sims.append(float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))))

    chunks: list[Chunk] = []
    buffer = [sentences[0]]
    i = 0
    for idx, sim in enumerate(sims):
        if sim < similarity_threshold:
            # Topic shift detected — close the current chunk
            chunks.append(Chunk(
                text=" ".join(buffer),
                index=i,
                strategy="semantic",
                metadata={"split_similarity": round(sim, 3)},
            ))
            i += 1
            buffer = [sentences[idx + 1]]
        else:
            buffer.append(sentences[idx + 1])

    if buffer:
        chunks.append(Chunk(
            text=" ".join(buffer),
            index=i,
            strategy="semantic",
            metadata={},
        ))
    return chunks


show(semantic_chunk(SAMPLE, similarity_threshold=0.5))

The similarity threshold is a tunable parameter — lower values produce fewer, larger chunks (more tolerant of topic drift); higher values produce more, smaller chunks (stricter about topic coherence). Start at 0.5 and adjust against an evaluation set.

When it wins: long essays, academic papers, dense technical writing where topics shift gradually without clear structural markers. News archives, podcast transcripts, meeting notes.

When it loses: short documents (not enough sentences to form clusters), already-structured content (document-structure chunking is cheaper and matches reality), heterogeneous corpora where no single similarity threshold works across documents.

Picking a strategy — the rough rule

Match the strategy to the dominant shape of the source:

Uniform prose, logs, chat transcripts → fixed-size.
Mixed Markdown / HTML / Notion-style docs → recursive.
Q&A, FAQs, dialogue, interviews → sentence-boundary.
Technical docs, textbooks, policy, specs with headings → document-structure.
Long essays, academic papers, meeting notes without structural markers → semantic.

Benchmark two or three candidates on a held-out evaluation set before committing. In practice, recursive-with-markdown-separators is a defensible default for 70% of production corpora in 2026, and the other four are specialised wins for the remaining 30%.

What this article deliberately does NOT cover

Chunking in production has a longer tail than the five strategies above. The parts not covered here, and where to find them:

A one-page decision table that maps corpus shape to strategy in 30 seconds.
Edge cases: multilingual text with mixed scripts, PDFs with tables that get destroyed by naive extractors, legal contracts with nested clause structures.
Chunk metadata schema — what fields to store alongside each chunk so retrieval can filter and the generator can cite.
Chunker versioning — rolling out a new strategy without re-embedding the entire corpus from scratch.
Delta reindex strategy — detecting which chunks actually changed when source documents are edited, so only those get re-embedded. Cuts embedding cost on small edits by ~99%.

Those live in Chapter 2 of The RAG Production Handbook — launching November 9, 2026. First 20 buyers at ₹699 (regular launch ₹999, rising to ₹1,499 on Dec 1).

Build a RAG pipeline from scratch — the actually simple version

aiunplugged — Sat, 10 Oct 2026 17:10:17 +0000

Originally published on aiunplugged.in — cross-posting for the Dev.to community.

Every RAG tutorial online starts with LangChain, LlamaIndex, or a hosted vector database signup. None of that is necessary to understand what a RAG pipeline actually is. This walkthrough builds one in a single Python file with three libraries — no framework, no cloud accounts, no config files.

What RAG actually is

RAG (Retrieval-Augmented Generation) is a pattern where an LLM answers a question using text pulled from a private document store at query time, not from what the model memorized during training. It exists because pretrained LLMs don't know a team's docs, a product's changelog, or last week's meeting notes, and fine-tuning every time the source changes is impractical.

When RAG is worth adding

RAG makes sense when at least one of these is true:

Source data changes often (docs, tickets, chat logs, code)
Source data is private (company internals, customer records)
Source data is large — thousands of pages — and answers must cite specific passages

If the entire dataset fits in a single 200K-token context window (roughly 500 pages) and doesn't change between requests, RAG is often overkill — sending the whole corpus with the question is simpler and sometimes cheaper. That trade-off is worth its own walkthrough, coming later in this series.

The 4-step pipeline

Every RAG system, regardless of framework or vendor, does some version of these four steps:

Chunk — split source documents into smaller passages
Embed — convert each chunk into a vector (a list of floats)
Store — save chunks + vectors in a vector database
Retrieve + generate — embed the user's question, find the closest chunks, hand them to an LLM

Everything else — reranking, hybrid search, query rewriting, evaluation — is optional refinement on top of this loop.

Setup

Three libraries, one pip install:

pip install chromadb sentence-transformers anthropic

chromadb — a small local vector database (no server, no signup, in-memory or disk-backed)
sentence-transformers — the embedding model, downloaded once on first run
anthropic — client for Claude, used for the answer step

The embedding model (all-MiniLM-L6-v2) has 22 million parameters (~90 MB download) and runs on CPU in milliseconds. The Claude client needs an API key from console.anthropic.com — Haiku 4.5 is $1 per million input tokens, so a full test session costs cents.

Set the key as an env var:

export ANTHROPIC_API_KEY=sk-ant-...

Step 1 — Chunk the documents

A "chunk" is a small passage the embedding model can turn into one vector. Chunks that are too big (thousands of tokens) blur meaning across topics. Too small (single sentences) lose surrounding context. A good starting point: 200-400 words per chunk with a 40-word overlap between adjacent chunks.

For this tutorial the source is a short list of imaginary product-doc passages held in a Python list:

docs = [
    "Aurora API rate limits: free tier is 60 requests per minute. Paid tiers start at 600 rpm.",
    "Aurora API authentication uses a bearer token in the Authorization header.",
    "Aurora supports Python, Go, and Node.js SDKs. Community SDKs exist for Ruby and Rust.",
    "Aurora billing is per-request. Failed requests (5xx) are not charged. 4xx requests are charged.",
    "Aurora response latency: p50 is 120ms, p99 is 800ms across all regions.",
    "Aurora data residency: EU customers can pin storage to Frankfurt. US customers use us-east-1 by default.",
]

Each entry is already a self-contained chunk, so no splitting is needed here. For real documents (Markdown files, PDFs, HTML pages), a character-based splitter works for a first pass:

def chunk_text(text: str, size: int = 1500, overlap: int = 200) -> list[str]:
    """Split text into overlapping character windows. Cheap and predictable."""
    chunks = []
    start = 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap
    return chunks

Character splitting is imperfect — it can cut mid-sentence — but for a first pipeline it beats spending an hour picking the "right" splitter. Better strategies (heading-aware, semantic, sentence-boundary) get their own deep-dive later in this series.

Step 2 — Embed the chunks

An embedding is a fixed-length vector that represents the meaning of a text passage. Chunks with similar meanings sit close together in vector space; unrelated chunks are far apart. That distance is the entire mechanism behind retrieval.

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embedder.encode(docs).tolist()

all-MiniLM-L6-v2 outputs 384-dimensional vectors. The .tolist() call converts the NumPy array into plain Python lists so Chroma can serialize them.

First run downloads the model to ~/.cache/huggingface/ — a one-time 22 MB download.

Step 3 — Store chunks and embeddings in a vector database

Chroma runs in-process. EphemeralClient() keeps everything in RAM (fine for tutorials and tests). PersistentClient(path="./chroma_data") writes to disk and survives restarts.

import chromadb

client = chromadb.EphemeralClient()
collection = client.create_collection(name="aurora_docs")

collection.add(
    ids=[f"doc_{i}" for i in range(len(docs))],
    documents=docs,
    embeddings=embeddings,
)

Every stored item needs a unique id. Chroma keeps the raw document text alongside the embedding, so retrieval later returns both the vector match and the original text to feed to the LLM.

Step 4 — Retrieve, then generate the answer

Retrieval is symmetric with storage: embed the user's question with the same model, ask Chroma for the closest N chunks, and hand them to the LLM as context.

import anthropic

def answer(question: str, k: int = 3) -> str:
    q_embedding = embedder.encode([question]).tolist()

    results = collection.query(
        query_embeddings=q_embedding,
        n_results=k,
    )
    context = "\n\n".join(results["documents"][0])

    llm = anthropic.Anthropic()
    reply = llm.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=400,
        messages=[{
            "role": "user",
            "content": (
                "Answer the question using ONLY the context below. "
                "If the answer is not in the context, say so.\n\n"
                f"Context:\n{context}\n\n"
                f"Question: {question}"
            ),
        }],
    )
    return reply.content[0].text

Two details worth pinning down:

Use the same embedding model for storage and retrieval. Mixing models silently breaks similarity math — vectors from two different models are not directly comparable, and search results become noise.
Instruct the model to say "I don't know." Without that clause, LLMs will invent an answer from prior training when retrieval misses. It is one of the cheapest reliability wins in RAG.

The full script

Everything above, in one file (rag.py):

import chromadb
import anthropic
from sentence_transformers import SentenceTransformer

docs = [
    "Aurora API rate limits: free tier is 60 requests per minute. Paid tiers start at 600 rpm.",
    "Aurora API authentication uses a bearer token in the Authorization header.",
    "Aurora supports Python, Go, and Node.js SDKs. Community SDKs exist for Ruby and Rust.",
    "Aurora billing is per-request. Failed requests (5xx) are not charged. 4xx requests are charged.",
    "Aurora response latency: p50 is 120ms, p99 is 800ms across all regions.",
    "Aurora data residency: EU customers can pin storage to Frankfurt. US customers use us-east-1 by default.",
]

embedder = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embedder.encode(docs).tolist()

client = chromadb.EphemeralClient()
collection = client.create_collection(name="aurora_docs")
collection.add(
    ids=[f"doc_{i}" for i in range(len(docs))],
    documents=docs,
    embeddings=embeddings,
)

def answer(question: str, k: int = 3) -> str:
    q_embedding = embedder.encode([question]).tolist()
    results = collection.query(query_embeddings=q_embedding, n_results=k)
    context = "\n\n".join(results["documents"][0])
    llm = anthropic.Anthropic()
    reply = llm.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=400,
        messages=[{
            "role": "user",
            "content": (
                "Answer the question using ONLY the context below. "
                "If the answer is not in the context, say so.\n\n"
                f"Context:\n{context}\n\nQuestion: {question}"
            ),
        }],
    )
    return reply.content[0].text

print(answer("Are failed requests charged?"))

Expected output:

No — failed requests that return a 5xx status are not charged. 4xx requests are charged.

That is a complete RAG loop.

What's still missing before this is production-grade

The pipeline above works, but a production RAG system adds five things:

A persistent vector store. EphemeralClient() loses data on restart. Swap for PersistentClient(path=...), or graduate to Qdrant, pgvector, or Pinecone when the collection grows past a few hundred thousand chunks. A comparison of vector databases is coming next in this series.
A smarter chunker. Markdown-aware, heading-aware, or semantic splitters preserve context better than fixed-size character windows. A deep-dive on chunking strategies follows.
Metadata filtering. Attaching {"source": "changelog", "date": "2026-08-01"} to each chunk lets retrieval filter by tag or date range, not similarity alone.
Reranking. Pull 20 candidates from the vector DB, then rerank them with a cross-encoder before sending the top 3 to the LLM. Cuts hallucinations and improves precision.
Evaluation. Something that measures whether the pipeline actually returned the right answer, not just "an" answer. Also getting its own article in this series.

Everything past those is optimization. The four-step loop above is the whole idea.

Common mistakes at this stage

Storing the whole document as one chunk. Retrieval then returns the entire document every time, and the LLM ignores the middle.
Chunking on newlines only. A single paragraph in a well-formatted document can be 2000 words. Fix chunk size in characters or tokens, not lines.
Using different embedding models for storage and query. All similarity math becomes noise; results feel random.
Sending the retrieved chunks without an instruction to only use them. The LLM blends prior training with the new context and hallucinates confidently.
Skipping the "I don't know" clause. Guarantees a plausible-sounding wrong answer whenever retrieval misses.

Bottom line

RAG is four steps: chunk, embed, store, retrieve+answer. Everything else — LangChain, LlamaIndex, hosted vector DBs, hybrid search, reranking, query rewriting — is scaffolding built on top of this loop. Once the version above works on real documents, layering the extras one at a time reveals exactly what each one changes and why.

If you've worked through this and are now wrestling with the production side (chunking that doesn't lose context, picking vector DBs, modeling costs at scale), I'm writing a handbook on exactly that — ships next month. More here.

Kubernetes Pods stuck in Pending — 8 causes and exact fixes

aiunplugged — Fri, 02 Oct 2026 18:45:41 +0000

A Pod stuck in Pending means Kubernetes cannot schedule it onto any node. The Pod is created but no container is running. Here are the 8 real causes ranked by frequency, with exact diagnostic commands.

First — always run this

kubectl describe pod <pod-name>

Scroll to the Events: section at the bottom. The scheduler explains WHY it can't place the Pod. Every fix below starts from this event message.

Cause 1 — Insufficient CPU or memory on nodes

Event message:

0/3 nodes are available: 3 Insufficient cpu.

The Pod requests more resources than any node has free.

Diagnose:

kubectl top nodes
kubectl describe nodes | grep -A 5 "Allocated resources"

Fix:

Reduce Pod's resources.requests.cpu / memory if over-requested
Scale cluster (add nodes)
Free capacity by removing unused Deployments

Cause 2 — Node selector or affinity doesn't match

Event message:

0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector.

Pod requires nodes with specific labels but none have them. (The scheduler uses the same NodeAffinity plugin for both nodeSelector and nodeAffinity, hence the combined message.)

Diagnose:

kubectl get nodes --show-labels
kubectl get pod <pod-name> -o yaml | grep -A 5 nodeSelector

Fix:

Add the required label to a node: kubectl label node <node> disktype=ssd
Or remove the nodeSelector from Pod spec if not needed

Cause 3 — Taints without matching tolerations

Event message:

0/3 nodes are available: 3 node(s) had untolerated taint.

Nodes have taints (e.g. NoSchedule) that Pod doesn't tolerate.

Diagnose:

kubectl describe nodes | grep -A 2 Taints

Fix:

Add matching toleration to Pod spec:

tolerations:
- key: "special"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

Or remove the taint from node: kubectl taint node <node> special-

Cause 4 — PersistentVolumeClaim not bound

Event message:

persistentvolumeclaim "data-pvc" not found

0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.

Pod references a PVC that doesn't exist or has no matching PV.

Diagnose:

kubectl get pvc
kubectl describe pvc <pvc-name>

Fix:

Create the missing PVC
Check StorageClass exists: kubectl get storageclass
Verify dynamic provisioning is working (CSI driver installed)

Cause 5 — Image pull secret missing (private registry)

Event message:

Failed to pull image: unauthorized

Pod's image is in a private registry, no credentials configured. Note: kubectl get pods will show STATUS as ErrImagePull or ImagePullBackOff for this case — the Pod's .status.phase is technically still Pending, but readers watching the STATUS column won't see the word "Pending." Include this in your triage when a Pod visibly appears not Pending but is stuck.

Diagnose:

kubectl get pod <pod-name> -o yaml | grep imagePullSecrets
kubectl get secrets | grep regcred

Fix:

kubectl create secret docker-registry regcred \
  --docker-server=<registry-url> \
  --docker-username=<user> \
  --docker-password=<pass>

Add to Pod spec:

spec:
  imagePullSecrets:
  - name: regcred

Cause 6 — Pod anti-affinity conflicts

Event message:

0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules.

Pod has anti-affinity requiring separation from other Pods, but no eligible node.

Diagnose:

kubectl get pod <pod-name> -o yaml | grep -A 20 affinity
kubectl get pods -o wide

Fix:

Add more nodes (so anti-affinity has room to spread)
Relax anti-affinity from requiredDuringScheduling to preferredDuringScheduling

Cause 7 — Namespace ResourceQuota rejects Pod creation

Important distinction: unlike causes 1-6 and 8, a ResourceQuota violation prevents the Pod object from being created at all. The API server's admission controller rejects the create request with 403 Forbidden, so there's no Pod stuck in Pending — the Pod never exists. Include this in your triage when kubectl create / kubectl apply returns an error and no Pod appears in kubectl get pods.

Error message (from the CLI, not the Pod events):

Error from server (Forbidden): pods "foo" is forbidden: exceeded quota: <quota-name>, requested: ..., used: ..., limited: ...

Diagnose:

kubectl describe quota -n <namespace>
kubectl apply -f pod.yaml   # observe the Forbidden error text

Fix:

Delete unused Pods in namespace to free quota
Increase quota: kubectl edit quota <quota-name>
Deploy to a different namespace with headroom

Cause 8 — Scheduler broken or overloaded

Rare, but happens on stressed clusters.

Diagnose:

kubectl get pods -n kube-system | grep scheduler
kubectl logs -n kube-system <scheduler-pod-name>

Fix:

Restart scheduler pod
Check for scheduler configuration issues
On managed K8s (EKS/GKE), this usually resolves itself

The universal debug checklist

Copy this and run through it every time:

# Basic status
kubectl get pod <pod-name>

# Detailed events (THE most important)
kubectl describe pod <pod-name>

# Node status
kubectl get nodes
kubectl describe nodes

# Recent events across cluster
kubectl get events --sort-by='.lastTimestamp' | tail -20

# Related resources
kubectl get pvc
kubectl get storageclass
kubectl get secrets

Reproduce this yourself (browser sandbox)

Free Kubernetes playground: https://killercoda.com/playgrounds/scenario/kubernetes

Create a pod that will be Pending:

kubectl run stuck-pod --image=nginx --overrides='{"apiVersion":"v1","spec":{"nodeSelector":{"nonexistent":"label"}}}'

Then:

kubectl describe pod stuck-pod

Screenshot the Events section. Note the "didn't match node selector" message.

Clean up:

kubectl delete pod stuck-pod

Prevention

Set sensible defaults from the start:

Resource requests — not too high, not too low. Start with 100m CPU, 128Mi memory. Adjust based on actual usage.
No nodeSelector unless truly needed. Default schedulers work well.
Test PVCs before deploying real workloads — verify storage class works.
Use kubectl top regularly to see node capacity trends.

Bottom line

Every Pending pod has a specific reason. The scheduler always tells you what it is via kubectl describe. Match the event message to one of the 8 causes above. Apply the corresponding fix. Done in 5 minutes.

Fix: kubectl exec fails with 'connection refused' — 5 real causes

aiunplugged — Sat, 26 Sep 2026 18:56:13 +0000

Trying to exec into a running pod and Kubernetes refuses:

error: unable to upgrade connection: pod does not exist

Or:

Error from server: error dialing backend: dial tcp 10.42.0.7:10250: connect: connection refused

The pod shows Running in kubectl get pods, yet exec doesn't work. Here are the real causes and how to fix each.

Diagnose first

Before guessing, gather facts:

For RBAC-related issues, also check what permissions the current user has:

# Confirm pod is actually running
kubectl get pod <pod-name> -n <namespace>

# Check node the pod is on
kubectl get pod <pod-name> -n <namespace> -o wide

# Check the node's status
kubectl get node <node-name>

# Check kubelet logs on the node (if you have SSH access)
journalctl -u kubelet -n 100

The exact error message narrows down which of the 5 causes below applies.

Cause 1 — kubelet not reachable from API server

kubectl exec requires the API server to open a tunnel to the kubelet on port 10250. If a firewall or security group blocks that port, exec fails while everything else works.

Symptom: error like dial tcp <node-ip>:10250: connect: connection refused or i/o timeout.

Fix:

Cloud clusters (EKS/GKE/AKS): check that the control plane security group can reach worker nodes on 10250/TCP. On EKS, this is the eksClusterSecurityGroup inbound rule on the node security group.
Self-hosted: verify iptables or ufw allows 10250 from control plane nodes.

# On the node, verify kubelet is listening
sudo ss -tlnp | grep 10250

# From the control plane, test connectivity
nc -zv <node-internal-ip> 10250

If the port is closed, add the firewall rule. If open but still failing, jump to cause 2.

Cause 2 — kubelet is running but crashed the streaming server

kubelet has a separate HTTPS server for exec/attach/logs. It sometimes crashes without taking down the whole kubelet.

Symptom: logs works but exec and attach fail. Or intermittent failures.

Fix — restart kubelet:

sudo systemctl restart kubelet
sudo journalctl -u kubelet -f

Watch for errors during startup. If kubelet keeps crashing, the underlying issue is often disk pressure, memory pressure, or a stale container runtime socket.

Cause 3 — pod is Running but container isn't ready for TTY

Distroless containers, scratch images, or containers without a shell installed have no /bin/sh — exec fails immediately.

Symptom: OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Fix — use an ephemeral debug container (Kubernetes 1.23+):

kubectl debug -it <pod-name> -n <namespace> \
  --image=busybox:1.36 \
  --target=<container-name>

This attaches a debug container to the same pod's namespace, giving you a shell that CAN inspect the target container's processes, network, and mounts. Works even for distroless.

Cause 4 — RBAC blocking your user from exec

You can list pods but not exec into them.

Symptom: Error from server (Forbidden): pods "<name>" is forbidden: User "<you>" cannot create resource "pods/exec"

Diagnose:

kubectl auth can-i create pods/exec -n <namespace>

If it returns no, your Role or ClusterRole is missing the pods/exec sub-resource.

Fix — grant exec permission:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-exec
  namespace: YOUR_NAMESPACE
rules:
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

Bind it to your user with a RoleBinding and retry.

Cause 5 — kubectl version mismatch with cluster

Very old kubectl talking to a much newer cluster (or vice versa) can hit exec-protocol incompatibilities.

Symptom: exec starts, prints a partial banner, then disconnects. Or error: unable to upgrade connection: unauthorized.

Diagnose:

kubectl version

The default output shows both client and server versions. Kubernetes officially supports client/server skew of +/- 1 minor version. Beyond that, exec is unreliable.

Fix — align kubectl version:

# macOS
brew install kubernetes-cli
# or pinned to a specific minor version:
brew install kubernetes-cli@1.29

# Linux — pinned install
curl -LO "https://dl.k8s.io/release/v1.29.0/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Match kubectl to your cluster's minor version and retry.

The universal exec debug flow

Every stuck-exec case, run in order:

# 1. Pod actually running?
kubectl get pod <pod> -n <ns>

# 2. Node healthy?
kubectl get node -o wide

# 3. Can I even see logs? (rules out kubelet-total-failure)
kubectl logs <pod> -n <ns> --tail=5

# 4. RBAC permission for exec?
kubectl auth can-i create pods/exec -n <ns>

# 5. Try debug container (bypasses distroless issue)
kubectl debug -it <pod> --image=busybox --target=<container>

Ninety percent of exec failures resolve after step 5.

Prevention

Bake these into every cluster setup:

Explicit security group rules for control-plane to node on 10250/TCP; documented in Terraform, not vibes-based
Ephemeral containers enabled on all managed clusters (default in K8s 1.25+; verify with kubectl api-resources | grep ephemeral)
Standardized debug image in a private registry — a common debug:latest with curl, bind-utils, netcat, iputils pre-installed
RBAC role called debugger with pods/exec permission, granted temporarily via kubectl auth reconcile for incident response
Regular kubectl version audits across the team — pin to cluster version in your dotfiles

Bottom line

kubectl exec failing usually points at one of five layers: network to kubelet, kubelet streaming server, missing shell in the container, RBAC, or version skew. Diagnose with kubectl get pod -o wide, kubectl auth can-i create pods/exec, and kubectl debug. The debug container is a strong Swiss army knife — reach for it early rather than fighting distroless exec issues directly.

Claude API vs OpenAI API for developers — 2026 practical comparison

aiunplugged — Fri, 18 Sep 2026 12:54:37 +0000

This post was originally published on aiunplugged.in.

Two dominant APIs power most 2026 AI apps: Anthropic's Claude and OpenAI's GPT. Both are excellent. The right choice depends on specific use cases. Here is a practical comparison based on daily developer use, with current 2026 model families and pricing.

The 30-second summary

Claude — better at long-form reasoning, careful code, safety-critical outputs, honest limits
OpenAI — better at general-purpose tasks, mature ecosystem, plugins, image generation
Both — production-ready, well-documented, actively improved

For code + reasoning tasks, Claude tends to win in 2026. For general apps, either works.

Model comparison (2026)

Claude models (Anthropic)

The current Claude 5 family plus Haiku 4.5:

Claude Fable 5.1 — newest release, high-capability with faster response times
Claude Opus 5 — most capable for deep reasoning, slower, most expensive
Claude Sonnet 5 — balanced default recommendation, strong on code and reasoning
Claude Haiku 4.5 — fast, cheap, still highly capable for lighter tasks

OpenAI models

The GPT-5.6 family is OpenAI's current flagship, with older GPT-5 and GPT-5.4 still available:

GPT-5.6-sol — flagship, most capable
GPT-5.6-terra — mid-tier, balanced cost/quality
GPT-5.6-luna — cheapest of the flagship family, high volume
GPT-5.4 — previous generation, still supported
GPT-5 — original GPT-5, cheapest input pricing in the modern lineup

Pricing per million tokens (verified Sep 2026)

Input / Output pricing (USD per million tokens):

Model	Input	Output
Claude Opus 5	$5	$25
Claude Sonnet 5	$2 - $3	$10 - $15
Claude Haiku 4.5	$1	$5
GPT-5.6-sol	$5	$30
GPT-5.6-terra	$2	$12
GPT-5.6-luna	$0.20	$1.20
GPT-5.4	$2.50	$15
GPT-5	$0.625	$5

Notes:

Sonnet 5 launched at $2/$10 as introductory pricing; sources report the standard rate as either $2/$10 or $3/$15 depending on when tracked
OpenAI: batch mode halves rates, fast mode doubles them, long-context (>272K tokens) bills roughly double
Cached input gets ~90% discount on GPT-5.6 and GPT-5; 50-75% on legacy models
Prices change frequently. Always verify current pricing at anthropic.com/pricing and openai.com/pricing before committing

Cheapest quality option: GPT-5.6-luna for high volume, Claude Haiku 4.5 for balance. GPT-5 at $0.625 input is the cheapest input-heavy option in the modern lineup.

Where Claude wins

Long context handling

Claude Sonnet 5 and Opus 5 handle 200K tokens (~500 pages) effectively. Their recall over long context is measurably better than comparable OpenAI models at similar sizes.

Use for:

Analyzing entire codebases
Processing long documents
Multi-turn conversations that accumulate

Careful code output

Claude tends to be more precise on code:

Fewer hallucinated APIs
Better adherence to instructions
More self-aware ("I'm not sure about X" instead of confident wrong answers)

Use for:

Refactoring
Debugging
Code review
Complex logic

Following complex instructions

Claude is more likely to follow detailed multi-step instructions accurately.

Use for:

Agentic workflows
Multi-step pipelines
Structured output requirements

Honest uncertainty

When Claude doesn't know something, it more often says so. OpenAI models tend to answer confidently even when wrong.

Use for:

Fact-critical outputs
User-facing apps where wrong answers cause harm
Safety-important use cases

Where OpenAI wins

Multimodal (image + audio)

The GPT-5.6 family handles images, audio, and text in a single call. Voice mode is polished.

Use for:

Image analysis apps
Voice-driven apps
Screenshot understanding

Ecosystem maturity

More libraries, examples, SDKs
More Stack Overflow answers
More third-party tools built on OpenAI first
Assistants API (built-in state, tools, threads)

Use for:

Prototyping with existing tools
When leveraging community examples matters

Image generation (DALL-E)

Native image generation via API. Claude doesn't offer image generation.

Use for:

Image gen features
Combined text + image workflows

Streaming reliability

Both stream. OpenAI's streaming has slightly better tooling support (more libraries).

Use for:

Chat UIs needing token-by-token streaming

Practical patterns

Building a chat interface

Both work. Choice comes down to:

Model quality per dollar (Claude Sonnet 5 currently strong for reasoning quality)
Streaming library support (both fine)
Tool use / function calling (both mature)

Code generation for developer tools

Claude Sonnet 5 or Opus 5. Consistently better code quality with fewer hallucinated APIs.

Summarization

Both. GPT-5.6-luna is cheapest. Claude Haiku 4.5 is competitive.

RAG (Retrieval-Augmented Generation)

Claude's long context wins for RAG where you stuff many documents into context. Better recall.

For traditional RAG (small chunks retrieved, then generation), both work equally.

Agentic workflows

Claude's better instruction-following makes it easier to build reliable agents. OpenAI Assistants API provides more built-in scaffolding.

Safety-critical applications

Claude has stronger built-in safety training. Refuses harmful requests more reliably. Better for consumer-facing products.

API differences that matter

System prompts

Claude: dedicated system parameter separate from messages

{
  "model": "claude-sonnet-5",
  "system": "You are a helpful assistant.",
  "messages": [{"role": "user", "content": "Hello"}]
}

OpenAI: system message inside messages array

{
  "model": "gpt-5.6-sol",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello"}
  ]
}

Tool use / function calling

Both support tool calling with similar patterns. Claude's tool use is more forgiving of schema variations. OpenAI's is stricter but has better docs.

Streaming

Both use Server-Sent Events. Response formats differ slightly. All popular SDKs abstract this.

Prompt caching

Both offer prompt caching to reduce cost on repeated inputs. Different implementations:

Claude: cache_control on message blocks, 5-minute default TTL
OpenAI: automatic caching on longer prompts (~500+ tokens); ~90% discount on GPT-5.6 cached input

For repeated system prompts, both cut costs significantly.

Which SDK to use

Both providers offer official SDKs:

Claude:

Python: pip install anthropic
JavaScript: npm install @anthropic-ai/sdk

OpenAI:

Python: pip install openai
JavaScript: npm install openai

Both are well-maintained, well-documented, and stable.

Rate limits

Both providers have tier-based rate limits that scale with spend:

Tier 1 (new accounts): low limits, may hit quickly
Higher tiers: unlock automatically as you spend

Neither is friendlier than the other on rate limits. If building high-volume, budget for tier progression.

When to use both

Some applications benefit from using both:

Primary: Claude for reasoning-heavy tasks
Fallback: OpenAI if Claude is rate-limited or down
Specialized: OpenAI DALL-E for images, Claude for text
A/B testing: compare quality on your specific tasks

LiteLLM or OpenRouter provide unified interfaces to both (and other providers). Useful for multi-model apps.

Testing quality for your use case

Do not rely on benchmarks. Test on YOUR actual prompts:

Pick 10 real prompts from your use case
Run each in Claude Sonnet 5, GPT-5.6-sol, and a cheaper option (Haiku 4.5 or GPT-5.6-luna)
Grade output quality subjectively
Note cost and latency
Pick winner based on quality/cost balance

This 30-minute test beats reading 100 comparison articles.

Reproduce this yourself

Get free credits to try both:

Anthropic — sign up at console.anthropic.com; check current signup credit offers
OpenAI — sign up at platform.openai.com; check current signup credit offers

Or use browser interfaces (no API required for testing):

Claude.ai — free tier available
ChatGPT — free tier available

Test the same prompt in both. Note which handles your specific case better.

Recommendation by profile

Building a code assistant / copilot: Claude Sonnet 5
Chat app for general users: either (OpenAI edges on ecosystem)
RAG on long documents: Claude Sonnet 5 (200K context)
Multi-modal image analysis app: GPT-5.6-sol
Image generation feature: OpenAI (Claude doesn't offer)
Safety-critical consumer product: Claude (better refusals)
Voice AI: OpenAI (native voice mode)
Agentic workflow: Claude (better instruction-following)
Cheapest possible for high volume: GPT-5.6-luna or GPT-5
Best free-tier interface: both are good

Bottom line

Neither Claude nor OpenAI is universally better. Both are excellent APIs in 2026.

For most developer-focused apps, Claude Sonnet 5 is the current best-in-class for quality-per-cost. For general apps needing multimodal or image gen, OpenAI wins.

Test on your specific use case. Do not choose based on marketing or benchmarks. And re-verify pricing before committing to production — this space moves monthly.

AWS Free Tier in 2026 — what actually stays free

aiunplugged — Sat, 12 Sep 2026 16:35:21 +0000

AWS Free Tier changed significantly on July 15, 2025 — the old "12-month free trial" model was replaced for new accounts with a 6-month plan plus credits. Existing accounts kept the legacy tier. On top of that, AWS began charging for public IPv4 addresses in Feb 2024. If your article on AWS Free Tier is older than that, its numbers are probably outdated. This guide covers the 2026 reality.

The two versions of AWS Free Tier that exist today

Legacy (accounts created before July 15, 2025):

Traditional 12-month free tier still applies
Access to Always Free services with monthly limits
Nothing changed for these accounts

New AWS Free Plan (accounts created on or after July 15, 2025):

100 USD credits at signup, up to 200 USD through onboarding activities
Plan lasts 6 months OR until credits are exhausted, whichever comes first
Account auto-closes when the plan ends
Also includes Always Free services

Always Free services (both plans): 30+ AWS services stay free forever within monthly usage limits, regardless of which plan you're on.

Always Free services — free forever within limits

These services stay free regardless of account age or plan type, subject to monthly usage limits.

Compute

Lambda — 1 million requests + 400,000 GB-seconds compute per month
DynamoDB — 25 GB storage + 25 read/write capacity units
SNS / SQS — 1 million requests per month
Step Functions — 4,000 state transitions per month

Networking

CloudFront — 1 TB data transfer + 10 million requests per month

AI / ML

Rekognition — 5,000 image analyses per month (first 12 months)
Comprehend — 50,000 units of text per month (first 12 months)
Polly — 5 million standard characters per month (first 12 months)

Monitoring

CloudWatch — 10 custom metrics, 5 GB log ingestion, 1 million API requests
X-Ray — 100,000 traces per month

Storage

Glacier — 10 GB retrieval per month (data storage costs extra)

Always Free service limits are per AWS account, not per region. Once you exceed the monthly quota, standard billing applies.

Legacy 12-month tier (pre-July 2025 accounts only)

If your account was created before July 15, 2025, the traditional 12-month free tier still applies to these services. After 12 months, standard rates kick in.

EC2 — 750 hours/month of t2.micro or t3.micro
S3 — 5 GB standard storage + 20,000 GET + 2,000 PUT requests
RDS — 750 hours/month of db.t2.micro / db.t3.micro
EBS — 30 GB storage + 2 million I/O
Elastic Load Balancer — 750 hours/month
ElastiCache — 750 hours/month of cache.t3.micro
Data Transfer — 15 GB out per month
Route 53 — first hosted zone free for 12 months (0.50 USD/month per zone after)
API Gateway — 1 million API calls per month (first 12 months)

Real cost after 12 months: if leaving these running, a small always-on EC2 + RDS + ELB stack costs around 40 to 80 USD per month.

New AWS Free Plan (post-July 2025 accounts)

New accounts don't get a 12-month allowance. Instead:

Start with 100 USD credits
Complete onboarding activities (up to 100 USD more)
Credits expire 12 months from account creation
Free Plan status ends after 6 months or when credits hit zero, whichever comes first
Account auto-closes at that point unless you upgrade to a Paid plan

You can use those credits on any AWS service, so it's more flexible than the old fixed 750-hour allocations. But the runway is shorter — most experimentation projects that used to fit in 12 months now need to fit in 6.

Hidden costs that surprise everyone

1. Public IPv4 addresses — charged since Feb 2024

This is the biggest change most guides miss. Since February 1, 2024, AWS charges 0.005 USD/hour (~3.60 USD/month) for every public IPv4 address, whether attached to a running EC2 instance or sitting idle.

Free tier exception: legacy free tier accounts (pre-July 2025) get 750 hours/month of public IPv4 usage free for the first 12 months.

What this means practically:

Every running EC2 with a public IP costs at least 3.60 USD/month beyond free tier
NAT Gateways, Load Balancers, and VPN connections with public IPs — all charged
New accounts on the AWS Free Plan don't get the 750-hour IPv4 exemption

2. Data transfer OUT

The second most common surprise. Free tier includes limited outbound bandwidth. Beyond that:

Internet egress: 0.09 USD/GB (US regions), 0.12 USD/GB (many others)
Cross-region: 0.02 USD/GB
Cross-AZ within same region: 0.01 USD/GB per direction

A hobbyist blog serving 1 TB/month costs 90 USD or more just in bandwidth.

3. RDS storage

RDS free tier includes 20 GB storage (legacy accounts only)
Beyond that: 0.115 USD/GB/month for gp2
Backup storage — first 100% of allocated is free, additional charged

4. NAT Gateway

Popular for private subnets. Costs:

0.045 USD/hour (~32 USD/month per gateway)
0.045 USD/GB data processed
Plus 0.005 USD/hour for its public IPv4 address

Not in free tier at all. Common surprise.

5. Support plans

Basic support: free
Developer support: 29 USD/month minimum
Business support: 100 USD/month or 3-10% of AWS bill

Free tier includes only Basic support.

Setting up billing alerts (critical)

Never skip this step. Set alerts BEFORE deploying anything.

AWS Console → Billing → Billing preferences
Enable "Receive Free Tier Usage Alerts"
Enter your email
Go to CloudWatch → Alarms → Create Alarm
Metric: EstimatedCharges (in USD)
Set threshold: 5, 10, 25 USD (three alarms recommended)
Send notification to your email

Now if usage spikes, you get emails before the bill arrives.

What to avoid on free tier

Don't run these on always-on free tier:

NAT Gateway — expensive, not free
Multiple availability zone deployments — cross-AZ transfer charges
CloudTrail data events — logging every S3 object access
Detailed monitoring — 1-minute EC2 metrics cost extra
VPC endpoints (7.30 USD/month per endpoint)
Multiple public IPv4 addresses — each one is 3.60 USD/month post-2024

Realistic free/cheap stack for a personal project

Given the 2024 IPv4 charges and the 2025 Free Tier restructure, here's what actually stays cheap for a personal project.

Option A — Static Website (essentially free)

S3 for storage
CloudFront for CDN
Route 53 for DNS (0.50 USD/month after 12 months)
Certificate Manager for SSL (free)
Total: essentially 0 USD for personal traffic (CloudFront's 1 TB always-free covers most personal blogs)

Option B — Full Stack Web App (legacy accounts only)

1x t3.micro EC2 (running Docker/Node) — free for 12 months, includes 750 hrs public IPv4
1x db.t3.micro RDS — free for 12 months
5 GB S3
CloudFront in front
Total: around 0 USD for first 12 months, then 40 to 80 USD/month + IPv4 charges

Option C — Serverless (works on both plan types)

Lambda for compute (1M requests always free)
DynamoDB for storage (25 GB always free)
API Gateway for HTTP endpoints (first 12 months free, then 3.50 USD per million calls)
CloudFront for CDN (1 TB always free)
Total: often 0 USD for hobbyist traffic, indefinitely

Serverless is the best free-tier fit today because it avoids the always-on EC2 costs and the IPv4 charge entirely.

Alternatives worth comparing

For personal projects, AWS free tier is not always the best deal. Verified 2026 limits:

Provider	Free Compute	Free Bandwidth	Notes
AWS (Always Free)	Lambda 1M req/month	15 GB egress + CloudFront 1 TB	Always Free services never expire
AWS (Legacy 12-mo)	750 hrs t3.micro	15 GB egress	Pre-July 2025 accounts only
AWS (New Free Plan)	100 to 200 USD credits	Included in credits	6 months or credit exhaustion
Cloudflare Pages	Unlimited static requests	Unlimited (unmetered)	500 builds/month; Functions billed via Workers plan
Vercel Hobby	1M function invocations	100 GB/month	Also 1M edge requests, 4 CPU hours
Netlify (legacy)	300 build minutes	100 GB/month	Pre-Sep 2025 accounts
Netlify (new)	300 credits/month	Credits shared with bandwidth	Post-Sep 2025 accounts, unified pool
Oracle Cloud	4 vCPU + 24 GB RAM ARM	10 TB egress	Always Free tier (verify with Oracle)
Google Cloud	1 e2-micro	200 GB egress	Always Free, region-restricted

Fly.io was previously listed as having a free tier. As of late 2024, Fly.io no longer offers a free tier — new users pay from usage.

For blogs and small static sites in 2026, Cloudflare Pages or Vercel Hobby typically beat AWS on both cost and simplicity.

Reproduce this yourself (safely)

Before deploying anything:

Complete the billing alert setup above
Never enter production credit card if just experimenting — use a low-limit card
Delete resources after testing to avoid accidental charges
Note whether your account is legacy or new — the free tier you get depends on this

For learning AWS without risk: use AWS Skill Builder (free courses) or Amazon-provided sandboxes at re:Invent events. These have hard spending limits.

Common surprise-bill scenarios

Left an EC2 instance running for 8 months → around 60 to 100 USD bill (plus IPv4 charges)
CloudFront hit by scraping bot → 500 GB egress = 45 USD
Forgot to delete RDS → around 15 USD/month × months
Set up NAT Gateway "just to try" → 32 USD/month + IPv4
Multiple EIPs allocated but not attached → 3.60 USD each per month

Real AWS bills for someone experimenting without alerts typically range 20 to 200 USD/month unexpectedly.

Bottom line

AWS Free Tier is real but requires attention, and the rules changed twice recently:

Feb 2024: public IPv4 addresses became a paid resource
July 2025: the 12-month free tier was replaced (for new accounts) with a 6-month credit-based plan
Always Free services remain the safest bet — they never expire and cover Lambda, DynamoDB, CloudFront, and 30+ others
Data transfer OUT is still the biggest hidden cost across everything
Billing alerts are mandatory, not optional

For most personal projects in 2026, Cloudflare Pages or Vercel Hobby offer more predictable cost profiles than AWS. Use AWS when you specifically need its services (Lambda, DynamoDB, ML APIs), not for basic web hosting.

Originally published on aiunplugged.in