DEV Community: saaro

The countdown is on: Kubernetes disables cgroup v1 – a migration checklist for 2026

saaro — Sun, 01 Nov 2026 12:01:11 +0000

The countdown is on: Kubernetes shuts down cgroup v1 – a migration checklist for 2026

Since Kubernetes 1.37 in August 2026 at the latest, the message is unmistakable: the kubelet no longer starts on a cgroup v1 node – unless an administrator explicitly sets the temporary failCgroupV1: false. What many teams saw as a side note in the release notes is now becoming operational reality. Kubernetes 1.34 reached the end of its support cycle on October 27, 2026, and with every upgrade to 1.35 or newer, the cgroup v2 migration is on the table. Those who do not act now risk nodes that simply fail to come up after the next upgrade.

Why cgroup v2?

cgroups (Control Groups) are the Linux kernel mechanism that Kubernetes uses to enforce CPU and memory limits for containers. cgroup v1 organised this control over several separate hierarchies – one per resource controller. This led to inconsistent behaviour across distributions and made features like In-Place Pod Resizing or reliable Memory QoS difficult.

cgroup v2 replaces this model with a single, unified hierarchy. The switch brings three concrete operational improvements:

Memory QoS with a two-tiered model: cgroup v2 introduces a soft limit with memory.high, which throttles and triggers reclaim before the OOM killer. cgroup v1 only knew the hard memory.limit_in_bytes – either the container fits within the limit, or the OOM killer cleans up. Starting with Kubernetes 1.36, the Memory QoS feature (still Alpha) can even enable graduated protection (TieredReservation): Guaranteed pods receive hard protection via memory.min, Burstable pods receive soft protection via memory.low.

Container-aware OOM handling: On cgroup v2 nodes, the kubelet sets memory.oom.group per container cgroup by default. An OOM event then kills all processes of a container together, rather than individual processes – this prevents partially malfunctioning multi-process containers.

Per-cgroup PSI metrics: Starting with Kubernetes 1.36, Pressure Stall Indicators (PSI) per cgroup are available (GA). Instead of only measuring overall node pressure, platform teams can now monitor the load of individual tenants or namespaces – and trigger alerts before the OOM killer strikes. This capability has no cgroup v1 counterpart.

The timeline: From warning to hard stop

The migration has been underway for years – but 2026 is the year of enforcement:

Release	Date	Milestone
Kubernetes 1.25	August 2022	cgroup v2 becomes stable (GA)
Kubernetes 1.31	August 2024	cgroup v1 enters maintenance mode
Kubernetes 1.35	December 2025	`failCgroupV1` defaults to `true` (Beta)
Kubernetes 1.37	August 26, 2026	Kubelet no longer starts on cgroup v1 nodes
Kubernetes 1.38	approx. December 16, 2026	Planned removal of cgroup v1 code

The official Kubernetes blog published a comprehensive guide on October 6, 2026, titled "The Shift to cgroup v2", summarising the current state. The message: failCgroupV1: false is a temporary escape hatch, not a permanent configuration mode. Those who set it buy time – but the direction is final.

Which nodes are affected?

The good news: most current distributions ship cgroup v2 by default – Ubuntu 22.04 and 24.04, Debian 12, Amazon Linux 2023, RHEL 9.4+, and Talos Linux 1.3+. The risk lies in older base images: Ubuntu 20.04, Debian 10, CentOS 7, and nodes manually reverted to v1.

The check is simple: stat -fc %T /sys/fs/cgroup/ on each node returns cgroup2fs for v2 and tmpfs for v1.

Migration checklist

Audit your fleet: Run stat -fc %T /sys/fs/cgroup/ on all nodes – note the v1 candidates.
Check the container runtime: Kubernetes 1.37 requires containerd 2.0 or newer. containerd 1.x is no longer supported.
Test legacy agents: Monitoring tools, security agents, and direct cgroup accesses (e.g., --max-old-space-size in Node.js apps) must support cgroup v2.
Update the base image: Upgrade nodes to a distribution with cgroup v2 default – or for self-managed images, set the kernel boot parameter systemd.unified_cgroup_hierarchy=1.
Validate workload behaviour: After the switch, check the CPU weight mapping (cgroup v1 shares → v2 weight) – the conversion formula has changed.
Evaluate Memory QoS: Those who want to use the Alpha features MemoryQoS and TieredReservation need cgroup v2. The official blog recommends testing the configuration in a non-production environment beforehand.

Conclusion

The cgroup v2 migration is no longer optional – it is the new standard. With Kubernetes 1.38 on the horizon (expected December 2026) and the planned removal of cgroup v1 code, the remaining time is tight. Platform teams that audit their fleet now, update legacy agents, and plan the switch will avoid the stress of a forced upgrade under time pressure. The effort pays off: cgroup v2 brings not only compliance with the deprecation path but also better resource isolation, finer-grained memory control, and the PSI transparency that is becoming increasingly essential for operating AI workloads and multi-tenant platforms.

Sources

GitOps with ArgoCD 2026: Patterns, Anti-Patterns, and Production Deployment Automation

saaro — Sun, 01 Nov 2026 12:00:35 +0000

GitOps has established itself as the standard operating model for Kubernetes in 2026, and with over 18,000 GitHub stars and CNCF graduation, Argo CD is the most widely used tool for leveraging Git as the single source of truth for deployment configurations. However, between a working installation and smooth production operations lie several typical pitfalls. This article shows which patterns have proven themselves in practice in 2026 – and which anti-patterns teams should avoid so that Argo CD becomes an amplifier, not a bottleneck.

App of Apps: The Central Organizing Pattern

The most important pattern that every Argo CD team should master early on is the App of Apps. Instead of manually managing dozens of individual Application YAML files, you define a parent Application that points to a directory in the Git repo – Argo CD automatically recognizes any Application definitions contained within and deploys them in one pass.

Typical structure for a small to medium-sized team:

infrastructure/
├── argocd/               # Parent app and Argo CD configuration
│   └── platform.yaml     # App of Apps
├── apps/                 # One folder per service
│   ├── web-app.yaml
│   ├── api-server.yaml
│   └── worker.yaml
├── manifests/            # Kubernetes manifests per service
│   └── web-app/
│       ├── deployment.yaml
│       └── service.yaml
└── environments/         # Environment overrides (Kustomize)
    ├── production/
    └── staging/

Anti-Pattern: Multiple Git repos per service. Anyone who creates a separate config repo for each microservice multiplies the administrative effort without gaining real benefits. A single monorepo for the entire GitOps configuration significantly reduces complexity and allows consistent cross-service changes in a single commit.

Sync Policy: Using Self-Healing and Pruning Correctly

Argo CD's strength lies in continuous reconciliation: a controller compares the desired state from Git with the actual state in the cluster. For production operations, two options are crucial:

selfHeal: true: Automatically reverts manual changes to the cluster. This prevents drift from ad-hoc hotfixes and forces teams to manage changes cleanly through Git.
prune: true: Removes resources that have been deleted from the Git repo from the cluster as well.

According to a practical report from OneUptime, self-healing is especially valuable for smaller teams that do not run 24/7 cluster monitoring. On the other hand, if you want a human approval process before automatic sync in production, disable automated and use Argo Rollouts or manual approvals as an additional security layer.

Anti-Pattern: Blindly enabling all sync options without understanding the consequences. Enabling prune: true in a namespace that also contains resources outside of Argo CD risks data loss. The Argo CD-controlled namespace should be clearly isolated.

Multi-Cluster and Multi-Environment with ApplicationSets

Teams deploying the same service to multiple clusters or environments benefit from ApplicationSets. A generator template automatically creates the appropriate Application definitions from a list of clusters and environments.

A typical use case: The staging service runs with one replica, the same service in production with three replicas – all defined in a single file, as Luca Berton shows in his production guide. This reduces copy-paste errors and keeps the configuration consistent.

Anti-Pattern: Manually copying Application YAMLs for each environment. As soon as a configuration exists in three or more copies, the probability of errors during updates increases exponentially.

Never Put Secrets in Git – But How Then?

Argo CD does not support native secret encryption. Committing Kubernetes Secret resources to Git anyway is a security problem – whether the repo is private or public. Common solutions in 2026:

SealedSecrets (Bitnami): Encrypted secrets that can only be decrypted by the cluster.
External Secret Stores (HashiCorp Vault, AWS Secrets Manager) with an init container that injects secrets at runtime.
External Secrets Operator: A Kubernetes operator that synchronizes secrets from external sources into the cluster.

Anti-Pattern: kubectl create secret in deployment scripts. While this bypasses the Git problem, it creates new ones: no audit trail, no rollback, no automatic sync.

Monitoring and Notifications from the Start

Argo CD offers an integrated notification system based on ConfigMaps. This allows alerts for failed syncs, OutOfSync states, or drift events to be routed to Slack, email, or webhooks. Small teams tend to postpone notifications – a mistake: especially in environments without dedicated on-call staff, the sync failure alarm is the first escalation level before manual interventions pile up.

Conclusion

Argo CD in 2026 is mature and production-proven. The patterns described – App of Apps, deliberate sync policy, ApplicationSets, clean secret management, and early notifications – help teams operate the tool effectively and securely. The most important principle remains: if the state is not in Git, it does not exist. Sticking to this rule prevents most anti-patterns from the outset.

Sources

The Platform Island: Why Nobody Deploys on Our Internal Developer Platform

saaro — Fri, 30 Oct 2026 15:21:33 +0000

Every second company has one. Most have invested six figures. And in almost every case, the most used feature is the password reset link. We're talking about Internal Developer Platforms (IDPs) – the in-house developer platforms that can do everything but are used by no one.

The Platform Island Phenomenon

You know the drill: the company buys or builds a platform. A team of platform engineers works on the architecture for months. There's a fancy UI, CI/CD pipelines, self-service interfaces. The platform is celebrated with a big launch – and then what happens: nothing.

The teams continue to work as before. They use their own scripts, their own workflows, their own directories. The platform becomes a digital ghost town.

Why?

Three reasons why platforms fail

1. The platform doesn't solve a real problem

The most common mistake: the platform was built because management wanted a platform – not because developers had a problem.

A platform that doesn't address an acute pain point will never be adopted. "Look, we have a nice UI for Kubernetes deployments" isn't a value proposition. "Your deployment used to take 45 minutes and three emails to the ops team – now it takes three minutes by a click" is.

2. Too much complexity, too little standardization

Many platforms try to map all possible workflows. The result: they can do everything, but nothing properly. The user interface is overloaded, the documentation is outdated, the interfaces are inconsistent.

The most successful platforms limit themselves to a few, but excellently implemented Golden Paths: well-defined, documented, tested standard workflows for the most common tasks.

3. No migration, no pressure

The biggest hurdle is the existing work: existing projects run with old scripts, Makefiles, and manual know-how. As long as they work, there's no reason to switch.

The platform gets built, but nobody migrates the existing projects. And without migration, the platform remains empty.

What platforms need instead

From observing working platforms (Backstage, internal developer portals at Spotify, Red Hat, Netflix), five principles can be derived:

1. Pain before Platform

Before you build a platform, find out where the pain is greatest. Measure the time teams lose on manual processes. Talk to the developers, not the managers.

A platform that addresses the biggest pain first will be adopted. A platform that can do "everything" will be ignored.

2. Golden Paths, not a workflow engine

Define exactly 3-5 standard workflows that work like a dream. Everything else is secondary.

3. Self-service with a safety net

Developers must be able to use the platform without having to write a ticket. But every action mmust be auditable and reversible.

4. Migration is not an afterthought

One third of the platform budget should go into migration. Automated migration tools, import interfaces, compatibility modes – everything that makes the switch easier.

5. Metrics and visibility

The platform must measure whether it is being used. Number of active projects, deployment frequency, time-to-production. These metrics belong in the management dashboard – and if they stagnate, that is a warning sign.

The counterexample: Spotify and Backstage

Spotify built Backstage because the infrastructure team in 50 teams with different languages, tools, and deployment processes had no standard workflow. The pain was real: a new service took weeks to get through all the checks.

Backstage automated exactly this one workflow: generate service template → connect CI/CD → comply with deployment policies → set up monitoring. No more, no less.

The result: Backstage was adopted internally at Spotify, later open-sourced, led to CNCF graduation – and is today the most widely used Internal Developer Platform in the world. Helen Greul, Head of Engineering for Backstage at Spotify, reported in an interview with The New Stack that Spotify achieved a voluntary 99% platform adoption rate. Her key insight: "With a product that's built for developers, with empathy for developers, they recognize the value."

Why? Because it perfectly automated one workflow – the one with the greatest pain.

Conclusion: Less is more

A successful platform is not a platform that can do everything. It is a platform that does one thing so well that no developer wants to work without it.

If your platform hasn't reached 80% adoption after a year, it's not the developers' fault. It's the platform design. Ask a different question: Not "What can the platform do?" but "What problem does the platform solve for whom – and how do I measure whether it is solved?"

Platform silos are not created by malicious developers, but by platforms that solve problems nobody has.

Sources

AI Agents in Production: State Management, Cost Control, and Observability 2026

saaro — Fri, 30 Oct 2026 12:00:47 +0000

The hype around AI agents is unbroken – but those who don't just demonstrate agents but operate them productively quickly hit the harsh reality: state management, exploding costs, and a lack of transparency. While the trade press reports on new frameworks and benchmarks, DevOps teams are quietly struggling with the same three problems that determine the success or failure of an agent platform.

The Cost Trap: When the Agent Becomes a Cash Machine

The biggest operational shock for teams bringing their first AI agents into production is the bill. What began as a chatbot costing a few cents per request quickly mutates into a five-figure monthly amount in agent operations. The reason: a single agentic workflow triggers not one, but 10 to 20 sequential LLM calls – planning, tool selection, execution, validation, error handling, and response generation.

An analysis by Anthropic in early 2026 found that inference costs at enterprise customers consume over 85 percent of the entire AI budget. The problem isn't the price per token – that has dropped drastically – but the sheer volume. A coding agent tasked with fixing a bug consumes an average of 18,000 tokens per task. A multi-step research agent uses 35,000 tokens. Without optimization, handling a complex support ticket with Claude Sonnet costs around $1.60 per case – with 10,000 tickets monthly, that's $16,000 for LLM inference alone.

Three Strategies for Cost Containment

The good news: there are proven levers that, combined, enable 60 to 80 percent savings.

1. Model Routing: Not every step in a workflow requires a frontier model. Routing systems like RouteLLM classify incoming requests and direct 70 percent of traffic to small, inexpensive models (under 7B parameters, under $1 per million tokens). Only 10 percent of tasks – complex architectural decisions or multi-step planning – reach the expensive frontier models. This 70/20/10 model reduces average cost per task by up to 80 percent, at 95 percent of GPT-4 quality, as a study by UC Berkeley and Canva showed.

2. Prompt Caching: System prompts, tool definitions, and context information repeat in every agent run. Prompt caching stores the computed attention tensors between runs. Providers offer discounts of 50 to 90 percent for this – Anthropic, for example, charges only $0.30 per million tokens for cached inputs instead of $3.00. In practice, this saves 40 to 60 percent of costs for tool-heavy agents.

3. Context Engineering: More context doesn't mean better results. Precise, filtered retrieval results lower token costs while improving response quality. Systems that limit their context windows to the essentials save 20 to 35 percent compared to the "maximum context" strategy.

State Management: The Invisible Bottleneck

The second operational pain point is state management. Agents are stateful by definition – they conduct conversations, remember intermediate results, and react to tool outputs. But in multi-agent systems, state quickly becomes a nightmare.

Those working with the OpenAI Agents SDK use Context Variables – Python objects available during a run but disappearing when it ends. Lasting persistence must be built yourself. Google ADK, on the other hand, comes with an explicit session-state model with namespaced storage (app:, user:, temp:) and pluggable backends like Firestore or Cloud SQL.

LangGraph is the only framework that uses checkpointing as its architectural foundation. Every state transition is automatically persisted in PostgreSQL or Redis. This enables time-travel debugging – developers can restart an agent from any earlier checkpoint, resume after a crash, or pause at human approval points. In regulated environments, this capability isn't a luxury but a compliance requirement.

The Silent Data Loss

An often-overlooked problem: in multi-agent systems, agents overwrite each other's states. If two agents write to the same context variable, the second value is silently overwritten – without warning, without a log. Such race conditions can only be identified in production through structured traces. Explicit state passing in ADK or LangGraph prevents this class of errors by design.

Observability: Why Logs Aren't Enough

The third pillar of operational management is observability. Standard logs fundamentally fail with AI agents. A "strange response" report can't be reproduced because the prompt was shaped by dynamic context retrieval, tool calls delivered different results, and the model chose a different reasoning path. Without structured traces that capture the exact prompt, tool parameters, tool responses, and LLM output at the time of inference, there's no reproducible path to the root cause.

The numbers are sobering: according to LangChain's State of Agent Engineering Survey 2026, 72 percent of companies run multi-agent systems in production, but only 37 percent operate online evaluations. The LLM observability market grew to $1.97 billion in 2025 and will rise to $6.8 billion by 2029 with a 36.5 percent CAGR.

The Three Pillars of Agent Observability

LangFuse v4 (March 2026) introduced an observations-first data model: every LLM call, tool execution, and agent step is a direct table entry queryable without hierarchy traversal. Chart load times dropped drastically, and LLM-as-a-judge evaluations at the observation level are now completed in seconds.

Arize Phoenix is built from the ground up on OpenTelemetry and the OpenInference format. The advantage: tracing data remains portable and can be forwarded to any OTel-compatible backend (Jaeger, Tempo, Datadog). Teams that have already invested in OTel infrastructure no longer run parallel observability stacks.

OpenTelemetry GenAI Conventions (still experimental, as of October 2026) define standardized span operations for agents: create_agent and invoke_agent. The market is converging here, even though the specification isn't stable yet. Those who adopt OTel now are prepared for the upcoming standard.

Conclusion

Operating AI agents in production is a different discipline than prototyping them. Those who make the leap from demo to production system must master three disciplines: cost management through intelligent routing and caching, robust state management with checkpointing and explicit state passing, and observability with structured traces instead of pure logs.

The tools for this are available and production-ready in 2026. LangGraph delivers the most mature checkpointing, ADK the most explicit state model, and platforms like LangFuse and Arize Phoenix close the transparency gap. The difference between an expensive prototype and a profitable AI platform isn't a framework comparison – it's the operational excellence behind it.

Sources

Agent Token Cost Optimization in 2026: Cut AI Inference Spend by 60-80% — AgentMarketCap (April 2026)
Agent Observability in 2026: LangFuse, Arize Phoenix, and OpenTelemetry — AgentMarketCap (April 2026)
AI Cost Optimization Strategies for 2026: A Practical Guide for Enterprise Teams — TrueFoundry (October 2026)
LangGraph vs CrewAI vs AutoGen 2026: Full comparison — N-iX (September 2026)
Multi-Agent Orchestration 2026: LangGraph, CrewAI, and AutoGen Compared — saaro.net (September 2026)

Obsidian and AI 2026: From Second Brain to Knowledge Base for AI Agents

saaro — Fri, 30 Oct 2026 12:00:11 +0000

Obsidian has been the first choice for years for anyone wanting to organize their thoughts, notes, and research in a local, Markdown-based system. But in 2026, the tool’s role has fundamentally changed: Obsidian is no longer just a “second brain” for humans – it is increasingly becoming a personal knowledge base for AI agents. The key to this is MCP (Model Context Protocol), and the community has created an impressive infrastructure in recent months that directly connects Obsidian vaults with Claude Code, Cursor, and other AI tools.

The starting point: Obsidian without built-in AI

Anyone looking for an AI feature in Obsidian in fall 2026 won’t find one in the tool itself. Obsidian Desktop is at version 1.14.4 (October 2026) and contains no built-in AI assistant. This is not an oversight but a deliberate design decision: Obsidian relies on a lean core and leaves AI integration to the community. The only official AI feature is the interpreter in the Obsidian Web Clipper, which can summarize, translate, or extract web pages when clipping – with its own API key for services like Anthropic, Google Gemini, or OpenAI.

All AI capability comes from the plugin ecosystem, which now includes over 1,500 community plugins. And this is exactly where decisive changes happened in 2026.

The plugin ecosystem: Copilot, Smart Connections, and Co.

Four plugins dominate AI integration in Obsidian:

Copilot for Obsidian (version 4.0.14, October 2026) is by far the most feature-rich plugin. It won the “Best LLM Integration” award at the Obsidian Gems of the Year in 2024. Version 4 introduces an agent mode that supports multiple LLMs – from OpenAI and Anthropic to local models and the coding agents Claude Code and Codex. Particularly practical: API keys are stored encrypted in the system’s native keychain, not in the vault itself.

Smart Connections is the easiest entry point: it comes with a built-in local embedding model and requires no API key. It displays related notes in a sidebar while you write – semantic search without cloud dependency.

AI Providers and Local GPT (by the same developer) form a duo for complete local operation. AI Providers stores the model configuration centrally; Local GPT adds context menu actions: continue text, summarize, check spelling – all via a local Ollama instance.

MCP: The game changer 2026

However, the most significant development in 2026 is MCP integration. The Model Context Protocol (MCP) is an open standard that allows AI assistants to access external data sources. Obsidian vaults can be connected as a structured API via this protocol – and the community has produced five competing implementations.

The range extends from REST API-based servers (such as mcp-obsidian with over 3,000 GitHub stars), which require the Obsidian Local REST API plugin, to pure file system servers (such as obsidian-mcp or MCPVault), which access the Markdown files directly – without needing Obsidian to be open.

The practical benefit is enormous: an AI agent can read architecture decisions from the vault while reviewing code. It can search meeting notes for last week’s API specification. Or it updates project documentation after making code changes. Particularly valuable is backlink traversal: the agent can follow Wikilinks ([[Note name]]) and traverse the local knowledge graph – “find all notes connected to [[Project-Alpha]] within two hops” becomes a real query.

Local AI with Ollama: Privacy as a feature

A decisive advantage of Obsidian in the AI era is data sovereignty. Unlike cloud-based note apps, Obsidian notes do not leave your computer – as long as you don’t connect cloud LLMs. With Ollama and local models (such as Gemma 2), you can build a completely private AI workflow: the vault stays local, the model runs locally, and no prompt leaves the device.

The combination of AI Providers, Local GPT, and Ollama is the easiest path to this. If you also want semantic search without cloud connectivity, add Smart Connections with its built-in local embedding model. The result is an AI-powered knowledge base that is fully under your own control.

Conclusion

In 2026, Obsidian has undergone a remarkable transformation: from a passive note-taking tool to an active knowledge platform for AI agents. The lack of a built-in AI assistant is not a gap but a strength – it forces a conscious choice of the right plugins and architecture. Whether you connect cloud LLMs via Copilot, set up an MCP server for Claude Code, or run a completely local stack with Ollama: Obsidian offers the right solution for every requirement. The crucial question remains how much data sovereignty you are willing to give up. If you want to keep it, you are on the safe side with Obsidian and a local setup.

Sources

Crossplane 2026: Kubernetes-Native Infrastructure as Code as a Platform Engine

saaro — Wed, 28 Oct 2026 12:00:47 +0000

When platform teams talk about Internal Developer Platforms (IDPs) today, names like Backstage, Argo CD, and Kubernetes often come up. One project frequently remains in the background, even though it forms the crucial layer beneath them: Crossplane. The CNCF-graduated framework turns cloud infrastructure into Kubernetes resources – and in 2026, with a series of releases and the new Modelplane project, it has definitively established itself as a central platform engine.

From Control Plane Framework to Platform Engine

Crossplane was open-sourced by Upbound in 2018 and joined the CNCF in 2020. What began as a specialized tool for "control planes" is now a graduated CNCF project – on the same governance level as Kubernetes, Prometheus, and Helm. More than 3,000 contributors from over 450 organizations have contributed over 100 releases to date, with cumulative downloads exceeding 100 million. Productive users include Nike, SAP, IBM, Elastic, Autodesk, and the NASA Science Cloud.

The decisive paradigm shift: Crossplane does not model cloud resources as external state files (like Terraform or OpenTofu), but as native Kubernetes objects. An S3 bucket, a PostgreSQL database, or an IAM role becomes a YAML manifest that kubectl apply expects and that Argo CD or Flux synchronize via GitOps. Kubernetes etcd handles state management and locking – no separate state backend configuration, no manual terraform plan.

Crossplane v2: The Architectural Leap

Crossplane v2.0 was released in August 2025 – the biggest architectural change since the project's inception. Three changes were particularly consequential:

Namespaced Resources: Managed Resources live in namespaces by default and thus inherit Kubernetes RBAC. Multi-tenancy becomes an inherent property, not an option configured later.

Composition Functions: Instead of static patch-and-transform templates, platform teams can now write Composition Functions in Go, Python, or CEL since v2.0. These functions compute the desired infrastructure state at runtime – with loops, conditions, and logic that pure YAML templates could never represent.

Every Kubernetes resource is composable: A single Composition can not only create cloud resources like RDS instances, but also create Deployments, Services, ConfigMaps, and monitoring rules in the same cluster at the same time. What previously required multiple tools and pipelines becomes a single declarative object.

Since then, Crossplane has continued to grow at a three-month cadence: v2.2 (March

Kubernetes Gateway API 2026: The Ingress successor matures

saaro — Tue, 27 Oct 2026 12:01:11 +0000

The Kubernetes Gateway API has definitively established itself in 2026 as the successor to the traditional Ingress resource type. With version 1.6, the GA release by AWS for the Load Balancer Controller, and a growing number of conformant implementations, the time for migrating from Ingress has come. The switch promises not only better type safety and clearer role separation, but above all portable routing definitions that work equally well between cloud providers and self-managed clusters.

TCPRoute and UDPRoute in the Standard Channel

With Gateway API v1.6.0 (Released on June 30, 2026), TCPRoute and UDPRoute have made the jump to the Standard Channel and are now available as stable v1 API resources. This closes one of the biggest gaps in the previous Ingress model: while HTTP and TLS traffic could already be mapped portably since v1.0, raw TCP/UDP workloads like databases, DNS, VoIP, or IoT telemetry had to resort to implementation-specific CRDs or classic Kubernetes Services. With TCPRoute and UDPRoute, all protocols can now be defined via uniform, portable resources.

The migration is simple: Instead of the previous v1alpha2 version, v1 resources are now used in the API group path gateway.networking.k8s.io/v1. The old v1alpha2 versions are deprecated with v1.6 and will be removed in a future release. If you are already using TCPRoute or UDPRoute, you should update the manifests promptly.

AWS Adopts Gateway API

A strong signal for industry maturity came in March 2026: AWS switched Gateway API support in the AWS Load Balancer Controller to GA. The controller now manages both Application Load Balancers (L7) and Network Load Balancers (L4) via the standardized Gateway API resources – and no longer via Ingress annotations. A typo in an annotation like alb.ingress.kubernetes.io/condtions.api-v2 (with a swapped 'i') previously led silently to a non-routing configuration because the entry was simply ignored. The new CRDs like TargetGroupConfiguration, LoadBalancerConfiguration, and ListenerRuleConfiguration catch such errors already upon application, as the API server validates the structure.

The strategic importance of AWS support can hardly be overstated: The largest public cloud provider is signaling that the standardized, portable API approach is the future – not the proprietary annotation path that AWS itself shaped for years. The AWS Load Balancer Controller in version 3.0.0 can also automatically obtain and rotate TLS certificates from the AWS Certificate Manager, without requiring manual ARN updates.

Experimental Extensions in v1.6: XBackend and API Group Separation

Alongside the graduation of TCPRoute and UDPRoute, v1.6 brings two important innovations in the Experimental Channel. With the introduction of a separate API group gateway.networking.x-k8s.io, experimental functionality will in future be distinguished from stable functionality already at the group level – a clear signal of the project's growing maturity.

The new XBackend resource enables external targets as a backend for the first time, e.g., for egress scenarios to cloud AI APIs. With this, the SIG Network is responding to the growing need to route AI workloads with the Gateway API – a topic that was intensively discussed at KubeCon EU 2026.

The resource model of the Gateway API is based on three clearly separated levels:

GatewayClass defines the provider (Envoy, AWS, Cilium, Istio, NGINX) – managed by the platform team.
Gateway configures the load balancer with ports, protocols, and TLS – managed by the infrastructure team.
Routes (HTTPRoute, GRPCRoute, TCPRoute, UDPRoute, TLSRoute) define the application routing rules – managed by the developer teams.

This role separation is the decisive advancement over Ingress, where cluster admin rights were usually necessary for routing changes. Since Gateway API v1.5 (February 2026), Gateway Merging is also stable: Multiple gateways can merge onto shared listeners. The platform team owns the listener with port and TLS certificate, while tenant teams attach their routes to it in their own namespaces – without overwriting each other.

What Migrating from Ingress Brings

The advantages of switching to the Gateway API are tangible:

Type Safety: CRDs with schema validation instead of strings in annotations
Header-based Routing: Natively in the Standard, without provider-specific hacks
Traffic Splitting: Canary Deployments as a first-class feature of HTTPRoute
gRPC Routing: GRPCRoute firmly in the Standard Channel since v1.5
Multi-Protocol Support: TCP, UDP, TLS, HTTP, and gRPC from one model
Portability: The identical HTTPRoute works on AWS, Envoy Gateway, Cilium, and Istio – only the GatewayClass changes

The migration itself can be done incrementally: Existing Ingress resources remain functional while HTTPRoutes are tested in parallel. The AWS Load Balancer Controller supports both APIs simultaneously, so teams are not under migration pressure. For new deployments, however, starting directly with the Gateway API is recommended.

Conclusion

The Gateway API has achieved all major milestones in 2026: TCPRoute and UDPRoute are stable, the largest cloud provider supports the model in production, and the implementation landscape includes over 20 conformant controllers. Anyone setting up a new Kubernetes cluster today should rely on the Gateway API from the start – using Ingress in new code is a technical legacy. Those who migrate benefit from better validation, clearer role separation, and portable routing definitions that prevent a provider switch from becoming a project.

Sources

Kubernetes Blog: Gateway API v1.6 – TCPRoute and UDPRoute Graduate to Standard — Official Kubernetes release about v1.6.0 (August 2026)
InfoQ: AWS Load Balancer Controller Reaches GA with Kubernetes Gateway API Support — Analysis of the AWS GA announcement (March 2026)
AWS Blog: AWS Load Balancer Controller Adds GA Support for Kubernetes Gateway API — Official AWS announcement (March 2026)
Luca Berton: Kubernetes Gateway API – Ingress Replacement 2026 Guide — Practical guide with YAML examples and provider comparison (2026)

RAG Observability 2026: How to Monitor Retrieval Pipelines in Production

saaro — Tue, 27 Oct 2026 12:00:34 +0000

RAG systems promise the best of both worlds: the timeliness of a knowledge base combined with the linguistic fluency of an LLM. But anyone operating a RAG system in production quickly realizes: the quality of the answers is not a constant. Embedding models drift, vector databases shift, chunk boundaries change due to content updates – and the HTTP status still remains at 200. This is exactly where RAG observability comes in, an area that has finally gone from nice-to-have to must-have in 2026.

The "Silent Failure" Problem

A RAG system can deliver a wrong or misleading answer with 200 OK and 500 milliseconds of latency – and classic API monitoring notices nothing. A recent analysis by the observability provider Ailog shows: Traditional monitoring only detects two out of eight relevant problem categories. Hallucinations, irrelevant retrieval results, embedding drift, or a desynchronized vector database remain invisible as long as the API responds.

The conclusion is sobering: Anyone who monitors a RAG system only via HTTP status, latency, and token consumption is flying blind. A glance at the dashboard can show green while users quietly drift away because the answers become unusable.

The Four Pillars of RAG Observability

In practice, a four-layer model for RAG monitoring has become established, which StackPulsar describes precisely in its guide "Measuring What Matters in Production":

1. Query Embedding Layer: Even the embedding of the user query can fail or drift. A sudden shift in the embedding vectors indicates changed query behavior or degradation of the embedding model.

2. Retrieval Layer: This is where most RAG errors occur. Metrics like Precision@K and Recall@K measure whether the correct documents were found. Since recall over the entire corpus is computationally intensive, a golden evaluation dataset with 200 to 500 hand-labeled queries is recommended, which is run regularly (nightly or with every deployment).

3. Context Utilization Layer: Finding relevant chunks is not enough – the LLM must actually use them. Context utilization efficiency indicates what percentage of the passed context is actually referenced in the answer. As a rule of thumb, 40 to 60 percent is considered healthy; lower values indicate noisy retrieval or incorrectly sized chunks.

4. Answer Layer: The metrics Faithfulness (Are all statements supported by the context?) and Answer Relevance (Does the answer address the question?) complete the picture. Frameworks like RAGAS, DeepEval, and TruLens calculate these automatically via LLM-as-a-Judge.

Tool Landscape 2026: From RAGAS to FutureAGI

The market for RAG evaluation has become highly differentiated in 2026. Atlan's comparison from 2026 summarizes the core metrics, while benchmarks like those from FutureAGI and Python Data Bench compare the tools in detail:

RAGAS remains the library of choice for dataset-based offline evaluation. It offers the established metrics Faithfulness, Answer Relevancy, and Context Precision and is open source (Apache 2.0).
DeepEval integrates evaluation seamlessly into Pytest – an approach that is particularly attractive for teams with CI/CD discipline. With over 40 built-in metrics, it also covers safety and agent scenarios.
Langfuse (MIT-licensed) combines open-source tracing with prompt management and is suitable for teams looking for a self-hosted observability solution.
Arize Phoenix relies on OpenTelemetry and OpenInference as the tracing standard – ideal for organizations that already have OpenTelemetry in their stack.
Galileo targets enterprise teams with a focus on risk and compliance, offering research-backed metrics and on-premise operation.
FutureAGI combines evaluation, observability, simulation, and guardrails in a single runtime environment as a platform.

Current cost comparisons show: For 1,000 evaluation runs with gpt-4o-mini as the judge model, around $0.30 to $0.40 USD are due. Using local models like Qwen3-8B can reduce costs by a factor of 5 to 10.

From Ad-hoc Check to Continuous Evaluation Operation

The most important paradigm shift in 2026: RAG evaluation is not a one-time quality gate before deployment. It must be embedded in operations as a continuous process. Every user correction, every rephrased question, and every escalation to a human is a signal that the retrieval or generation needs to be adjusted.

Modern RAG observability platforms offer feedback loops precisely for this: production errors are automatically turned into test cases in the evaluation dataset. This creates a self-improving system that detects quality degradation before users notice it.

Conclusion

RAG observability is not a luxury in 2026, but an operational necessity. Anyone seriously operating a RAG system in production must look beyond latency and token consumption and systematically monitor the four layers: query, retrieval, context, and generation. The tools for this are mature, open source, and affordable – the lack of monitoring is no longer a question of technology, but of prioritization. The teams that invest in RAG observability now will be the ones whose AI systems still deliver reliable answers tomorrow.

Sources

MCP Security 2026: Critical Vulnerabilities, Malicious Servers, and the New Attack Surface for AI Agents

saaro — Mon, 26 Oct 2026 12:00:58 +0000

The Model Context Protocol (MCP) has established itself as the standard plug-in for AI agents in 2026: once connected, Large Language Models can access internal APIs, databases, and workflows via MCP. However, while adoption in production is rising rapidly, autumn 2026 reveals a first serious security crisis. From the CISA-KEV list to critical privilege escalations to malicious servers that pass all common scanners – the headlines are piling up.

LiteLLM and the CISA-KEV List: First MCP Vulnerability on the Government Blacklist

In September 2026, the US agency CISA set a milestone: for the first time, an MCP-specific vulnerability landed on the Known Exploited Vulnerabilities (KEV) list. CVE-2026-59822 affects LiteLLM, a widely used open-source proxy that serves as a gateway between applications and LLM providers. The MCP Streamable HTTP endpoint accepted any arbitrary Bearer token as valid authentication – an improper authentication flaw (CVSS 8.8) that allowed attackers to establish full-fledged MCP sessions without valid credentials (Tech Insider).

The consequences are severe: a hijacked MCP access grants not only access to a single service but to all tools, databases, and workflows that the LiteLLM proxy accesses on behalf of AI agents. CISA set a deadline of three to seven days for federal agencies – an indicator of the urgency. This was not the first LiteLLM incident: already in June 2026, CVE-2026-42271 had appeared on the KEV list, a command execution via MCP stdio test endpoints (WZ-IT).

Vulnerabilities in the Protocol Ecosystem: From Session Spoofing to Privilege Escalation

But LiteLLM is only the tip of the iceberg. In October 2026, security research firm Adversa AI recorded 15 new MCP security resources in one month. Three examples illustrate the range:

Session Spoofing in the Grafana MCP Server: The Grafana MCP server validated session IDs only by format, not by issuer. Unauthorized callers could present any session ID and access Grafana instances using the server's service account credentials (CVE-2026-19516). Additionally, a manipulable X-Grafana-URL header allowed full SSRF access (Adversa AI).

Privilege Escalation in LiteLLM (CVSS 9.9): Already at the end of September 2026, BerriAI published an advisory on GHSA-7hp6-4w63-5g45. The same LITELLM_SALT_KEY was used both for encrypting secrets and for creating session tokens. An authenticated user with the internal_user role could inject a manipulated admin key, which the proxy unknowingly signed – thereby gaining full proxy_admin control, including command execution on the host via the MCP stdio endpoint. GitHub rated the vulnerability as CVSS 9.9 (critical). A follow-up advisory (GHSA-g5ff-637f-6q2m, CVSS 8.1) showed that even the internal_user_viewer role could read arbitrary files of the proxy process (WZ-IT).

OAuth Fallback Without Issuer Validation: An attacker could configure an MCP server to return a 404 status on client requests. This caused the MCP Python SDK (versions before 1.30.0/2.2.0) to fall back to a legacy OAuth path that completely skipped issuer validation. Client secrets, authorization codes, and PKCE verifiers ended up on the attacker's server (Adversa AI).

Blind Spots in Security Scanners: Two Malicious MCP Servers, Five Clean Scans

An alarming research result from Pluto Security (September 2026) shows how inadequate the current MCP scanning landscape is. The researchers built two functional malicious MCP servers: a credential thief that extracted SSH, AWS, and Kubernetes credentials and sent them to a collection server, and a remote code execution server that fetched and executed commands from a second server. Both disguised themselves as harmless weather tools.

Five public scanners were tested: Cisco mcp-scanner, Snyk agent-scan, Ventrova sentinel-scan-cli, mcp-shield, and NVIDIA SkillSpector. The result: All five scanners returned clean results ("SAFE", "0 findings", "Verified") for at least one of the malicious servers. The reason: The scanners primarily checked tool metadata (names, descriptions, schemas), but not the actual handler code. Only SkillSpector's LLM source code review detected both servers – but only at normal file size. Once the malicious code was hidden behind 16 MiB of harmless text, the LLM review skipped the file, and the scan reported "CAUTION / 0" with incomplete coverage (Pluto Security).

Conclusion

Autumn 2026 marks a turning point for MCP security. The protocol has left the experimental stage behind – and with it its naive security assumptions. The findings are sobering: authentication is often bypassable, privileges are insufficiently separated, and existing scanning tools do not reliably detect malicious MCP servers. The MCP community is reacting: CoSAI released version 2.0 of its MCP security model, and initiatives like the "Five Gates" before MCP tool production – identity, scope, limits, approval, and evidence – offer practical guidance. Anyone running MCP servers in production should apply patches immediately, not expose MCP endpoints unprotected to the network, and treat scans for what they are: a snapshot, not a free pass.

Sources

Adversa AI: Top MCP Security Resources October 2026 — Monthly overview of MCP security incidents and research
Tech Insider: CISA Flags First MCP Flaw: LiteLLM Hits CVSS 8.8 — Analysis of the first MCP-specific CISA-KEV entry
WZ-IT: LiteLLM Vulnerability GHSA-7hp6-4w63-5g45 — Detailed analysis of the privilege escalation and patch management
Pluto Security: Malicious Servers, Clean Scans — Research on MCP scanner gaps and malicious servers
Jannik Reinhard: Five Gates Before an MCP Tool Reaches Production — Production checklist for MCP tools

Backstage 2026: From Developer Portal to Context Infrastructure for AI Agents

saaro — Mon, 26 Oct 2026 12:00:22 +0000

Internal Developer Portals were long considered the heart of Platform Engineering – a unified interface for service catalogs, documentation, and deployment workflows. But a fundamental shift is emerging in 2026: Backstage, the dominant open-source framework with an 89 percent market share, is evolving from a passive portal into an active Context Infrastructure for AI-powered development teams. The latest releases v1.51.0 and v1.52.0 mark this shift with concrete, production-ready features.

AiResource and MCP Server: Backstage Speaks AI

By far the most important milestone is Backstage v1.51.0, released in May 2026. It introduces a new entity type AiResource – a structured category in the Software Catalog that represents AI tools, governance rules, and agent configurations as first-class citizens. Developers can define skill and rule subtypes that carry fields like disciplines, categories, agents, and dependsOn as native catalog properties.

In parallel, the MCP Server (Model Context Protocol) is registered as a first-class API subtype. Instead of a simple string definition, an MCP server entry now carries a spec.remotes list – a machine-readable reference that AI agents can query directly, instead of having to configure API endpoints manually. This turns Backstage into a registry not only for human developers, but also for software-driven agents.

v1.52.0: The Connections Service as Central Credential Management

June 2026 saw the release of Backstage v1.52.0 with the experimental @backstage/connections service (BEP-0014). It introduces a central system through which plugins can declare and retrieve credentials for external services – including runtime enforcement that blocks access to undeclared connection types.

This sounds unremarkable, but it is fundamental for AI integrations: An AI agent that needs to obtain a deployment approval or provision a service via Backstage requires reliable, auditable credentials. The Connections Service delivers exactly that – replacing the previous ad-hoc management of tokens and secrets via environment variables.

The AI Integration Drivers: Why Now?

Three developments are driving the transformation of Backstage into an AI Context Infrastructure:

First, in Q1 2026, the CNCF published its Platform Engineering Technology Radar and placed Backstage at the highest maturity level Adopt – alongside Helm and kro. The signal to platform teams is clear: Backstage is no longer an experimental playground, but a production-proven standard.

Second, a recent analysis by Roadie showed that AI-powered teams systematically build duplicate services. The reason: AI coding agents lack structured, queryable access to the existing service catalog. An agent asked to build a new notification service will do so – without knowing that three already exist. Backstage closes exactly this gap by exposing the catalog as a machine-readable knowledge base.

Third, the ecosystem is growing: The community counts over 200 open-source plugins, more than 3,000 companies use Backstage in production – including Netflix, American Airlines, and IKEA. In July 2026, Giant Swarm already documents Backstage v0.153.x as the basis for their managed Developer Portal. The maturity level now allows integrations that would have been experimental a year ago.

Context Engineering as a New Discipline

Behind the technical change lies a strategic one: Platform teams are recognizing that the Software Catalog must be optimized not only for humans, but also for machines. Roadie coins the term Context Engineering for this – the discipline of providing structured, up-to-date, and queryable context data that AI agents can use for decision-making.

An example: A developer asks their AI assistant to create a new microservice for payment processing. The assistant queries the Backstage catalog, finds an existing payment service with available capacity, and suggests reusing it – instead of generating another duplicate service. This requires that the catalog not only exists, but is also current, complete, and agent-readable.

Conclusion

In 2026, Backstage has made the leap from Developer Portal to AI Context Infrastructure. With AiResource, MCP server support, and the Connections Service, the latest releases provide the technical foundations to make the Software Catalog the central knowledge base for AI agents. The CNCF classification as "Adopt" and the growing number of production installations underscore its maturity.

For platform teams, this means: Those running Backstage today should evaluate the new AI entity types and design the catalog as an agent-readable interface. Those who do not yet have a Developer Portal will get not just a portal with Backstage 2026, but the infrastructural basis for the next wave of software development – where humans and AI agents work side by side.

Sources

Agentic RAG 2026: When AI Agents Decide How to Find Knowledge

saaro — Sat, 24 Oct 2026 12:00:56 +0000

Classical RAG pipelines (Retrieval-Augmented Generation) have become the standard over the past two years for supplying large language models with up-to-date domain knowledge. But the simple “Embed → Retrieve → Generate” approach quickly reaches its limits with multi-step questions, ambiguous queries, or heterogeneous data sources. By 2026, Agentic RAG has established itself as an architectural pattern that replaces this linear process with an agentic control loop—taking the quality and reliability of knowledge-based AI applications to a new level.

What distinguishes Agentic RAG from classical RAG

A classical RAG pipeline works deterministically: the user query is embedded, the top-k most similar chunks are retrieved from a vector database, inserted into the prompt, and the LLM generates an answer. This one-time pass is fast and cheap, but blind—the retriever does not know whether the chunks found are actually relevant, and the LLM cannot ask follow-up questions or change strategy.

Agentic RAG wraps this process in an agent control loop: an LLM-based orchestrator agent independently decides how and where to search for information. It can break the query into sub-questions, call multiple retrieval tools sequentially (vector database, SQL database, web search, MCP servers), evaluate the results, and start a new round if quality is insufficient. The decisive difference: the system transitions from a pipeline model to a state machine model.

An example from practice: an internal knowledge assistant is asked: “What is our return policy for business customers in Germany, and has it changed since last quarter?” A classical RAG pipeline would embed the entire query and hope that the top chunks cover both aspects. An agentic RAG system, by contrast, identifies two sub-questions, searches for the right source for each (one for the current policy, one for the version history), evaluates the results, and then creates a synthetic answer with citations.

The five components of an agentic RAG architecture

A production-ready Agentic RAG architecture consists of five interlocking components:

1. The orchestrator agent: The LLM-powered brain of the system. It receives the query, plans the next steps, calls tools, and decides whether the gathered information is sufficient for an answer. Frameworks such as LangGraph or LlamaIndex Agents provide the necessary infrastructure for this.

2. Retrieval tools: Unlike in classical RAG, the agent has a toolbox with multiple access methods—vector search for semantic similarity, SQL queries for structured data, web search for current information, and MCP servers for specialized sources. Each tool is described via a schema that the agent can interpret automatically.

3. Query planning and rewriting: The agent breaks complex queries into sub-questions, expands abbreviations, or generates alternative formulations. For ambiguous queries, it creates a mini-plan: “First find the policy document, then search for exceptions, then summarize both.”

4. Self-correction and reflection: After each retrieval step, the agent evaluates the results for relevance and completeness. If the chunks are insufficient, the query is reformulated and searched again. This pattern—known as Corrective RAG or Self-RAG—is the biggest quality lever compared with classical pipelines.

5. The generator: Once enough validated information has been collected, the agent produces the final answer with citations. Because the retrieval process was targeted and self-correcting, the prompt contains less noise, and hallucinations measurably decrease.

Agentic RAG in practice: frameworks and patterns

The technology stack for Agentic RAG consolidated in 2026. LangGraph (from LangChain) has established itself as the standard for state machine orchestration and is replacing older chain-based approaches. LlamaIndex remains the first choice for the retrieval layer and data indexing. For evaluation, specialized platforms such as Ragas (for faithfulness and answer relevancy), Arize Phoenix, and Langfuse are increasingly used.

The costs should not be underestimated: Agentic RAG typically consumes 3–10 times more tokens and has 2–5 times higher latency than one-shot RAG. However, the added value justifies these costs for multi-step questions, ambiguous queries, and safety-critical areas such as law, medicine, or finance. In June 2026, Microsoft published a reference architecture for Agentic RAG on Azure that concretely describes how to get started with LangGraph and LlamaIndex.

Production targets for Agentic RAG that have proven themselves in practice: Faithfulness ≥ 0.9, Answer Relevancy ≥ 0.85, Context Precision ≥ 0.8—measured with Ragas on a golden dataset.

Conclusion

Classical RAG was a breakthrough, but it treated the LLM as a passive consumer of the retrieval result. Agentic RAG reverses this relationship: the model actively decides what to retrieve, where, and how—and whether the results are good enough. By 2026, this pattern has arrived in the mainstream. Platforms such as LangGraph and LlamaIndex provide the tools, evaluation metrics are standardized, and reference architectures from major cloud providers lower the barrier to entry. Anyone who wants to run knowledge-based AI applications in production can no longer avoid agentic retrieval architectures.

Sources

FreeAcademy.ai: Agentic RAG Explained – How AI Agents Supercharge RAG in 2026 — Fundamental introduction to the architecture and components of Agentic RAG.
MarsDevs: Agentic RAG – The 2026 Production Guide (Patterns, Frameworks, and the Cost Reality) — Production-proven patterns, framework comparison, and cost analysis.
Microsoft Learn: Develop an Agentic RAG Solution on Azure — Official Azure reference architecture for agentic RAG (June 2026).
Bacancy Technology: What is Agentic RAG – Architecture, Use Cases & Guide in 2026 — Overview of architectural patterns and use cases.
arXiv: Agentic Retrieval-Augmented Generation – A Survey on Agentic RAG (2501.09136) — Scientific taxonomy of agentic RAG patterns and their classification.

LLM Wiki in Practice 2026: CI Pipelines, Health Checks, and Operating AI Knowledge Bases

saaro — Fri, 23 Oct 2026 12:00:43 +0000

Since Andrej Karpathy published his GitHub Gist on the "LLM Wiki" in April 2026, the pattern has spread rapidly. What started as an initial prompt draft has become a movement: thousands of developers and knowledge workers now operate AI-managed knowledge bases. But practice reveals where the pattern's limits lie – and how CI pipelines, health checks, and clearly defined operational concepts turn an experiment into reliable knowledge infrastructure.

From prompt experiment to daily practice

Anyone who copies Karpathy's Gist into their coding agent will have a working LLM wiki within 15 minutes. The directory structure raw/ → wiki/ + CLAUDE.md is set up quickly. The magic lies in the ingestion process: a new PDF or article lands in raw/, the agent analyzes the document, creates a summary in wiki/sources/, and updates cross-linked concept pages – all automatically.

However, after a few weeks of daily use, patterns emerge that the original Gist does not address. Kunal Ganglani, who used the pattern daily for three months, describes a clear scaling limit: From about 150 to 200 wiki pages onward, no current agent can keep the entire knowledge graph in its context window. The result is inconsistent updates and missed cross-references.

The 200-page limit and the index workaround

The solution is surprisingly simple: a master index (wiki/index.md) that lists every page with a one-line summary and is updated on every ingestion. The agent reads the index first, then selectively chooses the relevant pages for an operation. This technique extends practical capacity to over 300 pages. Anyone maintaining many page types and cross-disciplinary topics cannot avoid this workaround.

The choice of agent is also not a minor detail: While Claude Code and OpenAI Codex work reliably for most workflows, smaller models (under 32B parameters) hit their limits with complex multi-page updates. Qwen3 32B and Llama 3 70B are suitable for basic use cases; according to consistent community experience, more powerful models are needed for full wiki operations with many cross-references.

Health checks as a CI pipeline for knowledge

Karpathy's compiler analogy – raw/ is source code, the LLM is the compiler, wiki/ is the binary – can be thought through to its logical conclusion: If the wiki is the compiled artifact, it needs a test suite. The pattern's third core operation, lint, is exactly that.

A structured lint run checks:

Contradictions: Claims that collide between different sources or pages
Orphaned pages: Wiki pages without incoming links – they are cut off from the knowledge graph
Missing concepts: Terms mentioned across multiple pages but lacking a page of their own
Outdated statements: Claims superseded by newer sources

In practice, automating these checks as a cron job or GitHub Actions workflow has proven effective. A daily lint run, driven by a prompt such as "Check the wiki for contradictions and orphaned pages," keeps the knowledge base clean – without any human intervention. LMMarketCap aptly describes this quality layer as "CI/CD for your knowledge database."

Integration into daily development work

The LLM wiki pattern is not a replacement for RAG, but a complement for a specific use case: vertical synthesis. While RAG scales horizontally (searching through 100,000 support tickets), the LLM wiki excels at the in-depth processing of fewer but higher-quality sources. Synthesizing 50 research papers on one topic and uncovering contradictions in the process is a task for which RAG is naturally ill-suited – while the LLM wiki approach is optimized for it.

Integrating Obsidian as a frontend makes the wiki searchable, linkable, and versionable. Those who take the compiler model seriously also place the entire wiki in a Git repository. The lint process becomes a CI job; faulty "compilations" become visible before they pollute the knowledge base.

Conclusion

The LLM wiki has evolved from Karpathy's prompt sketch into a mature pattern for personal and team knowledge bases. Practice shows: the real art lies not in setting it up, but in operating it. Index-driven navigation beyond the 200-page limit, automated health checks as a CI pipeline, and the deliberate delineation from RAG are the levers that turn a promising experiment into reliable knowledge infrastructure. Those who master these operational concepts have a tool that becomes more valuable with every new source.

Sources

Kunal Ganglani – LLM Wiki Setup: Karpathy's Knowledge Base [2026 Guide] — Practical experience after 3 months of daily use, scaling limits, and agent comparison
Starmorph – How to Build Karpathy's LLM Wiki: The Complete Guide — Complete setup guide with Claude Code, Obsidian, and 3-layer architecture
LMMarketCap – LLM Knowledge Bases: Karpathy's Self-Improving Second Brain — 6-stage pipeline, context window math, and model recommendations
GitHub Gist – Karpathy's llm-wiki.md — Original prompt pattern (5,000+ stars, 5,000+ forks, April 2026)