DEV Community: Rost

RTX 5090 Price in Australia: Listings and History

Rost — Sun, 01 Nov 2026 23:51:35 +0000

On 29/10/2026 the cheapest in-stock RTX 5090s averaged A$12,249, and the RTX 5080s averaged A$2,832. A shop with no card in stock is left out of the average.

From the July 2025 check through the January 2026 check, the published figure is the average of the cheapest in-stock card at Centrecom, PC Case Gear, and Scorptec. On 29/10/2026 the RTX 5090 floors were Centrecom A$12,999 and Scorptec A$11,499, and PC Case Gear had none in stock. The RTX 5080 floors were Centrecom A$2,899, PC Case Gear A$2,899, and Scorptec A$2,699, and that average is rounded to the nearest dollar.

The January 2026 RTX 5080 average on this same measure was A$1,899. Memory capacity, bandwidth, and the Arc Pro B70 and Radeon AI Pro R9700 comparison stay on GPUs for AI in 2026: NVIDIA, AMD, Intel Compared.

Retailer listings

These rows are the 29/10/2026 check. A sold-out row is the price on the listing, not a card that could be added to a cart. Where several cards shared the cheapest in-stock price, each of those cards is a row.

Checked	Product	Price (AUD)	Metric	Retailer	Availability	Listing
29/10/2026	ASUS TUF Gaming GeForce RTX 5090 32GB	12,999	Cheapest in-stock at this shop	Centrecom	In stock	Centrecom
29/10/2026	Palit GeForce RTX 5090 GameRock, 32GB	11,499	Cheapest in-stock at this shop	Scorptec	In stock	Scorptec
29/10/2026	Palit GeForce RTX 5090 GameRock OC, 32GB	11,499	Named in-stock listing	Scorptec	In stock	Scorptec
29/10/2026	ASUS ProArt GeForce RTX 5090 32GB OC	12,999	Named in-stock listing	Scorptec	In stock	Scorptec
29/10/2026	ASUS GeForce RTX 5090 TUF Gaming 32GB	10,499	Listed price, sold out	PC Case Gear	Sold out	PC Case Gear
29/10/2026	MSI GeForce RTX 5090 Ventus 3X OC 32GB	10,499	Listed price, sold out	PC Case Gear	Sold out	PC Case Gear
29/10/2026	Palit GeForce RTX 5080 Infinity 3, 16GB	2,699	Cheapest in-stock at this shop	Scorptec	In stock	Scorptec
29/10/2026	Palit GeForce RTX 5080 Infinity 3, 16GB	2,899	Cheapest in-stock at this shop	Centrecom	In stock	Centrecom
29/10/2026	Palit GeForce RTX 5080 GamingPro OC V1, 16GB	2,699	Named in-stock listing	Scorptec	In stock	Scorptec
29/10/2026	ZOTAC GAMING GeForce RTX 5080 Solid Core OC, 16GB	2,699	Named in-stock listing	Scorptec	In stock	Scorptec
29/10/2026	Gigabyte GeForce RTX 5080 Windforce OC V2 16GB	2,899	Cheapest in-stock at this shop	PC Case Gear	In stock	PC Case Gear
29/10/2026	Palit GeForce RTX 5080 Infinity 3 16GB	2,899	Named in-stock listing	PC Case Gear	In stock	PC Case Gear

PC Case Gear's RTX 5090 category had eighteen listings, and every one of them was sold out. The two sold-out rows are the lowest of those prices. Scorptec's in-stock RTX 5090s start at the Palit GameRock rows, and the ASUS ProArt OC row is the highest in-stock price in this check.

Earlier checks

The June 2025 check is a shop minimum for the RTX 5090, and the July page carries the June RTX 5080 average forward. From July 2025 through January 2026 the figure is the three-shop average. October 2025 is the low of that RTX 5090 average, and November 2025 is the next row.

The March 2026 check uses an entry-to-premium range rather than that average, and it does not publish an RTX 5080 average. NVIDIA's Australian list prices from the June check are the first row of the table.

Period	RTX 5090 (AUD)	RTX 5080 (AUD)	Metric
Launch list	4,039	2,019	NVIDIA Australia list, from the June 2025 check
June 2025	5,299	2,009	5090: same minimum at five shops. 5080: average of cheapest in-stock, as carried into the July check
July 2025	4,532	1,985	Average of cheapest in-stock at Centrecom, PC Case Gear, and Scorptec
October 2025	4,282	1,782	Same three-shop average
November 2025	4,832	1,786	Same three-shop average
January 2026	5,566	1,899	Same three-shop average
March 2026	5,999–7,500	—	Entry-to-premium range for the 5090. No 5080 average in that check
29/10/2026	12,249	2,832	5090: average of the shops with stock (PC Case Gear had none). 5080: average of all three shops, nearest dollar

The January TL;DR rounds the three shop floors (Centrecom A$5,499, PC Case Gear A$5,799, Scorptec A$5,399) to A$5,566. The March range is a different measurement, so it is not a step on the average.

On the same day as the 29/10/2026 GPU listings, PC Case Gear listed an ASUS Ascent GX10 with 128 GB and 1 TB at A$9,649, with stock at the supplier rather than in the warehouse. That workstation check, and the January 2026 DGX Spark prices, are on DGX Spark Pricing in Australia.

Useful links

Comparing NVIDIA GPU suitability for AI

Decision LLMs: When a Model Should Decide — and When It Shouldn't

Rost — Mon, 26 Oct 2026 08:59:29 +0000

Ask practitioners what breaks in agent systems and the answer is rarely capability. The failure is quieter: the model decides, and nobody can tell sound from fluent.

A model output is a sample. A decision has properties a sample does not have — determinism for a fixed input, invariants it must satisfy, an audit trail that says why this and not the alternative. When those properties are missing, the "decision" is a high-confidence guess wearing the clothing of one.

This article gives that gap a mechanical treatment: a taxonomy of five decision roles, a rule for routing each decision to the smallest model competent for it, and the verification machinery that turns "the agent decided X" into something you can later prove was sound. It closes with a rubric you can run against any action in your own system.

"For local agents the gap I keep hitting is not capability, it is verification. The agent says it read a file, called a tool, or made a decision…" — r/LocalLLaMA, "Best Local Agents — Jun 2026"

"When an LLM agent fails, it's a black box and nobody knows why. The real bottleneck to adoption isn't getting a model from 92% to 96% accuracy — [it's knowing why it decided X]." — r/AI_Agents, "The gap between decision and execution"

The community has converged on three instincts: don't trust the model with the final call, route the decision to the smallest competent model, and make the decision explainable. What has been missing is the framework that makes the three actionable at once.

Why "it answers, but does it decide?"

A model output is a token distribution collapsed to one sequence, and the same prompt run twice can produce two sequences that diverge on exactly the token that matters. A decision, by contrast, is deterministic for a fixed input, satisfies explicit invariants, and leaves an audit trail recording which branch was taken and why.

When a model "decides" without machinery around it, none of those properties exist. There is no invariant the output is obligated to respect, no log of the branch taken, no way to replay the decision and check it held. The gap is not intelligence — it is that a probabilistic output has been placed in a slot designed for a deterministic one.

The practitioner instinct is to keep the model out of the decision entirely:

"The LLM never makes final decisions in the system. Its role is purely: logic refinement, stress testing, scenario generation, explanation." — r/AI_Agents, "Using an LLM as a 'reasoning agent' for validating price…"

That is the right posture stated as a maxim. To apply it, you need a way to classify which role the model should hold for a given action — which is what the taxonomy below provides.

Five decision roles

Every decision a system makes can be assigned to one of five roles. The role determines how much the model's output can be trusted, how it is audited, how it fails, and whether a human belongs in the loop.

Role	The model…	Trust level	Auditability	Typical failure mode	Human in the loop?
Decider	Commits the call	Lowest — must be constrained	Must emit a decision record	High-confidence wrong branch	Yes, for irreversible/high-stakes
Recommender	Produces options + a ranking	Low	Ranked list is inspectable	Confident wrong option	Optional; human picks
Evaluator (judge)	Scores an input against a rubric	Medium — rubric-bound	Score + rubric trace	Rubric drift; self-preference bias	No, but rubric is versioned
Generator	Produces the artifact	Not applicable — it is content	Content review	Off-spec, hallucinated specifics	Review of output
Router	Selects the path/model for the task	Medium — deterministic mapping	Routing log	Misrouted tier	No

Read the table as a constraint, not a menu. Most broken systems share one defect: a single model fills all five roles at once, so a role that demands determinism (decider) is served by a component optimized for fluency (generator). Where the model acts as generator, none of the decision machinery is needed — content review is. The failure to avoid is conflating these: "the agent decided to delete the environment" is usually a missing approval gate, not a decision-quality problem.

Route the decision to the smallest competent model

Once the role is fixed, the routing question is which model tier serves it. Practitioners report the payoff directly:

"Model routing: Haiku for classification/routing decisions, Sonnet for execution, Opus only for design/planning. Cost dropped ~60% once I stopped using Opus for everything." — r/AI_Agents, "What is your full AI Agent stack in 2026"

That number comes from treating model selection as a per-role decision instead of a per-system one. A routing call — "is this an email triage or a refund approval?" — is a classification, and classification does not need a frontier model; it needs a small, fast, low-temperature model, because it runs thousands of times. The tier mechanics — capability-based, cost-aware, and latency-aware selection, plus fallback chains — are covered in Model Routing: Stop Using One Model for Everything; the orchestration patterns around multiple models are in Multi-Model System Design.

The addition the decision framing contributes is confidence-based escalation, which makes "smallest competent" measured rather than assumed. Start on the smallest tier. If the model emits a calibrated confidence — or a structured self-assessment — and it clears a threshold, stop. If it does not, escalate one tier and record the escalation. Competence is then a property verified per call, not a property assumed per model.

One constraint stays absolute regardless of tier: the router role itself must not be a free language-model call. A deterministic spine — rules on task type, payload size, cost budget — selects the branch, and the model fills only the leaf decisions inside it. The spine is auditable and unit-testable; the model is contained to what it is actually good at.

Making the decision verifiable

This is where the "agent says it did X" gap closes. The fix is to make the decision a first-class structured artifact rather than a side effect of a chat turn:

A structured decision store. Every decision the model is allowed to make is written to a table, not a log line: the input, the role, the model and tier, the confidence, the alternatives considered, the timestamp. A decision is something you can SELECT later. If it lives only in a conversation transcript, it is a guess.
A decision log, separate from the execution log. The execution log says what happened; the decision log says what was decided and on what basis. The separation matters because the failure is usually in the decision, not the execution — the agent executed the deletion perfectly; the bug was deciding to delete. The mechanics of structured, append-only logging are the same ones used for compliance audit trails in LLM Guardrails in Practice.
Invariants and assertions. For decider and evaluator roles, the decision must satisfy invariants checked after the model emits it, in deterministic code. A price recommender can emit anything; the invariant "recommended price must be within [floor, ceiling]" is enforced by code. The model proposes; the invariant disposes.
Human-approval gates at the irreversibility boundary. The gate is a property of the action's blast radius, not of the model's confidence. A refund under a threshold is auto-approvable; one over it is not. A 99%-confident model still cannot approve an irreversible action it has no audit record for.

flowchart LR
    A["Action requires<br/>a decision"] --> B{"Classify<br/>the role"}
    B --> C["Route to smallest<br/>competent tier"]
    C --> D["Emit structured<br/>decision record"]
    D --> E{"Invariants<br/>hold?"}
    E -- "yes" --> F{"Irreversible or<br/>high-stakes?"}
    E -- "no" --> G["Reject, escalate,<br/>and log"]
    F -- "yes" --> H["Human approval gate"]
    F -- "no" --> I["Execute"]
    H -- "approved" --> I
    H -- "rejected" --> G

The through-line: the 92%→96% accuracy push is not the bottleneck — explainability is. A system at 92% with every decision logged, invariant-checked, and gated at the boundary ships safely, because the 8% of wrong decisions are findable before they do damage. A system at 99% with a black-box decision process does not, because the 1% is a live incident with no forensic record. Where the gate itself lives in an assistant — which component holds approval authority and how requests reach it — is laid out in AI Assistant Architecture: LLM, Memory, Tools, Routing, Observability.

Worked examples

Three systems, one role assignment each.

Price validation — the model as recommender, never decider. An agent reviews a supplier price change and produces a ranked list: accept, negotiate at X, reject. The final call is a decider slot filled by deterministic rules: below the cost floor, auto-accept; above the ceiling, route to a human; in between, apply the invariant "negotiated price must beat last accepted by Y%." The model never approves its own recommendation, and the recommendation sits in the decision store with its rubric trace if it goes wrong.

Agent task routing — the spine as decider, the model at the leaves. An inbound request (email, ticket, command) is assigned a path by payload shape and intent category — deterministic, unit-testable. A small model fills only the leaf decision inside the branch. A misrouting is a blast-radius event, which is exactly why the branch selection must be deterministic.

LLM-as-judge — the model as evaluator, the rubric as the constraint. One model scores another's output against a versioned rubric, and the score plus rubric trace are stored, not just the number. The known failure mode is self-preference bias — a model scoring its own generation style higher — and the mitigation is structural: version the rubric so a model is never silently scoring against criteria it shaped.

The honest boundary

There is a class of decision an LLM must not make, and it is not a question of scale: actions that are irreversible, high-stakes, and have no fallback — data deletion, fund movement, external commitments. Verification is impossible for an action you cannot roll back, which is why "autonomous agent" keeps hitting the same wall regardless of model quality.

The human-in-the-loop that is not theater sits at exactly that boundary and nowhere else. A human reviewing every low-stakes decision adds friction and no safety; a human absent from every high-stakes decision is the incident. The same principle appears in a narrower form in agent memory: self-reinforcing memory loops gate candidate memory writes behind explicit human approval before anything reaches durable state — the gate sized to what the write can break. For multi-agent systems, the corresponding question is who may act on whose behalf, and the identity, delegation, and audit-trail machinery for it is in A2A and MCP Agent Security.

The decision rubric

Run this against any action in your system:

Classify the role. Decider, recommender, evaluator, generator, or router? An unnamed role is a bug — a model in a decider slot wearing a generator's confidence.
Pick the tier for that role. Smallest competent, with confidence-based escalation so competence is measured per call.
Define the verification. Which invariants must hold? What goes in the decision store? What is the audit trace? A role with no verification path is a decider by default.
Set the human gate from blast radius, not confidence. Irreversible or high-stakes → gated. Otherwise → auto, with the decision logged so it is findable.

The same discipline, applied to engineering choices rather than runtime actions, is what decision records do — Decision Records for AI-Driven Development captures why a choice was made so it can be reviewed later; the rubric above does the same for the decisions a running system makes.

What's coming: the decider as a product

The framework predicts its own commoditization: once "route the decision to the smallest competent model" became a pattern, the next step was a model class that does nothing but decide. That step has now landed.

Through October 2026, several vendors shipped models built on a shared recipe — a frozen Qwen backbone, a rank-256 LoRA adapter, and a joint schema head — that emit typed decisions in a single forward pass: a choice from an enumerated set, a yes/no, or a score, with calibrated probabilities and no free-form text. There is no chain of thought to wait for, because there is no generation. Vendor-reported latency sits between roughly 8 and 210 ms across the class — from the smallest variant (d1-3B, about 8 ms) up to the largest (d3, about 209 ms) — and the footprints are small-GPU territory: d1-3B at about 1.67 GB, d3-flash at about 6.6 GB at Q4_K_M. Support also moved fast: llama.cpp merged native architecture support for several of these models within roughly two weeks of release.

Two consequences for the framework above. First, the routing rule gets a hardware expression it did not have before: the decider role can be served by a dedicated small model instead of a prompt mode of a general one, which makes the smallest-competent-tier decision partly automatic — the tier either exists for your decision type or it does not. Second, nothing about verification changes. A typed decision in 8 ms still needs the decision store, the invariants, and the gate at the irreversibility boundary; what the model class removes is generation cost and non-determinism, not the audit burden. Faster decisions multiply the number of decision records to audit — the bottleneck was never the model's speed, and it is not the model's intelligence either.

The other directions to watch are quieter. Tool-calling reliability for small models is approaching saturation — post-training on tool use is the cheapest high-impact step in the stack, which shifts attention to the rows above. Reliable planning horizons keep extending, which makes the decider role reachable for longer tasks and grows the verification burden with the horizon. Both feed the same conclusion: the scarce resource in agent systems is decisions you can later prove were sound, and everything else is catching up to that.

Qwen 4 Is Coming: What's Confirmed and What It Means for Local AI

Rost — Sat, 24 Oct 2026 22:58:09 +0000

Alibaba confirmed at Apsara that Qwen 4 is in training, no date attached, while community timing signals point at a launch in weeks rather than months.

For local AI, the consequential part is not the announcement but the 27B tier. Alibaba has shipped a capable dense model in that class in every recent generation, and the current one already runs on a single consumer GPU.

This article separates what Alibaba has actually confirmed from cadence inference, looks at what each 27B generation gained and what that implies for the next one, examines the architecture preview that can already be run, and works through the hardware and licensing picture for self-hosting. If your shortlist extends past this one family, the AI Models pages follow the rest of the landscape: architecture shifts, other model lines, and where the quantization tradeoffs sit.

What Alibaba has confirmed

On 22 September 2026, at the Apsara Conference in Hangzhou, Alibaba confirmed that Qwen 4 is in training. Qwen lead Liu Dayiheng described it as built on a "new-generation architecture" and arriving "very soon" — no date, no specifications, no benchmark results, and no licensing terms.

The widely quoted 5–10 trillion parameter figures refer to the longer-term Qwen 4.5 and Qwen 5 roadmap, not to confirmed Qwen 4 specifications. The current flagship, Qwen3.8-Max, is a ~2.4T-parameter MoE with roughly 95B active parameters and a 1M-token context window.

Two further items from the same announcement matter for the release itself:

Qwen3.8-Flash-Next (released late August 2026) is explicitly described in Alibaba's own materials as "a preview of the Qwen 4 architecture" — the one Qwen 4-adjacent model that can be run today. More on what it demonstrates below.
The full-stack roadmap includes the Zhenwu V900 AI accelerator from T-Head, a target of 20+ GW of Alibaba Cloud data-center capacity by 2032, and Qwen-Audio 3.1, Qwen3.8-LiveTranslate, and Qwen-Image 3.1 later in the year.

Everything beyond this — variant names, dates, benchmark scores — originates from community threads, tracker pages, or the prediction market, and is labelled accordingly in the rest of this article.

Release timing and cadence

Three timing windows have circulated, and none comes from Alibaba: a September window from a July "roadmap leak" (passed with no launch), an end-of-October claim on r/LocalLLaMA attributed to someone familiar with a launch partner, and an October-to-early-November cadence inference from tracker threads. The Manifold prediction market prices a launch before 1 November 2026 at roughly 74%. Read that as weeks, not a date.

The cadence of the current generation is the concrete precedent. When Qwen3.8 rolled out, the flagship Max launched on 3 August and the 27B open weights appeared on Hugging Face about ten days later. Flash-Next adds a new element: the architecture preview shipped before the flagship, which suggests Alibaba is decoupling "here is the architecture" from "here is the full model".

flowchart LR
    A["Qwen3.8-Flash-Next<br/>architecture preview<br/>(late Aug 2026)"] --> B["Flagship announcement<br/>(no confirmed date)"]
    B --> C["Open 27B-class weights<br/>~1-2 weeks after<br/>(cadence precedent)"]
    C --> D["Community quants<br/>days after weights"]

The practical consequence for local users: the keynote is not the moment weights become available. The Hugging Face drop is the signal to watch, and it has historically lagged the announcement.

How much better could Qwen 4 27B be?

Qwen3.8-27B is a dense vision-language model with strong repository-level coding, terminal automation, and instruction following, under Apache 2.0. Comparing the last three 27B generations on benchmarks where published scores are comparable gives a baseline for what the next one might do.

Benchmark	Qwen3.5-27B	Qwen3.6-27B	Qwen3.8-27B	Avg. gain	Qwen 4 forecast
Terminal-Bench 2.1	—	63.4	73.0	+9.6*	76–82
SWE-bench Pro	—	53.5	61.7	+8.2*	65–72
NL2Repo-Bench	—	36.2	42.3	+6.1*	45–52
CoWorkBench	—	61.0	70.7	+9.7*	74–80
IFBench	76.5	69.1	79.5	+1.5	81–86
GPQA Diamond	85.5	87.8	89.2	+1.9	90–93
LiveCodeBench v6	80.7	83.9	90.3	+4.8	92–95

Average gain is in percentage points per observed generation-to-generation transition. An asterisk marks benchmarks where only the Qwen3.6-to-Qwen3.8 transition is available; missing Qwen3.5 scores are not assumed to be zero. The Qwen 4 column is a forecast, not a leaked or measured result.

These scores are vendor-reported, and evaluation conditions and harnesses change between generations, so the progression is indicative rather than a controlled experiment. The pattern in it is consistent, though: gains concentrate in agentic coding and long-horizon professional work (Terminal-Bench, SWE-bench Pro, CoWorkBench), while scientific reasoning (GPQA) moves slowly as it approaches saturation.

A linear extrapolation would overstate the forecast. Benchmark gains get harder as scores rise, some evaluations are near saturation, and Qwen3.8's improvements came from specific training and post-training changes that need not repeat at the same magnitude. The forecast above assumes incremental progress on established reasoning benchmarks and larger gains on coding and agentic reliability — roughly 80 on Terminal-Bench 2.1 and 70 on SWE-bench Pro would already be a meaningful generational step. Worth watching alongside raw scores: whether those scores arrive with fewer reasoning tokens, since in local inference every extra token costs GPU time and memory bandwidth.

The quantization ladder it will land in

Whatever the next 27B's scores are, it will be deployed quantized. The current generation's ladder is the honest floor and ceiling for what a 27B-class model does at each bit level, drawn from cross-source evaluations:

Build	True bits/weight	Footprint	Aggregate score	Verdict
BF16 (FP16)	16	~54 GB	85.07 (reference)	The ceiling
Q4_K_XL	~5.2 (not 4)	17.6 GB	84.99 (99.9% of FP16)	Default choice
Ternary (Bonsai-class)	1.76	5.9 GB	84.78–86.2 (source-dependent)	Beats a plain Q2
1-bit QAT	1.125	3.9 GB	82.9%	Phone-class; ~4 points under Q4
Conventional IQ2_XXS	2.8	9.4 GB	72.6–72.7	The collapse to avoid

Two footnotes on that table. The advertised "4-bit" is really about 5.2 bits per weight once metadata and scaling factors are counted, which is why the Q4 row is nearly free relative to BF16. And the aggregate hides where the bits are actually spent: general knowledge, history, coding, and language hold at 97–100% of the FP16 score even at 1-bit, while arithmetic and multilingual tasks are what the bits buy — with the size of that category drop varying noticeably between evaluations.

This is consistent with the broader picture in The Efficient Frontier of Open Models in 2026: the 25–34B class is where open models deliver near-frontier capability per gigabyte, and a Qwen 4 27B that holds the Q4_K_XL position would inherit that slot.

Flash-Next: a working preview of the architecture

Flash-Next is the one piece of Qwen 4 evidence that can be measured today. Against Qwen3.8-27B, on Alibaba's own comparison:

Benchmark	Qwen3.8-27B	Flash-Next	Improvement
DeepSWE 1.1	42.2	58.7	+16.5
SWE-bench Pro	61.7	62.5	+0.8
NL2Repo-Bench	42.3	48.1	+5.8
CoWorkBench	70.7	73.9	+3.2
JobBench	33.4	55.7	+22.3
Toolathlon Verified	67.1	73.5	+6.4
IFBench	79.5	81.3	+1.8
GPQA Diamond	89.2	91.7	+2.5
LiveCodeBench v6	90.3	91.7	+1.6

The largest gains are again on agentic workloads — JobBench and DeepSWE — with modest movement on reasoning and competitive programming. One qualification matters when reading this as a Qwen 4 signal: Flash-Next is not a 27B dense model. It is a substantially larger MoE with ~125B main-model parameters, ~6B active, and ~51B additional n-gram embedding parameters, so its results cannot be attributed to architecture alone.

Three architectural changes in Flash-Next are the substance of the "new-generation architecture" description:

Qwen Sparse Attention (QSA), combined with Gated DeltaNet, which keeps a compact running representation of earlier context while retrieving detail from selected regions. Alibaba reports QSA kernel speedups of up to 7.6× in prefill and 4.9× in decoding at one-million-token context — kernel-level numbers, not end-to-end generation speed, but directly relevant to the KV-cache pressure that dominates long-context local inference (see KV-cache management for long context on 16 GB).
Gated Residual connections, which expand the residual stream into four dynamically controlled branches to improve information flow and training stability — invisible to users, but plausibly a contributor to long-horizon behavior.
N-gram embeddings, ~51B parameters accessed as lookup operations rather than dense matrix multiplies, which Alibaba suggests can be stored in system RAM and prefetched asynchronously. The parameters still occupy memory somewhere and real gains depend on memory bandwidth, but the design points at more capacity without proportionally more GPU compute.

How much of this reaches the smaller Qwen 4 models is unknown. For context on where these designs sit in the wider shift away from monolithic transformers, see What Comes After LLMs? — hybrid attention and explicit-capacity mechanisms are the same direction the post-transformer research landscape is moving in.

The model that doesn't exist

One item circulating in Qwen 4 rumor posts should be deleted from any notes it has reached: a "Qwen 4 Coder 32B-A3B" — a laptop-class coding model with a June 2026 release date, Apache 2.0 license, and an 82% SWE-bench Verified score. No such model was ever published through the Qwen blog, GitHub, or Hugging Face organization.

It even contaminated a benchmark aggregator: LLMCheck ingested the phantom model during a July 2026 data incident and removed it after an August reconciliation against official sources confirmed it does not exist. The 82% figure carries its own internal contradiction — it would have beaten the actual open-weight record at the time (80.2%, held by a 276B-parameter MoE that requires a Mac Studio-class machine), from a claimed 32B laptop model.

The real shipping coding model in this family is Qwen3-Coder-Next: an MoE with ~80B total and ~3B active parameters, Apache 2.0, at roughly 70.6% SWE-bench Verified. The numbering gap here is the likely origin of the hoax — the version sequence jumped 3.5 → 3.6 → 3.8, and a spec was invented to fill the gap before "4".

Running Qwen 4 locally

Assuming a 27B-class dense model with a comparable architecture, the memory targets are familiar:

Configuration	Raw weight memory	Practical fit
BF16	~54 GB	High-memory workstation
FP8	~27 GB	32 GB GPU, little headroom left
4-bit	~13.5 GB	24 GB GPU with runtime and cache space
3-bit	~10.1 GB	Marginal 16 GB deployment
2-bit	~6.8 GB	Large quality cost

For 24–32 GB cards (RTX 3090, 4090, 5090 class), a high-quality 4-bit quantization of a same-size 27B is the expected default, with 32 GB leaving room for context. On 16 GB cards the picture is tighter — a compact quantization fits weights but leaves little for context and runtime buffers, which is where a smaller next-generation model (14B-class) would matter more than an aggressively quantized 27B. Current 16 GB performance references live in the 16 GB VRAM benchmark comparison and in the Qwen 3.6 MTP vs standard decoding runs on an RTX 4080.

High-memory Apple Silicon is the other profile to watch for the Flash tier: a 64–128 GB unified-memory system can hold useful quantizations of a ~125B/6B-active MoE, and community experiments with SSD-based expert loading and hybrid-memory techniques for Flash-Next already exist — workable, but more complex and not equivalent to fully GPU-resident inference.

Three practical notes for launch week:

Runtime support may lag the release. New attention and embedding mechanisms usually need fresh kernels; Flash-Next already required specialized implementations. Watch llama.cpp, vLLM, SGLang, MLX, Ollama, and LM Studio — initial support may be limited to development builds.
Do not assume a uniform license. Qwen3.8-27B is Apache 2.0; Flash-Next ships under the Qwen Community License 1.0. Which license each Qwen 4 checkpoint carries matters for commercial use, redistribution, and fine-tuning.
Plan hardware against the current generation. The ~74% prediction-market figure is a probability, not a commitment. A purchase that needs Qwen 4 to justify itself should be justified against Qwen3.8-27B's ~16–24 GB practical target instead, with Qwen 4 as upside.

A checklist for the next model rumor

The Qwen 4 cycle generalizes. When the next frontier-model rumor lands:

Primary source check — is it on the lab's blog, GitHub, or Hugging Face organization? A post about the lab is one hop removed.
Traceable dates — a date with no named source is a cadence inference. Label it as one.
Numbering-gap suspicion — a model whose name fills a gap the lab's own versioning created (3.6 → 3.8 → "so 4?") deserves skepticism, especially with clean specifications attached.
Record-beating specs — a rumored benchmark above the actual current record is a red flag, not a feature.
Preview vs flagship — when an architecture preview exists, the flagship announcement is not the moment weights arrive. Plan for the lag.

Qwen 4 is confirmed to exist; almost nothing else about it is confirmed, and the reliable signals are the current generation's numbers, Flash-Next's measured results, and the ten-day weights cadence. The 27B tier will likely remain the single-GPU default for local AI, and the decision worth making now is not when it ships but which memory and quantization target it needs to beat to matter on the hardware already in use.

Tiered Context: A Knowledge Hierarchy for AI Agents

Rost — Fri, 23 Oct 2026 08:21:45 +0000

An agent with one flat memory file does not forget things; it cannot stop remembering things it was not asked about. The fix is a tiered context where each knowledge layer enters through a different trigger.

The failure mode is not forgetting, it is the opposite. In one flat store every fact lives in the same place, enters context the same way, and is therefore available the same way — always. Ask the agent about a client email and it narrates the router configuration while drafting the reply; ask it to troubleshoot a service and it mentions the sponsorship rate quoted last month. The memory pile is a monolith: any change to one concern touches every other concern, and the blast radius of every query is the entire corpus.

The standard advice is behavioral: teach the agent what is relevant to the current topic. That is the software equivalent of writing "please be careful" on a load-bearing wall — the agent can be instructed, and it will mostly comply, until the day it does not. The structural alternative is knowledge that cannot enter a conversation unless the activity that needs it is already happening. This article defines that structure as a five-layer knowledge hierarchy — what each layer is, what triggers its injection, what belongs in it, and where it fails — and applies it to a concrete multi-domain setup: one general profile, one for a homelab, one for personal matters, one for work, one for education.

Naming the hierarchy

There is no single industry-standard term, so the definition here is explicit. A knowledge hierarchy is a stratified arrangement of what the agent knows, where each stratum differs in three properties: how it enters the model's context, how large it is allowed to be, and how long it is expected to stay correct. At the knowledge-architecture level, the neighboring paradigms — PKM, RAG, wikis, and memory systems — are compared in PKM vs RAG vs Wiki vs Memory Systems, and the runtime contracts assistants actually implement are mapped in Memory Systems in AI Assistants. Both sit inside the Knowledge Management cluster, which covers the foundations, methods, and tools of human knowledge systems. This article adds the trigger-based classification and the profile boundary that make placement enforceable.

Two lineages explain the shape of the hierarchy.

The first is the CPU memory hierarchy. A processor does not store every byte it might ever need in the same register. It has L1 cache — small, always resident, accessed with zero decision-making — L2, main memory, and disk, each layer larger, slower to reach, and accessed only when the upper layers do not have the answer. The design rule is not "keep everything fast." It is most data rarely matters; pay access cost only when it does. An agent with one flat memory file is a CPU with everything crammed into L1: the register file is huge, so everything is visible, including the bytes the current instruction did not ask for. MemGPT made the same move for agents, modelling memory as tiers with explicit movement between fast and slow stores.

The second is the cognitive memory taxonomy, which divides knowledge into procedural memory (how to do things), semantic memory (facts about the world), and episodic memory (what happened, when, to whom). The taxonomy matters because the failure modes differ by type: procedural knowledge that goes stale is a bug in your workflow, semantic knowledge that goes stale is a wrong fact, and episodic knowledge that leaks is a privacy incident. A hierarchy organized by content type, not by location, tells you which failures are dangerous.

The organizing axis for this article is neither of the above. Classify knowledge by what causes it to enter context. The injection trigger is the mechanism, and the mechanism is what you can actually enforce. An agent system offers exactly four triggers:

Always. Injected into every turn, unconditionally.
Task trigger. Loaded when the current activity matches a declared condition.
Semantic recall. Surfaced when the current message is similar to something stored.
Explicit reference. Fetched because a task names the file or the query.

Every layer of the hierarchy is defined by which trigger feeds it. That is the whole design, and the rest of this article is an elaboration of one table.

The five layers

Layer	What it is	Injection trigger	Typical carrier	Size discipline	Fails as
0 — Identity	Who the agent is, how it talks, what it will not do	Always	SOUL.md or equivalent	Small; changes rarely	Bloat — identity becomes a fact store
1 — Resident memory	Durable facts that apply to every session	Always	MEMORY.md / USER.md or equivalents	Budgeted in characters; pointer-dense	Monolith — domain facts migrate in
2 — Procedural	How to do recurring tasks, plus where their data lives	Task trigger	Skills	Bounded per skill; procedures, not data	Data creep — skills become databases
3 — Reference	The system of record: tables, configs, contracts, runbooks	Explicit reference	Files, wiki	Unbounded; greppable, versioned	Staleness — nobody updates the record
4 — Retrieved memory	Episodic and relational facts: decisions, exchanges, state changes	Semantic recall	A memory backend with retrieval (e.g. a vector/graph store)	Budgeted per recall; tag-scorable	Bleed — recall surfaces the wrong domain

flowchart LR
    T0["Always"] --> L0["L0 Identity (SOUL.md)"]
    T0 --> L1["L1 Resident memory (MEMORY.md, USER.md)"]
    T1["Task trigger"] --> L2["L2 Procedural (skills)"]
    T2["Explicit reference"] --> L3["L3 Reference (wiki, runbooks)"]
    T3["Semantic recall"] --> L4["L4 Retrieved memory (domain banks)"]

Read the table down the trigger column, not across. Layer 0 and Layer 1 are the L1 cache: they pay for their always-resident privilege by being tiny. Layer 2 is instruction cache: pulled on a branch condition, the task matching the skill's trigger. Layer 3 is disk: it can be enormous because nothing enters context until a step of a task explicitly reads a named file. Layer 4 is the awkward one — it is main memory, accessed by similarity, which is the only trigger in the list that does not involve a decision made by you or by the task. Similarity is a probabilistic gate, and that is why it needs the strictest external controls, which the profile section provides. The cabinet image at the top of this article encodes the same idea: the higher a tier sits, the more freely its contents enter a conversation, and the glass partition marks where one domain's shelf ends.

Layer 0: identity

SOUL.md, or whatever your agent calls its system prompt. Its job is to define who, not what: tone, values, refusal boundaries, and memory policy. The memory policy belongs here because it is the only instruction that must be in force before any memory is consulted. An identity file that starts listing server names, client names, or rates has already failed — it has become Layer 1 content wearing Layer 0's badge, and it now rides into every conversation, including ones where it must not appear.

Layer 1: resident memory

The persistent memory files: general notes and the user profile. Two rules govern this layer, and they are the rules that separate a working hierarchy from a pile.

First, pointers, not facts. If a fact serves only one domain, it does not belong here. The correct entry is not the fact itself but a signpost:

Infra facts: wiki/infra/ + memory bank 'homelab'.
Clients and contracts: wiki/business/clients/ + bank 'business'.

The signpost is domain-neutral, so it is safe to carry everywhere. The fact it points at is loaded only by the activity that named the destination.

Second, a hard budget, enforced. If the store has no character limit, it fills, and the filling is always with the most recently discussed topic — so an infra week quietly evicts or surrounds the general context with infra. A budget that is nearly full forces triage, and triage is the only mechanism that keeps this layer honest. The Hermes implementation is a working example: MEMORY.md capped at 2,200 characters and USER.md at 1,375, injected as a frozen block at session start, is documented in Hermes Agent Memory System.

Layer 2: procedural knowledge

Skills are how the agent encodes how: the procedure, the pitfalls, and — critically for this article — the pointers into the deeper layers. A skill for homelab administration names which wiki directory holds the inventory and which memory bank to recall. A skill for client work names the contract store and the business bank. The skill's description is its trigger: it is the branch condition that decides whether the procedure, and therefore its pointers, enter context at all.

This is the layer that makes the hierarchy work operationally, because it is the seam between "what activity is happening" and "what data that activity may see." A task about a client email loads the client skill; the client skill names the business bank and the contracts directory; the homelab bank and the infra directory are simply never referenced from a path the task walked.

The failure mode is data creep: pasting the actual client list or the actual IP table into the skill body. Then the data is always loaded whenever the procedure is, and the procedure's trigger has become the data's exposure boundary. Skills carry the path to the data, and nothing else that is sensitive.

Layer 3: reference data

The wiki, the notes tree, the system of record. This is the layer with the fewest constraints and the fewest surprises, because it is the only layer with a deterministic access path: something enters context only because a step of a task explicitly read that file. Tables, inventories, contracts, runbooks, statistics — the stuff you would keep in a database or a configuration management system. When the corpus is LLM-maintained, the LLM Wiki pattern applies: synthesis happens at ingest time, and the result is stored as structured, linked pages.

The reason reference data belongs here and not in Layer 4 is that retrieval quality is the wrong tool for tables. Semantic recall is excellent for "what did client X confirm about the deadline" and terrible for "what is the port of the service on host Y" — the latter has exactly one correct row, and a reranker returning the three nearest paragraphs is a liability. Deterministic lookup wins for reference data; similarity wins for episodes. Keep them apart, and each layer's access pattern matches its content shape.

The failure mode is staleness, which is a maintenance problem, not an architecture problem: the record drifts from reality and the agent, doing everything right, works from a wrong table. The mitigation is the same one any system of record needs — one writer per file, change logs, and an audit cadence — plus the drift-detection and contradiction-check practices covered in LLM Wiki Maintenance: Drift, Contradictions and Review.

Layer 4: retrieved memory

A memory backend with per-turn automatic recall: the agent's episodic and relational memory. Decisions and their dates, "client X confirmed Y," "we tried Z and it did not work." This is where the dynamic, self-organizing part of the system lives — and where the security story gets interesting, because semantic recall is a probabilistic gate and probabilistic gates bleed.

"Billing question for client X" and "which host runs the billing server" are close in embedding space. Any single bank that holds both the client records and the infrastructure inventory will, with non-trivial probability, surface the latter in the former. You can narrow this with tags and recall restrictions, and you should — but a tag filter on a mixed corpus is a discipline, not a structure. The structural answer is to stop mixing: one bank per domain, so that the bleed radius of a recall is, by construction, one domain. Choosing the backend itself — capture policy, dependencies, self-hosting constraints — is a separate comparison, covered in Agent Memory Providers Compared.

Two mechanical details matter when you wire this layer. Write paths and read paths are separate controls: a bank can be included in recall while being excluded from retention, which is how you build a profile that reads everything and writes to nothing. And every bank should carry its own retention mission — a short statement of what that bank exists to hold. A business bank with the mission "client correspondence, contract terms, negotiation state; never infrastructure details" will extract accordingly, because extraction is an LLM call that reads the mission. OpenAI's personalization cookbook treats consolidation as the most sensitive and error-prone stage of a memory lifecycle — the retention mission is the contract that keeps that stage honest.

Profiles: the boundary that makes separation structural

Up to here, the hierarchy separates knowledge by layer — necessary but not sufficient. The five layers above are one agent's anatomy. The question this article started with is about domains: homelab, personal, work, education, and the general context that spans them.

The mechanism for domain separation is the profile — a full, independent instance of the agent: its own identity file, its own resident memory, its own skills directory, its own scheduled jobs, and, decisively, its own memory-bank wiring. A profile is not a tag on a conversation; it is a separate system that happens to share a machine. When you run agent -p homelab, you get a process whose entire Layer 0 through Layer 4 is scoped to the homelab domain, and whose recall pool cannot contain the business bank because the business bank is not in its configuration at all.

This is the difference between the two answers to the leak problem. "Do not disclose infrastructure in business conversations" is a prompt instruction — a probability. "The business profile's recall pool does not include the infrastructure bank" is a configuration — a guarantee. The hierarchy tells you what goes where; the profile boundary is what makes the placement un-bypassable from inside the conversation.

The worked example, five domains:

	common (default)	homelab	personal	work	education
Identity (L0)	Shared base + memory policy	Base + "infra operator" frame	Base + "private, no business tone"	Base + "professional, client-facing"	Base + "teacher, assume no prior knowledge"
Resident memory (L1)	Pointers only — to the four domain homes	Homelab pointers, machine facts	Personal pointers	Work pointers	Course pointers
Skills (L2)	General workflow skills	Infra skills, runbook skill, monitor skill	—	Client-operations skill, outreach skill	Lesson-planning skill
Reference (L3)	Shared index of the wiki	wiki/infra/	wiki/personal/	wiki/business/clients/	wiki/education/
Retained bank (L4)	None (read-only)	bank: homelab	bank: personal	bank: business	bank: education
Recall-only banks (L4)	homelab, personal, business, education	—	—	—	—

Two cells in that table do the heavy lifting.

The common profile's bank row is "none, read-only". The general conversation is where you ask cross-cutting questions, so it should be able to see every domain's recalled facts — but it must not write to any of them. Chatter in a general session is exactly the material you do not want extracted into a domain's durable memory, and the read-only wiring removes the choice from the model entirely. If the backend's wiring supports recall-only bank lists (separate from write banks), this is one configuration line per domain bank; if it does not, the common profile recalls from a dedicated general bank and you accept a weaker cross-domain view.

The domain profiles' recall rows are empty. Each domain profile sees exactly one bank. The homelab profile will never recall a client's contract term, not because it was told not to, but because the term is not in the pool it queries. This is the bidirectional separation the whole exercise is for: it works symmetrically by construction, with no instruction involved in either direction.

Three operational consequences follow, and they are why the profile approach costs more than the single-agent approach but is worth it for anyone whose domains are genuinely disjoint.

The trigger becomes the surface. If the homelab work happens in a terminal you launch with -p homelab, and client work happens on a messaging account routed to the work profile, then which hat you are wearing is a property of how you invoked the agent, not a flag you remember to state. Misdirection — accidentally running client tasks in the homelab context — becomes as visible as running the wrong shell.
Scheduled work inherits the boundary. Cron jobs and delegated sub-tasks run inside whichever profile owns them. A nightly infra check scheduled in the homelab profile cannot drift business content into the homelab bank, because it never has it.
The filesystem is shared, and that is the honest limit. Profiles separate context, not access. Every profile's terminal can read every profile's files; the boundary protects conversations, not disks. So the layers that hold genuinely sensitive material need their own discipline: credentials in an environment file or a secret store rather than in wiki plaintext, the wiki outside any public repository, and per-directory permissions where the OS helps. Treat the profile boundary as a firewall for context — excellent at its job — and do not let it be mistaken for one for data.

Bootstrapping from a flat pile

The hierarchy is usually reached from below: an agent that started with one memory file and now keeps everything in it. The migration order matters, because each step is only cheap once the previous one is done.

First, split the recall pool. Creating one memory bank per domain is a configuration change, not a content rewrite — and it is the step with the highest leverage, because recall bleed stops the day the banks split. Content can migrate afterwards; the boundary should not wait for it.

Second, budget Layer 1. Set a character cap on the resident files, then audit what is already in them against the cap. Domain-specific facts become pointers to where they will live; what remains is what genuinely applies to every session.

Third, give the demoted facts a destination. One wiki directory per domain, with the one-writer and change-log discipline, receives what Layer 1 releases. Skills pick up the pointers to it last, because a skill that names a directory that does not exist yet is worse than no skill.

Where the hierarchy rots

The architecture is only as good as its maintenance, and each layer rots in a characteristic way.

Layer 1 fills first. It is the layer with the most convenient write path ("just remember this"), so it is the first to accumulate domain facts in disguise. The budget makes this visible: when the store is 90% full, audit it for entries that should be pointers. The fix is always the same move — demote facts to Layer 3, promote the signpost.

Layer 2 creeps second. Skills are where procedures and data want to live together, because "the procedure references the data constantly" feels reasonable at write time. The check is mechanical: if a skill body contains a table that is the source of truth for anything, it belongs in the wiki, and the skill should carry a path.

Layer 4 drifts silently. Retrieved memory is append-oriented and self-explanatory, so stale episodes are never corrected, only outlived. The recall preamble ("these memories are historical context, not verified facts") is the standard mitigation. The cleaner mitigation is the same one Layer 3 uses: when an episode becomes a durable fact ("the client signed on this date at these terms"), it gets written into the system of record, where it is visible and editable, and the bank keeps only the narrative. A sharper variant of this drift — the agent's own inference written back as evidence — is worked through in Self-Reinforcing Memory Loops in AI Agents.

Profiles multiply. Five profiles is manageable; fifteen is a fleet. The discipline is that a new profile appears only when a new domain appears — not for a new project, a new tool, or a new mood. Project-level separation is what the reference layer's directory structure is for; domain-level separation is what profiles are for. Confusing the two is how people end up with a profile zoo and no clear recall story.

The takeaway: one placement rule

Put every piece of knowledge at the shallowest layer whose trigger matches its need: always-needed facts in resident memory as pointers, procedures in skills, reference data in files, episodes in a bank — and put each domain's bank behind a profile that no other domain's profile can recall. The agent then behaves like a well-designed system: the right knowledge is present because the right activity requested it, the wrong knowledge is absent because nothing in the conversation had a path to it, and the only thing that could force the two together is a change in configuration — something you make deliberately, and can therefore see when it happens.

The flat pile was never a discipline failure. It was an architecture where every fact shares one trigger, and that trigger is "always." Break the trigger apart, and the leaks stop being a question of the model's judgment. They stop being possible.

References

PKM vs RAG vs Wiki vs Memory Systems — the foundational comparison of personal knowledge stores, retrieval pipelines, and agent memory
Memory Systems in AI Assistants — the runtime memory stack this hierarchy sits inside
Retrieval vs Representation in Knowledge Systems — why the shape of your store determines what it can answer
AI Systems Memory hub

Trending Open Source AI Projects on GitHub, October 2026

Rost — Wed, 21 Oct 2026 09:34:50 +0000

GitHub's monthly trending page works for a language, because a language filter already defines the set. Open source AI asks a different question: the work spans five languages, and the useful cut is the job a repository does.

This snapshot was read from the public GitHub API on 16 October 2026, and inside each group the repositories are ordered by star count. Stars measure attention, and a skills folder can gather that attention faster than a serving engine, so the group comes before the sort. The license column is part of the ranking as well, because several of the most starred projects are fair-code, brand-restricted, or split across two licenses.

The Rust and Go roundups on this site were built from that same page, with a language filter doing the grouping. Here the eight job groups below replace the filter, and each section states the trade that separates the close options. The closing sections turn the tables into a shortlist you can run, and into a repeatable method for the next snapshot.

How to rank open source AI projects

Search topics, then assign a job

GitHub topics are the workable substitute for a language filter. Sort a topic such as ai-agents, llm, rag, local-llm, or agent-memory by stars, and keep repositories that were pushed in the last few months. Free-text search for "AI" is too wide, and the unfiltered trending page still mixes agents with unrelated tools.

Three public inputs are enough to rebuild the list. Use topic search for the pool, a scenario catalog such as Top Open Source AI for the job labels, and the repository LICENSE file for the last column. That catalog publishes a composite of its own: stars at 70 percent, contributors at 10 percent, and update recency at 20 percent. This article keeps the simpler rule, so the number in the table is the number on GitHub.

Useful starting queries:

Keep software, and set lists aside

A repository earns a row when someone can install it, or when it is a maintained harness that changes how an agent runs. Awesome-lists, prompt directories, and dumps of vendor system prompts collect stars and still leave nothing to deploy. They belong in a reading list, and they stay out of the tables below.

Add a trending column only from your own snapshots

The language roundups could show stars gained in a month because GitHub already computes that delta inside one language. Across these groups, a monthly delta needs two snapshots of your own. Save full_name and stargazers_count on the first of the month, then subtract. Until that series exists, total stars are the honest ranking key. "Trending" in the title means actively maintained projects with the attention to match, as of this date.

flowchart TD
  topics[Topic search on GitHub]
  catalog[Scenario catalog]
  licenseFile[License file]
  drop[Set lists and prompt dumps aside]
  group[Assign each repo to one job]
  rank[Sort by stars inside the job]
  topics --> drop
  catalog --> group
  drop --> group
  group --> rank
  licenseFile --> rank

Where the projects sit

These groups are layers you can stack. A coding assistant can call a gateway, a gateway can call Ollama or vLLM, and a memory service can sit beside either of them. Two products from the same layer are usually a second opinion, and they are rarely a better architecture.

flowchart TB
  agent[Personal agent]
  coder[Coding assistant]
  skills[Skills and harness]
  memory[Memory]
  docs[RAG and documents]
  flow[Workflow builder]
  gate[Model gateway]
  run[Local runtime]
  coder --> skills
  agent --> memory
  coder --> memory
  agent --> docs
  flow --> docs
  agent --> gate
  coder --> gate
  flow --> gate
  docs --> run
  gate --> run

Personal AI assistants

This group is for agents that take actions on a machine or across tools, beyond a single coding session. OpenClaw leads by a wide margin. Hermes is the other general agent at this scale, and AutoGPT is the older name that is still accumulating stars while its license has split. A deeper leaderboard for this pair alone, with downloads, token share, and CVE history, is the OpenClaw vs Hermes Agent comparison.

Project	Stars	License	Repository
OpenClaw	392,126	MIT	openclaw/openclaw
Hermes Agent	254,122	MIT	NousResearch/hermes-agent
Firecrawl	191,908	AGPL-3.0	firecrawl/firecrawl
AutoGPT	187,598	MIT and Polyform Shield	Significant-Gravitas/AutoGPT

Firecrawl is listed here because agents need pages, and it is the web-data project readers actually star. It is a tool, and the assistants call it.

OpenClaw

Metric	Value
Repository	openclaw/openclaw
Stars	392,126
License	MIT
Language	TypeScript

OpenClaw is a personal agent meant to run on your own operating system and carry out tasks, which is the plain reading of a project that calls itself an AI that does things on any platform. The star count is the largest in this article, and the MIT license is the permissive end of the set. Treat the install as a local automation runtime with broad access to the machine, and scope what it is allowed to touch before you leave it running.

Hermes Agent

Metric	Value
Repository	NousResearch/hermes-agent
Stars	254,122
License	MIT
Language	Python

Hermes Agent, from Nous Research, is a general agent designed to accumulate skills and context as it is used. It sits in the same job as OpenClaw, with a Python codebase and the same MIT terms. Choose it when you want that growth loop and you are already comfortable operating a Python agent. Running both Hermes and OpenClaw means two permission models to audit.

AutoGPT

Metric	Value
Repository	Significant-Gravitas/AutoGPT
Stars	187,598
License	MIT outside autogpt_platform; Polyform Shield inside it
Language	Python

AutoGPT is the project that made autonomous agents a mainstream GitHub category, and it is still one of the most starred names in the set. The license is no longer a single grant. Everything outside the autogpt_platform folder is MIT, and everything inside that folder is under the Polyform Shield License, which permits use but reserves the code's commercial value to the producer. Read that split before you build a product on the platform directory.

Firecrawl

Metric	Value
Repository	firecrawl/firecrawl
Stars	191,908
License	AGPL-3.0
Language	TypeScript

Firecrawl turns sites into data that agents can call. The star count puts it among the assistants even though the job is acquisition, which says a lot about what people are wiring up in 2026. AGPL-3.0 matters if you modify it and offer the result as a service. For a personal agent that only calls a local or vendor endpoint, the license is a smaller constraint.

AI coding assistants

Coding agents are the densest part of the 2026 list. OpenCode leads the fully open terminal and editor agents, with Codex and Gemini CLI close behind as the vendor CLIs that happen to be open source. OpenHands, Cline, and Goose are the other programs you can actually run.

Aider (49,567 stars, Apache-2.0), Continue (36,228, Apache-2.0), and Crush (28,669, FSL-1.1-MIT) are real coding tools in the next band. Crush's license is a Functional Source License that converts to MIT on the schedule written in the file. One coding agent is plenty. A second one is for a comparison week, and after that it is configuration you have to keep in sync.

Project	Stars	License	Repository
OpenCode	213,922	MIT	anomalyco/opencode
Codex	129,395	Apache-2.0	openai/codex
Gemini CLI	107,389	Apache-2.0	google-gemini/gemini-cli
OpenHands	91,004	MIT	OpenHands/OpenHands
Cline	70,370	Apache-2.0	cline/cline
Goose	55,324	Apache-2.0	aaif-goose/goose

OpenCode

Metric	Value
Repository	anomalyco/opencode
Stars	213,922
License	MIT
Language	TypeScript

OpenCode is an open source coding agent with a terminal workflow and a large plugin surface. It has become the default name people cite when they want a coding agent they can inspect, and the MIT license matches that use. The repository is anomalyco/opencode. Pick it when the agent itself should be something you can patch.

Codex

Metric	Value
Repository	openai/codex
Stars	129,395
License	Apache-2.0
Language	Rust

Codex is OpenAI's lightweight coding agent for the terminal, written in Rust and licensed Apache-2.0. It is the open client for a hosted model family, which is a different bargain from a local-first agent. Use it when your team already standardizes on OpenAI models and you want a CLI you can automate. The client being open does not move the model weights onto your disk.

Gemini CLI

Metric	Value
Repository	google-gemini/gemini-cli
Stars	107,389
License	Apache-2.0
Language	TypeScript

Gemini CLI brings Gemini into the terminal as an agent, under Apache-2.0. It plays the same role as Codex for a different provider, and the TypeScript codebase is easier to embed in existing Node tooling. Choose it when Gemini is the model you already pay for. Running it beside Codex is reasonable only if each repo has one default agent.

OpenHands

Metric	Value
Repository	OpenHands/OpenHands
Stars	91,004
License	MIT
Language	TypeScript

OpenHands is a software-development agent with a full task loop, now published from the OpenHands organization. The MIT license and the higher-level product make it a platform choice, closer to a hosted development environment you can self-host than to a single binary. Use it when you want issues, sandboxes, and a UI around the agent. For a one-person terminal workflow, OpenCode, Codex, or Goose is the smaller install.

Cline

Metric	Value
Repository	cline/cline
Stars	70,370
License	Apache-2.0
Language	TypeScript

Cline is an autonomous coding agent that ships as an SDK, an IDE extension, or a CLI. Apache-2.0 and the editor integration make it the practical pick for people who live in VS Code and still want a library they can call. It overlaps OpenCode more than it overlaps a chat UI. Standardize on one of them per repository so reviews stay readable.

Goose

Metric	Value
Repository	aaif-goose/goose
Stars	55,324
License	Apache-2.0
Language	Rust

Goose is an extensible local agent that installs, edits, runs, and tests code with whatever model you point at it. The repository now lives at aaif-goose/goose, and the Rust implementation plus Apache-2.0 make it a strong workstation agent. Choose Goose when the model provider should be swappable and the agent should run on the machine that holds the code. It is a quieter star count than OpenCode, and it is a more direct fit for a local model.

Agent skills and harnesses

Topic search will put this group above several real products. Superpowers and ECC are instructions, hooks, and workflows for agents you already run. Spec Kit is GitHub's own process toolkit for spec-driven development. They change agent behavior, and they are still software you compose with a coding assistant from the previous section.

Project	Stars	License	Repository
Superpowers	299,886	MIT	obra/superpowers
ECC	280,382	MIT	affaan-m/ECC
Spec Kit	141,439	MIT	github/spec-kit

Superpowers

Metric	Value
Repository	obra/superpowers
Stars	299,886
License	MIT
Language	Shell

Superpowers is an agentic skills framework and a software-development methodology, distributed mostly as shell and instruction files under MIT. The star count is what you get when a method spreads through coding-agent users in a few months. Install it on top of an agent you already trust, and expect to edit the skills to match the repository. The framework will follow whatever permissions that agent already has.

ECC

Metric	Value
Repository	affaan-m/ECC
Stars	280,382
License	MIT
Language	JavaScript

ECC is a harness for Claude Code, Codex, OpenCode, Cursor, and similar agents, covering skills, memory, and checks around how the agent works. It is the right shape if you want the same loop on more than one coding agent. It is also another moving layer between you and the model. Adopt it when the team has already picked the agents and wants one shared way to brief them.

Spec Kit

Metric	Value
Repository	github/spec-kit
Stars	141,439
License	MIT
Language	Python

Spec Kit is GitHub's toolkit for spec-driven development and the process around it. The maintainer is the platform most of these repositories already live on, and the MIT license is straightforward. Use it when you want a written spec before the agent edits code. It will stay quiet if the team treats the spec as optional paperwork.

AI memory

Memory is the layer that lets an agent resume. The starred projects split into two styles. claude-mem, Beads, and much of Hindsight attach memory to coding sessions. Mem0, Cognee, and Letta are services you call from application agents. Capture policy, dependencies, and self-hosting tradeoffs for these backends are compared in Agent Memory Providers.

Memvid (16,600 stars, Apache-2.0) and Zep (4,961 stars, Apache-2.0) are smaller options in the same job, Memvid as a single-file store and Zep as an integration-heavy memory API. One memory system per agent is the setup you can still explain later.

Project	Stars	License	Repository
claude-mem	101,052	Apache-2.0	thedotmack/claude-mem
Mem0	67,243	Apache-2.0	mem0ai/mem0
Hindsight	50,055	MIT	vectorize-io/hindsight
Cognee	32,364	Apache-2.0	topoteretes/cognee
Beads	27,917	MIT	gastownhall/beads
Letta	25,156	Apache-2.0	letta-ai/letta

claude-mem

Metric	Value
Repository	thedotmack/claude-mem
Stars	101,052
License	Apache-2.0
Language	TypeScript

claude-mem records what a coding agent does, compresses that history, and injects the relevant part into later sessions. It talks to Claude Code, OpenClaw, Codex, Gemini CLI, Hermes, Copilot, and OpenCode, which is why it leads this group. Apache-2.0 fits a sidecar you may want to modify. Use it when the thing you keep forgetting is the coding session itself.

Mem0

Metric	Value
Repository	mem0ai/mem0
Stars	67,243
License	Apache-2.0
Language	Python

Mem0 is a memory layer for agents and applications, with user, session, and agent state behind an API. It is the default name when the product is a support bot or a personal assistant, and you want that memory outside any one coding tool. Apache-2.0 and a hosted option sit side by side, so self-hosting is a choice you can make later. Prefer it over a coding-session hook when many agents should share the same facts about a user.

Hindsight

Metric	Value
Repository	vectorize-io/hindsight
Stars	50,055
License	MIT
Language	Python

Hindsight is Vectorize's agent memory project, aimed at agents that learn from what they have already done, and it integrates with both coding agents and application frameworks. The MIT license is the most permissive in this group. The project publishes strong benchmark claims, and those claims should be re-run on your own transcripts before you bet a product on them. Pick Hindsight when you want a dedicated memory service and you are willing to compare it with Mem0 on your data.

Cognee

Metric	Value
Repository	topoteretes/cognee
Stars	32,364
License	Apache-2.0
Language	Python

Cognee builds a memory and knowledge graph that agents can query, with an emphasis on running that layer on smaller models. It overlaps Mem0 and it also overlaps the RAG tools below, because a graph of your documents behaves like both memory and retrieval. Apache-2.0 keeps the embedding path replaceable. Choose it when the facts you care about are entities and relationships, and a flat vector index has already felt too lossy.

Beads

Metric	Value
Repository	gastownhall/beads
Stars	27,917
License	MIT
Language	Go

Beads stores coding-agent tasks, plans, and dependencies as files that travel with the git branch. The repository is now gastownhall/beads, and it is the same project the January Go roundup listed under its earlier path. MIT and a git-backed store make it the memory tool that your existing review process can see. Use it when context should fork with the branch. Use claude-mem or Mem0 when the memory has to outlive a single repository.

Letta

Metric	Value
Repository	letta-ai/letta
Stars	25,156
License	Apache-2.0
Language	Python

Letta is the stateful-agent platform that grew out of MemGPT. Memory here is part of the agent runtime, so the agent manages its own context as a feature of the platform. That is a deeper commitment than dropping Mem0 under an agent you already run. Apache-2.0 applies to the open platform. Choose Letta when you want the runtime and the memory designed together, and choose Mem0 when you want memory under agents you do not intend to replace.

RAG and knowledge bases

RAG projects turn documents into a context layer. RAGFlow is the end-to-end engine people are starring. LlamaIndex is the document framework, GraphRAG is Microsoft's graph approach, and WeKnora is Tencent's knowledge platform, which also appeared in the January Go roundup and has kept growing.

Project	Stars	License	Repository
RAGFlow	92,289	Apache-2.0	infiniflow/ragflow
LlamaIndex	52,513	MIT	run-llama/llama_index
GraphRAG	36,349	MIT	microsoft/graphrag
WeKnora	34,210	MIT	Tencent/WeKnora

RAGFlow

Metric	Value
Repository	infiniflow/ragflow
Stars	92,289
License	Apache-2.0
Language	Go

RAGFlow is a self-hosted RAG engine that parses messy documents and exposes retrieval to agents, with citations as part of the product. Apache-2.0 and a UI you can hand to a team are why it leads this group by a clear gap. Use it when the job is "put our files behind an answer box" and you want the parser, the index, and the API from one project. Reach for LlamaIndex when you are embedding retrieval inside an application you already control.

LlamaIndex

Metric	Value
Repository	run-llama/llama_index
Stars	52,513
License	MIT
Language	Python

LlamaIndex is a document-processing framework for agents and RAG pipelines. It is a library with many connectors, and the MIT license fits that role. You assemble the index, the store, and the agent yourself, which is the cost of the flexibility. Choose it for product code. Choose RAGFlow when you want the same job as a service you deploy.

GraphRAG

Metric	Value
Repository	microsoft/graphrag
Stars	36,349
License	MIT
Language	Python

GraphRAG builds a graph over a corpus and answers questions by walking that graph, as Microsoft Research's open implementation of the idea. The technique itself is explained in Advanced RAG: LongRAG, Self-RAG and GraphRAG. MIT licensing makes it easy to try beside a vector pipeline. Graph construction costs more up front than a chunk-and-embed index, and it pays that cost back on questions that span many documents. Use it when the corpus has structure you are currently forcing into chunks.

WeKnora

Metric	Value
Repository	Tencent/WeKnora
Stars	34,210
License	MIT
Language	Go

WeKnora is Tencent's knowledge platform for turning documents into a RAG index, an agent, and a wiki that the project describes as self-maintaining. The project license is MIT, with third-party components under their own terms. GitHub reports that license as Other, because the MIT grant sits under a Tencent preamble. It is the closest cousin to RAGFlow in this table, with a Go core. Prefer it when you want the wiki and the agent in the same box, and compare the parsers on a sample of your own PDFs before you migrate a library.

Workflow and agent builders

Builders are where a team draws the flow, or where a developer imports the framework that draws it for them. n8n is the largest by stars and the narrowest by license. Dify and Langflow are the AI-native studios. LangChain is the library underneath a great many of the others.

CrewAI (59,745 stars, MIT) and AutoGen (61,429 stars, CC-BY-4.0) belong in the same conversation as code-first multi-agent frameworks. AutoGen's Creative Commons attribution license is unusual for a code repository, and it is worth reading before you vendor it. LangChain remains the integration surface those frameworks still meet.

Project	Stars	License	Repository
n8n	207,535	Sustainable Use License	n8n-io/n8n
Dify	158,591	Modified Apache-2.0	langgenius/dify
Langflow	155,717	MIT	langflow-ai/langflow
LangChain	147,886	MIT	langchain-ai/langchain

n8n

Metric	Value
Repository	n8n-io/n8n
Stars	207,535
License	Sustainable Use License
Language	TypeScript

n8n is a workflow platform with hundreds of integrations and native AI steps, and it is the project people mean when they want Zapier-style automation they can host. The Sustainable Use License allows internal business use, personal use, and non-commercial use. Distribution is limited to free, non-commercial copies, and files marked .ee need a separate enterprise license. It is source-available fair-code, and it is still the right tool for internal automation if those limits match how you run it.

Dify

Metric	Value
Repository	langgenius/dify
Stars	158,591
License	Modified Apache-2.0
Language	TypeScript

Dify is a workspace for agent workflows and RAG pipelines, with model providers and tools in one deployable stack. The license starts from Apache-2.0 and adds conditions: operating a multi-tenant service, meaning multiple workspaces, needs a commercial license, and the console logo and copyright notice stay on the frontend. Single-tenant internal use is the case the extra clauses leave open. Choose Dify when a team needs one UI from prototype to production, and choose Langflow when the license has to stay plain MIT.

Langflow

Metric	Value
Repository	langflow-ai/langflow
Stars	155,717
License	MIT
Language	Python

Langflow is a visual builder for agents and RAG flows, with an MIT license and a star count essentially tied with Dify. The canvas is the product, and the generated flows are something a developer can still own. Use it when you want a diagram of the pipeline and a license a procurement checklist already understands. The trade is a younger application surface than Dify's workspace features.

LangChain

Metric	Value
Repository	langchain-ai/langchain
Stars	147,886
License	MIT
Language	Python

LangChain is the agent-engineering library that most integration docs still assume. MIT and the breadth of connectors keep it in every shortlist, and the abstraction cost is real once the chain is doing something simple. Use it when you are writing application code and you want the adapters already written. Use Dify or Langflow when the deliverable is a workspace, and use a direct provider SDK when the flow is a handful of calls.

Local inference and chat interfaces

This is the layer under everything else. Ollama is the workstation runner, Transformers is the model-definition library, Open WebUI is the chat front end, llama.cpp is the portable C++ engine, and vLLM is the server engine. Hugging Face Transformers is included because so many of the other projects import it, even though you do not "run" it the way you run Ollama. API maturity, hardware support, and production readiness across the runtimes are compared in Ollama vs vLLM vs LM Studio.

Project	Stars	License	Repository
Ollama	183,334	MIT	ollama/ollama
Transformers	167,734	Apache-2.0	huggingface/transformers
Open WebUI	155,825	Open WebUI License	open-webui/open-webui
llama.cpp	131,311	MIT	ggml-org/llama.cpp
vLLM	93,870	Apache-2.0	vllm-project/vllm

Ollama

Metric	Value
Repository	ollama/ollama
Stars	183,334
License	MIT
Language	Go

Ollama runs current open models locally behind a simple CLI and an OpenAI-compatible API. It was already a leader in the January Go roundup, and at 183,334 stars it is still the default workstation runtime. MIT and a single daemon are the reasons. Use it on a developer machine and on small servers. Move the same weights to vLLM when concurrency, batching, and GPU utilization become the problem.

Transformers

Metric	Value
Repository	huggingface/transformers
Stars	167,734
License	Apache-2.0
Language	Python

Transformers is Hugging Face's library for defining, loading, and running models across text, vision, audio, and multimodal checkpoints. Apache-2.0 and the model hub around it make this the library the rest of the stack imports. You feel it when you are training, converting, or debugging a model. You can ignore it for a long time if Ollama or vLLM already serves the checkpoint you need.

Open WebUI

Metric	Value
Repository	open-webui/open-webui
Stars	155,825
License	Open WebUI License
Language	Python

Open WebUI is the chat interface most people put in front of Ollama and OpenAI-compatible APIs. The Open WebUI License allows redistribution and then restricts branding. Above 50 end users in any rolling 30 days, you may not alter, remove, or replace the Open WebUI name and marks without written permission or an enterprise license. For a personal or small-team install that keeps the name, it is the polished front end. For a product you will rebrand, budget the enterprise conversation first. Docker deployment and the setup options are walked through in the Open WebUI guide.

llama.cpp

Metric	Value
Repository	ggml-org/llama.cpp
Stars	131,311
License	MIT
Language	C++

llama.cpp is the C and C++ inference engine that made local models practical on ordinary hardware. MIT licensing and the GGUF ecosystem are why Ollama and a crowd of smaller apps sit on top of it. Use the project directly when you want the engine, the quantization tools, or a binary with few dependencies. Use Ollama when you want that engine with model management already wrapped.

vLLM

Metric	Value
Repository	vllm-project/vllm
Stars	93,870
License	Apache-2.0
Language	Python

vLLM is a high-throughput serving engine for GPUs, with continuous batching and an OpenAI-compatible API. Apache-2.0 and the deployment focus make it the server counterpart to Ollama. Star count is lower because the audience is people who run GPUs, and that audience is the one that feels the difference. Choose vLLM when more than a few concurrent sessions share one machine.

LLM gateways

A gateway is the OpenAI-compatible front door in front of many providers. LiteLLM is the common choice, New API is the self-hosted distributor, and Bifrost is the smaller high-throughput gateway. Put a gateway in the stack when you have more than one provider, a budget per key, or a need to fail over. A single local Ollama daemon does not need one.

Project	Stars	License	Repository
LiteLLM	61,338	MIT for the core	BerriAI/litellm
New API	50,037	AGPL-3.0	QuantumNous/new-api
Bifrost	8,837	Apache-2.0	maximhq/bifrost

LiteLLM

Metric	Value
Repository	BerriAI/litellm
Stars	61,338
License	MIT outside enterprise/
Language	Python

LiteLLM exposes a large set of provider APIs through one OpenAI-shaped interface, with spend tracking, retries, and guardrails. Code outside the enterprise/ directory is MIT, and that directory has its own license. The Python SDK is what application code imports, and the proxy is what the platform team runs. Use the core when you want one client for many vendors. Check the enterprise directory before you assume the whole repository is MIT.

New API

Metric	Value
Repository	QuantumNous/new-api
Stars	50,037
License	AGPL-3.0
Language	Go

New API is a model hub that aggregates providers and re-serves them as OpenAI, Claude, or Gemini compatible endpoints. It is the gateway people deploy when they also want user accounts, quotas, and a distribution UI. AGPL-3.0 is the term that matters if you modify the hub and offer it as a service. For an internal gateway with upstream keys, it is a concrete alternative to LiteLLM's proxy, with a Go deployment and a stricter license.

Bifrost

Metric	Value
Repository	maximhq/bifrost
Stars	8,837
License	Apache-2.0
Language	Go

Bifrost is a Go gateway for models and tools, with failover, budgets, and an MCP gateway in the same process. The star count is far below LiteLLM, and the January Go roundup already had it on the trending list, so the project is younger than it is obscure. Apache-2.0 avoids the enterprise-directory split. Consider it when the gateway is on the request path of a busy service and you want a small Go binary. Keep LiteLLM when you need the long tail of provider adapters today.

A stack worth running

The tables are a map, and a workstation still needs a short list. For a local setup in October 2026, Ollama plus one coding agent covers the daily loop. Add Open WebUI if the branding clause fits the user count, Mem0 or claude-mem depending on whether memory belongs to an application or to a coding session, and RAGFlow when the documents are the product. Add LiteLLM or New API only after a second model provider shows up.

Layer	Default pick	Choose the alternative when
Runtime	Ollama	vLLM, once several users share a GPU server
Chat UI	Open WebUI	You need to remove the brand above 50 users
Coding agent	OpenCode, Codex, Gemini CLI, or Goose	You self-host the whole task loop: OpenHands
Memory	claude-mem or Mem0	The facts are a graph: Cognee. The facts follow a git branch: Beads
Documents	RAGFlow	You are embedding retrieval in your own service: LlamaIndex
Builder	Langflow	You accept Dify's multi-tenant clause, or n8n's fair-code terms for internal automation
Gateway	LiteLLM core	You want AGPL and a user-facing hub: New API

Skills packs such as Superpowers and ECC come after the agent is chosen. They are easy to star and easy to accumulate, and three harnesses on one repository will fight over the same instructions.

Keeping the ranking current

Repeat the topic queries at the top of this article, assign each new repository to one job, and re-sort that job by stars. Then open the license file. A project that jumped the table on a skills directory, a prompt list, or a weekend of attention can wait for the next snapshot before it replaces a tool you run.

Store the snapshot. A single CSV of full_name, stargazers_count, license, and pushed_at, taken monthly, turns the next edition of this page into a real trending column, the same kind of delta the language articles already show. Until then, the groups are the part that ages well, and the star counts are the part that goes stale on purpose.

Sources

Star counts, languages, and license identities were read from the GitHub repository API and the repository LICENSE files on 16 October 2026. Job labels also draw on the public scenario list at Top Open Source AI, which documents its own composite score separately from the sort used here.

Hindsight for Hermes Agent: Self-Hosted Memory with Local LLMs

Rost — Sun, 18 Oct 2026 02:50:36 +0000

A self-hosted AI agent needs more than a long context window. It needs a way to retain useful facts, retrieve earlier decisions, and revise what it thinks it knows as projects evolve.

Hermes Agent ships with built-in memory files, but an external provider can provide much richer retrieval across sessions. Hindsight is one of the more interesting options in that space. This guide installs it for Hermes with no cloud inference, first as an all-in-one Docker container and then against an existing PostgreSQL server with separate local model endpoints.

The guide covers deployment, choosing and wiring local embedding, reranking, and generative models, controlling what auto-retain writes into the bank, connecting Hermes, and the operational layer: backup, migration, and day-to-day bank hygiene. The examples use a deliberate port scheme — 8080 for the main llama.cpp server, 8081 for embeddings, 8082 for reranking, 11434 for Ollama — so change them to the ports you already use and keep the small servers off the main model's port.

What Hindsight Does

Hindsight is a persistent agent memory engine with three principal operations: retain, recall, and reflect. Retain extracts and stores information; recall combines semantic, keyword, graph, and temporal retrieval; reflect synthesizes an answer from accumulated memories. Background consolidation can turn overlapping observations into a more useful knowledge layer.

It is not just an embeddings database. A vector store can locate similar passages, but a memory system also has to preserve chronology, project boundaries, contradictory evidence, and changes in decisions. Those failure modes are explored in self-reinforcing memory loops.

For the wider market, see agent memory providers compared. Hermes' built-in MEMORY.md, USER.md, and memory modes are covered in the Hermes Agent memory system guide, so this article assumes you know how the built-in layer works.

Why Move Beyond Mnemosyne

Mnemosyne for Hermes offers appealing local-first ideas, including SQLite storage and hybrid retrieval. In practice, installation constraints, version compatibility, write-approval behaviour, and rapid changes around its 4.0 prereleases can make it difficult to operate predictably. Those are operational concerns, not evidence that Mnemosyne's memory model is fundamentally flawed.

Hindsight is not dependency-free either: its Docker image and local models can consume substantial disk space. Running its API as an independent service, with inference elsewhere, keeps upgrades and troubleshooting isolated from the agent itself.

Choose a Deployment

Component	A: All-in-one container	B: Existing PostgreSQL server
Hindsight API and UI	Docker container	Docker container
PostgreSQL	Embedded `pg0` in container	Existing PostgreSQL machine
Generative LLM	External llama.cpp	External llama.cpp
Embeddings	Default local or external	External Ollama or llama.cpp
Reranking	Default local or external	External rerank server
Best for	Quick evaluation	Long-running, isolated installation

Here, "all-in-one" means the Hindsight API, UI, and database are in one container; the inference LLM is still hosted elsewhere. This is intentional: Docker's official Hindsight image does not bundle a ready-to-use local generative llama.cpp server.

flowchart LR
    H[Hermes Agent] -->|Memory API| A[Hindsight API]
    A --> P[(PostgreSQL)]
    A --> E[Embedding endpoint]
    A --> R[Reranking endpoint]
    A --> L[Generative LLM endpoint]

Use LAN-reachable addresses, not localhost, for services running on a different host. Restrict these ports to trusted clients; the example dummy keys are compatibility placeholders, not authentication.

Option A: All-in-One Hindsight Container

This is the simplest way to evaluate Hindsight without filling the Hermes VM with Python, Torch, and local model dependencies. The container runs Hindsight and embedded PostgreSQL, while your main llama.cpp LLM handles memory extraction and reflection.

Two image tags matter here. ghcr.io/vectorize-io/hindsight:latest is the large image — roughly 9 GB on AMD64 — because it bundles a BGE embedder, a MiniLM reranker, and embedded PostgreSQL. latest-slim is roughly 500 MB and expects those two models to live elsewhere, which is what you want once you point Hindsight at the embedding and reranking setup in Serve and Wire Local Models. Once the health check passes, pin a specific tag: a moving tag is a poor match for a database you intend to keep.

Create a directory and a Compose file on the Hindsight host:

mkdir -p ~/services/hindsight
cd ~/services/hindsight
cat > compose.yaml <<'YAML'
services:
  hindsight:
    image: ghcr.io/vectorize-io/hindsight:latest
    container_name: hindsight
    restart: unless-stopped
    shm_size: 1gb
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "8888:8888"  # Memory API
      - "9999:9999"  # Control-plane UI
    environment:
      HINDSIGHT_API_WORKER_ID: hindsight-standalone
      HINDSIGHT_API_LLM_PROVIDER: openai
      HINDSIGHT_API_LLM_BASE_URL: http://WORKSTATION_IP:8080/v1
      HINDSIGHT_API_LLM_MODEL: LocalMegaLLM
      HINDSIGHT_API_LLM_API_KEY: dummy
      HINDSIGHT_API_LLM_MAX_CONCURRENT: "2"
    volumes:
      - hindsight-pg0:/home/hindsight/.pg0
volumes:
  hindsight-pg0:
YAML

Replace WORKSTATION_IP with the host running llama.cpp. On Linux, host.docker.internal resolves only if the container declares the host-gateway mapping shown above; without it, a base URL of http://host.docker.internal:8080/v1 points nowhere. The 1 GB shared-memory reservation is particularly useful for embedded PostgreSQL indexing; Docker's small default /dev/shm can cause failures. A named volume preserves the database when the container is recreated, and a bind mount alternative has to be owned by uid 1000, the user inside the image.

HINDSIGHT_API_WORKER_ID should stay stable across recreations — Docker's default is the container hostname, and a task that was mid-flight when a container died stays stuck under the old worker id.

docker compose up -d
docker compose logs -f --tail=100
curl -f http://localhost:8888/health

If your installed Hindsight version exposes health at a different path, inspect http://localhost:8888/docs and the container logs. The control plane is normally available at http://HINDSIGHT_HOST:9999. Once the basic deployment works, you can also point its embedding and reranking providers at the configuration in Serve and Wire Local Models rather than loading the default models inside the container.

Do not expose ports 8888 or 9999 directly to the internet. Use a private network or an authenticated reverse proxy. Embedded PostgreSQL is convenient for testing, but an externally managed database is easier to back up and maintain.

Option B: Hindsight Against an Existing PostgreSQL Server

For a long-running installation, I prefer keeping the Hindsight API, database, and inference endpoints separate. The Hindsight VM can remain small because it runs orchestration and HTTP services rather than heavy models.

flowchart LR
    H[Hermes VM] -->|port 8888| A[Hindsight VM]
    A -->|port 5432| P[(PostgreSQL host)]
    A -->|port 8080| L[Generative LLM]
    A -->|port 8081 or 11434| E[Embedding endpoint]
    A -->|port 8082| R[Reranking endpoint]

Prepare PostgreSQL and pgvector

Hindsight needs PostgreSQL 14 or newer and a vector extension in the hindsight database. The extension is per database: installing the package on the server does nothing until CREATE EXTENSION runs there. On a Debian or Ubuntu PostgreSQL server, install the pgvector package matching the actual PostgreSQL major version; for example, on PostgreSQL 16:

sudo apt update
sudo apt install postgresql-16-pgvector

The package name and availability depend on your configured apt repositories. If it is not available, follow the pgvector installation instructions, or install a compatible package from the PostgreSQL PGDG repository. Do not blindly change the PostgreSQL major version on a shared server.

Create a dedicated role and database as a PostgreSQL administrator:

CREATE ROLE hindsight LOGIN PASSWORD 'REPLACE_WITH_LONG_PASSWORD';
CREATE DATABASE hindsight OWNER hindsight;

Connect to the new database and enable the extension:

sudo -u postgres psql -d hindsight -c \
  'CREATE EXTENSION IF NOT EXISTS vector;'
sudo -u postgres psql -d hindsight -c \
  "SELECT extname, extversion FROM pg_extension WHERE extname = 'vector';"

The extension normally needs elevated privileges to install; the application role does not need to be a superuser. Hindsight uses pgvector by default and wants 0.8.0 or newer for iterative ANN scans; older servers reject that setting. HINDSIGHT_API_VECTOR_EXTENSION can alternatively select pgvectorscale (a DiskANN option Hindsight documents for very large banks) or vchord, both of which need extra packaging on this exact PostgreSQL build. Switching algorithms after memories exist stops startup and asks for a migration, so choose before the first retain. For a personal Hermes bank, pgvector is the one to install and then stop.

Allow only the Hindsight host through pg_hba.conf, for example:

host  hindsight  hindsight  192.168.1.50/32  scram-sha-256

Set PostgreSQL listen_addresses appropriately and use TLS or a trusted private network for cross-host connections. Back up this database independently of the Hindsight Docker container. If the server is shared with other applications, keep Hindsight in its own database, or set HINDSIGHT_API_DATABASE_SCHEMA to a schema you own; the default schema is public.

Run Hindsight Against PostgreSQL

Create compose.yaml on the Hindsight VM. This version deliberately does not start PostgreSQL locally:

services:
  hindsight:
    image: ghcr.io/vectorize-io/hindsight:latest-slim
    restart: unless-stopped
    ports:
      - "8888:8888"
      - "9999:9999"
    env_file: .env
    environment:
      HINDSIGHT_API_LLM_PROVIDER: openai
      HINDSIGHT_API_LLM_BASE_URL: http://WORKSTATION_IP:8080/v1
      HINDSIGHT_API_LLM_MODEL: LocalMegaLLM
      HINDSIGHT_API_LLM_API_KEY: dummy
      HINDSIGHT_API_LLM_MAX_CONCURRENT: "2"
      HINDSIGHT_API_DB_POOL_MIN_SIZE: "2"
      HINDSIGHT_API_DB_POOL_MAX_SIZE: "10"
      HINDSIGHT_API_WORKER_ID: hindsight-standalone

Store the external database URL in a file called .env:

HINDSIGHT_API_DATABASE_URL=postgresql://hindsight:YOUR_PASSWORD@POSTGRES_IP:5432/hindsight

chmod 600 .env
docker compose up -d
docker compose logs -f --tail=100

If your password contains @, :, /, or other reserved URL characters, URL-encode it before putting it in the connection string. Test that the Hindsight VM can reach PostgreSQL port 5432, and that pg_hba.conf allows the Hindsight host — a server bound only to 127.0.0.1 refuses the connection even when the URL looks right. Allow the first startup to run database migrations; migrations run on startup unless you set HINDSIGHT_API_RUN_MIGRATIONS_ON_STARTUP=false. With the API and database up, continue with Choose Models for Hindsight and Serve and Wire Local Models before connecting Hermes.

Choose Models for Hindsight

Deploy puts the API and database on the network. The next decision is which models fill the three inference roles: embeddings for recall candidates, a reranker to re-order them, and a generative LLM for retain, reflect, and consolidation. A practical default for a CPU Hindsight host whose GPU is already busy with conversation is in-process Qwen/Qwen3-Embedding-0.6B, in-process BAAI/bge-reranker-v2-m3, and a shared llama.cpp chat model for generative work.

Embedding models

Hindsight reads the embedding dimension on startup and sizes the bank to match, so the model you pick at first retain is the model the bank keeps until you rebuild it. Dimensionality and instruction behaviour matter more than leaderboard rank: a stronger 8B embedder is not worth a migration if the bank is already a 384- or 1024-dimensional one. Rows below are sorted from lightest to heaviest; RAM figures are approximate loaded footprint on CPU (FP16 / typical local load), not GGUF file size.

Model	Params	Dim	Context	Loaded	Runs on	Notes
`BAAI/bge-small-en-v1.5`	33M	384	512	~100–130 MB	CPU	Hindsight default; ships in `hindsight:latest`. Fastest local path. Score distribution still calibrates many `recall_min_scores` defaults after you leave it.
`BAAI/bge-base-en-v1.5`	109M	768	512	~400 MB	CPU	Mid-tier English BGE; locks a 768-dim bank.
`sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`	~118M	384	512	~200 MB	CPU	Hindsight-documented multilingual MiniLM; 50+ languages at the same 384-dim bank width as BGE-small.
`intfloat/multilingual-e5-small`	~118M	384	512	~200 MB	CPU	Default for the `onnx` provider; light multilingual CPU path when you have an ONNX export.
`nomic-ai/nomic-embed-text-v1.5`	~137M	768	8k	~500 MB	CPU	Common Ollama embedder; longer context than classic BGE, still English-first.
`google/embeddinggemma-2`	270M	768	8k	~540 MB	CPU	EmbeddingGemma 2 text-only (disable vision/audio towers). 100+ languages + code; MRL truncates to 512/256/128; instruction prefixes for search vs document. Full multimodal load is 740M.
`BAAI/bge-m3`	568M	1024	8k	~1.1 GB	CPU or GPU	Strong multilingual dual-encoder; same width as Qwen3-0.6B, so a same-dim swap still needs a full re-embed.
`Qwen/Qwen3-Embedding-0.6B`	0.6B	1024	32k	~1.2 GB	CPU	This guide's default. Instruction-aware (search instruction on the query side); comfortable on CPU. Local provider auto-applies the instruction; OpenAI-compatible servers need `HINDSIGHT_API_EMBEDDINGS_QUERY_PREFIX`.
`Qwen/Qwen3-Embedding-4B`	4B	2560	32k	~8 GB FP16 / ~2.5 GB quant	GPU	Qwen3 quality tier; locks a 2560-dim bank. Not viable when the conversational GPU is already saturated.

How the model is run is the other half of the choice. Hindsight's embedding providers, from simplest to most moving parts:

Provider	How it runs	Best when	Notes
`local`	In-process SentenceTransformers; no sidecar	Single Hindsight host, CPU embedder	Default path. Set `HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL`. No llama.cpp, Ollama, or TEI.
`onnx`	In-process ONNX Runtime	Lighter CPU than Torch; you already have an ONNX export	Needs `hindsight-api-slim[local-onnx]` and `HINDSIGHT_API_EMBEDDINGS_ONNX_*`.
`openai` / `tei`	External OpenAI-compatible or HF TEI endpoint	Embeddings off the API process; shared GPU/CPU host	What Serve and Wire Local Models configures for Ollama / llama.cpp / TEI. Only worth the sidecar if you want isolation or a different machine.

Qwen3-Embedding-0.6B does not need an external embeddings service. On a CPU Hindsight host it loads inside the API process with the local provider — Hugging Face downloads the weights on first start, and the provider applies the model's search instruction itself:

# Compose environment (or equivalent .env keys)
HINDSIGHT_API_EMBEDDINGS_PROVIDER: local
HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL: Qwen/Qwen3-Embedding-0.6B

Use an external OpenAI-compatible endpoint only when you want the embedder on another machine or off the API process. Ollama is one such host:

# Embeddings: Qwen3-Embedding-0.6B on Ollama
HINDSIGHT_API_EMBEDDINGS_PROVIDER: openai
HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL: http://192.168.1.123:11434/v1
HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL: qwen3-embedding:0.6b-q8_0
HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY: dummy

Replace the host, port, and tag with whatever ollama list shows on your inference machine.

HINDSIGHT_API_EMBEDDINGS_QUERY_PREFIX depends on the provider:

Path	`HINDSIGHT_API_EMBEDDINGS_QUERY_PREFIX`	`HINDSIGHT_API_EMBEDDINGS_PASSAGE_PREFIX`
`local` (`Qwen/Qwen3-Embedding-0.6B`)	Leave unset / empty — the in-process provider applies Qwen's search instruction itself	Leave unset / empty
External Ollama / llama.cpp / OpenAI-compatible	Official Qwen3 retrieval instruction (real newline before `Query:`)	Leave empty — stored memories stay raw text
Server already wraps inputs in Qwen's template	Leave both empty — otherwise the instruction is applied twice	Leave empty

For the external path, set:

# Real newline before Query:. Bash $'...' is the reliable way to pass it.
HINDSIGHT_API_EMBEDDINGS_QUERY_PREFIX=$'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:'
HINDSIGHT_API_EMBEDDINGS_PASSAGE_PREFIX=

In Compose YAML the same string needs an explicit \n (or a literal multiline value). Set the prefixes before the first retain: changing them later means old vectors and new queries no longer sit in the same space.

Two properties decide more than the leaderboard position. Instruction sensitivity: Qwen3 embeddings apply a search instruction on the query side; without the matching prefix on an OpenAI-compatible server, the stored side and the query side stop sitting in the same space. Dimension lock: because the width is read once at startup, moving from 0.6B to 4B later is a bank migration, not a config change — and even a same-dimension swap (BGE-M3 ↔ Qwen3-0.6B, or EmbeddingGemma 2 ↔ nomic at 768) changes the vector space, so bank vectors are not recomputed automatically. Plan a fresh bank or a manual re-embed/reindex; see Export, import, and migrating the memory bank. After leaving BGE-small, probe recall_min_scores against the new score distribution — thresholds calibrated on BGE-small can be too tight for Qwen3-0.6B and starve the reranker of candidates. For benchmarks and GGUF/Ollama background on the Qwen3 family, see Qwen3 Embedding and Reranker models on Ollama.

Reranking models

Embeddings recall a candidate set; the reranker scores each query/passage pair and re-orders it before Hindsight injects it. With the local provider, Hindsight loads a SentenceTransformers cross-encoder (or a generative reranker where supported). With an external Cohere-compatible endpoint, it posts the full Cohere-style body to HINDSIGHT_API_RERANKER_COHERE_BASE_URL. The reranker is optional, but once the embedder is fixed it is usually the largest recall-quality lever — and, on CPU, often most of the recall wall-clock. Rows are sorted from lightest to heaviest.

Model	Params	Type	Context	Runs on	Notes
`cross-encoder/ms-marco-MiniLM-L-6-v2`	23M	Cross-encoder	128	CPU	Hindsight default; ships in `hindsight:latest`. Sub-second for ~20 candidates. Keep as named fallback if a heavier BGE model is too slow.
`cross-encoder/ms-marco-MiniLM-L-12-v2`	~33M	Cross-encoder	512	CPU	Same family, higher accuracy; still English-leaning MS MARCO.
`cross-encoder/mmarco-mMiniLMv2-L12-H384-v1`	~33M	Cross-encoder	512	CPU	Hindsight-documented multilingual MiniLM reranker.
`BAAI/bge-reranker-base`	278M	Cross-encoder	512	CPU	Moderate upgrade over MiniLM; en/zh. Discussed often; skipped when v2-m3 already fits the latency budget.
`BAAI/bge-reranker-v2-m3`	568M	Cross-encoder	8k	CPU or GPU	Practical multilingual default on CPU. On a ~500-memory bank, ~2.6–2.8 s per recall at a 20-candidate floor (~140 ms/candidate; ~93–99% of recall time). Candidate caps 20/40/80; lowering the floor from 20 → 10 roughly halves rerank cost.
`Qwen/Qwen3-Reranker-0.6B`	0.6B	Generative (yes-token)	32k	GPU preferred	Instruction-aware, long context. Not a drop-in cross-encoder: needs recent SentenceTransformers generative-reranker support, and llama.cpp `/rerank` scoring has known wrong-score bugs (hindsight-benchmarks#11). Validate before adopting.
`BAAI/bge-reranker-v2-gemma`	2.5B	Cross-encoder	8k	GPU preferred	Strong multilingual quality; excessive for a few-hundred-memory personal bank when the GPU is already full.
`Qwen/Qwen3-Reranker-4B`	4B	Generative	32k	GPU	Higher-quality Qwen candidate; not viable without dedicated VRAM.
`Qwen/Qwen3-Reranker-8B`	8B	Generative	32k	GPU	Family top tier; same generative-scoring caveats, larger still.

Cloud API rerankers (Cohere, ZeroEntropy, Alibaba qwen3-rerank, TypeSafe, and similar) work in Hindsight but send memory text off-box. For a local-only cross-domain layout they are out of scope.

Two operational constraints dominate the pick. VRAM: if the conversational model already saturates the GPU (for example a Qwen3.8-27B Q5 on a 16 GB card with no headroom), every "GPU preferred" row is off the table and the live choice collapses to MiniLM or BGE-v2-m3 on CPU. Candidate count: HINDSIGHT_API_RERANKER_MAX_CANDIDATES (30 in the external-endpoint example below) bounds how many passages the model re-scores; each candidate is a full forward pass, so cut the floor before you jump a model tier. For Qwen3 via llama.cpp, the GGUF must keep the ranking head — an old convert_hf_to_gguf.py that drops it scores every passage near zero — and even a correct conversion still needs a known-good/known-bad passage test until the generative scoring path is confirmed.

Generative LLM tiers

Embedding and reranking models are not substitutes for the generative LLM used to extract and reason over memories. Hindsight supports a default LLM and operation-specific overrides, which can target the same llama.cpp server or independent endpoints.

Hindsight operation	Purpose	Practical model tier	Qwen examples
Retain	Extract facts, entities, decisions, and chronology	4B minimum for tests; 9B preferred	Qwen3.5-4B / Qwen3.5-9B
Recall	Semantic, keyword, graph and temporal retrieval	No generative LLM required	Qwen3-Embedding-0.6B + reranker
Reflect	Answer questions by synthesizing retrieved memories	9B to 27B+	Qwen3.5-9B / Qwen3.8-27B
Consolidation	Reconcile and compress related facts	9B preferred	Qwen3.5-9B / Qwen3.8-27B
Mental model refresh	Update longer-lived synthesized understanding	9B to 35B	Qwen3.5-9B / Qwen3.6-35B-A3B

These are operational recommendations, not official Hindsight minimum parameter counts. A small model may extract a port number correctly but miss that an earlier architectural decision has been superseded. Structured-output reliability and faithful handling of negation matter more than nominal parameter count alone. Hindsight's own temperature defaults are worth leaving alone — 0.1 for retain, 0.0 for consolidation, 0.9 for reflect — and for retain and consolidation, the non-thinking alias is worth serving if your llama.cpp build has one, because the small Qwen3.5 checkpoints spend many tokens in thinking mode.

For a deployment with an existing Qwen3.8-27B llama.cpp endpoint, start by sharing it across generative Hindsight tasks with low concurrency:

HINDSIGHT_API_LLM_PROVIDER=openai
HINDSIGHT_API_LLM_BASE_URL=http://WORKSTATION_IP:8080/v1
HINDSIGHT_API_LLM_MODEL=LocalMegaLLM
HINDSIGHT_API_LLM_API_KEY=dummy
HINDSIGHT_API_LLM_MAX_CONCURRENT=2
HINDSIGHT_API_RETAIN_LLM_MAX_CONCURRENT=1
HINDSIGHT_API_CONSOLIDATION_LLM_MAX_CONCURRENT=1

Later, route routine extraction to an independent 9B model and reserve the 27B for difficult reflection:

HINDSIGHT_API_RETAIN_LLM_PROVIDER=openai
HINDSIGHT_API_RETAIN_LLM_BASE_URL=http://MEMORY_LLM_HOST:8083/v1
HINDSIGHT_API_RETAIN_LLM_MODEL=MemoryLLM
HINDSIGHT_API_RETAIN_LLM_API_KEY=dummy

HINDSIGHT_API_REFLECT_LLM_PROVIDER=openai
HINDSIGHT_API_REFLECT_LLM_BASE_URL=http://WORKSTATION_IP:8080/v1
HINDSIGHT_API_REFLECT_LLM_MODEL=LocalMegaLLM
HINDSIGHT_API_REFLECT_LLM_API_KEY=dummy

The smaller 4B model is economical but needs testing; 9B is a sensible quality/performance compromise. If retain and consolidation share a small model, lower HINDSIGHT_API_CONSOLIDATION_LLM_BATCH_SIZE to 2 — the default batch of 8 is a long prompt for a small context window. Qwen3.6-35B-A3B is attractive for a separate well-provisioned inference server (it is a mixture-of-experts model with about 3 billion active parameters, so budget it as a small active model), while a 27B dense model is usually unnecessary for every routine retain event. Limiting background concurrency protects the Hermes conversational model from memory-maintenance bursts.

Serve and Wire Local Models

If the in-process local embedding and reranker settings in Choose Models are enough, you can skip the external Ollama and llama.cpp subsections and jump to wiring — or leave Compose on the local defaults shown there. External endpoints matter when the embedder or reranker should live on another host.

External embeddings: Ollama or llama.cpp

External embeddings are for when you want Qwen3-Embedding-0.6B (or another model) on a separate host. Two practical options are Ollama, which handles model acquisition and serving with little configuration, and llama.cpp, which gives precise control over GGUF quantization, CPU threads, device placement, and batching. Model background and benchmarks for both Qwen3 models are in Qwen3 Embedding and Reranker models on Ollama.

Qwen3-Embedding-0.6B returns 1024-dimensional vectors, and Hindsight reads that width on startup and sizes the database to match. A bank created with the bundled 384-dimensional BGE model cannot take this model — start from an empty database or volume, and decide the embedding model before the first retain. An embeddings provider switch later requires re-embedding stored records even if both models use the same dimensionality.

Option 1: Ollama for Qwen3-Embedding-0.6B

Install Ollama on the inference host using its official Linux installation documentation, then start the embedding model:

ollama pull qwen3-embedding:0.6b
ollama serve

For remote access, configure Ollama's OLLAMA_HOST=0.0.0.0:11434 in its service environment, restart the service, and firewall it to the Hindsight VM — the procedure is covered in Ollama remote access. A daemon bound only to 127.0.0.1 is invisible to the Hindsight container. First check the model name with ollama list, and test the standard embedding endpoint:

curl http://EMBED_HOST:11434/api/embed \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-embedding:0.6b","input":"A test memory"}'

Hindsight's OpenAI embeddings provider instead calls Ollama's OpenAI-compatible /v1/embeddings endpoint. Verify that endpoint works with your Ollama version before configuring Hindsight. Ollama is straightforward for embeddings; do not assume its API offers a compatible reranking endpoint. For reranking, prefer the in-process local BGE path below, or a separate llama.cpp / TEI endpoint after you have validated scores.

Option 2: llama.cpp for Qwen3-Embedding-0.6B

Download a verified Qwen3-Embedding-0.6B GGUF conversion. Use the GGUF filename you actually downloaded:

#!/bin/sh
# start-embedding.sh - CPU-only embedding server
set -eu
cd "$HOME/prj"
exec ./llama.cpp/llama-server \
  --host 0.0.0.0 --port 8081 \
  --model "$HOME/ggufs/Qwen3-Embedding-0.6B-Q8_0.gguf" \
  --alias qwen3-embedding-0.6b \
  --embedding --pooling last \
  --ctx-size 4096 --parallel 1 \
  --threads 4 --threads-batch 4 \
  --batch-size 512 --ubatch-size 256 \
  --n-gpu-layers 0

Qwen3 embeddings use last-token pooling, unlike the CLS pooling used by some BGE models. A CPU-only llama.cpp build is preferable if you want absolutely no CUDA initialization on a workstation whose GPU is occupied by a large conversational model. Do not enable --reranking on this embedding server: that combination is a known way to get a vector of zeros.

curl -s http://EMBED_HOST:8081/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-embedding-0.6b","input":"A test memory"}' \
  | jq '.data[0].embedding | length'

A full-dimension Qwen3-Embedding-0.6B output should contain 1024 floats. Check that the GGUF conversion and your Hindsight query/document instruction handling are compatible; instruction-sensitive embedding models can lose accuracy when prompted incorrectly.

Reranker: local BGE or Qwen3 via llama.cpp

Reranking is different from embeddings: the model scores each query/document pair and returns a relevance score. On a CPU-only Hindsight host — or whenever the conversational GPU has no VRAM headroom — the practical path is Hindsight's local provider with BAAI/bge-reranker-v2-m3 (or the lighter MiniLM default). No sidecar is required:

HINDSIGHT_API_RERANKER_PROVIDER=local
HINDSIGHT_API_RERANKER_LOCAL_MODEL=BAAI/bge-reranker-v2-m3
HINDSIGHT_API_RERANKER_MAX_CANDIDATES=20

Qwen3-Reranker-0.6B is a generative yes-token scorer, not a classic cross-encoder. It can be served through llama.cpp with a GGUF conversion designed for ranking, but treat that path as experimental until you confirm scores on a known-good/known-bad pair: an old convert_hf_to_gguf.py that drops the ranking head scores every passage near zero, and llama.cpp's native /rerank path has known wrong-score reports for this family.

#!/bin/sh
# start-reranker.sh - GPU or CPU as selected by --n-gpu-layers
set -eu
cd "$HOME/prj"
exec ./llama.cpp/llama-server \
  --host 0.0.0.0 --port 8082 \
  --model "$HOME/ggufs/Qwen3-Reranker-0.6B-Q8_0.gguf" \
  --alias qwen3-reranker-0.6b \
  --embedding --reranking --pooling rank \
  --ctx-size 4096 --parallel 1 \
  --batch-size 512 --ubatch-size 256 \
  --n-gpu-layers 99

Set --n-gpu-layers 0 to run it on CPU. The GPU version needs additional VRAM for compute buffers and context, not just model weights; avoid assuming that the file size is the total GPU allocation.

curl -s http://RERANK_HOST:8082/v1/rerank \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-reranker-0.6b", "query":"PostgreSQL storage", "documents":["Hindsight stores memories in PostgreSQL.","A recipe for bread."], "top_n":2}' \
  | jq

The PostgreSQL document should rank above the unrelated document; all-zero relevance scores mean the GGUF is the wrong conversion, and recall will look random even when the facts are stored. Check the exact results[].index and results[].relevance_score shape: Hindsight expects a Cohere-compatible rerank response, and llama.cpp compatibility can vary across builds.

Wire endpoints into Hindsight

Add the following environment settings to the Hindsight Compose service, or store them in its .env file. Pick one embeddings path — in-process local, Ollama, or llama.cpp — never two at once.

# Option A: in-process Qwen3 on the Hindsight host (CPU; no sidecar)
HINDSIGHT_API_EMBEDDINGS_PROVIDER=local
HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL=Qwen/Qwen3-Embedding-0.6B

# Option B: llama.cpp embedding service
# HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
# HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL=http://EMBED_HOST:8081/v1
# HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=qwen3-embedding-0.6b
# HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=dummy
# HINDSIGHT_API_EMBEDDINGS_OPENAI_BATCH_SIZE=8
# HINDSIGHT_API_EMBEDDINGS_QUERY_PREFIX=$'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:'
# HINDSIGHT_API_EMBEDDINGS_PASSAGE_PREFIX=

# Option C: Ollama on another host
# HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
# HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL=http://192.168.1.40:11434/v1
# HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=qwen3-embedding:0.6b-q8_0
# HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=dummy
# HINDSIGHT_API_EMBEDDINGS_QUERY_PREFIX=$'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:'
# HINDSIGHT_API_EMBEDDINGS_PASSAGE_PREFIX=

# Option: local BGE reranker (no sidecar; preferred on CPU)
HINDSIGHT_API_RERANKER_PROVIDER=local
HINDSIGHT_API_RERANKER_LOCAL_MODEL=BAAI/bge-reranker-v2-m3
HINDSIGHT_API_RERANKER_MAX_CANDIDATES=20

# Option: llama.cpp Cohere-compatible rerank endpoint (validate Qwen3 scores first)
# HINDSIGHT_API_RERANKER_PROVIDER=cohere
# HINDSIGHT_API_RERANKER_COHERE_BASE_URL=http://RERANK_HOST:8082/v1/rerank
# HINDSIGHT_API_RERANKER_COHERE_MODEL=qwen3-reranker-0.6b
# HINDSIGHT_API_RERANKER_COHERE_API_KEY=dummy
# HINDSIGHT_API_RERANKER_MAX_CANDIDATES=30

The variable names are provider-specific, and a near-miss is silent: HINDSIGHT_API_EMBEDDINGS_BASE_URL is the wrong key, and Hindsight quietly aims at OpenAI's own embedding endpoint instead of your local server. Use HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL. With the Cohere-compatible reranker path, Hindsight posts the full Cohere-style body to HINDSIGHT_API_RERANKER_COHERE_BASE_URL; if the container log shows a 404 for /v1/rerank/rerank, your build appends /rerank on its own and the value should stop at /v1.

For Options B and C, use the HINDSIGHT_API_EMBEDDINGS_QUERY_PREFIX value from Embedding models (Option A leaves it unset). Test query-to-passage retrieval on your actual memories after the first retain.

Control What Gets Remembered

With the stack running, the next failure mode is usually the write path: Hermes defaults to auto_retain: true, so every conversation turn can be extracted into the bank. Incorrect guesses, discarded plans, and exploratory statements become searchable facts, and later recall treats them as evidence — the loop described in self-reinforcing memory loops. Capture policy for Hindsight among other providers is summarized in agent memory providers compared.

Capture modes that exist today

Mode	Hermes / Hindsight settings	What you get
Continuous	`auto_retain: true`, `retain_every_n_turns: 1`	Every turn is retained and extracted
Batched auto	`auto_retain: true`, raise `retain_every_n_turns`	Fewer, larger retain payloads — still automatic, not a review queue
Explicit-only	`auto_retain: false` + tool `hindsight_retain`	Storage only when the agent calls the tool (for example after you say "remember this")
Session synthesis	End-of-session `hindsight_retain` of a summary, or bank reflect / consolidation	A curated write if you or the agent discipline the call — not a built-in flush-for-approval

Setting auto_retain to false disables automatic storage. It does not add a human approval dialog before each write. The agent can still call hindsight_retain whenever its tools allow; treat explicit retain as deliberate capture, not a pending queue.

The approval-queue gap

A common wish is a pending queue of candidate memories that you review and approve before they enter the bank. Hindsight under Hermes does not provide that contract today. Mnemosyne for Hermes implements provider-specific staged writes in some versions; that path is not a shared Hermes standard and does not apply to the Hindsight plugin. Until an upstream approval flow exists, the workable pattern is auto_retain: false, deliberate hindsight_retain, and inspection or deletion in the Hindsight control-plane UI before the next session relies on a bad fact. There is no missing write_approval key for Hindsight to hunt for in config.json.

Bank steering (missions, labels, directives, mental models)

People often call the next layer "finetuning." In Hindsight it is mostly bank configuration, not Hermes config.json alone: missions and extraction settings steer what retain keeps; labels structure entities; directives and mental models shape later synthesis.

Control	Where it lives	How it helps
Retain mission (`retain_mission` / Hermes `bank_retain_mission`)	Bank config	Plain-language "pay attention to X, ignore Y" injected into extraction
Extraction mode (`concise`, `verbose`, `custom`, …)	Bank config	`concise` (default) keeps long-term-worthy facts; `custom` replaces built-in rules via `retain_custom_instructions`
Entity labels	Bank config	Controlled `key:value` vocabulary at retain time → graph links and optional tags; improves retrieval structure, does not by itself reject wrong statements
Directives ("hard rules")	Bank directives API	Explicit instructions enforced during reflect (tag-scoped); constrain synthesis, not a write-approval gate
Mental models	Bank mental-model API	Standing answers to recurring questions, refreshed via reflect from matching memories; derived, so bad source facts still poison them
Reflect / observations mission	Bank config	Steers synthesis and consolidation, not the raw retain firehose

Missions, concise extraction, and labels reduce noise and improve retrieval. They do not replace turning off auto-retain when accuracy matters more than completeness. Prefer a short retain mission that names durable project decisions and ignores chat logistics; add entity labels only when you have a real taxonomy; add mental models for two or three recurring questions after the bank already holds clean facts.

Recommended hygiene profile

A concrete starting point for a personal Hermes bank:

{
  "mode": "local_external",
  "api_url": "http://HINDSIGHT_VM_IP:8888",
  "bank_id": "hermes",
  "memory_mode": "hybrid",
  "auto_recall": true,
  "recall_sync": false,
  "auto_retain": false,
  "retain_async": true,
  "recall_prefetch_method": "recall"
}

On the bank, set a retain mission and leave extraction in concise mode (via the control plane or Banks API). After an explicit retain, open the UI, confirm the observation, and delete mistakes before consolidation or mental-model refresh runs. Leave recall_prefetch_method at recall — reflect spends a chat-model call before every turn and injects synthesis rather than underlying facts.

Connect Hermes Agent to Hindsight

On the Hermes VM, install the catalog plugin, then run the setup wizard. Select Hindsight and the local_external mode; the plugin does not need to start embedded PostgreSQL or run the Hindsight model stack locally, and local_embedded would start a second Hindsight daemon beside the container you just ran.

hermes plugins install hindsight
hermes memory setup hindsight

There will be Interesting set of options of agent role using hindsight

hermes memory status

The external Hindsight connection lives in ~/.hermes/hindsight/config.json. An illustrative configuration that keeps continuous capture (Hermes defaults) is:

{
  "mode": "local_external",
  "api_url": "http://HINDSIGHT_VM_IP:8888",
  "bank_id": "hermes",
  "memory_mode": "hybrid",
  "auto_recall": true,
  "recall_sync": false,
  "auto_retain": true,
  "retain_async": true,
  "retain_every_n_turns": 1
}

For explicit-only retain and bank steering, use the hygiene profile in Control What Gets Remembered instead of leaving auto_retain on. recall_sync: false prioritizes interactive responsiveness but may inject recalled context on the following turn. Use synchronous recall when the current request must incorporate the latest memories before the model starts answering. retain_async: true avoids blocking ordinary responses while memories are processed.

The plugin's recall default is observations only; raw world and experience facts stay in the bank and out of the prompt until consolidation has written an observation, or until you widen recall_types to observation,world,experience. Widening it for the first week and watching the control plane, then narrowing, is a reasonable order.

Built-in Hermes memory stays active next to the provider; keeping MEMORY.md and USER.md on while you learn what Hindsight adds is fine, and the memory-system article is the place to decide whether the flat files should stay.

Restart the gateway if it is already running:

hermes gateway restart
hermes memory status

The status should identify Hindsight as the active provider. For a wider command reference, including Hermes' TUI and plugin commands, see the Hermes Agent CLI cheatsheet.

Validate Memory, Not Just Installation

Ask Hermes to remember a disposable fact about a test project, start another session, and ask for that fact again. Retention is asynchronous by default and extraction waits on the chat model, so give the main llama.cpp server a moment before judging. Then test a harder case: tell Hermes that an earlier decision has changed and ask which decision is current and why. Check Hindsight's UI and stored observations rather than assuming memory status proves ingestion and recall are correct.

If the answer comes back empty, inspect the control plane before changing models. A retain can succeed and still be invisible to recall while recall_types is observations only and consolidation has not run — widening recall_types is the first check. A connection refused in the embedding server log means the container cannot reach the embedding port, and all-zero reranker scores mean the GGUF is the wrong conversion.

Write-path choices (auto_retain, missions, approval gap) are covered in Control What Gets Remembered.

Troubleshooting and Operational Notes

Symptom	Likely cause	First check
Hermes says provider unavailable	Wrong `local_external` URL or network rule	`curl http://HINDSIGHT_VM_IP:8888/health`
Docker starts, PostgreSQL errors	Missing extension, wrong credentials, or insufficient shared memory	Container logs and `pg_extension`
Embeddings return 401 or 404	Incorrect OpenAI-compatible URL, wrong env key name, or model alias mismatch	Direct `/v1/embeddings` test and variable names
Reranking fails	Endpoint or response format mismatch, including doubled `/rerank` path	Direct `/v1/rerank` test and container log
Recall returns nothing usable	`recall_types` limited to observations, or zero reranker scores	Control plane, then `recall_types`, then reranker conversion
Existing bank reports dimension mismatch	Embedding model changed	Rebuild vectors/reimport with a consistent model
Hermes responses slow down	Synchronous recall or shared LLM contention	Turn off sync recall; cap retain concurrency
Irrelevant memories repeat	Weak extraction, noisy retrieval, or insufficient consolidation	Inspect observations and relevance scores

A practical small deployment is 2-4 vCPUs and 4-6 GB RAM for the Hindsight API VM when PostgreSQL and all inference are external. Disk allocation still depends on the chosen Docker image and retained image layers; allocate roughly 30-40 GB initially with room to expand. Size the PostgreSQL host based on actual memory volume, storage performance, and the other databases it serves.

Back up and restore the Hindsight bank

The memory bank is the only state that matters, and it lives in PostgreSQL. In Option B that is a database you administer; in Option A it is the embedded pg0 named volume inside the container. Back it up independently of the Hindsight container — recreating the container without first capturing the volume or the database discards the bank, and hermes backup does not reach a bank running in a separate service.

Option B — external PostgreSQL. Dump the hindsight database in custom format, then restore it into a scratch database to verify the archive before you trust it:

pg_dump -h POSTGRES_IP -U hindsight -Fc hindsight > hindsight-bank.dump

# restore into a scratch database to prove the dump is usable
createdb -h POSTGRES_IP -U postgres hindsight-verify
pg_restore -h POSTGRES_IP -U postgres -d hindsight-verify hindsight-bank.dump

Confirm the pgvector extension and a sample recall exist in the restored copy before you delete the original dump target. Keep the dump on a different host than the Hindsight VM, and run this as a PostgreSQL admin (a role that can create databases) rather than the application role.

Option A — all-in-one container. The embedded database is a named volume, so stop the container and tar the volume through a throwaway image:

docker stop hindsight
docker run --rm -v hindsight-pg0:/data -v "$PWD":/backup alpine \
  tar czf /backup/hindsight-bank.tgz -C /data .
docker start hindsight

The mount point inside the tar container (/data) is arbitrary; it simply aliases the named volume so tar can read it. To restore, stop the container, clear the volume, and untar the archive back into it, or point the container at the external PostgreSQL restore from Option B.

Take a snapshot at three moments rather than on a calendar schedule: before changing the embedding model, before a major Hindsight or PostgreSQL upgrade, and before a reflect or consolidation pass that rewrites the bank. A backup you have restored into a scratch bank is worth more than one you have only written.

Export, import, and migrating the memory bank

Backup and restore protect the bank in place. Export/import moves it between Hindsight instances, and that is what a model change actually forces you to do.

Changing the embedding model is not a configuration edit. The vectors in the bank were computed with the old model, and a different model — for example the bundled 384-dimensional BGE embedder versus 1024-dimensional Qwen3-Embedding-0.6B — writes into a different space and usually a different width. You cannot re-point the existing instance at the new embedder and keep the old vectors; the dimensions must be recalculated by re-embedding every memory with the new embedding model.

Because the embedder and its configuration are set per Hindsight instance, the clean path is to stand up a fresh instance with the new embedding configuration and migrate the bank into it, rather than rebuilding vectors inside the one that is already serving recall. The Hindsight API can move the bank too, but for a one-off change export/import is the easier and more inspectable route:

# export the bank from the old instance
docker exec hindsight \
  hindsight-admin export-bank \
    --bank hermes2 \
    --output /tmp/hermes2.zip

docker cp hindsight:/tmp/hermes2.zip ./hermes2.zip

ls -lh hermes2.zip
unzip -t hermes2.zip

# import the bank into the new instance
docker cp ./hermes2.zip hindsight-q06:/tmp/hermes2.zip

docker exec hindsight-q06 \
  hindsight-admin import-bank \
    --archive /tmp/hermes2.zip

unzip -t verifies the archive before you ship it to the new instance. After the import, re-point Hermes at the new instance's api_url in ~/.hermes/hindsight/config.json, restart the gateway, and run the memory validation from earlier: a retained fact should recall through the new embedder. Leave the old bank and its vectors in the old instance until the new one behaves, so you still have a working fallback.

Monitor, prune, and keep the bank stable

With the bank running, the day-to-day job is watching what the extraction model has decided to store and keeping derived state in check. Use the control plane UI (hindsight-embed -p hermes ui start) to inspect observations, relevance scores, and consolidation output rather than assuming memory status proves ingestion and recall are correct. Duplicated or stale memories are usually a weak extraction or an under-run consolidation, so fix the extraction tier or consolidation cadence before widening retrieval.

Two maintenance traps are worth testing before you rely on the bank. First, deleting an observation does not guarantee every derived representation disappears with it — a reflect fact or graph edge built from it can survive the delete, so verify deletion end to end, the same provenance concern covered in self-reinforcing memory loops. Second, keep HINDSIGHT_API_WORKER_ID stable across recreations and pin a specific image tag: a task that was mid-flight when the container died stays stuck under the old worker id, and a moving tag is a poor match for a database you intend to keep.

References

Upgrading to Go 1.27: What Breaks, What to Adopt First

Rost — Fri, 16 Oct 2026 09:33:44 +0000

Go 1.27 shipped in August 2026, and for most modules the upgrade is a single go.mod line followed by a go fix pass — but a handful of changes bite silently. Here is what actually breaks and what to adopt first.

The headline changes are structural: encoding/json is now backed by the new json/v2 implementation under the hood, the uuid package joined the standard library, and methods can finally declare their own type parameters. None of these force a rewrite, and the Go team kept the usual compatibility promise — almost everything compiles and runs unchanged.

Not everything is quiet, though. Several GODEBUG settings now fail the build at their old values, compress/flate produces different bytes, go test enforces a new stdversion check, and compiler-generated symbol names changed. If you need a syntax refresher on the parts that did not move, the Go cheatsheet stays valid; this guide covers the rest, with commands and fixes for each item.

The upgrade path in one diagram

The Go team recommends a build-first, fix-second order. go fix rewrites code to new APIs and idioms, but only a clean build tells you whether any of the runtime-level changes hit you.

flowchart TD
    A[Install Go 1.27 toolchain] --> B["go build ./..."]
    B --> C{Build passes?}
    C -- "No: removed GODEBUG value" --> D[Set final default or delete the line]
    D --> B
    C -- Yes --> E["go test ./..."]
    E --> F{"stdversion + vet clean?"}
    F -- "No: symbol too new" --> G[Raise go line or swap the symbol]
    G --> E
    F -- Yes --> H["go mod tidy"]
    H --> I["go fix ./..."]
    I --> J[Review JSON golden tests and diffs]

Start with the toolchain itself. If you manage toolchains through the go command, either pin the version in go.mod or install explicitly:

go install golang.org/dl/go1.27@latest
go1.27 download
go1.27 version

Alternatively, set GOTOOLCHAIN=go1.27.0 in the environment, or add toolchain go1.27.0 to go.mod and let the go command fetch it. The module-focused companion on Go workspace structure explains how the go and toolchain lines interact across multiple modules; the short version for a single module is below.

Step 1: bump the `go` line and clear removed GODEBUG settings

go mod edit -go=1.27
go mod tidy
go build ./...

Raising the go directive turns on the new language features (generic methods, promoted-field keys in struct literals, wider function type inference) and makes the go command strict about GODEBUG settings whose support has been removed. Go 1.27 removed these permanently:

asynctimerchan — timer and ticker channels are now always unbuffered
gotypesalias
tlsunsafeekm, tlsrsakex, tls3des, tls10server
x509keypairleaf

The new rule: the go command still accepts these settings in go.mod (godebug entries) or in source (//go:debug comments) if they are set to the final default value that existed before removal. An old value — the one that disabled a behaviour change — fails the build:

//go:debug asynctimerchan=0  // build error: old value of a removed setting
//go:debug asynctimerchan=1  // accepted, but delete the line instead

If your build fails here, the code depended on pre-Go-1.23 buffered timer channels. The fix is in the code, not the config: check Timer.Stop and Timer.Reset usage against the context, cancellation, and timeout patterns that rely on the synchronous channels. The same search applies to the TLS-related settings — anything pinning tls3des=1 or tls10server=1 was keeping old cipher suites alive, and Go 1.27 refuses to compile it.

Step 2: run `go test` — the stdversion trap

go build and go test no longer agree. Go 1.27's go test runs the stdversion vet check by default, which reports standard-library symbols that are too new for the go version in force in the referring file.

// go.mod still says "go 1.26"
func Base(p string) string {
    _, file, ok := strings.CutLast(p, "/") // added in Go 1.27
    if !ok {
        return p
    }
    return file
}

With a 1.27 toolchain, go build succeeds — the compiler has strings.CutLast — while go test fails the vet step because go.mod still declares 1.26. With a real 1.26 toolchain the file would not build at all. The lesson is procedural: never validate an upgrade with go build alone; run the full test suite, then bump the go line. The vet suite in go test also picked up a new printf finding: fmt.Errorf("...: %w", p) where p is a *E and E implements error now warns, because wrapping a pointer produces type assertions and errors.Is checks that fail. That failure mode shows up in error-handling code exactly as described in the Go error handling architecture patterns — check the operand type, not just the verb.

encoding/json: the v2 engine underneath

The biggest under-the-hood change: encoding/json/v2 (experimental since Go 1.25 as GOEXPERIMENT=jsonv2) graduated, and the classic encoding/json package is now backed by the v2 implementation. For v1 callers, behaviour is preserved — nil slices still encode as null, map keys still sort — with one visible difference: the exact text of some error messages changed. If tests or monitoring match on error strings, expect diffs.

Three things can still go wrong:

Error text. Any strings.Contains(err.Error(), ...) assertions keyed to the old v1 wording need updating.
Performance-sensitive paths may shift. Unmarshalling is faster, but the rewrite changes which operations allocate; benchmarks flagged in CI will show it.
Compatibility escape hatch. GOEXPERIMENT=nojsonv2 restores the original v1 implementation at build time. It is temporary — plan to remove it, not to ship with it.

A second-order effect for golden tests: compress/flate's encoder changed in the same release, so archive/zip, compress/gzip, compress/zlib, and image/png outputs may differ byte-for-byte from Go 1.26 even though they decode identically. Regenerate compressed fixtures rather than pinning old bytes.

Migrating to the v2 API deliberately

The v2 API is a separate import (encoding/json/v2) with Marshal, Unmarshal, and an options system. The supported migration path is to make v2 calls behave like v1 first, then tighten selectively:

import (
    "encoding/json/v2"
    "encoding/json/jsontext"
)

// Step 1: identical behaviour to encoding/json — a safe, mechanical change
out, err := jsonv2.Marshal(v, jsonv2.DefaultOptionsV1())

// Step 2: opt in to one v2 default at a time
out, err := jsonv2.Marshal(v,
    jsonv2.DefaultOptionsV1(),
    jsonv2.FormatNilSliceAsNull(false)) // nil slices now encode as []

Once you drop DefaultOptionsV1, v2 defaults apply, and they are stricter and more interoperable than v1:

Behaviour	v1 (`encoding/json`)	v2 default
Invalid UTF-8 in strings	Replaced with U+FFFD	Error
Duplicate names in an object	Allowed	Error
Nil slice / nil map	`null`	`[]` / `{}`
Struct field matching	Case-insensitive fallback	Case-sensitive
Unknown object members	Ignored	Ignored (`RejectUnknownMembers` to error)
Map key ordering	Always sorted	Unsorted by default
`time.Duration`	Built-in string encoding	No default encoding

The unsorted map keys are the one that surprises golden-file tests: v2 skips the sort because sorting costs time. When you need stable output — fixtures, snapshot tests, hashed representations — pass json.Deterministic(true), which restores sorted output for maps and structs. The ORM comparison article's models all carry json tags; those tags keep working in v2, but the format tag option was removed during the experiment and inline was renamed to embed, so grep your model definitions for both before migrating. For low-level needs — streaming, token-by-token processing, custom syntax handling — the companion encoding/json/jsontext package exposes an Encoder/Decoder state machine over JSON text.

stdlib `uuid`: dropping `github.com/google/uuid`

Go 1.27 adds uuid to the standard library, implementing RFC 9562. The API surface covers the common cases:

id := uuid.New()    // currently equivalent to NewV4
v7 := uuid.NewV7()  // time-sortable, 48-bit timestamp + random
u, err := uuid.Parse("550e8400-e29b-41d4-a716-446655440000")

github.com/google/uuid is the second most-depended-on Go module, so this migration matters at scale. The two APIs are close — types are [16]byte values comparable with == and usable as map keys, String(), MarshalText/UnmarshalText round-trip — and the upstream project is preparing go:fix inline shims so go fix can rewrite uuid.NewV4() calls to the standard package and delete the dependency.

The interesting adoption decision is NewV7. Version-7 UUIDs embed a millisecond timestamp in the most significant bits, so values sort in creation order (except when the system clock moves backwards). For database primary keys that means better B-tree locality than random UUIDv4, and created_at-like ordering for free in the identifier itself — the same property that drives time-sortable key discussion in multi-tenant data models.

Language changes: generic methods and friends

Generic methods are the release's most requested language feature: a method declaration may now declare its own type parameters.

// Go 1.27: math/rand/v2 gained a generic method alongside the function
func (r *Rand) N[Int intType](n Int) Int

Two restrictions keep the type system decidable: interface methods may not declare type parameters, and interface methods cannot be implemented by generic methods. Alongside them, struct literal keys may now be any valid field selector (not just top-level field names), and function type inference generalizes to all contexts where a generic function is assigned to a matching function-type variable. None of these break existing code; they remove workarounds — package-level generic helper functions and explicit instantiations.

go fix grew modernizers for adjacent cleanups: atomictypes, embedlit, slicesbackward, and unsafefuncs. One caution from the hands-on upgrade labs: run go fix -fix=atomictypes deliberately rather than in bulk on libraries, since it can change the types of package-level variables that other packages reference. The waitgroup modernizer was renamed to waitgroupgo; note that sync.WaitGroup.Go (from 1.25) already covers most of what it detects — see the bounded concurrency patterns for how it slots into worker-pool shutdown.

Things that quietly break

These are the changes that compile fine and still cost you a release day:

Function-literal symbol names. The compiler now generates simpler, inlining-independent names for closures, and may merge identical function literals. Tests that assert on symbol names in tracebacks or profiles will fail; code that compares function code pointers for equality is exposed in more cases.
gofmt alignment. A fix to column-alignment logic (it previously varied across CPU architectures) reflows a small number of already-formatted files. Expect whitespace-only diffs in a gofmt ./... commit after the upgrade.
go mod tidy rewrites go.mod. For modules on go 1.27, tidy now merges duplicate require blocks into at most two blocks (direct, indirect), preserving comments. Diff your go.mod and resolve any merge-conflict residue before the release commit, not after.
Traceback labels. Tracebacks for modules on go 1.27 include runtime/pprof goroutine labels in the header line; disable with GODEBUG=tracebacklabels=0 if labels carry sensitive data.
go tool trace binds to localhost. -http=:6060 now listens on localhost only; pass -http=0.0.0.0:6060 explicitly when you need remote access (and mind that a profiling port on a shared host is a risk — see the production profiling workflow for the hardened setup).

Experiments worth watching

Two experimental packages shipped in 1.27, both enabled with GOEXPERIMENT=simd at build time:

simd — a portable, vector-size-agnostic API (Int8s, Float32s, ...) that maps to hardware instructions where available. It supports AVX/AVX2/AVX-512 on amd64, NEON on arm64, and WebAssembly SIMD. The standard library already uses this machinery internally: Go's Swiss Table map implementation benefits from it.
simd/archsimd — the architecture-specific layer for code that needs to know the vector width.

On the crypto side, crypto/mldsa implements the post-quantum ML-DSA signature scheme from FIPS 204 in three parameter sets (MLDSA44, MLDSA65, MLDSA87), integrated into crypto/x509 and offered as TLS 1.3 signature schemes. Hybrid post-quantum key exchange can now be forced via Config.CurvePreferences even when the tlsmlkem/tlssecpmlkem GODEBUG fallbacks are set. Neither is a migration item today, but both shape the TLS configuration surface your services will grow into.

New APIs worth adopting immediately

A few additions solve everyday friction and cost nothing to adopt:

// strings.CutLast / bytes.CutLast — the LastIndex dance, replaced
before, after, found := strings.CutLast("a/b/c.tar.gz", ".")

// httptest.NewTestServer — in-memory fake network, auto-cleanup
srv := httptest.NewTestServer(t, mux) // no real port, no defer srv.Close()

httptest.NewTestServer creates an httptest.Server backed by an in-memory network instead of a TCP listener and registers its own t.Cleanup. Its real payoff is pairing with testing/synctest: HTTP round-trips run in synthetic time, making timeout and retry tests fast and deterministic — the combination is demonstrated in the synctest concurrency testing article.

Upgrade checklist

The order matters more than the items; each step assumes the previous one is green.

Step	Small module	Service / monorepo
1. Toolchain	`go install golang.org/dl/go1.27@latest`	Pin `toolchain go1.27.0`; align CI images
2. `go.mod`	`go mod edit -go=1.27 && go mod tidy`	Same, plus resolve `require`-block merge diffs
3. Removed GODEBUG	Grep `//go:debug` and `godebug:`	Also audit deployed configs and flags
4. `go build ./...`	Fix compile errors	Fix per-module; bisect failures
5. `go test ./...`	Fix `stdversion`/`printf` findings	Run before the `go` bump to catch symbol drift
6. `go fix ./...`	Review the diff	One modernizer at a time
7. JSON/golden tests	Regenerate fixtures	Check `flate`-based fixtures (zip/gzip/png) too
8. `GOEXPERIMENT=nojsonv2`	Skip	Temporary only; file issues for what forced it

Closing notes

Go 1.27's breaks cluster around edges most programs never touch — removed GODEBUG opt-outs, symbol-name assertions, compressed-byte fixtures — while its wins land where every program lives: faster JSON, standard UUIDs, generic methods. A clean upgrade is mostly discipline: go build, then go test, then go fix, in that order, with the go line raised before the test run rather than after it. The nojsonv2 escape hatch and the asynctimerchan acceptance-at-final-default rule buy time, but both are scheduled to disappear, so treat this upgrade as the deadline for the deferred fixes it inherits from 1.23–1.26.

Bounding Concurrency in Go: errgroup, Worker Pools, and Backpressure

Rost — Sat, 10 Oct 2026 07:53:44 +0000

Unbounded fan-out is the default failure mode of Go concurrency: every go statement is one more task in flight, with no ceiling and no owner. This article is about giving both.

Cancellation and bounding are separate controls. Cancellation decides when work stops; bounding decides how much of it exists at once. A service can cancel perfectly and still fall over because ten thousand goroutines each decided, independently, that now was a good time to call the database.

The pages around this one cover the other half of the problem. Go context.Context Done Right treats context as the control plane for stopping work; here the subject is the capacity plane — errgroup.SetLimit, semaphore channels, worker pools, and bounded queues — plus the one leak shape that survives errgroup.Wait even when every cancel path looks wired up.

Five primitives, one decision table

Primitive	Partial results	Fail-fast	Backpressure	Dynamic work	Best for
`sync.WaitGroup` + `errors.Join`	yes	no	no	no	wait-all batches where every result counts
`errgroup.WithContext`	no	yes	no	no	fan-out where the first error ends the request
`errgroup.SetLimit(n)`	no	yes	yes (blocks `Go`)	no	bounded fail-fast fan-out
Semaphore channel	yes	manual	yes (blocks acquire)	yes	custom admission control
Worker pool	yes	manual	yes (queue fills)	yes	steady streams, long-lived workers

Two questions pick the row. First: when one task fails, do you still want the others' results? If yes, fail-fast group semantics discard work you needed. Second: does work arrive continuously, or is it a fixed batch you can count before launching? Groups handle batches; pools and queues handle streams.

errors.Join is what makes the plain WaitGroup row viable — it collects every worker's error instead of keeping only the first, which pairs with the boundary-translation rules in Go Error Handling Architecture.

The leak that survives errgroup.Wait

The best-known errgroup failure mode has nothing to do with bounding. It is a cancellation gap, and it hides behind code that looks correct:

g, gctx := errgroup.WithContext(ctx)
results := make(chan int) // unbuffered

go func() { // producer — nobody owns this goroutine
    for i := 0; i < 100; i++ {
        results <- i
    }
    close(results)
}()

g.Go(func() error { // consumer — group member
    for r := range results {
        if r == 5 {
            return fmt.Errorf("save failed")
        }
    }
    return nil
})

err := g.Wait() // returns at the consumer's error

When the consumer returns, gctx cancels and Wait returns. The producer is parked on results <- i — and context cancellation does not unblock a channel send. Nobody owns that goroutine, so it stays parked for the life of the process. Run this shape in a loop and the leak is exactly linear:

Variant	Live goroutines after 500 iterations
Send without a `ctx.Done` arm	501 (+500 leaked, one per iteration)
Send wrapped in `select` with `<-gctx.Done()`	502 (+1 baseline)

// the fix: every channel operation inside a cancellable task
// gets a Done arm
select {
case results <- i:
case <-gctx.Done():
    return
}

The same gap exists on the receive side. The rule is mechanical: every channel operation inside code that can be cancelled gets a <-ctx.Done() arm, and every goroutine has an owner that waits on it or cancels it. goleak in CI catches the cases review misses — it slots into the same automated quality gate as the tools in Go Linters: Essential Tools for Code Quality.

Two related shapes are worth naming. If the producer is a group member, the same code produces not a leak but a deadlock — Wait blocks forever waiting for the parked send, which at least fails loudly. And a producer that sends with select but receives on a channel whose consumer has quit leaks the same way; Done arms belong on both sides.

errgroup.SetLimit: bounded fail-fast fan-out

SetLimit turns an errgroup into an admission-controlled group. Each Go call beyond the limit blocks until a slot frees:

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(8)
for _, item := range items {
    g.Go(func() error {
        return process(ctx, item)
    })
}
err := g.Wait()

Three semantics matter in production. Go blocks, so the loop above applies backpressure to the producer — that is usually what you want, but it means the caller's goroutine can stall, so a request handler feeding a large batch needs its own timeout budget around the whole loop. SetLimit(0) blocks every Go call forever. And changing the limit while goroutines are active panics — the limit is fixed for the life of the group.

SetLimit inherits group semantics: the first error cancels the context and Wait returns the first error, discarding results of work still in flight. It is the right primitive when failing the whole operation on first failure is correct — fetching one page of data, calling a set of independent health checks, fanning a request out to shards.

Semaphore channels: backpressure as a feature

A buffered channel of capacity n is a semaphore with no dependencies:

sem := make(chan struct{}, 8)
var wg sync.WaitGroup
for _, item := range items {
    wg.Add(1)
    sem <- struct{}{} // blocks when 8 tasks are in flight
    go func() {
        defer wg.Done()
        defer func() { <-sem }()
        _ = process(ctx, item)
    }()
}
wg.Wait()

The acquire happens before the go statement, so goroutines are only ever created when a slot exists. The buffer size is the contract: it is simultaneously the concurrency ceiling and the queue of pending admissions.

The primitive earns its place when you need admission control the groups do not express — context-aware acquisition, per-downstream pools, or partial results on failure:

select {
case sem <- struct{}{}: // slot acquired
case <-ctx.Done():      // caller gave up before getting in
    return ctx.Err()
}

Measured on the same workload, the semaphore channel and SetLimit(8) finish within 1.5% of each other — the choice between them is semantics, not performance.

Worker pools: for streams, not batches

A fixed pool of long-lived workers separates admission (the queue) from execution (the workers):

func pool(ctx context.Context, workers int) chan<- func() {
    jobs := make(chan func())
    var wg sync.WaitGroup
    for i := 0; i < workers; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for {
                select {
                case f, ok := <-jobs:
                    if !ok {
                        return
                    }
                    f()
                case <-ctx.Done():
                    return
                }
            }
        }()
    }
    return jobs
}

Pools fit work that arrives continuously — queue consumers, polling loops, request workers — where a per-batch group would be created and torn down endlessly. Since Go 1.25, sync.WaitGroup.Go removes the Add/Done pairing boilerplate for the common shape. The shutdown contract is the part to get right: closing jobs terminates workers after the queue drains; cancelling ctx abandons queued work. Both are legitimate, but only one can be the documented behavior.

Bounded queues: block or drop

A buffered channel is also a queue with a ceiling. Once the buffer fills, the producer blocks — and that moment is the design decision. Blocking propagates pressure upstream to whoever feeds the queue; dropping sheds load at the cost of lost work. A monitoring pipeline can drop; an order pipeline must block or spill to durable storage.

select {
case queue <- event: // admitted
default:            // queue full: drop policy lives here
    dropped.Add(1)
}

Whatever the policy, make it explicit and measured. A silent default branch is where events go to disappear.

What bounding costs — measured

The same workload, 10,000 tasks of 5 ms simulated I/O each, Go 1.27.1:

Strategy	Peak goroutines	Wall time
Unbounded `go` per task	~10,000 launched	18 ms
`errgroup.SetLimit(8)`	8	6.9 s
Semaphore channel (cap 8)	8	6.6 s

The unbounded run wins on wall time because nothing in the experiment pushes back — ten thousand timer goroutines simply sleep in parallel. That is exactly the trap. The cost of unbounded fan-out is never CPU in your own process; it is ten thousand simultaneous connections, a downstream that starts timing out under the burst, and a memory graph that follows the queue depth of whoever you are calling. On a real dependency, the unbounded run does not finish in 18 ms — it times out. The bounded runs pay 6.6 seconds because 10,000 tasks ÷ 8 slots × 5 ms is 6.25 s of unavoidable serialization, and the cap is the point: it turns an uncontrolled burst into a predictable 6.25-second drain.

One refinement matters when the cap is higher than a handful of slots: keep the cap at or below what the downstream can sustain concurrently, and derive it from that limit (connection pool size, rate limit, worker capacity) rather than from a feeling about "reasonable" parallelism.

Measuring and testing bounded code

Three signals cover most of it. Goroutine count trends (/sched/goroutines:goroutines exported via the Prometheus Go collector) catch leaks as a slope. Scheduler latency (/sched/latencies:seconds) catches saturation — runnable goroutines waiting for CPU time is real pressure regardless of the count. A pprof delta of two goroutine dumps pinpoints where the parked goroutines live.

For the tests, bounded workers are the case Testing Concurrent Go Code with synctest was built for: fake time makes a full 10,000-task drain run in milliseconds, synctest.Wait replaces sleep-and-hope for quiescence, and a bubble fails fast when a worker stays durably blocked — the leak repro above dies in a test bubble instead of in production.

At service scale, the same bounding question reappears between services, where the semaphore becomes a rate limit and the circuit breaker guards the downstream — Circuit Breaker Pattern in Go covers that layer, and Go Microservices for AI/ML Orchestration shows the queue-backed forms the orchestration tier takes.

Choosing

flowchart TD
    A[Work to run concurrently] --> B{Fixed batch or continuous stream?}
    B -- continuous stream --> C[Worker pool or bounded queue]
    B -- fixed batch --> D{Partial results needed<br/>on first failure?}
    D -- yes --> E[WaitGroup + errors.Join<br/>or semaphore channel]
    D -- no --> F[errgroup.WithContext]
    F --> G{Need a concurrency ceiling?}
    G -- yes --> H[errgroup.SetLimit]
    G -- no --> I[group without limit<br/>only when the batch is provably small]
    C --> J{Full queue: block or drop?}
    J -- block --> K[Backpressure upstream]
    J -- drop --> L[Shed load, count the drops]

The table and the flowchart compress to one habit: before writing go, name the ceiling and the owner. The ceiling is SetLimit, a semaphore, or a bounded queue; the owner is a Wait, a ctx.Done arm on every channel operation, or both. Every goroutine-creation site that can answer those two names in one glance is one that will not page anyone at 3 a.m.

Useful links

Clone a VM in Virt-Manager with virt-clone and virt-sysprep

Rost — Fri, 09 Oct 2026 11:36:19 +0000

Virt-Manager (the libvirt GUI) copies a KVM disk and assigns the clone a UUID and MAC. The guest keeps its machine-id and SSH host keys until virt-sysprep clears them.

This walkthrough uses an Ubuntu 26.04 guest and a 14 GB qcow2 image stored at /mnt/extra2/vms. The disk commands are the same for other Linux guests. virt-sysprep does not operate on Windows guests.

The host already needs KVM and libvirt. Install KVM on Ubuntu 24.04 is that setup, and it pulls in virtinst, which provides virt-clone. virt-sysprep, virt-customize, virt-cat, and guestfish come from a separate package:

sudo apt install -y libguestfs-tools

What actually clones a VM

libvirt splits the job across three tools. virt-clone copies the disk images and defines a new domain with the same virtual hardware. It changes the domain name, the UUID, and the host-side MAC. Passwords, the hostname, static addresses, and files inside the guest are unchanged, which is the behaviour virt-clone(1) documents.

virt-sysprep opens the disk with libguestfs and edits the guest filesystem. The virt-sysprep(1) manual is explicit that the edit is in place: there is no output-file flag, so the image you pass to -a is the image that changes. Copy or clone first if you still need the source.

qemu-img info reports whether that file is standalone or the top of a backing chain. qemu-img convert writes a new file with the chain resolved.

flowchart LR
    A[guest shut down] -->|virt-clone| B[new domain and disk]
    B -->|qemu-img info| C{backing file?}
    C -->|yes| D[qemu-img convert]
    C -->|no| E[standalone qcow2]
    D --> E
    E -->|copy, then virt-sysprep| F[template]
    F -->|per-instance customize| G[imported guest]

Flatten before sysprep when a backing line is present. Sysprep the copy, then derive each running guest from that file.

Cloning the disk in Virt-Manager

Shut the guest down. virt-clone rejects a running domain, and a copy of a live qcow2 is not a consistent filesystem.

In Virt-Manager, right-click the guest and choose Clone. Set the new name. On the storage step, give each disk a new path. Leaving the original path in place defines a second domain that opens the same file, so both guests write the same blocks.

The new file lands beside the source. A default pool uses /var/lib/libvirt/images/. A disk that already lives on /mnt/extra2/vms is cloned into that directory. Note the path virt-clone prints, or read it back:

virsh domblklist ubuntu26.04-base14gb-clone

The command-line equivalent, with the name and the disk path chosen explicitly:

virt-clone --original ubuntu26.04-base14gb \
  --name ubuntu26.04-base14gb-clone \
  --file /mnt/extra2/vms/ubuntu26.04-base14gb-clone.qcow2

--auto-clone invents the name and the disk path. That is convenient for a throwaway copy. A template path is easier to keep when you pass --name and --file yourself. Omit --mac, or pass --mac RANDOM, and virt-clone generates an address in the QEMU range 52:54:00:xx:xx:xx.

--reflink asks for a lightweight copy. The Ubuntu 26.04 virt-clone manual limits that flag to raw images on the same btrfs filesystem, so it skips these qcow2 files. A qcow2 overlay is a separate setup, covered with qemu-img create -b below.

Does your clone have a backing file?

Run this on the new disk before you treat it as independent:

qemu-img info --backing-chain /mnt/extra2/vms/ubuntu26.04-base14gb-clone.qcow2

A standalone image lists file format, virtual size, and disk size. An overlay also lists a backing file: line pointing at the image it reads through. virtual size is the guest's disk size, here about 14 GB. disk size is how much space the file currently occupies, which is small for a fresh overlay and grows as the guest writes.

virt-clone is not the usual source of that line. An overlay comes from qemu-img create -b, from a disk-only snapshot, or from a tool that created a qcow2 delta on purpose. The info check is what tells those cases apart from a full copy.

Keep the overlay only when the base will stay where it is and will not be written again. Every write from the clone lands in the small file; reads that miss in the overlay come from the base. Moving or editing the base breaks every guest that names it.

flowchart TB
    subgraph linked
        O[base.qcow2 read-only]
        C1[vm1.qcow2 deltas only]
        C2[vm2.qcow2 deltas only]
        C1 -->|backing file| O
        C2 -->|backing file| O
    end
    subgraph flat
        F1[vm1-flat.qcow2 standalone]
        F2[vm2-flat.qcow2 standalone]
    end

To make a standalone file, convert the overlay. -p prints progress. The destination is a new file; the overlay and the base are left as they are:

qemu-img convert -p -O qcow2 \
  /mnt/extra2/vms/ubuntu26.04-base14gb-clone.qcow2 \
  /mnt/extra2/vms/ubuntu26.04-base14gb-flat.qcow2

Run qemu-img info on the flat file and confirm there is no backing file: line. A template is that standalone file. Overlays are how you stamp guests from it later, not the template itself.

A btrfs reflink is a third case. cp --reflink=auto can share physical extents between two files, and qemu-img info still shows no backing file, because each path is a complete image. That copy is independent at the qcow2 layer even though the filesystem has not duplicated every block yet.

virt-sysprep: resetting the identity

Copy the standalone image, then sysprep the copy. --reflink=auto shares extents when the filesystem supports it and otherwise makes a normal copy, including on ext4. -n is a dry run that discards its writes:

cp --reflink=auto \
  /mnt/extra2/vms/ubuntu26.04-base14gb-flat.qcow2 \
  /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2

virt-sysprep -n -a /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2
virt-sysprep -a /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2 \
  --hostname ubuntu-template

On a default Ubuntu pool the image is often mode 0600 and owned by libvirt-qemu. The manual recommends giving your user write access rather than running as root. When the qemu user owns the file, sudo virt-sysprep -a ... is the invocation that can open it. The guest must be shut down, and nothing else should have the image open.

--hostname is a customize option, and customize is part of the default operation set, so it runs in the same invocation. There is no separate default operation that clears /etc/hostname on its own. --operations replaces that default set when you pass it; --operations defaults,-ssh-userdir keeps the defaults and turns one operation off.

These are the operations that decide whether the clone collides with the original. The list is the one in the current virt-sysprep manual, which is worth reading because a short --operations list silently drops the rest:

Operation	Default	Effect on the guest
machine-id	yes	Clears the machine ID. systemd generates a new one on the next boot when `/etc/machine-id` is empty.
ssh-hostkeys	yes	Removes `/etc/ssh/ssh_host_*`. The next boot creates new host keys.
ssh-userdir	yes	Removes `.ssh` under `/root` and `/home/*`, including `authorized_keys` and private keys stored in the guest.
dhcp-client-state	yes	Removes DHCP client leases.
udev-persistent-net	yes	Removes udev rules that map an old MAC to a fixed name such as `eth0`.
net-hwaddr	yes	Removes `HWADDR` from Fedora and RHEL `ifcfg-*` files. It does not edit Ubuntu netplan.
lvm-uuids, lvm-system-devices	yes	Assigns new LVM UUIDs and removes `/etc/lvm/devices/system.devices`, which otherwise pins the guest to the source disk's WWIDs.
logfiles, bash-history, tmp-files	yes	Removes logs, shell history, and temp files.

ssh-userdir is the one that surprises people cloning a machine they still log into. The template should not carry the source's private keys, so leave the operation on and inject a key per instance later. For a one-off copy that must keep the existing authorized_keys, pass --operations defaults,-ssh-userdir.

user-account is not in the default set, and passwords stay unless you pass --password or --root-password. fs-uuids is also off by default. The manual says it does not update every reference, including /etc/fstab, and that enabling it is likely to make the guest unbootable.

Ubuntu Server images usually have cloud-init, and virt-sysprep has no cloud-init operation. An old instance id under /var/lib/cloud makes the clone skip first-boot setup. Clean it on the template, and tell cloud-init to leave the hostname that --hostname wrote:

virt-customize -a /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2 \
  --run-command 'if command -v cloud-init >/dev/null 2>&1; then cloud-init clean --logs --seed; fi' \
  --write '/etc/cloud/cloud.cfg.d/99-preserve-hostname.cfg:preserve_hostname: true'

The same manual warns against cloning a guest that uses internal full-disk encryption for distribution: every copy shares the volume key. It also notes that deleted files are unlinked, not scrubbed. --scrub overwrites a named file, and virt-sparsify can drop freed space from the image.

How the clone gets its IP address

virt-sysprep has no operation that rewrites an address. A guest that used DHCP asks again. The clone has a new MAC, an empty machine-id, and no old lease file, so the DHCP server on libvirt's default network treats it as a new client.

A static addresses: block in netplan is copied with the disk, and both guests then claim it. Read the files before the first boot. virt-ls and virt-cat are the one-shot tools; guestfish --ro -a <disk> -i is the same inspection as a shell, after which ls /etc/netplan and cat run against the guest:

virt-ls -a /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2 /etc/netplan
virt-cat -a /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2 \
  /etc/netplan/00-installer-config.yaml

Installer images often use 00-installer-config.yaml or 01-netcfg.yaml. Cloud images use 50-cloud-init.yaml, which cloud-init will recreate unless network configuration is disabled. Netplan merges every YAML file in the directory, so a leftover static file still applies after you add a DHCP file.

To make the template a DHCP client, remove those files and upload one replacement. Use the interface name virt-cat showed. On a guest installed under KVM that name is often ens3 or enp1s0, and virt-clone keeps the same virtual PCI slot, so the name stays valid:

cat > /tmp/99-dhcp.yaml <<'EOF'
network:
  version: 2
  ethernets:
    ens3:
      dhcp4: true
EOF

virt-customize -a /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2 \
  --delete '/etc/netplan/*.yaml' \
  --upload /tmp/99-dhcp.yaml:/etc/netplan/99-dhcp.yaml \
  --write '/etc/cloud/cloud.cfg.d/99-disable-network.cfg:network: {config: disabled}'

For a static address, gateway, and DNS, use the same upload with the YAML from How to Change a Static IP Address in Ubuntu Server. Each instance then needs its own file, applied to the instance disk rather than the template.

The name in the YAML has to exist inside the guest. A disk moved from physical hardware often names the NIC enp2s0, while the same system under virtio presents a different name, and netplan then configures an interface that is not there. For a template that should boot on more than one machine type, pin the kernel to eth0 and use that key in netplan. Ubuntu sources /etc/default/grub.d/*.cfg while building grub.cfg:

virt-customize -a /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2 \
  --write '/etc/default/grub.d/99-eth0.cfg:GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX net.ifnames=0 biosdevname=0"' \
  --run-command 'update-grub'

update-grub runs inside the libguestfs appliance. If grub-probe cannot see the boot disk there, the command fails and the cmdline is unchanged. In that case, set the interface name in netplan to the name the guest already uses, and skip the eth0 switch.

Per-instance customization with virt-customize

Boot instances from copies of the clean file. Do not point a domain at the template itself.

A full copy has no runtime dependency on the template:

cp --reflink=auto \
  /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2 \
  /mnt/extra2/vms/ubuntu26.04-vm1.qcow2

An overlay stores only the blocks vm1 writes. -F qcow2 records the base format so QEMU does not have to probe it:

qemu-img create -f qcow2 \
  -b /mnt/extra2/vms/ubuntu26.04-base14gb-clean.qcow2 \
  -F qcow2 \
  /mnt/extra2/vms/ubuntu26.04-vm1.qcow2

The base path is stored inside the overlay. It has to keep that path, and it has to stay read-only, for every guest built this way.

The virt-sysprep security notes describe a second pass on each new disk. Sysprep writes a random seed into the template; guests cloned from that file would otherwise start with the same seed, which makes first-boot SSH host keys and TCP sequence numbers predictable. --enable customize rewrites the seed. --hostname and --ssh-inject are customize options, so they belong on the same command. Substitute the account that actually exists in the image for ubuntu:

virt-sysprep --enable customize \
  -a /mnt/extra2/vms/ubuntu26.04-vm1.qcow2 \
  --hostname vm1 \
  --ssh-inject "ubuntu:file:${HOME}/.ssh/id_ed25519.pub"

Hand the instance file to libvirt, not the template. If you created it as your user outside a storage pool, libvirt-qemu needs read access or the domain fails at startup with a permission error on the qcow2.

Importing into Virt-Manager

For each instance disk, open Virt-Manager, choose New Virtual Machine, and select Import existing disk image. Point it at ubuntu26.04-vm1.qcow2. The OS entry should be Ubuntu 26.04 when the host's osinfo database has it.

The matching short id is ubuntu26.04. Check the host database before relying on it:

osinfo-query os | grep ubuntu26

If that prints nothing, the installed database is older than the 26.04 entry. --os-variant ubuntu24.04 still selects a virtio-friendly machine type for a guest that is already installed. --import tells virt-install the disk already contains an OS.

sudo virt-install --name ubuntu26.04-vm1 \
  --memory 4096 --vcpus 2 \
  --disk path=/mnt/extra2/vms/ubuntu26.04-vm1.qcow2,format=qcow2 \
  --import \
  --os-variant ubuntu26.04 \
  --network network=default \
  --noautoconsole

--network network=default attaches the NAT network whose dnsmasq instance hands out the DHCP lease. After boot:

virsh domifaddr ubuntu26.04-vm1 --source lease
virsh dumpxml ubuntu26.04-vm1 | grep "<mac"

--source lease reads that dnsmasq table. It stays empty for a static address, for a bridge, or for a NIC that is not on a libvirt virtual network. Those guests need qemu-guest-agent in the image and virsh domifaddr ubuntu26.04-vm1 --source agent, or an ARP lookup with --source arp. virsh net-dhcp-leases default lists every lease on the NAT network when you want the pool view rather than one domain.

Choosing a clone strategy: full copy, overlay, or reflink

Strategy	Space per VM	Time to create	Constraint	Use when
Full copy, or `cp --reflink=auto`	about 14 GB logical; reflink shares blocks until they change	minutes, or near-instant with reflink	reflink needs a filesystem that supports it, such as btrfs	a guest that should outlive the template
qcow2 overlay	megabytes at first	seconds	base path must remain, and the base must stay read-only	many similar guests from one sealed image
`virt-clone --reflink`	raw image, shared extents	near-instant	raw disks on the same btrfs filesystem; qcow2 is skipped	raw images already stored on btrfs

A guest you expect to keep for months is a full copy or a reflink. A short-lived guest can be an overlay, as long as the base file is still there when it runs. Proxmox is the other self-hosted path when you want a datacenter-style template and clone model on its own web UI. Multipass launches Ubuntu cloud images for a single user and applies cloud-init itself, which is a different workflow from sealing a disk you already installed.

Troubleshooting: when the clone will not boot

No address. virt-cat the netplan file and compare the interface key with ip link on the console. A match: macaddress entry still set to the source MAC drops the new NIC. A key that names an interface the guest does not have does the same. Fix the file on the disk, or apply the eth0 cmdline above and rebuild the instance from the template.

Address up, same as the original. The static addresses: block is still in one of the netplan files. virt-sysprep did not remove it. Upload a replacement and delete the other YAML files in /etc/netplan.

SSH says the host key changed, or accepts the clone as the original. Host keys are shared when ssh-hostkeys did not run on the file the domain opens. Confirm with virsh domblklist that the domain uses the sysprepped disk, then check qemu-img info so you did not sysprep an overlay while the guest boots from a different path. If the clone kept the original IP on purpose, clients that stored the old key in known_hosts will warn until that line is removed. That warning is the regeneration the manual describes.

Key login fails on a clone of a machine you could reach before. Default sysprep removed .ssh from the home directories. Inject a key with --ssh-inject, or re-run with --operations defaults,-ssh-userdir if this copy should have kept them.

LVM volumes missing after boot. The guest has /etc/lvm/devices/system.devices from the source, and the virtual disk's WWID changed. The default lvm-system-devices operation removes that file. A hand-picked --operations list has to include it, and lvm-uuids, when the guest uses LVM.

One clone or a template

A single extra machine is virt-clone, a backing-file check, and virt-sysprep on the new disk. A fleet is one sealed standalone file, then a copy or an overlay per guest, with --enable customize on that guest's own disk before it boots.

Overlays make the sealed file part of the runtime path of every guest that names it. Full copies and reflinks do not. Pick that dependency with the lifetime of the guest, and leave fs-uuids off so the bootloader and /etc/fstab still match the filesystem.

Useful links

References

virt-clone(1) on Ubuntu 26.04, including --auto-clone, --file, and --reflink
virt-sysprep(1) operations, in-place edits, and the random-seed note
virt-customize(1) for --hostname, --ssh-inject, --upload, and --run-command
qemu-img info, convert, and create -b / -F
virt-install(1) --import and --os-variant
osinfo-db Ubuntu 26.04 entry, short id ubuntu26.04
machine-id(5) on how an empty /etc/machine-id is replaced at boot

Self-Hosted S3 Alternatives in 2026 (RustFS, Garage, SeaweedFS, Ceph)

Rost — Thu, 08 Oct 2026 08:35:20 +0000

Amazon S3 has become the de facto API for object storage. After the open source MinIO repository was archived in April 2026, seven credible self-hosted alternatives remain.

These options are not interchangeable. Some are full object-storage engines that own their data layout; others are thin gateways that expose files you already have. That split — native store versus filesystem-backed gateway — decides more about backups, recovery, and migrations than any feature list does.

This article compares RustFS, Garage, SeaweedFS, Ceph RGW, VersityGW, S3Proxy, and rclone. Each section covers the storage model, the S3 feature coverage, where the project fits, and the limitation that usually decides the choice. A decision guide for single-server and multi-server deployments closes the article.

For the broader picture — object storage, databases, search, and AI-native data layers — see the Data Infrastructure for AI Systems.

Self-Hosted S3 Storage Comparison

The most important question is whether you need a distributed object store or an S3 interface in front of storage that already exists.

Software	Storage model	Distributed	S3 compatibility	Plain files on disk	Complexity	Best use
RustFS	Native object store	Yes	High, tested subset	No	Low to medium	General MinIO replacement
Garage	Native object store	Yes	Good, narrower API	No	Low to medium	Small distributed clusters
SeaweedFS	Distributed blob and file store	Yes	High	No	Medium	Huge file counts and mixed storage
Ceph RGW	Ceph object gateway	Yes	High	No	High	Large storage infrastructure
VersityGW	S3 gateway over POSIX	Backend dependent	Good	Yes	Low	S3 over normal files
S3Proxy	S3 translation layer	Backend dependent	Moderate	Yes with filesystem-nio2	Medium	Gateway and protocol translation
rclone serve s3	S3 gateway	Backend dependent	Basic, experimental	Yes with local backend	Very low	Simple and lightweight deployments

RustFS is the closest direct replacement for a conventional MinIO deployment. Garage fits small distributed clusters. SeaweedFS and Ceph solve broader storage problems. VersityGW can expose an existing POSIX filesystem through S3 while leaving the object payloads as ordinary files. For a feature-by-feature matrix of Garage, MinIO, and AWS S3, see Garage vs MinIO vs AWS S3.

Two Different Types of S3 Storage

A native object store owns its storage layout:

flowchart TD
    A[Application] -->|S3 API| B[Object Store]
    B --> C[Internal Metadata]
    B --> D[Chunks or Object Layout]
    C --> E[Disks]
    D --> E

RustFS, Garage, Ceph RGW, and SeaweedFS belong to this family. The files visible on the underlying disks are implementation details, and applications should not manipulate them directly.

A filesystem-backed S3 gateway has a different relationship with storage:

flowchart TD
    A[Application] -->|S3 API| B[S3 Gateway]
    B --> C[POSIX Filesystem]
    C --> D[bucket]
    D --> E[path]
    E --> F[ordinary-file.dat]

With software such as VersityGW, the filesystem can remain the authoritative representation. An object such as:

s3://archive/documents/2026/report.pdf

can correspond directly to:

/storage/archive/documents/2026/report.pdf

RustFS: The Closest MinIO Replacement

RustFS is a distributed object storage system written in Rust and released under the Apache-2.0 license. It provides an S3-oriented operating model familiar to MinIO users, which makes it the first project to evaluate when replacing MinIO.

Its current S3 implementation covers the operations most applications expect: buckets, objects, multipart uploads, conditional requests, object metadata, tags, versioning, lifecycle management, object lock, presigned URLs, bucket policies, CORS, notifications, replication configuration, and server-side encryption.

RustFS publishes an S3 compatibility matrix that is a tested subset, not a claim of complete Amazon S3 coverage. The snapshot verified against commit 1e6f5f1e on 9 August 2026 lists 455 implemented standard tests, 5 lifecycle-behavior tests, 17 unimplemented standard tests, and 270 excluded cases that do not block the compatibility gate. The executable lists under scripts/s3-tests are the source of truth when those counts move.

Where RustFS fits

RustFS is a candidate when an application expects a proper S3 object store.

Typical uses include application object storage, backup repositories, media storage, AI and data-pipeline artifacts, internal S3 infrastructure, Kubernetes workloads, and replacement of existing MinIO endpoints.

It works with AWS SDKs and common S3 clients. Applications normally need a custom endpoint, credentials, region settings where required, and path-style addressing.

The important limitation

RustFS owns the storage representation. The files below its data directory are RustFS internals.

That makes RustFS the wrong answer when the requirement is that every S3 object remain an ordinary file you can inspect, copy, rsync, or recover without the object-storage software. That requirement selects a filesystem-backed gateway.

Garage: Distributed S3 Without Ceph-Scale Complexity

Garage is an S3-compatible distributed object store for small and medium self-hosted installations. Deuxfleurs develops it under the AGPL v3 licence and has run it in production since the project's initial releases in 2020.

Garage is built to run across several ordinary servers, including servers in different physical locations. Replication is part of the design.

flowchart LR
    A[Applications] --> B[S3 Endpoint]
    B --> C[Garage Node 1]
    B --> D[Garage Node 2]
    B --> E[Garage Node 3]
    C <--> D
    D <--> E
    E <--> C

Garage is attractive when you have three small servers rather than a rack of dedicated storage hardware. A working setup path, from Docker through cluster layout, replication, and TLS via a reverse proxy, is in the Garage quickstart.

Where Garage fits

Garage makes sense for homelabs, small hosting platforms, geographically distributed servers, replicated backups, and clusters where Ceph would be excessive.

Its scope is narrower than AWS S3. Garage's compatibility table leaves out bucket policies, ACLs, and bucket versioning. Choose Garage over RustFS when lightweight multi-node replication is the primary requirement. Choose RustFS when broader S3 behavior and MinIO-like application compatibility matter more.

SeaweedFS: More Than an Object Store

SeaweedFS started by storing enormous numbers of files efficiently. It is now a distributed storage platform exposing S3, a FUSE filesystem, WebDAV, SFTP, and HDFS. SeaweedFS 4.48 was released on 28 September 2026.

Volume servers pack blobs into append-only volume files. Masters track volumes rather than individual files. A filer provides hierarchical filesystem semantics. The design targets workloads with millions or billions of relatively small objects.

S3 support

SeaweedFS documents versioning, object lock, lifecycle rules, tagging, CORS, checksums, presigned URLs, multipart uploads, bucket policies, IAM, STS, and several server-side encryption modes.

The same cluster can serve S3 table buckets and a built-in Iceberg REST catalog, so Spark, Trino, DuckDB, and similar engines can use Iceberg tables without a separate Hive Metastore or Glue. The project's one-command S3 example listens on port 8333.

Where SeaweedFS fits

Consider SeaweedFS for web crawlers and archives, image and document repositories, billions of small objects, data lakes, mixed S3 and filesystem workloads, and storage that grows by adding volume servers.

There are more components to understand than in a straightforward RustFS installation. SeaweedFS can provide filesystem semantics through its filer and FUSE mount, but objects are stored inside its volume format. It does not satisfy the requirement that every S3 object remain an independent ordinary file on the host filesystem.

Ceph RGW: The Infrastructure-Scale Option

Ceph is broader than S3 storage. A cluster can provide block devices through RBD, a filesystem through CephFS, and object storage through RADOS Gateway (RGW). RGW presents an S3-compatible API backed by RADOS.

flowchart TD
    A[Applications] -->|S3| B[Ceph RGW]
    B --> C[RADOS]
    C --> D[OSD 1]
    C --> E[OSD 2]
    C --> F[OSD 3]

Ceph supports replication, erasure coding, failure domains, multiple gateways, multisite deployments, access policies, and cluster management tooling.

Where Ceph fits

Ceph makes sense when object storage is one part of a larger storage platform: Kubernetes persistent volumes, virtual-machine disks, shared filesystems, large S3 repositories, and clusters that need explicit failure domains.

For a single server with several terabytes, installing Ceph only to obtain an S3 endpoint is difficult to justify. If you already operate Ceph, RGW is the obvious S3 choice. If you do not, RustFS or Garage reaches useful object storage with less machinery.

VersityGW: S3 Over Ordinary Files

VersityGW is the option to evaluate when the filesystem itself matters. It is an Apache-2.0 licensed S3 gateway that can expose POSIX filesystems, ScoutFS, Azure Blob Storage, other S3 services, and custom backends.

Its POSIX backend maps S3 buckets and objects onto directories and files. For example:

s3://website-assets/images/logo.png

maps to a path such as:

/storage/website-assets/images/logo.png

The logo.png payload remains a normal file.

Why plain files are useful

You can inspect storage with ordinary Unix tools:

find /storage -type f
du -sh /storage/*

You can read an object without the S3 server:

cat /storage/archive/2026/source.html

Conventional filesystem backup tools apply directly: rsync, restic, borg, tar, and zfs send. If the gateway software disappears, the payloads are still files in a directory tree.

S3 metadata on a POSIX backend

S3 metadata does not fit in a POSIX file by itself. VersityGW stores it in extended attributes by default. Since March 2026, object metadata is a single user.metadata xattr holding JSON, because one xattr per key hit filesystem key-length limits. A versitygw utils convert-xattr-metadata command rewrites older per-key attributes. Reads still fall back to the old layout when user.metadata is absent.

Filesystems without xattrs can use --sidecar (or VGW_META_SIDECAR) to keep metadata in a separate directory tree. The project marks sidecar and --nometa as less tested than xattrs. --nometa disables metadata storage and is aimed at read-only hosting of a dataset that already exists; bucket policies and ACLs are metadata, so they are unavailable in that mode.

Object versions can live in a separate directory. The project's POSIX example passes --versioning-dir for older versions while the current payload stays at its natural path.

Changing a file behind the gateway can leave S3 metadata, including ETags, inconsistent with the new bytes. A workable operating model is:

Application access:
    Read through S3
    Write through S3

Administrator access:
    Read the filesystem directly
    Inspect the filesystem directly
    Back up the filesystem directly
    Avoid modifying objects behind the gateway

VersityGW 1.8.0, already the version a September 2026 COSI driver release targeted, adds a standalone IAM service (versitygw iam), STS AssumeRoleWithWebIdentity, and IAM policy conditions.

S3Proxy: A General Translation Layer

S3Proxy implements the S3 API and translates requests onto several backends: local files, Azure Blob Storage, Google Cloud Storage, OpenStack Swift, SFTP, and other S3 services. It requires Java 17 or newer.

flowchart TD
    A[S3 client] --> B[S3Proxy]
    B --> C[filesystem-nio2]
    C --> D[Local filesystem]

The recommended on-disk provider is filesystem-nio2. The older filesystem provider is deprecated. Object keys become filesystem paths, and metadata uses user extended attributes, so the payloads stay ordinary files.

The filesystem backend refuses object versioning and server-side encryption. Versioning stays unsupported (supportsVersioning() is false, and those requests remain HTTP 501) because a version layout written to a real disk would become a format the project would have to keep. Server-side encryption is refused for the same class of reason: reporting AES256 over a plaintext file someone can read would be a false claim. The in-memory transient-nio2 backend implements both, and the data dies with the process. The project README also lists bucket policies, object tagging, and ACLs other than private and public-read as unsupported.

S3Proxy is the tool when you need that translation across backends. For a straightforward S3-to-POSIX gateway, VersityGW is the smaller surface.

rclone serve s3: Lightweight but Experimental

rclone serve s3 exposes an rclone backend through an S3-compatible API. The documentation still labels the command experimental.

For a local filesystem:

rclone serve s3 \
  --addr :9000 \
  --auth-key ACCESS_KEY,SECRET_KEY \
  local:/srv/s3

Without --addr, the server binds 127.0.0.1:8080. --auth-key enables Signature Version 4. Multipart uploads are written in part-number order to a temporary object and renamed into place on completion, so a failed upload does not replace an existing object.

Objects written through a local backend become files under that path. The same command can sit in front of other rclone remotes. It is a reasonable choice for internal tooling, a lab, or a temporary migration hop. It is a weak permanent S3 layer for many unrelated production applications.

What About MinIO?

The MinIO GitHub repository was archived on 13 February 2026, briefly unarchived, and archived again on 25 April 2026. It is read-only. The repository notice points operators at AIStor, MinIO's commercial distribution, and states that the community edition is source-only.

An existing MinIO deployment can keep running, particularly when it is isolated and its behavior is already understood. The full timeline, the operational risk, and a phased migration plan are in MinIO CE in 2026.

For the S3 behavior of the system you may be leaving, MinIO as an AWS S3 alternative documents how MinIO maps onto S3. The MinIO command line cheatsheet is the audit companion for a cluster that is still up. For a new installation, an actively maintained alternative is the default.

Plain Files vs Native Object Storage

Use RustFS, Garage, SeaweedFS, or Ceph when the object store should control durability, metadata, distribution, and layout. That buys richer S3 semantics, replication or erasure coding, horizontal scaling, and versioning where the project implements it. The cost is dependence on the engine's on-disk representation.

Use VersityGW, S3Proxy, or rclone when an existing filesystem should remain authoritative. That buys plain recoverable files, conventional backups, direct inspection, and compatibility with ZFS, XFS, ext4, and NFS. Distributed durability then has to come from the filesystem or the storage array, not from the S3 process. On a single server or a NAS, that trade is often the right one.

Choosing the Right Self-Hosted S3 Server

For a single server that should resemble MinIO, start with RustFS.

For three or more modest servers where replication is the main requirement, and the application can live without bucket policies and versioning, choose Garage.

For enormous numbers of small objects, mixed filesystem access, or a data-lake workload that may grow into Iceberg tables, consider SeaweedFS.

When the same cluster must also provide block devices and a distributed filesystem, choose Ceph RGW.

When the requirement is specifically an S3 API over ordinary files — a NAS, an archive, a document repository, a crawler, or a backup target — start with VersityGW.

When the job is translating S3 onto Azure, GCS, Swift, SFTP, or another S3 service, use S3Proxy.

When the deployment is small, temporary, and can tolerate an experimental server, rclone serve s3 may be enough.

Recommended Architecture for a Single Storage Server

If applications should reach the data only through S3, RustFS owns the object layer and the filesystem is an implementation detail:

flowchart TD
    A[Applications] -->|S3| B[RustFS]
    B --> C[Local Storage]
    C --> D[Filesystem or RAID]

If the files themselves must survive the gateway, keep the data in a POSIX tree:

flowchart TD
    A[Applications] -->|S3| B[VersityGW]
    B --> C[POSIX Filesystem]
    C --> D[ext4, XFS, ZFS, NFS]
    D --> E[Plain Files]

In the second design, redundancy is a ZFS mirror, RAID, snapshots, filesystem replication, or a conventional backup tool. The S3 service can be replaced without rewriting the data format.

Final Thoughts

Maintained self-hosted S3 still exists after MinIO's public repository went read-only. The choice that matters is whether S3 owns the bytes or only fronts files that already exist. Every comparison above follows from that answer.

References

Go Web Frameworks in 2026: net/http, chi, Gin, Echo, Fiber Compared

Rost — Mon, 05 Oct 2026 23:53:52 +0000

Go ships a production-capable HTTP server in its standard library, and since Go 1.22 the built-in router handles method matching and path parameters on its own. The framework question has narrowed, not disappeared.

Five stacks cover almost every Go backend decision in 2026: the standard library's net/http, chi, Gin, Echo, and Fiber. Each takes a different position on the same tradeoff — what the library gives you versus what you owe it in dependencies, conventions, and ecosystem compatibility. This article compares them on routing power, handler ergonomics, middleware, and measured performance, then ends with a decision table rather than a single verdict.

If you are starting from zero and want the full build process around your choice — validation, authentication, testing, deployment — the step-by-step version lives in Building REST APIs in Go. Here the focus stays on the stacks themselves.

The post-1.22 ServeMux reset

The reason the framework debate changed shape is one addition to the standard library. Go 1.22 gave http.ServeMux method-aware patterns and path parameters:

mux := http.NewServeMux()
mux.HandleFunc("GET /users/{id}", func(w http.ResponseWriter, r *http.Request) {
    id := r.PathValue("id")
    // ...
})
mux.HandleFunc("POST /users", createUser)

Before this, any API needing GET /users/{id} required a third-party router, which is how Gin, chi, and Echo became defaults. In 2026, the stdlib covers static routes, method matching, single-segment and wildcard parameters, and middleware via plain function wrapping. What it still does not give you: route groups, composable middleware chains, struct binding, or per-route param types.

The contenders

	net/http (stdlib)	chi v5.3.2	Gin v1.12.0	Echo v4.16.0	Fiber v2.52.15
HTTP engine	net/http	net/http	net/http	net/http	fasthttp
Handler signature	`func(w, r)`	`func(w, r)`	`func(c *gin.Context)`	`func(c echo.Context) error`	`func(c *fiber.Ctx) error`
http.Handler compatible	yes	yes	yes	yes	no
Middleware ecosystem	growing, stdlib-shaped	stdlib-shaped	largest, Gin-specific	large, Echo-specific	growing, fasthttp-specific
Binding/validation	manual	manual	built-in tags	built-in	via helpers
HTTP/2	yes	yes	yes	yes	no
Dependencies	0	1	~6	~8	~4

All five use trie- or radix-style routers with path parameters and grouped routes. The structural divide is the last row of the handler-signature story: chi and stdlib speak plain http.Handler; Gin, Echo, and Fiber wrap requests in their own context type. Fiber is the outlier once more because it sits on fasthttp, not net/http at all.

What I benchmarked

Rather than quoting synthetic router benchmarks, I ran the same service on all five stacks. The rig:

Go 1.27.1, Gin v1.12.0, chi v5.3.2, Echo v4.16.0, Fiber v2.52.15
Intel Core Ultra 7 265, Linux, 20 threads, localhost, no TLS
Three routes per stack: GET /health, GET /users/{id}, GET /api/v1/users/{uid}/orders/{oid}
Load: 8 keep-alive workers for 3 seconds against the two-parameter route, after a warmup pass
Every handler marshals a small JSON document per request with encoding/json — so the numbers include serialization, not just routing

The stdlib version of the hot route:

mux.HandleFunc("GET /api/v1/users/{uid}/orders/{oid}", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    w.Write(orderBody(r.PathValue("uid"), r.PathValue("oid")))
})

and the Fiber equivalent:

app.Get("/api/v1/users/:uid/orders/:oid", func(c *fiber.Ctx) error {
    c.Set("Content-Type", "application/json")
    return c.Send(orderBody(c.Params("uid"), c.Params("oid")))
})

Two honesty notes before the numbers. First, this is a localhost micro-workload: no database, no TLS handshake, no middleware chain. Real services bottleneck on I/O — the framework's share of a request that queries PostgreSQL via GORM, Ent, Bun, or sqlc is a rounding error. Second, each stack got its own server process and an identical handler shape, so the comparison is between stacks, not between hand-tuned configurations.

Results

Throughput on the dynamic two-parameter route:

Stack	Requests/sec	Avg latency
Fiber (fasthttp)	205,958	0.034 ms
Echo	169,826	0.042 ms
Gin	169,715	0.042 ms
net/http ServeMux	169,708	0.042 ms
chi	163,017	0.044 ms

The four net/http-based stacks land within 4% of each other. Fiber's fasthttp engine is about 21% ahead on this localhost workload — real, but the kind of gap that a single database roundtrip (0.5–5 ms) makes irrelevant in production.

To see where the per-request cost actually goes, I also benchmarked handler invocation without the network, calling each router directly through http.Handler.ServeHTTP (Fiber through its fasthttp handler):

Stack	Time/op	Bytes/op	Allocs/op
Gin	289 ns	192 B	3
Echo	333 ns	192 B	3
net/http ServeMux	430 ns	240 B	5
Fiber	541 ns*	192 B	3
chi	592 ns	897 B	7

* The Fiber figure carries harness overhead — the test has to copy the request into fasthttp's reusable context each iteration, which the real server avoids by reading from the wire. Treat it as an upper bound; the server-level throughput above is the fair comparison for Fiber.

The allocation profile is the more durable story. Gin and Echo pool their request contexts, so a request through them costs 3 allocations. The stdlib's 5 allocations go to the request-scoped plumbing itself. chi stores route parameters in the request context.Context, which shows up as 897 B and 7 allocations per request — the price of staying a plain http.Handler with rich route info.

Fiber and the fasthttp tradeoff

Fiber is the fastest stack on the server benchmark, and the speed comes with a compatibility bill. fasthttp is a from-scratch HTTP implementation with its own request/response types, so:

net/http middleware does not attach. Any library expecting an http.Handler or *http.Request needs a conversion or a Fiber-native port.
No HTTP/2. fasthttp does not implement it; Go's net/http gets h2 from the standard library for free over TLS.
WebSocket, streaming, and http.Flusher-style tricks all take the fasthttp path, which differs from the stdlib behavior most Go documentation assumes.

What Fiber buys in exchange: the fastest numbers in the table above, an Express-like API, and lower allocations than the framework benches suggest once the wire parsing advantage is included. It fits when responses are small, routing dominates, HTTP/2 is terminated upstream, and the middleware you need exists in the Fiber ecosystem. Those conditions hold for some edge services and hold for almost nothing else.

chi: the boring middle path

chi is a router and a middleware pipeline, nothing more. Handlers are func(w, r), middleware is func(http.Handler) http.Handler, and everything written for the stdlib attaches without adapters:

r := chi.NewRouter()
r.Use(RequestID, RealIP, Logger, Recoverer)
r.Route("/api/v1", func(r chi.Router) {
    r.Get("/users/{uid}/orders/{oid}", getOrder)
})

That is the whole pitch: route groups and a middleware stack on top of the standard interface. The allocation table prices it honestly — richer route context per request than the stdlib alone, in exchange for composable middleware and nested routes. For services that want structure without a framework identity, chi plus net/http is the configuration most Go teams converge on.

Gin vs Echo, in code

Gin and Echo are near-twins on performance — within noise in both benchmarks — so the choice between them is API taste and error-handling philosophy.

Echo centralizes errors in an HTTPErrorHandler. Handlers return error, and one function maps errors to responses:

func getOrder(c echo.Context) error {
    o, err := store.Order(c.Param("uid"), c.Param("oid"))
    if err != nil {
        return echo.NewHTTPError(http.StatusNotFound, "order not found")
    }
    return c.JSON(http.StatusOK, o)
}

Gin handlers return nothing; errors accumulate on the context and you render them where you choose:

func getOrder(c *gin.Context) {
    o, err := store.Order(c.Param("uid"), c.Param("oid"))
    if err != nil {
        c.JSON(http.StatusNotFound, gin.H{"error": "order not found"})
        return
    }
    c.JSON(http.StatusOK, o)
}

Echo's model maps cleanly onto Go error handling architecture with wrapped errors and centralized boundary handling — one place translates errors.Is results into status codes. Gin's model matches the more common pattern of deciding per handler. Beyond errors, Gin's deciding advantages are ecosystem size and built-in struct binding with validation tags; Echo's are a consistent context API and stronger built-in middleware. Both integrate Swagger generation through their own adapters — the per-framework setup is covered in Adding Swagger to Your Go API.

One more axis: context propagation. Gin and Echo wrap context.Context inside their own context types, so passing cancellation into handlers requires an extraction step. The patterns for keeping cancellation, timeouts, and values straight across any of these wrappers are in Go context.Context Done Right.

The timeouts nobody sets

Every stack in this comparison leaves server timeouts at zero defaults unless you set them — and zero means no limit. A client that opens a connection and sends nothing holds a goroutine and a file descriptor indefinitely:

srv := &http.Server{
    Addr:              ":8080",
    Handler:           mux,
    ReadHeaderTimeout: 5 * time.Second,
    ReadTimeout:       10 * time.Second,
    WriteTimeout:      30 * time.Second,
    IdleTimeout:       120 * time.Second,
}
srv.ListenAndServe()

ReadHeaderTimeout is the one to set first — it bounds slowloris-style connection exhaustion without breaking long uploads the way a blanket ReadTimeout can. WriteTimeout must cover the slowest legitimate response, which makes it a policy decision tied to your timeout budgets rather than a copy-paste constant. The same discipline applies per framework: Gin, Echo, and Fiber all accept server-level timeout configuration, and none of them enable it by default.

Choosing

flowchart TD
    A[New Go HTTP service] --> B{Internal tool, plugin,<br/>or zero-dependency requirement?}
    B -- yes --> C[net/http ServeMux]
    B -- no --> D{Want built-in binding,<br/>validation, big middleware catalog?}
    D -- no --> E[chi + net/http]
    D -- yes --> F{Team convention?}
    F -- error-returning handlers --> G[Echo]
    F -- largest ecosystem, Gin idioms --> H[Gin]
    A --> I{Small responses, extreme throughput,<br/>no HTTP/2, upstream TLS termination?}
    I -- all true --> J[Fiber]
    I -- no --> E

Situation	Pick
Internal service, CLI endpoint, plugin, library	net/http alone
Most APIs — structure without framework lock-in	chi on top of net/http
Public API, larger team, validation and binding out of the box	Gin
Public API, centralized error translation, consistent context API	Echo
Proven routing/serialization bottleneck, tiny JSON, no HTTP/2	Fiber

If the honest default for a new service in 2026 has to be named: start with net/http, add chi when middleware or route groups start to ache, and move to Gin or Echo only when binding, validation, or team convention justifies a framework identity. Fiber stays a specialist tool — worth benchmarking against when throughput is proven to be the problem, priced against the compatibility bill it charges.

Migrating between stacks

Migration cost tracks the handler-signature divide. stdlib ↔ chi is mechanical: handlers keep the same signature, routes move from mux.HandleFunc patterns to r.Get, middleware attaches to the chi stack. Gin ↔ Echo is a context-type translation — c.Param, c.JSON, and middleware logic port nearly line-for-line. Anything moving to or from Fiber is a rewrite of the HTTP layer, because *http.Request-based code, net/http middleware, and h2 assumptions do not survive the fasthttp boundary; the least painful route is keeping business logic in framework-free packages and regenerating the transport layer, which is also the structure that makes the next migration cheap.

Useful links

Best Newsletter Platforms in 2026: Pricing, Hosting and SEO

Rost — Mon, 05 Oct 2026 10:38:35 +0000

Newsletter platforms split into thin email senders and full publishing systems with archives and memberships. That split decides hosting, SEO, and migration more than any editor does.

This comparison covers Buttondown, EmailOctopus, beehiiv, Kit, Ghost, Substack, MailerLite, Brevo, and Sender. Prices are in USD, reflect the public pricing in October 2026, and normally rise as the mailing list grows on subscriber-based plans.

Each platform section describes what the product actually does, its pricing tiers, and where the public archive lives. The closing sections cover the three archive ownership models, canonical URLs, sender authentication, and API publishing, because those decisions outlast any single vendor choice.

Platform models: email-first, publishing-first, hybrid

Email-first services assume that another system owns the website and the content. Publishing-first services want the newsletter platform itself to become an important public destination. A few deliberately sit between those two models, and the architectural choice carries consequences for SEO, archives, and migration.

flowchart TD
    A[Newsletter platforms] --> B[Email-first]
    A --> C[Publishing-first]
    A --> D[Hybrid]

    B --> B1[EmailOctopus]
    B --> B2[Brevo]
    B --> B3[Sender]
    B --> B4[MailerLite]

    C --> C1[Substack]
    C --> C2[beehiiv]
    C --> C3[Ghost]

    D --> D1[Buttondown]
    D --> D2[Kit]

Pricing and features at a glance

Platform	Free plan	Paid entry point	Public archive	External canonical support	API	Main model
Buttondown	100 subscribers	From $9/mo	Yes	Yes	Yes	Hybrid
EmailOctopus	2,500 subscribers, 10,000 emails/mo	$10/mo, or $9/mo annual	Individual web copies	No documented per-issue override	Yes	Email-first
beehiiv	2,500 subscribers, unlimited sends	From $43/mo annual	Yes	Limited for external-site workflows	Yes	Publishing-first
Kit	10,000 subscribers, unlimited broadcasts	From $33/mo annual	Yes	Limited	Yes	Hybrid
Ghost(Pro)	No permanent free hosted plan	From $18/mo annual	Yes	Yes	Yes	Publishing-first
Substack	Unlimited free publishing	10% of paid revenue	Yes	Limited	Limited	Publishing-first
MailerLite	250 subscribers, 2,500 emails/mo	From $12/mo	Yes	Limited	Yes	Email-first
Brevo	300 sends/day	From $9/mo	Web campaign copies	No comparable publication archive	Yes	Email-first
Sender	2,500 subscribers, 15,000 emails/mo	From $7/mo annual	Web campaign copies	Limited	Yes	Email-first

The free allowance is a useful starting point, but it should not dominate the decision. A newsletter with 500 subscribers may be operationally better served by a $10 email platform than by a free system that also wants to become the primary CMS.

Buttondown

Buttondown stays focused on email while adding publishing features: Markdown support, custom sending domains, a hosted archive, an API, and external canonical URLs for individual issues. A newsletter can therefore keep a public Buttondown presence while the permanent version lives on the publisher's own site.

flowchart LR
    A[Canonical website issue] --> B[Buttondown campaign]
    B --> C[Subscribers]
    B --> D[Buttondown archive]
    D --> A

Buttondown pricing

Plan or tier	Price	Limits / notable features
Free	$0	First 100 subscribers
Up to 1,000 subscribers	$9/mo	Core newsletter features
Around 5,000 subscribers	$29/mo	Subscriber-based pricing
Around 10,000 subscribers	$79/mo	Subscriber-based pricing
Around 20,000 subscribers	$139/mo	Subscriber-based pricing
Tagging and segmentation	+$9/mo	Optional add-on
Analytics	+$9/mo	Optional add-on
RSS-to-email	+$9/mo	Optional add-on
Paid subscriptions	+$9/mo	Optional add-on
Multiple newsletters	+$29/mo	Optional add-on
Automations	+$29/mo	Optional add-on
Custom-domain archive	+$29/mo	Optional add-on
Whitelabeling	+$79/mo	Optional add-on

The modular pricing keeps a simple newsletter inexpensive, but stacking several add-ons raises the monthly cost quickly. Buttondown fits publishers whose website remains the authoritative copy and who want distribution plus a secondary public archive rather than a replacement for the website.

EmailOctopus

EmailOctopus takes the conventional email-marketing approach: subscriber management, forms, campaigns, templates, automations, analytics, and API access, without trying to become a publishing destination. For a website that already has its own CMS or static-site generator, that is usually exactly what is required.

Each sent campaign receives a public web version, but EmailOctopus does not automatically build a publication-style archive containing every issue. Its normal model is for the publisher to create the archive page separately.

EmailOctopus pricing

Plan	Price	Subscribers	Email allowance	Notable differences
Starter	$0	2,500	10,000/mo	EmailOctopus branding, reports retained 30 days, 1 form and 1 landing page
Pro monthly	$10/mo	Tier-based	Tier-based	Branding removed, permanent reports, unlimited forms and landing pages
Pro annual	From $9/mo	Tier-based	Tier-based	Same Pro features, 10% annual discount
Connect Starter	$0	2,500	Unlimited via own SES account	Amazon SES delivery, EmailOctopus branding
Connect Pro	From $8/mo annual	Tier-based	Unlimited via own SES account	Full design control and permanent reports

EmailOctopus Connect is the option for users already operating AWS infrastructure: EmailOctopus manages contacts and campaigns, while Amazon SES performs the actual delivery.

A sensible website-first archive structure is:

example.com/newsletter/
    2026-08/
    2026-09/
    2026-10/

with EmailOctopus responsible for sending those issues to subscribers. The hosted web copies are useful for "view in browser" links and sharing, but the permanent archive does not need to depend on them.

beehiiv

beehiiv treats newsletters as a publishing business rather than an email list. It combines email, a hosted website, public posts, subscription forms, recommendations, analytics, advertisements, paid subscriptions, digital products, and growth tooling, and the free tier already includes a public website with unlimited email sends.

That positioning makes beehiiv attractive when the newsletter itself is the publication. For an established publication with its own CMS, the same breadth creates overlap with the existing content system.

beehiiv pricing

beehiiv was in the middle of a plan-name transition in October 2026, so customers may see the newer Free/Lite/Pro naming or the previous Launch/Scale/Max terminology depending on account and renewal state.

Plan	Price	Subscriber capacity	Notable features
Launch / Free	$0	Up to 2,500	Unlimited sends, website, custom domain, analytics, recommendations, API access
Scale	From $49/mo monthly or $43/mo annual	Tier-based, up to 100,000	Automations, Ad Network, paid subscriptions, digital products, advanced analytics
Max	From $109/mo monthly or $96/mo annual	Tier-based	Branding removal, RSS-to-send, dynamic content, more publications, priority support
Enterprise	Custom	100,000+ / custom	SSO, dedicated infrastructure options, account management

Email sending is unlimited across the normal newsletter plans, so costs are driven mainly by subscriber count and feature tier rather than campaign volume. beehiiv charges a 0% platform fee on paid subscriptions on the relevant paid plans, although normal payment-processing fees still apply.

Kit

Kit, formerly ConvertKit, sits between creator marketing software and newsletter publishing. Its free tier is unusually generous: up to 10,000 subscribers with unlimited email broadcasts, landing pages, forms, tagging, segmentation, and API access, designed to keep audience growth inexpensive. Paid plans add more sophisticated automation and analytics.

Kit also provides public newsletter pages, which lets broadcasts function as web content rather than existing only inside inboxes. It makes the most sense when email subscribers are a core business asset and features such as sequences, segmentation, products, and creator-oriented automation matter.

Kit pricing

The prices below use the 1,000-subscriber tier and annual billing for the paid plans.

Plan	Price	Subscribers	Notable features
Free	$0	Up to 10,000	Unlimited broadcasts, forms and landing pages, tagging, segmentation, API
Creator	$33/mo annual	1,000 at quoted tier	Visual automations, sequences, integrations, RSS campaigns, branding removal
Creator monthly equivalent	About $39/mo	1,000	Same feature tier without annual discount
Pro	$66/mo annual	1,000 at quoted tier	Advanced analytics, subscriber scoring, referral system, unlimited users
Pro monthly equivalent	About $79/mo	1,000	Same feature tier without annual discount

Kit's paid prices increase with subscriber count.

Ghost

Ghost is a complete open-source publishing platform with newsletters built in: a website, CMS, memberships, newsletters, subscriptions, themes, APIs, and strong SEO controls. It can be self-hosted or purchased as the managed Ghost(Pro) service.

Unlike most hosted newsletter platforms, self-hosted Ghost is open source, and hosting, database operation, and bulk email delivery then become separate infrastructure costs. Ghost is strong on SEO architecture: posts can specify external canonical URLs, and themes can access that canonical URL directly, which allows a Ghost archive to function as an index to content whose authoritative copy lives elsewhere.

newsletter.example.net/
    Issue A -> example.com/newsletter/issue-a/
    Issue B -> example.com/newsletter/issue-b/

For organizations that need both a CMS and newsletters, Ghost is one of the most complete systems available. Deploying it only to send email, however, is unnecessarily heavy.

Ghost(Pro) pricing

The prices below are annual-billing equivalents, and Ghost(Pro) pricing also changes with audience size.

Plan	Price	Included members at entry tier	Notable features
Starter	$18/mo	1,000	Website, custom domain, newsletter, unlimited sends
Publisher	$29/mo	1,000	Custom themes, paid subscriptions, advanced analytics, integrations
Business	$199/mo	Higher limits	15 staff users, priority support, larger usage limits
Custom	Quote	Custom / unlimited	Dedicated IP, SLA, advanced configuration

Substack

Substack removes most infrastructure decisions. A publication gets email delivery, public posts, archives, subscriber management, comments, recommendations, community features, and paid subscriptions without a monthly platform charge, so its pricing model is fundamentally different from subscriber-tiered services.

Substack's public archive is central to the product: the platform expects its own web pages to be part of the publication rather than temporary copies of messages sent from another CMS. That is convenient for newsletter-first publishing, but it gives an external website less control over the public archive and the SEO architecture.

Substack pricing

Plan / usage	Cost	Notes
Free publication	$0	Unlimited free subscribers and publishing
Free newsletters	$0	No monthly platform fee
Paid subscriptions	10% of subscription revenue	Charged by Substack
Payment processing	Additional	Stripe processing charges also apply

This makes Substack extremely inexpensive for a large free newsletter. The economics change once significant paid subscription revenue is involved: a newsletter producing $10,000 per month in subscription revenue would pay approximately $1,000 per month to Substack before payment-processing fees.

MailerLite

MailerLite combines traditional email marketing with websites, landing pages, signup forms, and automation. Its free tier became substantially smaller in June 2026, dropping to 250 active subscribers and 2,500 email sends per month; in return, the free plan includes access to a broader set of design and publishing tools than many basic free newsletter products.

MailerLite provides web versions of campaigns and a newsletter archive component that displays previous campaigns on a MailerLite-hosted page. That makes its public archive functionality stronger than EmailOctopus, although MailerLite is still primarily an email-marketing platform rather than an independent publishing ecosystem.

MailerLite pricing

Plan	Price	Subscribers / sending	Notable features
Free	$0	250 subscribers, 2,500 emails/mo	3 automations, 3 forms, 1 landing page, 1 small website
Comfort	From $12/mo	Subscriber-tiered; monthly allowance is 10x tier size	More templates, dynamic email, auto-resend, preference center
Power	Subscriber-tiered	Unlimited sending under fair-use policy	Higher feature limits and more advanced functionality
Enterprise	Custom	200,000+ subscribers	Custom account arrangement

Brevo

Brevo is broader than a newsletter platform. Its product includes marketing email, transactional email, SMS, WhatsApp, automation, web push, sales tools, and customer-data functionality, and a newsletter is only one part of that platform. Its pricing is based primarily on sending volume rather than the number of stored contacts.

Brevo pricing

Plan	Price	Sending / contacts	Notable features
Free	$0	300 emails/day, up to 100,000 stored contacts	Campaigns, transactional email, basic automation and reporting
Starter	From $9/mo	From 5,000 emails/mo	No daily sending limit; multiple volume tiers
Standard	From $18/mo	From 5,000 emails/mo	Unlimited-contact automation, A/B testing, advanced reports
Professional	From $499/mo	From 150,000 emails/mo	Multichannel features, 10 users, advanced analytics
Enterprise	Custom	Custom	Enterprise controls and support

The free plan's 300-email daily limit is unusual. For a list with 250 subscribers it is perfectly usable; for a newsletter with 1,500 subscribers it becomes inconvenient because one full-list campaign cannot be delivered under the free daily allowance. Brevo becomes considerably more attractive when the same organization also needs application-generated email or multichannel marketing.

Sender

Sender competes more directly with EmailOctopus and MailerLite. It offers newsletters, forms, landing pages, automation, segmentation, and transactional email, with one of the more generous free allowances in this group.

Sender pricing

Plan	Price	Subscribers	Email allowance	Notable features
Free Forever	$0	2,500	15,000/mo	Automation, forms, landing pages, transactional email; Sender branding
Standard annual	From $7/mo	From 1,000	Roughly 12x subscriber tier/mo	Branding removal, SMS, more templates and experiments
Standard monthly	From about $9/mo	From 1,000	Tier-based	Same plan without annual discount
Professional	From $14/mo annual	From 1,000	Roughly 24x subscriber tier/mo	Advanced automation and reporting, priority support
Enterprise	Custom	Custom	Custom	Enterprise sending infrastructure

At the free tier, 2,500 subscribers and 15,000 monthly sends allow approximately six full-list newsletters per month. Sender therefore compares well with EmailOctopus on raw free capacity, with the tradeoff that it has fewer integrations and less of a publication-oriented ecosystem than platforms such as Kit or beehiiv.

Public archives: three ownership models

Archive functionality decides what happens to a multi-year back catalogue when the platform changes. Three models cover most cases.

Publication archive

Platforms such as Substack, beehiiv, and Ghost treat each newsletter issue as part of a public publication on the provider's domain:

provider.example/publication/issue-1
provider.example/publication/issue-2
provider.example/publication/issue-3

The provider effectively becomes both newsletter infrastructure and CMS.

Hosted campaign copies

Email-marketing services usually create a browser-readable version of an email after it has been sent:

email-provider.example/campaign/abc123

These URLs are useful for "view in browser" links and sharing. They should not necessarily be treated as the permanent content archive.

Website-owned archive

A publisher with an existing website can keep the permanent copies itself:

example.com/newsletter/issue-1/
example.com/newsletter/issue-2/
example.com/newsletter/issue-3/

The email provider then becomes replaceable infrastructure: changing newsletter vendors does not require changing historical URLs. For established websites this is the cleaner model, and it is the newsletter application of the POSSE principle, which favors the copy you control over the copy a platform controls. For a static site, the archive is regular site content, as in the Hugo deployment to AWS S3 guide.

Canonical URLs and newsletter SEO

If the complete issue appears both on an independent website and on a provider-hosted page, search engines can see two very similar documents. A canonical URL provides an explicit preferred version:

<link rel="canonical"
      href="https://example.com/newsletter/issue-42/">

Buttondown and Ghost are particularly strong for this architecture because both provide mechanisms for associating content with an external canonical URL. Other newsletter services tend to assume that either their own hosted page is the publication or that their web version is simply an email mirror.

This does not mean that every duplicate issue creates an SEO problem: search engines routinely consolidate duplicate or near-duplicate pages, and explicit canonical control simply makes the intended architecture clearer. Once the archive is in place, the self-hosted SEO tools comparison covers verification and monitoring, the WordPress SEO plugins comparison covers the metadata layer for WordPress-based sites, and IndexNow can tell search engines that a new issue has been published.

Custom sending domains and sender authentication

A serious newsletter should normally use an address under a domain controlled by the publisher:

newsletter@example.com

rather than:

examplepublication@gmail.com

A custom domain can be authenticated with SPF, DKIM, and DMARC and can remain stable when the newsletter platform changes, so infrastructure can move from one provider to another while readers continue to see the same sender identity. This section covers the sending side; for mailbox hosting on the same domain, see the hosted email comparison instead.

A practical setup sequence for a new sending domain:

Choose the sending domain, usually the main domain or a subdomain, and note it in the newsletter platform.
Create the sending address in the platform and record the DKIM selector it publishes.
Add the DNS records the platform provides: an SPF TXT record listing the sending service, the DKIM TXT record under the selector, and a DMARC TXT record at _dmarc.
Verify that each record resolves before sending:

   # SPF: which servers may send for the domain
   dig TXT example.com +short

   # DKIM: the key published under the platform's selector
   dig TXT selector1._domainkey.example.com +short

   # DMARC: the policy record for the domain
   dig TXT _dmarc.example.com +short

If a record is missing or mistyped, the platform's dashboard usually flags the domain as unverified, and sends either bounce or land in spam.

API access and automated publishing

For an occasional newsletter, manually creating a campaign is not a problem. For a technical publication, API access becomes useful because newsletter generation can be integrated into the normal content pipeline:

flowchart LR
    A[Markdown source] --> B[Static site generator]
    B --> C[Website]
    A --> D[Newsletter build step]
    D --> E[Newsletter API]
    E --> F[Subscribers]

The same source material can generate the permanent page, the newsletter excerpts, the campaign metadata, and the links without repeatedly copying content between systems. The send step is a plain HTTP call; endpoint paths and payload fields differ per platform, but the pattern is the same:

curl -sS -X POST "https://api.example.com/v1/campaigns" \
  -H "Authorization: Bearer $NEWSLETTER_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d @issue.json

This is one reason API-oriented platforms such as Buttondown, EmailOctopus, Kit, and Ghost remain attractive even when their visual editors are not the main selling point.

Choosing the architecture, not the editor

The platform choice reduces to one decision: which system owns the content, and which system merely delivers it. A newsletter without an existing website saves a considerable amount of setup by letting a publishing-first platform own the posts, the archive, and the subscriptions. An established website usually gets a simpler long-term architecture by keeping the permanent archive on its own domain and treating the provider as delivery infrastructure that can be replaced without touching historical URLs.

Whatever the split, three details are cheap to settle before the first send and expensive to unwind later: a custom sending domain with SPF, DKIM, and DMARC in place, an explicit decision about where each issue's canonical copy lives, and an export path that keeps the subscriber list in a standard format.