DEV Community: Amon Ochuka

Building a RAG Knowledge Assistant: Chunking, Embeddings, Vector Search, and Answers That Cite Their Sources

Amon Ochuka — Tue, 27 Oct 2026 21:23:00 +0000

Ask a general-purpose LLM about your company's refund policy and it will answer confidently. It will also be wrong, because it has never seen your policy.

That gap is why Retrieval-Augmented Generation (RAG) exists. Instead of hoping the model "knows" your data, you find the relevant text first and hand it to the model along with the question. The model stops guessing and starts reading.

Guidely is an internal knowledge assistant I'm building at Zone01 Kisumu, with a FastAPI backend and a React/Vite frontend. You upload company docs, it indexes them, and when you ask a question it returns a short answer with the sources it used. In this post I'll break down how the pipeline works, what each stage costs you, and where RAG systems quietly fail.

The Problem With Asking an LLM Directly

A bare LLM has two weaknesses for this job. It doesn't know your private documents, and when it doesn't know something it tends to produce a plausible answer anyway. For a support tool, a confident wrong answer is worse than no answer, because people act on it.

Pasting every document into the prompt isn't the fix either. Context windows are limited, long prompts are slow and expensive, and models are worse at finding a needle in a huge haystack than in a small one. You want to send the model only the few passages that matter.

Why Not Just Use Keyword Search?

Keyword search is the obvious shortcut, and it works until it doesn't:

Vocabulary mismatch breaks it. "How do I reset my password?" and "I forgot my login credentials" share almost no words, yet they ask the same thing.
It returns pages, not answers. Users still dig through results to find the relevant paragraph.
It can't explain itself in plain language. You get a list of links, not a readable answer.

Semantic search fixes the first problem, and an LLM fixes the other two. RAG combines both.

The Architecture

The design has two phases. Indexing happens offline, when documents are added or changed. Querying happens online, on every question.

Indexing:

Parse the uploaded file into raw text.
Chunk the text into small overlapping pieces.
Embed each chunk into a vector.
Store the vectors, plus the chunk text and file name, in a vector index.

Querying:

Embed the question with the same model.
Search the index for the top-k most similar chunks.
Generate an answer from those chunks and the question.
Return the answer together with the sources that were used.

The important property is that the model only sees what retrieval hands it. Everything downstream depends on that handoff.

Chunking: The Most Underrated Decision

Embedding a 40-page PDF as one vector produces mush. So documents are split into chunks, typically 500 to 1,000 tokens, with a small overlap between neighbours so a sentence cut at a boundary still has its context on the other side.

Chunk size is a trade-off. Too big, and retrieval gets vague because one chunk covers several topics. Too small, and each chunk loses the context it needs to make sense on its own. Splitting on natural boundaries like headings and paragraphs beats cutting at a fixed character count, because it keeps ideas intact.

Every chunk also keeps metadata: the file it came from and its position in that file. That metadata is what later lets the UI say exactly where an answer came from.

Embeddings and the Vector Store

An embedding turns text into a list of numbers so that similar meaning means nearby vectors. That is what lets the password question match the login question despite sharing no words.

For storage, Guidely uses FAISS: it runs locally, needs no account or external service, and is fast. With normalized vectors and inner-product search, the score is cosine similarity. An exact flat index is perfectly fine up to hundreds of thousands of chunks. Approximate indexes matter later, and hosted options like Pinecone or Weaviate make sense once you need managed infrastructure. Start with the simplest thing that works.

One rule matters here: the question and the chunks must be embedded with the same model. Vectors from different models live in different spaces and can't be compared.

Generating a Grounded Answer

The "G" in RAG is where you enforce honesty. The prompt gives the model the numbered chunks and a strict set of instructions:

Answer using only the provided context.
Cite sources by number.
If the context doesn't contain the answer, say you don't know.

Each of those does real work. "Only the context" reduces hallucination. Numbered sources let the model cite, and let the UI map each citation back to a real file and snippet. The "I don't know" escape hatch stops the model from inventing an answer when retrieval fails. Setting the temperature to zero keeps the answers stable and reproducible.

The API then returns clean JSON: the answer, plus the list of sources with file names and snippets. Always showing sources turns "trust me" into "check for yourself."

Things That Bite You in Production

Re-embedding everything. Embedding costs money and time, and documents rarely change. Hash each file's content at index time and skip it if the hash is unchanged. Log cache hits against new embeddings. If you re-ask a question and the embedding count goes up, something is broken.

Exact terms and IDs. Pure vector search is weak on error codes, product names, and identifiers, which is exactly where keyword search is strong. Hybrid search runs both and merges the rankings, and it is often the single biggest quality upgrade you can make.

Coarse ranking. Vector search is fast but approximate. A common upgrade is to retrieve a wide net, say the top 20, then use a cross-encoder reranker that reads the question and each chunk together to pick the best few. You trade a little latency for better precision.

Follow-up questions. A user asks "what about for contractors?" and retrieval has nothing to search for. Rewrite follow-ups into standalone questions before embedding them, using the conversation history.

Prompt injection. Your documents become part of the prompt, so a document containing "ignore previous instructions" is an attack vector. Keep system instructions separate from retrieved content, label the context clearly as data, and never let retrieved text trigger actions without validation.

Failure states. Real systems hit empty queries, missing API keys, corrupted files, model timeouts, and searches that find nothing. Return proper 4xx and 5xx responses with clear messages, and show friendly error states in the UI. "No relevant documents found" is a good answer. A confident hallucination is the worst one.

How Do You Know It Works?

"It seems to work" isn't a metric. Guidely tracks a small set of checks, some logged automatically by the backend and some validated by hand:

Retrieval@3 (manual): for 10 to 20 questions whose answers exist in the sample docs, how often does the correct passage appear in the top 3 chunks? Target: 80% or more.
Answer reference coverage (manual): does every answer cite at least one valid source? Target: 90% or more.
Source precision (manual): does the highlighted snippet actually support the answer? Target: 80% or more.
Latency (auto-logged): time from question to final JSON. Target: median under 3 seconds and p95 under 5 seconds with a warm cache.
Embedding cache effectiveness (auto-logged): repeated queries on unchanged docs should cause zero re-embedding.

The most useful habit is evaluating retrieval separately from generation. If the right chunk isn't in the top 3, no prompt will save you. Check retrieval first, then check the answers, and when quality drops you'll know which half of the pipeline to blame.

Lessons Learned

Retrieval quality is the ceiling. The LLM can only be as good as the context you hand it.
Chunking and prompts matter more than model choice. Upgrading the model rarely fixes a bad pipeline.
Test embedding and retrieval before the UI. Wire up the interface last, once search returns the right chunks.
Cache from day one. Hash-based skipping is easy to build early and painful to retrofit.
Always show your sources. Citations are what make the tool usable by real people.
Log everything cheap. Latency, cache hits, retrieved chunk IDs, and error counts make debugging far easier.

Conclusion

RAG isn't magic. It's search plus a well-instructed reader, and once you see it that way, every improvement becomes an ordinary engineering problem: better chunks, hybrid search, reranking, query rewriting. If you're building your first one, start small: five documents, FAISS, and a strict prompt. Get retrieval working and measured before you touch anything else, and the rest of the system gets much easier to reason about.

Building a Secure Auth System in Go: JWT, 2FA, OAuth, Sessions, and Rate Limiting

Amon Ochuka — Tue, 27 Oct 2026 21:17:29 +0000

Authentication seems simple at first glance. Then you add "remember me", OAuth, 2FA, and proper session revocation, and it gets complicated fast.

Wapi is a live streaming app tailored for African content, with a backend written in Go using Gin. Its auth system covers many production concerns, from token rotation to two-factor login. In this post I'll walk through the pieces, why each one matters, and the mistakes that are easy to make along the way.

The Token Strategy: Access and Refresh Tokens

Wapi uses the classic dual-token approach. Short-lived access tokens authenticate API requests, and longer-lived refresh tokens obtain new access tokens without forcing the user to log in again.

The login endpoint returns both tokens along with the user profile. When the access token expires, the client calls the refresh endpoint, which issues a new access token and a new refresh token.

That second part is the important one: refresh token rotation. Every time a refresh token is used, it is invalidated and replaced. This limits the damage if a refresh token is stolen, because the thief and the real user can't both keep using the same token.

Rotation also gives you theft detection. If someone presents a refresh token that has already been rotated, one of two things happened: a client retried badly, or a token was copied. Either way, the safe response is to revoke the entire session the token belonged to and force a fresh login.

JWTs Done Right

JSON Web Tokens are convenient and easy to misuse. A few rules go a long way:

Keep access tokens short-lived. Minutes, not hours. The shorter the lifetime, the smaller the window an attacker has with a stolen token.
Sign properly. Use a strong secret (HMAC) or an asymmetric key pair (RSA or ECDSA). Pin the expected signing algorithm when parsing, so a token can't claim a weaker one.
Never put sensitive data in the payload. A JWT is signed, not encrypted. Anyone holding the token can read its claims.
Validate everything. Check the signature, the expiry (exp), and the issuer and audience where they apply.
Store minimal claims. Usually just the user ID, a token ID (jti), and the expiry. Don't bloat the token, since it travels with every request.

In a Gin application this lives in middleware. It extracts the Bearer token from the Authorization header, validates it, and attaches the user ID to the request context so downstream handlers never have to parse tokens themselves.

There is one trade-off to be honest about. A pure JWT is stateless, so the server can't "un-issue" it before it expires. That is why access tokens must be short-lived, and why real revocation happens at the refresh token and session level. If you need instant revocation of access tokens, for example after a ban, you can check the token's session ID against a store on each request. That reintroduces state, so it's a deliberate trade of some of JWT's simplicity for control.

Session Management

Tokens alone aren't enough. Users and the system both need to be able to see and end sessions. Wapi exposes endpoints to list active sessions, revoke a specific session, and revoke all sessions at once.

This matters for several scenarios:

"Sign out of all devices" after a lost phone or a password change
Killing suspicious sessions once a login looks wrong
Meeting security expectations that users increasingly take for granted

To support it, you track each refresh token (or session ID) in the database along with metadata: device or user agent, IP address, creation time, and last-used time. Validating a refresh token then means checking that it exists and hasn't been revoked, and logging out means revoking it. The metadata is what turns a bare list of tokens into something a user can understand, such as "Chrome on Windows, last active two hours ago".

Store a hash of the refresh token rather than the token itself, the same way you would treat a password. If the database leaks, the stored values shouldn't be usable.

Two-Factor Authentication (TOTP)

Adding 2FA is one of the biggest security upgrades you can make. Wapi implements TOTP (Time-based One-Time Passwords), which works with Google Authenticator, Authy, 1Password, and similar apps.

The flow has four stages:

Setup. An authenticated user requests 2FA setup. The server generates a secret and returns an otpauth:// URI, which the client renders as a QR code. The secret is stored (encrypted) against the user but not yet active.
Verification. The user scans the QR code and enters the code their app displays. The server checks it against the stored secret, and only then switches 2FA on. This confirms the user's app is configured correctly before they can lock themselves out.
Login with 2FA. When 2FA is enabled, a correct email and password no longer produce tokens. The server returns a short-lived challenge instead, and the user completes login by submitting a TOTP code. Only then are access and refresh tokens issued.
Disabling. Turning 2FA off requires re-authentication, so someone with a briefly unattended session can't quietly weaken the account.

Two details deserve attention. First, the login challenge should be a single-purpose credential: short expiry, usable only to complete 2FA, and never accepted as an access token. Second, 2FA verification needs its own strict rate limit. A six-digit code has only a million possibilities, and brute-forcing that is realistic if attempts are unthrottled.

Finally, plan for account recovery. Users lose phones, so you'll want backup codes, generated at setup, stored hashed, and single-use.

OAuth 2.0 (Google and Facebook)

Social login is convenient for users and has its own pitfalls. Wapi uses the standard authorization code flow:

The client hits the server's login endpoint for a provider, which redirects to the provider's authorization page.
The user authenticates with the provider and consents.
The provider redirects back to the server's callback endpoint with an authorization code.
The server exchanges the code for tokens with the provider and fetches the user's profile.
The server finds or creates a local account linked to that provider identity.
The server issues its own access and refresh tokens.

Step 6 is where Wapi does something smart. Instead of putting tokens in the redirect URL, it redirects the browser back to the frontend with a short-lived one-time code, and the frontend exchanges that code for tokens through a separate endpoint. URLs end up in browser history, server logs, and Referer headers, so tokens don't belong there. A code that is single-use and expires in seconds is a much safer thing to leak.

A few OAuth details are easy to miss:

Validate the state parameter to protect against cross-site request forgery on the callback.
Use PKCE where the provider supports it.
Be careful when linking accounts by email. Only auto-link to an existing local account if the provider confirms the email is verified. Otherwise an attacker can register at a provider with someone else's address and take over their account.

Rate Limiting for Abuse Prevention

Auth endpoints are prime targets for credential stuffing and brute-force attacks. Wapi applies different limits depending on the endpoint:

Endpoint type	Limit	Keyed by
Signup, login, password reset	10 per minute	IP
OAuth initiation	5 per minute	IP
2FA verification (suggested)	Stricter than login, for example 5 attempts per 15 minutes	IP and user or session

The limiter is built on golang.org/x/time/rate, with buckets stored per key. When a limit is hit, the API returns 429 Too Many Requests with a clear error code so clients can back off gracefully.

An in-memory limiter is fine for a single instance, but note what happens when you scale out. Each instance counts separately, so your effective limit is multiplied by the number of instances. When you run more than one, move the counters to Redis, using a sliding window or token bucket, so every instance shares the same view. Also remember that IP-based limits are blunt: users behind a shared NAT share a bucket, and attackers rotate IPs. That matters even more for a mobile-heavy audience, where carrier-grade NAT can put many real users behind one IP. That is why the 2FA limit also keys on the user or session.

Security Best Practices to Keep in Mind

Password hashing. Never store plaintext passwords. Use bcrypt, argon2, or scrypt with a sensible cost factor, and raise it over time as hardware improves.
Email verification. Verify addresses before allowing sensitive actions.
Password reset. Use time-limited, single-use tokens delivered by email, and invalidate existing sessions when a reset completes.
Account enumeration. Return the same response whether or not an email exists, for login errors and for password reset requests.
CORS. Lock allowed origins down to the ones you actually use.
Secure headers. Add standard security headers in production.
Token storage on the client. HttpOnly, Secure, SameSite cookies protect tokens from theft via XSS in a way localStorage can't. The trade-off is that cookie-based auth needs CSRF protection, which SameSite helps with. If you send tokens in an Authorization header instead, keep them in memory and accept the refresh-on-reload cost.

Putting It All Together

A robust auth system isn't about implementing every feature at once. It's about layering defenses. Start with solid password authentication and short-lived JWTs. Add refresh token rotation and session management so users can end sessions remotely. Layer on 2FA for accounts that matter, add OAuth for convenience, and wrap everything in rate limiting.

What stands out in Wapi's implementation is that these pieces are wired up and work together rather than existing as isolated features. The routes are organized logically, the middleware is reusable, and each flow closes the gaps the previous layer leaves open. It's a solid reference for anyone building auth in Go.

Scaling Real-Time Stream Chat in Go: In-Memory Hubs, WebSockets, and Redis Pub/Sub

Amon Ochuka — Tue, 27 Oct 2026 21:10:33 +0000

Live stream chat is deceptively hard. What starts as "just broadcast messages to connected clients" becomes a distributed systems problem the moment you scale past a single server instance.

Wapi is a live streaming app tailored for African content, with a backend written in Go. Its chat takes a pragmatic approach to scaling: an in-memory WebSocket hub per process, with an optional Redis Pub/Sub broadcaster for cross-instance fan-out. In this post I'll break down why the design works, what it costs you, and where it starts to strain.

The Problem With Single-Instance Chat

If you only ever run one backend instance, keeping all connected clients in memory and broadcasting to them directly is the simplest and fastest option. You get low latency, no external dependencies, and a system that is easy to reason about. Go is well suited to this because goroutines and channels make managing thousands of concurrent connections cheap.

In production, though, you'll almost always want multiple instances behind a load balancer, for redundancy, for rolling deploys, and for capacity. With a pure in-memory hub, a client connected to instance A never receives messages sent by a client connected to instance B. Your "live" chat fragments based on which instance each user was routed to, and users in the same stream see different conversations.

Why Not Just Use Sticky Sessions?

The obvious shortcut is to route every viewer of a given stream to the same instance. It can work, but it has real drawbacks:

Hot streams break it. A popular stream can have more viewers than a single instance can hold, and you can't split it.
Deploys get painful. Restarting an instance drops every connection pinned to it, and all those clients reconnect at once.
Load balancing gets complicated. You need routing logic that understands stream IDs, not just connections.

Fan-out between instances removes all three problems. Any instance can serve any viewer, and the instances keep each other informed.

The Architecture

The design is a hybrid. Each instance runs its own hub that manages WebSocket connections and broadcasts locally. If REDIS_URL is configured, each instance also publishes its messages to Redis and subscribes to messages from other instances. A message arriving from Redis is re-broadcast to local connections, but only if it originated on a different instance, so it never echoes back.

This gives you:

Local speed: no network hop between clients on the same instance
Horizontal scalability: add instances without breaking chat
Graceful degradation: without Redis, it falls back to single-instance mode
Simplicity: no dedicated message queue, just Redis Pub/Sub

How the In-Memory Hub Works

The hub's job is to track which clients are watching which stream, and to deliver each message to everyone watching that stream. Internally it is a map from stream ID to a set of connected clients.

The important design choice is how that map is protected. Instead of guarding it with locks, the hub runs in a single goroutine and receives all changes through channels: one for clients registering, one for clients leaving, and one for messages to broadcast. Because only that goroutine ever touches the map, there are no data races and no lock contention, and the event loop is easy to follow.

Each client gets its own buffered outbound channel, drained by a dedicated write goroutine that pushes messages down the WebSocket. A matching read goroutine handles incoming messages. This read/write pump pair per connection is the standard pattern for WebSockets in Go, and it keeps slow network writes off the hub's critical path.

The Slow Client Problem

Every real-time system eventually meets the slow consumer: a client on a bad mobile connection whose outbound buffer fills up. If the hub blocks while delivering to that client, one bad connection stalls chat for everyone in the stream.

For an audience that is largely on mobile data, this isn't an edge case, it's the normal case.

The fix is to never block. If a client's buffer is full, the hub drops that client, closes its channel, and lets the client reconnect. It sounds harsh, but it protects the many from the few. Combined with history loading on reconnect, the user barely notices.

Adding Redis for Cross-Instance Fan-Out

On startup, each instance generates a unique instanceID, for example with uuid.NewString(). When it publishes a message to Redis, it attaches this ID to the payload. When it receives a message from Redis, it checks the ID first:

Same ID: the message came from this instance, and local clients already got it directly from the hub. Ignore it.
Different ID: another instance produced it. Hand it to the local hub for delivery.

Without that check, every message would be delivered twice on the instance that sent it.

A design choice worth noting is how channels are laid out. Publishing to one channel per stream (something like chat:stream:<id>) and subscribing with a single pattern subscription keeps the subscriber code simple. Each instance holds one subscription rather than managing a subscribe and unsubscribe for every stream as viewers come and go.

One more detail matters for correctness: the Redis listener should not write to the hub's client map directly. It should send messages into the same broadcast channel that local messages use, so the single-goroutine guarantee holds.

Why Pub/Sub and Not Something Heavier

Redis Pub/Sub is fire-and-forget. If a subscriber is disconnected when a message is published, that message is gone for that subscriber. For payments that would be unacceptable. For live chat it is usually fine, for three reasons:

Chat is ephemeral by nature, and a dropped message during a brief blip is rarely noticed.
Every message is also persisted to the database, so history is complete even if live delivery isn't.
The operational cost is tiny compared with running Kafka or a full message broker.

If you later need replay, consumer groups, or delivery guarantees, Redis Streams, NATS JetStream, or Kafka are the natural next steps. Start simple and upgrade when a real requirement forces you to.

Putting It Together

The full flow looks like this:

Client connects via WebSocket to something like /api/v1/streams/:id/ws, with auth.
Connection is upgraded and the client is registered with the local hub for that stream.
Client sends a message. The service layer validates it (rate limit, ban check, permissions), persists it to the database for history, then pushes it to the local hub and publishes it to Redis with the instance ID.
Other local clients receive it instantly from the hub.
Other instances receive it from Redis, see a different instance ID, and broadcast it to their own local clients.

Things That Bite You in Production

Ordering. Within a single instance, messages are delivered in the order the hub processes them. Across instances there is no global ordering guarantee. Two users on different instances can send messages at nearly the same moment, and viewers may see them in different orders. For most chats this is acceptable, but if it matters, assign ordering on the server using database timestamps or sequence IDs and let clients sort on that.

Authentication and origin checks. WebSocket upgrades bypass a lot of the protections you rely on for normal HTTP requests. Validate the token during the upgrade, check the Origin header, and re-check permissions when bans or role changes happen mid-session, not just at connect time.

Reconnection storms. When an instance restarts, every client it held reconnects at once. Clients should use exponential backoff with jitter so they don't all retry in lockstep and overwhelm the remaining instances.

Heartbeats. Half-open connections are common, especially on mobile. Use WebSocket ping/pong with read deadlines so dead connections get cleaned up rather than lingering in the hub's map and leaking memory.

Message size and rate limits. Cap payload sizes and rate limit per user, for example 5 messages per second. Stream chat gets hot fast, and one spammer can saturate both the hub and Redis.

Observability. Track connected clients per instance, messages published and received per second, dropped slow clients, and Redis publish errors. When chat feels laggy, these numbers tell you whether the problem is the hub, the network, or Redis.

Lessons Learned

Filter your echoes. The instance ID check is non-negotiable. Without it, you get duplicate messages.
Never block the hub. Drop slow clients rather than letting them stall everyone else.
Persist for history. WebSockets are ephemeral. Write every message to your database so clients can load history on reconnect.
Don't let Redis block chat. If a publish fails, log it and carry on. Local delivery should never depend on Redis being up.
Plan for disconnects. Networks drop. Build reconnection with backoff into the client from day one.
Know the limits. Pub/Sub doesn't persist messages, and every instance receives every message for the streams it subscribes to. That is fine at moderate scale, but at very high volume you'll want to shard channels or move to a different transport.

Conclusion

The hybrid in-memory plus Redis approach is a sweet spot for most applications. It keeps the fast path local, scales horizontally with one small dependency, and degrades gracefully when Redis is unavailable. It doesn't overcomplicate your architecture before you need it, and the upgrade path to Streams or a full broker stays open. If you're building real-time features in Go, this pattern is worth stealing.

Why AI Can't Save You From Not Understanding Your Own System

Amon Ochuka — Sat, 18 Jul 2026 17:01:40 +0000

System design matters more than most people realize, and most of us practice it every day without noticing.

AI has made building software faster than it's ever been. It can scaffold a project in seconds, generate a CRUD layer, explain a concept you're fuzzy on, and get you from idea to working prototype in minutes. If you're learning to code right now, this is an incredible advantage that developers five years ago simply didn't have.
But there's a quiet trap hiding inside that speed: AI is only as useful as your ability to follow what it built.
This week I ran into a bug that made that lesson very concrete.

The Bug That Wasn't Where It Looked

I was working on an escrow platform's backend — a Go API with layered architecture: handlers, services, repositories, middleware. One endpoint kept returning a plain 400 Bad Request, no error body, nothing useful in the response.
The obvious place to look was the endpoint itself. The route existed. The handler looked correct. The service logic seemed fine. I stared at that one function for longer than I'd like to admit, because that's where the symptom was showing up.
The actual problem wasn't there at all.
It turned out a stale access token from an earlier test was silently failing verification before the request ever reached my handler. The middleware sitting in front of the route was rejecting it, quietly, without the kind of error message that would've pointed me anywhere useful.
The fix wasn't in the endpoint. It was in understanding the full path a request takes through the system — middleware, then handler, then service, then repository,and tracing it one layer at a time, adding a log line at each boundary until I found exactly where execution stopped.

Why AI Couldn't Just Solve This For Me

Here's the part I want to sit with, because I think it's the actual lesson, not just an anecdote.
If I'd asked an AI assistant "why is this endpoint returning 400," it could reason about the code I showed it,and it did, correctly, in this exact case. But it could only reason about what I gave it. It doesn't have a live view of my running server, my actual request/response cycle, or my database state. It can't run my curl command and watch what happens.
It can suggest the most likely causes based on patterns, but it can't independently verify which one is actually true in my specific running system.
That verification step — the actual debugging — still has to happen inside your own understanding of how the pieces fit together. AI can help you form hypotheses faster. It cannot replace tracing the real, live behavior of your own application.

This is true at a small scale (one 400 error) and it's even more true at the system design level. AI is excellent at generating a plausible-looking architecture — a service layer here, a repository there, a middleware chain that looks clean on paper. But plausible and correct for your actual system are not the same thing. If you don't understand why the architecture is shaped the way it is, you won't notice when it's subtly wrong for your use case, and you definitely won't be able to debug it when something breaks in the seams between the pieces AI generated separately.

System Design Is Something You Already Practice

Here's the thing I think gets missed: system design isn't some separate, advanced discipline you unlock later. If you've ever asked yourself questions like:

"Should this validation live in the handler or the service?"
"What happens if this database call fails halfway through?"
"Does this component need to know about that one, or should they stay decoupled?"

— you were already doing system design. Most developers do this daily, in small decisions, without ever labeling it that way. AI can generate answers to these questions for you. But if you don't understand why one answer is better than another for your specific system, you're not actually equipped to catch it when the AI's answer is wrong, or right in general but wrong for your case.

A Concrete Example: Layered Validation

To make this less abstract, here's a real pattern from the same project. I have three related resources — a deal, an artifact belonging to that deal, and a verification belonging to that artifact.
Early on, nothing stopped a client from requesting artifact ID from deal A while passing deal ID for deal B in the URL — the IDs would resolve independently, and as long as each one individually existed in the database, the request would "succeed," even though the artifact didn't actually belong to that deal.

The fix wasn't complicated once it was understood:

if artifact.DealID != dealID { return nil, ErrArtifactNotFound }

Every layer that owns a nested resource needs to check that the parent chain actually matches, not just that each individual ID exists somewhere in the database. This is the kind of bug that's easy for AI to catch once you point at it directly, but it's very easy to miss in the first pass if you're not thinking about your system's ownership hierarchy at all, and just asking AI to "add an endpoint to get a verification by ID."
The endpoint worked. It just wasn't correct, and "works" and "correct" diverge exactly at the boundaries AI can't see unless you show it the full picture.

So What Does This Actually Mean Day to Day?

I don't think the answer is "don't use AI for architecture", that would throw away a genuinely useful tool. I think the answer is closer to this:
Use AI to move faster, not to move blind.
Concretely, that's looked like:

Asking why a suggested pattern works, not just accepting that it does
Deliberately tracing a request through every layer it passes through, even when things are working, so I actually know the shape of my own system
Treating a bug as a chance to understand the architecture better, not just to patch the symptom
Writing my own validation logic (like the ownership check above) by reasoning through the actual failure case, rather than only generating code and hoping it's complete

None of this is about distrusting AI. It's about recognizing that AI's usefulness for system design has a hard ceiling: it can only be as good as the mental model you bring to the conversation. If you don't understand your own system, you can't tell a subtly wrong suggestion from a genuinely good one, you can only tell "does this compile" from "does this error," and that's a much weaker signal than actually knowing your architecture.

So remember...
You don't need to memorize every line of code you write, and you shouldn't try to. But you do need to understand how your system flows — request in, through which layers, touching which data, back out — because that's the only thing that lets you debug with confidence instead of guessing, and it's the only thing that lets you actually evaluate whether an AI-suggested design is right for your system, not just right in general.
That's what turns building software from assembling suggestions into actually engineering something.

The Rise, Pause, and Rise of CRUD Apps

Amon Ochuka — Wed, 27 May 2026 07:19:46 +0000

In the early days of the web, websites were mostly digital brochures. They were static HTML pages. You could read them, but you couldn't interact with them. If a business wanted to change a price, a developer had to manually edit code and re-upload the file via FTP.
It was slow, expensive, and didn't scale.

Then came the legendary LAMP stack (Linux, Apache, MySQL, PHP). Suddenly, the web became dynamic. The industry quickly realized that almost every business problem on earth;whether running a bank, an online store, or a social network,boiled down to four fundamental actions:

Create: Letting a user sign up or post a status.
Read: Showing a user their profile or a product list.
Update: Letting a user change their password.
Delete: Letting a user close an account.

By the mid-2000s, frameworks like Ruby on Rails, Django, and Laravel arrived. They turned CRUD into a science. With a single command line prompt (rails generate scaffold), a developer could spin up a fully functioning database table, back-end logic, and front-end forms in seconds.
This was the first great rise. CRUD apps became the undisputed infrastructure of the global economy. They weren't flashy, but they were predictable and deeply tied to the absolute truth of the database.

The Pause: The "CRUD is Boring" Mirage
Then, things got weird. Around 2015, the developer community collectively decided that simple, server-rendered CRUD apps were "boring." We entered the era of massive Single Page Applications (SPAs), microservices, GraphQL, and serverless hype.We started treating the database like something to be hidden behind ten layers of abstraction. Instead of a simple PHP or Rails script fetching a row from MySQL, we introduced a mountain of complexity:

The SPA Takeover: We moved the entire application logic to the browser. Suddenly, a simple user profile required React, Redux for state management, Webpack for bundling, and a massive JavaScript file that took 5 seconds to load on a mobile phone.
The API Chasm: We separated the front end and the back end entirely. Now, to display a simple list of products, you needed a REST or GraphQL API layer, complex CORS configurations, and token-based authentication (JWT) just to talk to your own database.
Microservice Madness: We broke monolithic apps into dozens of tiny services. A basic update operation now had to hop across three different servers, manage distributed transactions, and use Kafka queues just to change a user's email address.

We spent a decade over-engineering the front end just to realize that users still just want to fill out a form and hit submit. We swapped reliable server logs for confusing JavaScript console errors. We paused the simplicity of CRUD, and in return, we got total JavaScript fatigue.

The Crypto & Blockchain Detour: An Append-Only Trap
As the web-development world was over-engineering the front end, the Web3 and blockchain movement emerged, promising to decentralize the entire internet. Enthusiasts claimed that the "shared ledger" would replace traditional databases entirely.

But reality caught up quickly.

A blockchain is fundamentally an append-only ledger. In CRUD terms, it is brilliant at Create and Read, but terrible at Update and Delete.

Because data on a blockchain is immutable, it is incredibly slow and quite expensive for everyday operations. Worse, it creates a massive legal nightmare. Privacy frameworks like Europe's GDPR mandate that users have the "right to be forgotten"; a legal requirement to Delete data. Because you cannot delete data from a block, tech companies quickly realized they couldn't run a compliant business on a blockchain alone.

The blockchain didn't kill the database; instead, decentralized applications (dApps) still had to be wrapped in traditional CRUD architectures to cache data, update states quickly, and manage local user profiles efficiently.

The Second Rise: AI and the Return to Absolute Truth
Following the blockchain hype came the AI boom. Large Language Models (LLMs) can write code, analyze data, and build entire workflows in seconds. Prominent tech voices started declaring that apps were dead, and the era of the user interface was over.

But once again, the industry hit the same foundational wall: AI is probabilistic, but business is deterministic.

AI can write poetry, but it still needs a SQL database to know where to ship your package. An LLM cannot guess a bank balance. It cannot "predict" a shipping address. AI models are stateless; they don't know anything for sure; they just guess the next most likely word based on probability.

Tech is the User, CRUD is the Foundation
This realization has triggered the second rise of the CRUD app. We aren't throwing CRUD away; we are realizing it is the guardrail that keeps both AI and Web3 tethered to reality.
The relationship has completely flipped: AI and blockchain are no longer trying to replace the CRUD app; they are becoming the users of it.

When an AI agent "cancels your subscription," it executes a standard Update or Delete command via a regular API to a traditional database.

When a smart contract processes a transaction, a traditional CRUD system reads the event to update a user's dashboard instantly.

The Ultimate Backup
Most importantly, CRUD apps are our absolute fallback. AI models hallucinate, prompt injections corrupt logic, and decentralized networks suffer from latency or high gas fees.
When the advanced tech stack glitches or goes offline, humans cannot be left staring at a broken terminal. We need a direct, unabstracted, deterministic interface to view, edit, and fix the underlying data manually.

We need a CRUD app as our ultimate backup.

Conclusion: Boring is Beautiful Again
The tech industry went on a decade-long detour. We ran away from CRUD because it felt too simple, hiding it behind over-engineered JavaScript frameworks, distributed ledgers, and autonomous AI.

But the loop has closed. We are entering an era where "boring" software is beautiful again. Frameworks like Next.js, Remix, and HTMX are booming because they bring us back to simple, server-rendered data management.

No matter how smart the AI gets or how decentralized the ledger becomes, the world will always need a rock-solid database to store the absolute truth. And humans will always need a CRUD app to make sure that the database is backed up.

Long live the CRUD app.

Kiboko to Sherehe(polite): From Strict Go Rules to JavaScript Chaos

Amon Ochuka — Thu, 07 May 2026 17:53:56 +0000

Growing up in a typical strict African household always had a very clear structure, well at least for me.
There were rules. Not suggestions. Not “best practices.” Rules. You never really negotiated them, you just executed them.

You woke up early. You went to school. You came back on time. You didn’t always get the chance to “try things.” You mostly followed instructions.

If you messed up, there was usually a very immediate feedback loop.
Let’s just call it… real-time debugging. ("Kiboko")

Now fast forward to adulthood (or campus life, which feels like a soft launch into adulthood).
Suddenly:

No one is asking where you are
You can sleep at 3AM and call it “freedom”
Food is no longer provided;it’s a budgeting problem
And somehow, people are going out on weekdays like it’s normal("sherehe polite")

It feels like moving from a tightly controlled environment into an open-world sandbox with no manual.

And that’s exactly how learning JavaScript felt after coming from Go.

Go felt like home

Go is structured, strict, and predictable.It feels like that kind of upbringing where:

There is a correct way to do things
Mistakes are immediately pointed out
You don’t get “creative freedom” with rules
Everything has a clear, disciplined path

If you write something wrong, Go doesn’t argue with you.
It simply refuses to proceed.
Very “strict parent energy.”

JavaScript feels like campus life

Suddenly, everything is allowed.
Almost too allowed.

You can write things that technically work but feel emotionally wrong:

"5" + 1 // "51"
"5" - 1 // 4

And the language just… accepts it.
No questions. No intervention. No moral guidance.(This is where I would have been properly debugged)

Coming from Go, it feels like moving from mama's hot home-cooked meals, to a hostel where someone is boiling noodles in a sufuria at 2AM, while watching YouTube at full volume.

It works.But you’re not always sure it should.

The adjustment phase

At first, JavaScript feels messy.
Too flexible. Too forgiving. Too many ways to do the same thing.
But slowly, something changes.
You realize the language isn’t enforcing discipline on you, you’re expected to build it yourself.

That’s the shift.Go gives you structure upfront.
JavaScript forces you to learn structure through experience.

One is guided discipline.The other is learned discipline

Final thoughts

Growing up strict teaches you order.Campus life teaches you independence.
Go gives you rules.
JavaScript gives you freedom, and occasionally lets you suffer for bad decisions.

And maybe that’s the real lesson:
Structure is important, but so is learning how to create it for yourself when no one is watching.

Why AI Is Unlikely To Be A Game-Changer In African Economies

Amon Ochuka — Tue, 17 Feb 2026 07:34:27 +0000

Everyone is talking about AI transforming Africa, but the reality on the ground is very different, especially in countries facing electricity shortages and monopolies in key sectors. While AI has enormous potential globally, in many African contexts it is unlikely to become the economic miracle some people hope for, at least in the near term.

Electricity Shortages Are a Major Bottleneck

AI systems, whether cloud-hosted or on-device, consume a significant amount of electricity. Training even a single large AI model can use as much power as a small town in a month. Many African countries struggle with unreliable grids and frequent load shedding. Even highly skilled developers cannot run AI consistently without stable electricity, and relying on generators or solar power adds both cost and operational complexity, making many AI projects financially unviable.

Infrastructure Needs Go Beyond Power

AI requires more than electricity. Fast internet, reliable data storage, cloud computing, and ongoing maintenance are essential. Many regions still lack basic connectivity, and data centers are often concentrated in a few major cities. This makes AI adoption uneven and prevents smaller startups from scaling their solutions effectively.

Monopolies Stifle Innovation

Key sectors like telecommunications, banking, and energy are dominated by a few large players. Small AI startups often struggle to access APIs or market opportunities, as monopolies prioritize existing profit models over risky new technologies.

Talent Gaps Limit Scale

AI requires skilled engineers and data scientists. While Africa has a growing developer community, high-level AI talent remains limited, and access to training and compute resources is uneven, slowing adoption.

Localized Opportunities Exist

AI can succeed where infrastructure is reliable, such as mobile money in Kenya or agri-tech in parts of Nigeria. These are notable exceptions, not the rule.

Conclusion

AI is a powerful tool, but tools don’t operate in a vacuum. Without reliable electricity, fair competition, and solid infrastructure, AI is unlikely to drive broad economic growth. Prioritizing fundamental challenges should come first, before expecting AI to revolutionize economies.