DEV Community: John Eliud Odhiambo

Where Were the Logs? Building a Log Ingestion Engine That Never Silently Drops a Log

John Eliud Odhiambo — Thu, 29 Oct 2026 16:13:18 +0000

Three hours were lost in this example scenario. Two hours were spent on root cause analysis (where an engineer would log onto each server, open up various log files, and search manually). There was no error reported from any of the applications on the monitoring dashboard, meaning that no panels highlighted the service that was failing. There was also no clear indication of which service was failing the most. The compliance team wanted to see audit trails, but there were none to provide.

The brief is the running example I built this project from, and I later published the project as a repository.

TL;DR: Go service which receives batched JSON on /logs, validates at field level, enriches and routes to three SQLite destinations via RabbitMQ with a bounded in-memory fallback. Admission is durable via a write-ahead spool fsynced before 202; unacked spool entries are replayed on startup. Batch writer with bounded retries and rotating dead-letter file, idempotent writes per log_id. Load test sustained 1,000 logs/s acceptance with 12 pipeline workers, batch 100, on a laptop; default 4 workers do not sustain the target.

1. The problem

There are twelve services, each producing bespoke log files onto individual disks. There is no central search, no live dashboard, and no audit trail. It can take an hour to determine which service (or machine), or which file had the issue. By using grep on a laptop, it can be found in seconds. However, using grep on all twelve different formats takes an entire afternoon.

This resulted in three requirements:

A central search of all log files.
Signals in minutes about which service has an issue and how big the issue is.
An audit trail that is persistent and no data is lost when restarting the service.

The log ingestion engine has been built to provide some of the searchable requirements (see section 7).

2. The goal and the ground rules

Overview

A centralised logging platform to receive logs using an API, validate and enrich log data, route by service, store it reliably, and provide signals so we can see issues occur before our customers do.

Constraints

The brief says we must limit our libraries. I made a few changes here:

Used the rate-limiting library golang.org/x/time/rate rather than implementing my own token bucket. The interesting engineering was implementing per-IP keys, request shaping, and Retry-After.
The frontend uses React with Vite, Tailwind CSS as a utility CSS framework, Space Grotesk from the Google Fonts CDN, and Chart.js for charts, instead of hand-written HTML/CSS.

Storage is kept within the brief:

Three databases using SQLite.
One file per destination.

The configuration is simple with only .env files, integer values for duration, and ROUTING_RULES. The cost of using SQLite on real logs is discussed in section 7.

3. Architecture at a glance

This diagram represents the architecture shipped with the project:

flowchart TB
    client(["Clients"])

    subgraph ingress["HTTP ingress"]
        direction TB
        rateLimit["Rate limiter<br/>per IP, in-memory"]
        logsEndpoint["POST /logs"]
        validate["Validate batch<br/>body size + per-entry errors"]
    end

    spool[("Write-ahead spool<br/>fsync before ack")]
    rawQueue[("raw_logs<br/>bounded channel")]

    subgraph processing["Pipeline worker pool"]
        direction TB
        enrich["Enrich<br/>received_at · source_ip · environment"]
        route["Route<br/>service → destination rules"]
    end

    subgraph publishing["Publisher"]
        direction TB
        rabbitPublisher["RabbitMQ publisher<br/>persistent + confirms<br/>5s publish timeout<br/>auto-reconnect with backoff"]
        fallback[("Bounded in-memory fallback<br/>used while disconnected")]
    end

    broker[("RabbitMQ<br/>exchange_service1..3<br/>queue_service1..3<br/>routing key: log.write")]

    subgraph delivery["Consumers and storage"]
        direction TB
        supervisor["Supervisor<br/>one connection<br/>declares topology, reconnects"]
        amqpConsumer["AMQP consumers<br/>one connection per queue<br/>ack after stored"]
        fallbackConsumer["Fallback consumer"]
        batchWriter["Batch writer<br/>per-destination buffers<br/>retry with backoff"]
        sqlite[("SQLite<br/>logs_service1.db<br/>logs_service2.db<br/>logs_service3.db")]
        deadLetters[("Dead-letter NDJSON<br/>one shared file<br/>rotated by size")]
        replay["cmd/replay"]
    end

    client --> rateLimit --> logsEndpoint --> validate --> spool --> rawQueue
    rawQueue --> enrich --> route --> rabbitPublisher
    rabbitPublisher -->|"connected"| broker
    rabbitPublisher -->|"publish fails"| fallback
    fallback -.->|"fallback full: dead letter"| deadLetters
    broker -->|"deliver"| supervisor
    supervisor --> amqpConsumer
    fallback -->|"stored directly, never re-published"| fallbackConsumer
    amqpConsumer --> batchWriter
    fallbackConsumer --> batchWriter
    batchWriter -->|"stored"| sqlite
    batchWriter -->|"retries exhausted"| deadLetters
    deadLetters -.-> replay
    replay -.->|"replay"| sqlite

There are 6 stages within the pipeline:

Post
Validate
Buffer
Enrich/Route
Publish/Fallback
Consume/Store

flowchart LR
    post["1 Post<br/>rate limit,<br/>accept the batch"]
    validate["2 Validate and spool<br/>per-entry errors,<br/>fsync admission"]
    buffer["3 Buffer<br/>bounded raw_logs<br/>channel (10,000)<br/>202 or 503"]
    enrichRoute["4 Enrich and route<br/>stamp fields,<br/>pick destination"]
    publish["5 Publish<br/>RabbitMQ, or the<br/>in-memory fallback"]
    store["6 Consume and store<br/>batched SQLite writes,<br/>dead letters on failure"]

    post --> validate --> buffer --> enrichRoute --> publish --> store

Each stage handles one specific type of failure/slowdown.

The rate limiter handles abusive clients.
raw_logs handles separating quick HTTP traffic from slow enrichment.
The publisher fallback handles outages of the broker. The logs are each written to storage once, as persistence is idempotent on a per log_id basis. They are never ping-ponged between the broker and memory.
The batch writer spreads the cost of inserting into SQLite.
The dead letter file handles unrecoverable failure.

The fallback is consumed by a dedicated consumer and written directly to storage. No re-publishing occurs. As consumers only acknowledge once data has been stored, it provides at-least-once delivery. Persistence within the database is idempotent on a per log_id basis, so duplicate data can be handled.

4. The stages

4.1 Ingestion and validation

This will validate the Content-Type header, ensure the body is a certain size using http.MaxBytesReader, parse the JSON, and ensure that there are four fields for each log entry. These are:

Timestamp (RFC3339)
Service (at most 100 characters)
Level (INFO/WARN/ERROR/DEBUG)
Message (at most 10,000 characters)

Validation runs first. Valid entries are appended to the write-ahead spool and fsynced to disk. The handler then attempts to enqueue the batch onto the bounded raw_logs channel. If the enqueue succeeds within CHANNEL_SEND_TIMEOUT, the handler returns 202 with the batch ID and per-entry rejection details; if the channel is full or the send times out, the handler returns 503. The 202 therefore means the batch is durably admitted for replay and now being processed, not yet in SQLite. The spool guarantees the batch can be replayed on restart. In the 503 case, the spool entry is retained so it can be replayed later; the client may retry and create duplicates, which receive new log_ids and are therefore stored as separate rows (idempotency does not collapse them).

Design decision: I decided to accept partial batches. This will mean that if a log has an invalid time, it will only lose one log instead of one hundred. Each valid log entry will have a UUID added as the log_id field. This means they can be safely resent and played back.

Design decision: The spool fsync is performed inline on the request path to make a 202 durable. This breaks the “no blocking” ideal and adds latency on the critical path, but it guarantees admission durability. The fsync is small per batch, and the cost is accepted for durability; compaction of the spool log and .acked file occurs only when every spooled ID has been acknowledged.

func ValidateLogEntries(logEntries []model.LogEntry) (validEntries []model.LogEntry, rejectedEntries []model.RejectedEntry) {
 for entryIndex, logEntry := range logEntries {
  fieldErrors := validateLogEntry(logEntry)
  if len(fieldErrors) > 0 {
   rejectedEntries = append(rejectedEntries, model.RejectedEntry{
    Index:  entryIndex,
    Errors: fieldErrors,
   })
  } else {
   validEntries = append(validEntries, logEntry)
  }
 }
 return validEntries, rejectedEntries
}

Valid and invalid log entries will be separated by ValidateLogEntries whilst retaining their original index.

Every log entry needs to have a time, service, level, and message. POST /logs can return 202, 400, 413, 415, 503, or a 429 from the limiter.

4.2 Protecting the boundary

Rate limiting middleware using a lazy in-memory token bucket, keyed on the source IP address. Token buckets are purged explicitly when idle.
Rejected requests will not consume a token.

Design decision: The AllowWithoutConsumingCapacity method is used. This method checks how much delay is required. If the delay is positive, the reservation is cancelled, and the request is rejected before any work is done, returning 429 with a Retry-After header in whole seconds. The rejected request keeps its token.

reservation := entry.limiter.Reserve()
delay := reservation.Delay()
if delay > 0 {
 reservation.Cancel()
 return delay, false
}

Rate limiting middleware uses a token bucket for each IP address. 429 status code is returned with a delay in whole seconds.
No token is consumed on rejection. X-Forwarded-For is trusted only when a trusted proxy is configured; with no proxy list, the socket address is used for identity.

4.3 The `raw_logs` buffer

I have a bounded channel that defaults to 10,000 entries for RAW_LOGS_BUFFER. Validations happen quickly, but enrichments happen at their own pace.

Design decision: Producers wait for a bounded 100 ms rather than fail instantly. PushBatch rejects oversized batches immediately, then takes the producer lock and waits up to CHANNEL_SEND_TIMEOUT 100 ms for room for the entire batch. This is blocking with a bounded timeout, providing backpressure.

for len(rawLogs) > cap(rawLogsQueue.channel)-len(rawLogsQueue.channel) {
 select {
 case <-capacityCheck.C:
 case <-sendContext.Done():
  return ErrQueueOverloaded
 }
}

Writing straight to SQL would be on the slow path. Using the bounded channel means they are separated. I use back pressure by returning a 503.

4.4 Enrichment

Adds received_at, source_ip, and environment (defaulting to production) from config. Also adds log_id, batch_id, and retry_count (starting at 0) to every log. No I/O operation, so no failure case.

Design decision: All data is captured at the edge as it is based on request context. This means I only have access to the socket address, forwarded chain, and received time. These are stamped and then passed through. If any of these fields are missing, a sensible default is used.

return model.EnrichedLog{
 LogEntry:    rawLog.LogEntry,
 LogID:       logEnricher.logIDOrDefault(rawLog.LogID),
 BatchID:     rawLog.BatchID,
 ReceivedAt:  receivedAt,
 SourceIP:    sourceIP,
 Environment: logEnricher.applicationEnvironment,
 RetryCount:  0,
}

received_at uses the handler clock; source_ip uses the socket address or the first valid X-Forwarded-For header from behind a trusted proxy. All of these are captured at the edge.

4.5 Routing

The destination is stamped based on a validated mapping from the .env file using the ROUTING_RULES=claims=service2,billing=service3 format. If an unknown service is provided, it will default to service1. The config is loaded using only godotenv with no merging.

Design decision: This was a design decision to keep the routing rules in the .env file. I accept unknown services, as I can have new services logging and then add a new rule. All destination names are checked on startup and will fail if an unknown value is found. This means any typo will cause a crash on startup rather than the service being incorrectly routed.

destination, exists := serviceRouter.rules[enrichedLog.Service]
if !exists {
 slog.Warn("service has no routing rule, using default destination", "service", enrichedLog.Service, "default_destination", string(serviceRouter.defaultDestination))
 enrichedLog.Destination = serviceRouter.defaultDestination
}

A validated map of ROUTING_RULES has been created to route to either of the 3 destinations based on the service. The default destination has been set in the config and will show a warning if there is no rule to match.

4.6 Queuing with RabbitMQ

3 direct exchanges (exchange_service1..3) are bound to durable queues with a routing key log.write. Messages are published using amqp.Persistent, with a 5-second timeout (PUBLISH_TIMEOUT) and confirm mode. The publish will only return nil once the broker has acknowledged. The publisher will retry with backoff.

Design decision: Each destination uses an individual exchange and queue so that messages do not queue up in one queue, starving the others. We always get confirms and cannot allow fire-and-forget.

if publishError := publisher.publishOperation(enrichedLog); publishError != nil {
 publisher.markDisconnected()
 slog.Warn("publishing to rabbitmq failed, using in-memory fallback", "error", publishError, "destination", enrichedLog.Destination, "batch_id", enrichedLog.BatchID)
 return publisher.publishToFallbackAndRecord(enrichedLog)
}

publisher.recordPublishedMessage()
return nil

The broker is a persistent buffer, so it can handle bursts of activity and restarts. If it goes offline, messages will be sent to an in-memory store with a maximum size. These messages will be written to the store and will never be republished. The publisher will retry with backoff.

Timeouts on confirms do not necessarily mean the broker has failed. In ambiguous situations, the publisher will fall back to using the in-memory store. This means that a message that has been accepted by the broker may be stored twice. However, storage is idempotent based on the log_id, so duplicate entries will be merged into a single row.

4.7 Storage, retries and dead letters

A single batch writer for SQLite. Consumes:

AMQP consumers
Fallback consumer
cmd/replay

Flush per destination at 100 logs WRITE_BATCH_SIZE or every 1 second WRITE_FLUSH_INTERVAL, one transaction per destination. Failed flush retries: retry one after 1s, retry two after 5s, retry three after 10s. Four attempts over ~16s. If all fail, the whole batch is handed to a dead letter file as NDJSON, rotated by size.

The dead letter file is appended to in NDJSON format. It will be rotated by closing and renaming the file. Each entry is written with a new line and flushed. If a crash occurs while writing, the last line may be truncated. cmd/replay reads line by line and stops with an error on the first unparsable line, so a truncated tail causes replay to fail.

As the same destination databases are opened by cmd/replay, it is safe to run during the service. SQLite will serialise writes to the database per transaction. Both processes will use WAL and a busy timeout. This means the replay process will wait for/obtain a write lock, and the file will never be corrupted.

Design decision: Two triggers have been used to flush the data. This provides a good balance between latency and efficiency. Persistence is idempotent, as each log_id will be persisted using ON CONFLICT (log_id) DO NOTHING on the unique index. Duplicate rows will not occur, as messages will not be duplicated when redelivered, and dead letters will not be duplicated when replayed. The dead letter file can safely be replayed twice.

deadLetterError := batchWriter.deadLetterBatch(enrichedLogs, storeError)

The writer will flush each destination once every 100 logs or 1s. If an error occurs, it will retry with a backoff. The retry_count will be stamped, and the message will be dead-lettered. Recovery can be done using cmd/replay, which will use the same idempotent StoreBatch.

The dead letter store contains NDJSON format for each batch. It contains the full enriched log, storage exception, and time of failure.

4.8 Observation

The in-memory registry contains the following metrics:

Receipts
Request counts per level/service
Rate-limited requests
Dead letters
Write errors
1-minute moving average of throughput

Design decision: Metrics are collected by the pipeline. The limiter collects the number of times it rejected a request. The writer collects dead letters and write errors. The queue backlog can come from either the RabbitMQ management API or estimated from publish minus store if the API isn't available.

A single registry containing metrics from each stage. Backlog will be collected from local estimate if the management API isn't available. The metrics can be viewed as a JSON snapshot or via server-sent events.

5. Designing for failure

The table maps scenarios to outcomes.

Scenario	The client sees	Where the log ends up	Section and path
RabbitMQ unreachable	`202` still, if memory capacity lasts	Stored directly by the fallback consumer, never re-published	4.6
RabbitMQ down and fallback full	`202` still	Dead-letter NDJSON entry with the reason	4.6, publish error path to the sink
SQLite write failure	Nothing, by design	Retried at 1, 5, then 10 seconds, then the whole batch dead-lettered	4.7
Queue overloaded past the send timeout	`503`	Durably spooled, not yet enqueued; will be replayed on restart, client may retry and create duplicates handled by idempotency	4.3
Body or batch oversized	`413`	Not ingested, visibly rejected	4.1
Wrong content type	`415`	Not ingested, visibly rejected	4.1
Per-IP limit exceeded	`429` + `Retry-After` in whole seconds	Not ingested, visibly rejected, token kept	4.2
Ungraceful crash	Transport error	In-memory buffer and writer batches lost; spooled batches survive on disk and are replayed on restart. What that costs is section 7	7

Two failure mechanisms.

Closed/shutting down writer: Consumer nack's messages immediately, and the broker will redeliver shortly. This only happens if the writer is shutting down.
SQLite flush failure: Message will be scheduled up to 16s before dead-lettering. This is spread out to reduce retries.

6. Proof

I have 172 individual tests across 15 packages that are CI'd with the race detector. Timings are simulated using clock injection as I am testing out how the retry system deals with exhaustion in milliseconds.

Integration tests under cmd/server/integration_test.go use real components. This includes end-to-end functionality, zero dead letters during an outage, flush functionality triggered both ways, and retry to dead letter when the store fails.

I also have a load that simulates a constant request rate with cmd/loadtest. It uses fifty IPs in the documentation range as sources, and will output statistics on how many requests have gone through per status and latency percentiles.

I ran this with 12 pipeline workers, batch size 100, and 8 senders. This gave me 300 requests, each with 100 entries. This gave me 30,000 logs accepted in total. I ran this on my laptop with RabbitMQ and SQLite, with spool fsync enabled.

Sustained target run	Result
Sustained ingestion	1000.0 logs/s, zero rejections
Response latency	mean 2.9 ms, p50 2.2 ms, p95 7.1 ms, p99 14.1 ms
End-to-end accounting	300 requests accepted, all 30,000 accepted logs stored when the queue drained 47 seconds later, no dead letters
Server footprint	~26 MB RSS, ~9 percent CPU

This proves:

Sustained acceptance rate of 1,000 logs/s on one laptop with local broker and SQLite, confirmed by row-count reconciliation.
End-to-end with 30 seconds of sending time and 47 seconds of draining. Ingestion accepted at 1,000 logs/s while storage lagged; the backlog sat in the RabbitMQ queues. 30,000 logs in 77 seconds, about 390 logs/s including tail. This end-to-end rate reflects the publisher confirm loop, not consumers or SQLite. The test was mapped to a single service, so a single SQLite writer did the work; spreading across destinations does not change the per‑log confirm latency.

Does not prove:

Rate limiter saturation.
Multiple instances.

NOTE: When using 12 workers, you can get a throughput of 1,000 logs per second, but if you use the default of 4, you won't. You should aim for around 100 logs per second per worker. The confirmation loop is the limiting factor.

7. Trade-offs and what I would change

Log scale makes columnar storage ideal. Failure logic is contained within the code in SQLite. Batch writer means I can batch inserts. I would use ClickHouse in production. Two triggers mean the contract remains intact.
I guarantee at-least-once delivery by having an idempotent storage per log_id. If a crash occurs between inserting a record and sending the ack message, then the records will be delivered again. The command and replay functionality means the dead letters are applied again. The log_id is set during ingestion, and using ON CONFLICT DO NOTHING ensures only one row will exist for both.
Dead letters are applied at the batch level. After 16s of failed flushes, up to 99 logs that have done nothing wrong will be moved to a dead letter file. These are stored and can be restored. They are also grouped into batches.
In-memory state will be lost if there is an ungraceful crash. The writer has a buffer of less than 1s per destination, and the fallback will be lost. The broker will have the confirmed but unacknowledged messages. Spooled batches survive on disk and are replayed on restart, so admission is durable even if the process crashes; the cost is potential duplicates on retry.
The spool adds fsync latency on the request path and requires compaction of the spool log and .acked files on restart. This is a deliberate trade-off for durable admission.
Having a single point of state makes scaling out difficult. I have a limiter, counters, and backlog estimation, which are held in memory. The second replica will have its own buckets; these will be reset when the service is restarted. Having a shared rate limiting store such as Redis and storing/persisting metrics allows scaling out.
No search endpoint. Initially, this wasn't searchable. Everything is stored centrally and monitored. Logs can be viewed through SQLite and dead letter tools. Next step is to implement full-text search on the logs table in SQLite.
No alerts. Counters are in place to trigger, but paging on a threshold with cooldown and dedupe is a monitoring-stack integration.

What I would improve:

Search endpoint
State for scaling out
Alerts

Performance tip: The max we can publish per log is the publish-confirmed rate. If I were to publish the entire batch per confirmation, then this would increase the max per worker.

8. Lessons

Defence of the ingest boundary. Inexpensive inline decisions, and things to bound and make visible. All components have capacity and timeout so we can never block or drop.
Batching in async pipelines. We can be honest about our options. Either take the whole batch, or fail the whole request. The producer lock ensures we can do this under concurrency. When shutting down, we drain the channel, flush the writer, close the store, and lastly stop the dead letter writer.
Designing for queue/store failures. Each failure point has a different decision to make. If the broker fails, we use a one-time fallback store. If SQLite fails, we will retry with a limit, using stamps and dead-lettering out. If we fully fallback, we get dead letters. We can chain these decisions. It makes it easier to have a stage that can't fail.
Registry for event-to-action. The least appealing but most useful decision. We can pull from our registry. During integration testing, the broker failed, but the metrics endpoint still responded, as the fallback estimate was in the same registry. We can instrument what we are already doing. The signals will fail at the same time as the pipeline, meaning they are exactly when we need them.

Review Questions

#	Question	One-line answer
1	Which fields are required, and which status codes can `POST /logs` return?	`timestamp` (RFC 3339), `service` (100 characters), `level` (four values), `message` (10,000 characters); `202`, `400`, `413`, `415`, `503`, plus `429` from the limiter.
2	How does the rate limiter work, and what happens when the limit is exceeded?	A per-IP token bucket in middleware before the handler; `429` with `Retry-After` in whole seconds, and the rejected request keeps its token.
3	Why an in-memory channel instead of writing straight to SQLite?	The channel keeps the slow path out of the request and turns queue pressure into `503` backpressure with a timed, all-or-nothing send.
4	Which enrichment fields are added, and where does each value come from?	`received_at` from the handler's clock, `source_ip` from the socket address or a trusted proxy's `X-Forwarded-For`, `environment` from configuration, all captured at the edge.
5	How are routing rules structured, and how is a database chosen?	ROUTING_RULES from .env maps service to one of three destinations, stamped on the log, with the configured default and a warning when no rule matches.
6	Why RabbitMQ, and what changes when it is unavailable?	The broker is the disk-backed buffer absorbing bursts; down means the bounded in-memory fallback stores directly, never re-publishes, and the publisher reconnects with backoff.
7	How does batch writing work, and what is the retry strategy?	One writer flushes per destination at 100 logs or one second inside a transaction, retries once per backoff delay stamping `retry_count`, then dead-letters the batch.
8	What reaches the dead-letter file, and how is it recovered?	Batch-level entries as NDJSON with the log, the error and the timestamp, rotated by size, restored by `cmd/replay` through the same idempotent `StoreBatch`.
9	Which metrics are tracked, and what does `GET /metrics.json` return?	Receipt, level and service counts, error rate, throughput, queue backlog, rate-limited requests, dead letters and write failures, as one atomic JSON snapshot.
10	Which alert conditions exist, and how is alert spam prevented?	None are built: the counters are ready to trigger against, and alerting with its cooldown design is a monitoring-stack integration named in section 7.

A Story About Pluralization In Code (2 Items vs 2 Boxes)

John Eliud Odhiambo — Mon, 05 Jan 2026 13:20:47 +0000

I was building a small console application. A simple cart. It was nothing fancy but a chance to experiment with input, calculations, and formatted output. Here’s the first version I wrote:

import java.util.Scanner;

public class Cart {
    static void cart() {
        Scanner scanner = new Scanner(System.in);

        System.out.print("Enter item name: ");
        String item = scanner.nextLine();

        System.out.print("Enter price for each: ");
        double price = scanner.nextDouble();

        if (price <= 0) {
            throw new IllegalArgumentException("Price must be positive.");
        }

        System.out.print("Enter quantity to buy: ");
        int quantity = scanner.nextInt();

        if (quantity <= 0) {
            throw new IllegalArgumentException("Quantity must be positive.");
        }

        double total = price * quantity;

        String itemLabel = quantity == 1 ? item : item + "s";

        System.out.printf(
            "You bought %d %s. Your grand total is KES %.2f.%n",
            quantity, itemLabel, total
        );
    }

    static void main(String[] args) {
        cart();
    }
}

When I ran it with something simple like apple:

Enter item name: apple
Enter price for each: 100
Enter quantity to buy: 2

The output was:

You bought 2 apples. Your grand total is KES 200.00.

Perfect, simple and predictable. It worked exactly as expected.

The First Spark Of Curiosity

But then I tried a word that wasn’t so straightforward:

Enter item name: box
Enter price for each: 100
Enter quantity to buy: 2

And the program printed:

You bought 2 boxs. Your grand total is KES 200.00.

I paused and thought I could maybe just add some rules:

Words ending in x get appended with es. This would also apply for s, sh, and ch.
Words ending with y become ies.
But what about words like mouse and child?

It was exciting as it felt like executing the real craft of programming beyond just making something work.

First Attempt Towards A Smarter Solution

I decided to write a small helper function to handle the obvious cases:

static String pluralize(String word, int quantity) {
    if (quantity == 1) return word;

    if (word.endsWith("s") || word.endsWith("x") || word.endsWith("z")
        || word.endsWith("ch") || word.endsWith("sh")) {
        return word + "es";
    }

    if (word.endsWith("y") && word.length() > 1 
        && !"aeiou".contains("" + word.charAt(word.length() - 2))) {
        return word.substring(0, word.length() - 1) + "ies";
    }

    return word + "s";
}

Now, I could run tests with:

System.out.println(pluralize("box", 2));    // boxes
System.out.println(pluralize("city", 2));   // cities
System.out.println(pluralize("class", 2));  // classes
System.out.println(pluralize("apple", 2));  // apples
System.out.println(pluralize("mouse", 2));  // mouses

The results were mostly correct; boxes, cities, classes but mouse to mouses stood out. Irregular nouns weren’t going to be solved by simple rules. And every new edge case made the code longer and more fragile.

This felt like the 'even numbers are hard' meme

At this point, I realized that my 'make it perfect' curiosity was bumping into the limitations of language itself.

Discovering ICU4J

Searching for solutions, I found ICU4J's Java library used in production for internationalization and pluralization and it looked promising. However after reading the documentation more carefully, I realized ICU4J doesn’t actually change the spelling of words. It decides when to use a singular versus plural form. For example, it can help you display:

You have 1 item
You have 2 items

But it does not automatically turn box into boxes or child into children. It solves the category problem, not the word-inflection problem.

The Tempting Idea

It was disappointing after realizing that rule-based pluralization was incomplete and that even production-grade libraries like ICU4J weren’t meant to pluralize raw nouns.

Just as I was about to give up, another thought crossed my mind:

What if I just ask an AI to do this?

It felt reasonable as modern AI models are excellent at language. They know that:

box pluralizes to boxes
class to classes
mouse to mice
child to children

In theory, this meant making an API call, passing the noun, and getting back the correct plural. This guaranteed no grammar rules, and no irregular noun lists. Just letting the model handle the complexity.

For a moment, this felt like the perfect solution.

Why the AI Idea Actually Makes Sense (At First)

To be fair, this approach isn’t foolish. In fact, it has real strengths because an AI model:

Understands irregular nouns
Handles borrowed words and edge cases
Adapts to language naturally
Requires very little code on my end

For a small personal project, the idea is genuinely attractive. I wouldn’t need to maintain pluralization logic at all. I’d just delegate the problem to the AI via an API.

But then I paused and started thinking like a system designer, not just a coder.

The First Red Flag: Cost

Pluralization is a low-value, high-frequency operation. If every time a user adds an item to a cart the app needs to:

Make a network request
Pay for tokens
Wait for a response

That cost adds up quickly. Paying for AI inference to decide whether to print boxes or items is hard to justify, especially when that same operation could be done locally with zero cost.

For small scripts or experiments, this might be fine. At scale, it becomes expensive very fast.

The Second Red Flag: Latency

Pluralization sits on the critical path of user feedback. It happens:

While rendering UI
While printing output
During fast interactions

An AI call introduces:

Network latency
Timeout risk
Retry logic
Failure modes you don’t control (500 errors)

Something as simple as printing a sentence suddenly depends on the availability of an external service. That outright felt wrong.

The Third Red Flag: Non-Determinism

This was by far the biggest concern. Language is flexible, and AI reflects that. For example:

cactus to cacti
cactus to cactuses

Both are correct but if the application prints:

You bought 2 cacti

and later:

You bought 2 cactuses

You just introduced inconsistency. An answer that is 'sometimes right' is often worse than a simpler one that is always consistent.

The Fourth Red Flag: Control and Safety

To pluralize a word using AI, I have to send user input to an external service. This raises the questions:

What if the item name contains sensitive information?
What if this data must stay on-device?
What happens if the API changes behavior?

Suddenly, a simple console program is tied to:

Network access -API keys
Privacy considerations

At this point, the picture became clear. Opting to use AI for pluralization solves a hard linguistic problem but introduces cost, latency, inconsistency, dependency and still does not guarantee perfect results.

The Practical Tradeoff

At this point, I had to step back and ask myself what the real goal of my program is.

Showing the number of items bought
Showing the total cost

The exact spelling of the product name in plural wasn’t critical.

That’s when it clicked almost every app I use does the same thing. Shopping sites, notification systems, and dashboards don’t attempt to pluralize arbitrary nouns. They simply use a safe, controlled noun:

You bought 2 items
You have 5 notifications
Your cart contains 3 products

It is predictable, safe, and scales well since there are no edge cases to handle and also works perfectly if you later translate your application into other languages.

Final Code Snippet

Here’s how my original shopping cart code evolved after all this reflection:

double total = price * quantity;

String message = quantity == 1 ? "item" : "items";

System.out.printf(
    "You bought %d %s. Your grand total is KES %.2f.%n",
    quantity, message, total
);

Output:

You bought 1 item. Your grand total is KES 100.00.
You bought 2 items. Your grand total is KES 200.00.

No edge cases, no dictionary of irregular nouns, no fragile rules. This approach is exactly what large production systems do.

Lessons Learned

Curiosity is important. Trying to improve the code teaches you a lot about programming, language, and tradeoffs.
Language is messy and even simple pluralization can have dozens of edge cases.
Production code values simplicity, correctness, and consistency.
Using ICU4J helps in the right context and should be used for plural-aware messages and not for spelling every noun correctly.
Opting for controlled vocabulary wins,is safe, predictable, and maintainable.

Thought

The journey from apple to apples to understanding why 2 boxes is rarely shown in apps is more than a lesson in pluralization. It is a lesson in designing software systems that works reliably, even when human language does not cooperate.

Sometimes, the simplest solution like displaying 2 items is also the most elegant.

The Hidden Environmental Cost of Your Cloud Storage

John Eliud Odhiambo — Mon, 28 Jul 2025 17:54:27 +0000

We live in the age of infinite storage. Photos, videos, and documents get uploaded to the cloud without a second thought. Google Drive, iCloud, and Dropbox all sell on the idea of limitless space. But here’s the thing: the cloud isn’t magic. It’s a physical network of massive, power-hungry data centers, and they’re leaving a bigger carbon footprint than you might realize.

This made me write about the hidden environmental cost of cloud storage. Why "just uploading it" isn’t as clean as it seems, and what we can do about it.

"The Cloud" Is Just Someone Else’s Computer (And It’s Using A Lot Of Energy)

When you upload a file to the cloud, it doesn’t just float in space; it gets stored in a data center, a room full of servers running 24/7.

A single data center can use as much electricity as 50,000 homes.
Globally, data centers account for 1-2% of the world’s electricity use, and that number is growing.
Cooling these servers alone takes billions of gallons of water annually.

Most of that energy isn’t even renewable. Many cloud providers still rely on fossil fuels to keep their data centers running. So every time you back up another 4K video of your pet, you’re indirectly contributing to that demand.

The "Dark Data" Problem (Why Your Forgotten Files Matter)

Think about how much stuff you’ve uploaded over the years. A lot of that just sits there, untouched. That’s called dark data. These are files stored but never used.

Estimates suggest up to 60% of stored data is dark data.
Keeping it all online means servers are working overtime to store useless files.
Deleting unnecessary files could cut energy waste significantly, but nobody does it.

It’s like leaving every light in your house on 24/7, even in rooms you never enter. Sure, one person doing it doesn’t matter, but multiply that by billions of users and suddenly it’s a big problem.

What Can We Do? (Realistic Solutions)

Cloud storage isn’t going away, but we can be smarter about it. Here’s how:

Clean Up Your Digital Hoarding

Delete duplicates, blurry photos, and old files you’ll never need.
Use tools like Google Photos' Free Up Space or iCloud’s storage manager.
Unsubscribe from services you don’t use.

Choose Greener Cloud Providers

Some companies are pushing for renewable energy:

Google & Microsoft aim for carbon-neutral data centers.
Apple's iCloud runs on 100% renewable energy.
Smaller providers like GreenGeeks focus on sustainability.

Local Backups Aren't Dead

Instead of uploading everything, consider:

External SSDs for large files you rarely access.
Network-attached storage (NAS) for home backups.
Physical archives (hard drives still have a place).

Final Thought: The Cloud Isn’t Evil, But It’s Not Invisible Either

The convenience of cloud storage is undeniable, but like any tech, it comes with trade-offs. Small changes add up. If enough people clean up their unused files and push for greener tech, the cloud's environmental impact can shrink.

So next time you get that storage full notification, maybe don't just blindly upgrade, but take a minute to delete what you don't need. The planet (and your wallet) might thank you.

Understanding Load Shedding

John Eliud Odhiambo — Tue, 24 Jun 2025 13:07:35 +0000

Introduction

In today’s digital landscape, applications must handle unpredictable traffic spikes without collapsing. Whether it’s a sudden surge in users or a distributed denial-of-service (DDoS) attack, systems need mechanisms to protect themselves from being overwhelmed. One such mechanism is load shedding, a defensive strategy that prioritizes critical functionality by selectively rejecting non-essential requests when the system is under stress.

In this article, we’ll explore:

What is load shedding, and why is it necessary?
A practical demo that shows load shedding in action.

By the end, you should understand how to apply this technique to keep your services resilient under pressure.

What Is Load Shedding?

Load shedding is the deliberate termination or deferral of non-critical requests to prevent system overload. It ensures that essential services remain available even when demand exceeds capacity.

Why Is It Necessary?

Without load shedding, a system might experience:

Resource exhaustion: Too many requests consume CPU, memory, or network bandwidth.
Cascading failures: One overloaded service can bring down dependent systems.
Degraded performance: Even critical requests slow down, leading to timeouts and errors.

By shedding non-essential load, a system will:

Preserve core functionality (e.g., login, payment processing).
Prevent total outages by avoiding system collapse.
Improve user experience by failing fast instead of slow degradation.

How Load Shedding Works In Code

Let us examine a practical implementation in Go and JavaScript. The demo consists of:

Essential services (always available).
Non-essential services (shed under high load).
A global request tracker to enforce limits.

Tracking Active Requests

The shed_wrapper.go file defines a middleware that monitors concurrent requests:

var (
    maxRequests     = int32(50)  // Maximum allowed concurrent requests
    currentRequests int32        // Global counter for active requests
)

maxRequests: A threshold beyond which we start shedding.
currentRequests: An atomic counter to track active requests safely.

The Shedding Logic

The ShedWrapper middleware wraps HTTP handlers and applies load shedding:

func ShedWrapper(h http.HandlerFunc, canShed bool) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // Increment request count
        if atomic.AddInt32(&currentRequests, 1) > maxRequests {
            atomic.AddInt32(&currentRequests, -1) // Revert increment
            if canShed { // Only shed if allowed
                w.WriteHeader(http.StatusServiceUnavailable)
                fmt.Fprintln(w, "Service unavailable due to high load.")
                return
            }
        }
        defer atomic.AddInt32(&currentRequests, -1) // Decrement on exit
        h(w, r) // Proceed if under limit
    }
}

Key Decisions:

canShed flag: Determines whether a request can be rejected (e.g., non-essential vs. essential).
Atomic operations: Ensure thread-safe increments/decrements.
Graceful rejection: Returns HTTP 503 (Service Unavailable) for shed requests.

Essential vs. Non-Essential Services

Essential Handler (essential_handler.go) Always processes requests, even under load:

fmt.Fprintf(w, "Essential service responded! (Load: %v)", simulatedLoad)

Non-Essential Handler (non_essential_handler.go) Rejects requests when load exceeds 50:

if simulatedLoad > 50 {
    w.WriteHeader(http.StatusServiceUnavailable)
    fmt.Fprintln(w, "Non-essential service unavailable!")
}

Simulating Traffic Spikes

The frontend (script.js) lets users simulate load:

async function simulateLoad(requestCount) {
    for (let i = 0; i < requestCount; i++) {
        fetch('/simulate-load'); // Triggers 2-second delay per request
    }
}

Each /simulate-load request (simulate_handler.go) artificially increases load:

func SimulateLoadHandler(w http.ResponseWriter, r *http.Request) {
    time.Sleep(2 * time.Second) // Simulate processing
    w.WriteHeader(http.StatusOK)
}

Putting It All Together

Demo Workflow

User selects a load level (e.g., 75 concurrent requests).
Essential services remain available even under high loads, demonstrating prioritization.

The system tracks active requests:

If under 50, all services respond normally.
If over 50, non-essential requests are rejected.

Try It Yourself

You can experiment with this implementation by checking out the GitHub repository. The demo includes:

A frontend to simulate load.
Backend logic for essential/non-essential services.
Configurable thresholds for testing.

Real-World Applications of Load Shedding

While our demo is simplified, load shedding is used in many production systems:

E-Commerce Platforms

Critical: Checkout, payment processing, inventory management.
Non-Critical: Product recommendations, reviews, wishlist updates.
During Black Friday sales, non-essential features might be temporarily disabled to ensure checkout remains stable.

Cloud APIs

Critical: Authentication, billing, core compute services.
Non-Critical: Logging, metrics, secondary APIs.
If a cloud provider faces unexpected demand, it may throttle non-critical API calls.

Social Media Platforms

Critical: Posting content, direct messaging.
Non-Critical: "Like" counts, friend suggestions.
During viral events, platforms might delay updating engagement metrics to prioritize content delivery.

Financial Systems

Critical: Stock trading, fund transfers.
Non-Critical: Transaction history, analytics.
At market open, trading systems prioritize order execution over historical data queries.

Advanced Considerations

While our demo uses a fixed threshold, real-world systems might:

Adjust maxRequests dynamically based on CPU/memory usage.
Implement retries for shed requests with exponential backoff.
Use circuit breakers to fail fast when dependencies are struggling.

Conclusion

Load shedding is a proactive way to handle traffic surges. By:

Monitoring request volume,
Prioritizing critical functions, and
Gracefully degrading non-essential features,

we can build systems that fail well instead of collapsing entirely. Try the demo yourself to see load shedding in action!

Building a Dynamic User Profile Dashboard with GraphQL

John Eliud Odhiambo — Sat, 24 May 2025 18:50:38 +0000

Introduction

In today's web development landscape, efficient data fetching is crucial for creating responsive and user-friendly applications. This article explores a project that leverages GraphQL to create a personalized profile dashboard for Zone01 users, demonstrating how modern web technologies can be combined to deliver a seamless user experience.

Project Overview

The GraphQL Profile Dashboard is a web application that connects to Zone01's GraphQL API to fetch and display user data in an interactive and visually appealing manner. The project showcases several key features:

Secure JWT Authentication: Users can log in with their Zone01 credentials to access their personalized dashboard
Dynamic Data Visualization: Interactive SVG-based charts display user progress and statistics
Responsive Design: A clean, modern interface that adapts to different screen sizes
Dark/Light Theme Toggle: User preference-based theming with persistent settings

Technical Implementation

Authentication Flow

The application implements a secure authentication flow using JWT (JSON Web Tokens). When users provide their credentials, the application makes a request to Zone01's authentication endpoint, receives a JWT token, and stores it in the browser's localStorage for subsequent API requests.

This approach provides a secure way to maintain user sessions without storing sensitive credentials on the client side. The authentication status is monitored throughout the application, ensuring that protected routes are only accessible to authenticated users.

GraphQL Integration

At the core of this application is its integration with Zone01's GraphQL API. Unlike traditional REST APIs, GraphQL allows the client to specify exactly what data it needs, reducing over-fetching and under-fetching of data.

The application uses a custom GraphQL client that handles:

Authentication header inclusion
Error handling and token expiration
Structured query execution

Here's an example of how the application queries user data:

const USER_PROFILE_QUERY = `query{
  user{
    id
    firstName
    lastName
    login
    email
    campus
    auditRatio
    totalUp
    totalDown
    totalXp: transactions_aggregate(where:{type:{_eq:"xp"},eventId:{_eq:75}}){
      aggregate{
        sum{
          amount
        }
      }
    }
    // Additional fields...
  }
}`;

This query fetches all the necessary user information in a single request, which is then processed and displayed across different sections of the dashboard.

Data Visualization

One of the standout features of this application is its use of custom SVG-based data visualizations. Rather than relying on heavy third-party libraries, the project implements lightweight, custom visualization components:

Line Graphs: For tracking XP progress over time
Bar Graphs: For comparing project completion statistics
Pie Charts: For displaying audit ratios
Donut Charts: For showing pass/fail ratios

These visualizations are created using SVG elements, providing a responsive and interactive experience while maintaining a small footprint.

Theme Management

The application implements a theme management system that allows users to toggle between light and dark modes. This preference is stored in localStorage, ensuring that the user's preferred theme persists across sessions.

The theme implementation uses CSS variables and class-based styling, making it easy to maintain and extend:

:root {
  --transition: all 0.5s ease;
  --text-color: #000;
  --dark-text-color: #fff;
  --primary-color: rgb(62, 62, 255);
  --light-bg-color: #fff;
  --dark-bg-color: #121212;
  --box-shadow: rgba(0, 0, 0, 0.35) 0px 5px 15px;
  --dark-box-shadow: rgba(255, 255, 255, 0.15) 0px 5px 15px;
}

User Experience

The dashboard is designed with user experience at its core. Upon logging in, users are presented with a clean, organized interface that provides immediate access to their most important information:

Basic Information: User details including name, email, and campus
Audit Statistics: Visualization of audit ratios and activity
XP Statistics: Progress tracking with interactive charts
Project Statistics: Overview of completed and current projects
Skills: Visualization of skill development across different areas

Navigation is streamlined through a sidebar menu that allows users to quickly jump to different sections of their profile.

Challenges and Solutions

Authentication Security

Implementing secure authentication required careful consideration of token storage and transmission. The application uses localStorage for token storage, which provides a balance between persistence and security. All API requests include the JWT in the Authorization header, ensuring secure communication with the backend.

Dynamic Data Loading

Fetching and processing complex nested data from GraphQL required a structured approach to data handling. The application implements a loading state system that provides visual feedback during data fetching operations, enhancing the perceived performance of the application.

SVG Rendering Compatibility

Creating cross-browser compatible SVG visualizations presented challenges, particularly with text rendering and animation. The solution involved implementing fallback options and ensuring that visualizations degrade gracefully in environments with limited support.

Conclusion

This GraphQL Profile Dashboard demonstrates how modern web technologies can be combined to create a powerful, user-friendly application. By leveraging GraphQL for efficient data fetching, implementing custom SVG visualizations, and focusing on user experience, the project delivers a polished and performant solution for displaying user profile data.

The modular architecture and clean separation of concerns make the codebase maintainable and extensible, providing a solid foundation for future enhancements. Whether you're looking to implement GraphQL in your own projects or seeking inspiration for data visualization techniques, this project offers valuable insights and practical examples.

Live Demo

Experience the GraphQL Profile Dashboard in action at https://graphql-uvsx.onrender.com. Note that you'll need Zone01 credentials to log in and explore the full functionality.

Will A Project Manager Expect Nine Expectant Mothers To Deliver Nine Babies In One Month? A Dive Into Project Management

John Eliud Odhiambo — Sat, 10 May 2025 19:30:03 +0000

There has always been a joke in project management that tries to paint a picture of how project managers plan and manage projects, as well as people. This has always been about project managers (PMs) expecting nine expectant mothers to deliver nine babies in one month. The purpose of this article will be to give my opinion on how truthful or false this statement is, what role PMs play in any project, my experience as a technical project manager during my internship, and my advice to any newbie PM.

NOTE: The article will be more specific about technical project management based on my software development background, but most opinions will be applicable to the broader project management field.

What Is Project Management?

In simple terms, it's all about the process of planning, organizing, and overseeing to achieve a specific goal within defined constraints, scope, time, and budget.

Who Is A PM, And What Role Do They Play?

I define a PM as someone responsible for creating a clear vision for the project, building trust with all stakeholders, and ensuring everyone understands their role in the bigger picture, as well as identifying and managing risks.

What Is Agile Methodology?

This is one of the approaches used in project management that emphasizes flexibility and adaptability. Work is broken into short cycles, often referred to as sprints, that last for a few weeks. This allows teams to deliver parts of the project incrementally, gather feedback quickly, and make adjustments as needed throughout the process.

The Project I Am Working On

In the spirit of bringing like-minded people together, my team has been tasked with developing a web platform that will provide a medium for enabling creatives and activists to identify and network with fellow creatives, discover places of interest, and collaborate on projects. I won't share many details for now due to organizational regulations.

It takes great people to come up with a great product, and this is reflected in this project as we have a team comprising two designers, four developers, two PMs, the design team lead, the development team lead, and the PM team lead. The size of the team already justifies how high the stakes are for the project.

The Sprint

With agile as our approach for the project delivery, the first objective was to align with all stakeholders, including the client, for familiarity. This is crucial to clearly define everyone's role and foster the spirit of collaboration. To effectively manage the team, we leveraged project management tools like Jira, Confluence, GitHub Projects, as well as meeting tools like Google Meet. Jira and GitHub Projects are used to assign tasks, supervise individual progress on the assigned tasks, alongside estimate the schedule for each assigned task. It is from this that a PM can determine if everything is progressing as scheduled. An alternative to Jira can be Asana. To better manage our design team workflows, we opted for Jira and GitHub Projects for the development team.

Apart from issuing tasks, a lot of documentation on every event that takes place, be it client meetings, team stand-ups, and so on, needs to be captured. For this, we used both Confluence for managing team events such as daily stand-ups, retrospectives, and meeting notes, and Google Docs for sharing project essentials with the client. For every meeting that happens, a good PM should always ensure they have a notebook and a pen. In the age of artificial intelligence, some note-taking assistants can also do this, but are often limited to some selected languages and dialects.

It is also at the sprint phase where we establish our work processes that include daily stand-ups. A stand-up is a very brief meeting among the internal team, usually about fifteen minutes, conducted as the first thing in the morning, that seeks to set the mood for every team member for the day. It highlights the problems/blockers faced by an individual and their priority for the day. During this meeting, a PM needs to act fast to address any blockers that are delaying an individual's progress.

For the project the team was working on, the first sprint deliverable was for our design team to have high fidelity prototypes for both web and mobile screens of the landing page, signup, and login. The second sprint was for our development team to implement the design templates into actual code for both the frontend and backend.

The Highs: Moments That Made Me Believe

The toughest task our team has is to ensure that what we deliver is in line with the client's vision, as well as manage client expectations. Among the presentations we have had with the client was a review of what was achieved during sprint one. I was very nervous during the entire presentation since any feedback from the client, whether positive or negative, at this point would have a change in the trajectory of the project's overall direction. Apart from some minor adjustments to the designs suggested by the client, the meeting ended on a high note as the presented deliverables were approved. The same also applied to the second sprint deliverable meeting. Such moments where the client expresses their gratitude are what every PM should strive for. This will involve frequent meetings with the client and asking a lot of questions to ensure you share the same vision as the client. I believe a PM should have a vast understanding of the project, as compared to the client, because you get to interact with every stakeholder in the project. The team spirit and morale are always boosted on such occasions.

The Lows: Moments That Made Me Doubt

Despite the bed of roses, there is always a period of uncertainty and pressure. One such moment was during the second sprint when we faced version control merge conflicts while integrating the backend and frontend, with barely thirty minutes to a presentation meeting. As a PM, I initially assumed (the worst mistake a PM can make) that the handoff would be seamless after all, we had stand-ups, documentation, and alignment meetings. But I quickly learned that even with the best planning, miscommunications can occur, especially when team members interpret requirements differently. To mitigate this, we have embraced continuous communication, especially among our internal team, to ensure we all know what each person is doing.

Time estimation is also a tricky part. Early in the project, we underestimated the time required for deliverables, which caused a delay in one of our internal milestones. We had to regroup, reassess, and communicate this shift in expectations to the client with transparency. This was tough but taught me the value of managing not only tasks but also expectations, both for the team and stakeholders.

Lessons I Have Learned So Far

One of the biggest lessons I have learned is that communication is important to any project. It is not about overcommunicating, but about ensuring that everyone knows the WHY behind what they are doing. A team that understands the end goal is more motivated and aligned.

Another important realization is that leadership as a PM isn’t about having all the answers. It is about being resourceful enough to ask the right questions and build bridges when gaps appear.

It is also crucial to adapt. What worked in the first sprint might not work for the next sprint. Flexibility in processes and even timelines can make the difference between a bottleneck and a breakthrough.

Do I Expect Nine Expectant Mothers to Deliver Nine Babies in One Month?

Let me finally address the elephant in the room. A good PM needs to understand that just because you have more hands doesn’t mean the work gets done faster and to the required standards. Sometimes, it’s about the right skills, the right timing, and respecting the natural rhythm of work.

Advice To New PMs

Be curious by learning the basics of the tools your team is using. It helps in making informed decisions.
Apart from chasing deadlines, build relationships by getting to know your team, understanding their strengths, and supporting them as people. Have open sessions with your team to get a better perspective of them.
Don’t be afraid to ask whether it's clarity from a client or technical help from your dev lead.
Always document decisions, changes, and feedback. This will come in handy at some points of the project cycle.
Celebrate small wins as they build momentum and remind everyone of the progress being made.
Conduct A/B tests among a number of users on occasions where a decision has to be made from multiple options. This builds trust, especially from the client.

As we progress more into the project, I will document the experience and publish another article based on a continuation of this. I hope this was worth reading. Thanks.

FETCHING ADVENT OF CODE INPUT DYNAMICALLY IN GO

John Eliud Odhiambo — Wed, 04 Dec 2024 20:04:23 +0000

Advent of Code is a fun way for programmers to test and improve their problem-solving skills. While solving the puzzles, you might want to automate the fetching of your personalized puzzle input directly using its URL instead of copying the input to a text file that will be available locally. However, trying to access the input URL using a simple HTTP request, results in the message below:

Puzzle inputs differ by user. Please log in to get your puzzle input.

This article explains why this happens and how to correctly fetch your inputs dynamically using Go programming language.

The Problem: Why Can't We Fetch The Input Directly?

Advent of Code requires you to log in to access your personalized puzzle inputs. When you log in through the browser, Advent of Code sets a session cookie in your browser. This cookie is used to identify your account and provide your unique input.

If your HTTP requests don’t include this session cookie, the Advent of Code server cannot recognize you as a logged-in user, hence the error message.

Solution: Using the Session Cookie in HTTP Requests

We must include the session cookie in our HTTP requests to fetch the puzzle input. Here is a step-by-step guideline:

Log in to Advent of Code.
Open your browser's Developer Tools (Press F12 key) and navigate to the Network tab.
Refresh the Advent of Code page and look for the cookie header in the request headers.

Extract the value of the session cookie.

NOTE: It's important to keep your session cookie a secret since someone else can access your Advent of Code account if they get access to it.

Code To Fetch The Input

Below is a simple program we will use to fetch our puzzle input dynamically:

Setting Up The Base URL

We start by defining the base URL for fetching inputs and creating a function to read the input for a specific day.

const baseURL = "https://adventofcode.com/2024/day/%s/input"

func readInput(day string) {
    url := fmt.Sprintf(baseURL, day)
    fmt.Println(url)
}

Creating The HTTP Request

Next, we create an HTTP request and include the session cookie.

client := &http.Client{}
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        fmt.Printf("Error creating HTTP request: %v\n", err)
        return
    }

    // Add the session cookie
    req.Header.Add("Cookie", "session=[YOUR_SESSION_TOKEN]")

http.NewRequest: Creates an HTTP GET request for the input URL.

req.Header.Add: Adds a header to the request with the session token for authentication. (Replace [YOUR_SESSION_TOKEN] with your actual token).

Sending The Request And Handling The Response

Now we send the HTTP request and read the server's response.

resp, err := client.Do(req)
    if err != nil {
        fmt.Printf("Error making HTTP request: %v\n", err)
        return
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        fmt.Printf("Unexpected HTTP status: %d\n", resp.StatusCode)
        return
    }

client.Do(req): Sends the HTTP request and stores the response.

defer resp.Body.Close(): Ensures the response body is closed after reading.

resp.StatusCode: Checks the HTTP status code. A code other than 200 indicates an error.

Reading And Printing The Input

Finally, we read the response body and print the puzzle input.

body, err := io.ReadAll(resp.Body)
    if err != nil {
        fmt.Printf("Error reading response body: %v\n", err)
        return
    }

    fmt.Println(string(body))

io.ReadAll(resp.Body): Reads the response body.

string(body): Converts the body from a slice of bytes to a string for easy display.

Defining The Main Function

We invoke the readInput function from the main function to fetch the input for day 1.

func main() {
    readInput("1") // Fetches input puzzle for day 1
}

Enhancing Security

Hardcoding the session token in our code isn’t safe. Instead, we should store it as an environment variable using the steps below:

Export the session token using the terminal:

export AOC_SESSION="[YOUR_SESSION_TOKEN]"

Modify the code to read the session token from the environment variable. (Ensure to have "os" among your imports):

req.Header.Add("Cookie", "session="+os.Getenv("AOC_SESSION"))

This helps, the session token stay outside the source code, reducing the risk of accidental exposure.

Full Program Code

Here's the complete program for reference:

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
)

const baseURL = "https://adventofcode.com/2024/day/%s/input"

func readInput(day string) {
    url := fmt.Sprintf(baseURL, day)

    client := &http.Client{}
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        fmt.Printf("Error creating HTTP request: %v\n", err)
        return
    }

    // Use session token from environment variable
    req.Header.Add("Cookie", "session="+os.Getenv("AOC_SESSION"))

    resp, err := client.Do(req)
    if err != nil {
        fmt.Printf("Error making HTTP request: %v\n", err)
        return
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        fmt.Printf("Unexpected HTTP status: %d\n", resp.StatusCode)
        return
    }

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        fmt.Printf("Error reading response body: %v\n", err)
        return
    }

    fmt.Println(string(body))
}

func main() {
    readInput("1")
}

Output

Things To Keep In Mind

Session Expiry: Session tokens may expire after a while. If you encounter issues, log in again and retrieve a fresh token.
Privacy: Never share your session token publicly, including in blog posts or GitHub repositories.

Conclusion

You can dynamically fetch your Advent of Code inputs by including your session cookie in HTTP requests.

Feel free to share your tips or ask questions in the comment section. Happy coding, and good luck with Advent of Code 2024!

MY EXPERIENCE ATTENDING GOPHERCON AFRICA 2024

John Eliud Odhiambo — Tue, 12 Nov 2024 01:18:04 +0000

What is Gophercon?

GopherCon is a global series of conferences dedicated to the Go programming language that provides a space for learning, networking, and collaboration within the Go community.

As the first-ever GopherCon Africa ended after a two-day successful conference, I reflected on what a monumental event it was for the Go programming community on the continent. Despite being the first of its kind, GopherCon Africa 2024 exceeded all expectations by bringing together developers of diverse experience levels, offering state-of-the-art talks, and creating countless opportunities for networking and growth for anyone who attended.

A Promising Beginning for the African Go Community

This being the inaugural GopherCon in Africa made it special. It felt like the start of something big and something that would grow and shape the future of the Go language across the continent. Attendees, speakers, and organizers all shared a sense of pride in being part of something so significant.

What made the event particularly successful was the way it struck a balance between serving seasoned developers and those just starting with Go. I earlier feared that the talks might cater only to advanced developers, leaving beginners behind, but the speakers did an exceptional job of making their content accessible to all, especially by giving room for asking as many questions as possible. Whether being interested in the technical talks or wanting to understand the broader vision for Go's future, there was something for everyone.

A Venue Fit for Innovation

The venue itself was nothing short of world-class. It was not just a place to sit and listen but a space that fostered meaningful conversations, learning, and connection. The environment played a huge role in the success of the event with modern facilities and thoughtful setups for both the talks and networking sessions. The venue elevated the experience and allowed me to focus on the content and connections.

State-of-the-Art Talks

The talks were the heart of the event, and they certainly delivered. I had the privilege of hearing from industry leaders who presented on a wide range of topics, from event-driven architecture to advanced testing strategies. One talk that stood out to me was a session on "Empowering the Future with Go: Africa’s Role in the Go Community." It was inspiring to hear how Africa can lead the charge in growing Go adoption globally. Adesina Hassan’s quote "If there’s no community around you, build one" hit home for me. It was a reminder that the future of Go in Africa is in our hands, and we all have a part to play.
The technical depth of the sessions was complemented by the speakers' ability to make the content relatable and actionable. Below are some topics that were covered at the conference:

Empowering the Future with Go: Africa’s Role in the Go Community
The keynote underscored the importance of building communities and encouraged developers to initiate Go user groups locally. The speakers’ message emphasized Africa’s unique role in advancing Go, highlighting how community-driven support can make a lasting impact.
Event-Driven Architecture with Go and Kafka
This session explored how Go can be used to create robust, event-driven architectures with Kafka. The talk covered the benefits of decoupling services, which improves scalability and resilience, and demonstrated how to use Kafka’s features to build a seamless data flow in distributed systems.
Effective Testing Strategies for Large Data Inserts
We learned strategies to ensure efficient, resilient, and scalable data operations. From unit and integration testing to load testing and benchmarking, the session emphasized the need for data integrity checks, latency testing, and optimizing indexing strategies for handling large data volumes.
Advanced Monitoring and Observability with OpenTelemetry and APItoolkit
This hands-on workshop provided a deep dive into leveraging OpenTelemetry alongside APItoolkit to develop comprehensive monitoring and observability for modern applications. Attendees learned practical techniques to instrument distributed systems and analyze telemetry data for improved insights.
SDK Secrets: Unlocking API Magic
This session was a deep dive into building efficient and user-friendly Software Development Kits (SDKs) for APIs. Key focus areas included error handling, concurrency, and designing SDKs that meet real user needs with structured logging, testing, and a strong emphasis on developer empathy.
Unveiling the Power of Interfaces: Go’s Secret Weapon
Go’s approach to interfaces stands out due to its implicit implementation and structural typing. This talk not only highlighted the benefits of Go’s interfaces for achieving modular and testable code but also delved into advanced techniques, comparing Go’s interface capabilities with those of other languages.
Taming Code Chaos: Reducing Entropy in Go Design Systems
Addressing common sources of code entropy, such as unstructured code and inconsistent styles, this session focused on modular architecture, dependency injection, and interface-driven design to create maintainable and scalable systems. The practical refactoring techniques discussed were valuable for increasing team efficiency.
Mastering Intra-service Communication in Go with gRPC
This technical session examined how Go and gRPC simplify communication in microservices architectures, addressing issues like latency, performance, and reliability. Comparing gRPC with REST, the talk demonstrated how gRPC can streamline cross-service interactions, particularly for high-performance applications.
Testing in Go: Unlocking Confidence and Efficiency
This talk emphasized the importance of test-driven development and table-driven tests, explaining how testing improves focus and builds confidence for developers. I learned practical techniques, including subtests, parallelism in testing, and using popular test packages like Testify and GoConvey to create robust test suites.
The State of Go in Africa
In a special panel discussion, industry leaders discussed the unique challenges and growth opportunities for Go across the continent. Panelists shared insights on Go adoption and how Go is being introduced in universities and boot camps across the region.

Each of these sessions provided actionable insights and real-world examples, reinforcing why Go continues to grow in popularity for both small and large-scale applications.

The Importance of Communities and Open-Source Contributions

A key theme throughout the event was the importance of communities. I repeatedly heard that joining (or starting) a Go community is one of the best ways to grow as a developer. Being part of a community helps one learn faster, share knowledge, and stay up-to-date with the latest developments in the Go ecosystem. And it’s not just about personal growth; communities help raise awareness about Go, making it more accessible to others and driving adoption.

Open-source contributions were also highlighted as an avenue to not only improve our technical skills but also gain visibility in the wider tech world. Africa, in particular, faces the challenge of limited job opportunities for Go developers. By contributing to open-source projects, we can showcase our skills to global audiences, helping to bridge that gap. Many speakers encouraged us to jump into open-source, not only to learn but to help build the Go ecosystem and demonstrate to employers what Go and African developers are capable of.

Networking

One of the best parts of GopherCon Africa was the networking opportunities. The event provided a rare chance to meet veteran Gophers face-to-face, exchange ideas, and even brainstorm potential collaborations. There were students, junior developers, industry veterans, startup founders, and entrepreneurs each bringing a unique perspective. The diversity of attendees made the networking sessions even more valuable, as we got to learn from each other’s experiences, challenges, and successes.
In one of the sessions, I had a conversation with a fellow developer about the increasing use of Go in fintech. We discussed how Go’s efficiency and scalability make it the go-to language for many fintech companies across Africa. This was just one example of the countless conversations that expanded my understanding of Go’s applications and the industries it’s shaping.

Creating More Awareness Around Go in Africa

One thing that became clear during the event is that there is still much work to be done to create awareness around Go in Africa. Go is already making waves in industries like fintech, cloud computing, and telecommunications, but there are still many developers and companies who aren’t aware of its potential. The event made it clear that we need to continue advocating for Go, both through community efforts and by encouraging companies to adopt it in their tech stacks.

Looking Forward: The Future of Go in Africa

As GopherCon Africa came to a close, I left feeling both inspired and optimistic. There is a real sense that this is just the beginning. The language’s simplicity, performance, and scalability make it perfectly suited for solving many of the challenges African developers face. And with a growing community of passionate developers, I am confident that Go will continue to thrive here.
We have the opportunity to drive Go adoption forward by contributing to open-source, building communities, and showing the world what African developers are capable of. I’m excited to see what the future holds, and even more excited to be part of it.

To many more successful GopherCons in Africa!

DEV Community: John Eliud Odhiambo

Where Were the Logs? Building a Log Ingestion Engine That Never Silently Drops a Log

1. The problem

2. The goal and the ground rules

Overview

Constraints

3. Architecture at a glance

4. The stages

4.1 Ingestion and validation

4.2 Protecting the boundary

4.3 The raw_logs buffer

4.4 Enrichment

4.5 Routing

4.6 Queuing with RabbitMQ

4.7 Storage, retries and dead letters

4.8 Observation

5. Designing for failure

6. Proof

7. Trade-offs and what I would change

8. Lessons

Review Questions

A Story About Pluralization In Code (2 Items vs 2 Boxes)

The First Spark Of Curiosity

First Attempt Towards A Smarter Solution

Discovering ICU4J

The Tempting Idea

Why the AI Idea Actually Makes Sense (At First)

The First Red Flag: Cost

The Second Red Flag: Latency

The Third Red Flag: Non-Determinism

The Fourth Red Flag: Control and Safety

The Practical Tradeoff

Final Code Snippet

Lessons Learned

Thought

The Hidden Environmental Cost of Your Cloud Storage

"The Cloud" Is Just Someone Else’s Computer (And It’s Using A Lot Of Energy)

The "Dark Data" Problem (Why Your Forgotten Files Matter)

What Can We Do? (Realistic Solutions)

Final Thought: The Cloud Isn’t Evil, But It’s Not Invisible Either

Understanding Load Shedding

Introduction

What Is Load Shedding?

Why Is It Necessary?

How Load Shedding Works In Code

Tracking Active Requests

The Shedding Logic

Key Decisions:

Essential vs. Non-Essential Services

Simulating Traffic Spikes

Putting It All Together

Demo Workflow

Try It Yourself

Real-World Applications of Load Shedding

E-Commerce Platforms

Cloud APIs

Social Media Platforms

Financial Systems

Advanced Considerations

Conclusion

Building a Dynamic User Profile Dashboard with GraphQL

Introduction

Project Overview

Technical Implementation

Authentication Flow

GraphQL Integration

Data Visualization

Theme Management

User Experience

Challenges and Solutions

Authentication Security

Dynamic Data Loading

SVG Rendering Compatibility

Conclusion

Live Demo

Will A Project Manager Expect Nine Expectant Mothers To Deliver Nine Babies In One Month? A Dive Into Project Management

What Is Project Management?

Who Is A PM, And What Role Do They Play?

What Is Agile Methodology?

The Project I Am Working On

4.3 The `raw_logs` buffer