DEV Community: Josephat Macharia

No cloud to lean on: building a reliable data platform on a bank's own servers

Josephat Macharia — Sun, 01 Nov 2026 08:35:07 +0000

Replication, failover, graceful shutdown, autoscaling and alerting usually arrive as managed services. Inside a bank, you build them from Kubernetes, an operator or two and a lot of small decisions. Here is how each one works, with the code.

Field notes from moving a transaction reconciliation platform from AWS onto Kubernetes inside banks' data centres

The constraints
Five words, one goal
Reliability: no work is lost silently
Fault tolerance: replication, redundancy, failover, graceful degradation
High availability: what survives what
Scalability on a fixed number of servers
Monitoring and observability without the internet
High availability is not a backup
Checklist

The constraints

In the cloud, our reconciliation platform ran on managed pieces: a managed database with automatic failover, managed queues, managed object storage, autoscaling groups and a monitoring SaaS. Banks wanted the same platform on their own hardware and their hardware comes with rules:

No internet. Images come from the bank's registry, backups go to the bank's storage and the only way out is the bank's mail relay.
A fixed number of servers. Three to seven machines, sized months in advance. Nothing appears when load goes up.
No managed services. Every database, queue and object store runs inside the cluster, so failover is our problem.
Little access. The people on call may not have cluster access and our engineers get in only through a bastion, when the bank allows it.

The platform itself is a typical event-driven data system. Users upload files through a web portal. Workers clean them, an ingestion engine loads them into an Iceberg data lake with Spark and reconciliations match transactions across sources. Every hop between services goes through a message queue.

Figure 1. The platform inside one Kubernetes cluster. The only things that leave are backups, email and image pulls and each goes to a system the bank controls.

Five words, one goal

These terms get used loosely, so here is what each one meant for us. They overlap and they all serve one goal we wrote at the top of the design: losing any one machine loses no data and stops nothing for more than seconds.

Concept	The question it answers	What it looks like here
Reliability	Does the system do the right thing, every time?	Work is acknowledged only after it is done; locks can't be taken twice; shutdown never drops a file
Fault tolerance	What happens when a part fails?	Replicated database and queues, spread replicas, automatic failover, failures contained to one job
High availability	How long are users affected?	Seconds for a lost server; zero for upgrades and node maintenance
Scalability	Can it keep up when the work grows?	Replica counts follow queue depth; heavy jobs wait for room instead of failing
Observability	Will someone find out and can they tell why?	Alerts by email, searchable logs, dashboards, a daily health report

Reliability: no work is lost silently

The rule underneath everything else: work moves through the queue and a message leaves the queue only when its work is done. A service that is down or busy leaves messages waiting. Nothing is lost while it recovers.

Acknowledge after the work, never before

A message broker like RabbitMQ hands a message to one consumer and marks it unacknowledged. It stays in the queue until the consumer says what happened. That gives you three outcomes and each one needs a deliberate answer.

Figure 2. Every path out of "unacknowledged" is deliberate. The message disappears only on success; everything else either retries or lands somewhere a person will see it.

Here is the pattern in a small consumer that forwards events from a queue to an HTTP endpoint. It acknowledges only after the endpoint answers 2xx and on any failure it puts the message back for another attempt:

Python (kombu)

def forwarder(endpoint, http_timeout):
    def on_message(body, message):
        try:
            response = requests.post(endpoint, data=to_payload(body), timeout=http_timeout)
            response.raise_for_status()
            message.ack()                      # only now does it leave the queue
        except Exception:
            logger.exception("Failed to forward message; requeueing")
            message.reject(requeue=True)       # back to Ready, tried again
    return on_message

Two settings make that safe at the broker. Quorum queues in RabbitMQ drop a message after 20 failed deliveries by default. A consumer that requeues on purpose while its downstream is down (the portal restarting, say) would lose messages after a few minutes of retries, so we lift the limit with a policy and rely on dead-letter queues and alerts for work that truly can't succeed:

curl -X PUT "$api/policies/%2F/redelivery" -H 'content-type: application/json' \
  -d '{"pattern": ".*", "apply-to": "quorum_queues",
       "definition": {"delivery-limit": -1}, "priority": 0}'

Watch out for the poison message. Unlimited redelivery has a dark side. A message whose processing kills the consumer is redelivered to the next consumer, which dies too.

Figure 3. The poison loop we hit on a test server. One large file was redelivered from pod to pod, killing each one at its memory limit.

The fix was sizing, not code: the file cleaner's memory limit went from 2 GiB to 4 GiB and a day later the pods still had no restarts. The lessons are general, though:

Size memory limits for the largest input, not the average one. An OOM kill is instant: no exception handler runs, nothing is logged by the app and the message goes back to the queue.
Alert on OOM kills and on restarts, so a loop shows up in minutes rather than as "the system is slow today".
Stream large inputs where you can, so memory stays flat whatever the file size.

Consumers that can't stall silently

A web service has an obvious health check: send it a request. A queue consumer has nothing to probe and its worst failure leaves the process running. The connection drops, the consume loop blocks forever, the pod shows Running and the queue grows behind it.

Figure 4. A heartbeat written only by the loop that does the work turns "running but useless" into a restart.

The consumer touches a file at the end of each pass through its loop. Long jobs run on their own threads, so a big file never holds the loop up and the heartbeat really does mean "this process is still taking work":

Python

LIVENESS_FILE = pathlib.Path("/tmp/heartbeat")

while not stopping:
    try:
        conn.drain_events(timeout=5)
    except socket.timeout:
        pass
    conn.heartbeat_check()
    LIVENESS_FILE.touch()          # only a working loop gets here

Kubernetes checks the file's age. A startup probe gives the process time for its first beat; after that, three stale checks in a row restart it:

Kubernetes

startupProbe:
  exec:
    command: ["sh", "-c", 'f=/tmp/heartbeat; [ -f "$f" ] && [ $(( $(date +%s) - $(stat -c %Y "$f") )) -lt 120 ]']
  periodSeconds: 10
  failureThreshold: 30          # up to 5 minutes for the first heartbeat
livenessProbe:
  exec:
    command: ["sh", "-c", 'f=/tmp/heartbeat; [ -f "$f" ] && [ $(( $(date +%s) - $(stat -c %Y "$f") )) -lt 120 ]']
  periodSeconds: 30
  failureThreshold: 3

Probes can cause outages too. RabbitMQ's usual readiness check, rabbitmq-diagnostics ping, starts a whole separate Erlang runtime to answer. On a busy node that timed out while the broker was healthy, which pulled it out of the Service and refused every client. A plain tcpSocket check is a kernel-level connect with no process to start and it stays reliable under the same load.

Locks that can't be taken twice

Two reconciliations on the same account at the same time would write conflicting results, so each run locks its accounts first. The obvious way ("check whether a lock exists, then write one") lets two triggers that arrive together both see "unlocked". The lock has to be a single conditional write and it has to record who owns it:

SQL (illustrative)

-- Take the lock only if nobody holds a live one. Exactly one caller gets a row back.
INSERT INTO account_locks (account_id, owner, expires_at)
VALUES (:account, :owner, now() + interval '1 hour')
ON CONFLICT (account_id) DO UPDATE
  SET owner = EXCLUDED.owner, expires_at = EXCLUDED.expires_at
  WHERE account_locks.expires_at < now()          -- take over only a stale lock
RETURNING account_id;

-- Release only your own lock, never someone else's.
DELETE FROM account_locks WHERE account_id = :account AND owner = :owner;

Lock several accounts in a fixed order and on a conflict release the ones already taken. Two runs grabbing the same accounts in different orders is the classic deadlock.
Every lock-holding workflow releases its locks on failure, on every step. Otherwise a failed run blocks the account until the lock expires.
Keep the lock table in the replicated, backed-up database. We moved this state from DynamoDB in the cloud to a Postgres table behind the same interface. Adding a separate store would have meant one more thing to make highly available and back up.

Things that must happen once

Some work must never run twice at the same moment and Kubernetes happily starts several copies of everything.

Database migrations. On a first install, or when scaling up from zero, every portal pod starts together and concurrent migrate runs on an empty database fail on duplicate keys. Each pod's init container takes a Postgres advisory lock first. One pod migrates and the others wait and then find nothing left to do:

Python (init container)

import django
django.setup()
from django.core.management import call_command
from django.db import connection

with connection.cursor() as cursor:
    # Held by this session until the process exits.
    cursor.execute("SELECT pg_advisory_lock(hashtext('portal-migrate'))")
call_command("migrate", interactive=False)

The scheduler. Two copies of a periodic-task scheduler fire every task twice. It runs as one replica with the Recreate strategy, so even an upgrade stops the old pod before starting the new one. Everything else rolls with a surge, so a new pod is ready before an old one goes:

# scheduler: never two at once
strategy:
  type: Recreate

# portal: never fewer than the desired count during a rollout
strategy:
  type: RollingUpdate
  rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }

Graceful shutdown: finish the work in hand

Pods are stopped all the time: every upgrade, node patch and scale-down. Kubernetes sends SIGTERM, waits for the grace period, then kills the process. A worker that ignores SIGTERM loses whatever it was doing on every deployment.

Figure 5. Graceful shutdown has two good endings. Either the work finishes and is acknowledged, or it is left unacknowledged and redone elsewhere. The bad ending, a half-finished file marked as done, can't happen.

Python

stopping = False

def on_sigterm(signum, frame):
    global stopping
    stopping = True                  # finish the current message, then leave

signal.signal(signal.SIGTERM, on_sigterm)

with Connection(broker_url, heartbeat=60) as conn, conn.Consumer(queues, callbacks=[on_message]):
    while not stopping:
        try:
            conn.drain_events(timeout=5)   # returns at least every 5s to check the flag
        except socket.timeout:
            pass

Two details are easy to miss:

exec the real process in a shell entrypoint (exec celery worker ...). Otherwise SIGTERM goes to the shell, which ignores it and the worker is killed at the end of the grace period with no warning.
Know when your framework acknowledges. Celery acknowledges a task when it starts by default, so a task cut off by SIGKILL is gone. acks_late fixes that, but then a task runs again after a crash, which is only safe when every task can run twice. We left it off until every task is idempotent.

Component	On SIGTERM	Grace period
File and report workers	Stop consuming, finish and acknowledge the job in hand	900s / 1,800s
Celery workers	Stop taking tasks, finish running ones	900s
Ingestion engine	A `preStop` drain: stop picking up runs, wait for its own to finish	1,800s
Event forwarder, notifier	Finish the current message or email	60s
Message broker node	Wait until every queue keeps a majority without it (see failover below)	Up to the grace period
SQL engine worker	Finish running query tasks	Until its tasks end

Fault tolerance

Fault tolerance is four techniques working together. Replication keeps the data in more than one place. Redundancy keeps more than one copy of each service running, on different machines. Failover moves the work to a healthy copy when one dies. Graceful degradation keeps the rest working, more slowly if it must, when something can't be replaced.

Replication: the database

Postgres runs as a three-instance cluster managed by the CloudNativePG operator: one primary and two streaming replicas, each on a different server. Every service connects to one Service name (postgres-rw) and the operator always points it at the current primary.

We chose it because it uses the Kubernetes API itself to decide which instance is primary. There is no extra consensus store (Patroni-based setups need etcd or similar) and no failover scripts of our own. WAL archiving and point-in-time recovery are built in, it's a CNCF project and it's on OpenShift's OperatorHub, which matters to banks that run OpenShift.

Figure 6. Synchronous, quorum-style replication. A commit the application was told about survives the loss of any one server.

CloudNativePG Cluster

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres
spec:
  instances: 3
  primaryUpdateStrategy: unsupervised   # roll replicas, then switch over, no human needed
  smartShutdownTimeout: 15              # default 180s: see below
  enablePDB: true
  affinity:
    enablePodAntiAffinity: true         # one instance per server
  postgresql:
    synchronous:
      method: any                       # synchronous_standby_names = 'ANY 1 (...)'
      number: 1
      dataDurability: preferred         # degrade to async rather than stop writes

preferred rather than required is a graceful-degradation choice. With required, every write stops while no replica is up and a rolling update or a lost server would stop the platform. The window where only one copy exists is short and an alert says when it opens.

One default worth changing: smartShutdownTimeout. A stopping primary waits for open connections to close before a replica takes over. Django keeps its database connections open, so the default 180 seconds meant three minutes of errors on every primary restart. Fifteen seconds is plenty.

Replication: the message queues

RabbitMQ runs as a three-node cluster with quorum queues: each queue is a Raft group copied on all three nodes. A message is confirmed to the publisher once a majority has it, so the queue keeps working and loses no confirmed message, while any two nodes are up.

Figure 7. A quorum queue tolerates one lost node out of three. Two lost nodes stop it, by design, rather than risk two diverging copies.

rabbitmq.conf

# Every node finds the others through the Kubernetes API.
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.address_type = hostname

# Every durable queue a client declares without a type becomes a quorum queue,
# so no client code has to change.
default_queue_type = quorum

# A queue declared while the cluster was still forming gains replicas as nodes join.
quorum_queue.continuous_membership_reconciliation.enabled = true
quorum_queue.continuous_membership_reconciliation.target_group_size = 3

Setting the default at the broker was the important part. Every service that declares queues kept its code unchanged. Celery was the one exception. Without declaring quorum queues itself, it asks for a global QoS setting that quorum queues reject and it can't consume at all. One setting fixed it: CELERY_QUORUM_QUEUES, read from the environment.

Redundancy: copies on different machines

Three copies of a service on one server are one failure away from zero. Every workload with more than one replica prefers to spread across servers and has a PodDisruptionBudget, so a node drain (patching, a hardware swap) can't take every copy at once.

Figure 8. Spreading puts each copy on a different server. The disruption budget makes a drain wait whenever taking the next pod would leave too few copies.

Both come from two small Helm helpers that every Deployment includes:

Helm helpers

{{- define "chart.spreadReplicas" -}}
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:   # prefer, so a 1-server install still schedules
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app: {{ .app }}
{{- end -}}

{{- define "chart.pdb" -}}
{{- if gt (int .replicas) 1 }}   # a PDB over one replica would block drains forever
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ .app }}
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: {{ .app }}
{{- end }}
{{- end -}}

Both helpers handle the small case on purpose. Some installs are a single server for user acceptance testing. A hard anti-affinity rule would leave pods Pending there and a disruption budget over one replica would make every drain hang.

Failover: when a copy dies

The database fails over by itself. When the primary's pod or server is lost, CloudNativePG promotes the most up-to-date replica within seconds and moves postgres-rw to it. Clients see their connections drop and reconnect. With synchronous replication, no committed transaction is lost. The old primary rejoins as a replica when it comes back.

The broker needs care at planned shutdowns. Stopping a RabbitMQ node while another is already down would take queues below a majority. So before a node stops, a preStop hook waits until every quorum queue would still have a majority without it, then moves its queue leaders away:

StatefulSet container

terminationGracePeriodSeconds: 600
lifecycle:
  preStop:
    exec:
      # Wait for quorum without this node, then hand off its leaders.
      # Gives up before the grace period ends, so a node can still stop
      # if another one is down for good.
      command: ["/bin/sh", "-c",
        "rabbitmq-upgrade await_online_quorum_plus_one -t 540; rabbitmq-upgrade drain -t 30 || true"]

Stateless workers with long-running state are the hard case. The ingestion engine runs two or more pods, each running reconciliation workflows that hold account locks. If a pod dies mid-run, someone has to notice, mark its runs as failed and free its locks. Nobody else can tell a dead pod from a busy one, so each pod writes a heartbeat row and every pod checks the others':

Figure 9. Peer failure detection with heartbeats. Any surviving pod can clean up after a dead one and it frees exactly the locks the dead pod's runs held.

The subtle part is not judging others while you're the one in trouble. If the database is down for two minutes, every pod's heartbeat fails and when it returns every heartbeat looks expired. A pod therefore judges others only after its own heartbeats have succeeded for a full timeout period. By then every live pod has written a fresh row too:

Python

class RunnerHeartbeat:
    def __init__(self, table, runner_id, clock=time.monotonic):
        self.table, self.runner_id, self.clock = table, runner_id, clock
        self.healthy_since = clock()

    def beat(self):
        try:
            self.table.put(runner_id=self.runner_id, expires_at=int(time.time()) + HEARTBEAT_TTL)
        except Exception:
            self.healthy_since = self.clock()      # our own view is unreliable: start over
            logger.exception("Runner heartbeat write failed")

    def may_reap(self):
        """True once our own heartbeats have succeeded for a full TTL:
        any live runner has then written a fresh row too."""
        return self.clock() - self.healthy_since >= HEARTBEAT_TTL

Crash and alert, don't re-run. A dead pod's runs are marked Crashed, not retried automatically. A reconciliation stopped half-way may have written part of its results and running it again isn't known to be safe. Nothing is lost silently: the run shows as Crashed, the account is unlocked and a person is told. Automatic retry is a feature you earn by making the work idempotent.

Graceful degradation: fail small, slow down rather than fall over

Some failures can't be hidden. The aim then is that they stay small and that overload produces waiting rather than errors.

Figure 10. Isolation as graceful degradation. Running each heavy job in its own pod keeps a failure the size of the job.

Heavy jobs fail alone. We considered Spark standalone (driver inside the service) and Spark's own Kubernetes mode, whose spark-submit exits 0 even when the job fails. A plain Kubernetes Job already gives a status, a deadline, retries, clean-up and logs.
Too much work waits instead of failing. Without a limit, a burst of uploads starts every Spark Job at once. Those the cluster has no room for sit Pending with their deadline running and time out without having run. A queue in front of them (next section) turns a burst into a backlog.
Downstream outages become retries. When the portal restarts, the event forwarder's requests fail and the messages are requeued until it's back. Users see their notifications a minute late, not never.
Replication degrades to async rather than stopping writes (Figure 6).
Telemetry never blocks the app. Services send traces and metrics to a gateway asynchronously. If the gateway or log store is down, the services carry on and the gateway buffers what it can.
CPU limits slow a pod down; memory limits kill it. That asymmetry decides how to set each one, as the scalability section explains.

High availability: what survives what

High availability is the sum of the above, measured from the user's side. We kept one table in the architecture docs that answers "what happens if X fails?" and every design change had to keep it true:

If this fails	What happens	Users notice
A database pod or its server	A replica is promoted; no committed transaction is lost; services reconnect	A few seconds of errors
A message broker node	Every queue keeps a majority on the other two; no confirmed message is lost	Nothing
A portal pod or its server	The other replicas keep serving; the pod is rescheduled	Nothing
A workflow pod mid-run	Its runs are marked Crashed within 3 minutes, their locks freed, an alert sent	That run shows as Crashed
A pod being stopped (upgrade, drain)	It finishes its current work first	Nothing
A heavy job	Only that job fails, with its error in the run	One failed upload
The whole cluster	Reinstall and restore from off-cluster backups	An outage, within the agreed recovery time
The object store's disk	Not covered by HA yet: restore from the last bucket copy	An outage

That last row matters. A useful HA table lists what isn't covered, so the bank can accept it in writing or fund the fix. Ours is the object store, which still sits on a single disk. It's on the open-items list the bank signs at go-live.

Upgrades are the most common "failure" of all, so they get the same treatment. Setup work runs as ordinary Jobs named after the Helm revision, not as Helm hooks. That way helm upgrade --install --wait --atomic works end to end and a failed upgrade rolls itself back:

helm upgrade --install platform . -f values-bank.yaml -n platform --wait --atomic --timeout 20m

Scalability on a fixed number of servers

In the cloud, scaling means adding machines. A bank's cluster has the servers it has. Autoscaling there doesn't add capacity. It moves the capacity you have to whichever queue is backed up, which pays off when work arrives in bursts, as uploads and exports do.

Scale on the work waiting, not on CPU

CPU is the wrong signal for a queue consumer. A worker waiting on the database or the object store uses little CPU while messages pile up behind it.

Figure 11. Queue depth and CPU for the same worker during an upload burst. Only one of them says "more workers needed".

KEDA scales each worker Deployment on its queue length (and for one worker on CPU too, whichever asks for more). It manages the Horizontal Pod Autoscaler itself and it reads queue sizes with a broker user that can see queues but not use them:

KEDA ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: file-cleaner
spec:
  scaleTargetRef:
    name: file-cleaner
  minReplicaCount: 3
  maxReplicaCount: 8
  advanced:
    restoreToOriginalReplicaCount: true
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown: { stabilizationWindowSeconds: 300 }   # 5 quiet minutes before removing pods
  triggers:
    - type: rabbitmq
      metadata:
        protocol: http
        host: http://rabbitmq:15672/%2F
        queueName: file-cleaning
        mode: QueueLength
        value: "5"                                         # one pod per 5 messages waiting
      authenticationRef: { name: keda-rabbitmq-readonly }

The workflow engine is different. Its trigger queues empty almost at once into orchestrator runs, so queue length says nothing about its load. It scales on runs waiting or running, through a Postgres role that can do exactly one thing, call a function returning that count:

triggers:
  - type: postgresql
    metadata:
      userName: keda                     # EXECUTE on one function, nothing else
      dbName: orchestrator
      query: "SELECT keda_active_flow_runs()"
      targetQueryValue: "6"              # one pod per maxConcurrentRuns

One detail for reading the numbers: the autoscaler ignores differences under 10%. With 2 pods of 6 runs each, 13 runs (6.5 per pod) doesn't add a third pod; 14 does.

Admission control for heavy jobs

Scaling the workflow engine up means more Spark Jobs and those need real CPU and memory. Without a limit, more workflow pods would add load, not capacity. So the Spark Jobs go through a Kueue queue that holds a fixed budget of CPU and memory. A Job is created suspended and starts when the running Jobs leave room for it. Its deadline starts only then, so waiting no longer counts as failing:

Kueue ClusterQueue

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: spark
spec:
  queueingStrategy: BestEffortFIFO    # in order, but a small job may pass a large one that doesn't fit yet
  resourceGroups:
    - coveredResources: ["cpu", "memory"]
      flavors:
        - name: spark
          resources:
            - name: cpu
              nominalQuota: "8"       # the room kept for Spark beside everything else
            - name: memory
              nominalQuota: 24Gi

Kueue's admission webhook refuses to create any Job in the namespace while Kueue is down. That covers setup Jobs and backup runs as well as Spark Jobs. A new dependency is also a new failure mode, so Kueue runs two replicas with a disruption budget, only manages namespaces labelled for it and has its own alert after two minutes down.

Requests, limits and the throttling trap

Every container has CPU and memory requests (what the scheduler reserves) and limits (the ceiling). The two kinds of limit behave very differently:

	CPU limit	Memory limit
When exceeded	The pod is throttled: it waits for the next 100 ms period	The container is killed (OOMKilled)
What users see	Slowness	Restarts, redelivered work (Figure 3)
How we set it	Enough for the process's real parallelism	The largest input plus headroom; alert at 90%

The parallelism point caught us. Celery's prefork pool runs four child processes and the workers had a CPU limit of half a core. Four processes shared 500m, so every task ran at an eighth of a core. The Grafana panel "held back by CPU limit" showed it plainly. A 2-core limit matched the pool and the requests stayed small so the scheduler still packs pods tightly.

Throttling alerts have their own trap. Throttling is measured as throttled periods ÷ periods. A pod that is idle almost all the time but spikes briefly, a health check that imports a large app, say, can show 60% throttling while using 2% of its limit. Alerting on that ratio alone pages people about pods that are fine. Our alert fires only when the pod is also using a real share of its limit:

PromQL (alert rule)

(
  sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="platform"}[5m]))
  / sum by (namespace, pod) (rate(container_cpu_cfs_periods_total{namespace="platform"}[5m]))
) > 0.25
and on (namespace, pod)
(
  sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{namespace="platform",container!=""}[5m]))
  / sum by (namespace, pod) (max by (namespace, pod, container) (kube_pod_container_resource_limits{namespace="platform",resource="cpu"}))
) > 0.5

Monitoring and observability without the internet

Everything above assumes someone notices when something goes wrong. In a bank there's no internet, often no Slack and the people on call may have no cluster access. Our monitoring had to work entirely inside the cluster and reach people only through the bank's mail relay.

Figure 12. Three signals, one place for alerts. Logs are collected from every pod, instrumented or not; traces and metrics pass through one redacting gateway; every alert is evaluated in Prometheus and leaves by email.

Collect from everything, redact in one place

Logs come from the container log files, not from the apps. The database, the broker and third-party components never send telemetry and they're often the ones that fail. A Vector agent on every node tags each line with its namespace, app, pod and parsed log level.
Traces and metrics go through one gateway (the OpenTelemetry Collector). It strips what a bank must not store, such as SQL text, credentials, cookies and URL query strings, once for every service. It also buffers through a log-store restart.
Each service is named by its app label everywhere, so its logs, traces, metrics and alerts join on one name.
The log store keeps its data in the platform's own object store and Postgres. There's no extra disk to provision or back up.

Alerts that mean something

Prometheus evaluates the rules and Alertmanager sends email through the bank's relay. A bank's cluster can't reach Slack or PagerDuty, but it always has a mail relay. Each alert links to a runbook: what it means, what's affected, how to check, how to fix, when to escalate. A few of the rules show the thinking.

A queue with messages and nobody consuming it. This catches a dead consumer within minutes, whatever the reason:

- alert: QueueNoConsumers
  expr: |
    max by (namespace, vhost, queue) (rabbitmq_detailed_queue_messages_ready{queue!~".*(retry|dead-letter).*"}) > 0
    and on (namespace, vhost, queue)
    max by (namespace, vhost, queue) (rabbitmq_detailed_queue_consumers) == 0
  for: 10m
  labels: { severity: critical }
  annotations:
    summary: "Queue {{ $labels.queue }} has messages but no consumer for 10m"
    runbook_url: "https://runbooks.example/#queuenoconsumers"

A backup that silently stopped. This one alerts on the absence of success, not on a failure. A CronJob that never runs (suspended, a refused admission, a broken schedule) never fails either:

- alert: BackupMissing
  expr: |
    time() - (
      max by (namespace, cronjob) (kube_cronjob_status_last_successful_time{cronjob=~".*-backup-.*"})
      or max by (namespace, cronjob) (kube_cronjob_created{cronjob=~".*-backup-.*"})   # a new CronJob counts from its creation
    ) > 26 * 3600
  for: 30m
  labels: { severity: critical }

Duplicates held back. kube-prometheus-stack ships generic rules for the same failures. While our more specific alert fires for an object, the generic one for it is inhibited, so on-call gets one email per problem:

inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: [alertname, namespace]
  - source_matchers: [alertname="PlatformPodCrashLooping"]
    target_matchers: [alertname="KubePodCrashLooping"]
    equal: [namespace, pod, container]

Other choices that keep the noise down:

Error-line alerts come from a metric, not a log query. Vector counts ERROR lines per app and Prometheus scrapes the count, so log alerts share the routing, grouping and silencing of every other alert.
Retry and dead-letter queues are left out of backlog alerts. They hold messages by design. Dead letters get their own alert.
A Job alerts only after it has used up its retries. A setup Job that succeeds on its second try stays quiet.

Check from the users' side

Every pod can be healthy while nobody can log in: a DNS change, a load balancer, an expired certificate. A blackbox exporter fetches the portal, the object store and the other endpoints, both inside the cluster and at their public addresses. It records whether each answered, how fast and when its certificate expires. Vector also turns the portal's access log into request counts and response times, which works behind any ingress controller or OpenShift's router.

Mark every change on every chart

The first question in an incident is "what changed?". Grafana marks every Helm install, upgrade and rollback and every database failover, on every chart. They come from metrics Prometheus already has (kube-state-metrics exposes the Helm release Secrets and CloudNativePG's exporter exposes the primary), so nothing has to be recorded by hand.

A daily report, which is also a dead man's switch

Nothing reports to us from a bank on its own. So a CronJob builds a support bundle inside the cluster every morning, with read-only access. That covers pod states, events, recent logs, memory and CPU pressure, queue depths, database activity and Helm values with every credential redacted. It keeps the last 14 bundles in a bucket and emails a short report: each component's state, the problems found and what to do about each. It contains no logs and no data.

The report has a second use. When an install's report doesn't arrive by mid-morning, we ask. Its monitoring, its relay or the platform itself may be down and a platform that is down can't send alerts. A signal that must keep arriving is the only way to detect the failure of the alerting path itself.

Case study: the disk alert that wasn't about volumes. Volume-filling alerts fired on a single-server install, yet the volumes held little. On k3s, local-path volumes have no quota, so they report the node's whole disk and the disk was 82% full of old container images from months of upgrades. Pruning brought it to 58%. The lasting fix wasn't a cron job, which banks dislike on their servers. It was the kubelet's own image garbage collection, set to start at 75% disk use and stop at 65%.

# /etc/rancher/k3s/config.yaml (OpenShift: a KubeletConfig with imageGCHighThresholdPercent)
kubelet-arg:
  - "image-gc-high-threshold=75"
  - "image-gc-low-threshold=65"

High availability is not a backup

Replication protects against a machine dying. It does nothing against a bad upgrade, a deleted table or a corrupted volume: those are copied faithfully to every replica within milliseconds. That needs copies stored somewhere else and a restore that is scripted.

Figure 13. Replication and backups solve different problems. Point-in-time recovery always builds a new cluster, so the archive it reads from is never overwritten and a recovery that goes wrong can be repeated.

Three copies leave the cluster, all to the bank's own S3-compatible storage. That might be a storage appliance or a separate MinIO, ideally in another data centre. It is never the in-cluster object store, which is one of the things being backed up.

What	How	Kept
Postgres WAL and base backups	CloudNativePG's Barman Cloud plugin: each WAL segment when full or after 5 minutes; a base backup daily	35 days
Every database and role	A nightly `pg_dumpall` CronJob: a plain, portable copy for the scripted restore	35 days
Every bucket	A nightly or hourly `rclone copy`, which never deletes on the target	Never deleted

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-postgres
spec:
  schedule: "0 1 * * *"
  concurrencyPolicy: Forbid          # a slow night never overlaps the next one
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          containers:
            - name: upload
              command: ["/bin/sh", "-c"]
              args:
                - |
                  set -e
                  rclone copy /dump target:$BUCKET/postgres
                  rclone delete --min-age 35d target:$BUCKET/postgres

Two stores must agree

An Iceberg table has two halves: data files in the object store and a catalog in Postgres that says which files make up each table. Postgres's WAL can be hours newer than the nightly bucket copy. Restored to its latest point, the catalog would point at data files the restored store doesn't have.

So the bucket copy writes a LAST_BACKUP marker with its start time to the backup target. Recovery puts Postgres back to that moment, not later. Keeping the marker off-cluster means the consistent point is still known after the whole cluster is gone. For a smaller gap, copy the buckets more often than you dump the database.

A restore you haven't run is a hope, not a plan. The restore is a script that scales everything down, refuses to continue while anything else is connected to the database, restores and scales back up. The go-live checklist includes a timed restore drill, so the recovery time the bank signs off on is a measured number.

Checklist

Every hop between services goes through a durable, replicated queue; messages are acknowledged after the work
Memory limits are sized for the largest input and OOM kills and restarts alert
Queue consumers have a heartbeat written by the working loop and a liveness probe that checks it
Locks are single conditional writes with an owner, released on every failure path
Migrations take a lock; the scheduler runs exactly once (Recreate)
Every process handles SIGTERM: stop taking work, finish, acknowledge, exit; grace periods fit the longest job
The database replicates synchronously to at least one replica, on another server
Queues are quorum queues; broker nodes wait for quorum before stopping
Replicas spread across servers and have disruption budgets, without breaking single-server installs
Heavy jobs run in their own pods, admitted when there is room
Workers scale on work waiting, not CPU; CPU limits match the process's real parallelism
Alerts arrive by a channel the site allows, each with a runbook; duplicates are inhibited
Something must keep arriving every day, so a silent install is noticed
Backups leave the cluster, the database and object store restore to the same moment and the restore has been timed
The "what survives what" table is current, including what isn't covered

None of these ideas are new. What changes on a bank's servers is that no provider applies them for you, so each one becomes a line in a chart, a probe or a runbook. Writing them down, with the reason next to each, is what lets the next engineer change one without breaking the others.

Apache Iceberg in a data lake: how it works and how to keep it fast, cheap and simple

Josephat Macharia — Fri, 30 Oct 2026 03:20:48 +0000

An S3 bucket full of Parquet files is not a table. Iceberg turns it into one. Here is what happens under the hood, the traps we fell into and the settings that made our queries faster and our bill smaller.

Field notes from running Iceberg tables for transaction reconciliation on S3, Glue, Spark and Athena

The problem Iceberg solves

A classic data lake is just folders of files. To answer a query, the engine lists the folders, opens the files and hopes nobody was halfway through writing one. There are no atomic commits, no safe way to update a row and the schema is whatever the last writer decided.

Iceberg adds a thin layer of metadata files that record exactly which data files make up the table at each moment. The data stays as plain Parquet on S3. Everything clever happens in the metadata.

How Iceberg works: a tree of metadata

Figure 1. Every Iceberg table is a tree. The catalog holds one pointer. Everything below it is an immutable file on S3, so a reader always sees one consistent snapshot.

A commit writes new data files, then new metadata, then swaps the catalog pointer in one atomic step. Readers see all of the change or none of it.
A snapshot is an immutable list of files. Old snapshots stay until you expire them, which is what gives you time travel and rollback.
Engines never list S3 folders. They read the manifests, which already know every file's partition and per-column min/max values.

How a read finds its data

That last point is where the speed comes from. Before opening a single Parquet file, the engine uses metadata to throw away files that can't contain a match. It skips data in three layers:

Partition pruning. Manifests record each file's partition values. With a table partitioned by account_id, months(transaction_date), a filter on those columns drops whole accounts and months before any file is opened.
File-level min/max statistics. Manifests also store each column's min and max for every file. A filter like source_ref = 'TX-88213' skips every file whose range can't contain that value.
Parquet row-group statistics. Inside each file that survives, rows are stored in chunks called row groups, each with its own min/max per column. Whole chunks are skipped the same way.

Figure 2. Three layers of skipping. The first two use only Iceberg metadata, so they cost almost nothing. A query that skips all three layers reads the whole table.

Partition pruning, with a real query

Our table is partitioned by (client_id, account_id, months(transaction_date)). Every file belongs to exactly one partition, a tuple such as (42, 7, 2026-10) and Iceberg writes that tuple into the file's manifest entry. Take this query:

SELECT id, amount
FROM db.transactions
WHERE client_id = 42 AND account_id = 7
  AND transaction_date >= TIMESTAMP '2026-10-01'
  AND transaction_date <  TIMESTAMP '2026-10-13';

Figure 3. Partition pruning works in two passes over metadata. The manifest list rules out whole manifests and each remaining manifest rules out individual files. No data file is touched.

Translate the filter. The query never mentions a month. Because the partition spec says months(transaction_date), Iceberg derives month = 2026-10 from the date range on its own. This is hidden partitioning.
Read the manifest list. It stores, for each manifest, the lowest and highest value of every partition field it covers. m1 only covers July to September and m3 only covers clients 55 to 91, so neither is opened.
Read the surviving manifest. Each entry in m2 names one data file and its exact partition tuple. Only the five files in (42, 7, 2026-10) match.
Hand the five files to the next layer. All of this came from a handful of small metadata files.

When it doesn't help: the filter isn't on a partition column (reporting_tag, id, a map key), or the partition column is hidden inside a function the planner can't see through. The MERGE case study (Figure 14) is that second trap at full price.

File-level min/max statistics, with a real manifest

Partition pruning stops at the partition. The five files left in (42, 7, 2026-10) all have the same partition tuple, so it can't tell them apart. The next layer can. Each manifest entry also stores, for every column, the file's min value, max value, null count and row count. The query now adds one more filter:

  AND amount > 100000

Figure 4. Each bar is one file's min → max range, read from its manifest entry. A file is opened only if every filter could match inside its ranges. Here only f4 passes both.

Check the date. f1 ends on Oct 6, f2 ends on Oct 11 and f5 starts on Oct 16, so none can hold Oct 12. f3 and f4 both span Oct 12 because they came from different write batches and nothing stops two files' ranges from overlapping.
Check the amount. f3's largest amount is 87,000, so amount > 100000 can't match. f4's maximum is 250,000, so it might.
Open f4 only. Inside it, the row-group statistics in the next section narrow things further.

When it doesn't help:

Values are spread across every file. Unsorted data gives every file a range of Oct 1 → Oct 31, so all five are opened. That's why the sort order matters.
The value is inside a map. All keys share one value column, so its min/max mixes every key together (Figure 21).
The column has no statistics. By default Iceberg only collects them for the first 100 columns (write.metadata.metrics.max-inferred-column-defaults). For strings it keeps only the first 16 characters, which can blur long prefixes such as TXN-2026-10-….
The filter is a not-equals. is_reversal != true can't be ruled out from a min/max range, but is_reversal = false can.

Row-group statistics, with a real file

The third layer happens inside a Parquet file, so it helps to see one. Parquet doesn't store a file as one block of rows. It cuts the rows into row groups, about 128 MB each by default (write.parquet.row-group-size-bytes). Inside each row group, every column is stored separately as a column chunk. At the end of the file sits a small footer that records, for every row group and every column, the min value, the max value and the null count.

Take a 512 MB file of October transactions, sorted by transaction_date and this query:

SELECT id, amount
FROM db.transactions
WHERE transaction_date = DATE '2026-10-12';

Figure 3. Left: how the file is laid out. Right: what its footer says. Only the highlighted chunks are fetched from S3: 3 of the 16 chunks.

Here is what the reader does, in order:

Reads the footer first. It's a few KB at the end of the file, fetched with one small S3 range request.
Checks the filter against each row group's range. Oct 12 can't be in Oct 1 → Oct 8, so row group 1 is skipped. Row group 2 covers Oct 8 → Oct 16, so it might contain matches. Row groups 3 and 4 start after Oct 12, so they're skipped.
Fetches only the column chunks it needs from row group 2: transaction_date to filter, plus id and amount to return. The metadata chunk is never downloaded. This is the columnar part of Parquet.
Filters the rows in memory. "Might contain" isn't "does contain". The reader still checks each row in row group 2 and keeps the ones dated Oct 12.

The amount column shows why the order matters. A filter like amount > 300000 skips all four row groups, because every max is below it. But amount = 5000 falls inside every range, so nothing can be skipped. Statistics only help on columns whose values are clustered and in our tables that means the column the data is sorted by. Shuffle the same rows randomly and every row group reports Oct 1 → Oct 31, so the Oct 12 query reads all four (Figure 18 shows that case).

	Layer 2: file statistics	Layer 3: row-group statistics
Where they live	Iceberg manifests	The footer of each Parquet file
What gets skipped	Whole files, without opening them	Row groups inside a file that's already been opened
Cost to check	Nothing extra; read during planning	One small read per surviving file
Matters most when	A partition has many files	A partition is one or a few large files

Lesson: a query is fast when its WHERE clause hits partition columns and columns whose values are tightly grouped inside each file. Everything in the "faster queries" section below is about making that true.

How writes commit and why they collide

Iceberg uses optimistic concurrency. Each writer assumes it is alone, does its work, then tries to swap the pointer. If someone else committed first, it has to re-check and retry.

Figure 3. Conflicts are counted per table. Two jobs writing to different tables never wait on each other. Ten jobs writing to one table take turns.

Watch out: if you see many CommitFailedException retries, you have too many writers on one table. Batch small writes into fewer commits, keep independent write paths in separate tables and raise commit.retry.num-retries only as a safety net.

Updates and deletes: two strategies

Parquet files can't be edited. So when you UPDATE, DELETE or MERGE, Iceberg either rewrites files or writes small "delete files" next to them.

Figure 4. Copy-on-write makes writes pay. Merge-on-read makes reads pay and the bill grows with every unmerged delete file until compaction folds them back in.

Workload	Choose	Why
Frequent small `MERGE`s or updates (reconciliation, CDC)	Merge-on-read	Rewriting 512 MB to flip one flag is wasteful. Compact on a schedule.
Rare, large batch updates; read-heavy dashboards	Copy-on-write	Readers never pay a merge cost.
Append-only event logs	Either	No deletes, so the choice doesn't matter.

Set it per operation with write.update.mode, write.delete.mode and write.merge.mode.

Why does an update create a delete file?

Because Parquet files can't be edited, Iceberg has no real "update". Every UPDATE is carried out as two steps: delete the old version, then insert the new one. The write mode decides how the delete half is done.

Mode	Delete the old version	Insert the new version
Copy-on-write	Rewrite file A without the old row	The new row goes into that rewritten file
Merge-on-read	Write a delete file: "file A, row 51207 is gone"	Write a new data file holding the new row

So a delete file is merge-on-read's cheap way of saying "this old version no longer counts" without rewriting a 1M-row file. A plain DELETE works the same way minus the insert: it produces only a delete file. A MERGE is the same again: matched rows are deleted and re-inserted.

The three kinds of delete file

Kind	What it says	Who writes it	Cost at read time
Position delete	"file A, row 51207"	Spark, Athena and Trino `UPDATE` / `DELETE` / `MERGE`. The engine already scanned for the rows, so it knows exactly where they are.	Low: skip known positions
Equality delete	"any row where `id = 42`"	Streaming upserts (for example Flink), which don't look the row up first	High: every row of every older data file in scope is compared
Deletion vector (format v3)	A compact bitmap of deleted row positions, one per data file	The same engines, on v3 tables	Lowest: one small file per data file, replaced on each commit instead of piling up

Why the new row isn't deleted too

Figure 5. Every commit gets an increasing sequence number. A position delete names the exact file it applies to. An equality delete only applies to data files with a lower sequence number. Either way, the new row written in the same commit is safe.

The flow of a `DELETE`

Figure 6. A DELETE from plan to physical removal. Figure 7 shows how an UPDATE differs.

The flow of an `UPDATE`

An update follows the same path as a delete, with two differences. It always writes the new row versions, so there is no metadata-only shortcut. And at step 3 the two write modes split: copy-on-write rewrites files, while merge-on-read writes a delete file plus a new data file. A MERGE works the same way: matched rows take this path and unmatched rows are plain inserts.

Figure 7. An UPDATE from plan to physical removal. The two write modes only differ at step 3; everything after that is shared, though compaction matters far more for merge-on-read.

Merge-on-read, step by step: what happens to the delete file?

A common misreading is that the one-row file gets merged into file A when you read it and the delete file then disappears. It doesn't. Reads never change anything on disk. The files are only combined when compaction runs and only removed from S3 when the old snapshot expires.

Figure 8. The lifecycle of one merge-on-read update. The delete file stops being used at step 3 and stops costing money at step 4.

The one-row file is never appended to file A. Parquet files are immutable. Compaction writes a brand-new file A′ containing A's surviving rows plus the updated row and it may pack other small files from the same partition in with them.
The delete file is dropped from the table, then from S3. Once every data file it points at has been rewritten, the new snapshot no longer lists it. It is physically deleted only when expire_snapshots removes the last snapshot that uses it.
Until compaction runs, every read pays. Ten MERGEs on one partition mean ten delete files to apply on every query. That's why merge-on-read tables need compaction on a schedule.
Newer format, same idea. Iceberg format v3 replaces per-commit position delete files with one compact "deletion vector" per data file. That makes reads cheaper, but you still need compaction and snapshot expiry.

Watch out: a deleted row stays on S3 until both compaction and snapshot expiry have run. If you handle data-removal requests (GDPR-style "erase my data"), your deadline includes those two jobs, not just the DELETE.

Tip: a delete whose filter lines up with partitions is almost free. On a table partitioned by account_id, months(transaction_date), DELETE ... WHERE account_id = 42 AND transaction_date < TIMESTAMP '2026-01-01' can drop whole files without writing anything.

When `UPDATE` and `DELETE` get slow

Almost every slow update or delete comes down to one of two things: step 2 has to scan too much, or step 3 has to write too much.

Situation	Why it's slow	What to do
The `WHERE` clause has no partition filter	Step 2 scans every file in the table to find the rows	Always include the partition columns, for example account and date range
Filtering on a map key, JSON field or a function of a column	No file can be skipped, so the scan is as wide as the partition filter allows	Promote the field to a real column; compare raw columns to literals
Scattered changes on a copy-on-write table	One changed row per file still rewrites the whole file. 1,000 rows spread over 1,000 files means 1,000 × 512 MB rewritten	Use merge-on-read for scattered changes, or sort the table so rows that change together live in the same files
Many small statements, such as an update per row in a loop	Each statement scans, writes, commits and adds a snapshot and its own delete files	Collect the changes and apply them in one `MERGE` (below)
Several jobs changing the same partitions at once	Conflicts are detected at commit time, after all the work is done, so the loser redoes the whole statement	One writer per table or partition at a time; batch changes; keep filters narrow so conflict checks are scoped
Delete files have piled up	Step 2 has to apply all existing delete files while it scans, before it can even find the rows	Compact merge-on-read tables on a schedule
Rewriting most of a partition with merge-on-read	You get a delete file listing almost every row, plus a full copy of the new rows. Reads pay for both until compaction	For bulk changes, use copy-on-write for that operation, or `INSERT OVERWRITE` the affected partitions
Equality deletes from streaming upserts	Every read compares rows against them	Compact often; prefer writers that produce position deletes

How `MERGE` works

MERGE takes a source of changes (a staging table or query) and applies it to a target table in one statement. It joins the two, decides per row whether to update, delete or insert, writes the results once and commits once.

Figure 9. A MERGE is an update, a delete and an insert in one statement. However many rows it changes, it scans the target once and produces one snapshot.

Is `MERGE` the better way to change lots of rows?

For a batch of row-specific changes, yes. If you have 10,000 transactions that each need a different new value, a MERGE from a staging table does all of them in one pass. The same changes as 10,000 UPDATE statements repeat the whole flow 10,000 times.

Figure 10. The win comes from batching, not from the keyword. Each statement carries a fixed cost: scan, commit, snapshot and conflict check. MERGE pays it once.

It isn't automatically better in every case:

One set-based change: UPDATE t SET status = 'closed' WHERE batch_id = 7 is already a single pass. A MERGE would do the same work.
Most of a partition changes: INSERT OVERWRITE of that partition is cheaper. It writes new files and drops the old ones, with no delete files at all.
The target isn't pruned: a MERGE without partition filters on the target scans the whole table, which can be slower than a well-filtered UPDATE.

Writing a fast `MERGE`

MERGE INTO db.transactions t
USING (
  SELECT * FROM db.staging WHERE batch_id = 7      -- deduplicated: one row per id
) s
ON  t.id = s.id
AND t.account_id = 42                               -- prune the target
AND t.transaction_date >= TIMESTAMP '2026-09-01'
AND t.transaction_date <  TIMESTAMP '2026-10-01'
WHEN MATCHED AND s.op = 'delete' THEN DELETE
WHEN MATCHED AND t.status IS DISTINCT FROM s.status  -- skip rows that didn't change
  THEN UPDATE SET status = s.status, updated = s.updated
WHEN NOT MATCHED THEN INSERT *;                     -- Spark syntax; Athena needs a column list

Put the target's partition filters in the ON clause. They prune the target scan and also narrow the conflict check, so concurrent jobs on other partitions don't clash.
Deduplicate the source to one row per key. Duplicates make the MERGE fail.
Skip no-op rows with a condition such as IS DISTINCT FROM. Without it, unchanged rows still get a delete and a re-insert.
Keep the source small and filtered. A small source can be broadcast to every worker instead of shuffling both sides of the join.
Run one MERGE per table per batch, not several in parallel on the same partitions.

Case study: the `ON` clause that scanned 130 GB

That first bullet cost us more than any other mistake. Our reconciliation rules are MERGE statements on the transactions table itself. The source query finds matched pairs and the MERGE flags them. The source was carefully filtered to one client, a few accounts and a 30-day window. The ON clause looked like this:

MERGE INTO db.transactions r
USING (
  SELECT client_id, array_agg(id) AS matched_ids
  FROM db.transactions
  WHERE client_id = 42 AND account_id IN (7)          -- source is pruned
    AND CAST(transaction_date AS DATE) >= date_parse('2026-09-01', '%Y-%m-%d')
    AND CAST(transaction_date AS DATE) <= date_parse('2026-09-30', '%Y-%m-%d')
  ...
) t
ON  r.client_id = t.client_id                         -- join condition, not a filter
AND contains(t.matched_ids, r.id)                     -- can't prune anything
WHEN MATCHED THEN UPDATE SET matched = true;

A MERGE is a join between the source and the target. Filters inside the USING subquery only prune the source's own scan and they don't carry over to the target. On the target side, nothing compared a partition column to a constant. r.client_id = t.client_id is a join condition and contains(array, r.id) is opaque to the planner. So Athena read every partition of the target to look for matches.

Figure 11. The same MERGE before and after. The source was always pruned. The target was not, until its filters were written into the ON clause.

The fix was to repeat the source's static filters on the target alias, inside ON:

ON  r.client_id = t.client_id
AND r.client_id = 42                                  -- static filters on the target:
AND r.account_id IN (7)                               -- these prune the target scan
AND CAST(r.transaction_date AS DATE) >= date_parse('2026-09-01', '%Y-%m-%d')
AND CAST(r.transaction_date AS DATE) <= date_parse('2026-09-30', '%Y-%m-%d')
AND contains(t.matched_ids, r.id)

One rule, one account, 30-day window	Before	After
Data scanned	129.72 GB	984 MB (99.2% less)
Runtime	4 min 31 s	20 s (about 13× faster)
Athena cost per run, at $5 per TB scanned	about $0.63	about $0.005

We then applied the same pattern to every rule file. Across the board, data scanned per query fell by about 98%, from over 100 GB to under 10 GB. Because a reconciliation run executes dozens of these rules one after another, the savings multiply.

The same pass also made some smaller fixes:

Removed redundant date_trunc('month', CAST(transaction_date AS DATE)) filters. They added nothing the day-range filter didn't already say and a function wrapped around the column is the form planners are least likely to prune on.
Trimmed unused columns from the UNNEST subqueries. Parquet is columnar, so every column you don't select is bytes never read and fewer bytes to shuffle into the join.
Dropped an ORDER BY in the source. A MERGE doesn't care about the order of its source rows, so sorting them was pure cost.
Wrote is_reversal = false instead of is_reversal != true. Both exclude NULL, but the equality form is a plain value that file statistics can check.

Lesson: every filter on the source of a MERGE belongs on the target too, written against the target alias in the ON clause. Then check the "Data scanned" figure in the query stats. In Athena, that number is also your bill.

Things to watch for

Most Iceberg problems are slow leaks. Each write leaves a little behind: a small file, a delete file, a snapshot, a manifest. Nothing breaks on day one. By month three queries are slow and the S3 bill has doubled.

Figure 11. Small files are the most common Iceberg tax. Every file costs an S3 request, a manifest entry and some planning time, whatever its size.

Symptom	Usual cause	Fix
Queries slow down week by week	Thousands of small files from frequent small writes	Compact with `rewrite_data_files`; batch writes; set `write.target-file-size-bytes`
Reads slow after many `MERGE`s	Merge-on-read delete files piling up	Compact with a delete-file threshold; run `rewrite_position_delete_files`
Query planning takes seconds before any data is read	Too many snapshots or manifests	`expire_snapshots`, `rewrite_manifests`, cap `write.metadata.previous-versions-max`
S3 bill grows while the table doesn't	Unexpired snapshots; orphan files from failed jobs	`expire_snapshots`, then `remove_orphan_files`
A filter scans the whole partition anyway	Filtering on a value inside a map or JSON column, or on a function of a column	Promote the field to a real column; compare raw columns to literals
Millions of tiny partitions	Partitioning on a high-cardinality column (user, day for a quiet table)	Use coarser transforms like `months(ts)` or `bucket(16, id)`
Commit retries and timeouts under load	Many concurrent writers on one table	Fewer, larger commits; separate tables for separate write paths

Faster queries

1. Partition by how you filter and keep partitions big

Partition on the two or three columns almost every query filters on. Aim for each partition to hold at least a few hundred MB. Iceberg's hidden partitioning means you declare a transform once and users never see it:

CREATE TABLE db.transactions (...)
USING iceberg
PARTITIONED BY (client_id, account_id, months(transaction_date));

-- Users filter on the real column; Iceberg maps it to the month partition.
SELECT * FROM db.transactions
WHERE account_id = 42
  AND transaction_date >= TIMESTAMP '2026-09-01'
  AND transaction_date <  TIMESTAMP '2026-10-01';

Got it wrong? Partition specs can evolve without rewriting old data: ALTER TABLE db.transactions ADD PARTITION FIELD bucket(16, merchant_id).

2. Sort inside partitions so min/max statistics mean something

Figure 12. Each bar is one file's min-to-max range for merchant_id. Iceberg can only skip a file when the value you want falls outside its range, so random order makes the statistics useless.

ALTER TABLE db.transactions WRITE ORDERED BY merchant_id, source_reference;
-- Compaction then re-sorts existing data:
CALL catalog.system.rewrite_data_files(table => 'db.transactions', strategy => 'sort');

A worked example: our four-column sort order

Our transactions table is partitioned by (client_id, account_id, months(transaction_date)) and declares this sort order:

ALTER TABLE db.transactions
WRITE ORDERED BY transaction_date, client_id, account_id, reporting_tag;

It reads like clustering on four columns. In practice it clusters on one. Partitioning happens before sorting and every data file belongs to exactly one partition. So inside any file, client_id and account_id never change and sorting by a constant does nothing. transaction_date is a timestamp and almost unique, so reporting_tag only orders rows that share the exact same timestamp.

Figure 13. What the sort order does once partitioning has run. Only transaction_date changes the order of rows inside a file.

Rule of thumb: a sort column only helps if it still varies inside a partition and if the columns before it repeat often enough to leave ties for it to break.

To see what this means for queries, take one partition (one client, one account, October) holding six files and two filters our reconciliation rules use all the time:

Query A, one day, every tag: transaction_date >= TIMESTAMP '2026-10-12' AND transaction_date < TIMESTAMP '2026-10-13'
Query B, one tag, the whole month: reporting_tag = 'chargebacks'

Each rectangle below is one file's min/max box: its date range across, its tag range down. Iceberg reads a file only if the query's line or band touches its box.

Figure 14. Rows written in sort order. Files become date stripes, so date filters skip well. Every stripe spans the alphabet of tags, from CardPayments to withdrawals, so a tag filter skips nothing.

Then compaction changes the picture. Our nightly job rewrites the last three months with strategy => 'sort', sort_order => 'zorder(transaction_date, client_id, account_id, reporting_tag)'. A sort_order passed to rewrite_data_files overrides the table's own order for the files it writes. Z-order interleaves the bits of each column, so each file ends up covering a block of dates and a block of tags. The constant partition columns drop out again, which leaves date and tag sharing the ordering.

Figure 15. The same partition after z-order compaction. The dotted line is the order the curve visits the blocks. Date skipping gets weaker and tag skipping starts to work.

Filter	Linear by date	Z-order (date, tag)
A: one day, every tag	1 of 6 files	3 of 6
B: one tag, whole month	6 of 6	2 of 6
A and B together	1 of 6	1 of 6
`client_id` + `account_id` + month only	Partition pruning does all the work. Sort order doesn't matter.

There is one more layer. With a 512 MB target file size, a single account's month often fits in one to three files, so file-level skipping has little to choose from. The sort still pays off inside the file. Parquet stores min/max per row group and sorted rows give each row group a narrow date range.

Figure 16. Inside one file. Even when a partition is a single file, sorting by date lets the reader skip most of it. Unsorted, it reads everything.

What we took from this:

Partition columns in the sort order do nothing. Listing client_id and account_id is harmless, but it suggests clustering that isn't happening.
A column behind a timestamp in a linear sort does nothing. If tag filters matter, either z-order on date and tag, or put the low-cardinality column first: WRITE ORDERED BY reporting_tag, transaction_date gives each tag its own date-sorted run. The cost is that a one-day, every-tag query then touches one run per tag.
Don't let two layouts drift apart. New data lands linear by date. Compaction later turns it into z-order. Choose one layout on purpose, based on which filter your queries use most and set it in both places.
The sort only helps filters on the raw column. CAST(transaction_date AS DATE) or DATE_FORMAT(...) can hide the range from the planner (see tip 5 below).
Check the layout, don't guess it. In Spark, the files metadata table shows each file's ranges:

SELECT file_path, record_count,
       readable_metrics.transaction_date.lower_bound AS date_min,
       readable_metrics.transaction_date.upper_bound AS date_max,
       readable_metrics.reporting_tag.lower_bound    AS tag_min,
       readable_metrics.reporting_tag.upper_bound    AS tag_max
FROM catalog.db.transactions.files
WHERE partition.client_id = 42 AND partition.account_id = 7
ORDER BY date_min;
-- Narrow, non-overlapping date ranges: date skipping works.
-- tag_min = 'CardPayments' and tag_max = 'withdrawals' on every file: tag skipping doesn't.

3. Don't hide hot fields inside maps or JSON

A MAP<STRING,STRING> column is handy for odd extra fields. But Parquet stores every map key in one column and every value in another, so the min/max statistics mix all keys together.

Figure 17. Same data, two layouts. The map version must read and rebuild the whole map for every row. The promoted column lets Iceberg skip the file from metadata alone.

How a map column behaves in each of the three skipping layers:

Partition pruning still works, as long as the query also filters on partition columns. That is the only layer left.
File min/max is useless. The "metadata values" range covers rrn, status, amount and every other key mixed together, so no file can ever be ruled out.
Row-group statistics are useless too, for the same reason. Engines like Athena (Trino) and Spark don't push a map lookup into the scan at all. They read the column, rebuild the map for each row, then filter.

	Top-level typed column	Key inside a map
Files skipped by min/max	Yes	No
Bytes read	Only that column	The entire map, all keys and values, for every row in the scanned partitions
CPU	Compare values directly	Rebuild the map per row, look up the key, often `CAST` the string
As a join key	The engine can use the other side's values to skip files on this side (dynamic filtering)	Can't. `ON element_at(l.metadata, 'rrn') = r.x` is an expression, so every row is evaluated

A map is often one of the widest columns in the table, so reading all of it to check one key can be most of a query's scan.

What to do about it:

Promote the keys you filter or join on to top-level columns. In Iceberg, ALTER TABLE ... ADD COLUMNS only changes metadata, so it's cheap. Fill the new column at ingestion, backfill old rows once with an UPDATE or MERGE and keep the map for rare keys.
Give them real types, for example amounts as decimal(32,4), so min/max statistics and comparisons are correct.
Add Bloom filters for exact lookups on high-cardinality columns like IDs: 'write.parquet.bloom-filter-enabled.column.rrn'='true'. Check that your engine version actually reads them.
Rewrite the queries to use the new columns. Filters and join keys then get file skipping and dynamic filtering for free.

4. Joins: cheap when pruned, expensive when not

A join isn't slow by itself. On a data lake, its cost comes from three things:

Bytes scanned on each side. If both tables are filtered on their partition columns, each side is already small before the join starts. Put the partition filters on every table in the join, not just one.
The join key. On a plain column, Athena and Trino build a filter from the smaller side's keys and use it to skip files on the bigger side. A map lookup, CAST or function in the ON clause turns that off.
Rows exploded before the join. UNNEST and CROSS JOIN UNNEST(...) on arrays multiply rows before the join and can cost more than the join itself. Filter first, unnest last.

So joins on well-pruned plain columns are fine. A join keyed on map values over a wide date range is where queries get slow and it's slow for the map reasons above, not because it's a join.

5. Compare raw columns to literals

transaction_date >= TIMESTAMP '2026-09-01' prunes reliably. Wrapping the column in a function or cast, as in CAST(transaction_date AS date) = ..., may not, depending on the engine. Put the transformation on the literal side.

Cheaper, leaner storage

Iceberg never deletes anything by itself. Every rewrite leaves the old files in place for old snapshots. You need a maintenance routine and the order matters.

Figure 18. Compaction alone makes queries faster but storage bigger. Storage only shrinks once expired snapshots release the old files.

-- Spark (Iceberg procedures)
CALL catalog.system.rewrite_data_files(table => 'db.transactions',
  where => 'transaction_date >= current_date() - INTERVAL 7 DAYS');
CALL catalog.system.rewrite_manifests('db.transactions');
CALL catalog.system.expire_snapshots(table => 'db.transactions',
  older_than => TIMESTAMP '2026-10-22 00:00:00', retain_last => 10);
CALL catalog.system.remove_orphan_files(table => 'db.transactions');

-- Athena equivalents
OPTIMIZE db.transactions REWRITE DATA USING BIN_PACK;
VACUUM db.transactions;

Settings that shrink the bill

Setting	Suggested value	Effect
`write.parquet.compression-codec`	`zstd`	Usually smaller files than snappy for similar read speed. It's the default in recent Iceberg versions.
`write.target-file-size-bytes`	128–512 MB	Fewer files, fewer S3 requests
`write.metadata.delete-after-commit.enabled`	`true`	Old `metadata.json` files are cleaned up automatically
`write.metadata.previous-versions-max`	10–50	Caps how many old metadata files are kept
`write.metadata.metrics.default`	`truncate(16)`, with `none` for wide text columns you never filter on	Smaller manifests on wide tables
Column types	`decimal`, `date`, `bigint` instead of `string`	Better compression and min/max that compares correctly

What map columns cost in storage

Key names are stored on every row. The strings "rrn", "status" and so on repeat for each row. Parquet's dictionary encoding and compression shrink them well because they repeat, so this overhead is real but modest.
Values compress worse than typed columns. Everything is a STRING, so an amount is stored as text instead of a decimal. Mixing many keys in one column also makes a messier dictionary than separate columns would.
The bigger cost is reading, not storing. Athena bills per byte scanned and every filter on a map key scans the whole map.

Simpler queries, easier changes

Performance tuning is wasted if nobody can safely change the queries afterwards. These habits kept our SQL short and our migrations boring.

Typed columns beat map lookups in readability too

Before

SELECT *
FROM transactions t
WHERE CAST(element_at(t.metadata,
        'ReconAmount') AS decimal(32,4)) > 1000
  AND COALESCE(t.metadata['rrn'],
        t.metadata['reference']) = ?

After

SELECT *
FROM transactions t
WHERE t.recon_amount > 1000
  AND t.rrn = ?

The second version is shorter, prunes files and fails loudly at write time if the data is the wrong type, not silently at query time.

More habits

Lean on hidden partitioning. Never ask users to add a separate month = '2026-09' filter. They filter on the timestamp; Iceberg does the rest.
Evolve schemas freely, but never reuse a column name for new meaning. Iceberg tracks columns by ID, so add, rename and drop are safe metadata changes. Readers of old dashboards are not.
One table per write path, views for reading. Separate tables keep writers from colliding (Figure 3). A UNION ALL view gives analysts one place to query and engines push filters down into it.
Debug with time travel. SELECT ... FOR VERSION AS OF 4129... shows the table before a bad job. CALL catalog.system.rollback_to_snapshot(...) undoes it.
Keep table properties in code. One CREATE TABLE template, applied to every environment, stops tables drifting apart.

One table or three?

We keep transactions in three tables, one for each source: bank statements from banks and payment providers, our own ledger and external for other third-party feeds. They share one schema. Every so often someone asks whether one transactions table with a kind column would be simpler. It's a fair question and the answer depends on how Iceberg commits, rolls back and prunes.

Tables, not buckets: all three tables live in the same S3 bucket under one warehouse path. In Iceberg, the unit that matters is the table, meaning its own metadata tree and its own commit pointer. Whether those files sit in one bucket or three changes access policies, lifecycle rules and cost reports, not query speed.

Writes: one pointer per table

Each table has exactly one "current metadata" pointer and every commit swaps it. Writers to different tables never wait on each other. Writers to the same table take turns: the loser re-reads the latest metadata, checks for conflicts and tries again (see the commit figure in "How writes commit").

Figure 23. Each source has its own ingestion jobs, reconciliation MERGEs and compaction. Split into three tables, those commits run side by side. Merged into one, every commit from every source queues on one pointer.

A retry is usually cheap when the writers touched different partitions, because only metadata is rewritten. But retries add latency, they can run out (we already raised commit.retry.num-retries to 20) and a MERGE that hits a real conflict fails after doing all its work. Merging three tables roughly triples the commit traffic on the one pointer that's left.

Mistakes: the rollback blast radius

Rollback works per table. rollback_to_snapshot moves the table's pointer back, so every commit after that point disappears, whichever source wrote it.

Figure 24. A bad ledger load lands at 10:00. With separate tables, undoing it touches only the ledger table. With one table, the rollback also erases the reconciliation results and the ingest that other sources committed afterwards.

-- Three tables: undo only the bad ledger load.
CALL catalog.system.rollback_to_snapshot('db.transactions_ledger', 8812734455019);

-- One table: there is no "undo only ledger" snapshot to roll back to.
-- You either roll everything back and rerun the other sources, or write a targeted DELETE.

Reads: a draw, if you partition right

For reads, the layout matters more than the table count. One table with kind as the first partition field prunes to exactly the same files as three tables. The risks come from getting that layout wrong, or forgetting the filter.

Figure 25. Data read by a query for bank rows only. Separate tables make the filter impossible to forget, because it's the table name.

Cross-source queries cost the same either way. Matching bank rows to ledger rows reads bank files and ledger files whether they come from two tables or from two partitions of one table.

The trade-offs side by side

	Three tables	One table with `kind`
Commit contention	Isolated per source	All sources share one pointer
Rollback and time travel	Per source	All sources together
Read performance	Good	Same, only if `kind` is the first partition field
Forgetting the source filter	Impossible	Silent: 3× the data and mixed results
Maintenance	Tuned per table; a big source never delays a small one	One larger job; per-source tuning needs `where` filters
Schema changes	Can differ per source, but must be kept in sync by hand	One schema to change; source-only columns are null elsewhere
Access control	Grant per table	Needs row-level filters
Querying all sources	Needs a `UNION ALL` view	One name
Number of objects to run	3 tables, 3 maintenance jobs, 3 sets of DDL	1 of each
Storage cost	The same. Bytes don't care which table they're in.

Separate tables win when

each source has its own ingestion schedule and writers
heavy MERGEs run per source
volumes differ a lot between sources
you need to undo one source without touching the rest

One table wins when

there are many small sources (thirty tables is real overhead)
one job writes every source in a single commit
most queries span all sources
schemas are identical and stable

Our sources match the left column, so we keep three tables. A view gives people who want "all transactions" a single name to query, without bringing back shared commits:

CREATE VIEW db.transactions_all AS
SELECT 'bank'     AS kind, * FROM db.transactions_bank
UNION ALL
SELECT 'ledger'   AS kind, * FROM db.transactions_ledger
UNION ALL
SELECT 'external' AS kind, * FROM db.transactions_external;

-- kind is a constant in each branch, so the engine drops the branches that can't match:
SELECT count(*) FROM db.transactions_all WHERE kind = 'ledger';   -- reads only the ledger table

Rule of thumb: split tables along the lines where writes are independent. Join them back with views for reading.

Coming from a relational database

If you've spent years tuning Postgres, MySQL or SQL Server, most of your instincts still apply. SQL is SQL. But a few habits that served you well there work against you on a lake. The shift underneath all of them: a database owns its storage and keeps it tidy for you; a lake is files on S3 and you design and maintain the layout yourself.

1. "Add an index" becomes "design the layout"

Iceberg has no secondary indexes. Nothing points from a value to a row. Speed comes from not reading data at all: partitioning, sort order and min/max statistics let the engine skip files and row groups. The layout is the index and you pick it once, for your most common filters.

Figure 26. A database seeks to the row. A lake skips everything it can prove irrelevant, then scans what's left. Point lookups on a column nobody sorted or partitioned by are a full scan.

So the question "why is this lookup slow?" has a different answer. In a database you'd check for a missing index. On a lake you check whether the filter column is a partition column, a sort column, or neither.

2. "`SELECT *` is harmless" becomes "every column costs money"

Figure 27. Parquet is a column store. A query reads only the columns it names, so query cost grows with the number of columns, not just rows. Athena bills by bytes scanned, so a lazy SELECT * on a wide table shows up on the invoice.

3. "Update a row" becomes "write a new version of the table"

Database pages change in place. Lake files never change. Every UPDATE, DELETE or MERGE writes new files and commits a new snapshot, however few rows it touches.

Figure 28. The fixed cost of a statement is files and a commit, not a page. Row-by-row loops, which a database shrugs off, are the most expensive way to change a lake. Batch the change into one set-based MERGE.

More habits to unlearn

Relational habit	Lake mindset	Why
Normalize; joins are cheap	Wide tables are normal; join only on pruned inputs	A join of two big tables shuffles data read from S3 across the cluster. Without a selective filter on each side, both sides are read in full.
Stash extras in a JSON column and index it	Promote hot fields to typed columns	There's no GIN index. A value inside a map or JSON blob can't be skipped and the whole blob is read for every row.
Wrap columns in functions; an expression index covers it	Compare bare columns to literals	No expression indexes either. A function around a partition column can hide it from pruning and that's how we once scanned 130 GB (Figure 14).
`PRIMARY KEY` and `UNIQUE` keep data clean	Your pipeline keeps data clean	Iceberg doesn't enforce keys, uniqueness or foreign keys. Inserting the same row twice gives you two rows. Deduplicate in a `MERGE` or in the load job.
Autovacuum tidies up	Schedule maintenance yourself	Compaction, snapshot expiry and orphan removal are jobs you own. Skip them and queries slow down while storage grows.
Deleted rows free space	Space frees when snapshots expire	Old snapshots still reference old files for time travel. Until they expire, a `DELETE` makes storage grow, not shrink.
Thousands of small concurrent writes	Few, large writes per table	Each commit swaps one pointer per table, so concurrent writers take turns. Small writes also leave small files behind.
A transaction can span tables	Atomic per table only	Each table commits on its own. Make every step idempotent so a failed run can simply be rerun.
Storage is the expensive part	Scans are the expensive part	S3 storage is cheap per GB. What you pay for is bytes scanned and compute time, so layout and filters matter more than disk size.
The planner will sort it out	Check "data scanned" on every new query	The optimizer can only skip what the layout and filters allow. The bytes-scanned number tells you at once whether pruning worked.

Things you can stop worrying about

Schema changes are instant. Adding, renaming or dropping a column is a metadata change. No table lock, no rewrite, no migration window.
Readers never block writers. A query reads one fixed snapshot while new commits land next to it.
Time travel is built in. FOR TIMESTAMP AS OF or a snapshot ID answers "what did this look like yesterday?" without a backup restore.
You can change the partitioning later. New data uses the new spec and old data stays readable as it is.
Storage grows with you. There's no disk to resize and no tablespace to fill.

The mindset in one line: design the layout around your filters, read as few bytes as possible, write in batches and treat maintenance as part of the system, not an afterthought.

Checklist

Partitions match the filters in most queries and hold at least a few hundred MB each
A sort order is set on the columns you look up by
Fields you filter or join on are real typed columns, not map keys or JSON paths
Merge-on-read tables are compacted at least daily
expire_snapshots and remove_orphan_files run on a schedule, in that order
Writes are batched; independent write paths use separate tables
Compression is zstd and target file size is 128–512 MB
Table DDL and properties live in version control

Iceberg makes a data lake behave like a database, but it doesn't clean up after you. Design the layout around how you query, keep the metadata tidy and the lake stays fast and cheap as it grows.

When Your Codebase Blocks Agile

Josephat Macharia — Fri, 22 Aug 2025 13:23:03 +0000

One of the biggest mistakes companies make when they adopt Agile is assuming that changing the process is enough. Daily stand-ups, sprints and Kanban boards are introduced — but the code remains untouched, structured for a world that predates Agile.

The result? An architecture mismatch that quietly undermines agility. Teams may be “Agile” in ceremony, but in practice, they are slowed down so much that they can hardly respond to change in any meaningful sense.

The Hidden Bottleneck

Agility is about responding quickly to new requirements, shifting priorities and customer needs. But if your codebase is still organized in a way that resists change, every new feature feels like a battle. Instead of building, teams spend disproportionate time:

Untangling tightly coupled modules

Reworking brittle dependencies

Fighting against legacy design decisions

This isn’t a process problem. It’s an architectural problem.

Why Architecture Shapes Agility

A system’s architecture sets the boundaries for speed, adaptability and collaboration. If the architecture reflects an older way of working — where long release cycles and rigid silos were the norm — Agile practices will only expose those weaknesses faster.

Agile thrives when the codebase itself is:

Modular – enabling teams to work independently without constant conflicts.

Flexible – designed for extensions and iterations, not rigid perfection.

Evolvable – allowing refactoring and improvement as part of everyday work.

Without these qualities, your Agile transformation risks becoming Agile in name only.

The Road Ahead

If you are considering a move to Agile, be prepared: restructuring the fundamental architecture of your system is almost certainly in your future. The sooner you address it, the faster your teams can deliver value without being slowed down by the very code they’re meant to improve.

Agility isn’t just about process. It’s about aligning process and architecture so that your teams — and your software — can truly move at the speed of change.

Rethinking Estimation In Agile Development

Josephat Macharia — Thu, 15 May 2025 18:35:50 +0000

Estimation

Perhaps we should rethink and reassess our ritual of story estimation.

Direct from the Scrum creator himself:

Most story estimation is a waste of everybody's time and a recipe for organisational disappointment.

One of the ongoing debates in agile circles is that of story sizing. For years, it has seemed to be a settled topic, but there are always new people entering the field that don't know that it's settled, go out looking for how long something is going to take and find a bunch of information that says you should use story points (or t-shirt sizing, or something similar) to figure out how big something is and get an idea of how long it will take.

So, let's see if we can bring people up to speed. When XP was developed, its creators thought it was important to be able to have an indicative value of the relative complexity of work.
They started with ideal hours, then padded them and then settled on points to abstract away from time, because they found that sizing by hours didn't work very well. Points were never meant to be a predictor of duration.

After a number of years, just about everybody involved in formalising story points realised it didn't work particularly well, for lots of reasons and suggested we move to something else.

Here's Ron Jeffries on how his views of story points have changed:
"I like to say that I may have invented story points and if I did, I’m sorry now."
From Story Points Revisited

For a lot of people, that isn't enough. Knowing that the creators thought it was a mistake isn't enough to enable people to change.
To change, people also need to know what to change to. For that, a detailed breakdown of what to do instead is really valuable. So, here it is:

Why estimating doesn't work:
✔ Parkinson's Law (the idea that work expands to fill the time allotted for its completion. This may mean you take longer than necessary to complete a task or you procrastinate and complete the task right before the due date.)
✔ Optimism
✔ Inherent uncertainty of creative work (This uncertainty stems from the open-ended nature of creative challenges, where there is rarely a single "right" answer or a clear, pre-defined path to a successful outcome. )
✔ Humans are awful at estimating anything bigger than about 2 days of work

What to do instead:
💡 Make stories smaller. No, even smaller. (Hint: See "2 days" above)
💡 Actively seek to understand the uncertainty
💡 Use exploratory stories
💡 Write genuine user stories
💡 Remember that user stories are a placeholder for a conversation
💡 Go talk to the user
💡 Utilizes short iterations of 1-2 weeks while embracing continuous releases.
💡 Incorporates sprint planning and review while allowing flexibility and no commitment.
💡 No notion of story points or “points per sprint.”

The Fallacy of Estimates

Estimates are inherently based on guesswork.
Estimation leads to dysfunction and false commitments.
It's not that things are late 80% of the time; it's that we don't know how to do estimates 80% of the time, so our estimates are always wrong.
Making decisions based on erroneous estimations leads to significant process breakdown.
Story points have no meaning, any more than time-based estimates would have.

Velocity

What if you need to understand velocity? Simple: count the stories. That's your velocity. Once you make stories really small, their variability decreases and you get through more. The Law of Large Numbers will help eliminate some of the variability you used to see in your work, so counting stories becomes a perfectly reasonable way to measure velocity, if you need to (you probably don't).

Velocity is a ratio that compares the hypothetical time needed to complete a project without interruptions to the actual time it takes in practice.
Essentially, it's a decimal figure ranging from 0 to 1, similar to a percentage.
Our wrong estimates are tied into this.
We wrongly attribute decreased velocity to team inefficiency rather than acknowledging systemic barriers (e.g., bureaucratic processes and unnecessary tasks).
Velocity is another destructive force, often misinterpreted as a measure of efficiency or productivity.

Sprint "Spillover"

In Agile methodology, there's no concept of Sprint "spillover."
Agile embraces change, even during coding, allowing for feedback-driven adjustments.
Insisting on estimation often means misunderstanding Agile, resorting to what is termed as "fake Scrum."
True Agile doesn't require committing to a fixed set of stories within a Sprint; instead, it focuses on committing to overall goals, emphasizing adaptability over rigid planning, similar to traditional waterfall approaches.

Conclusion

Some people will say, "Aha! You said estimating was a waste, but you must estimate in order to figure out if a story is less than 2 days." You're right. But that's the extent of estimation's value. If you understand what's needed well enough, you'll be able to get the story to be smaller than 2 days. If you don't, you won't.

It's time to move away from the culture of estimates and focus on delivering valuable software.
If we are obsessed with estimates, let's employ a difficulty scale that is difficult to translate into time.
- For example, categorize tickets as trivial, easy, normal, hard, don't know, etc.
- This could provide valuable information for prioritization.

The Rewrite Trap: A Programmer's Survival Guide

Josephat Macharia — Sun, 22 Dec 2024 16:47:25 +0000

In the world of software development, there's a constant temptation that lurks in the shadows of every legacy codebase: the urge to throw everything away and start fresh.

It's a siren song that has led many projects to their doom, yet it continues to seduce developers across the industry.

Let's dive deep into the art of code refactoring and explore why the "rewrite everything" approach is often a costly mistake.

The Temptation of the Blank Slate

There’s a unique allure in starting fresh.
Like an architect yearning to demolish and rebuild, programmers often see existing codebases as messy, convoluted, or just plain ugly.

It’s natural to feel this way.
Reading someone else’s code (or even your own, months later) is harder than writing new code from scratch. But this impulse to bulldoze can be catastrophic.

Here’s the truth: existing code is more than just text on a screen. It’s the culmination of countless hours of debugging, testing and adapting to real-world problems.

Each line however inelegant—represents accumulated knowledge.

Tossing it aside without due diligence is like throwing away a treasure map because you dislike the handwriting.

Why Complete Rewrites Are Dangerous

Rewriting an entire codebase might seem like a clean solution, but history shows it’s often a grave mistake.

Consider these cautionary tales:

Netscape's decision to rewrite their browser from scratch led to a three-year gap between versions 4.0 and 6.0, giving Internet Explorer the opportunity to dominate the browser market.
Borland's attempt to rewrite Quattro Pro ended in failure.
Even Microsoft, with all its resources, had to abandon a complete rewrite of Word.

Rewriting not only wastes time and money but also introduces unknown bugs while losing the hard-earned stability of the original code.

Refactor, Don’t Rewrite

So, what’s the alternative? Incremental refactoring. This approach allows you to:

Preserve Battle-Tested Code: Even ugly code has survived production, been reviewed and patched over time. Respect the lessons it holds.
Make Incremental Changes: Small, focused updates make it easier to track the impact of each change. It’s far less intimidating to fix two failing tests than to face a hundred.
Leverage Existing Tests: Never discard old tests without understanding their purpose. These tests may uncover subtle issues you might overlook.

The Right Way to Approach Refactoring

Before refactoring, ask yourself: Why am I doing this?

Performance Issues: Profile the code first. Identify bottlenecks and optimize strategically.
Difficulty Adding Features: If tight coupling or spaghetti code is the issue, focus on encapsulation and better interfaces.
Code Maintainability: Start small, rename variables, clean up formatting and clarify comments. Gradual improvements add up.

Practical Tips for Refactoring Success

Understand the Existing System: Take stock of the codebase and tests. Identify strengths to preserve and weaknesses to address.
Refactor in Small Steps: Focus on one area at a time. Verify changes by running tests after each iteration.
Avoid Personal Bias: Restructure for functionality, not style. If it isn’t broken, don’t fix it.
Beware of New Technology: Don’t refactor just to use the latest framework or language. Unless there’s a clear, measurable benefit, it’s not worth the risk.
Respect the Past: Every “ugly” workaround or patch likely exists for a reason. Investigate before replacing it.

Conclusion: Progress Without Pain

Refactoring isn’t just about writing better code, it’s about making smarter decisions.

By respecting the lessons embedded in your existing system and taking a methodical incremental approach, you’ll avoid the pitfalls of hubris and set yourself up for long-term success.

Remember, code doesn’t need to be perfect, it just needs to work well for the people who use and maintain it.

So next time you’re tempted to rewrite everything, stop and think: What’s the real problem I’m solving? The answer might save you countless hours—and a lot of headaches.

Introduction To RabbitMQ

Josephat Macharia — Sun, 31 Oct 2021 14:25:26 +0000

What is RabbitMQ

RabbitMQ is a message broker that accepts and forwards messages.
It accepts messages from producers, and delivers them to consumers. It acts like a middleman which can be used to reduce loads and delivery times taken by web application servers.

Message flow in RabbitMQ

The producer publishes a message to an exchange. When creating an exchange, the type must be specified.
The exchange receives the message and is now responsible for routing the message. The exchange takes different message attributes into account, such as the routing key, depending on the exchange type.
Bindings must be created from the exchange to queues. In this case, there are two bindings to two different queues from the exchange. The exchange routes the message into the queues depending on message attributes.

4.The messages stay in the queue until they are handled by a consumer

5.The consumer handles the message

Types of exchanges in RabbitMQ:

Fanout
This exchange basically sends a copy of messages to all the queues it knows. (in the dashboard every queue should be bind ed to one exchange)
Direct
A direct exchange delivers messages to queues based on a message routing key. The routing key is a message attribute added to the message header by the producer. Think of the routing key as an “address” that the exchange is using to decide how to route the message.
Topic
Topic exchanges route messages to one or many queues based on matching between a message routing key and the pattern that was used to bind a queue to an exchange.

4.Headers
A header exchange is an exchange that route messages to queues based on message header values instead of routing key. If a match is found, it routes the message to the queue whose binding value is matched and if a match is not found, it ignored the message.

RabbitMQ Terminologies and Concepts

Producer: A producer is a user application that sends messages to the consumers.

Consumer: A consumer is a user application that receives messages.

Message: It’s a simple serialized object from the producer!

Exchange: An exchange is a very simple thing. On one side it receives messages from producers and the other side pushes them to queues. This is a place that messages come first and its main purpose is to decide which consumer application, the message should go!

Queue:
A queue is a buffer that stores messages basically, the exchange routes the message to queues, then the consumer clients receive messages from the queue. We have queues as many as the consumer’s applications
The producers send the message to the exchange first, the exchange decides which queues, the message should go.

When to Use RabbitMQ

For long-running tasks, when you need to run reliable background jobs. e.g such as file scaning, image scaling or PDF
communication and integration within, and between applications, i.e as middleman between microservices; where a system simply needs to notify another part of the system to start to work on a task, like ordering handling in a webshop (order placed, update order status, send order, payment, etc.)

17 days of code complete Day 2 - Chapter 3 and 4

Josephat Macharia — Fri, 29 Oct 2021 06:18:08 +0000

On day 2 we will look into chapter 3 and 4.
For more in depth notes of the book, please purchase the book!

Chapter 3 Measure Twice, Cut Once Upstream Prerequisites

This chapter describes the work that must be done to prepare for software construc-
tion. As with building construction, much of the success or failure of the project has
already been determined before construction begins. If the foundation hasn’t been
laid well or the planning is inadequate, the best you can do during construction is to
keep damage to a minimum.
This chapter discusses software equivalent of blueprints
and construction permits. You might not have much control over preparations, so the focus of this chapter is on assessing what you have to work with when
construction begins.

Key take away

It is good to plan to specify requirements up front.
Essentially, you’ll need to pick the right balance for your project. If it’s something more stable, where the requirements are well understood and unlikely to change, a more sequential approach is appropriate. But if the requirements may change, or the structure isn’t as well understood, then a more iterative approach will be more beneficial.
The overarching goal of preparation is risk reduction
Most common project risks in software devel-
opment are poor requirements and poor project planning, thus preparation tends to
focus on improving requirements and project plans.
From a technical point of
view, planning means understanding what you want to build so that you don’t waste
money building the wrong thing.
Explicit requirements keep you from guessing
what the user wants.
Explicit requirements also help to avoid arguments. You decide on the scope of the
system before you begin programming. If you have a disagreement with another pro-
grammer about what the program is supposed to do, you can resolve it by looking at
the written requirements.
Software being what it is, iterative approaches are useful much more often than sequential approaches are.

Handling Requirements Changes During Construction

Use the requirements checklist at the end of the section to assess the quality of your requirements If your requirements aren’t good enough, stop work, back up, and make them right before you proceed.
Make sure everyone knows the cost of requirements changes, work up a revised schedule and cost estimate so that you can decide whether you want to do it now or later.
Set up a change-control procedure If your client’s excitement persists, consider establishing a formal change-control board to review such proposed changes. It’s all right for customers to change their minds and to realize that they need more capabilities. The problem is their suggesting changes so frequently that you can’t keep up

Chapter 4 Key Construction Decisions

This chapter focuses on preparations that individual programmers and technical leads are responsible for, directly or indirectly. It discusses the software equivalent of how to select specific tools for your tool belt and how to load your
truck before you head out to the job site.

Programmers may be similarly influenced by their languages. The words available in a
programming language for expressing your programming thoughts certainly deter-
mine how you express your thoughts and might even determine what thoughts you
can express.

Your programming tools don’t have to
determine how you think about programming.
Programmers who program “in” a language limit their thoughts to constructs that the language
directly supports. If the language tools are primitive, the programmer’s thoughts will also be primitive.
Programmers who program “into” a language first decide what thoughts they want to express, and then they determine how to express those thoughts using the tools provided by their specific language.
Your position on the technology wave determines what approaches will be effective or even possible. Identify where you are on the technology wave, and adjust your plans and expectations accordingly.
Establish programming conventions before you begin programming. It’s nearly
impossible to change code to match them later.
Every programming language has strengths and weaknesses. Be aware of the specific strengths and weaknesses of the language you’re using.

My thoughts

Its hard for programmers to adopt to the test first approach.
Ensuring the programmer has stepped through their code in the debugger before they check it in is not easy.

How To Build APIs That Are Resilent From 3rd Party Failures

Josephat Macharia — Thu, 28 Oct 2021 20:27:52 +0000

There are many possible reasons for failures, from an outage of a third-party service (API, DB, etc.) to a hardware failure, to the “classic” software bug (after all, software developers are humans too, aren’t we? 🤔).

Fault tolerance is a requirement, not a feature.

This post is an a attempt to address handling system failures during 3rd Party API communications.

What are some of the failure points?

The network connectivity between the application and the third party system might be disrupted causing communication timeouts or lost information.
The third party system has an internal error or machine failure causing the application not to receive a response.
Third party system has not been paid for, e.g late in account payment renewal.
We have an internal error or machine failure. Depending on the timing of this failure, two things could happen:
```
 1. Inability to send the request.

 2. Inability to receive the response from the third party.
```

How do we respond to a user request when failure occurs?

The diagram below attempts to show a process that would be followed to achieve resilience in API communication failure

In each of the options described above a timeout, internal error or connection error will result in a request not retrieving the optimal response for customers.

What are the possible actions to take when retry will not succeed and is unlikely to help?

Notifications: when such 3rd party failure occurs, notifications should be sent for quick action such as calling the customer back to manage expectation and give way out that lets them know that you have their back.
Showing descriptive messages that will aid to let the customer know what to do. e.g refresh, retry

To improve the resilience of the application we should consider the following patterns:

1.Retry
2.Caching
3.Persistence
4.Circuit Breaker

Retry

For a reliable and robust retrying solution it should be:

Smart - To answer this, we use exponential backoff retrying.
Customizable - Standard way to handle errors, we should control which errors should be retried and which shouldn’t by throwing exceptions for the errors that should be retried and by catching the others.

Caching

We record each request in the database before sending the information to an external party.
We should have a status attribute that tracks which part of the process the request is in.
We update the request with a new status based on the response, normally either created, declined or successful.

Persistence

To get more inspiration Uber manages the reprocessing of their data using kafka topics and dead-letter queues. See https://eng.uber.com/reliable-reprocessing/.

Uber’s idea would fit many of our needs, so we can design a solution based on it.

First, our error handling solution should identify that there is an error. Then, it needs to grab the relevant context of the error (in this example, the handled HTTP request).
After that, it should send that context to some persistence layer, for later re-execution of the service flow.
When a failure occurs in one of the HTTP requests, the request context (request body, query params, etc.) will be sent as a Kafka message to the first retry Kafka topic.

As for the re-execution part, a polling of the first retry topic is added and if that processing fails too, it will be sent to the second topic and so forth, until it’s sent to the dlq (“dead-letter queue”) topic, for manual analysis. (The number of retry topics is arbitrary in this example, it can be any number).
Let’s see how the solution I described applies to service A below:

Summary

If we implement the solutions I’ve described, we can be much more confident now when our system has failures, the system requires fewer manual interventions when it fails, and most importantly, we sleep well at night!

I hope this post helped you by providing you with some new ideas for handling failures in your applications.

Feel free to comment below with any questions/thoughts :-)

When To Use NoSQL Document Databases: MongoDB, CouchDB

Josephat Macharia — Sat, 08 May 2021 10:33:59 +0000

Why NoSQL Databases Were Invented

When traditional databases were introduced they were able to handle the size by running on bigger machines
NoSQL means that the database does not employ tables, rows and columns for data organization
Relational databases could horizontally scale which means being able to copy their data to other machines so reads could be distributed.
There was no easy way to distribute writes
To fix this problem, leading technological companies introduced new databases referred to as NoSQL.
Therefore NoSQL databases were designed to scale large datasets horizontally which means scaling large datasets was the primary motivator to create NoSQL databases.

Types Of NoSQL Databases

Key/value
Graph databases
Column-oriented
Document-oriented

Industry examples of Document Databases

MongoDB
CouchDB
Cosmos DB
CrateDB
Couchbase Server

What Is A Document

A way to organize and store data as a set of field value pairs.

Features Of Document databases

Relationships are obvious using embedded arrays and documents.
The biggest benefit to developers of using a document-oriented database is that developers use structure like maps, hashes, dictionaries, and JSON objects.
The document model use a similar structure resulting in the representation that is easier and more natural, as it is the same in the code and the application.
This means there is no complex mapping between your applications data object and the database.
Today, we know that modern data sets can grow to be huge, that data is polymorphic by nature, and that our modern applications have to be resilient and always up.
NoSQL databases, and in particular a document-oriented database like MongoDB, are a more natural way to work with modern data.

Why Use Document Databases

Applications that take records that are fairly complex and need to access those records in a fairly complex way
Allows one to have a lot of records with a lot of engineering
Allows you to have different categories of data and access them the same way
For rapid application development
When you need a flexible schema

When To Use Document Databases

When the schema is easily changeable.
When you want to scale (especially horizontal): they are built for scale since their horizontal partitioning is inbuilt since they expect a lot of scale.
When you require simplicity e.g fewer tables, fewer relationships
When you want to aggregate data since they are built for aggregation

When Not To Use Document Databases

When you require to make a lot of updates since consistency is not guaranteed, the data may not be consistent (two nodes might have different data for the same ID)
They are not read optimized since they read each block and filter out the rows
They do not have implicit information of relations in data, therefore joins are hard if you have more than one table.

How Monitoring Works: Prometheus, Grafana and StatsD

Josephat Macharia — Tue, 27 Apr 2021 07:15:04 +0000

What is Monitoring?

The Periodic tracking (for example, daily, weekly, monthly, quarterly, annually) of any activity’s progress by systematically gathering and analyzing data and information

Why monitor?

Helps to answer the following:(Imagine or picture this):

What is happenning right now?
When did things change or what has changed (has it gone up or down did it change when we did this deploy)
When did it happen
How often does it happen e.g how many times a second, a day when do people use this feature, how many queries am i sending to the database
How long did it happen.
Enhance reliability: software bugs can be prevented via:
- testing
- measuring
- monitoring
- analyzing.

What is a metric?

The unit that is being monitored for a specific target.

Things to monitor?

request latency
request count
number of exceptions
cpu utilization

How to Monitor?

we can use traditional methods but why reinvent the wheel use monitoring tools.

What do monitoring tools offer?

collection
- Describes how the measurements are taken.
storage
- how to get metrics you measured from where you measured into something that will store them long term (usually called time series databases).
- This storage also needs to be capable of fetching that stuff quickly for analysis/graphing/alerting etc.
graphing
- visualizing and analyzing metrics collected.
aggregation windows
- how long you can keep metrics before you run out of space or the storage becomes slow.
alerting
- channels used to notify users when things happen.

What are some of the common measurement/metric types?

Counter
- Use this for situations where you want to know “how many times has x happened”.
- for example total number of HTTP requests, or the total number of bytes sent in HTTP
Gauge
- A representation of a metric that can go both up and down.
- “what is the current value of x now”.
- what is the current value of logged in users now.
Histogram

This represents observed metrics sharded into distinct buckets.
Think of this as a mechanism to track “how long something took” or “how big something was”.

Monitoring patterns for reporting metrics

Pull model
- the monitoring system "scrapes" the application at a predefined HTTP endpoint.
Push model, the application sends the data to the monitoring system.

An example of a monitoring system working in the pull model is Prometheus. StatsD is an example of a monitoring system where the application pushes the metrics to the system.

Monitoring tools

We will focus on the following:

StatsD
Prometheus

Whats is StatsD

System for instrumenting metrics
statsd is semi real time and little historical
a funnel that collects data

Why it was creatd?

Devops guys did not understand the internals of an application apart from its CPU utilization and network IO.

How it works?

StatsD will accept measurements from all over your network with UDP.
Aggregates them into 10-seconds chunks.
Sends them off to somewhere that will store the data e.g Graphite

What are the components of StatsD?

API Client --> this integrates with your actual application.
UDP Protocol --> API client just wraps over the UDP protocol.
Daemon --> this deamon runs on your actual application service.
Metrics collection: you fit your application with metrics which will send UDP packets to a local daemon the local daemon will aggregate these packets together and send them in batch to a backend

What metric types are available?

counters - how many x times something happened
timers - it took this long e,g on average
gauges - % completion
sets - keep track of unique values

How does it integrate with my app

Push metrics from your code
clients are available for many languages e.g ntest statsd

Which backends it can integrate with

graphite
mysql, influxdb, mongodb

Prometheus

A monitoring tool that actively scrapes data, stores it, and supports queries, graphs, and alerts, as well as provides endpoints to other API consumers like Grafana or even Graphite itself.

Why it exists

Created to monitor highly dynamic container environments like kubernates and docker swarm etc
However it can used in a traditional non container infrastructure.
It has become the mainstream monotoring tool of choice in container and microservice world.

Where can i use it?

When running multiple servers that run containerized applications and there are x(processes) running on that infrastructure and things are interconnected.
In complex infrastructure with lots of servers distributed

What can go wrong in complex infrastructure?

One service can crush and cause failure to others.This can be difficult to debug manually.
Application downtime
Errors
Overloaded and running out of resources
Response latency

What is the remedy for problems in complex infrastructure

Have a tool that constantly monitors all the services.
Alerts the maintainers as soon as one service crashes e.g you know what happened.
Identify problems before they occur and alert the system admin responsible for that infrastructure to prevent that issue e.g
- when the application is about to run out of storage space
- when an application becomes too slow, have tool that detects network spikes

What it offers

Automated monitoring and alerting for devops workflow

Architecture/How it works

Prometheus Server

Does the actual monitoring work and made up of three components
1. Storage: time series database --> stores all the metrics data e.g current cpu usage, no_of_requests
2. Data retrieval worker responsible for getting or pulling those metrics and pushing them to the database
3. Server API That accepts queries for that stored data and used to display the data/visualize inside prometheus or other graph tool

What is the work of prometheus alert Manager

Responsible for firing alerts via different channels e.g email, slack.
Determines who receives? how it triggers.
Reads the rule, if the condition is met then an alert is fired.

Where is data stored

Stores the data on disk also integrates with remote storage.

What are the characteristics

Designed to be reliable even when other services have an outage.
Standalone and self containing --> does not depend on network storage

What it Monitors

A particulat thing e.g a linux server, windows, database server
Things it monitors are called targets
- For linux server it can be, CPU status, memory disk, space usage
- For application server it can be exceptions count, number of request and request duration

There are 3 primitive metrics types

Counter --> How many times x happened
Gauge --> what is the current value of x now or what is the current capacity of disk space now
Histogram/Timer ---> How long or how big

How it collects metrics

It pulls metrics from a http endpoint whose host adress exposes a /metrics endpoint. for that to work:
- The target must expose the /metrics endpoint
- Data available at /metrics must be in a format that prometheus understands

Target Endpoints and exporters

Some servers are already exposing the /metrics endpoint so no extra work is needed to get metrics from them
Many services dont have native prometheus endpoint and need an extra component known as an exporter

Exporter is a script or service that fetches metrics from ur target and converts them into a format prometheus understands
and exposes these metrics to its own /metrics endpoint

How to monitor a linux/windows server

Download a node exporter
untar and execute
converts metrics of the server
exposes /metrics endpoint
configure promethues to scrape this endpoint

How to monitor an application

At application level you can monitor the following:

How long request are taking
- How many requests Use client libraries to expose /metrics endpoint

Why care

In push mechanism applications/servers push to a centralized platform
high load of network traffic: when working with many microservices and you have each service pushing their metrics to the monitoring system it creates a high load of traffic in your infrastructre and your monitoring can be your bottleneck
must install a daemon on each target to push data to the monitoring server.
Better detect/insight if service is up and running

note: for shortlived gateway it offers a pushgateway

How it know what to scape and when?

All that is confugured in promethues.yml:
- define what targets to scrape and at what interval.
- It uses a service discovery to discover those endpoints.

What is it does not do well

Dificult to scale when you have 100 of servers you might want to have multiple prometheus servers that somewhat aggregate all these metrics data and configuring that can be dificlut

Workarounds

Increase prometheus server capacity.
Limit number of metrics.

Example

Graphana

Allows you to query, visualize alert and understand your metrics no matter where they are stored.

Enables you to create, explore and share dashboards with your team.

Example:
Explore metrics

Ad-hoc queries are queries that are made interactively, with the purpose of exploring data. An ad-hoc query is commonly followed by another, more specific query

1.tns_request_duration_seconds_count

is a counter, a type of metric whose value only ever increases. Rather than visualizing the actual value, you can use counters to calculate the rate of change, i.e. how fast the value increases.
1. Add the rate function to your query to visualize the rate of requests per second. Enter the following in the Query editor and then press Shift + Enter.

rate(tns_request_duration_seconds_count[5m])

Immediately below the graph there’s an area where each time series is listed with a colored icon next to it. This area is called the legend.

PromQL lets you group the time series by their labels, using the sum function.

Add the sum function to your query to group time series by route:

sum(rate(tns_request_duration_seconds_count[5m])) by(route)

Add a logging data source

Grafana supports log data sources, like Loki. Just like for metrics, you first need to add your data source to Grafana.

Loki

Grafana Loki is a set of components that can be composed into a fully featured logging stack.

Grafana supports log data sources, like Loki. Just like for metrics, you first need to add your data source to Grafana.

Explore logs

{filename="/var/log/tns-app.log"}

Grafana displays all logs within the log file of the sample application. The height of each bar encodes the number of logs that were generated at that time.

{filename="/var/log/tns-app.log"} |= "error"

Logs are helpful for understanding what went wrong

you can correlate logs with metrics from Prometheus to better understand the context of the error.

Build a Dashboard

A dashboard gives you an at-a-glance view of your data and lets you track metrics through different visualizations.

Dashboards consist of panels, each representing a part of the story you want your dashboard to tell.

Every panel consists of a query and a visualization. The query defines what data you want to display, whereas the visualization defines how the data is displayed.
Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels).

Annotate events

When things go bad, it often helps if you understand the context in which the failure occurred. Time of last deploy, system changes, or database migration can offer insight into what might have caused an outage. Annotations allow you to represent such events directly on your graphs.

query annotations
{filename="/var/log/tns-app.log"} |= "error"

Alerts

Alerts allow you to identify problems in your system moments after they occur. By quickly identifying unintended changes in your system, you can minimize disruptions to your services.

Alerts consists of two parts:

Notification channel - How the alert is delivered. When the conditions of an alert rule are met, the Grafana notifies the channels configured for that alert.
Alert rules - When the alert is triggered. Alert rules are defined by one or more conditions that are regularly evaluated by Grafana.

Supported Alert Notification Channels
Email
Slack
Kafka
Google Hangouts Chat
Microsoft Teams
etc.

Others to look at

TICK stack
Google cloud Monitoring

Document Databases e.g MongoDB

Josephat Macharia — Tue, 27 Oct 2020 14:25:36 +0000

This course focuses on mapping the knowledge of:

architects designing a database schema
developers accessing the database from an application
from SQL or RDBMS (Relational DataBase Management Systems) to MongoDB

17 days of code complete Day 1 - Chapter 1 and 2

Josephat Macharia — Mon, 14 Sep 2020 21:19:04 +0000

In the coming days i will be giving a summary and my own review and thoughts of Code Complete book by Steve McConnell chapter by chapter. I hope you find this book review helpful.

Chapter 1

What Is Software Construction?

In common usage, “construction” refers to
the process of building. The construction process might include some aspects of
planning, designing, and checking your work, but mostly “construction” refers to
the hands-on part of creating something

Construction activities:

coding and debugging
detailed design
unit testing
integration testing

Tasks involved in construction activities:

Verifying that the groundwork has been laid so that construction can proceed successfully.
Determining how your code will be tested.
Designing and writing classes and routines.
Creating and naming variables and named constants.
Selecting control structures and organizing blocks of statements.
Unit testing, integration testing, and debugging your own code.
Reviewing other team members’ low-level designs and code and having them review yours.
Polishing code by carefully formatting and commenting it.
Integrating software components that were created separately.
Tuning code to make it smaller and faster.

Nonconstruction activities:

Management.
Requirements development.
Software architecture.
User-interface design.
System testing and maintenance

Why Is Software Construction Important?

Putting construction in its context with other activities helps
keep the focus on the right tasks during construction and appropriately
emphasizes important nonconstruction activities.
Construction is a large part of software development.
Construction is the central activity in software development.
individual programmer’s productivity can improve enormously.

My thoughts:

When tuning code to make it smaller and faster one should be pragmatic about choosing algorithms, the fastest is not always the best for the job
Design to test, build the tests before you implement the code.build testability into the software from the very beginning.
Continuous integration and delivery should be part of the construction process to prevent the integration hell, although integrating code frequently does not by itself offer any guarantees about the quality of the new code or functionality.
Although not mandatory including acceptance testing to be part of the construction process minimizes the amount of bugs that could be realized.
Well written code should be self documenting and does not need comments.

Chapter 2: Metaphors for a Richer Understanding of Software Development

A software metaphor is more like a searchlight than a roadmap. It doesn’t tell
you where to find the answer; it tells you how to look for it. A metaphor serves
more as a heuristic than it does as an algorithm.

An algorithm is predictable, deterministic, and not subject to chance

A heuristic is a technique that helps you look for an answer. Its results are
subject to chance because a heuristic tells you only how to look, not what to find.
It doesn’t tell you how to get directly from point A to point B; it might not even
know where point A and point B are. In effect, a heuristic is an algorithm in a
clown suit. It’s less predictable, it’s more fun, and it comes without a 30-day
money-back guarantee

The main difference between the two is the level of indirection from the solution.
An algorithm gives you the instructions directly. A heuristic tells you how to discover the instructions for
yourself, or at least where to look for them.

How to Use Software Metaphors

Use them to give you insight into your programming problems and processes.
Use them to help you think about your Programming activities and to help you imagine better ways of doing things.

The Importance of Metaphors

Metaphors have the virtue of an expected behavior that is understood by all.
Unnecessary communication and misunderstandings are reduced.
Learning and education are quicker.
In effect, metaphors are a way of internalizing and abstracting concepts allowing one’s thinking to be on a higher plane and low-levelmistakes to be avoided.

Common Software Metaphors

Software Penmanship: Writing Code
- “writing code.” The writing metaphor suggests that developing a program is like writing a casual letter—you sit down with pen, ink, and paper and write it from start to finish.
Software Farming: Growing a System
- Creating software as something like planting seeds and growing crops. You design a piece, code a piece, test a piece, and add it to the system a little bit at a time. By taking small steps, you minimize the trouble you can get into at any one time.
- If you buy into the farming metaphor, you might find yourself talking about fertilizing the system plan, thinning the detailed design, increasing code yields through effective land management, and harvesting the code itself.
Software Oyster Farming: System Accretion

Learn how to add to your software systems a small amount at a time. Other words closely related to accretion are “incremental,” “iterative,” “adaptive,” and “evolutionary.

4.Software Construction: Building Software

If you’re building a first-class software product, you
might build your own scientific functions for better speed or accuracy. You
might build your own container classes, user interface
classes and database
classes to give your system a seamless, perfectly consistent
look and feel.
Both building construction and software construction both benefit from
appropriate levels of planning. If you build software in the wrong order, it’s hard
to code, hard to test, and hard to debug. It can take longer to complete, or the
project can fall apart because everyone’s work is too complex and therefore too
confusing when it’s all combined.
The more experienced you have with the
kind of software you’re building, the more details you can take for granted. You
just want to be sure that you plan enough so that lack of planning doesn’t create
major problems later.
In software you might generally use flexible,
lightweight development approaches, but sometimes rigid, heavyweight
approaches are required to achieve safety goals and other goals.
Making structural changes in a program costs more than adding or deleting peripheral features.

5.Toolbox Metaphor

A good craftsman knows the
right tool for the job and knows how to use it correctly. Programmers do too.
The more you learn about programming, the more you fill your mental toolbox
with analytical tools and the knowledge of when to use them and how to use
them correctly.
The toolbox metaphor helps to keep all the methods, techniques, and tips in
perspective—ready for use when appropriate.

To extend the “building a house metaphor” construction workers do not use a single tool — the hammer — to build a house. Instead, they have a complete toolbox full of different implements for the varied tasks involved in construction. Likewise, in software engineering, or data science, the person who only knows one method will not get very far.You should be open to learning from others, respond constructively to feedback, and fill your toolbox with the appropriate techniques.

DEV Community: Josephat Macharia

No cloud to lean on: building a reliable data platform on a bank's own servers

The constraints

Five words, one goal

Reliability: no work is lost silently

Acknowledge after the work, never before

Consumers that can't stall silently

Locks that can't be taken twice

Things that must happen once

Graceful shutdown: finish the work in hand

Fault tolerance

Replication: the database

Replication: the message queues

Redundancy: copies on different machines

Failover: when a copy dies

Graceful degradation: fail small, slow down rather than fall over

High availability: what survives what

Scalability on a fixed number of servers

Scale on the work waiting, not on CPU

Admission control for heavy jobs

Requests, limits and the throttling trap

Monitoring and observability without the internet

Collect from everything, redact in one place

Alerts that mean something

Check from the users' side

Mark every change on every chart

A daily report, which is also a dead man's switch

High availability is not a backup

Two stores must agree

Checklist

Apache Iceberg in a data lake: how it works and how to keep it fast, cheap and simple

The problem Iceberg solves

How Iceberg works: a tree of metadata

How a read finds its data

Partition pruning, with a real query

File-level min/max statistics, with a real manifest

Row-group statistics, with a real file

How writes commit and why they collide

Updates and deletes: two strategies

Why does an update create a delete file?

The three kinds of delete file

Why the new row isn't deleted too

The flow of a DELETE

The flow of an UPDATE

Merge-on-read, step by step: what happens to the delete file?

When UPDATE and DELETE get slow

How MERGE works

Is MERGE the better way to change lots of rows?

Writing a fast MERGE

Case study: the ON clause that scanned 130 GB

Things to watch for

Faster queries

1. Partition by how you filter and keep partitions big

2. Sort inside partitions so min/max statistics mean something

A worked example: our four-column sort order

3. Don't hide hot fields inside maps or JSON

4. Joins: cheap when pruned, expensive when not

5. Compare raw columns to literals

Cheaper, leaner storage

Settings that shrink the bill

What map columns cost in storage

Simpler queries, easier changes

Typed columns beat map lookups in readability too

More habits

One table or three?

Writes: one pointer per table

Mistakes: the rollback blast radius

Reads: a draw, if you partition right

The trade-offs side by side

Coming from a relational database

1. "Add an index" becomes "design the layout"

2. "SELECT * is harmless" becomes "every column costs money"

3. "Update a row" becomes "write a new version of the table"

More habits to unlearn

Things you can stop worrying about

Checklist

When Your Codebase Blocks Agile

Rethinking Estimation In Agile Development

Estimation

The Fallacy of Estimates

The flow of a `DELETE`

The flow of an `UPDATE`

When `UPDATE` and `DELETE` get slow

How `MERGE` works

Is `MERGE` the better way to change lots of rows?

Writing a fast `MERGE`

Case study: the `ON` clause that scanned 130 GB

2. "`SELECT *` is harmless" becomes "every column costs money"