DEV Community: longtk26

Kubernetes Architecture and Primitives: From Containers to Orchestration

longtk26 — Sat, 31 Oct 2026 08:00:35 +0000

Running containers with standalone Docker commands works well on a single machine, but operating a container fleet across multiple hosts introduces critical operational challenges: recovering from container crashes, executing zero-downtime rollouts, and dynamically routing network traffic. Kubernetes solves this by operating as a declarative orchestration system that continuously reconciles the cluster's actual state with your defined desired state. Through its centralized cluster architecture, Kubernetes delegates administrative control to a dedicated control plane while worker nodes manage container execution and health monitoring.

In this guide, we break down Kubernetes architecture from first principles to transition your mental model from isolated Docker containers to automated orchestration. We examine how the control plane exposes its API through the API server, uses the scheduler to place unassigned Pods onto suitable nodes, and packages control loops into the controller manager. We then trace how worker node components—specifically the kubelet ensuring container health and kube-proxy maintaining network rules—execute workloads, before analyzing workload hierarchies from Pods to Deployments, stable service networking, and automated health checks.

Why Kubernetes? From Manual Docker to Automated Orchestration
- The Operational Limits of Standalone Docker
- Declarative Reconciliation vs. Imperative Management
Cluster Architecture: Control Plane and Worker Nodes
- The Control Plane: The Cluster Brain
- Worker Nodes: The Execution Fleet
The Workload Hierarchy: Pods, ReplicaSets, and Deployments
- 1. The Pod: The Atomic Unit of Scheduling
- 2. ReplicaSets: Declarative Self-Healing and Scale
- 3. Deployments: Declarative Rolling Updates and Rollbacks
Service Networking: Stable Virtual IPs and Traffic Distribution
- The Service Primitive and EndpointSlices
- How kube-proxy Routes Traffic
- Service Types and Exposure Scope
Health Probes: Startup, Liveness, and Readiness
- The Three Probe Types
- 1. Startup Probes: Guarding Slow Initialization
- 2. Liveness Probes: Self-Healing Deadlocks
- 3. Readiness Probes: Dynamic Traffic Gating
- Probe Configuration and the Cascading Restart Trap
Auxiliary Primitives and the End-to-End Request Lifecycle
- Supporting Primitives: Configuration, Isolation, and Quotas
- Ingress and the End-to-End Request Path
Summary & What's Next
- Core Primitives Summary
- What's Next: Hands-On Practice

1. Why Kubernetes? From Manual Docker to Automated Orchestration

Docker packages an application and its runtime dependencies into a reproducible, isolated container. Running docker run on a local workstation or a single virtual machine provides a reliable developer experience, but managing standalone Docker containers in production quickly runs into severe operational walls. To use an analogy: Docker is a single musical instrument; Kubernetes is the conductor coordinating the entire symphony.

The Operational Limits of Standalone Docker

When deploying distributed microservices across multiple physical or virtual hosts, manual container operations fail at scale:

Host failures and downtime: If a physical server hosting a Docker daemon crashes, all containers on that host terminate immediately. Standalone Docker cannot detect external node failures or automatically resurrect containers on healthy hardware.
Bespoke operational tooling: Teams managing raw containers inevitably build fragile maintenance layers—custom shell scripts, external cron jobs, and host-level systemd units—to inspect health statuses, trigger restarts, and map ephemeral container ports.
Manual scaling bottlenecks: Handling sudden traffic spikes requires operators to provision new VMs, run manual deployment commands, and reconfigure reverse proxies or load balancers with the new host IP addresses.

Declarative Reconciliation vs. Imperative Management

Kubernetes fundamentally shifts operations from an imperative model ("run this container now") to a declarative reconciliation model ("maintain three healthy replicas of this workload").

Rather than executing continuous shell commands to keep services online, you declare your desired system state in version-controlled manifests. As outlined in The Kubernetes Visual Handbook, Kubernetes operates via an infinite reconciliation loop where controllers constantly observe actual state, compare it against desired state, and act to eliminate any discrepancies.

Furthermore, Google Cloud's GKE cluster architecture documentation highlights that the control plane actively schedules workloads onto nodes while managing workload lifecycles, scaling, networking, and storage. If a worker node crashes or an application process fails its health checks, the control plane immediately detects the deficit and reschedules replacement workloads across the remaining cluster pool without human intervention.

Operational Dimension	Standalone Docker	Kubernetes Orchestration
Management Paradigm	Imperative commands (`docker run`, `docker restart`)	Declarative state reconciliation loops
Failure Recovery	Manual intervention or custom watchdog scripts	Automated self-healing and workload rescheduling
Multi-Host Scheduling	None (bound to a single Docker host)	Native scheduling based on resource requests and node capacity
Service Discovery	Static port forwarding and manual proxy updates	Cluster-internal DNS and dynamic virtual IPs
Application Updates	Disruptive manual container replacement	Automated, zero-downtime rolling updates

2. Cluster Architecture: Control Plane and Worker Nodes

To transition from operating standalone Docker daemons to managing distributed infrastructure, we must first examine how Kubernetes decouples administrative orchestration from container execution. As detailed in the Kubernetes components documentation, a cluster is split into two primary operational tiers: the control plane and one or more worker nodes.

The control plane makes global decisions about the cluster (such as scheduling workloads and responding to node failures), while worker nodes supply the actual compute capacity to run your containerized workloads.

The Control Plane: The Cluster Brain

The control plane maintains the desired state of all workloads across your machines, acting as the decision-making engine:

kube-apiserver: The stateless entry point and central communication hub for the entire cluster. According to Google Cloud's GKE cluster architecture guide, all internal components and external client requests (kubectl) interact strictly through this REST API. It handles authentication, authorization, validation, and admission control. Crucially, it is the only component that directly reads from or writes to etcd.
etcd: A strongly consistent, distributed key-value store that acts as the cluster's single source of truth. As outlined in the Kubernetes architecture documentation, etcd holds all cluster state, resource specifications, and runtime metadata. Because kube-apiserver is stateless, multiple API server replicas can run behind a load balancer while sharing this common datastore.
kube-scheduler: When you submit a Pod manifest without a designated node assignment, the scheduler assigns it to a suitable machine. As described in The Kubernetes Visual Handbook, the scheduler evaluates nodes through filtering (eliminating nodes that do not meet CPU, memory, or policy constraints) and scoring (ranking remaining nodes to pick the optimal candidate).
kube-controller-manager: A single binary combining multiple control loops that continuously compare the current state of the cluster with the desired state stored in etcd, issuing corrections whenever drift occurs.

Worker Nodes: The Execution Fleet

While the control plane handles coordination, worker nodes provide the compute infrastructure where user workloads execute. As highlighted in DevOpsCube's Kubernetes architecture guide, each worker machine hosts three primary elements:

kubelet: The primary node agent running as a system daemon. It registers the node with the API server, watches for Pod assignments targeted at its host, and interacts directly with the local runtime to ensure containers match their declared PodSpec definitions and remain healthy.
Container Runtime: The underlying software (such as containerd or CRI-O) responsible for pulling container images, configuring namespaces and cgroups, and running the container processes.
kube-proxy: A network proxy running on each node that synchronizes host network rules (such as iptables or IPVS). It translates abstract virtual IPs into direct routing pathways across dynamic worker workloads.

Component	Placement	Statefulness	Core Responsibility
`kube-apiserver`	Control Plane	Stateless	Exposes the Kubernetes API and validates cluster read/write operations
`etcd`	Control Plane	Stateful	Serves as the distributed, consistent key-value backing store
`kube-scheduler`	Control Plane	Stateless	Selects target nodes for unassigned Pods using filtering and scoring
`kube-controller-manager`	Control Plane	Stateless	Executes core reconciliation loops to maintain desired cluster state
`kubelet`	Worker Node	Stateless agent	Communicates with the API server and enforces container specifications
`kube-proxy`	Worker Node	Stateless agent	Manages local node network rules to direct traffic to target Pods
Container Runtime	Worker Node	Local runtime	Executes container lifecycles on the node OS

3. The Workload Hierarchy: Pods, ReplicaSets, and Deployments

To run applications reliably across a cluster, Kubernetes does not manage individual containers directly. Instead, it layers primitives into an abstraction hierarchy: Deployments manage ReplicaSets, which manage Pods, which in turn house your application containers.

1. The Pod: The Atomic Unit of Scheduling

A Pod is the smallest deployable atomic unit in Kubernetes. While a Docker workflow treats a single container as the deployment artifact, Kubernetes wraps one or more tightly coupled containers into a Pod.

All containers inside the same Pod share:

A single network namespace and loopback interface (localhost), allowing intra-Pod processes to communicate via local ports.
A shared Pod IP address routable within the cluster network.
Shared storage volumes mounted into multiple containers simultaneously.

Crucially, raw Pods are ephemeral and non-durable. Worker nodes host Pods, and the local kubelet agent ensures containers defined in PodSpecs are running and healthy. However, if an entire host node fails, a standalone Pod is not rescheduled or migrated—it is permanently lost. Production workloads therefore require higher-level managers.

2. ReplicaSets: Declarative Self-Healing and Scale

A ReplicaSet guarantees that a specified number of identical Pod instances remain running at all times. It enforces this contract using an infinite reconciliation loop:

flowchart LR
    Obs["1. Observe<br/>Query API server for<br/>matching Pod labels"]
    Comp["2. Compare<br/>Match active count<br/>against spec.replicas"]
    Act["3. Act<br/>Create missing Pods or<br/>prune excess Pods"]

    Obs --> Comp
    Comp --> Act
    Act --> Obs

    style Obs fill:#0369a1,stroke:#38bdf8,stroke-width:2px,color:#ffffff;
    style Comp fill:#b45309,stroke:#fbbf24,stroke-width:2px,color:#ffffff;
    style Act fill:#15803d,stroke:#4ade80,stroke-width:2px,color:#ffffff;

Observe: Query the cluster state to count running Pods matching a specific label selector.
Compare: Evaluate the actual active Pod count against the desired target (spec.replicas).
Act: Issue requests to the API server to create new Pods when counts drop below target, or terminate excess Pods if counts exceed target.

If an underlying worker node crashes, the control plane notices the lost replicas during its reconciliation cycle and schedules replacement Pods onto healthy worker machines, providing automated self-healing.

3. Deployments: Declarative Rolling Updates and Rollbacks

While ReplicaSets manage instance counts, they cannot smoothly upgrade running workloads. If you update a container image inside a standalone ReplicaSet, existing Pods continue running untouched until manually deleted.

A Deployment solves this by managing multiple ReplicaSets declaratively. When you apply a new container image or configuration, the Deployment controller provisions a new ReplicaSet, scales it up incrementally, and simultaneously scales down the old ReplicaSet. This mechanism provides zero-downtime rolling updates and instant rollbacks.

Here is a complete, minimal Deployment specification defined in JSON:

{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "metadata": {
    "name": "web-api",
    "labels": {
      "app": "web-api"
    }
  },
  "spec": {
    "replicas": 3,
    "selector": {
      "matchLabels": {
        "app": "web-api"
      }
    },
    "template": {
      "metadata": {
        "labels": {
          "app": "web-api"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "server",
            "image": "nginx:1.27-alpine",
            "ports": [
              {
                "containerPort": 80
              }
            ]
          }
        ]
      }
    }
  }
}

You submit this declaration with kubectl apply -f deployment.json. In this hierarchy, you define the desired operational state on the Deployment. The Deployment orchestrates the ReplicaSets, the ReplicaSets ensure the target quantity of Pods exist, and worker node runtimes execute your containers.

4. Service Networking: Stable Virtual IPs and Traffic Distribution

While Deployments solve replication and rolling updates, they introduce a fundamental networking problem: Pod IPs are ephemeral. Whenever a node dies, a deployment rolls out a new image, or a horizontal autoscaler shrinks your workload, the affected Pods are destroyed along with their dynamic IP addresses. If upstream callers or other internal microservices attempt to hardcode direct Pod IP connections, communication breaks the moment a Pod restarts.

The Service Primitive and EndpointSlices

To decouple client requests from ephemeral Pod IPs, Kubernetes introduces the Service resource. A Service acts as a durable abstraction layer, assigning a permanent virtual IP (ClusterIP) and a persistent cluster-internal DNS name to an arbitrary collection of Pods selected by labels.

Rather than routing directly to pods, the Service points to dynamic EndpointSlices that track the real-time IP addresses of ready, healthy Pods. As detailed in the Kubernetes documentation on probes, when a container fails its readiness probe, the control plane immediately strips the Pod IP from the corresponding EndpointSlice. The container stays running, but incoming traffic ceases until the probe passes again.

How kube-proxy Routes Traffic

The virtual IP assigned to a Service is not bound to any physical network interface. Instead, kube-proxy maintains network rules on every worker node to implement the Service abstraction. Running on worker nodes alongside the kubelet as described in The Kubernetes Visual Handbook, kube-proxy configures local packet-filtering mechanisms—such as Linux iptables or IPVS—to intercept requests directed to the Service's virtual IP and rewrite the destination to one of the healthy backend Pod IPs.

Here is a minimal Service manifest in JSON routing port 80 traffic to Pods carrying the label app: web-api:

{
  "apiVersion": "v1",
  "kind": "Service",
  "metadata": {
    "name": "web-api"
  },
  "spec": {
    "type": "ClusterIP",
    "selector": {
      "app": "web-api"
    },
    "ports": [
      {
        "port": 80,
        "targetPort": 8080
      }
    ]
  }
}

Service Types and Exposure Scope

Kubernetes defines multiple Service types depending on whether your workload is cluster-internal or needs to accept ingress from the outside world:

Service Type	Exposure Scope	Routing Mechanism	Primary Use Case
ClusterIP	Internal cluster only	Default static virtual IP reachable strictly inside worker nodes.	Inter-service communication, backend APIs, internal databases.
NodePort	External to cluster	Allocates a dedicated port (usually 30000–32767) across every worker node's external IP.	Development testing or direct node ingress without cloud load balancers.
LoadBalancer	External to cluster	Provisions a cloud provider load balancer (e.g., AWS NLB, GCP Load Balancer) that forwards traffic to NodePorts.	Production-grade public traffic ingress for internet-facing applications.

5. Health Probes: Startup, Liveness, and Readiness

In standalone Docker, the container runtime considers a container healthy as long as PID 1 remains active. However, real-world services can deadlock, exhaust thread pools, or remain unresponsive during lengthy startup routines while keeping their main process alive. To automate remediation and prevent traffic black-holing, the kubelet executes three dedicated health probes: Startup, Liveness, and Readiness.

The Three Probe Types

Each probe serves a distinct operational objective across the container lifecycle:

Probe Type	Evaluation Timing	Kubelet Action on Failure	Operational Objective
Startup	At container launch until initial success	Kills and restarts the container	Shields slow-initializing applications from premature termination
Liveness	Post-startup, throughout container life	Kills and restarts the container	Recovers from deadlocks, infinite loops, and unrecoverable broken states
Readiness	Post-startup, throughout container life	Removes container IP from Service `EndpointSlices`	Isolates instances during heavy load, cache hydration, or temporary faults

1. Startup Probes: Guarding Slow Initialization

When applications require extended time to hydrate in-memory caches, parse schemas, or execute database migrations, standard health checks can trigger premature restart loops. As detailed in the Kubernetes documentation on probes, a startup probe disables both liveness and readiness checks until the container successfully completes initialization. If the startup probe does not succeed within its configured failure threshold, the kubelet terminates the container and enforces the Pod's restart policy.

2. Liveness Probes: Self-Healing Deadlocks

Once an application marks itself as started, the kubelet runs liveness probes to confirm ongoing progress. If an unhandled exception locks a worker thread or a resource leak causes an internal deadlock, the application fails this check. When consecutive failures exceed the configured threshold, the kubelet kills the container and initiates a restart to restore service availability.

3. Readiness Probes: Dynamic Traffic Gating

A running process is not necessarily capable of processing live requests. An application might temporarily saturate its thread pool or face downstream throttling. According to the Kubernetes probe specification, failing a readiness probe prompts the EndpointSlice controller to remove the Pod IP from the Service endpoints without terminating the process. Once the Pod recovers and passes the probe, traffic routing automatically resumes.

Probe Configuration and the Cascading Restart Trap

We configure health checks inside the container specification of a workload manifest using standard HTTP, TCP socket, or exec command probes:

{
  "name": "api-server",
  "image": "nginx:1.25",
  "ports": [{ "containerPort": 80 }],
  "startupProbe": {
    "httpGet": { "path": "/healthz", "port": 80 },
    "failureThreshold": 30,
    "periodSeconds": 10
  },
  "livenessProbe": {
    "httpGet": { "path": "/healthz", "port": 80 },
    "initialDelaySeconds": 5,
    "periodSeconds": 10
  },
  "readinessProbe": {
    "httpGet": { "path": "/ready", "port": 80 },
    "periodSeconds": 5
  }
}

As described in the official probe configuration guide, initialDelaySeconds determines the grace period before the initial check runs, and periodSeconds defines the evaluation interval.

A dangerous architectural anti-pattern is testing external dependencies (such as a database or downstream microservice) inside a liveness probe. If that shared dependency encounters transient latency or downtime, every Pod's liveness check fails simultaneously. The kubelet will restart every replica across the cluster at once, turning a temporary dependency hiccup into a catastrophic, self-inflicted cascading outage.

6. Auxiliary Primitives and the End-to-End Request Lifecycle

Beyond Pods, Deployments, and Services, a production-grade Kubernetes cluster relies on auxiliary primitives to manage security boundaries, runtime configuration, durable storage, and resource contention.

Supporting Primitives: Configuration, Isolation, and Quotas

Modern applications decouple immutable code from environment-specific configuration and storage. Kubernetes handles these concerns with specialized API resources:

Namespaces: Provide virtual isolation within the same physical cluster, allowing multiple teams or deployment stages to coexist without naming collisions.
ConfigMaps and Secrets: Inject plain-text configuration and sensitive credentials (such as tokens or API keys) into containers via environment variables or mounted volumes, decoupling runtime state from container images. According to the Google Cloud GKE architecture guide, Kubernetes persists the state of objects like ConfigMaps, Secrets, and Deployments in the etcd key-value store.
PersistentVolumes (PV) and PVCs: Abstract underlying cloud or physical block storage away from application Pods, ensuring data persists across container restarts and rescheduling.
Resource Requests and Limits: Enforce compute governance. Requests specify the baseline CPU and memory guaranteed for scheduling onto a node, while Limits set strict ceilings preventing rogue workloads from exhausting node memory.

Auxiliary Primitive	Primary Function	Failure / Misconfiguration Risk
Namespace	Multi-tenancy and resource boundary isolation	Unintended cross-team resource clashes or quota exhaustion
ConfigMap / Secret	Decoupling environmental configuration from container images	Application crashes if keys are missing; unencrypted secrets exposure
PV / PVC	Durable stateful storage decoupled from Pod lifecycles	Pod scheduling blocks (`Pending`) if physical storage cannot bind
Requests / Limits	Capacity allocation and runaway memory containment	Overcommitment causes node instability or OOM-killed containers

Ingress and the End-to-End Request Path

While a ClusterIP Service provides an internal virtual IP and a NodePort exposes raw ports, managing hundreds of individual ports becomes unmaintainable. An Ingress acts as a centralized Layer 7 reverse proxy, consolidating HTTP and HTTPS routing, path-based dispatching, and TLS termination behind a single public entry point.

As detailed in DevopsCube's Kubernetes Architecture analysis, a Kubernetes cluster divides responsibilities: control plane nodes maintain the cluster state while worker nodes execute container workloads. The official Kubernetes architecture documentation explains that kube-proxy runs on nodes to maintain network rules and implement Service virtual routing, while the kube-apiserver acts as the control plane front end.

Here is how an incoming user request traverses the cluster from edge to execution:

Edge Entry: An external client issues a request resolved by DNS to an external Load Balancer routing into the cluster's Ingress Controller.
Path Matching: The Ingress Controller inspects the HTTP host header and URI path, mapping the request to the target cluster Service.
Network Translation: The packet reaches a worker node where kube-proxy network rules translate the Service's virtual IP into a healthy, ready Pod IP.
Execution: The target Pod receives the request, processes it inside the container runtime, and writes persistent updates to its mounted PVC.

7. Summary & What's Next

We have transitioned from viewing containers as standalone, isolated runtimes to understanding how Kubernetes coordinates them into a cohesive, self-healing system. At its foundation, a Kubernetes cluster divides responsibility between an administrative control plane and worker nodes that execute your workloads. Instead of managing container lifecycles manually, you declare your desired system state in manifests, and Kubernetes relies on continuous reconciliation loops to observe actual state and correct discrepancies automatically.

Core Primitives Summary

Primitive	Role in Cluster	Operational Takeaway
Control Plane	Manages cluster state, scheduling, and API requests	Central authority running the API server, scheduler, and controller managers.
Worker Node	Runs containerized workloads	Hosts the `kubelet`, container runtime, and `kube-proxy` proxying traffic.
Pod	Smallest deployable compute unit	Colocates tightly coupled containers sharing network and storage namespaces; ephemeral by default.
Deployment	Manages ReplicaSets declaratively	Provides automated rollouts, rollbacks, and replica scaling without manual intervention.
Service	Stable network abstraction	Gives ephemeral Pods a persistent virtual IP and distributes traffic across healthy instances.
Health Probes	Startup, liveness, and readiness checks	Prevents traffic from reaching unready containers and restarts stalled processes.

What's Next: Hands-On Practice

To move from architecture to implementation, test these primitives in a lightweight local cluster:

Spin up a local cluster: Use local orchestrators like Minikube or Kind (kind create cluster) to run a multi-node cluster environment directly on your workstation.
Deploy your first workload: Write a basic Deployment YAML exposing an NGINX container, apply it with kubectl apply -f deployment.yaml, and scale replicas using kubectl scale.
Expose and test networking: Attach a ClusterIP or NodePort Service to verify internal DNS resolution and traffic distribution across Pod IPs.
Inspect and debug: Practice cluster troubleshooting using key diagnostic commands:

# Inspect workload health and scheduled events
kubectl get pods -o wide
kubectl describe pod <pod-name>

# Stream logs and open an interactive shell inside a running container
kubectl logs -f <pod-name>
kubectl exec -it <pod-name> -- /bin/sh

References

Cluster Architecture — Kubernetes
Architecture | The Kubernetes Visual Handbook — k8s.info
GKE cluster architecture | Google Kubernetes Engine (GKE) | Google Cloud Documentation — Google Cloud Documentation
Kubernetes Components — Kubernetes
Kubernetes Architecture Explained (With Illustrated Diagrams) — DevOpsCube – Easy DevOps, SRE Guides & Reviews
Liveness, Readiness, and Startup Probes — Kubernetes
Configure Liveness, Readiness and Startup Probes — Kubernetes

Originally published at https://leon-pham.vercel.app/posts/kubernetes-architecture-and-primitives

Step-by-Step Guide: Fine-Tuning an IELTS Examiner LLM with Unsloth, QLoRA

longtk26 — Tue, 20 Oct 2026 16:02:54 +0000

In Part 1: Concepts & Architecture, we explored what fine-tuning is, how it differs fundamentally from RAG, and how QLoRA enables training large models on consumer hardware.

In this practical, hands-on tutorial, we will build a domain-specialized LLM from scratch: an official IELTS Examiner AI. We will fine-tune an 8-Billion parameter foundation model (Qwen 2.5 7B) directly on a free Google Colab instance, export the model to GGUF format.

The Tool Stack: Why Unsloth + QLoRA?
Step 1: Dataset Engineering for IELTS Evaluation
Step 2: Colab Environment Setup & GPU Check
Step 3: Loading the 4-bit Base Model
Step 4: Attaching LoRA Adapters
Step 5: Formatting Prompts & Chat Templates
Step 6: Configuring SFTTrainer & Training
Step 7: Qualitative Inference & Testing
Step 8: Exporting to GGUF & Serving with Ollama
Troubleshooting & Pro Tips

1. The Tool Stack: Why Unsloth + QLoRA?

Fine-tuning an 8B parameter model natively requires over 110 GB of VRAM. On Google Colab's free tier, we are constrained to a single NVIDIA Tesla T4 GPU (15.3 GB VRAM).

To make this feasible, we use:

Tool	Role	Why We Selected It
Unsloth	Fine-tuning Acceleration	Cuts VRAM consumption by 70% and speeds up training by 2x without any loss of accuracy.
BitsAndBytes	4-bit NormalFloat (NF4)	Compresses the 16-bit base model (~16 GB) down to just ~5.5 GB, fitting comfortably on the T4 GPU.
PEFT (Hugging Face)	Parameter-Efficient Fine-Tuning	Trains only low-rank adapter matrices ($r=16$), representing just 0.53% of all model parameters (40.3M trainable parameters out of 7.6B).
TRL (`SFTTrainer`)	Supervised Training Loop	Handles batching, sequence packing, loss masking, and gradient accumulation.
llama.cpp + Ollama	Quantization & Local Serving	Compiles the fine-tuned model into a single 4.5 GB `.gguf` file that runs with low latency on your local laptop (CPU, Mac M-series, or NVIDIA GPU).

Step 1: Dataset Engineering for IELTS Evaluation

The model's grading accuracy and feedback quality depend entirely on the consistency of the training data. For IELTS, our model must evaluate essays strictly according to the four official Cambridge assessment criteria:

Task Response (TR)
Coherence & Cohesion (CC)
Lexical Resource (LR)
Grammatical Range & Accuracy (GRA)

ChatML Data Schema (`sample_ielts_dataset.jsonl`)

Each training row is a JSON object with a "messages" array:

{
  "messages": [
    {
      "role": "system",
      "content": "You are an official IELTS Senior Examiner. Evaluate the candidate's response using the official IELTS Band Descriptors: Task Response (TR), Coherence and Cohesion (CC), Lexical Resource (LR), and Grammatical Range and Accuracy (GRA). Provide overall and component band scores, diagnostic feedback, specific sentence-level corrections, and a Band 8.5+ model revision."
    },
    {
      "role": "user",
      "content": "IELTS Writing Task 2 Prompt:\nIn many cities, the use of private cars is increasing rapidly. What are the problems associated with this, and what measures can be taken to solve them?\n\nCandidate Essay:\nIn modern cities, people buy more cars than before. This phenomenon cause many issues for society and environment..."
    },
    {
      "role": "assistant",
      "content": "### 📊 Estimated Band Score: 6.0\n\n#### Component Scores Breakdown:\n- **Task Response (6.5):** Clearly identifies problems (traffic congestion, air pollution) and plausible solutions...\n- **Coherence & Cohesion (6.0):** Information is logically arranged, but transitions are mechanical...\n- **Lexical Resource (5.5):** Adequate for basic communication, but relies on informal phrasing...\n- **Grammatical Range & Accuracy (5.5):** Frequent errors with articles and subject-verb agreements...\n\n#### ✍️ Diagnostic Sentence Improvements:\n1. *Original:* 'This phenomenon cause many issues...'\n   *Correction:* 'This phenomenon causes numerous detrimental consequences for both society and the environment.'\n\n#### 🌟 Band 8.5+ Model Rephrase:\n'The proliferation of private motor vehicles precipitates severe urban complications...'"
    }
  ]
}

Step 2: Colab Environment Setup & GPU Check

Open Google Colab and set your runtime to GPU: Runtime > Change runtime type > T4 GPU.

Installing Pre-built Wheels (Avoiding Source Compilation)

# Check GPU hardware
!nvidia-smi

# Fast installation using pre-compiled binary wheels (~1 minute)
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps trl peft accelerate bitsandbytes

⚠️ Critical Gotcha: Never run !pip install "xformers<0.0.27" on modern Colab. Doing so triggers a 40-minute source compilation (Building wheels for collected packages: xformers). Installing via the command above installs pre-built binary wheels in under 90 seconds.

Step 3: Loading the 4-bit Base Model

We load Qwen2.5-7B-Instruct in 4-bit precision via Unsloth's FastLanguageModel. Qwen 2.5 is exceptional at linguistic reasoning, grammatical parsing, and instruction following.

from unsloth import FastLanguageModel
import torch

max_seq_length = 2048  # Ample context for full essay + comprehensive grading feedback
dtype = None           # Auto-detection (Float16 on T4)
load_in_4bit = True    # 4-bit NormalFloat (NF4)

model_name = "unsloth/Qwen2.5-7B-Instruct-bnb-4bit"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=model_name,
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

Memory Footprint: The entire 7.6B model is loaded into only ~5.5 GB of VRAM, leaving ~9 GB of free VRAM for activations and optimizer states.

Step 4: Attaching LoRA Adapters

Next, we wrap the base model with low-rank trainable matrices targeting both the attention projections and the feed-forward MLP layers:

model = FastLanguageModel.get_peft_model(
    model,
    r=16,                  # LoRA rank (16-32 provides optimal expressive capacity)
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha=16,         # Scaling factor (usually equals r or 2*r)
    lora_dropout=0,        # Optimized to 0 by Unsloth for maximum speed
    bias="none",
    use_gradient_checkpointing="unsloth", # Drastically cuts activation memory
    random_state=3407,
)

Step 5: Formatting Prompts & Chat Templates

We apply the model's native chat template to convert the multi-role JSON conversation into standard token sequences:

from datasets import load_dataset, Dataset

# Load your JSONL dataset
# dataset = load_dataset("json", data_files="sample_ielts_dataset.jsonl")

def formatting_prompts_func(examples):
    convos = examples["messages"]
    texts = [
        tokenizer.apply_chat_template(convo, tokenize=False, add_generation_prompt=False)
        for convo in convos
    ]
    return {"text": texts}

dataset = dataset.map(formatting_prompts_func, batched=True)

Step 6: Configuring SFTTrainer & Training

Now we configure the SFTTrainer with an 8-bit AdamW optimizer and cosine learning rate schedule:

from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    packing=False,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,  # Effective batch size = 2 * 4 = 8
        warmup_steps=5,
        max_steps=60,                  # Adjust based on dataset size (e.g. 100-300)
        learning_rate=2e-4,             # Standard LoRA learning rate
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=5,
        optim="adamw_8bit",             # Saves 75% optimizer memory
        weight_decay=0.01,
        lr_scheduler_type="cosine",
        seed=3407,
        output_dir="ielts_outputs",
    ),
)

trainer_stats = trainer.train()

Step 7: Qualitative Inference & Testing

We switch the model to inference mode (which Unsloth accelerates by an additional 2x) and test on a completely unseen essay:

FastLanguageModel.for_inference(model)

test_messages = [
    {
        "role": "system",
        "content": "You are an official IELTS Senior Examiner. Grade the candidate's essay using the official IELTS Band Descriptors (Task Response, Coherence and Cohesion, Lexical Resource, Grammatical Range and Accuracy). Provide estimated band scores, granular feedback for each criterion, sentence-level corrections, and a Band 8.5+ model revision."
    },
    {
        "role": "user",
        "content": "Prompt: Some people say that advertising encourages us to buy things we do not really need. Others think that advertisements inform us about new products that can improve our lives. Discuss both views and give your opinion.\n\nCandidate Essay:\nAdvertising is everywhere today. In television and internet, we can see many ads. Some people think advertisements make people waste money on useless items, but others think ads provide useful knowledge. I agree that ads are helpful if we are smart consumers."
    }
]

inputs = tokenizer.apply_chat_template(
    test_messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs,
    max_new_tokens=800,
    temperature=0.3, # Low temperature ensures calibrated, deterministic grading
    use_cache=True
)

response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
print(response)

Step 8: Exporting to GGUF & Serving with Ollama

1. Save Adapter & Export 4-bit GGUF in Colab

# 1. Save LoRA weights locally
model.save_pretrained("ielts_lora_adapter")
tokenizer.save_pretrained("ielts_lora_adapter")

# 2. Automatically merge weights into 16-bit and quantize to GGUF Q4_K_M
model.save_pretrained_gguf("ielts_examiner_gguf", tokenizer, quantization_method="q4_k_m")

This compiles ielts_examiner_gguf_gguf/Qwen2.5-7B-Instruct.Q4_K_M.gguf (~4.5 GB) and an automatic Modelfile.

2. Backup to Google Drive

from google.colab import drive
drive.mount('/content/drive')

!cp -r ielts_examiner_gguf_gguf /content/drive/MyDrive/

3. Run Locally with Ollama

Download the folder from Google Drive to your local machine, open your terminal, and run:

# Register the model with Ollama
ollama create ielts-examiner -f Modelfile

# Start an interactive session
ollama run ielts-examiner

You now have a private, dedicated IELTS Examiner AI running locally on your laptop with zero API fees!

10. Troubleshooting

Problem	Cause	Solution
`Building wheels for collected packages: xformers` hangs	Incompatible pip version attempting to build CUDA C++ from source	Avoid pinning `xformers<0.0.27`; use the modern install command with pre-built wheels shown in Step 2.
CUDA Out-of-Memory (OOM)	Sequence length too long or batch size too large	Reduce `per_device_train_batch_size` to `1` or `2`, increase `gradient_accumulation_steps` to `8`, and ensure `use_gradient_checkpointing="unsloth"`.
Grade Inflation (Scores too high)	Dataset consists primarily of high-band essays	Ensure at least 30% of your training dataset consists of Band 4.5–5.5 essays with rigorous feedback explaining their weaknesses.
Lost Files after Colab Disconnect	Colab runtime disk is temporary	Always mount Google Drive at the beginning of the notebook and save checkpoints directly to `/content/drive/MyDrive/`.

Conclusion & Resources

By combining QLoRA with Unsloth, we transformed an 8B foundation model into a specialized IELTS assessment engine within a single hour on free cloud hardware.

LLM Fine-Tuning: Concepts, Architecture, and Fine-Tuning vs. RAG

longtk26 — Tue, 20 Oct 2026 16:01:45 +0000

Fine-tuning has emerged as one of the most critical skills for modern AI engineers. While large general-purpose foundation models (like GPT, Claude, or Qwen) are impressive out of the box, building production-grade domain applications often demands specialized behavior, deterministic formatting, or deep domain expertise that prompting alone cannot reliably achieve.

In this guide, we break down the foundational concepts of LLM fine-tuning, compare it with Retrieval-Augmented Generation (RAG), examine parameter-efficient training techniques (LoRA/QLoRA), and help you decide exactly when and when not to fine-tune.

What is Fine-Tuning?
- The LLM Training Lifecycle
Why Fine-Tune? Core Motivations
Fine-Tuning vs. RAG
- Core Mental Model: Open-Book vs. Closed-Book Exam
- Detailed Comparison
- The Hybrid Approach: RAG + Fine-Tuning
- Decision Framework: Which Should You Choose?
How an LLM Processes Input and Generates Output (Under the Hood)
- The End-to-End Inference Lifecycle
- The Tokenization Mechanism: From Words to Numbers
- Internal Processing: Embeddings, Positional Encoding & Attention
- The Prediction Loop: Logits, Softmax & Autoregressive Decoding
- Sampling Controls: Temperature, Top-p & Stop Tokens
Methods of Fine-Tuning
- Full Parameter Fine-Tuning (FPFT)
- Parameter-Efficient Fine-Tuning (PEFT) & LoRA
- QLoRA: 4-bit Quantized LoRA
- Hardware Comparison: FPFT vs. LoRA vs. QLoRA
Essential Hyperparameters & Micro-Concepts
- Epochs vs. Steps vs. Batch Size
- Gradient Accumulation & Effective Batch Size
When is Fine-Tuning Useful? (And When is it Not?)
- Best Scenarios for Fine-Tuning
- Scenarios Where You Should NOT Fine-Tune
Summary & What's Next

1. What is Fine-Tuning?

At its core, fine-tuning is the process of taking a pre-trained model (which already understands grammar, world facts, and language patterns from trillions of tokens) and training it further on a curated, domain-specific dataset to adapt its weights for a specific task, persona, or style.

The LLM Training Lifecycle

Modern language models undergo a three-stage lifecycle:

flowchart LR
    A["1. Pre-training<br/>Web Corpus<br/>Base Knowledge"] --> B["2. SFT / Instruct<br/>Curated QA<br/>Instruction Following"]
    B --> C["3. Alignment<br/>RLHF / DPO<br/>Safety & Preference"]

    style A fill:#0369a1,stroke:#38bdf8,stroke-width:2px,color:#ffffff;
    style B fill:#15803d,stroke:#4ade80,stroke-width:2px,color:#ffffff;
    style C fill:#b45309,stroke:#fbbf24,stroke-width:2px,color:#ffffff;

Pre-training (Base Models): Models like Llama-3-8B-Base or Qwen2.5-7B read massive swaths of the internet. Their objective is raw next-token prediction. They complete sentences, but they cannot converse or reliably follow instructions.
Supervised Fine-Tuning (SFT / Instruct Models): The base model is trained on question-answer or multi-turn conversational datasets. This transforms a raw text completer into an assistant (Llama-3-8B-Instruct).
Domain-Specific Fine-Tuning (Our Focus): You take an Instruct model and fine-tune it on proprietary tasks—for example, evaluating IELTS essays according to strict Cambridge band descriptors.

2. Why Fine-Tune? Core Motivations

Why go through the effort of fine-tuning when we can simply write a prompt for GPT-4 or Claude?

mindmap
  root((Why Fine-Tune?))
    Style & Persona
      Strict tone consistency
      Unwavering examiner persona
      Specialized grading rubrics
    Output Determinism
      Reliable JSON schemas
      Predictable structured output
      Zero prompt leakage
    Efficiency & Cost
      Distill 70B behavior into 7B
      Drastically shorter prompts
      Lower latency and API costs
    Edge & Privacy
      Run on-premise or locally
      Zero data sent to third parties
      Full sovereignty over weights

Strict Style and Persona Consistency: Prompt engineering often suffers from "drift" or breaks character over extended contexts. Fine-tuning hardcodes the persona directly into the model's weights.
Complex Output Formatting: When your application requires a strict multi-section output, fine-tuned models produce the exact schema reliably without needing fragile retry loops.
Prompt Compression & Latency Reduction: With zero-shot prompting, you might need to send 2,000 tokens of rubrics, instructions, and few-shot examples with every single API request. Fine-tuning bakes those instructions into the model, cutting input prompt lengths by up to 80% and slashing latency and inference costs.
Performance on Edge Hardware: You can fine-tune a compact 7B or 8B model to rival or outperform a 70B+ generic model on your specific vertical task, allowing you to deploy on lightweight hardware or on-premise servers.
Data Privacy & Governance: Fine-tuning open-weight models allows enterprises to keep all sensitive customer data within their own Virtual Private Cloud (VPC) or local machine.

3. Fine-Tuning vs. RAG

One of the most frequent question in AI is: "Should I use RAG or Fine-Tuning?"

Core Mental Model: Open-Book vs. Closed-Book Exam

RAG (Retrieval-Augmented Generation) is an Open-Book Exam: The student has good general reasoning skills. When asked a question, they look up external manuals, company wikis, or database records and synthesize an answer based on what they just retrieved.
Fine-Tuning is Studying for the Exam (Internalized Expertise): The student has memorized patterns, stylistic habits, and specialized procedural skills. When asked to perform, the reaction is instinctive and fast.

Detailed Comparison

Dimension	Retrieval-Augmented Generation (RAG)	Supervised Fine-Tuning (SFT)
Primary Purpose	Injecting dynamic, up-to-date factual knowledge	Modifying form, behavior, tone, style, and reasoning patterns
Knowledge Freshness	Real-time: Update your vector database, and the model instantly accesses new documents	Static: Knowledge is frozen at training time; updating requires retraining
Hallucination Mitigation	High: Grounded directly in retrieved source chunks with explicit citations	Moderate: Can still hallucinate facts if trained on noisy or contradictory data
Cost Profile	Low upfront setup; higher recurring per-query cost (longer context embeddings & tokens)	Higher upfront compute cost; lower recurring query cost (shorter prompts, smaller models)
Latency	Slower (Vector search + Embedding generation + Large context window)	Faster (Direct inference with compact prompts)
Context Window Consumption	High (Passages and reference chunks occupy prompt tokens)	Low (Knowledge and rules are internal to the model)

The Hybrid Approach: RAG + Fine-Tuning

In modern production systems, RAG and Fine-Tuning are not mutually exclusive—they are complementary:

flowchart TD
    UserQuery["User Prompt<br/>Essay + Exam Question"] --> RAG["RAG Module<br/>Official Rubrics Database"]
    RAG --> ContextAugmented["Augmented Prompt<br/>User Essay + Rubrics"]
    ContextAugmented --> FTModel["Fine-Tuned 8B Model<br/>Calibrated Examiner Persona"]
    FTModel --> Output["Assessment Report<br/>Accurate & Structured"]

    style RAG fill:#0369a1,stroke:#38bdf8,stroke-width:2px,color:#ffffff;
    style FTModel fill:#15803d,stroke:#4ade80,stroke-width:2px,color:#ffffff;
    style Output fill:#7e22ce,stroke:#c084fc,stroke-width:2px,color:#ffffff;

Use Fine-Tuning to teach the model HOW to behave (e.g. grade like an official Cambridge examiner, never give generic praise, generate exact JSON).
Use RAG to feed the model WHAT facts or reference materials to consult (e.g. the specific exam version question, updated vocabulary).

Decision Framework: Which Should You Choose?

Use this decision tree to pick the right strategy:

flowchart TD
    Start["Improve LLM Performance"] --> Q1{"Need dynamic or<br/>private facts?"}
    Q1 -- Yes --> Q2{"Changes daily<br/>or frequently?"}
    Q2 -- Yes --> UseRAG["✅ Use RAG<br/>(Vector DB + Search)"]
    Q2 -- No --> Q3{"Need custom tone<br/>or structure?"}
    Q3 -- Yes --> UseHybrid["✅ Use Hybrid<br/>(RAG + Fine-Tuning)"]
    Q3 -- No --> UseRAG

    Q1 -- No --> Q4{"Works with<br/>few-shot prompts?"}
    Q4 -- Yes --> UsePrompt["✅ Use Prompting<br/>(Fastest & Low Cost)"]
    Q4 -- No --> Q5{"Have 500+ quality<br/>examples?"}
    Q5 -- Yes --> UseFT["✅ Fine-Tune<br/>(PEFT / QLoRA)"]
    Q5 -- No --> CollectData["⚠️ Curate Data<br/>(Bootstrap Pairs First)"]

    style Start fill:#1e293b,stroke:#94a3b8,stroke-width:2px,color:#ffffff;
    style UseRAG fill:#0369a1,stroke:#38bdf8,stroke-width:2px,color:#ffffff;
    style UseHybrid fill:#7e22ce,stroke:#c084fc,stroke-width:2px,color:#ffffff;
    style UsePrompt fill:#15803d,stroke:#4ade80,stroke-width:2px,color:#ffffff;
    style UseFT fill:#c2410c,stroke:#fb923c,stroke-width:2px,color:#ffffff;
    style CollectData fill:#b45309,stroke:#fbbf24,stroke-width:2px,color:#ffffff;

4. How an LLM Processes Input and Generates Output (Under the Hood)

To understand how fine-tuning alters a model's behavior, we must first understand how an LLM natively receives text, processes it through its neural architecture, and generates a response.

The End-to-End Inference Lifecycle

When you ask an LLM to evaluate an IELTS essay, the following cycle occurs:

Architectural Flow Breakdown:

flowchart LR
    In["1. User Prompt<br/>(Raw String)"] --> Tok["2. Tokenizer<br/>(BPE / IDs)"]
    Tok --> Emb["3. Embedding<br/>+ RoPE"]
    Emb --> Trans["4. 28 Layers<br/>(Attention + LoRA)"]
    Trans --> Head["5. LM Head<br/>(Logits)"]
    Head --> Soft["6. Softmax<br/>(Temp T=0.3)"]
    Soft --> Sam["7. Next Token<br/>(Top-p=0.9)"]
    Sam --> Loop{"Stop Token<br/>&lt;|im_end|&gt;?"}
    Loop -- No --> Trans
    Loop -- Yes --> Out["8. Output Essay<br/>Review Report"]

    style In fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#f8fafc;
    style Tok fill:#1e293b,stroke:#818cf8,stroke-width:2px,color:#f8fafc;
    style Emb fill:#1e293b,stroke:#a855f7,stroke-width:2px,color:#f8fafc;
    style Trans fill:#14532d,stroke:#4ade80,stroke-width:2px,color:#f8fafc;
    style Head fill:#1e293b,stroke:#38bdf8,stroke-width:2px,color:#f8fafc;
    style Soft fill:#7c2d12,stroke:#fb923c,stroke-width:2px,color:#f8fafc;
    style Sam fill:#581c87,stroke:#c084fc,stroke-width:2px,color:#f8fafc;
    style Loop fill:#334155,stroke:#94a3b8,stroke-width:2px,color:#f8fafc;
    style Out fill:#0f172a,stroke:#4ade80,stroke-width:2px,color:#f8fafc;

The Tokenization Mechanism: From Words to Numbers

Neural networks cannot process raw characters or words directly; they only perform linear algebra on numbers.

Why Not Words or Characters?

Word-level tokenization: The vocabulary would be infinite. It would fail completely on typos, hyphenated words, or domain jargon (e.g. "lexicalization").
Character-level tokenization: Sequences become too long (each letter is an input), and individual characters convey minimal semantic meaning.

The Modern Solution: Subword Tokenization (Byte-Pair Encoding - BPE)

Modern models (like Qwen 2.5 and Llama 3) use Byte-Pair Encoding (BPE). Frequent words remain single tokens, while rare words are broken into meaningful subword chunks:

Original Word	Subword Breakdown	Why?
`"education"`	`["education"]`	Common word; assigned its own dedicated token ID.
`"unquestionably"`	`["un", "question", "ably"]`	Split into common prefix, root, and suffix tokens.
`"overemphasizing"`	`["over", "emphas", "izing"]`	Common subword stems combined flexibly.

Example: Tokenizing an IELTS Prompt

When you send:

IELTS Writing Task 2: Plastic pollution is damaging oceans.

The tokenizer consults its dictionary table and produces an array of integer Token IDs:

[38384, 18274, 8431, 220, 17, 25, 27329, 21890, 374, 21950, 24590, 13]

Special Tokens in ChatML Templates

In conversational models, special delimiter tokens tell the model who is speaking. Without special tokens, the model cannot distinguish between instructions and candidate essays:

<|im_start|>system : Marks the start of the Senior Examiner rubric persona.
<|im_start|>user : Marks the candidate essay and prompt.
<|im_start|>assistant : Signals the model that it is now its turn to generate the score report.
<|im_end|> : The Stop Token. Signals that generation is complete.

Internal Processing: Embeddings, Positional Encoding & Attention

Once text is tokenized into IDs:

Embedding Lookup Matrix (W_e in R^(V * d_model)): Each token ID is mapped to a continuous, high-dimensional vector. For example, in an 8B model with d_model = 4096, the integer 38384 becomes a vector of 4,096 floating-point numbers encoding its base semantic meaning.
Rotary Positional Embeddings (RoPE): Self-attention on its own is order-agnostic ("dog bites man" would look identical to "man bites dog"). RoPE mathematically rotates the embedding vectors in complex coordinate space so the model knows the exact position and distance of each word in the sequence.
Multi-Head Self-Attention (The Core Engine): Each token projects into Query (Q), Key (K), and Value (V) vectors:

Attention(Q, K, V) = softmax((QK^T) / (sqrt(d_k)))V

This allows the token "congestion" to attend back to "traffic" and forward to "delays", capturing deep contextual grammar and semantic dependencies across the entire essay.

The Prediction Loop: Logits, Softmax & Autoregressive Decoding

LLMs do not produce a whole paragraph in a single forward pass. They are autoregressive: they generate text one single token at a time.

1. From Hidden States to Logits

After passing through all 28–36 transformer layers, the model extracts the final hidden state vector corresponding to the very last token.

This vector passes through the Language Model Head (lm_head), which calculates raw confidence scores (called logits) for each of the 152,000+ words in the model's vocabulary.

2. From Logits to Probabilities (Softmax)

Because raw logits can be negative, zero, or arbitrary numbers, the model passes them through the Softmax function to turn them into clean percentages (probabilities) between 0% and 100% that all add up to 1.0 (100%).

In simple terms:

Words with higher logit scores receive higher probabilities.
The Temperature (T) setting controls how bold or conservative the model is: a low temperature concentrates probability onto the top word (more focused and deterministic), while a higher temperature spreads probability more evenly (more creative).

Sampling Controls: Temperature, Top-p & Stop Tokens

Once we have a probability distribution over the 152,000 candidate tokens, how do we pick the winner?

Candidate Tokens:
  "###"        -> 64.2%  <-- Most likely next token!
  "Overall"    -> 21.5%
  "The"        -> 8.1%
  "Candidate"  -> 4.3%
  "banana"     -> 0.000001% (Cut off by Top-p)

1. Temperature (`T`)

Low Temperature (T = 0.1 - 0.3): Sharpen the distribution! The highest probability token becomes overwhelmingly favored. This provides deterministic, consistent grading—essential for IELTS examiner scoring.
High Temperature (T = 0.8 - 1.2): Flatten the distribution! Lower-probability words get a higher chance to be chosen, creating creative, diverse storytelling (but increasing the risk of hallucinations).
Greedy Decoding (T = 0): Strictly pick the token with the highest probability (argmax).

2. Top-p (Nucleus Sampling)

Instead of considering all 152,000 tokens, the model pools only the top tokens whose cumulative probability adds up to p (e.g., p = 0.90). The remaining 10% "long tail" of weird or irrelevant words is discarded, eliminating bizarre hallucinations.

3. Stop Tokens (`<|im_end|>`)

How does the generation loop know when to finish?
When the model concludes its final revision paragraph, it predicts the special token <|im_end|>. The inference engine recognizes this stop token, halts the loop, and triggers the Detokenizer to convert all generated token IDs back into human-readable text on your screen.

5. Methods of Fine-Tuning

Full Parameter Fine-Tuning (FPFT)

In Full Parameter Fine-Tuning, every single parameter in the model is updated.

Problem: An 8-Billion parameter model stored in 16-bit float requires:
- Model weights: 8 * 2 GB = 16 GB
- Gradient tensors: 16 GB
- Optimizer states (AdamW tracks 2 momentum variables per weight): 8 * 8 GB = 64 GB
- Activations & buffers: ~ 16 GB
- Total VRAM required: > 110 GB! (Requires multi-A100 server clusters).

Parameter-Efficient Fine-Tuning (PEFT) & LoRA

In 2021, Microsoft researchers published LoRA (Low-Rank Adaptation).

The core insight: The weight updates ΔW during domain adaptation have a low "intrinsic dimension" (rank). Instead of updating the full weight matrix W_0 in R^(d * k), LoRA decomposes the update into two low-rank matrices A and B:

ΔW = B * A

Where B in R^(d * r) and A in R^(r * k), with rank r << \min(d, k).

   Original Weight Matrix W (Frozen)            LoRA Adapter Matrices (Trainable)
        [ d x k ]                                    B [ d x r ]   x   A [ r x k ]
  (Millions of parameters)                           (e.g., r = 16: ~0.5% of parameters)

During training, W_0 is frozen (requires no optimizer states).
Only A and B receive gradient updates.
Memory usage for optimizer states drops by over 95%!

QLoRA: 4-bit Quantized LoRA

Introduced by Dettmers et al. (2023), QLoRA takes LoRA further by combining:

4-bit NormalFloat (NF4): Quantizes the frozen base model weights from 16-bit down to 4-bit without noticeable perplexity degradation.
Double Quantization (DQ): Quantizes the quantization constants themselves, saving an additional 0.37 bits per parameter.
Paged Optimizers: Uses CUDA Unified Memory to automatically page memory between GPU and CPU RAM during gradient spikes.

With QLoRA, an 8-Billion parameter model requires only ~5.5 GB VRAM for weights, allowing full fine-tuning on a single free Google Colab T4 GPU (15 GB VRAM)!

Hardware Comparison: FPFT vs. LoRA vs. QLoRA

Technique	Precision of Base Model	Trainable Parameters	GPU VRAM Required (8B Model)	Can run on Free Colab T4?
Full Fine-Tuning	16-bit FP16/BF16	100% (8.0B)	~110 GB+	❌ No (Requires multiple A100s)
Standard LoRA	16-bit FP16/BF16	~0.5% – 2% (40M)	~24 GB – 32 GB	❌ No (Requires A10G / L4 / V100)
QLoRA	4-bit NF4	~0.5% – 2% (40M)	~7 GB – 10 GB	✅ Yes (Runs smoothly on T4 15GB)

6. Essential Hyperparameters & Micro-Concepts

When beginning LLM fine-tuning, you will encounter parameters and configuration options that look familiar from traditional machine learning, but carry distinct rules of thumb in modern LLM training.

Epochs vs. Steps vs. Batch Size

To understand how fine-tuning progresses, you must grasp how data flows into the model:

flowchart TD
    Dataset["Full Dataset<br/>(800 Samples)"] --> Split["Split into Mini-Batches"]
    Split --> Step1["Step 1: 8 Samples<br/>Forward + Update"]
    Step1 --> Step2["Step 2: 8 Samples<br/>Forward + Update"]
    Step2 --> Dots["... 100 Total Steps ..."]
    Dots --> EpochEnd["All 800 Seen<br/>= 1 Complete Epoch"]

    style Dataset fill:#0369a1,stroke:#38bdf8,stroke-width:2px,color:#ffffff;
    style Split fill:#1e293b,stroke:#94a3b8,stroke-width:2px,color:#ffffff;
    style Step1 fill:#1e293b,stroke:#64748b,stroke-width:2px,color:#ffffff;
    style Step2 fill:#1e293b,stroke:#64748b,stroke-width:2px,color:#ffffff;
    style Dots fill:#1e293b,stroke:#64748b,stroke-width:2px,color:#ffffff;
    style EpochEnd fill:#15803d,stroke:#4ade80,stroke-width:2px,color:#ffffff;

Epoch: One complete cycle through the entire training dataset. If your dataset has 1,000 examples, 1 epoch means the model has seen all 1,000 examples exactly once.
Step (Iteration): A single weight update. One step consists of:
1. Passing a batch of data through the model (Forward Pass).
2. Calculating loss and computing gradients (Backward Pass).
3. Updating the model weights using the optimizer.

The simple calculation formula is:

Total Training Steps = (Total Examples * Number of Epochs) / Effective Batch Size

Example: If you have 800 essays, train for 3 epochs, and use an effective batch size of 8:

Total Steps = (800 * 3) / 8 = 300 steps

Gradient Accumulation & Effective Batch Size

On consumer GPUs like the Colab T4 (15 GB VRAM), setting a physical batch size of 16 or 32 for an 8B model will immediately trigger an Out-Of-Memory (OOM) error.

However, a tiny batch size (like 1 or 2) makes the gradient updates noisy. How do we solve this? Gradient Accumulation.

Per-Device Batch Size (B_device): The number of samples loaded into GPU memory at any single moment (e.g., 2).
Gradient Accumulation Steps (G): Instead of updating weights after every single batch, the optimizer computes gradients for G batches, accumulates (sums) the gradients in memory, and only updates the weights once every G steps!

Effective Batch Size = Per-Device Batch Size * Gradient Accumulation Steps * Number of GPUs

7. When is Fine-Tuning Useful? (And When is it Not?)

Best Scenarios for Fine-Tuning

Domain Assessment & Scoring: e.g., IELTS, TOEFL, SAT essays, or coding interview grading adhering to specific rubrics.
Complex Schema Adherence: Enforcing strict, unvarying JSON, YAML, or XML output without preamble or conversational fluff.
Domain Vocabulary & Tone: Medical triage dialogues, legal clause rewriting, financial auditing tone.
Latency & Edge Deployments: Shrinking prompt sizes so a compact model can be deployed locally on laptops with Ollama.

Scenarios Where You Should NOT Fine-Tune

Dynamic / Fast-Changing Knowledge: Stock prices, company product catalogs, weather, news. Use RAG instead.
General Fact Retrieval: Fine-tuning on facts frequently induces subtle hallucinations. Use RAG instead.
Insufficient Data: If you have fewer than 100 examples, few-shot prompting in Claude or GPT-4 will almost always outperform a fine-tuned model.

8. Summary & What's Next

Core Concept	Summary Takeaway
What is it?	Adapting model weights to internalize style, structure, and specialized reasoning.
RAG vs. FT	RAG brings the facts (open book); Fine-Tuning shapes the behavior and skill (retained mastery).
Epochs & Steps	LLMs only need 1–3 epochs; gradient accumulation achieves large effective batch sizes on limited VRAM.

In Part 2: Step-by-Step Implementation Guide, we will implement a complete end-to-end training pipeline using Unsloth, QLoRA, and Google Colab, training our own custom IELTS Examiner AI.

Understanding Observability Stack with Grafana, Prometheus, Loki, and Alloy

longtk26 — Thu, 27 Aug 2026 17:09:00 +0000

Understanding Observability Stack with Grafana, Prometheus, Loki, and Alloy
- Table of Contents
- Introduction
- How Observability Components Fit Together
- Simple Data Flow
- Core Stack Components & Their Usages
- 1. Prometheus: Metrics Database
- 2. Grafana Loki: Logs Database
- 3. Grafana: Dashboards & Visualization
- 4. Exporters: System Metric Translators
- 5. Grafana Alloy: Telemetry Collector
- Component Summary Table
- Conclusion

Introduction

When you deploy applications to a server, you need a reliable way to know if everything is running smoothly. This is where observability comes in.

Observability helps you answer simple but essential questions:

Is the server running out of CPU or memory?
How many requests per second is the API handling?
Why did a user get an error, and what exact line in the logs explains it?

In a modern observability stack, telemetry is primarily divided into two main types of data:

Metrics: Numerical values recorded over time (e.g., CPU usage %, memory in MB, request count, response time).
Logs: Timestamped text messages output by applications or the operating system (e.g., error stack traces, informational events).

To collect, store, and visualize this data, we combine five core open-source components: Prometheus, Grafana Loki, Grafana, Exporters, and Grafana Alloy.

How Observability Components Fit Together

Here is a simple overview of how telemetry data moves from your applications all the way to your browser:

Simple Data Flow

Emit Data: Applications and system exporters produce metrics and log messages.
Collect: Grafana Alloy runs locally, gathers the metrics and logs, adds useful labels (such as app="my-api"), and forwards them.
Store:
- Prometheus receives and stores the numerical metrics.
- Loki receives and stores the log entries.
Visualize: Grafana connects to both Prometheus and Loki, allowing you to view graphs, inspect logs, and set up alerts on a single screen.

Core Stack Components & Their Usages

Let's break down each component and what it is used for.

1. Prometheus: Metrics Database

Prometheus is an open-source database built specifically for time-series metrics.

What it does: Stores numbers paired with timestamps and labels.
Example Data:

  http_requests_total{app="web", status="200"} 1540
  cpu_usage_percentage{host="server-1"} 42.5

How you query it: Uses PromQL (Prometheus Query Language) to compute rates, averages, and totals over time:

  rate(http_requests_total[5m])

Main Usage: Measuring CPU/RAM usage, request throughput, error counts, and response latency.

2. Grafana Loki: Logs Database

Grafana Loki is an open-source log aggregation system designed by Grafana.

What it does: Stores text log streams from all your applications and containers in one searchable place.
Key Advantage: Unlike traditional search engines that index every single word, Loki indexes only the labels (like app="backend", env="prod"). This makes it fast, lightweight, and very cheap on disk and memory.
How you query it: Uses LogQL to search and filter log lines:

  {app="backend"} |= "error"

Main Usage: Searching application logs, investigating exceptions, and reviewing debug traces.

3. Grafana: Dashboards & Visualization

Grafana is the web UI where you view and interact with all your monitoring data.

What it does: Connects to data sources like Prometheus and Loki to render graphs, charts, counters, and log panels.
Key Features:
- Unified Dashboards: Display CPU graphs, database queries, and live error logs side-by-side.
- Alerting: Sends notifications to Slack, Discord, or email when thresholds are exceeded (e.g., when CPU > 90%).
- Explore Mode: Quickly run ad-hoc queries without needing to create a permanent dashboard.
Main Usage: The central control room for developers and operators to monitor system health.

4. Exporters: System Metric Translators

Software like Linux OS, PostgreSQL, or Redis does not natively output Prometheus metrics. Exporters act as translators: they read system stats and expose them as a /metrics page that Prometheus can read.

Node Exporter: Measures host-level stats (OS CPU, RAM, disk space, network traffic).
cAdvisor (Container Advisor): Measures Docker container stats (per-container CPU, RAM limits, container status).
Database Exporters (Postgres / Redis Exporters): Measure active database connections, query throughput, and cache hit rates.
Main Usage: Providing visibility into servers, containers, and databases without modifying their source code.

5. Grafana Alloy: Telemetry Collector

Grafana Alloy is Grafana's modern, lightweight telemetry agent.

What it does: Runs on the server where your applications live. It gathers /metrics from apps and exporters, tails log files from Docker containers, and pushes everything to Prometheus and Loki.
Why it is useful: Instead of installing separate tools for metrics and logs (like older setups with Promtail and Prometheus Agent), Alloy handles both in a single lightweight program.
Main Usage: Local scraping, enriching data with common labels, and shipping telemetry to the databases.

Component Summary Table

Here is a quick summary of all five components:

Component	Role	What It Does	Telemetry Type	Query Language
Prometheus	Metrics Database	Stores numerical measurements over time	Metrics	PromQL
Grafana Loki	Logs Database	Stores and indexes log streams	Logs	LogQL
Grafana	Visualization UI	Displays charts, graphs, and sends alerts	Dashboards & Alerts	N/A (Uses PromQL & LogQL)
Exporters	Metric Translators	Reads OS, Docker, and DB stats and outputs metrics	Metrics	N/A (HTTP `/metrics`)
Grafana Alloy	Telemetry Collector	Scrapes metrics and tails logs, then pushes to databases	Metrics & Logs	N/A (Agent Configuration)

Conclusion

I hope this blog is helpful to you.
Understanding these five fundamental building blocks, you can get started with modern observability. Thank you for reading.

Overview: Change Data Capture (CDC)

longtk26 — Fri, 17 Apr 2026 14:03:41 +0000

Overview: Change Data Capture (CDC)

Welcome! In this post, we'll explore Change Data Capture (CDC) — one of those foundational concepts that can genuinely transform the way you think about data synchronization in distributed systems. Let's dive in!

What is Change Data Capture?
Use Cases
- Cache Invalidation
- Data Integration
- Real-Time Analysis
Pros and Cons
- Pros
- Cons
When Should You Use CDC?
When Should You NOT Use CDC?
Wrapping Up

What is Change Data Capture?

Change Data Capture (CDC) is a method used in databases to track and record changes made to data.

CDC captures modifications like inserts, updates, and deletes, and makes them available for downstream systems to consume — either for analysis, replication, or synchronization purposes. CDC helps maintain data consistency across different systems by recording alterations in real time.

Think of it like a digital detective sitting quietly inside your database, logging every change that happens and when it occurred — so that other systems never fall behind.

Use Cases

Cache Invalidation

Automatically invalidate entries in a cache as soon as the corresponding record changes or is removed.

If your cache (such as Redis, Memcached, or Infinispan) runs as a separate process, the invalidation logic can be placed into a dedicated service — completely decoupled from your main application. In more advanced setups, you can even use the data from the change event itself to update the affected cache entries directly, rather than just evicting them.

Data Integration

Data is often stored in multiple places, especially when used for different purposes or in slightly different shapes. Keeping those multiple systems in sync can quickly become a maintenance nightmare.

Example: When a system uses a separate search service like Elasticsearch, it's critical to keep the search index synchronized with the primary database whenever records are created or updated.

CDC makes this elegant: changes are captured from the database and automatically propagated to the search service. Instead of sprinkling Elasticsearch update calls throughout your business logic, a lightweight event-processing layer handles it asynchronously. This simplifies the application design while improving scalability and maintainability.

Real-Time Analysis

CDC powers real-time analytics platforms by continuously feeding data changes into analytical systems. Organizations can then derive insights from fresh data and respond swiftly to changing conditions — all without expensive batch jobs or manual ETL pipelines.

Pros and Cons

Pros

1. Decouples Application Logic

The main application no longer needs to explicitly publish events or sync data to downstream systems.
Reduces duplicated logic across services — one source of truth emits events, and consumers react independently.

2. Near Real-Time Data Synchronization

Changes from the database are streamed almost instantly.
Ideal for search indexing, analytics pipelines, and cache management.

3. Non-Intrusive to Existing Systems

Works at the database level (via WAL / binlog), so there's usually no need to heavily modify existing application code.
Particularly useful for integrating with legacy systems where source code changes are risky or costly.

4. Event-Driven Architecture Enablement

Naturally integrates with streaming platforms like Apache Kafka.
Makes it easier to build reactive microservices and real-time data pipelines.

5. Data Consistency (Eventually Consistent)

Ensures downstream systems gradually converge toward the source of truth over time.
Change events are durable and ordered, making them reliable for replication.

6. Supports Multiple Consumers

One stream of change events can feed multiple services simultaneously — search indexing, analytics, auditing, cache invalidation, and more.

Cons

1. Increased System Complexity

Requires additional infrastructure: a message broker (e.g., Kafka), connectors, and monitoring tooling.
Debugging issues that span multiple components can be significantly harder than debugging a simple API call.

2. Schema Evolution Challenges

Changes in the database schema (adding/removing columns, renaming tables) can break downstream consumers if not handled carefully.
A solid schema management strategy — such as a schema registry — becomes essential at scale.

3. Operational Overhead

Someone needs to manage connectors, track offsets, handle failures, and configure retries.
This requires a level of DevOps maturity that smaller teams may not yet have.

4. Learning Curve

Teams need to invest time in understanding event-driven design principles, CDC concepts, and the tooling involved.
The mental model shift from "call an API" to "react to events" can take time to internalize.

5. Data Filtering and Transformation Are Limited at the Source

Raw database change events may not perfectly match what downstream consumers need.
An additional transformation or processing layer is often required to shape, filter, or enrich events before they reach their destination.

When Should You Use CDC?

Use CDC when:

You need to sync data to systems like Elasticsearch, a data warehouse, or a cache layer
You are building an event-driven or microservices architecture
You want to reduce coupling between services
You need audit logs or historical change tracking
You have multiple downstream consumers of the same data
You want near real-time data propagation without modifying core application logic

** Examples:**

Sync user records from PostgreSQL → Elasticsearch for full-text search
Stream order events into an analytics pipeline for real-time dashboards

When Should You NOT Use CDC?

Avoid CDC when:

Your system requires strong consistency — CDC is eventually consistent and introduces some propagation delay
The system is simple or monolithic and doesn't benefit from event streaming
You only have one consumer and a direct API call or a database trigger would suffice
Your team lacks experience in operating distributed systems
Infrastructure cost and operational complexity are a concern
Your database is already under heavy load (CDC adds overhead via WAL reading)

Examples:

A simple CRUD app with basic search — just update Elasticsearch directly from the application
A low-scale system with no real-time data requirements

Wrapping Up

CDC is a powerful pattern, but like any tool, it shines brightest when applied to the right problem. If you are building systems where data needs to flow reliably across multiple components in near real time, CDC is absolutely worth the investment.

Thank you so much for reading! I hope this overview gave you a clear and practical understanding of Change Data Capture. If you found this helpful, feel free to explore the Debezium setup guide to see how to put these concepts into practice with real infrastructure. Happy building! 🚀

Debezium: CDC in Practice

longtk26 — Fri, 17 Apr 2026 14:03:35 +0000

If you've read the Overview CDC post, you already know why Change Data Capture is such a compelling pattern. Now it's time to get our hands dirty. In this guide, we'll walk through a complete, working Debezium setup — from standing up Kafka and Debezium Connect with Docker, all the way to watching live database changes flow into Elasticsearch in real time. Let's go! 🛠️

What is Debezium?
Setup: PostgreSQL Connector for Tracking Database Changes
- Step 1 – Create the Docker Network
- Step 2 – Prepare the Custom Debezium Connect Image
- Step 3 – Prepare the Docker Compose File
- Step 4 – Start the Services
- Step 5 – Configure the PostgreSQL Connector via Debezium UI
Setup: Synchronization from PostgreSQL to Elasticsearch
- Step 1 – Start Elasticsearch and Kibana
- Step 2 – Register the Connectors
- Step 3 – Verify the Pipeline
Wrapping Up

What is Debezium?

Debezium is an open-source project that provides a low-latency data streaming platform for Change Data Capture (CDC). It connects to your database, monitors its transaction log (e.g., PostgreSQL's Write-Ahead Log), and emits a stream of change events for every row-level modification.

Your applications then consume these events and react accordingly — syncing to a search index, invalidating a cache, updating a data warehouse, or triggering downstream workflows.

Key characteristics:

Open source under the Apache 2.0 license
Built on top of Kafka Connect — battle-tested, scalable, and extensible
Supports many popular databases: PostgreSQL, MySQL, MongoDB, SQL Server, Oracle, and more
Guarantees at-least-once delivery of change events with configurable offset management

Setup: PostgreSQL Connector for Tracking Database Changes

Step 1 – Create the Docker Network

All services in this setup share a common Docker bridge network. Create it once before starting any containers:

docker network create base-network

Step 2 – Prepare the Custom Debezium Connect Image

We need a custom Debezium Connect image that includes the Confluent Elasticsearch Sink Connector plugin. Create a Dockerfile inside a folder named connect/:

FROM debezium/connect:3.0.0.Final

# Install the Confluent Elasticsearch Sink Connector for Kafka Connect.

ARG ELASTICSEARCH_SINK_VERSION=15.1.1

RUN set -eux; \

    curl -fsSL "https://hub-downloads.confluent.io/api/plugins/confluentinc/kafka-connect-elasticsearch/versions/${ELASTICSEARCH_SINK_VERSION}/confluentinc-kafka-connect-elasticsearch-${ELASTICSEARCH_SINK_VERSION}.zip" -o /tmp/es-sink.zip; \

    mkdir -p /kafka/connect/confluentinc-kafka-connect-elasticsearch; \

    unzip -q /tmp/es-sink.zip -d /kafka/connect/confluentinc-kafka-connect-elasticsearch; \

    rm -f /tmp/es-sink.zip

Note: If you don't need Elasticsearch, skip the custom image and use the official debezium/connect:3.0.0.Final image directly in your Compose file.

Step 3 – Prepare the Docker Compose File

Create a docker-compose.yml at the project root with the following content:

version: '3.9'
services:
  kafka-ui:
    container_name: kafka-ui
    image: ghcr.io/kafbat/kafka-ui:latest
    ports:
      - "8070:8080"
    depends_on:
      - kafka
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
      DYNAMIC_CONFIG_ENABLED: 'true'
    networks:
      - base-network

  kafka:
    image: apache/kafka:4.0.2
    container_name: kafka
    ports:
      - "9092:9092"
      - "9094:9094"
    environment:

      ############################################
      # KRaft Metadata & Node Identity
      ############################################

      # Unique ID for this Kafka node
      KAFKA_NODE_ID: 1

      # Unique Kafka cluster identifier (required for KRaft mode)
      KAFKA_CLUSTER_ID: 'energy-tracker-cluster-1'

      # Node roles — this instance acts as BOTH a broker and a controller
      KAFKA_PROCESS_ROLES: 'broker,controller'

      ############################################
      # Controller Quorum Configuration (Raft)
      ############################################

      # List of controller nodes that form the Raft quorum
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:9093'

      # Listener name used by controllers for internal Raft communication
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'

      ############################################
      # Network Listeners (Internal, External & Controller)
      ############################################

      # Define all listener endpoints for this Kafka node
      KAFKA_LISTENERS: >
        PLAINTEXT://0.0.0.0:9092,
        EXTERNAL://0.0.0.0:9094,
        CONTROLLER://0.0.0.0:9093

      # What each listener advertises to connecting clients
      # Internal (Docker containers): kafka:9092
      # External (host machine): localhost:9094
      KAFKA_ADVERTISED_LISTENERS: >
        PLAINTEXT://kafka:9092,
        EXTERNAL://localhost:9094

      # Maps listener names to their security protocols
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >
        CONTROLLER:PLAINTEXT,
        PLAINTEXT:PLAINTEXT,
        EXTERNAL:PLAINTEXT

      # Which listener brokers use for internal broker-to-broker communication
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'

      ############################################
      # Single-Node Cluster Safety Settings
      # (replication factor > 1 would fail on a single node)
      ############################################
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

      ############################################
      # Broker Defaults & Quality-of-Life Settings
      ############################################

      # Speed up consumer group startup
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

      # Auto-create topics if they don't already exist
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

      # Default number of partitions per topic
      KAFKA_NUM_PARTITIONS: 1

    networks:
      - base-network

  debezium-connect:
    build:
      context: ./connect
    container_name: debezium-connect
    ports:
      - "8083:8083"
    environment:
      - BOOTSTRAP_SERVERS=kafka:9092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_connect_statuses
      - CONFIG_STORAGE_REPLICATION_FACTOR=1
      - OFFSET_STORAGE_REPLICATION_FACTOR=1
      - STATUS_STORAGE_REPLICATION_FACTOR=1
      - REST_ADVERTISED_HOST_NAME=debezium-connect
      - KEY_CONVERTER=org.apache.kafka.connect.storage.StringConverter
      - VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
      - VALUE_CONVERTER_SCHEMAS_ENABLE=false
      - PLUGIN_PATH=/kafka/connect,/usr/share/java,/usr/share/confluent-hub-components
    depends_on:
      - kafka
    networks:
      - base-network

  debezium-ui:
    container_name: debezium-ui
    image: debezium/debezium-ui
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CONNECT_URIS=http://debezium-connect:8083
    depends_on:
      - debezium-connect
    networks:
      - base-network

networks:
  base-network:
    external: true

Service summary:

Service	Role
`debezium-connect`	Hosts the Kafka Connect worker. Connectors registered here read the PostgreSQL WAL and publish change events to Kafka topics.
`debezium-ui`	A web UI that simplifies connector management by wrapping the Debezium Connect REST API.
`kafka`	The message broker that durably stores change event streams. Runs in KRaft mode (no Zookeeper required).
`kafka-ui`	A web UI for browsing Kafka topics, messages, and consumer groups — invaluable for debugging.

Step 4 – Start the Services

docker compose up -d --build

Once the containers are healthy:

Debezium UI → http://localhost:8080
Kafka UI → http://localhost:8070

Step 5 – Configure the PostgreSQL Connector via Debezium UI

Before creating the connector, make sure you have a users table in your PostgreSQL database.

Open Debezium UI at http://localhost:8080 and follow the steps shown in the screenshots below to create a new PostgreSQL source connector:

Click Create Connector

Choose the PostgreSQL connector type

Fill in the basic connection settings (hostname, port, database name, credentials)

Configure advanced settings (table filter, plugin name: pgoutput)

Validate the configuration and finish

Confirm the connector is running

Now try inserting a record into the users table and open Kafka UI at http://localhost:8070 to see the change event appear on the Kafka topic:

Setup: Synchronization from PostgreSQL to Elasticsearch

With Debezium already streaming changes to Kafka, adding Elasticsearch as a sync target is straightforward — we just need to:

Start an Elasticsearch (+ Kibana) stack
Register a sink connector that reads from the Kafka topic and writes to an Elasticsearch index

Step 1 – Start Elasticsearch and Kibana

Create a separate docker-compose-elk.yml file:

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.15
    container_name: elasticsearch
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
    networks:
      - base-network

  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.15
    container_name: kibana
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - base-network

networks:
  base-network:
    external: true

Start the stack:

docker compose -f docker-compose-elk.yml up -d --build

Elasticsearch → http://localhost:9200
Kibana → http://localhost:5601

Step 2 – Register the Connectors

Note: Make sure the users table exists in your PostgreSQL database before registering the connectors.

Use the following curl commands to register both connectors via the Debezium Connect REST API.

Elasticsearch Sink Connector — reads the users Kafka topic and writes to the users Elasticsearch index:

curl --location 'http://localhost:8083/connectors' \
--header 'Content-Type: application/json' \
--data '{
  "name": "elasticsearch-sink-connector",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "topics": "users",
    "connection.url": "http://elasticsearch:9200",

    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",

    "schema.ignore": "true",
    "key.ignore": "false",

    "write.method": "upsert",
    "behavior.on.null.values": "delete",

    "transforms": "extractKey,removeDeleted",
    "transforms.extractKey.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
    "transforms.extractKey.field": "id",

    "transforms.removeDeleted.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
    "transforms.removeDeleted.blacklist": "__deleted",

    "errors.tolerance": "all",
    "errors.deadletterqueue.topic.name": "dlq-users",
    "errors.deadletterqueue.topic.replication.factor": "1"
  }
}'

PostgreSQL Source Connector — reads WAL changes from PostgreSQL and publishes them to the users Kafka topic:

curl --location 'http://localhost:8083/connectors' \
--header 'Content-Type: application/json' \
--data '{
  "name": "postgres-source-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgresdb",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname": "template",
    "topic.prefix": "users_table",
    "table.include.list": "public.users",
    "plugin.name": "pgoutput",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "transforms": "route,unwrap",
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "users_table.public.users",
    "transforms.route.replacement": "users",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.delete.handling.mode": "rewrite"
  }
}'

Step 3 – Verify the Pipeline

After registering both connectors, check their status in Debezium UI and verify that a consumer group has been created in Kafka UI:

Now run some INSERT, UPDATE, or DELETE statements on the users table in PostgreSQL and follow the data through the pipeline:

1. Run a search in Kibana DevTools before inserting any data:

2. Insert a new record into the users table:

3. Check the users topic in Kafka UI — the change event should appear:

4. Query the users index in Elasticsearch via Kibana DevTools — the new document should be there:

That's the full pipeline working end-to-end: PostgreSQL → Debezium → Kafka → Elasticsearch. 🎉

Reference

Repository: Setup CDC

Wrapping Up

You've just set up a fully functional CDC pipeline with Debezium! From a single row update in PostgreSQL, a change event flows through Kafka and lands in Elasticsearch — all automatically, with zero application code involved.

This is just the beginning. From here you can explore:

Adding more sink connectors (e.g., a data warehouse, another database)
Schema evolution strategies using a schema registry
Monitoring connector health and lag in production

Thank you so much for following along! I hope this guide saved you hours of trial and error. If you haven't read it yet, the Overview CDC post provides great background context on the concepts behind what we built here. Happy streaming! 🚀

Subtitle to Audio Tutorial

longtk26 — Sun, 18 Jan 2026 16:17:50 +0000

Subtitle to Audio Tutorial

Hi everyone! In this tutorial, we will explore how to convert subtitles into audio using text-to-speech (TTS) technology. This process can be particularly useful for creating audiobooks, podcasts, or enhancing accessibility for video content.

Subtitle to Audio Tutorial
- Table of Contents
- Introduction
- Workflow Overview
- Problems with workflow
  - 1. Limited execution time of lambda functions
  - 2. Limited resources within lambda functions
  - 3. Synchronous time and speed between subtile and audio
  - 4. Popping noises between audio segments
- Solutions
  - Algorithm diagram
  - Key Optimizations
- Conclusion

Introduction

When working with applications that involve video content and translation, it is often necessary to translate an original subtitle file (e.g., SRT format) into another language.
To enhance user experience, we can convert these translated subtitles into audio using TTS technology.
This tutorial will guide you through the process of converting subtitles to audio, addressing common challenges and providing solutions.

Workflow Overview

Convert from subtitle to audio is a job that need several steps to complete that why we will handle it asynchronously.
We choose lamba functions for saving cost and easy in implementation.
The workflow consist of the following steps:
1. Choose a subtitle file from current system: Assume our system already show list of subtitle files that user can choose from.
2. Choose voice settings: User can choose voice settings like male or female voice.
3. Request to convert subtitle to audio: After user choose subtitle file and voice settings we will send request to backend to start the conversion process.
4. Server request to s3 to get subtitle file: Because in real project we usually store files in s3, so our server need to get subtitle file url from s3 first.
5. Server request to lambda function to convert subtitle to audio: After get subtitle file url server will send request to lambda function to start convert subtitle to audio.
6. Lambda function process subtitle file: Lambda function will download subtitle file from s3, parse subtitle file to get text and timing information, then use google text to speech to convert text to audio, finally combine audio segments according to timing information and save the final audio file to s3.

Problems with workflow

1. Limited execution time of lambda functions

AWS Lambda functions have a maximum execution time limit (currently 15 minutes). If the subtitle file is large or the conversion process takes too long, the lambda function may time out before completing the task.

2. Limited resources within lambda functions

Lambda functions have limited CPU and memory resources. If the TTS engine requires more resources or inprogress the size of audio become to large, the conversion process may fail or produce error out of memory.

3. Synchronous time and speed between subtile and audio

Ensuring that the audio segments align perfectly with the timing information in the subtitle file can be challenging. Any discrepancies can lead to audio that is out of sync with the original video content.

4. Popping noises between audio segments

When combining multiple audio segments generated from subtitles, popping noises may occur at the boundaries between segments.

Solutions

This section will show you solutions for the problems mentioned above and successully in converting subtitle file more than 30 minutes to audio only in some minutes. Also implement it with at least resources, we only use default lambda function settings (128MB memory) and timeout is 15 minutes.

Algorithm diagram

Link algorithm diagram: Image

The flowchart above illustrates the complete subtitle-to-audio conversion algorithm with optimized memory management and streaming architecture. Here's how it works:

1. Initialization Phase

The process begins by downloading and parsing the SRT subtitle file from the provided URL
It pre-calculates the total PCM (Pulse Code Modulation) audio size based on subtitle duration and timing information
An S3 Multipart Upload is initialized to enable streaming upload without holding the entire file in memory
A WAV header is created and added to the part buffer, initializing tracking variables (part_number=1, content_id index=0, current_part=b"")

2. Batch Processing Loop

Instead of processing all subtitles at once, the algorithm processes them in batches of approximately 15 subtitles
For each batch, it retrieves the next set of subtitle entries to process
This batching approach prevents memory overflow by keeping only a small subset of subtitles in memory at any time

3. Audio Generation for Each Subtitle

For each subtitle in the current batch:
- Calculate speech speed: Determines the appropriate speaking rate to fit the subtitle text within its designated time window
- Generate audio using Google Cloud TTS: Converts the subtitle text to audio using the selected voice settings
- Apply fade effects: Adds 250ms fade-in and fade-out to prevent popping noises between audio segments
- Calculate and add silence padding: Adds appropriate silence gaps to ensure precise alignment with subtitle timing

4. Streaming Upload Mechanism

As audio segments are generated, PCM data is appended to the part buffer
When the buffer reaches 5MB or when processing the last batch, the current part is uploaded to S3 immediately
After each upload, the buffer is cleared and the part_number is incremented
This streaming approach ensures that Lambda memory usage remains constant (around 5-10MB) regardless of the total audio file size

5. Finalization Phase

Once all subtitle batches have been processed and all parts uploaded, the algorithm completes the S3 Multipart Upload
The final audio file is assembled from all uploaded parts and becomes available on S3
Error handling paths are included to manage upload failures or conversion issues (shown by the red error node)

Key Flow Control

The algorithm uses two main decision loops:
- "More batches to process?": Controls the outer loop for batch processing
- "More subtitles in batch?": Controls the inner loop for processing individual subtitles within a batch
"Buffer size >= 5MB OR last batch?": Triggers S3 part uploads at optimal intervals

Key Optimizations

Batch Processing: Subtitles are processed in batches (default: 15 subtitles per batch) instead of loading all at once. This prevents memory overflow in Lambda.
S3 Multipart Upload:
- Initialize multipart upload at the start
- Upload parts incrementally when buffer reaches 5MB
- Each part is uploaded immediately and buffer is cleared
- Complete multipart upload at the end
- This approach ensures Lambda never holds the entire audio file in memory
Streaming Architecture: Audio data flows directly from TTS → batch buffer → S3 parts, with minimal memory footprint at any given time.
Fade Effects: 250ms fade-in/out applied to each audio segment to eliminate popping noises between segments.
Memory Management: After each part upload, the buffer is reset, keeping memory usage constant regardless of total audio length.
Avoid FFmpeg and audio processing libraries:

We should avoid using FFmpeg (both the binary and Python wrappers like pydub, ffmpeg-python) in Lambda functions because they significantly increase deployment package size.
FFmpeg binaries are typically 50-100MB, and audio processing libraries add another 20-50MB, easily exceeding Lambda's package size limits.
Instead, we should work directly with WAV file format using Python's built-in struct module and pure Python code to manipulate raw PCM audio bytes.
This approach keeps the deployment package minimal (<15MB) while maintaining full control over audio processing operations like fade-in/fade-out, silence padding, and concatenation.

Conclusion

Thank you for following this tutorial on converting subtitles to audio using TTS technology.
I hope this information can help you to implement this feature in your own projects efficiently. You could use these ideas and use any AI tools for generating code to implement it.
This blog is written by my experience in real project that we need to save cost with lambda function, if you have any good ideas or suggestions please feel free to share with me!
You could find more on my blogs: Website

Understanding the Factory Method Pattern

longtk26 — Sat, 04 Jan 2025 15:18:34 +0000

Introduction

Hi everyone, I am writing this post to share my knowledge as I continue learning about design patterns. Today, I will present the Factory Method Pattern, which is a design pattern commonly used in real-world applications. If there are any mistakes in my post, please feel free to comment below, and I will gladly fix and update it.

Factory method pattern provides an interface for creating objects in a superclass, but allow subclasses to alter the type of objects that will be created.

Problem

Assume you have a bank application, and you’re building a feature for transferring money through various methods like bank transfer, paypal transfer,…

Before using the Factory Method pattern, let’s examine the scenario without it.

I will give an example implemented in Java.

Situation: Person1 sends money to Person2 using a transfer method (Bank Transfer or PayPal Transfer).

Folder structure:

problem/
├─ BankApp.java
├─ service/
│  ├─ PaypalTransferPayment.java
│  ├─ BankTransferPayment.java
├─ data/
│  ├─ Person.java

In the main application, create two persons with default amounts of money.

package problem;

import problem.data.Person;

public class BankApp {
    public static void main(String[] args) {
        Person person1 = new Person("John", 1000);
        Person person2 = new Person("Jane", 500);
    }
}

Create BankTransferPayment and PaypalTransferPayment classes.

package problem.service;

import problem.data.Person;

public class BankTransferPayment {
    public void processPayment(Person fromAccount, Person toAccount, float amount) {
        fromAccount.withdraw(amount);
        toAccount.deposit(amount);
        System.out.println("Bank transfer payment success.");
    }
}

package problem.service;

import problem.data.Person;

public class PaypalPayment {
    public void processPayment(Person fromAccount, Person toAccount, float amount) {
        fromAccount.withdraw(amount);
        toAccount.deposit(amount);
        System.out.println("Paypal transfer payment success.");
    }
}

Implement the logic in the main function.

package problem;

import problem.data.Person;
import problem.service.BankTransferPayment;
import problem.service.PaypalPayment;

public class BankApp {
    public static void main(String[] args) {
        Person person1 = new Person("John", 1000);
        Person person2 = new Person("Jane", 500);

        String paymentMethod = "BANK_TRANSFER";

        if (paymentMethod.equals("BANK_TRANSFER")) {
            BankTransferPayment bankTransferPayment = new BankTransferPayment();
            bankTransferPayment.processPayment(person1, person2, 100);

            System.out.println("===Method bank_transfer===");
            System.out.println(person1.getName() + " has " + person1.getAmount());
            System.out.println(person2.getName() + " has " + person2.getAmount());
        } else if (paymentMethod.equals("PAYPAL")) {
            PaypalPayment paypalPayment = new PaypalPayment();
            paypalPayment.processPayment(person1, person2, 100);

            System.out.println("===Method paypal===");
            System.out.println(person1.getName() + " has " + person1.getAmount());
            System.out.println(person2.getName() + " has " + person2.getAmount());
        }
    }
}

Problems with the current implementation:

Repetitive code: The processPayment method logic is repeated for every payment method.
Tightly coupled code: The application needs to create the payment method objects itself, making it hard to extend the application.
Scalability issues: If new payment methods are added, the source code becomes more complex and harder to maintain.

Solution

The solution to the above situation is to use factory method pattern. So, how do we apply it ?

In the example above:

Each if-else block calls the processPayment method, which leads to repetitive code.
Objects are created based on the payment type condition, making the code messy with excessive if-else statements.

To solve these issues, the Factory Method pattern will be implemented step by step.

Folder structure (solution):

solution/
├─ BankApp.java
├─ service/
│  ├─ payments/
│  │  ├─ Payment.java
│  │  ├─ PaymentFactory.java
│  │  ├─ BankTransferPayment.java
│  │  ├─ PaypalTransferPayment.java
├─ data/
│  ├─ Person.java

Step 1: Create Payment interface, declares common method processPayment

package solution.service.payments;

import solution.data.Person;

// Step 1: Create an interface for the payment
public interface Payment {
    void processPayment(Person fromAccount, Person toAccount,float amount);
}

Step 2: Create BankTransferPayment and PaypalTransferPayment classes implement Payment interface.

package solution.service.payments;

import solution.data.Person;

// Step 2: Create a class that implements the Payment interface
public class BankTransferPayment implements Payment {
    @Override
    public void processPayment(Person fromAccount, Person toAccount, float amount) {
        fromAccount.withdraw(amount);
        toAccount.deposit(amount);
        System.out.println("Bank transfer payment success.");
    }
}

package solution.service.payments;

import solution.data.Person;

public class PaypalPayment implements Payment{
    @Override
    public void processPayment(Person fromAccount, Person toAccount, float amount) {
        fromAccount.withdraw(amount);
        toAccount.deposit(amount);
        System.out.println("Paypal transfer payment success.");
    }
}

Step 3: Create PaymentFactory class. This class is responsible for creating objects based on payment type condition.

package solution.service.payments;

public class PaymentFactory {
    public Payment createPayment(String paymentType) {
        if (paymentType == null) {
            return null;
        }
        if (paymentType.equalsIgnoreCase("BANK_TRANSFER")) {
            return new BankTransferPayment();
        } else if (paymentType.equalsIgnoreCase("PAYPAL")) {
            return new PaypalPayment();
        }
        return null;
    }
}

Step 4: Use the Factory in the Main Application.

Modify the main function to use the Factory Method pattern.

package solution;

import solution.data.Person;
import solution.service.payments.Payment;
import solution.service.payments.PaymentFactory;

public class BankApp {
    public static void main(String[] args) {
        Person person1 = new Person("John", 1000);
        Person person2 = new Person("Jane", 500);

        String paymentMethod = "PAYPAL";

        Payment payment = new PaymentFactory().createPayment(paymentMethod);
        payment.processPayment(person1, person2, 100);
    }
}

Benefits of Using the Factory Method Pattern

The code is cleaner and more structured.
Repetitive calls to processPayment in multiple if-else blocks are eliminated.
Object creation is delegated to the factory, improving maintainability.

Bonus

To make the PaymentFactory class comply with the Open/Closed Principle (from SOLID principles), you can implement a dynamic registration mechanism using the Strategy Pattern.

Updated PaymentFactory.java:

package solution.service.payments;

import java.util.HashMap;
import java.util.Map;

public class PaymentFactory {
    private Map<String, Payment> paymentMaps;

    public PaymentFactory() {
        this.paymentMaps = new HashMap<>();
    }

    public Payment createPayment(String paymentType) {
        return paymentMaps.get(paymentType);
    }

    public void registerPayment(String paymentType, Payment payment) {
        paymentMaps.put(paymentType, payment);
    }

    public PaymentFactory initializePaymentMethods() {
        Payment bankTransferPayment = new BankTransferPayment();
        Payment paypalPayment = new PaypalPayment();

        this.registerPayment("BANK_TRANSFER", bankTransferPayment);
        this.registerPayment("PAYPAL", paypalPayment);

        return this;
    }
}

Using the Updated Factory in the Main Application.

package solution;

import solution.data.Person;
import solution.service.payments.Payment;
import solution.service.payments.PaymentFactory;

public class BankApp {
    public static void main(String[] args) {
        Person person1 = new Person("John", 1000);
        Person person2 = new Person("Jane", 500);

        String paymentMethod = "BANK_TRANSFER";

        Payment payment = new PaymentFactory()
                .initializePaymentMethods()
                .createPayment(paymentMethod);
        payment.processPayment(person1, person2, 100);
    }
}

By applying this approach, the code adheres to the Open/Closed Principle, enabling the addition of new payment methods without modifying the PaymentFactory logic.

I hope this post will be helpful to you.

References:

guru-design-patterns

DEV Community: longtk26

Kubernetes Architecture and Primitives: From Containers to Orchestration

Table of Contents

1. Why Kubernetes? From Manual Docker to Automated Orchestration

The Operational Limits of Standalone Docker

Declarative Reconciliation vs. Imperative Management

2. Cluster Architecture: Control Plane and Worker Nodes

The Control Plane: The Cluster Brain

Worker Nodes: The Execution Fleet

3. The Workload Hierarchy: Pods, ReplicaSets, and Deployments

1. The Pod: The Atomic Unit of Scheduling

2. ReplicaSets: Declarative Self-Healing and Scale

3. Deployments: Declarative Rolling Updates and Rollbacks

4. Service Networking: Stable Virtual IPs and Traffic Distribution

The Service Primitive and EndpointSlices

How kube-proxy Routes Traffic

Service Types and Exposure Scope

5. Health Probes: Startup, Liveness, and Readiness

The Three Probe Types

1. Startup Probes: Guarding Slow Initialization

2. Liveness Probes: Self-Healing Deadlocks

3. Readiness Probes: Dynamic Traffic Gating

Probe Configuration and the Cascading Restart Trap

6. Auxiliary Primitives and the End-to-End Request Lifecycle

Supporting Primitives: Configuration, Isolation, and Quotas

Ingress and the End-to-End Request Path

7. Summary & What's Next

Core Primitives Summary

What's Next: Hands-On Practice

References

Step-by-Step Guide: Fine-Tuning an IELTS Examiner LLM with Unsloth, QLoRA

Table of Contents

1. The Tool Stack: Why Unsloth + QLoRA?

Step 1: Dataset Engineering for IELTS Evaluation

ChatML Data Schema (sample_ielts_dataset.jsonl)

Step 2: Colab Environment Setup & GPU Check

Installing Pre-built Wheels (Avoiding Source Compilation)

Step 3: Loading the 4-bit Base Model

Step 4: Attaching LoRA Adapters

Step 5: Formatting Prompts & Chat Templates

Step 6: Configuring SFTTrainer & Training

Step 7: Qualitative Inference & Testing

Step 8: Exporting to GGUF & Serving with Ollama

1. Save Adapter & Export 4-bit GGUF in Colab

2. Backup to Google Drive

3. Run Locally with Ollama

10. Troubleshooting

Conclusion & Resources

LLM Fine-Tuning: Concepts, Architecture, and Fine-Tuning vs. RAG

Table of Contents

1. What is Fine-Tuning?

The LLM Training Lifecycle

2. Why Fine-Tune? Core Motivations

3. Fine-Tuning vs. RAG

Core Mental Model: Open-Book vs. Closed-Book Exam

Detailed Comparison

The Hybrid Approach: RAG + Fine-Tuning

Decision Framework: Which Should You Choose?

4. How an LLM Processes Input and Generates Output (Under the Hood)

The End-to-End Inference Lifecycle

Architectural Flow Breakdown:

The Tokenization Mechanism: From Words to Numbers

Why Not Words or Characters?

The Modern Solution: Subword Tokenization (Byte-Pair Encoding - BPE)

Example: Tokenizing an IELTS Prompt

Special Tokens in ChatML Templates

Internal Processing: Embeddings, Positional Encoding & Attention

The Prediction Loop: Logits, Softmax & Autoregressive Decoding

1. From Hidden States to Logits

2. From Logits to Probabilities (Softmax)

Sampling Controls: Temperature, Top-p & Stop Tokens

1. Temperature (T)

2. Top-p (Nucleus Sampling)

3. Stop Tokens (<|im_end|>)

5. Methods of Fine-Tuning

Full Parameter Fine-Tuning (FPFT)

Parameter-Efficient Fine-Tuning (PEFT) & LoRA

QLoRA: 4-bit Quantized LoRA

Hardware Comparison: FPFT vs. LoRA vs. QLoRA

6. Essential Hyperparameters & Micro-Concepts

ChatML Data Schema (`sample_ielts_dataset.jsonl`)

1. Temperature (`T`)

3. Stop Tokens (`<|im_end|>`)