DEV Community: ZoomApps

Building a WhatsApp AI Chatbot That Hands Off to a Human: Architecture and Pitfalls

ZoomApps — Mon, 02 Nov 2026 06:01:21 +0000

Getting a language model to answer WhatsApp messages takes an afternoon. Getting it to stop answering at the right moment, pass the conversation to a person with enough context, and never talk over that person afterwards takes considerably longer. Most of the bugs we have seen in production chatbots are not about answer quality. They are about state: who owns the conversation right now, which messages have already been processed, and what happens when the phone, the model or the manager goes quiet.

This post walks through an architecture that holds up under real traffic, with the parts that usually break.

The shape of the system

At a high level there are five components:

Transport. Receives incoming messages and sends replies. This is either the official WhatsApp Business Platform with webhooks, or a session linked to an existing number through Linked Devices (the same mechanism WhatsApp on a computer uses).
Inbox queue. Every incoming event goes into a durable queue before anything else touches it.
Conversation store. One record per chat: owner (bot or human), recent history, flags, timestamps.
Assistant worker. Pulls from the queue, builds the prompt from the knowledge base and the history, decides between "answer" and "hand off".
Operator side. Wherever managers actually reply: the phone itself, a shared inbox, or a CRM.

The rule that saves you later: the transport layer does nothing except normalise events and enqueue them. No model calls inside a webhook handler. Webhook senders retry on slow responses, and an LLM call that takes eight seconds will happily produce duplicate replies.

Normalise and deduplicate first

Whatever transport you use, convert events into one internal shape:

from dataclasses import dataclass

@dataclass
class InboundEvent:
    message_id: str      # provider's ID, used for dedup
    chat_id: str         # stable per customer
    from_me: bool        # sent from the business number
    kind: str            # "text", "voice", "image", "document"
    text: str | None
    media_ref: str | None
    ts: int              # provider timestamp, seconds

Then deduplicate on message_id before enqueueing. A Redis SET NX with a TTL of a day is enough:

def accept(event: InboundEvent) -> bool:
    key = f"seen:{event.message_id}"
    if not redis.set(key, 1, nx=True, ex=86400):
        return False          # duplicate delivery, drop it
    queue.push(event)
    return True

Webhook retries, reconnects of a linked session and your own worker restarts will all deliver the same message twice. Without dedup the customer gets two slightly different answers to one question.

Note the from_me field. On a linked number, messages the manager types on the phone arrive as events too. You need them, because they are the most reliable signal that a human has taken over (more on that below).

Debounce bursts before calling the model

People do not write in single messages. They write "hi", then "how much is", then "delivery to Almaty?" within ten seconds. If the worker answers each fragment, you get three replies, the first two of them useless.

A simple debounce: when a message arrives for a chat, schedule processing a few seconds later and reset the timer on every new message.

DEBOUNCE_SEC = 4

def on_event(event):
    store.append_pending(event.chat_id, event)
    scheduler.schedule(
        job_id=f"reply:{event.chat_id}",
        run_at=now() + DEBOUNCE_SEC,
        replace_existing=True,
    )

When the job fires, it takes all pending messages for that chat as one turn. Three to five seconds is the usual sweet spot: shorter and you still split sentences, longer and the bot feels sluggish. Voice notes need special handling here, because transcription adds its own delay; start the debounce timer from the moment the transcript is ready, not from the moment the audio arrived.

Voice notes are text with a confidence problem

A large share of WhatsApp traffic is voice. The pipeline is straightforward: download the media, transcribe it, treat the transcript as a text message. The pitfall is noise. A voice note recorded on a busy street produces a transcript that is grammatically fine and semantically wrong, and the model will confidently answer the wrong question.

Two mitigations that work:

Keep the per-segment confidence or average log-probability from the speech model and, below a threshold, have the assistant ask the customer to clarify instead of answering.
Store the original audio reference with the transcript, so a manager who takes over can listen to it rather than trust the text.

Conversation ownership is a state machine

This is the core of the handoff design. Each chat has exactly one owner at a time:

            customer asks for human / trigger fires
   BOT  ───────────────────────────────────────────▶  PENDING_HUMAN
    ▲                                                     │
    │                                     manager replies │
    │                                                     ▼
    └──────────── idle timeout or manual release ─────  HUMAN

BOT: the assistant answers.
PENDING_HUMAN: the assistant has told the customer a person will reply, and stays silent. Managers are notified.
HUMAN: a manager is active. The assistant is fully muted for this chat.

Store it explicitly and check it at the last possible moment, right before sending, not only when the job starts:

def send_reply(chat_id, text, generated_at_version):
    conv = store.get(chat_id)
    if conv.owner != "BOT":
        return  # a human took over while the model was thinking
    if conv.version != generated_at_version:
        return  # new messages arrived, this answer is stale
    transport.send(chat_id, text)

The version check matters more than it looks. A model call can take several seconds; during that time the customer may add a message or a manager may jump in. Sending a reply generated from an outdated state is the single most common source of "the bot interrupted me" complaints from managers.

What triggers a handoff

Hard-coding "if the message contains 'operator'" catches only a fraction of cases. A better approach is to let the model return a structured decision alongside the answer:

{
  "action": "handoff",
  "reason": "custom_terms",
  "summary": "Wants 40 units with installation in Shymkent, asks for a discount on volume.",
  "reply_to_customer": "Thanks, I'm passing this to a manager who can confirm volume terms."
}

Typical handoff reasons worth defining up front:

the customer explicitly asks for a person;
the customer is ready to buy and the next step needs a human (payment, contract, scheduling);
non-standard terms: discounts outside the price list, unusual quantities, custom deadlines;
complaints and negative sentiment;
the question is outside the knowledge base after one clarification attempt.

Make the list configurable per business rather than baked into the prompt. A clinic and a furniture shop draw the line in very different places.

Equally important are hard boundaries that never depend on the model's judgement. The assistant should not promise discounts beyond the price list, quote deadlines that depend on workload, or give professional conclusions. Encode these both in the system prompt and as post-generation checks: a regex for percentages and currency amounts that do not appear in the knowledge base catches a surprising number of invented prices.

Detecting that a human has taken over

If managers reply from a CRM or a shared inbox, you know when they send something. If they reply from the phone itself, which is common with small teams, you only know through from_me events from the transport.

The rule: any from_me message that was not sent by the bot moves the chat to HUMAN. To tell the bot's own messages apart, record the provider message IDs of everything the bot sends:

def on_from_me(event):
    if redis.sismember(f"bot_sent:{event.chat_id}", event.message_id):
        return                      # our own echo
    store.set_owner(event.chat_id, "HUMAN", since=event.ts)

Do not rely on comparing message text; managers copy bot phrases more often than you would expect.

Giving the manager context

A handoff that dumps the customer on a person with no context forces them to ask everything again, which defeats the purpose. Send the manager:

the model's one-paragraph summary from the handoff decision;
the reason code;
the last N messages verbatim, including transcripts of voice notes;
anything already collected: name, city, items and quantities mentioned.

The summary is generated once at handoff time and stored, so it stays consistent if several managers look at the chat.

Releasing the chat back to the bot

The forgotten transition is HUMAN back to BOT. If you never release, the bot slowly goes silent across your whole customer base. If you release too eagerly, the bot answers in the middle of a human negotiation.

What works in practice:

An idle timeout measured from the last manager message, not the last customer message. Something between 2 and 12 hours depending on the business.
A manual release command from the operator side.
On release, the bot does not proactively write anything. It simply answers the next incoming message, with the human part of the conversation included in its history so it does not contradict what the manager promised.

That last point is easy to miss. If the manager agreed to a delivery date, the bot needs to see that agreement, or it will cheerfully quote the standard terms on the next question.

Knowledge base instead of a decision tree

The reason menu-driven bots age badly is maintenance: every new service or price change means editing the tree. With an LLM, the price list and service descriptions become the knowledge base, and a new line in the price list is available immediately.

For a typical small business catalogue (a few hundred items) you often do not need vector search at all. The whole price list fits in the context window, and putting it there directly removes a class of retrieval failures where the right item simply is not fetched. Move to retrieval when the catalogue grows past what fits comfortably, and even then keep exact-match lookup for SKUs and product names alongside embeddings.

Feed corrections back from real conversations. Log every handoff with reason outside_knowledge_base; reviewing that list weekly is the cheapest way to improve answers.

Operational pitfalls

The linked phone goes offline. With Linked Devices the link runs through the phone. If it loses connectivity, the session stops receiving messages. Monitor last-event time per number and alert when it exceeds a sensible window during working hours. When the link comes back, the replayed backlog hits your dedup layer, which is another reason that layer has to exist.

Bulk messaging gets numbers banned. Answering inbound messages on a business number is normal usage. Sending to cold lists is what triggers restrictions. Keep the bot strictly reactive.

Late answers. If processing fails and recovers an hour later, do not answer an hour-old question as if nothing happened. Drop or rephrase replies to messages older than a cutoff, and route those chats to a human instead.

Several numbers. Branches often have different prices. Key the conversation store and the knowledge base by number, and allow a shared base with per-number overrides.

Logging and privacy. Decide retention up front and mask phone numbers in application logs.

A checklist before going live

Dedup on provider message ID before the queue
Debounce of 3 to 5 seconds per chat, started after transcription for voice
Explicit owner state per chat, checked again right before sending
Version check to drop stale model answers
Structured handoff decision with reason and summary
from_me detection with a record of bot-sent IDs
Idle-based release measured from the last manager message
Hard checks against invented prices and deadlines
Alert on silent numbers, cutoff for late answers
A sandbox chat where answers are reviewed before the first real customer

None of these pieces is complicated on its own. The difficulty is that skipping any one of them produces failures that look like "the AI said something weird", when the real cause is a race between a model call and a manager's thumb. If you would rather start with a setup that already handles connection, voice notes and handoff, and test it on your own price list first, take a look at our WhatsApp AI chatbot that works on your existing number.

Bitrix24 REST API in Practice: Webhooks, Batch Calls and Rate Limits

ZoomApps — Thu, 29 Oct 2026 06:02:09 +0000

The Bitrix24 REST API looks simple on day one. You create an inbound webhook, call crm.deal.list, get JSON back and the prototype works. The trouble starts on the first bulk sync, when a few thousand records need to move and the portal answers with QUERY_LIMIT_EXCEEDED, or a week later, when someone notices that deals updated over the weekend never reached the warehouse system.

This post collects what we have learned building exchanges between Bitrix24 and external systems: how the two kinds of webhooks behave, how to use batch without shooting yourself in the foot, how the rate limits actually work, and the handful of field-level quirks that cause silent data corruption.

Two kinds of webhooks, two different jobs

Bitrix24 uses the word "webhook" for two unrelated things.

Inbound webhook. A URL with a secret embedded in it that lets your code call REST methods on behalf of a specific user:

https://yourcompany.bitrix24.kz/rest/1/x7k2abc9def0ghij/crm.deal.get.json?id=42

The 1 is the user ID, the next segment is the secret. Everything this URL does runs with that user's permissions and is limited to the scopes you ticked when creating it (crm, user, task and so on). Two practical consequences:

Create a dedicated technical user for integrations. If the webhook belongs to a sales manager who later leaves the company and gets deactivated, the integration dies with their account.
Grant the minimum scopes. The URL is a bearer credential. It ends up in logs, environment files and occasionally in screenshots sent to a chat.

Outbound webhook (event handler). Bitrix24 sends a POST to your URL when something happens: ONCRMDEALADD, ONCRMDEALUPDATE, ONCRMCONTACTDELETE and many more. The body is form-encoded, not JSON, and looks roughly like this once parsed:

event=ONCRMDEALUPDATE
data[FIELDS][ID]=1287
ts=1761712345
auth[domain]=yourcompany.bitrix24.kz
auth[application_token]=k4q9...

Notice what is missing: the deal itself. You get an ID and nothing else. Your handler has to call crm.deal.get to find out what changed, which means every event costs at least one API request.

A minimal handler in Python (Flask) that does the right things:

import os
from flask import Flask, request, abort
from queue_client import enqueue  # Redis, RabbitMQ, a DB table, anything durable

app = Flask(__name__)
APP_TOKEN = os.environ["B24_APP_TOKEN"]

@app.post("/b24/events")
def b24_event():
    form = request.form
    if form.get("auth[application_token]") != APP_TOKEN:
        abort(403)

    event = form.get("event")
    entity_id = form.get("data[FIELDS][ID]")
    if not event or not entity_id:
        abort(400)

    enqueue({"event": event, "id": int(entity_id), "ts": form.get("ts")})
    return "", 200

Three rules are baked in here:

Check application_token. The handler URL is public; without the check anyone can push fake events.
Do not call the API inside the handler. Acknowledge fast and queue the work. A slow or failing handler loses events, and Bitrix24 keeps no replay log for you.
Store only the ID. By the time the worker runs, the deal may have changed again, and you want the latest state anyway.

Events are a hint, not a guarantee

Treat outbound events as "something probably changed, go look". Deliveries get lost during your downtime, network blips or a deploy that broke the route. Two mitigations we use together:

Offline events. For applications, Bitrix24 can store events on its side; you pull them with event.offline.get and confirm processing. Push becomes pull and survives your downtime.
A periodic reconciliation pass. Every N minutes, request entities with DATE_MODIFY after your last checkpoint and run them through the same worker. Cheap, and it catches whatever the event stream dropped.

The echo loop

If your integration writes back to the same entity it listens to, you will create a loop: you update deal 1287, Bitrix24 fires ONCRMDEALUPDATE, your worker fetches it, sees a "change", pushes it to the external system, which pushes a status back, which updates the deal again.

Break it explicitly. Options that work:

Compare a hash of the fields you care about with the last hash you synced. If nothing relevant changed, stop.
Check MODIFY_BY_ID (or updatedBy in crm.item.*) and ignore changes made by your technical integration user.
Write a sync marker into a custom field, but only if the business is fine with an extra field on the card.

The first option is the most robust, because it also filters out updates that touched fields you do not sync.

batch: up to 50 calls in one request

batch takes a set of named commands and executes them on the server in one HTTP round trip. The limit is 50 commands per batch.

import requests

WEBHOOK = "https://yourcompany.bitrix24.kz/rest/1/x7k2abc9def0ghij/"

def call(method, params=None):
    r = requests.post(WEBHOOK + method + ".json", json=params or {}, timeout=30)
    data = r.json()  # read the body even on 503: the error code says which limit you hit
    if "error" in data:
        raise RuntimeError(f"{data['error']}: {data.get('error_description')}")
    return data

cmd = {
    f"d{deal_id}": f"crm.deal.get?id={deal_id}"
    for deal_id in [101, 102, 103]
}
resp = call("batch", {"halt": 0, "cmd": cmd})
results = resp["result"]["result"]        # successful commands
errors = resp["result"]["result_error"]   # failed commands, keyed the same way

Things to know before relying on it:

Check result_error every time. With halt: 0 the batch continues past failures, and the top level of the response looks successful. A batch of 50 updates where 7 failed returns HTTP 200. If you only read result, those 7 are lost silently.

halt: 1 is not a transaction. It stops at the first error, but the commands that already ran are not rolled back. There are no transactions in the REST API. Design updates so that re-running them is safe.

Commands are query strings. Each value in cmd is a URL-encoded string, not JSON. Hand-built strings break on &, = or Cyrillic. Use an encoder for PHP-style nested arrays (fields[TITLE]=..., filter[>ID]=...); plain urlencode on a nested dict gets it wrong.

Results can feed later commands. You can reference an earlier result inside the same batch:

cmd = {
    "contact": "crm.contact.add?fields[NAME]=Aigerim&fields[PHONE][0][VALUE]=%2B77001234567&fields[PHONE][0][VALUE_TYPE]=WORK",
    "deal": "crm.deal.add?fields[TITLE]=Order%20123&fields[CONTACT_ID]=$result[contact]",
}

Handy, but it makes the batch order-dependent and harder to retry. We use it for create-and-link pairs and avoid it everywhere else.

A batch is one request for the rate limiter, but not free. It counts as one hit against the request frequency limit, but every command inside still consumes server execution time, which is limited separately (see below). Fifty heavy list calls with complex filters in one batch can exhaust the time budget faster than fifty simple get calls spread over a minute.

How the rate limits actually work

There are two independent limits, and integrations usually discover them in this order.

1. Request frequency: a leaky bucket

Each portal has a request counter that behaves like a leaky bucket. Requests fill it, and it drains at a fixed rate per second. While the bucket has room, requests go through instantly. When it is full, you get HTTP 503 with error QUERY_LIMIT_EXCEEDED. The common published figures are a drain rate of 2 requests per second with a burst capacity of 50 on standard plans, and higher values on top plans and on-premise installations. Check the current numbers for your plan, they have changed over the years.

A script firing 200 requests in a tight loop gets the first 50 through, then hits errors, and a naive retry loop only adds more requests. The fix is a client-side limiter that mirrors the server's bucket, plus backoff when the server disagrees:

import time, random, threading

class LeakyBucket:
    def __init__(self, rate=2.0, capacity=50):
        self.rate, self.capacity = rate, capacity
        self.level, self.last = 0.0, time.monotonic()
        self.lock = threading.Lock()

    def acquire(self):
        while True:
            with self.lock:
                now = time.monotonic()
                self.level = max(0.0, self.level - (now - self.last) * self.rate)
                self.last = now
                if self.level + 1 <= self.capacity:
                    self.level += 1
                    return
                wait = (self.level + 1 - self.capacity) / self.rate
            time.sleep(wait)

bucket = LeakyBucket(rate=1.8, capacity=40)  # stay below the real limit

def call_limited(method, params=None, attempts=6):
    for attempt in range(attempts):
        bucket.acquire()
        try:
            return call(method, params)
        except RuntimeError as e:
            if "QUERY_LIMIT_EXCEEDED" not in str(e) and "OPERATION_TIME_LIMIT" not in str(e):
                raise
        time.sleep(min(60, 2 ** attempt) + random.random())
    raise RuntimeError(f"{method}: rate limited after {attempts} attempts")

Two details matter. The limiter runs below the real limit, because telephony, marketplace apps and other scripts share the same bucket. And it must be shared across workers: four processes with their own in-memory bucket send four times the allowed rate. Beyond one process, keep the counter in Redis.

2. Execution time per method

Newer versions of the platform also limit how much server time a single method may consume over a sliding window. Responses carry a time block, with fields such as operating and operating_reset_at telling you how much time the method has used. When a method exceeds its budget, it is blocked for a while with OPERATION_TIME_LIMIT, while other methods keep working.

This bites bulk exports. crm.deal.list filtering on a custom field with a large select is expensive, and a loop of them eats the budget. Log the time block during test runs to see which calls are heavy before production does.

Pagination that does not get slower on every page

List methods return 50 records per page with next and total. The obvious loop passes start=0, 50, 100.... On large tables this degrades, because the server still counts and offsets through everything before your page, and it also computes total on every call.

The faster pattern: sort by ID, filter by >ID from the last record you saw, and pass start=-1 to turn off the count.

def iter_deals(select):
    last_id = 0
    while True:
        resp = call_limited("crm.deal.list", {
            "order": {"ID": "ASC"},
            "filter": {">ID": last_id},
            "select": select,
            "start": -1,
        })
        rows = resp["result"]
        if not rows:
            return
        yield from rows
        last_id = int(rows[-1]["ID"])

Each request is now an indexed range scan, and page 2,000 costs the same as page 1. On exports of hundreds of thousands of records this decides whether the job finishes overnight at all.

Always pass select. Without it you get the default field set, which excludes custom UF_CRM_* fields, and you will wonder why they are empty. With ["*", "UF_*"] you get everything, which is expensive. List the fields you need.

Field-level pitfalls

These do not throw errors. They corrupt data quietly.

Multi-fields append by default. PHONE, EMAIL, WEB and IM are arrays of {ID, VALUE, VALUE_TYPE}. If you call crm.contact.update with PHONE: [{"VALUE": "+77001234567"}], the number is added, not replaced. Run that on every sync and a contact collects fifty copies of the same phone. To change a value, pass the existing item's ID; to delete one, pass its ID with an empty VALUE.

Two naming conventions. crm.deal.* uses UPPER_SNAKE (STAGE_ID, ASSIGNED_BY_ID). The newer universal crm.item.* methods with entityTypeId (2 for deals, 3 for contacts, 4 for companies, plus IDs for smart processes) use camelCase (stageId, assignedById) and wrap results in item or items. Mixing them in one codebase leads to fields that are "set" but never saved. Pick one family per entity and stick to it.

Stage IDs depend on the pipeline. In the default pipeline a stage is NEW or WON; in additional pipelines it becomes C3:NEW, C3:WON, where 3 is the category ID. Hardcoding WON works until sales creates a second pipeline.

Dates come with a timezone offset. Values look like 2026-10-29T11:00:00+05:00. Parse them as aware datetimes. Comparing them as strings or stripping the offset produces off-by-hours bugs in checkpoints, which means missed or duplicated records in reconciliation.

List fields store IDs, not labels. A custom enum field returns "45", not "Wholesale". Load the field definitions with crm.deal.userfield.list once and map both ways.

What to log

Log method, entity ID, direction, duration, the time block, the error code and a correlation ID linking event, queue job and API call. Add one alert on the age of the oldest unprocessed job. Integrations rarely crash; they drift until the numbers stop matching a week later.

Checklist

Dedicated technical user, minimal webhook scopes
Event handler: token check, enqueue, return 200
Reconciliation pass by DATE_MODIFY to catch lost events
Echo-loop protection by field hash or author check
batch with result_error checked on every call
Shared client-side limiter below the published rate, backoff on 503
>ID pagination with start=-1 and explicit select
Multi-fields updated by ID, stage IDs resolved per pipeline
Logs with timing data and an alert on queue age

The REST API itself is the same for cloud and on-premise portals; only authorisation and access setup differ, so these patterns carry over. If you would rather hand the exchange to a team that builds queues, retries and monitoring into every project, here is how we approach Bitrix24 REST API integration with warehouse, logistics and banking systems.

Sizing VRAM for a Self-Hosted LLM: 7B to 70B on a Rented GPU Server

ZoomApps — Tue, 27 Oct 2026 18:29:47 +0000

Most "will it fit" questions about self-hosted models get answered with a rule of thumb: two bytes per parameter, add some headroom, done. That rule tells you whether the weights load. It does not tell you whether the server survives fifty concurrent users with long prompts, which is the question that actually decides what card you rent.

This post walks through the arithmetic I use before picking hardware for inference and fine-tuning, with a small Python calculator, a working vLLM launch config and the mistakes that cost the most time.

The four things that live in VRAM

During inference, GPU memory is split between:

Model weights. Fixed size, determined by parameter count and precision.
KV cache. Grows with context length and with the number of sequences served in parallel. This is the part people forget.
Activations and temporary buffers. Relatively small for inference, but spikes during prefill of long prompts.
Runtime overhead. CUDA context, cuBLAS workspaces, NCCL buffers if you use several cards. Budget roughly 1 to 2 GB per GPU.

Training adds gradients and optimizer state on top, which changes the picture completely. We will get to that.

Weights: the easy part

Bytes per parameter by precision:

Precision	Bytes/param	7B	13B	34B	70B
FP16 / BF16	2	14 GB	26 GB	68 GB	140 GB
FP8 / INT8	1	7 GB	13 GB	34 GB	70 GB
4-bit (AWQ, GPTQ)	~0.55	~4 GB	~7 GB	~19 GB	~39 GB

The 4-bit column is not exactly 0.5 because quantized formats store scales and zero points per group of weights, and some layers (embeddings, the output head) often stay in higher precision. Check the actual size of the safetensors files rather than trusting the multiplication.

So a 70B model in BF16 does not fit on one 80 GB card, needs two of them with tensor parallelism, and fits on a single 141 GB card with room for cache. The same model in 4-bit fits on a 48 GB card, but with very little left over for users.

KV cache: the part that decides the card

For every token in every active sequence, each transformer layer stores a key and a value vector. The formula:

kv_bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_element

The 2 is key plus value. num_kv_heads is the important term: models with grouped-query attention (GQA) share key/value heads across several query heads, which shrinks the cache dramatically.

Real numbers, FP16 cache:

Llama 2 7B (32 layers, 32 KV heads, head_dim 128, no GQA): 2 × 32 × 32 × 128 × 2 = 512 KiB per token.
Llama 3 8B (32 layers, 8 KV heads, head_dim 128): 2 × 32 × 8 × 128 × 2 = 128 KiB per token.
Llama 3 70B (80 layers, 8 KV heads, head_dim 128): 2 × 80 × 8 × 128 × 2 = 320 KiB per token.

Now multiply by load. Say you serve Llama 3 70B with an 8,192-token context and want 32 sequences in flight:

320 KiB * 8192 tokens = 2.5 GiB per sequence
2.5 GiB * 32 sequences = 80 GiB

That is 80 GiB of cache on top of 140 GB of weights. The "it fits on two 80 GB cards" plan turns into "it loads on two cards and serves about four users at full context". The model starts, the demo works, and throughput collapses under real traffic.

In practice not every request uses the full context, and paged attention (vLLM, SGLang, TensorRT-LLM) allocates cache in blocks only for tokens that exist. So size against your real prompt length distribution, not the theoretical maximum. Log prompt plus completion lengths from a pilot and use the 90th percentile.

A calculator you can adapt

from dataclasses import dataclass

GiB = 1024 ** 3

@dataclass
class Model:
    params_b: float      # billions of parameters
    layers: int
    kv_heads: int
    head_dim: int

def inference_vram_gib(m: Model, weight_bytes: float, kv_bytes: float,
                       ctx_tokens: int, concurrent: int,
                       overhead_gib: float = 2.0, activation_frac: float = 0.05):
    weights = m.params_b * 1e9 * weight_bytes / GiB
    kv_per_token = 2 * m.layers * m.kv_heads * m.head_dim * kv_bytes
    kv = kv_per_token * ctx_tokens * concurrent / GiB
    activations = (weights + kv) * activation_frac
    total = weights + kv + activations + overhead_gib
    return round(weights, 1), round(kv, 1), round(total, 1)

llama3_8b = Model(8.0, 32, 8, 128)
llama3_70b = Model(70.6, 80, 8, 128)

print(inference_vram_gib(llama3_8b, 2, 2, 8192, 16))    # BF16 weights, FP16 cache
print(inference_vram_gib(llama3_70b, 2, 2, 8192, 32))
print(inference_vram_gib(llama3_70b, 0.55, 1, 8192, 32)) # 4-bit weights, FP8 cache

Output:

(14.9, 16.0, 34.4)
(131.5, 80.0, 224.1)
(36.2, 40.0, 82.0)

Read the last line carefully. Quantizing weights to 4-bit and the cache to FP8 brings a 32-user 70B deployment from "three or four 80 GB cards" down to "one 80 GB card, slightly over budget". Drop concurrency to 24 or context to 6k and it fits. Whether the quality loss from 4-bit weights is acceptable is a separate question that you answer with your own eval set, not with a benchmark table.

The activation_frac of 5% is a rough allowance. Long-prompt prefill on large batches can need more; vLLM handles this by reserving memory during its profiling run at startup, which is why its numbers differ from naive math.

Turning the math into a vLLM config

vLLM takes a fraction of GPU memory, loads weights, and gives everything left to the KV cache. The flags that matter:

vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching

--gpu-memory-utilization 0.90 leaves 10% for the CUDA context and fragmentation. Going to 0.95 works until a monitoring agent or a second process grabs memory, then you get OOM at 3 a.m.
--max-model-len caps context. If you leave it at the model's native 128k, vLLM may refuse to start because one maximum-length sequence does not fit in the cache, or it starts with very few concurrent slots.
--max-num-seqs limits how many sequences are scheduled at once. Too high and requests get preempted and recomputed when the cache fills; too low and the GPU idles.
--kv-cache-dtype fp8 halves cache memory. Supported on Hopper and Ada cards; measure output quality before shipping it.
--enable-prefix-caching reuses cache for shared prompt prefixes. If every request starts with the same 2,000-token system prompt, this is a large win.

At startup vLLM logs the number of GPU blocks it allocated for the cache and the maximum concurrency it can sustain at full context. Read that line before running a load test. It is the single best check of whether your sizing was right.

Fine-tuning needs a different budget

Full fine-tuning with AdamW in mixed precision costs about 16 bytes per parameter before activations:

BF16 weights: 2
BF16 gradients: 2
FP32 master weights: 4
Adam first and second moments in FP32: 8

For a 7B model that is about 112 GB, so full fine-tuning even a small model needs several cards or ZeRO/FSDP sharding plus optimizer offload. Activations come on top and scale with batch size and sequence length; gradient checkpointing trades roughly 30% more compute for a large cut there.

LoRA freezes the base weights and trains small adapter matrices, so optimizer state almost disappears. A 7B model with LoRA fits on a 24 to 48 GB card. QLoRA loads the base model in 4-bit and trains adapters on top: the original QLoRA work fine-tuned a 65B model on a single 48 GB GPU. For most business use cases (tone, format, domain vocabulary) LoRA on an 8B or 70B base is the sensible starting point.

Matching models to cards

Rough mapping for inference with moderate concurrency, assuming you size the cache as above:

7B to 13B, BF16: a 48 GB card such as the L40S handles this comfortably with plenty of cache. Good fit for internal assistants and RAG.
30B to 34B, BF16: one 80 GB A100 or H100.
70B, 4-bit: one 80 GB card with limited concurrency, or a 48 GB card for a low-traffic internal tool.
70B, BF16 or FP8 with real traffic: two 80 GB cards with tensor parallelism, or a single H200 with 141 GB, which avoids inter-GPU communication entirely.

Memory bandwidth matters as much as capacity. Token generation is bandwidth-bound: every new token reads all the weights once. A card with more HBM bandwidth generates tokens faster at the same model size, which is a big part of why H100 and H200 are worth the premium for latency-sensitive serving.

Pitfalls that cost real time

Multi-GPU without NVLink. Tensor parallelism across PCIe-only cards works, but the all-reduce on every layer eats into throughput. Ask how the cards are connected before you plan a two-card split.

Driver and CUDA mismatch. A wheel built for one CUDA version on a host driver that is too old gives cryptic errors. Pin your container base image and check nvidia-smi for the driver's maximum supported CUDA version first.

Disk and download time. A 70B BF16 checkpoint is around 140 GB. On a fresh server, downloading it can take longer than the first experiment. Put model weights on local NVMe, not a network mount, or load times stretch into tens of minutes.

Benchmarking with one user. A single-stream test measures latency, not capacity. Use a load generator with your real prompt length distribution and watch time-to-first-token as concurrency grows.

Ignoring data residency. If the prompts contain customer records, where the GPU physically sits can be a legal constraint, not a preference. Check it before choosing a provider, not after the security review.

Rent or buy

The utilisation threshold is the deciding number. A card that is busy less than about half the time is usually cheaper to rent, because owned hardware depreciates while idle. Prototyping, evaluation of several model sizes and periodic fine-tuning runs almost always fall below that line. A model serving production traffic around the clock for several years can justify buying.

The practical sequence: rent, measure real utilisation and real VRAM use for a month with the calculator above as your starting estimate, then decide. If you need hardware located inside Kazakhstan with CUDA, PyTorch and Docker already set up, billed by the hour for experiments or monthly for long runs, here is how our GPU server rental with H100, H200, A100 and L40S cards works.

Finding duplicate photos on iPhone without uploading a single one

ZoomApps — Mon, 26 Oct 2026 17:22:05 +0000

Most photo cleaners send your library somewhere or lock the useful part behind a weekly subscription. We wanted the opposite for our app, Fast Cleaner: everything runs on the iPhone, and the free tier covers a normal daily cleanup. Here is how the duplicate finder works and what we learned shipping it to 50 languages.

1. One embedding per photo with Apple Vision

VNGenerateImageFeaturePrintRequest returns a feature vector for an image. Two photos of the same scene end up close to each other, even with small changes in framing or exposure.

let request = VNGenerateImageFeaturePrintRequest()
let handler = VNImageRequestHandler(cgImage: thumb, options: [:])
try handler.perform([request])
let print = request.results?.first as? VNFeaturePrintObservation

We request a 160×160 thumbnail from PhotoKit instead of the full image. Vision resamples to its own input size anyway, and the smaller request loads about 30% faster. Embeddings are cached on the device, so the second scan only processes new photos.

2. Compare only what can be similar

Comparing every photo with every other one is O(n²). On a 50,000-photo library that is far too slow. We bucket photos by the month they were taken and compare pairs inside a bucket. Duplicates and burst-like series are almost always taken close in time.

var d: Float = 0
try a.computeDistance(&d, to: b)
if d <= 0.3 { unionFind.union(i, j) }

A distance threshold of 0.3 gave the best balance between "same moment" and "similar but different". Union-find turns the pairs into groups.

3. Pick the best shot, offer the rest

Inside each group a simple weighted score picks the photo to keep:

face capture quality (VNDetectFaceCaptureQualityRequest): 0.35
sharpness, Laplacian variance on luminance: 0.30
resolution, log-scaled: 0.15
open eyes and smile from face landmarks: 0.15
newer date as a tiebreak: 0.05

Photos without faces get a neutral face score, so landscapes are not penalised.

4. Things that bit us

Memory. Batches of 50 images inside autoreleasepool, otherwise large libraries run out of memory.
The simulator lies. In the iOS simulator Vision returned the same vector for every image, so all distances were 0 and every photo landed in one group. We now detect "collapsed" embeddings (different sizes, distance exactly 0) and skip such buckets.
Main thread. Reading the photo library during view rendering caused watchdog kills on big libraries. All PhotoKit reads moved off the main thread.
Deletion safety. Deletions go through PhotoKit, so everything lands in Recently Deleted for 30 days.

Pricing without a weekly plan

Scanning, the report, swipe-to-clean and up to 300 MB of cleanup a day are free. Pro removes the limit and adds video compression: $1.99/month (first month $0.99), $9.99/year or $19.99 once.

Fast Cleaner is on the App Store: https://apps.apple.com/app/id6772155675 — more at https://zoomapps.kz/fastcleaner/