DEV Community: Stream

What Is Shadow Banning, and How Does It Work as a Moderation Technique?

Sarah Lindauer — Mon, 26 Oct 2026 17:22:03 +0000

TL;DR

Shadow banning limits a user's reach without telling them. They can keep posting and see their own content as normal, while other people see it less or not at all.
Platforms use it mostly against spam accounts and bots, because a banned spammer opens a new account and a shadow-banned one never learns anything changed.
A shadow ban only works if every read path applies the same server-side check, including search, push notifications, and channel previews.
It does not change behavior. Users told why their content was removed break the rules less often, so shadow bans suit spam and bots, and everyone else should be told what they did.

Shadow banning is a moderation technique that reduces a user's or their content's visibility without clearly telling them a restriction has been applied. The user can keep posting, and everything looks normal from their own account. Other people see their content less often, or not at all.

Why bother with this technique? Why not just ban the user outright? Because a bad actor can keep acting badly when they can see they’ve been banned. A spammer whose account gets suspended opens a new one and starts again. Under a shadow ban, they never find out. They keep posting, nobody sees it, and it can take days to realize the links have stopped getting clicks.

The cost is transparency. A user whose reach collapses can't tell whether they were shadow-banned, caught by an algorithm change, or posted something nobody cared about, which is why the term gets attached to every dip in engagement on every social media platform.

How Does a Shadow Ban Work?

The platform accepts the content as normal, stores it, and then leaves it out when it builds what other people see. Nothing changes on the author's side. The request that created the post returned success, and the post appears in their own feed and on their profile. The restriction applies on the read side, when someone else's feed, channel, search results, or notifications are assembled.

Online communities have used shadow banning since the 1980s, then under names like hellbanning, stealth banning and ghost banning. Jeff Atwood describes the intended effect on trolls: nobody ever responds, so they get bored and leave.

On social media platforms, the term now covers a wider set of restrictions:

Keep a post out of recommendations, trending pages and algorithmic feeds, so it only reaches existing followers
Drop an account from search results and hashtag pages
Rank its posts and comments much lower
Collapse its replies behind a "show more" control
Hide its comments from everyone except the author
Stop sending notifications when the account interacts with someone

The restriction can apply to an entire account, a single post, a feature like search, or a set period. X's enforcement options show the range of what's possible on one platform. A restricted post can be excluded from search, trends, and the For You timeline; limited to the author's profile; downranked in replies; or have likes and reposts switched off. Since 2023, a post restricted this way carries a label, so the author and everyone else can see that its reach was limited.

Why Shadow Ban Someone Instead of Banning Them Outright?

Because it works on spam accounts and bots, as long as the chat is busy enough that one user getting no replies looks normal. Stream moderation docs give livestreams as the example. In a channel with six people, the silence gives it away.

Reduced visibility also suits content that isn’t bad enough to delete. A post that only just violates guidelines, or one carrying misinformation that breaks no rule, can be demoted instead. TikTok does this with a notice. A video can stay on your profile and still be ineligible for the For You page, with the reason in its analytics and an appeal button.

How it has played out on the big platforms:

Reddit ran shadow bans as its only account-level punishment until 2015, when it called them “great for dealing with bots/spam rings, but woefully inadequate for real human beings” and switched to suspensions that state the reason and the duration. People had been posting for weeks, wondering why nobody answered them.
Instagram’s Restrict feature is a shadow ban on one relationship. A restricted person’s comments on your posts are visible only to them, and Instagram built it after young people said they were reluctant to block or report peers who bullied them.
Facebook hides some rule-breaking comments and deletes others, warning the author. A study of 412 million comments found that deletion reduced how often the author broke community guidelines afterward, while hiding had no significant effect.

What a shadow ban doesn’t do is change behavior. A study of 32 million Reddit posts found that users given a reason for a removal were less likely to have posts removed later. If you want someone to post differently, tell them what they did.

What Triggers a Shadow Ban?

A shadow ban is an enforcement action, and a separate decision triggers it. The decision can come from:

An automated classifier that scores a message, image, or video against the platform's content moderation policies and crosses a threshold.
Spam signals such as the same link posted hundreds of times, an unusually high posting rate, a brand-new account with no profile photo, or a pattern of other users muting and blocking the account.
User reports, once enough of them land on one account.
Repeated community guidelines violations, where earlier strikes were warnings or removals, and the next step is reduced distribution.
A human moderator who reviewed the account and decided a quiet restriction is the right call.

On Stream, rules are the automated route. A rule combines conditions, a threshold, and an action. The spam rule in our docs fires when a user posts five messages labeled as scams or platform bypass within an hour, or fifty messages of any kind in that hour, and the action can be a ban with shadow_ban set to true and a duration attached, or a shadow_block on the content itself. A cooldown period stops the rule from re-firing the moment a moderator lifts the ban.

The manual route is a moderator clicking shadow ban in the dashboard, or your backend making the same call:

await client.moderation.ban({
  target_user_id: "user_to_shadow_ban",
  banned_by_id: moderatorId,
  channel_cid: "livestream:game-night",
  reason: "Repeated promo links",
  timeout: 1440,
  shadow: true,
});

Leave out channel_cid, and the ban covers the whole app. timeout is in minutes, so this one expires after a day, and the reason is stored with the ban so whoever looks at it later knows why it exists.

What Does It Take to Enforce a Shadow Ban Properly?

Every read path must apply the same check, and the check must run on the server. Hiding a message in the main message list does nothing for:

The push notification that already went out with its text
The search index that returns it
The channel preview that shows it as the last message
The history request that returns it when the user pulls to refresh.

Each is a separate code path, and the restriction has to hold on every one of them, or the reader sees the content and the author gets a hint something is off.

The decision is keyed to who is asking:

In Stream Chat, messages from a shadow-banned user carry shadowed: true for everyone except the author, whose copy looks normal even in the raw API response. Your app hides the flagged messages for everyone else. queryBannedUsers returns who issued each ban, the reason, the expiry, and whether it was a shadow ban.

In the EU, Article 17 of the Digital Services Act requires a statement of reasons for any visibility restriction, demotion included. The one exception is deceptive high-volume commercial content, that is, spam.

How Can You Tell If an Account Has Been Shadow Banned?

From the outside, you can’t be certain, only narrow it down. X’s own list of reasons a post might not be seen runs from rule enforcement to other users muting you, ranking, experiments, and technical problems, and they all look the same from the author’s side. Search “shadow banned” alongside LinkedIn, TikTok, or Instagram and you’ll find people certain that’s why their posts aren’t showing up.

What you can check:

Open the post’s direct link logged out or from a second account. If it loads, the post is still public, and any restriction is on distribution.
Search for the account from that second account. Reachable by link but missing from search points to a search restriction.
Check the platform’s status page. Instagram’s Account Status shows whether a professional account is eligible for recommendation on the Explore page, and TikTok flags a video that’s ineligible for the For You feed in its analytics.
Compare with similar accounts. If they dropped the same week, the algorithm changed.

Twitter said in 2018 that it didn't shadow ban, and the Twitter Files released in 2022 showed search and trends blocklists that it called visibility filtering, and its critics called content suppression. A study of 40 million tweets found that tweets with external links got as little as an eighth of the visibility of tweets without them, and even then the authors couldn’t prove the recommendation algorithm was the cause, because they can’t inspect it.

If you’re running the community, keep shadow bans for spam accounts and bots, put an expiry on every one, and tell everyone else what they did. On Stream, both are the same ban call with shadow set one way or the other, and the review queue and appeals sit in the same dashboard.

How Does a Message Queue Power Reliable Chat Delivery?

Sarah Lindauer — Mon, 26 Oct 2026 16:58:42 +0000

TL;DR

A message queue is a buffer between services. A producer publishes a message, the broker stores it, and a consumer processes it and sends an acknowledgment, so neither service has to be available when the other runs.
A queue gives each message to one consumer, and pub/sub gives a copy to every subscriber. Chat delivery uses both.
Delivery is at-least-once, so a unique key on recipient ID and message ID makes a redelivered message harmless. The outbox pattern keeps the saved message and its queue event in sync.
Queues keep order per key, so the conversation ID is the key. A failing message blocks the ones behind it in that conversation.

Connections drop. Servers restart. The friend you just messaged is away from their desk.

Chat messages are asynchronous by nature. You send one now, and it gets read later, whether that's in two seconds or tomorrow morning. The backend in between is a distributed system, and it works the same way. Each service does its part and passes the message along without waiting for the next one.

A message queue is where that handoff happens. If a connection drops or a server restarts mid-process, the message stays in the queue, and the next attempt picks it up.

What Is a Message Queue?

A message queue is a buffer that services use for asynchronous communication. One service publishes messages to the queue, and the queue holds them until another service consumes and acknowledges them. The two services never call each other directly, so neither has to be available when the other one runs.

A message passes through these components:

The producer creates a message and publishes it to the broker. Its request ends when the broker confirms it has stored the message.
The message broker is the server that hosts the queue. It persists each message and delivers it to a consumer, either by pushing it over an open connection or by answering the consumer's poll.
The consumer is the service that does the work. It receives a message from the broker, processes it, and sends an acknowledgment back.

The publisher confirm and the consumer acknowledgment are independent. A confirm tells you the broker has the message, and says nothing about whether any consumer has processed it.

Message brokers can be self-hosted or hosted:

Self-hosted: RabbitMQ, IBM MQ, Apache ActiveMQ, Apache Kafka
Hosted: Amazon SQS, Google Pub/Sub, Azure Service Bus

Your code connects to a broker over a protocol such as AMQP (which RabbitMQ uses) or through an API such as JMS (which Java applications use with IBM MQ).

In a microservices architecture, a new chat message can fan out to several services, such as delivery, push notifications, moderation, and search indexing. If the chat API called each one directly, one slow service would hold up every send. With a queue, the API can enqueue a message and return, keeping the services decoupled.

A queue is also scalable on the consumer side. When messages arrive faster than they're processed, they wait in the queue, and you add more consumers to work through the backlog. The broker balances the load by giving each message to one consumer, so throughput increases without changing the producer.

How Is a Message Queue Different From Pub/Sub?

A message queue delivers each message to exactly one consumer, while publish/subscribe (pub/sub) delivers a copy to every subscriber.

Chat delivery needs both messaging patterns. A new message must reach several services, and a single queue would send it to only one. So each message goes through both:

The backend publishes one MessageCreated event.
Delivery, push notifications, and search indexing each subscribe to that event and get a copy in their own queues.
A pool of workers shares each queue, so one worker handles each copy.

This is an event-driven architecture. The chat API doesn't know which services consume the event, so you can add a subscriber, such as analytics, without changing the API.

How Does a Message Queue Deliver a Chat Message?

The chat API commits the message to its database, a relay publishes it to the message queue, and a delivery worker consumes it and writes it to the recipient’s inbox. Say Alice sends Bob “Can you send me the deployment notes?” from her phone, and Bob has the app open on his laptop. The message goes through these steps:

Alice’s app generates a UUID for the message before sending anything and reuses that ID as the idempotency key on every resend.
The chat API inserts the message and an outbox row in a single database transaction, then responds to Alice.
A relay publishes the outbox row to the broker, keyed by conversation ID, and marks the row as published once it receives the publisher confirm.
The broker delivers the message to one delivery worker and keeps it, unacknowledged, until that worker reports back.
The worker inserts an inbox row for Bob, keyed on recipient ID and message ID, commits, and then acknowledges the message. In a group conversation, this step fans out, creating one inbox row per member.
The worker sends the message to the real-time gateway, which holds Bob’s WebSocket connection and pushes the message to his laptop.

The JSON body of Alice’s request carries the ID her app generated:

{
  "message_id": "019a1d6e-7c2b-7f10-9a44-3b5e2c8d1f07",
  "conversation_id": "chat-42",
  "sender_id": "alice",
  "text": "Can you send me the deployment notes?"
}

If Alice’s connection drops before the response arrives, her app resends the same body. The message ID is the primary key, so the second insert conflicts with the first, and the API returns the message it already stored. The Stream iOS SDK queues a send that fails offline and retries it when the device comes back online, even if the app was killed in between.

Step 2 is the transactional outbox pattern. Inserting the message and then publishing to the broker would be a dual write, with no transaction spanning the database and the broker. If the process crashed after the insert and before the publish, Bob’s delivery pipeline would never receive a message that Alice can see in her history. The outbox pattern stores the event in the same transaction as the message, so the event is sent if and only if the transaction commits.

BEGIN;
INSERT INTO messages (id, conversation_id, sender_id, seq, body)
VALUES (:message_id, 'chat-42', 'alice', 1842, :body);
INSERT INTO outbox (event_id, event_type, message_id, payload)
VALUES (:event_id, 'MessageCreated', :message_id, :payload);
COMMIT;

Alice’s request returns as soon as this transaction commits. Everything after it is asynchronous messaging between backend services, so a slow or restarting worker delays Bob’s copy without failing Alice’s send.

In queue terms, the relay in step 3 is the producer and the delivery worker is the consumer. The relay finds new outbox rows in one of these ways:

A polling publisher queries the outbox table at intervals, which works with any SQL database.
Transaction log tailing, a form of change data capture, reads the database’s own log, such as the Postgres WAL or the MySQL binlog, and publishes each outbox insert it finds.

Each row becomes one message, with the row’s JSON as the payload and headers that carry metadata such as the message ID and a timestamp.

The relay waits for the publisher confirm because a client that has written a publish to its socket can’t assume the message reached the server. It can still crash after the confirm arrives and before it marks the row as published, so that it may publish the same event twice. That’s acceptable, because the worker’s insert in step 5 is written so that a second run changes nothing.

The worker’s ack in step 5 only proves that a backend service finished its write. Knowing whether Bob’s laptop received the message, or whether Bob read it, takes separate events sent from his device.

In Stream Chat, a message is sent when it reaches our server, delivered when the recipient’s device confirms it with a message.delivered event, and read when the recipient marks the channel read. Delivery tracking is off by default, and you turn it on per channel type with delivery_events.

What Happens When a Worker Crashes or a Message Is Delivered Twice?

The broker delivers the message again, so your write must be safe to run twice. If a worker crashes or loses its connection before it acknowledges a message, the broker gives that message to another worker. This is at-least-once delivery, and it’s where a queue’s fault tolerance comes from.

Each message broker has its own trigger for redelivery:

Broker	Redelivers when	Default timer
RabbitMQ	The AMQP channel or connection closes, or the worker nacks with requeue	30-minute ack timeout on quorum queues
Amazon SQS	The visibility timeout expires before the message is deleted	30 seconds
Kafka consumer group	A new consumer takes over the partition and starts from the last saved position	None
Google Pub/Sub	The ack deadline passes	10 seconds

Say a worker commits Bob’s inbox row and dies before its ack reaches the broker. The broker can’t see your database. It only knows the ack never arrived, so it gives the message to a second worker, which runs the same insert:

INSERT INTO recipient_inbox (recipient_id, message_id, conversation_id, seq, state)
VALUES ('bob', :message_id, 'chat-42', 1842, 'pending')
ON CONFLICT (recipient_id, message_id) DO NOTHING;

The unique key on (recipient_id, message_id) makes the second insert do nothing; the second worker acks, and Bob sees one message. The ack has to come after the commit. A worker that acks first and crashes before committing tells the broker the work is done, while Bob has no inbox row.

The alternative is at-most-once delivery. With RabbitMQ’s automatic acknowledgment mode, a message counts as delivered as soon as it’s sent, trading safety for higher throughput, and a worker that crashes loses it. Broker features labeled “exactly once,” such as Kafka’s idempotent producer and SQS FIFO deduplication, only prevent a producer from retrying and writing the message twice. They don’t cover the worker’s database write, so the unique key is still needed.

How Do Message Queues Keep Chat Messages in Order?

Queues keep order per key, so you give every message in a conversation the same key. Chat needs a stable order inside each conversation, and it doesn't matter how messages from chat-42 and chat-91 interleave. Each broker has its own name for that key:

Kafka uses the record key. Records with the same key go to the same partition, so set the key to the conversation ID.
In Amazon SQS FIFO queues (first-in, first-out), messages that share a MessageGroupId are processed in strict order.
Google Pub/Sub has an ordering key, but it only takes effect when you enable message ordering on the subscription.
In RabbitMQ, a consistent hash exchange sends all messages with the same routing key to the same queue, and a single active consumer on each queue processes them one at a time.

With strict ordering, a message that keeps failing blocks every message behind it. In an SQS FIFO queue, later messages in the same group are held back while one is in flight, so the conversation stalls until the failing message is moved to the dead-letter queue.

A Kafka partition blocks the same way, because a consumer group's position in each partition is a single integer and it can't skip a failing record and come back to it later. Moving the record to a separate retry topic lets the partition continue, and the newer messages are then processed ahead of it.

Do You Need to Build This Delivery Pipeline Yourself?

Building it means running message queues, an outbox relay, delivery workers, and a sync service for as long as your app has chat. If you'd rather have your engineers working on the rest of your app, Stream Chat runs the server side on our Go backend and global edge network, which we've benchmarked at 5 million concurrent connections in one channel, and our SDKs handle the client side.

When your own backend needs chat events, our webhooks retry failed deliveries and can send events to SQS or SNS. You can set up any of our SDKs from the Chat docs.

What Actually Happens When Android's Hardware Encoder Hits SVC

Pratim Mallick — Wed, 21 Oct 2026 03:57:53 +0000

A livestream can look perfectly healthy and still get flagged as broken. That's the situation I kept running into on Android, and it took a while to figure out why.

I work on the video team at Stream, where I spend most of my time on the Android SDK. Video calls and live streams on Stream use a system called scalable video coding (SVC), which lets a phone encode video once and lets the server deliver different resolutions to viewers based on their bandwidth. It's an efficient design, especially for mobile, where every extra encode costs battery and CPU.

The setup I want to walk through here started simple, with one publisher, one viewer, and everything worked as expected. Then a second viewer joined, and the quality score on that stream dropped and never recovered. Nothing about the stream had visibly changed.

That single symptom sent me down a debugging path with two convincing false leads before I found the actual cause, which turned out to be a limitation baked into Android's hardware video encoders themselves.

This piece is about what that debugging process looked like.

What SVC actually is

To back up for a second on how SVC actually works: a single encoded stream carries multiple layers inside it. Spatial layers are different resolutions, something like 180p, 360p, and 720p. Each spatial layer also carries temporal layers, which are different frame rates, something like 7.5, 15, and 30 fps.

The publisher encodes all of this once. The server, in Stream's case the SFU, looks at each subscriber's bandwidth and forwards only the layers that subscriber can handle.

Simulcast solves the same problem differently.

The publisher encodes three separate streams at three separate resolutions, and the server picks which stream to forward. It works, but it costs three encoders' worth of CPU and battery on the sending device. On mobile, where compute and battery are both limited, SVC is the better fit, assuming the encoder can actually produce the layers it's asked for.

That assumption turned out to be the entire problem.

The setup that worked

Here's where the debugging starts.

One publisher on Android, one subscriber on the other end. The SFU tells the publisher to send a single spatial layer with three temporal layers inside it, a mode we call L1T3. The hardware VP9 encoder handles this without issue. Network conditions were good, the stream looked good, and the SFU scored it well. This was the baseline.

The moment it broke

Then a second subscriber joined the call. With two subscribers now needing different resolutions, the SFU asked the publisher to switch modes, this time requesting three spatial layers and three temporal layers, a mode called L3T3_KEY.

As soon as that switch happened, the SFU's quality score for the stream dropped and never recovered.

The obvious thing to check first was the SFU's scoring system, which grades every published stream. This stream scored badly the moment the encoder switched configurations, and stayed bad for the rest of the call.

Something about producing three layers instead of one was breaking the stream, but at that point I had no idea what.

False lead #1: the encoder thrashing loop

The first thing I noticed in the logs was that the hardware encoder was getting torn down and recreated over and over, roughly every 200 milliseconds. That kind of churn is never good, so I dug into why it was happening.

Android's hardware VP9 encoder can't change its resolution or its buffer type while running. Any time WebRTC's quality scaler asks it to adjust resolution, the encoder has to shut down completely and start over, rather than adjusting in place. Software encoders don't have this restriction. Hardware encoders do.

Here's the loop that creates.

The encoder starts at 720p. It gets torn down and rebuilt for buffer type reasons. When it comes back up, it has a pile of frames waiting, which throws off the bandwidth estimator's read on the connection. The quality scaler decides 720p isn't sustainable and drops the encoder to 540p. Now the quantization parameter is calibrated for 720p bitrate but running at 540p, so the scaler tries to climb back to 720p. That triggers another full teardown and rebuild. The whole cycle repeats every five to ten seconds, and what you see as a viewer is the publisher's resolution oscillating between 720p and 540p.

I confirmed this by turning off the quality scaler entirely, using WebRTC's degradation preference setting. That stopped the resolution oscillation completely. But the SFU still scored the stream as poor. The thrashing loop was real, and we fixed it, but it wasn't the reason the stream kept getting marked as low quality.

False lead #2: the phantom bitrate

The next clue came from the stats. WebRTC reported the stream's target bitrate at around 71 kbps, while the encoder was clearly outputting close to 1.4 Mbps at 720p. That gap looked like it explained the quality problem on its own.

It turned out to be a reporting quirk in libWebRTC, not a real problem.

WebRTC calculates target bitrate using an index that assumes simulcast, where three separate streams each have their own SSRC (a unique stream identifier), and the target bitrate for a given layer comes from summing values across those indexes. In SVC, there's only one stream and one SSRC, so the index is always zero. That means WebRTC's target bitrate calculation always lands on the base layer's slice of the total bitrate, around 70 kbps, regardless of what the encoder is actually sending.

The wrong number comes from SendStatisticsProxy in libWebRTC. It maps each SSRC to a simulcast index, then reports the bitrate of that spatial layer as the target bitrate.

// send_statistics_proxy.cc
// Store target bitrates per substream stats.
for (auto& [ssrc, substream] : stats_.substreams) {
  std::optional<size_t> simulcast_index;
  for (size_t i = 0; i < rtp_config_.ssrcs.size(); ++i) {
    if (rtp_config_.ssrcs[i] == ssrc) {
      simulcast_index = i;
      break;
    }
  }
  if (!simulcast_index.has_value()) {
    substream.target_bitrate = std::nullopt;
    continue;
  }
  substream.target_bitrate =
      DataRate::BitsPerSec(allocation.GetSpatialLayerSum(*simulcast_index));
  if (substream.target_bitrate == DataRate::Zero()) {
    substream.target_bitrate = std::nullopt;
  }
}

The function it calls, GetSpatialLayerSum, adds up all the temporal layers on one spatial layer.

// video_bitrate_allocation.cc
// Get the sum of all the temporal layer for a specific spatial layer.
uint32_t VideoBitrateAllocation::GetSpatialLayerSum(
    size_t spatial_index) const {
  RTC_CHECK_LT(spatial_index, kMaxSpatialLayers);
  return GetTemporalLayerSum(spatial_index, kMaxTemporalStreams - 1);
}

This is correct for simulcast. SSRC 0, 1, and 2 map to index 0, 1, and 2, so S0, S1, and S2 each get their own target bitrate.

It is wrong for VP9 SVC. There is only one SSRC, so simulcast_index is always 0. The stat becomes GetSpatialLayerSum(0), which covers the S0 layer only. S1 and S2 never count toward the target bitrate. On L3T3, that S0 slice is the ~71 kbps I saw.

Once I understood the formula, the mismatch made sense.

The encoder was sending the full bitrate as intended. The stats weren't built to report it correctly for a single-SSRC stream. Another dead end, but a useful one to know if you're ever debugging SVC stats and the numbers don't add up.

The real root cause

With both of those explanations ruled out, I checked whether the encoder was actually producing three layers in the first place.

It wasn't.

The hardware VP9 encoder was only ever producing one layer, the highest one requested, and giving that single layer the entire bitrate. It never produced the two lower spatial layers. The SFU expected three layers because it had asked for L3T3, got one flat 720p stream instead, and marked the stream as poor, which was the accurate read of what was happening.

To rule out a device-specific issue, we tested across Samsung and Pixel devices, covering both Qualcomm and Tensor chipsets. Every device showed the same behavior. Hardware VP9 encoding on Android has this limitation across the platform, confirmed on every device and chipset we tested, regardless of brand.

Software VP9, running through libvpx, doesn't have this problem. It produces all three spatial layers correctly and doesn't need to tear itself down to change resolution. The tradeoff is CPU and battery, which is a real cost, and the reason nobody wants to run software encoding by default in the first place.

What this looks like at scale

To see how much this actually mattered outside of a controlled test, we looked at real production data from one of Stream's livestreaming customers using VP9 across platforms.

In that data, Android streams using the hardware VP9 encoder were consistently scored as poor. Streams using the VP9 software did noticeably better, and the remaining problems stemmed from CPU and battery limits rather than missing layers.

Neither path fully solves the problem.

Hardware can't produce the layers SVC needs. Software can, but not without a real device cost. And because quality scores show up in the UI as network indicators, most people watching would assume the publisher had a bad connection.

In most of these cases, the network was fine. The hardware just wasn't capable of what was being asked of it.

Can you detect this ahead of time?

Given all of this, the natural next question is whether you can detect this ahead of time and avoid asking a device to do something its encoder can't handle.

Android's MediaCodec APIs let you query a fair amount about a given hardware encoder: supported resolutions, bitrates, profile levels, and temporal layering schemas. Android 17 (API 37) added a new method, getSupportedLayeringSchemas(), which sounded promising. In practice, it only returns temporal schemas like webrtc.svc.l1tN. There's no API that reports whether a hardware encoder can produce spatial layers.

That means there's no way to check in advance whether a given device's hardware VP9 encoder actually supports the SVC mode you're about to ask it for. The only way to find out is to ask it and inspect the resulting bitstream.

Where this leaves us: broadcast mode

Given that neither hardware nor software VP9 fully solves this on Android, we've been building an alternative approach on the SFU side, which we're calling broadcast mode. It's still in development and not yet in production.

The idea is to stop asking the publisher's device to produce multiple layers at all. Instead, the publisher sends one clean stream at whatever hardware codec and resolution the device handles best. The server then transcodes that single stream into the quality ladder viewers need, generating the lower resolutions server-side instead of on the device.

This removes the CPU drain, the missing spatial layers, and the encoder recreation problem, since the phone only has to do one thing it's already good at. The tradeoff moves to the server, where transcoding adds real compute cost. That's not something we'd turn on for every use case, but it fits broadcast-style setups with one publisher and many viewers well, which is exactly the case where this problem shows up most.

We're planning to make this available as an opt-in SDK option for livestream and broadcast use cases once it's ready.

Takeaways

A few things I'd want any developer building live video on Android to take away from this.

Hardware VP9 on Android only supports temporal SVC. Spatial layers are silently dropped, not degraded, dropped.
Software VP9 gives you real SVC, but the CPU and battery cost is real too.
There's currently no API on Android that lets you check spatial SVC support ahead of time. You find out from the bitstream itself.
If you're seeing SVC quality problems on Android that seem to show up only once more than one viewer joins, check this first.

Inside Stream's Real-Time Architecture for Trading & Investing Apps

Sarah Lindauer — Fri, 25 Sep 2026 20:10:55 +0000

TL;DR

Trading apps run two real-time systems: a transaction plane (orders, execution, ledger) and an experience plane (chat, feeds, video). Users judge speed by the second one.
Stream Feeds caches fanout-on-read results as a materialized feed with per-source delta cursors, holding ~15ms p50 read latency even at 4.5M concurrent users during simulated market-open load.
Stream Chat's edge network and offline-first SDKs deliver sub-50ms API responses and scale to 5M people in a single channel.
Stream Video's SFU cascading handles a 100K-viewer stream at 225 Gbps with 0% packet loss, comfortably under the 150ms threshold for natural conversation.

Trading has zero tolerance for latency. Exchanges measure their matching engines in microseconds, brokers measure order handling in milliseconds, and firms pay for co-location just to shave a few hundred microseconds off the round-trip to the exchange. Users bring the same expectation to the app on their phone. When a stock moves, the quote has to change immediately.

Stream's Feeds, Chat, and Video are built for the speed of a trading app. They sit alongside the trading system rather than inside it. That same speed has to hold through the bursts, like a market open, an earnings release, or a sharp move in a widely held stock, when every real-time feature gets hit at once.

This piece breaks down the real-time architecture trading and investing apps need. Here's how the three hold together as one system on the other side of a trade.

Two Kinds of Real-Time Inside One App

A trading app contains two kinds of real-time infrastructure:

The transaction plane handles market data, order management, risk checks, routing, execution, and the ledger. It cares about ordering, idempotency, auditability, and exact financial state. A duplicated or reordered execution report is a serious bug.
The experience plane handles activity feeds, notifications, chat, presence, and live video. It cares about global fanout, low perceived latency, media quality, and clean recovery when a phone drops off the network for a minute. In Stream's terms, that's three products (Feeds, Chat, and Video) that a trading platform has to get working together.

The end user doesn't see that split. Both planes show up in the same interface. A price move updates a quote in the transaction plane, and the same move triggers a price alert, a run of community posts, and an analyst going live in the experience plane.

You can have the best transaction plane in the market, but the experience plane is what puts that data in front of the user in real time. If the alert arrives late or the chat lags, they'll assume your trading is slow too.

The backend is authoritative. The experience plane's job is to take data from the backend and put it in front of the user, with enough context to be useful.

Say a user has an open limit order on a stock they follow, and the price hits their limit during a busy morning. In the transaction plane, the sequence looks like this:

The market-data path receives the trades that carry the price to the user's limit and updates the cached quote.
The venue executes the resting limit order and sends an execution report back to the broker.
The OMS moves the order from working to filled, after checking that it hasn't already processed that execution ID.
The ledger records the cash movement and the new position, and the portfolio read model is rebuilt from those entries.
The backend publishes domain events. OrderFilled for this user, and PriceThresholdCrossed for everyone whose watchlist alert sits at that price.

The transaction plane is tailor-made for this. It is both correct and fast. A duplicated execution report or an out-of-order update is a financial error, so it's built around idempotency and strict ordering. The experience plane then turns those events into what the user sees:

The OrderFilled event becomes an activity in the user's portfolio timeline and a notification on their phone, showing the price and quantity recorded in the ledger.
The PriceThresholdCrossed event becomes a price alert for every user who has set one at that level, which, at a widely watched price, means thousands of alerts at once.
The user opens the stock's community channel. Recent messages load instantly from on-device storage, and new ones arrive via WebSocket as others react to the same move.
An analyst on the platform starts a live briefing on the move. That one doesn't come from the trading backend at all. The analyst starts the session from their own tools, a notification goes out to everyone who follows them, and the user joins the video session with the channel's chat alongside it.

From the user's side, this all happened at once. The fill confirmation, the price alert, the discussion, and the briefing appeared within the same minute and looked like a single product.

Underneath, the transaction plane ensured the fill was real and the numbers were right, and the experience plane ensured the user saw it while it still mattered. Neither one can do the other's job. The feed never decides whether the order filled, and the OMS was never built to fan a message out to two million phones.

Effectively, you have two different feeds:

A market-data feed carries sequenced quotes, trades, and book updates with gap recovery, and it belongs to the transaction plane.
An activity feed carries alerts, posts, research, and portfolio events. The feeds discussed here are the second kind.

Source event	What the user sees	Real-time feature
`PriceThresholdCrossed`	Watchlist or price alert	Notification feed
`OrderFilled`	Portfolio activity item	Activity feed
`ResearchPublished`	Personalized research timeline	Activity feed
`AnalystPosted`	Community post and discussion	Activity feed and chat
`AdvisorMessageSent`	Private client conversation	Chat
`MarketBriefingStarted`	Live commentary with chat alongside	Video and chat
`KYCSessionRequested`	Identity-review call	Video

The trading backend decides what happened, and the experience plane's job is distribution. That means getting the event to the right people, in the right feed, channel, or call, quickly enough that it's still useful when it arrives.

How that holds up under trading-hours load is where the architecture matters.

What Counts as Fast in Each Plane

The user cares that the app keeps up with the market. That is the standard. But under the hood, we have two systems operating on two different latency timescales.

In the transaction plane, the reference point is other machines. The venue matches orders in microseconds, and the quote on the user's screen updates as fast as the market data feed can deliver. The user isn't competing with any of that, and doesn't need to. What they need is to see what happened as soon as possible, so they can decide what to do while the move is still playing out. The quote already gets that treatment. The alert, the chat, and the video have to get it too.

In the experience plane, the question is simpler. Did the alert, feed, chat message, or video frame arrive before the person holding the phone noticed any delay? Two well-established numbers tell you where users start to feel a delay:

Though Nielsen Norman Group's response-time limits date from 1993, they are still the reference everyone uses. Around 0.1 seconds feels instantaneous. Around 1 second is noticeable, but the user's thought flow remains intact. They're general usability findings, not trading targets, and they're the right test for feed reads and optimistic UI updates, where the app is trying to look like it responded before it finished the work.
ITU-T Recommendation G.114 provides telecom planning guidance on one-way delay in a conversation. It treats 150 ms one-way as acceptable for most conversational voice, and notes that highly interactive tasks can suffer well below that. It's planning guidance, not a threshold where delay suddenly becomes noticeable, and it's the number that matters for an advisor call.

Here's how the layers stack up:

Layer	Measured against	What fast looks like
Exchange matching and market-data distribution	Other exchanges and other participants	Microseconds to low milliseconds
Broker order handling	The venue and the market-data feed	Milliseconds
Feed and chat APIs	The person reading the screen	Low milliseconds per request
The app's interface	The person reading the screen	Under 0.1 s to feel instant, under 1 s to keep flow
Voice and video	The person in the conversation	Under 150 ms one-way, with low jitter and loss

Everything in the experience plane gets judged against the bottom three rows, and it gets judged hardest at market open, when all of those systems are under the most load at the same time.

How Activity Feeds Handle Fanout at Market Open

In a trading app, most of what happens ends up in a feed:

Price and watchlist alerts that the trading backend fires.
The user's own order and portfolio activity.
Research from the analysts and instruments they follow.
Posts, comments, and reactions from other investors.
Per-ticker, per-sector, or per-topic activity.
Ranked “for you” style commentary.

Every one of those is a fanout problem. One event goes in, and it has to come out across many different feeds, each a different mix of what that user follows. That's hard to scale, and you’ll only find it fails at market open.

Push Versus Pull Fanout

There are two basic ways to build a feed:

With fanout on write, or push, every new activity gets copied into each follower's feed the moment it's created. Reads are quick, because the feed is already sitting there. The cost lands on the write, and it grows with the follower count. If an account has two million followers, every post is two million writes. That's the celebrity problem.
With fanout on read, or pull, you store the activity once and build the user's feed when they open it. Writes are trivial. Reads do all the work, pulling and merging activities from everyone the user follows, and that gets slower the more they follow.

An investing app has both problems at once. A well-known analyst, a fund, or the feed for a heavily traded ticker can have millions of followers, so pure push makes every one of their posts expensive. Pure pull moves the cost onto reads instead, and reads arrive all at once at 9:30 when everyone opens the app to see what happened overnight.1

How the Materialized Feed Works in Stream

The Stream Feeds API uses fanout-on-read, but caches the result. Each activity is stored once, and when a feed is read, the API fetches what's changed since that feed's last read and merges it into what's already cached. That cached result is the materialized feed.

Here's how it works:

Each activity is stored once, on write, with no per-follower copying.
When a feed is read, the API fetches only what's changed since that feed's last read and merges it into what's already cached, rather than rebuilding the feed from scratch.
While it's building the top N items, it keeps track of which followed feeds have nothing recent enough to make the cut, so it stops checking them.
The fields used for ranking and aggregation are pulled from the activity record rather than requiring the full object. Aggregation copies the fields it needs into the feed operation itself, and ranking works off a lightweight internal activity summary that holds just the ranking fields.
Aggregation keys get computed on write, so aggregated feeds open quickly. Ranked feeds can't be fully precomputed, so a fast expression parser re-ranks them at read time.
The cached result is stored in TiKV with a cursor per followed source, so a later read only pulls what's new since that cursor. TiKV also handles the multi-Raft clustering and rebalancing we used to run ourselves.

	Fanout on write	Fanout on read	Materialized feed
Write cost	High, one copy per follower	Low, a single write	Low, a single write
Read cost	Low, the feed is precomputed	High, assembled at read time	Low, cached with delta fetching
Celebrity accounts	Millions of writes per activity	No problem	No problem — nothing is copied per follower, same as fanout on read
Inactive users	Wasted writes to feeds nobody reads	Nothing wasted	Nothing wasted — same as fanout on read
Read latency	~1 to 5 ms	~100 to 500 ms or more	~10 ms

Reads then go through a few layers of caching before they touch storage:

Redis client-side caching keeps hot data in the API process itself, which saves a round trip to Redis on busy feeds.
Ristretto and singleflight handle cache stampedes. When a hot key expires, one request rebuilds it, and everyone else waits for that result instead of piling onto the database.
For shared ranked feeds with very high read volume, there's an opt-in hot feed cache that serves one cached ordering to every reader. A “most discussed tickers today” feed at market open is exactly what it's for.
On the WebSocket side, servers use cuckoo filters to cheaply check whether a user is even connected before doing any work to push an update.

What the Feeds Benchmark Measured

Say we have an app with 100 million users, and 4.5 million of them are online at the same time, reading and writing feeds at up to 1,500 requests per second. That's a market-open crowd for a large retail broker. We ran that test against a dataset of 500 million activities and 200 million follow relationships:

Feed reads came back in about 15 ms at p50.
Feed writes took about 31 ms at p50.
p99 stayed under 140 ms at the heaviest setting.
Every request succeeded.

So the alert or the post is in the user's feed well inside the 0.1 seconds where a person starts to notice a delay, and it stays that way when everyone opens the app at once.

How Chat Stays Fast at the Edge

In a trading app, chat is where everyone ends up at the same moment. The channel next to an earnings call, the ticker community during a sharp move, or the support inbox at market open. A message has to get from one person's phone to everyone in the room before the market moves on.

This means:

The message has to reach a server that can act on it without the sender having to pay for intercontinental round trips first.
One message has to reach millions of people in the same channel within milliseconds, at the moment they all joined.

A user in Singapore talking to an origin in Virginia pays around 200 ms for a round trip, and a new TLS connection needs several of those before the first message goes anywhere. So a Stream client connects to the nearest edge server instead. The handshake, authentication, and rate limiting all occur within a few milliseconds, and traffic flows from the edge to the origin over already-open connections — encrypted HTTP/2 between regions, and plain HTTP/1.1 for the last hop within the origin's own region.

The API itself often responds in under 50 ms. On the device, SDKs keep recent channels and messages in local storage, so the app opens on its last state, and a sent message appears immediately while the request finishes in the background.

The chat next to a widely watched earnings call can have millions of people in it, and every message has to reach them all. Servers use cuckoo filters to skip users who aren't connected to them, serialize each message once, and send it to each recipient individually, with the message's header and body packed together in a single call. Clients on networks that block WebSockets can fall back to HTTP long polling, though that's opt-in.

What the Chat Benchmark Measured

Say we have 5 million people in one channel, joining at about 4,000 per second, with three messages a second going out to all of them. That's the chat for a stock everyone owns at the moment the results come out. We ran that test on a single channel:

Around 15 million message deliveries per second at peak.
Average message send latency remained well under 40 ms.
No loss of stability for the whole run.

So sending a message into the biggest room in the app stays fast even at full scale, and the room keeps working when the whole user base joins it at once.

How Video Handles a Whole Audience Joining at Once

Video in a trading app is either a small call or a large broadcast. One-to-one advisor consultations and KYC calls need to stay under 150 ms one-way with reliable audio. Earnings commentary and analyst briefings have a few publishers and the whole user base watching, usually with chat alongside.

Serving the broadcast involves:

Pushing more bandwidth than a single server can handle to viewers on three continents.
Handling thousands of joins in the same few seconds, because a briefing starts the moment the results come out.

Forwarding One Stream to 100,000 Viewers

WebRTC was designed for peer-to-peer communication, where each publisher sends a separate copy of their stream to each other participant. That stops working past a handful of people. With a selective forwarding unit (SFU), each publisher uploads once, and the SFU forwards streams to whoever needs them, so each phone's connection count drops from O(n²) to O(n).

A 100,000-viewer 1080p stream needs more than 200 Gbps, which is more than one SFU can serve, so we cascade. The publisher's video goes to one SFU, which then forwards it to other SFUs in other regions, each serving around a thousand viewers in their area. Viewers only subscribe to the streams visible on screen, and simulcast lets each viewer receive the quality their connection can handle.

SFU capacity scales in steps based on how quickly it's needed, so a sharp rise in load can double or triple the number of SFUs running. Join-time thundering-herd protection prevents a burst of simultaneous joins from overwhelming signaling. For advisors and institutional users on locked-down corporate networks, TURN runs on our SFU servers over TCP on port 443, with a dedicated TURN network behind it and static IPs.

What the Video Benchmark Measured

Say we have 100,000 people watching one analyst's 1080p stream, joining at 10,000 per minute, with peaks of 600 per second, spread across six regions in the US and Europe. We ran that test:

225 Gbps of peak traffic across 132 cascading SFUs.
A stable 30 fps for the whole run, with 0% packet loss and 4 ms jitter.
Zero API failures.

So the briefing starts the moment the results come out, the whole audience is in within seconds, and everyone's watching at full quality.

Where This Leaves the Build Decision

None of this changes who owns the trading system. Orders, positions, balances, and execution remain with your platform or brokerage infrastructure provider, and nothing in the experience plane can assert financial state.

The question is whether you build the experience plane yourself. Running it in-house involves:

Global WebSocket fanout with edge termination and regional failover.
Feed materialization and cache-stampede protection that holds at market open.
Offline storage and delta sync across iOS, Android, and web.
SFU cascading, simulcast, TURN, and autoscaling sized for join spikes you can't schedule.
Capacity in several regions for load that arrives in bursts a few times a day.

Every one of those is a general problem, and we've already solved each of them at a scale most trading apps won't reach. Your team keeps its time for the parts of the app that carry financial risk, and the alerts, chat, and video run on infrastructure that's been tested with 4.5 million users online, 5 million people in a single channel, and 100,000 people watching a single stream.

The user judges your app by whether it keeps up with the market, and the experience plane is the part they see. If the alert arrives late or the chat lags, they'll assume the trading is slow too, however fast the order path underneath it is.

If you're building or replacing the feeds, chat, or video in a trading app, start with our Feeds, Chat, and Video architecture pages and read the benchmarks against your own peak concurrency. Or talk to our team about what your market open looks like.

Build a Live Virtual Classroom with React and Stream Video

Sarah Lindauer — Fri, 21 Aug 2026 22:28:28 +0000

Live classes are now table stakes for education products.

Whether you’re running a tutoring marketplace, a corporate training platform, or a complete online school, sooner or later a stakeholder asks for “a Zoom inside our app,” but with the rules of a classroom. The teacher controls the room, students ask to speak instead of talking over each other, and everything works on the phone a student actually has in their hand.

Building that from raw WebRTC means months of work on signaling, SFUs, device handling, and reconnection logic before you write a single classroom feature.

In this tutorial, you’ll build it in an afternoon with Stream’s React Video SDK: a virtual classroom where a teacher runs the class from a laptop, and students join from their phones, with hand-raising, a moderated microphone, and a live roster.

What You’ll Build

Here’s what the finished product looks like, with the teacher on the left and the students on the right.

Live virtual classroom demo

This includes:

Two roles with real permissions. The teacher can mute any student or the whole room, and Stream’s backend enforces it. Students can’t grant themselves the same powers by poking at the client.
Raise hand. Students tap a button; the teacher sees a queue ordered by who asked first, and can lower any hand from the roster.
A classroom mic policy. Students join muted by default and unmute when called on.
A live roster. Showing everyone’s mic and camera state in real time.
Mobile-ready student view. One responsive UI that renders full-screen on a phone.

You get video layouts, screen sharing, device selection, and dominant-speaker detection from the SDK’s built-in components, so the code you write is almost entirely classroom logic.

Worth knowing if you're handling student data: Stream is SOC 2 Type II, ISO 27001, HIPAA, DPF, and GDPR compliant, so the privacy paperwork a school district asks for is already sorted.

Prerequisites

Node 20+
A free Stream account — create an app in the dashboard and grab your API key and API secret
Working knowledge of React

Project Setup

Scaffold a Vite React app and install the two Stream SDKs plus a minimal backend:

npm create vite@latest classroom -- --template react
cd classroom
npm install @stream-io/video-react-sdk @stream-io/node-sdk express cors dotenv concurrently
npm install -D @vitejs/plugin-basic-ssl

You need two SDKs because there are two sides to authentication:

@stream-io/video-react-sdk runs in the browser.
@stream-io/node-sdk runs on your server, where your API secret lives. The secret must never be shipped to the client, as anyone who holds it can impersonate any user in your app.

The basic-ssl plugin enables HTTPS in dev. You won't need it on localhost, but you will when you test on a real phone. Browsers only expose the camera and microphone to secure origins.

Put your credentials in .env at the project root:

STREAM_API_KEY=your_api_key
STREAM_API_SECRET=your_api_secret
PORT=3001

Step 1: The Token Server

Every user connects to Stream with a token your backend mints. This is also the natural place to decide who gets to run the classroom, because call permissions in Stream are attached to server-assigned roles, not to anything the client claims about itself.

Create server/index.js:

import 'dotenv/config';
import express from 'express';
import cors from 'cors';
import { StreamClient } from '@stream-io/node-sdk';

const apiKey = process.env.STREAM_API_KEY;
const apiSecret = process.env.STREAM_API_SECRET;

const client = new StreamClient(apiKey, apiSecret);
const app = express();
app.use(cors(), express.json());

const CALL_TYPE = 'default';
const TOKEN_TTL_SECONDS = 8 * 60 * 60;

app.post('/api/join', async (req, res) => {
  try {
    const { name, role, callId } = req.body;
    if (!name || !callId || !['teacher', 'student'].includes(role)) {
      return res.status(400).json({ error: 'name, callId and role (teacher|student) are required' });
    }

    const userId = `${role}-${name.trim().toLowerCase().replace(/[^a-z0-9]+/g, '-')}`;
    await client.upsertUsers([{ id: userId, name: name.trim() }]);

    const call = client.video.call(CALL_TYPE, callId);
    await call.getOrCreate({
      data: { created_by_id: userId, custom: { classroom: true } },
    });
    await call.updateCallMembers({
      update_members: [{ user_id: userId, role: role === 'teacher' ? 'host' : 'user' }],
    });

    const token = client.generateUserToken({
      user_id: userId,
      validity_in_seconds: TOKEN_TTL_SECONDS,
    });

    res.json({ apiKey, token, userId, callId });
  } catch (err) {
    console.error('join failed:', err);
    res.status(500).json({ error: 'Could not join the classroom. Check the server logs.' });
  }
});

const port = process.env.PORT || 3001;
app.listen(port, () => console.log(`Token server running on <http://localhost>:${port}`));

The line that makes the whole classroom work is updateCallMembers. Stream’s built-in default call type includes a host role that carries capabilities such as mute-users and update-call-permissions.

By adding the teacher as a member with role: 'host', and students as plain users, you’ve established the classroom’s power structure server-side. Later, when the teacher’s client calls muteUser(), Stream’s backend checks this membership before acting. A student sending the same request gets rejected, no matter what UI they’ve hacked together.

In production, you’d derive role from your own auth system (the session of the logged-in teacher) rather than trusting the request body (here, we’re taking it from the join form to keep the focus on the video features).

Wire up the dev workflow in package.json, so one command runs both processes:

"scripts": {
  "dev": "concurrently -n server,web \"npm run server\" \"vite\"",
  "dev:lan": "LAN=1 concurrently -n server,web \"npm run server\" \"vite\"",
  "server": "node server/index.js",
  "web": "vite"
}

In vite.config.js, proxy /api to the token server, and when the LAN flag is set, serve over HTTPS on all network interfaces so a phone on your Wi-Fi can join:

// vite.config.js

import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
import basicSsl from '@vitejs/plugin-basic-ssl'

// LAN=1 serves over HTTPS on all interfaces so phones on the same network
// can join — browsers only allow camera/mic access on secure origins.
const lan = !!process.env.LAN

export default defineConfig({
  plugins: [react(), ...(lan ? [basicSsl()] : [])],
  server: {
    host: lan ? true : 'localhost',
    proxy: {
      '/api': 'http://localhost:3001',
    },
  },
})

Step 2: Connecting to the Call

The client flow is:

Exchange a name and role for a token
Create a StreamVideoClient
Join the call
Hand everything to the SDK’s provider components:

// src/App.jsx

import { useEffect, useState } from 'react';
import {
  StreamVideo,
  StreamVideoClient,
  StreamCall,
  StreamTheme,
} from '@stream-io/video-react-sdk';
import { JoinScreen } from './components/JoinScreen';
import { Classroom } from './components/Classroom';

export default function App() {
  const [session, setSession] = useState(null); // { apiKey, token, userId, name, role, callId }
  const [client, setClient] = useState(null);
  const [call, setCall] = useState(null);

  useEffect(() => {
    if (!session) return;

    const videoClient = new StreamVideoClient({
      apiKey: session.apiKey,
      user: { id: session.userId, name: session.name },
      token: session.token,
    });

    const classroomCall = videoClient.call('default', session.callId);
    // Students join muted so a full class doesn't turn into crosstalk;
    // they ask to speak with the raise-hand button instead.
    if (session.role === 'student') {
      classroomCall.microphone.disable();
    }
    classroomCall.join({ create: true }).then(() => {
      setClient(videoClient);
      setCall(classroomCall);
    });

    return () => {
      classroomCall.leave().catch(() => {});
      videoClient.disconnectUser();
      setClient(null);
      setCall(null);
    };
  }, [session]);

  if (!session) return<JoinScreen onJoin={setSession} />;
  if (!client || !call) return<div className="loading-screen"><p>Joining {session.callId}…</p></div>;

  return (
    <StreamVideo client={client}>
      <StreamTheme>
        <StreamCall call={call}>
          <Classroom role={session.role} callId={session.callId} onLeave={() => setSession(null)} />
        </StreamCall>
      </StreamTheme>
    </StreamVideo>
  );
}

Two details worth calling out:

classroomCall.microphone.disable() before join() is the entire “students join muted” feature. The SDK’s device manager applies the setting before any audio is published, so students never broadcast that first half-second of background noise.
The cleanup function leaves the call and disconnects the user, so navigating away doesn’t strand a ghost participant in the room.

JoinScreen is an ordinary form with name, class code, and a teacher/student radio that POSTs to /api/join and passes the response up.

Don’t forget the Stream SDK stylesheet in src/main.jsx:

// src/main.jsx
import '@stream-io/video-react-sdk/dist/css/styles.css';
import './index.css';

Step 3: The Teacher’s View

The teacher gets a laptop-oriented layout with the video stage, a side panel with the roster and raised-hand queue, and a control bar.

Almost all of the heavy lifting is SDK components:

// src/components/TeacherView.jsx

import {
  SpeakerLayout,
  ToggleAudioPublishingButton,
  ToggleVideoPublishingButton,
  ScreenShareButton,
  useCall,
  useCallStateHooks,
} from '@stream-io/video-react-sdk';
import { useRaisedHands } from '../hooks/useRaisedHands';
import { ParticipantsPanel } from './ParticipantsPanel';

export function TeacherView({ callId, onLeave }) {
  const call = useCall();
  const { useParticipantCount } = useCallStateHooks();
  const participantCount = useParticipantCount();
  const raisedHands = useRaisedHands();

  return (
    <div className="teacher-view">
      <header className="class-header">
        <div>
          <span className="live-dot" aria-hidden="true" />
          <h1>{callId}</h1>
        </div>
        <span className="participant-count">{participantCount} in class</span>
      </header>

      <div className="teacher-main">
        <div className="stage">
          <SpeakerLayout participantsBarPosition="bottom" />
        </div>
        <ParticipantsPanel raisedHands={raisedHands} />
      </div>

      <footer className="control-bar">
        <ToggleAudioPublishingButton />
        <ToggleVideoPublishingButton />
        <ScreenShareButton />
        <button
          className="bar-button"
          onClick={() => call?.muteAllUsers('audio')}
          title="Mute every student's microphone"
        >
          Mute all
        </button>
        <button className="bar-button leave" onClick={onLeave}>
          End class
        </button>
      </footer>
    </div>
  );
}

SpeakerLayout gives you dominant-speaker switching, a scrollable filmstrip of everyone else, and automatic screen-share handling. When the teacher clicks the ScreenShareButton to present slides, the layout promotes the shared screen without any code from you.

call.muteAllUsers('audio') is the “settle down, everyone” button. It’s a server-side operation. Stream stops the audio at the backend, and every muted client’s UI updates to reflect it.

Step 4: The Roster, With Per-Student Mute

The roster is where Stream’s reactive state is ideal. useParticipants() re-renders whenever anyone joins, leaves, or changes what they’re publishing, so mic/cam indicators are always live:

// src/components/ParticipantsPanel.jsx

import { SfuModels, useCall, useCallStateHooks } from '@stream-io/video-react-sdk';

export function ParticipantsPanel({ raisedHands }) {
  const call = useCall();
  const { useParticipants } = useCallStateHooks();
  const participants = useParticipants();
  const { hands, lowerHand } = raisedHands;

  return (
    <aside className="participants-panel">
      <section className="hand-queue">
        <h2>Raised hands</h2>
        {hands.length === 0 ? (
          <p className="empty-note">No hands up right now.</p>
        ) : (
          <ol>
            {hands.map((hand) => (
              <li key={hand.userId}>
                <span className="hand-badge" aria-hidden="true">✋</span>
                <span className="hand-name">{hand.name}</span>
                <button className="chip-button" onClick={() => lowerHand(hand.userId)}>
                  Lower hand
                </button>
              </li>
            ))}
          </ol>
        )}
      </section>

      <section className="roster">
        <h2>Everyone</h2>
        <ul>
          {participants.map((p) => {
            const micOn = p.publishedTracks.includes(SfuModels.TrackType.AUDIO);
            const camOn = p.publishedTracks.includes(SfuModels.TrackType.VIDEO);
            const handUp = hands.some((h) => h.userId === p.userId);
            return (
              <li key={p.sessionId}>
                <span className="roster-name">
                  {p.name || p.userId}
                  {p.isLocalParticipant && ' (you)'}
                  {handUp &&<span aria-label="hand raised"> ✋</span>}
                </span>
                <span className="roster-state">
                  <span className={micOn ? 'on' : 'off'}>{micOn ? 'Mic on' : 'Muted'}</span>
                  <span className={camOn ? 'on' : 'off'}>{camOn ? 'Cam on' : 'Cam off'}</span>
                </span>
                {!p.isLocalParticipant && micOn && (
                  <button className="chip-button" onClick={() => call?.muteUser(p.userId, 'audio')}>
                    Mute
                  </button>
                )}
              </li>
            );
          })}
        </ul>
      </section>
    </aside>
  );
}

A participant’s publishedTracks tells you exactly what media they’re sending right now, which is more trustworthy than tracking mute state yourself. And the per-student Mute button is one line: call.muteUser(p.userId, 'audio'). Because the teacher holds the host role from Step 1, Stream authorizes it; the student’s device stops publishing audio, and their own UI flips to muted, instantly.

Step 5: Hand Raising with Custom Events

Stream doesn’t ship a “raise hand” button, but it ships something better: sendCustomEvent(), a real-time channel for everyone on the call to send any JSON payload you like.

That means hand raising is your feature, with your rules. Here, that means a queue ordered by who asked first, which either side can clear.

The whole feature is one hook:

// src/hooks/useRaisedHands.js

import { useCallback, useEffect, useState } from 'react';
import { useCall, useCallStateHooks } from '@stream-io/video-react-sdk';

export function useRaisedHands() {
  const call = useCall();
  const { useLocalParticipant } = useCallStateHooks();
  const localParticipant = useLocalParticipant();
  const [hands, setHands] = useState([]); // [{ userId, name }] in raise order

  const localUserId = localParticipant?.userId;
  const handIsRaised = hands.some((h) => h.userId === localUserId);

  useEffect(() => {
    if (!call) return;
    const unsubscribe = call.on('custom', (event) => {
      const { type, userId } = event.custom ?? {};
      if (type === 'hand-raised') {
        const raiser = { userId: event.user.id, name: event.user.name || event.user.id };
        setHands((prev) => (prev.some((h) => h.userId === raiser.userId) ? prev : [...prev, raiser]));
      }
      if (type === 'hand-lowered') {
        const target = userId ?? event.user.id;
        setHands((prev) => prev.filter((h) => h.userId !== target));
      }
    });
    return unsubscribe;
  }, [call]);

  // Re-broadcast for late joiners so they see hands already in the air.
  useEffect(() => {
    if (!call || !handIsRaised) return;
    const unsubscribe = call.on('call.session_participant_joined', () => {
      call.sendCustomEvent({ type: 'hand-raised' });
    });
    return unsubscribe;
  }, [call, handIsRaised]);

  const raiseHand = useCallback(() => {
    if (!call || !localParticipant) return;
    setHands((prev) =>
      prev.some((h) => h.userId === localUserId)
        ? prev
        : [...prev, { userId: localUserId, name: localParticipant.name || localUserId }],
    );
    call.sendCustomEvent({ type: 'hand-raised' });
  }, [call, localParticipant, localUserId]);

  const lowerHand = useCallback(
    (userId = localUserId) => {
      if (!call) return;
      setHands((prev) => prev.filter((h) => h.userId !== userId));
      call.sendCustomEvent({ type: 'hand-lowered', userId });
    },
    [call, localUserId],
  );

  return { hands, handIsRaised, raiseHand, lowerHand };
}

You should notice three design decisions here:

The queue preserves order. New hands append to the array, and the teacher’s panel numbers them so “who was first?” is never a judgment call.
hand-lowered carries a userId. When a student lowers their own hand, the target defaults to the sender. When the teacher lowers a student’s hand, the event names the student, and because the student’s client processes the same event, the student's “✋ Lower hand” button automatically resets to “Raise hand.” One event type, both directions.
Late joiners get synced. Custom events are fire-and-forget, so someone who joins after a hand has been raised would miss it. The second effect handles this. Whenever a new participant joins, anyone who has raised a hand quietly rebroadcasts. The prev.some() dedupe check makes replays harmless.

Step 6: The Student’s View Designed for a Phone

Students in a live class are overwhelmingly on phones, so the student UI is built phone-first.

The teacher’s video is big, and there’s a thumb-reachable control bar with one prominent amber raise-hand button:

// src/components/StudentView.jsx

import {
  SpeakerLayout,
  ToggleAudioPublishingButton,
  ToggleVideoPublishingButton,
  useCallStateHooks,
} from '@stream-io/video-react-sdk';
import { useRaisedHands } from '../hooks/useRaisedHands';

export function StudentView({ callId, onLeave }) {
  const { useParticipantCount, useMicrophoneState } = useCallStateHooks();
  const participantCount = useParticipantCount();
  const { isMute } = useMicrophoneState();
  const { handIsRaised, raiseHand, lowerHand } = useRaisedHands();

  return (
    <div className="student-view">
      <header className="class-header compact">
        <div>
          <span className="live-dot" aria-hidden="true" />
          <h1>{callId}</h1>
        </div>
        <span className="participant-count">{participantCount}</span>
      </header>

      {isMute &&<p className="muted-note">You're muted — raise your hand to ask a question.</p>}

      <div className="stage">
        <SpeakerLayout participantsBarPosition="bottom" />
      </div>

      <footer className="control-bar">
        <ToggleAudioPublishingButton />
        <ToggleVideoPublishingButton />
        <button
          className={`hand-button${handIsRaised ? 'raised' : ''}`}
          onClick={() => (handIsRaised ? lowerHand() : raiseHand())}
        >
          ✋ {handIsRaised ? 'Lower hand' : 'Raise hand'}
        </button>
        <button className="bar-button leave" onClick={onLeave}>
          Leave
        </button>
      </footer>
    </div>
  );
}

useMicrophoneState().isMute drives the “You’re muted” banner, and it remains accurate regardless of the reason the student is muted. They joined that way, they muted themselves, or the teacher muted them from the roster.

Because it’s the same React SDK across desktop and mobile browsers, “mobile support” is a CSS problem (not an architectural one).

The stylesheet does two things. Below 700px, the student view goes edge-to-edge full-screen (with env(safe-area-inset-bottom) padding so the controls clear the iPhone home indicator):

.phone-frame {
  width: 390px;
  height: min(800px, 94vh);
  border: 10px solid #000;
  border-radius: 44px;
  overflow: hidden;
}

@media (max-width: 700px) {
  .phone-frame {
    width: 100%;
    height: 100dvh;
    border: none;
    border-radius: 0;
  }
}

If you’re shipping to app stores rather than the mobile web, the same call, roles, and custom events apply across Stream’s React Native, iOS, and Android SDKs. A student on the native app and one on the mobile web join the same classroom.

Try It

For a quick single-machine test:

npm run dev

Open http://localhost:5173, join as Teacher, then open a second tab and join as a Student (the student renders inside a phone frame so you can see the mobile layout without leaving your desk).

For the real thing, with the teacher on your laptop and students on actual phones, start in LAN mode:

npm run dev:lan

This is where the basic-ssl plugin earns its keep. If you serve plain HTTP to another device, students can join the call, but getUserMedia fails silently on their phones: no camera, no mic, and no obvious error. Over HTTPS, it just works.

On each phone, open https://<your-computer's-LAN-IP>:5173 (find the IP with ipconfig getifaddr en0 on macOS or hostname -I on Linux), accept the self-signed certificate warning, join as a Student, and allow camera and mic access.

Then run the class:

Student taps ✋ Raise hand → the teacher's queue shows them, numbered.
Teacher calls on them; student unmutes and asks their question.
Teacher hits Lower hand, then Mute on the roster. The student's phone will flip back to muted on its own.
Mute all when the whole back row forgets they're live.

Where To Take It Next

This is a solid starting point. Natural next steps for a production classroom:

Stricter mic policy. Instead of trusting students to stay muted, revoke the send-audio capability with call.updateUserPermissions() and grant it only when the teacher calls on a raised hand, turning the hand queue into a true request-to-speak flow.
Class chat. Stream’s Chat SDK shares the same user tokens, so a channel per classroom is a small addition.
Recording and attendance. call.startRecording() captures the lesson; call session events to give you join/leave timestamps for attendance.
Breakout groups. Each group is just another call; move students between call IDs.

You built a working virtual classroom in a few hundred lines of React code: server-enforced teacher moderation, an ordered raise-hand queue via custom events, and a student experience that's at home on a phone.

None of it required touching WebRTC internals. The Stream SDK handled the media layer, so every line you wrote was classroom logic.

Ready to add live classes to your own product? Create a free Stream account, grab your API keys, and you can have this running against your own app before your next standup. The React Video SDK docs cover everything this tutorial touched on, including call types, permissions, custom events, and where to go deeper.

How to Build an AI Voice Agent with Python and WebRTC: Step-by-Step Guide

Sarah Lindauer — Fri, 21 Aug 2026 22:15:17 +0000

Voice should be the default interface for AI. It is the sci-fi dream: talk into your watch and have an assistant answer your questions, book your movie tickets, and remember your dog's birthday.

That future isn’t quite here yet, but AI voice agents are already embedded in everyday systems. Think customer support lines, language tutors, drive-thru ordering, and in-app copilots. Each lets you talk instead of type, using the same thing under the hood: an AI voice agent.

A few years ago, such an agent would have taken an entire team to build and encompassed a full codebase. Now, if you’re a Python developer, building one is a single-file problem.

So that’s what we’ll do here. We’ll build a working voice agent you can talk to in your browser during a Stream call. But we also want to help you grok the concepts, so we’ll cover the architecture voice agents share, explain why the transport layer, not the AI, is the hard part, and walk through the design decisions that separate an agent that feels like a conversation from one that feels like a voicemail system.

By the end, you’ll have a ~60-line Python script you can run, understand, and adapt for your own use case.

What Is an AI Voice Agent?

An AI voice agent is a program that takes part in a live audio conversation as a full participant.

It listens to a continuous stream of a person's speech, uses a language model to decide how to respond, and answers in synthesized speech, in real time, taking turns the way a person would.

That definition rules out most voice features you've already seen. A voicemail transcriber runs speech-to-text in batch after the recording ends; an agent transcribes a live stream while you're still talking. A "read this aloud" button is one-directional TTS. And the classic voice assistant interaction (press button, say command, wait, hear result) is really a request/response API with audio attached; one utterance in, one utterance out, session over.

A voice agent, by contrast, holds a single continuous two-way session for the length of the call. Being a real conversation partner inside that session comes down to four specific behaviors:

Turn detection. Deciding when you've finished your turn versus paused mid-thought, sometimes called endpointing. A half-second of silence after "so what I'm thinking is" means keep listening; the same silence after "what do you think?" means respond.
Barge-in. Listening even while it's speaking, over a full-duplex audio channel, so when you cut in, it can stop mid-sentence, discard the unspoken rest of its reply, and yield the turn.
Latency. Keeping the gap between your last word and its first under about a second, ideally much less, or the exchange stops feeling live. (The next section puts hard numbers on this.)
State. Carrying the conversation history forward, so "no, the second one" resolves against options it listed a minute ago.

Get any of these wrong and users notice immediately, not as a bug, but as a vague sense that they're talking to a machine that isn't listening.

The rest of this article is, one way or another, about engineering these four behaviors: the pipeline that produces the words, and the transport and design decisions that make the timing work.

Voice agent demo

The Architecture: STT → LLM → TTS, in a Loop

Almost every voice agent is built from the same three components:

Speech-to-text (STT) transcribes the user’s audio into text as they speak.
A large language model (LLM) takes the transcript (plus conversation history and instructions) and generates a response.
Text-to-speech (TTS) converts that response into audio that the user hears.

These run as a continuous loop for the duration of the call, all riding on a transport layer that moves audio between the user’s microphone and your Python process:

Each stage of this pipeline is a solved problem.

STT providers like Deepgram or ElevenLabs transcribe streaming audio with word-level timestamps in real time. Any modern LLM can hold up its end of a conversation. TTS voices from providers like ElevenLabs are close enough to human that the difference rarely matters.

So, if the pieces are all commodity, why aren’t voice agents trivial to build?

Why Real-Time Transport Matters

In human conversation, the gap between one person finishing and the other starting is about 200 - 300 milliseconds. Anything beyond roughly a second reads as hesitation, and anything beyond two, people start saying “Hello? Are you there?”

Now add up the voice agent loop. The user’s audio has to travel from their microphone to your server, get transcribed, wait for the LLM’s first tokens, get synthesized back into audio, and travel back to the user’s speaker. Every one of those stages takes time, and the user experiences the sum.

That budget math has two consequences:

Everything must stream. You can’t wait for the full transcript before calling the LLM, or the full LLM response before starting TTS. Each stage has to start producing output while consuming input. What matters is time-to-first-audio, not time-to-complete-response.
The network hops can’t be an afterthought. If STT, LLM, and TTS each cost 200 - 400ms and you can’t do much about it, the transport is where your remaining budget lives. Sending audio over a request/response protocol like HTTP, or even WebSockets, which run over TCP, means head-of-line blocking. One delayed packet holds up everything behind it while the connection waits for retransmission. For live audio, it’s disastrous because a 200ms-old audio packet is worthless. Better to drop it and stay current.

This is exactly the problem WebRTC was built for. It runs audio over UDP with codecs and jitter buffers designed for one goal: keep latency minimal and quality acceptable, in that order of priority. It’s the same technology underneath Google Meet-style video calls, and it’s the reason a voice agent can respond in under a second while a WebSocket-based one struggles.

But WebRTC alone only solves the protocol. There’s still geography. If your user is in Singapore and your server is in Virginia, physics adds 200ms+ of round trip before anything else happens. This is where Stream’s global edge network comes in. The user connects to the nearest of Stream’s edge servers, and audio travels the long haul over Stream’s optimized backbone rather than the public internet. Your agent joins the call the same way, from wherever it’s running.

The practical upshot is that you shouldn’t build any of it. Not the transport, the edge routing, the codecs, or jitter buffers. You should get it as infrastructure and spend your time on what your agent actually does. That’s the design philosophy behind Vision Agents, Stream’s open-source Python framework for real-time voice and video AI, which is what we’ll use to build ours.

Building the Agent

Time to write some code. The finished agent is a single file, and we’ll walk through all of it. You’ll need Python 3.10–3.13, uv, and API keys from four providers (all have free tiers):

Stream for the WebRTC transport
Deepgram for speech-to-text
Google AI Studio for the Gemini LLM
ElevenLabs for text-to-speech

Each of these is swappable. Prefer OpenAI or Anthropic for the LLM, or Cartesia for TTS? Or xAI’s Grok-4 for the LLM paired with Fish Audio for TTS? It’s a one-line change.

Project Setup

Create a project and install Vision Agents with the plugins we need:

mkdir ai-voice-agent && cd ai-voice-agent
uv init
uv add "vision-agents[getstream,deepgram,gemini,elevenlabs]" python-dotenv

Then create a .env file with your keys:

STREAM_API_KEY=your_stream_api_key
STREAM_API_SECRET=your_stream_api_secret
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key
ELEVENLABS_API_KEY=your_elevenlabs_api_key

The Agent

Here’s the complete agent:

# agent.py

"""A real-time AI voice agent: STT -> LLM -> TTS over WebRTC."""

from dotenv import load_dotenv

from vision_agents.core import Agent, AgentLauncher, Runner, User
from vision_agents.plugins import deepgram, elevenlabs, gemini, getstream

load_dotenv()

INSTRUCTIONS = (
    "You are a friendly voice assistant. "
    "You are on a live audio call, so keep responses short and conversational — "
    "one or two sentences unless the user asks for detail. "
    "Never use markdown, lists, or emoji: everything you say is spoken aloud."
)

async def create_agent(**kwargs) -> Agent:
    return Agent(
        # The transport layer: joins the call over WebRTC via Stream's
        # edge network, so audio hops to the nearest edge server instead
        # of crossing the world to a single origin.
        edge=getstream.Edge(),
        # How the agent appears to other participants on the call.
        agent_user=User(name="Voice Assistant", id="agent"),
        instructions=INSTRUCTIONS,
        # The pipeline: swap any of these three for another provider
        # without touching the rest of the file.
        stt=deepgram.STT(eager_turn_detection=True),
        llm=gemini.LLM(),
        tts=elevenlabs.TTS(),
    )

async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        # Speak first so the user knows the agent is live.
        await agent.simple_response("Greet the user briefly and ask how you can help.")
        # Keep handling turns until the call ends.
        await agent.finish()

if __name__ == "__main__":
    Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()

That’s an entire AI voice agent. Let’s break down what each part does.

The Instructions

This is a standard LLM system prompt, but with voice-specific constraints:

INSTRUCTIONS = (
   "You are a friendly voice assistant. "
   "You are on a live audio call, so keep responses short and conversational — "
   "one or two sentences unless the user asks for detail. "
   "Never use markdown, lists, or emoji: everything you say is spoken aloud."
)

Text-chat habits are wrong for audio. An LLM that answers with a five-bullet markdown list produces TTS output that reads the asterisks aloud. Telling the model it’s speaking and to use speech patterns (e.g., short sentences) matters more than any fancy prompt engineering you’ll do here. When you adapt this agent, this string defines your agent’s actual job.

The Agent Definition

async def create_agent(**kwargs) -> Agent:
   return Agent(
       edge=getstream.Edge(),
       agent_user=User(name="Voice Assistant", id="agent"),
       instructions=INSTRUCTIONS,
       stt=deepgram.STT(eager_turn_detection=True),
       llm=gemini.LLM(),
       tts=elevenlabs.TTS(),
   )

This is the architecture diagram from earlier, expressed as six keyword arguments:

edge=getstream.Edge() is the transport layer. It handles WebRTC session negotiation, connects to Stream’s edge network, publishes the agent’s audio track to the call, and subscribes to the user’s audio track. Basically, everything from the “Why transport matters” section above is condensed into this single line.
agent_user is the agent’s identity on the call. It shows up as a named participant, just like a human would.
stt, llm, and tts are the three pipeline stages. Each is a plugin; Vision Agents wires them together, so audio flows from the call into Deepgram, transcripts flow into Gemini, and Gemini’s streaming output flows into ElevenLabs and back onto the call.

The eager_turn_detection=True flag on the STT is a latency optimization, which we’ll talk about in the design decisions below.

Joining a Call

agent.join(call) is an async context manager:

async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        await agent.simple_response("Greet the user briefly and ask how you can help.")
        await agent.finish()

Inside the with block the agent is live on the call, and when the block exits it cleans up the connection.

The first thing we do is simple_response(...). This sends a prompt to the LLM and speaks the result, so the agent greets the user instead of sitting in silence waiting to be spoken to. Then agent.finish() hands control to the framework’s conversation loop: listen, transcribe, respond, speak, repeat, until the call ends.

Run It

uv run agent.py run

The first launch takes a few extra seconds. Vision Agents warms up the pipeline components and downloads a small local voice activity detection model (Silero VAD) it uses to tell speech from silence. Then it creates the call, prints the join link, and opens Stream's demo UI in your browser for you:

Open the link if the browser didn't launch on its own, allow microphone access, and say hello. You join under a randomly assigned demo name (we drew “Electric Empress”), and the agent sits alongside you as Voice Assistant, the identity we set in agent_user. It speaks first because join_call fires that simple_response greeting before the conversation loop starts:

Note the latency readout in the call's top bar, 21 ms on our run. That's the transport slice of the latency budget below, measured live rather than estimated.

Two things are worth trying before you move on:

Watch the terminal while you ask a question, and you'll see the loop from the architecture section run in order: transcript finalized, response tokens streaming in, synthesized audio heading back out.
Interrupt the agent mid-answer. It stops talking and hands you the turn, and the next section explains where both behaviors come from.

Key Design Decisions

The script works, but understanding why it works and where the knobs are is what lets you adapt it. It helps to see a full turn end-to-end first. Here's everything that happens between your voice and the agent's:

Everything in this section is about the timing inside that diagram, from when the turn gets declared over, to what happens when you talk over the reply, to where the milliseconds go.

Four decisions matter most.

1. Turn Detection: Knowing When to Speak

The hardest question in voice AI isn’t “what should the agent say?” It’s “when should it start saying it?”

Humans signal the end of a turn with intonation, grammar, and rhythm. A naive agent that responds after every pause will interrupt you mid-thought (“So I was thinking… [AGENT STARTS TALKING]”). One that waits too long feels laggy. This is a genuine ML problem, and voice stacks solve it with dedicated end-of-turn models rather than simple silence timers.

In our agent, turn detection comes built into Deepgram’s STT, and the eager_turn_detection=True flag enables a clever latency trick. When the model suspects your turn is over, it starts the LLM speculatively. If the turn really is over, the response is already in flight, and you’ve saved a few hundred milliseconds. If you keep talking, the draft is discarded, and nothing is lost:

If you pick an STT provider without built-in turn detection (say, a local Whisper model), Vision Agents lets you plug in a standalone detector instead:

from vision_agents.plugins import smart_turn

agent = Agent(
    ...,
    stt=fast_whisper.STT(),
    turn_detection=smart_turn.TurnDetection(),
)

2. Interruption Handling: Letting the User Barge In

In real conversation, people talk over each other constantly. They correct, redirect, or say “no, silly!” An agent that plows through its full response while you’re trying to interrupt it is instantly infuriating.

Vision Agents handles this automatically. When the user starts speaking while the agent is mid-response, the framework stops TTS playback, flushes any already-synthesized audio from the outgoing track (so stale speech doesn’t bleed into the next turn), and goes back to listening. You get barge-in behavior without writing any code.

You can also control interruptibility per utterance when you script specific moments:

await agent.say("One moment while I check that.", interrupt=True)

3. Noise Suppression: Garbage In, Garbage Out

Turn detection and interruption handling are only as good as the audio feeding them. A TV in the background, a second person talking nearby, or keyboard noise can all trigger false barge-ins (the agent stops talking because it “heard” the user) or corrupt transcripts.

The right place to fix this is as early in the chain as possible, before the audio ever reaches your pipeline. Stream’s client SDKs apply noise suppression and echo cancellation on the capture side, so what arrives at the STT is the user’s voice, not their environment. If your users will call from cafés, cars, or open-plan offices, this is the difference between an agent that works in the demo and one that works in production.

4. The Latency Budget: Where the Milliseconds Go

Every stage of the loop spends part of your ~1-second budget. Roughly, for a well-tuned pipeline:

Stage	Typical cost	The lever
Capture + transport (user → agent)	~30–70 ms	WebRTC + edge network
STT finalization + turn detection	~100–300 ms	Eager turn detection hides most of this
LLM time-to-first-token	~200–500 ms	Model choice; smaller = faster
TTS time-to-first-byte	~100–300 ms	Provider/model choice
Transport back (agent → user)	~30–70 ms	WebRTC + edge network

Two things to notice. First, the transport rows are small because of WebRTC and edge routing. Doing this over plain HTTP from another continent would mean those rows would dominate the table. Second, everything streams, so these stages overlap. The user starts hearing the response while the LLM is still generating its end.

There’s one more architectural option on the latency frontier: speech-to-speech models like Gemini Live or OpenAI Realtime, which skip the STT and TTS stages entirely by processing and producing audio natively. In Vision Agents, that’s a one-line change: llm=gemini.Realtime(), and drop the stt/tts arguments. You gain latency and lose control. No transcript to inspect, no ability to filter or modify the LLM’s text before it’s spoken, less choice over the voice. For most applications, the pipeline’s control is worth its milliseconds, but it’s good to know the trade exists.

Building Your Own AI Voice Agent

Everything specific to your agent lives in two places: the INSTRUCTIONS string (what the agent’s job is) and the three pipeline components (which providers do the work). The transport, turn-taking, and interruption machinery doesn’t change when you change the agent’s purpose.

From here, some natural next steps:

Give it tools. Vision Agents supports function calling and MCP servers, so your agent can look up orders, book appointments, or query your API mid-conversation.
Swap providers. Try openai.LLM() or anthropic.LLM(), or a different ElevenLabs voice, and see how the agent’s character changes. Or, swap in a fully local option if you need to run offline.
Deploy it. uv run agent.py serve starts an HTTP server that spawns an agent for each incoming call. The first step from local demo toward production, with Docker and Kubernetes guides in the Vision Agents docs.

With the transport handled, a real-time voice agent is an afternoon project, and the interesting work is deciding what yours should do.

Build a Voice Agent That Calls to Confirm Fraud Alerts

Sarah Lindauer — Mon, 10 Aug 2026 15:55:50 +0000

This tutorial builds a voice agent that places an outbound call to a cardholder, reads back a suspicious transaction, and either clears it or freezes the card, all in Python with Vision Agents.

You’ll use speech-to-text from Deepgram, an LLM for the conversation, Cartesia for the voice, and Twilio to place the call. The build is for a fictional issuer, Meridian Bank.

The interesting constraint is that a fraud-alert call and a scam call open identically ("we noticed suspicious activity on your card"), and the only thing that separates them is what the agent does next.

A real bank confirms a transaction and never asks the customer to prove themselves with sensitive data. A scammer asks for exactly that. So the rules that keep this bot legitimate are the same rules that keep it from sounding like a scam, and we enforce them in the agent's instructions.

Let’s build it.

What We're Building

Meridian Bank's fraud system flags a charge and places an outbound call to the cardholder.

The agent:

Identifies itself as Meridian Bank's fraud line and states why it is calling.
Reads the flagged transaction back: merchant, amount, location.
Asks the customer to confirm or deny it.
If confirmed, it clears the hold so the purchase completes.
If denied, it freezes the card, opens a dispute, and hands off to a human specialist.

It runs over Stream's edge network for low latency, with Twilio bridging the phone call.

The trust boundary from the opening shows up concretely in steps one through three: 1) the agent identifies itself, 2) reads a transaction back, and 3) takes a yes or no.

It never turns the call into a request for the customer's card number, PIN, password, codes, or money. We'll enforce that in the instructions.

Why Vision Agents Fits This

A fraud callbot is a plumbing problem more than a model problem.

You need to place a phone call, understand speech coming through a phone codec (mulaw at 8kHz, with background noise and interruptions), respond fast enough that it feels like a conversation and not an interactive voice menu, and take real actions like freezing a card.

The interesting part is the fraud logic. Everything between the phone line and the model is undifferentiated work.

Vision Agents collapses that work into one Agent object with swappable parts. Speech-to-text, the LLM, and text-to-speech are constructor arguments you can change in a single line. Turn detection is built into Deepgram, so the agent knows when the caller stopped talking. Actions are ordinary async Python functions you register with a decorator. The phone call bridges in through the Twilio plugin without changing the agent itself.

The customer can also join from anywhere, not just a phone.

The agent runs on a Stream call, and Stream ships frontend SDKs for React, iOS, Android, Flutter, React Native, JavaScript, and Unity. So the same fraud agent can take a Twilio phone call, answer from a "verify this charge" button inside your mobile banking app, or run in a browser.

You build the agent once and let the customer reach it however they already talk to you.

Setup

Vision Agents uses uv and targets Python 3.13 for the examples. Install the framework with the plugins this build needs:

uv add "vision-agents[getstream, deepgram, cartesia, openai, twilio]"

Get a free Stream API key from the Stream dashboard. The free tier includes 333,000 participant minutes per month, which is more than enough to build and test this.

Set your keys in a .env file:

STREAM_API_KEY=...
STREAM_API_SECRET=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
OPENAI_API_KEY=...
TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
NGROK_URL=...

The NGROK_URL is the public hostname Twilio will use to reach your machine during development. More on that in the phone section.

The Core Agent

Start with the agent itself, no phone yet. This is the part you can run in a browser to hear it work before wiring up telephony.

from vision_agents.core import Agent, User
from vision_agents.plugins import getstream, deepgram, cartesia, openai


async def create_agent() -> Agent:
    llm = openai.LLM(model="gpt-5.5")

    agent = Agent(
        edge=getstream.Edge(),
        agent_user=User(id="agent", name="Meridian Fraud Line"),
        instructions=FRAUD_INSTRUCTIONS,
        stt=deepgram.STT(model="nova-3", eager_turn_detection=True),
        llm=llm,
        tts=cartesia.TTS(model_id="sonic-3.5"),
    )
    return agent

A few things to note, all of which matter in practice:

llm = openai.LLM(model="gpt-5.5") is the conversation. This is a one-line swap: any current OpenAI model works, and you can drop in Gemini, Claude, or an OpenAI-compatible endpoint by changing this line alone. gpt-5.5 is OpenAI's current flagship at the time of writing.

stt=deepgram.STT(...) sets up speech-to-text. eager_turn_detection=True lowers latency by letting the model decide sooner that the caller has finished talking. It costs a few more tokens but the call feels more natural, which matters when someone is already on edge about a possible fraud charge. Pass model= explicitly. The plugin's default has changed between releases, so pinning the model keeps your behavior stable.

tts=cartesia.TTS(...) sets up the voice. Pin model_id for the same reason.

edge=getstream.Edge() is the transport. It runs on Stream's global edge network, which Stream reports at sub-second join times and audio latency under 30 milliseconds on the transport hop, with a P50 voice round-trip around 240 milliseconds across the full speech-to-text, LLM, and text-to-speech pipeline. Twilio will bridge the phone call into this same call later.

To run it in a browser during development, wire it to the CLI launcher:

from vision_agents.core import Agent, AgentLauncher, Runner, User
from vision_agents.plugins import getstream, deepgram, cartesia, openai


async def create_agent() -> Agent:
    # ... as above ...
    return agent


async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        await agent.simple_response(
            "Introduce yourself as Meridian Bank's fraud line and explain you're "
            "calling about a recent transaction that needs to be confirmed."
        )
        await agent.finish()


if __name__ == "__main__":
    Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()

Run it:

uv run main.py run

This opens a browser UI where you can talk to the agent directly. You will hear it introduce itself and start the confirmation flow. Get this feeling right before you put it on a phone, because debugging tone and pacing is much easier in the browser than over a live call.

The Instructions Are the Product

For most voice agents, the system prompt is a personality. For a fraud bot, it is the compliance boundary, and it is the single most important part of this build.

The instructions have to do two things at once: run the confirmation flow, and hold the line on what the agent will never ask for.

FRAUD_INSTRUCTIONS = """
You are the automated fraud line for Meridian Bank. You are placing an outbound
call to a cardholder about a transaction our system flagged as unusual. Keep
replies short and conversational. Do not use special characters or formatting.

HOW TO OPEN THE CALL
- Say you are calling from Meridian Bank's fraud department.
- Say you are calling to confirm a recent transaction, nothing more.
- Do not create panic. This is a routine check, not an accusation.

HOW TO VERIFY THE TRANSACTION
- Read back only the flagged transaction: merchant, amount, and location.
- Ask the customer to confirm whether they made this purchase, yes or no.
- If they made it, tell them you'll clear the hold and the purchase will go through.
- If they did not, tell them you'll freeze the card, open a dispute, and connect
  them to a specialist.

WHAT YOU MUST NEVER DO
- Never ask for the full card number, CVV, PIN, or expiration date.
- Never ask for the online banking password or Social Security number.
- Never ask for a one-time passcode or verification code. No real fraud line
  ever asks for these. Asking would make you indistinguishable from a scammer.
- Never ask the customer to move, send, or reverse money.
- Never send or read out a link for them to click.
- We already have the customer's account on file. You do not need to collect
  identifying secrets to do your job.

IF THE CUSTOMER IS UNSURE OR THE CALL GETS COMPLICATED
- Offer to let them hang up and call the number on the back of their card. A
  real bank encourages this. Then escalate to a human specialist.
"""

The line that does the heavy lifting is the reasoning attached to the one-time passcode rule. A model told simply "do not ask for codes" can be argued out of it by a plausible-sounding edge case. A model told why (asking would make it indistinguishable from a scammer) holds the boundary under pressure, because the instruction now carries its own justification.

This mirrors what real issuers publish.

A legitimate fraud alert confirms a transaction with a yes or no. It does not extract secrets. Banks state plainly that they will never ask you to verify a full account number, CVV, PIN, or a verification code, and consumer protection guidance is absolute that no real fraud department will ever ask for a one-time code. Encoding that directly is what makes the agent credible to the person on the other end.

Giving the Agent Actions

The agent needs to do things, not just talk.

In Vision Agents, you register plain async Python functions as tools with the @llm.register_function() decorator, and the LLM calls them when the conversation calls for it.

One rule to know before you write them: registered functions must be async. If you pass a synchronous function, registration raises a ValueError. The framework checks with inspect.iscoroutinefunction at registration time, so this fails fast rather than at runtime.

Here are the four actions the fraud flow needs. Wire them onto the LLM before you attach it to the agent:

async def create_agent() -> Agent:
    llm = openai.LLM(model="gpt-5.5")

    @llm.register_function(
        description="Confirm a flagged transaction was legitimate and clear the hold."
    )
    async def confirm_transaction(transaction_id: str) -> dict:
        # Call your core banking system to release the hold.
        return {"status": "confirmed", "transaction_id": transaction_id}

    @llm.register_function(
        description="Freeze the customer's card after they deny a transaction."
    )
    async def freeze_card(card_last4: str) -> dict:
        return {"status": "frozen", "card_last4": card_last4}

    @llm.register_function(
        description="Open a fraud dispute for a denied transaction."
    )
    async def open_dispute(transaction_id: str) -> dict:
        return {"status": "dispute_opened", "transaction_id": transaction_id}

    @llm.register_function(
        description="Hand the call off to a human fraud specialist."
    )
    async def escalate_to_human(reason: str) -> dict:
        return {"status": "escalating", "reason": reason}

    agent = Agent(
        edge=getstream.Edge(),
        agent_user=User(id="agent", name="Meridian Fraud Line"),
        instructions=FRAUD_INSTRUCTIONS,
        stt=deepgram.STT(model="nova-3", eager_turn_detection=True),
        llm=llm,
        tts=cartesia.TTS(model_id="sonic-3.5"),
    )
    return agent

The function name and its parameters become the tool schema the LLM sees, and the description tells it when to call each one. Notice these functions take a transaction ID and a card's last four digits, never the full number. The tool signatures enforce the same boundary as the instructions: there is no parameter anywhere that could hold a full PAN, CVV, or code, so even a confused model cannot collect one.

In a real deployment, the function bodies call your core banking and case-management systems. The LLM chains them on its own: a denial leads it to call freeze_card, then open_dispute, then escalate_to_human in sequence. By default the framework allows up to three tool-calling rounds per turn; raise it with openai.LLM(model="gpt-5.5", max_tool_rounds=5) if a flow needs more steps.

Putting It on the Phone

Now the outbound call. Twilio places the call and streams the audio to your agent over a WebSocket, and Vision Agents bridges that stream into the same Stream call the agent already runs on. The agent code above does not change. You are adding a thin telephony layer around it.

The flow for an outbound call is:

Register the call and mint a one-time token for the media stream.
Tell Twilio to dial the customer and point its audio at your websocket URL.
When Twilio connects, validate the token, attach the phone audio to the Stream call, and let the agent run the conversation.

import asyncio
import os
import uuid

import click
import uvicorn
from dotenv import load_dotenv
from fastapi import FastAPI, WebSocket
from twilio.rest import Client

from vision_agents.plugins import twilio

load_dotenv()

NGROK_URL = os.environ["NGROK_URL"]

app = FastAPI()
call_registry = twilio.TwilioCallRegistry()


async def initiate_outbound_call(from_number: str, to_number: str) -> str:
    twilio_client = Client(
        os.environ["TWILIO_ACCOUNT_SID"], os.environ["TWILIO_AUTH_TOKEN"]
    )
    call_id = str(uuid.uuid4())

    async def prepare_call():
        agent = await create_agent()
        phone_user = User(name=f"Cardholder {call_id[:8]}", id=f"phone-{call_id}")
        await agent.edge.create_users([agent.agent_user, phone_user])
        stream_call = await agent.create_call("default", call_id)
        return agent, phone_user, stream_call

    twilio_call = call_registry.create(call_id, prepare=prepare_call)
    url = f"wss://{NGROK_URL}/twilio/media/{call_id}/{twilio_call.token}"

    twilio_client.calls.create(
        twiml=twilio.create_media_stream_twiml(url),
        to=to_number,
        from_=from_number,
    )
    return call_id

The websocket handler takes over once Twilio connects. It validates the token, accepts the media stream, waits for the prepared agent, and bridges the phone into the call with attach_phone_to_call:

@app.websocket("/twilio/media/{call_sid}/{token}")
async def media_stream(websocket: WebSocket, call_sid: str, token: str):
    twilio_call = call_registry.validate(call_sid, token)

    twilio_stream = twilio.TwilioMediaStream(websocket)
    await twilio_stream.accept()
    twilio_call.twilio_stream = twilio_stream

    try:
        agent, phone_user, stream_call = await twilio_call.await_prepare()
        twilio_call.stream_call = stream_call

        await twilio.attach_phone_to_call(stream_call, twilio_stream, phone_user.id)

        async with agent.join(stream_call, participant_wait_timeout=0):
            await agent.simple_response(
                text="Introduce yourself as Meridian Bank's fraud line and ask the "
                "customer to confirm the flagged transaction."
            )
            await twilio_stream.run()
    finally:
        call_registry.remove(call_sid)

Finally, a small entry point starts the server and places the call. The token in the media URL is what stops a random WebSocket connection from hijacking a call, so it is validated before any audio flows:

async def run_with_server(from_number: str, to_number: str):
    config = uvicorn.Config(app, host="localhost", port=8000, log_level="info")
    server = uvicorn.Server(config)
    server_task = asyncio.create_task(server.serve())

    while not server.started:
        await asyncio.sleep(0.1)

    await initiate_outbound_call(from_number, to_number)
    await server_task


@click.command()
@click.option("--from", "from_number", required=True, help="Your Twilio number.")
@click.option("--to", "to_number", required=True, help="The number to call.")
def main(from_number: str, to_number: str):
    asyncio.run(run_with_server(from_number, to_number))


if __name__ == "__main__":
    main()

Place a call:

uv run outbound.py --from "+15551234567" --to "+15559876543"

Twilio sends audio as mulaw at 8kHz. The plugin handles the conversion to and from the format your pipeline expects, so you do not deal with codecs directly. Latency is higher in local development than in production. For a real deployment, run close to your users; on Stream's edge, that means deploying to the region nearest them.

What About Compliance?

A few things are worth saying plainly, and none of this is legal advice.

In the US, the TCPA restricts automated calls, but fraud-prevention calls from a financial institution have generally been treated as exempt under the FCC's rules because they are informational and time-sensitive rather than marketing. The rules around this shift periodically, so a real deployment needs a compliance review rather than a blog post's summary.

On the technical side, keep card data out of your transcripts and logs. The pipeline in this tutorial never captures a full card number because it never asks for one, which is the cleanest way to stay out of trouble: the safest sensitive data is the data you never collect. The tool signatures back this up by only accepting a transaction ID and a card's last four digits.

The behavioral rules in the instructions are not “polish you add at the end”. They are what make the call trustworthy to the person answering it. A bot that confirms a transaction and offers to let the customer call the number on the back of their card is behaving exactly like a real bank. A bot that asks for a verification code is behaving exactly like a scammer.

The gap between those two is the entire product.

Where to Go Next

You have an outbound fraud-alert agent that places a real phone call, confirms a transaction, and takes action on the result, with the trust boundary enforced in both the instructions and the tool signatures.

From here:

Swap providers to fit your market. The speech-to-text, LLM, and text-to-speech pieces are all one-line changes. Sarvam AI covers Indian languages, for example.
Add answering-machine handling for the outbound path so the agent leaves a callback message instead of talking to voicemail.
Move to a realtime speech-to-speech model if you want the lowest possible latency and can accept coarser transcripts.
Skip the scaffolding entirely. If you use Claude Code, Cursor, or Codex, Stream's Agent Skills let you drop the voice agent into your project and describe what you want in plain English:

npx skills add GetStream/agent-skills -s stream

The phone build in this tutorial is grounded in the framework's own phone example, which pairs telephony with a knowledge base if you want the agent to answer follow-up questions.

Start Building

Clone the repo, wire in your own fraud logic, and place a test call to your own phone: github.com/GetStream/Vision-Agents.

When you're ready to move past a test number, Stream's Voice AI is where this goes to production: hosted STT, LLM, and TTS on a 60+ location edge network, with a P50 voice round-trip around 240 milliseconds.

If you want your fraud agent deployed in the same region as your existing Stream chat and video, so a support call, a video session, and a fraud verification all run on one stack, join the hosted-agents waitlist. You can start building on the open-source framework right now either way.

Build a Marketplace Voice Shopping Agent with Kimi K2.5 and Vision Agents

Sarah Lindauer — Mon, 10 Aug 2026 15:47:00 +0000

A buyer opens your marketplace app and says: "Find me running shoes under $120, size 10, from a seller who ships in two days."

Text chatbots stall on requests like that. They need structured follow-ups, live inventory, and seller comparisons across multiple turns.

A voice agent with tool access handles it naturally.

This tutorial walks through a voice shopping concierge for a fictional multi-seller marketplace called Pronto Market. Kimi K2.5 powers the reasoning and function calling. Deepgram transcribes speech. ElevenLabs speaks the replies. Vision Agents orchestrates the pipeline over Stream's WebRTC edge, so you can test the whole flow in a browser in under 30 minutes.

What You'll Build

A conversational voice agent named Riley that:

Searches a multi-seller product catalog by query, category, price, and rating
Compares sellers on ship time, return policy, and reviews
Adds items to a cart after confirming size and price
Looks up order status by order ID or email
Runs in your browser via Stream, using the same CLI flow as the Vision Agents quickstart

Kimi has no native realtime speech API today, so we use a custom pipeline (STT → LLM → TTS) instead of a single speech-to-speech model. That tradeoff buys you two things Kimi excels at: reliable function calling across multi-step shopping flows, and the freedom to swap STT or TTS providers without rewriting your agent.

If you want a camera-and-vision take on Kimi instead of a voice-and-commerce one, see our earlier walkthrough on building a video agent with Kimi K2.5.

The catalog and order data live in a local Python module. That means a database, Stripe checkout, and external marketplace API aren’t required to get started.

The Stack

Layer	Choice	Why
LLM	Kimi K2.5 (`kimi-k2.5`)	Strong reasoning, native tool use, 256K context
STT	Deepgram	Fast transcription with eager turn detection
TTS	ElevenLabs	Natural, retail-friendly voice output
Transport	Stream WebRTC	Sub-500ms edge delivery, browser demo out of the box
Framework	Vision Agents	Provider-agnostic orchestration, open source

Audio flows: user speaks → Deepgram STT → Kimi K2.5 (with tools) → ElevenLabs TTS → user hears the reply.

A note on model versions: This tutorial uses kimi-k2.5, released January 2026. Moonshot has since shipped newer models (k2.6, k2.7) and rotates the k2 series over time. If kimi-k2.5 no longer resolves when you run this, check the Moonshot model list and swap the model string. The code stays the same.

Requirements

You'll need API keys from:

Moonshot Platform for MOONSHOT_API_KEY
Deepgram for DEEPGRAM_API_KEY
ElevenLabs for ELEVENLABS_API_KEY
Stream for STREAM_API_KEY and STREAM_API_SECRET

You'll also need Python 3.12 and uv installed.

Step 1: Scaffold the Project

uv init pronto-voice-shop && cd pronto-voice-shop
uv add "vision-agents[openai,deepgram,elevenlabs,getstream]" python-dotenv
uv add "getstream>=3.4.0,<3.5"

The last line pins the Stream Python SDK. Fresh installs currently resolve getstream 3.5.0, which breaks getstream.Edge() on vision-agents 0.6.x. Pin to 3.4.x until that compatibility issue is fixed upstream.

uv init creates main.py and pyproject.toml. We'll work in two files: catalog.py for the mock data and tools, and agent.py for the agent itself. Rename the generated file:

mv main.py agent.py

Create a .env file in the project root and add your keys:

# .env
STREAM_API_KEY=your_stream_api_key
STREAM_API_SECRET=your_stream_api_secret
MOONSHOT_API_KEY=your_moonshot_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
ELEVENLABS_API_KEY=your_elevenlabs_api_key

Vision Agents auto-loads these per plugin via load_dotenv(). MOONSHOT_API_KEY is required. The agent reads it directly in setup_llm() and will not start without it.

Step 2: Add the Mock Marketplace Catalog

Create catalog.py in the project root. This file holds sellers, products, a session cart, and sample orders. Everything is in memory, so the tutorial runs without external services.

# catalog.py

SELLERS: dict[str, dict] = {
    "peak-gear": {
        "name": "Peak Gear Co.",
        "rating": 4.8,
        "return_policy": "30-day free returns",
        "avg_ship_days": 2,
    },
    "trailhead-supply": {
        "name": "Trailhead Supply",
        "rating": 4.5,
        "return_policy": "14-day returns, buyer pays shipping",
        "avg_ship_days": 4,
    },
    "urban-run": {
        "name": "Urban Run",
        "rating": 4.6,
        "return_policy": "45-day returns on unworn items",
        "avg_ship_days": 1,
    },
}

PRODUCTS: list[dict] = [
    {
        "id": "boot-001",
        "name": "TrailRunner GTX",
        "category": "footwear",
        "description": "waterproof hiking boots gore-tex trail",
        "price": 129.99,
        "sizes": ["9", "10", "11"],
        "seller_id": "peak-gear",
        "rating": 4.7,
        "ships_in_days": 2,
    },
    {
        "id": "shoe-001",
        "name": "City Sprint",
        "category": "footwear",
        "description": "daily running shoes breathable mesh",
        "price": 89.99,
        "sizes": ["8", "9", "10", "11"],
        "seller_id": "urban-run",
        "rating": 4.6,
        "ships_in_days": 1,
    },
   {
        "id": "jacket-001",
        "name": "StormShell Rain Jacket",
        "category": "outerwear",
        "description": "waterproof packable rain jacket hiking",
        "price": 79.99,
        "sizes": ["S", "M", "L", "XL"],
        "seller_id": "trailhead-supply",
        "rating": 4.3,
        "ships_in_days": 4,
    },
    {
        "id": "pack-001",
        "name": "Daypack 28L",
        "category": "bags",
        "description": "day hiking backpack hydration compatible",
        "price": 59.99,
        "sizes": ["one-size"],
        "seller_id": "trailhead-supply",
        "rating": 4.2,
        "ships_in_days": 4,
    },
]

ORDERS: list[dict] = [
    {
        "order_id": "ORD-1042",
        "email": "alex@example.com",
        "status": "shipped",
        "tracking": "1Z999AA10123456784",
        "items": [{"product_id": "shoe-005", "size": "10", "quantity": 1}],
        "estimated_delivery": "2026-06-27",
    },
    {
        "order_id": "ORD-1038",
        "email": "jamie@example.com",
        "status": "processing",
        "tracking": None,
        "items": [{"product_id": "boot-001", "size": "11", "quantity": 1}],
        "estimated_delivery": "2026-06-29",
    },
]

CART: list[dict] = []


def _product_by_id(product_id: str) -> dict | None:
    for product in PRODUCTS:
        if product["id"] == product_id:
            return product
    return None


def search_products(
    query: str,
    category: str = "all",
    max_price: float = 500.0,
    min_rating: float = 0.0,
) -> list[dict]:
    query_lower = query.lower()
    results: list[dict] = []

    for product in PRODUCTS:
        if category != "all" and product["category"] != category:
            continue
        if product["price"] > max_price:
            continue
        if product["rating"] < min_rating:
            continue

        haystack = " ".join(
            [
                product["name"],
                product["description"],
                product["category"],
                SELLERS[product["seller_id"]]["name"],
            ]
        ).lower()

        if query_lower in haystack or any(
            word in haystack for word in query_lower.split() if len(word) > 2
        ):
            seller = SELLERS[product["seller_id"]]
            results.append(
                {
                    "id": product["id"],
                    "name": product["name"],
                    "price": product["price"],
                    "sizes": product["sizes"],
                    "seller": seller["name"],
                    "seller_id": product["seller_id"],
                    "rating": product["rating"],
                    "ships_in_days": product["ships_in_days"],
                }
            )

    results.sort(key=lambda item: (-item["rating"], item["ships_in_days"]))
    return results[:5]


def get_seller_info(seller_id: str) -> dict:
    seller = SELLERS.get(seller_id)
    if seller is None:
        return {"error": f"Unknown seller: {seller_id}"}
    return {
        "seller_id": seller_id,
        "name": seller["name"],
        "rating": seller["rating"],
        "return_policy": seller["return_policy"],
        "avg_ship_days": seller["avg_ship_days"],
    }


def add_to_cart(product_id: str, size: str, quantity: int = 1) -> dict:
    product = _product_by_id(product_id)
    if product is None:
        return {"error": f"Product not found: {product_id}"}
    if size not in product["sizes"]:
        return {
            "error": f"Size {size} not available. Available: {product['sizes']}"
        }

    line = {
        "product_id": product_id,
        "name": product["name"],
        "size": size,
        "quantity": quantity,
        "unit_price": product["price"],
        "line_total": round(product["price"] * quantity, 2),
    }
    CART.append(line)

    return {
        "cart_size": len(CART),
        "added": line,
        "cart_total": round(sum(item["line_total"] for item in CART), 2),
    }


def get_order_status(order_id: str = "", email: str = "") -> dict:
    for order in ORDERS:
        if order_id and order["order_id"].lower() == order_id.lower():
            return order
        if email and order["email"].lower() == email.lower():
            return order
    return {"error": "No order found. Check the order ID or email and try again."}

Step 3: Register Tools on Kimi K2.5

Replace the contents of agent.py with the code below. The setup_llm() function wires Kimi through the OpenAI-compatible Chat Completions API and registers four marketplace tools.

# agent.py
import os

from dotenv import load_dotenv
from vision_agents.core import Agent, AgentLauncher, Runner, User
from vision_agents.plugins import deepgram, elevenlabs, getstream, openai

import catalog

load_dotenv()

MARKETPLACE_INSTRUCTIONS = """\
You are Riley, a voice shopping assistant for Pronto Market, a multi-seller marketplace.

Voice rules:
- One or two short sentences per turn. Max 25 words.
- Ask one clarifying question at a time.
- Before adding to cart, confirm product, size, and price.
- When comparing sellers, mention rating and ship time, not long feature lists.

Use tools to search, compare sellers, update the cart, and check orders. Never invent inventory.
"""


def setup_llm() -> openai.ChatCompletionsLLM:
    llm = openai.ChatCompletionsLLM(
        model="kimi-k2.5",
        base_url="https://api.moonshot.ai/v1",
        api_key=os.environ["MOONSHOT_API_KEY"],
    )

    @llm.register_function(
        description=(
            "Search marketplace products by query, category, max price, "
            "and minimum seller rating"
        )
    )
    async def search_products(
        query: str,
        category: str = "all",
        max_price: float = 500.0,
        min_rating: float = 0.0,
    ) -> list[dict]:
        return catalog.search_products(query, category, max_price, min_rating)

    @llm.register_function(
        description="Get seller profile: rating, return policy, average ship time"
    )
    async def get_seller_info(seller_id: str) -> dict:
        return catalog.get_seller_info(seller_id)

    @llm.register_function(description="Add a product to the buyer's cart")
    async def add_to_cart(product_id: str, size: str, quantity: int = 1) -> dict:
        return catalog.add_to_cart(product_id, size, quantity)

    @llm.register_function(
        description="Look up order status by order ID or buyer email"
    )
    async def get_order_status(order_id: str = "", email: str = "") -> dict:
        return catalog.get_order_status(order_id, email)

    return llm


async def create_agent(**kwargs) -> Agent:
    return Agent(
        edge=getstream.Edge(),
        agent_user=User(name="Riley, Pronto Market", id="agent"),
        instructions=MARKETPLACE_INSTRUCTIONS,
        llm=setup_llm(),
        stt=deepgram.STT(eager_turn_detection=True),
        tts=elevenlabs.TTS(),
    )


async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        await agent.simple_response(
            text=(
                "Greet the buyer warmly. You are Riley from Pronto Market. "
                "Ask what they are looking for today."
            )
        )
        await agent.finish()


runner = Runner(AgentLauncher(create_agent=create_agent, join_call=join_call))


if __name__ == "__main__":
    runner.cli()

Kimi K2.5 supports function calling with automatic tool invocation. When a buyer asks about boots under $150, Kimi calls search_products, reads the results, and speaks a short summary back through ElevenLabs. See the Kimi integration docs and function calling guide for more patterns.

Step 4: Run It

Start the agent from the project directory:

uv run agent.py run

The CLI prints a join link. Open it in your browser, allow microphone access, and start talking.

Try this script:

"I'm looking for waterproof hiking boots under $150."
"Which seller ships fastest?"
"Add size 10 to my cart."
"Where's order ORD-1042?"

Example dialogue:

Riley: Hi, I'm Riley from Pronto Market. What are you shopping for today?

You: Waterproof hiking boots under $150.

Riley: I found three options. TrailRunner GTX from Peak Gear is $129.99, ships in two days. Want size 10?

You: Which seller ships fastest?

Riley: Urban Run ships in one day, but they sell running shoes, not boots. For boots, Peak Gear ships fastest at two days.

That back-and-forth is Kimi chaining tool calls across turns while keeping replies short enough for voice.

From Demo to Production

The mock catalog in catalog.py is a stand-in for your real backend. In production, swap each tool function for calls to your own services:

search_products → catalog search API (Algolia, Elasticsearch, or your OMS)
get_seller_info → seller profile service
add_to_cart → cart session API (the same endpoints you'd expose to a text agent)
get_order_status → order management system

Text-based agentic commerce is standardizing around protocols like the Agentic Commerce Protocol (ACP) and Google's Universal Commerce Protocol (UCP). Voice fits the same architecture: the input modality changes, but the checkout session, inventory lookup, and order tracking stay the same. Your voice agent calls the same REST or MCP endpoints a chat agent would.

To reach buyers outside the browser:

Phone: extend this agent with Twilio phone integration
Mobile app: join the same Stream call from Stream's client SDKs
Managed hosting: Stream Voice AI runs production voice agents on Stream's global edge with co-located STT, LLM, and TTS. Join the waitlist for early access.

Why Kimi K2.5 for Marketplace Voice

Multi-tool reasoning. A single shopping request often needs search, seller comparison, and cart update in sequence. Kimi K2.5 handles that reliably.

256K context window. You can load full seller policy documents, return FAQs, or category guidelines into the system prompt without setting up RAG first. Useful when marketplace rules vary by seller.

OpenAI-compatible API. Kimi plugs into Vision Agents through openai.ChatCompletionsLLM with a custom base URL. Same pattern works for other OpenAI-compatible providers if you want to benchmark latency or cost.

Room to grow. The same Agent class supports video processors later. A follow-up tutorial could let buyers hold up a product to the camera and ask "do you have this in size 10?"

We like Kimi for marketplace flows because the model stays grounded when tool results come back mid-conversation. It summarizes inventory instead of inventing products, which matters when money is on the line.

A tuning note: Moonshot's published guidance for K2.5 is temperature 1.0 for thinking mode and 0.6 for instant mode, with top_p 0.95. The code above doesn't set temperature, so you get the provider default. If you want shorter, more deterministic replies for voice, start at the instant-mode setting and test from there.

Give it a try, swap in your catalog API, and let Riley handle the voice layer while Kimi does the reasoning.

Star Vision Agents on GitHub if this helped. Pull requests and issue reports welcome.

How To Build a Social Media App: Types, Features, Monetization

Sarah Lindauer — Fri, 20 Mar 2026 20:32:54 +0000

People leave social media apps for many reasons, such as a lack of trust in leadership, to improve mental health, or because of political alignment. Many are just looking for a more authentic social platform. To be that platform, you'll need to start with a roadmap and build your social media network from the ground up.

Let's take a high-level overview of what you need to get started.

Overview of the Social Media App Market

Various social media platforms have come and gone over the years, with some staying around long term, like Snapchat, and others becoming popular before losing steam, like Clubhouse. As of winter 2023, older platforms like Facebook, Instagram, and YouTube are still on top, but TikTok is growing fast. Twitter usage has fluctuated in recent years, dipping in 2018 and 2019 before seeing growth again, while Pinterest usage declined recently.

Business of Apps says 4.88 billion people worldwide have social media accounts, and smartphone users spend 70 hours using social media apps monthly. According to a report by Grand View Research, Inc., the social networking app market is projected to rise to $310.37 billion US by 2030. It's expected to have a compound annual growth rate of 26.2% from 2023 to 2030. Grand View Research attributes this to an increase in demand for digital marketing services globally. It's also due to increased social media adoption from all age groups, increased use of smartphones, and increased internet access.

Musk's Twitter purchase sent people away from the app, including prominent users, though many searched for alternatives to join. In the wake, alternative online communities have gained users. Some growth has been at a sustainable pace, like Discord and Reddit, and some has been temporary, like the spike seen at Mastodon, which slowed in the months following.

Among the top social media trends are user-generated content and highly personalized content. TikTok excels at both of these since users primarily create and share video content, and the app's #ForYou page is the main way users discover content. TikTok's algorithms excel at helping users find content that resonates with them and feels genuine.

Monetization of user accounts is keeping them on platforms like TikTok and Facebook that pay them for the number of views their content garners as more people look to earn money on social media. Influencer marketing continues to grow as a marketing tactic for many companies that have found authenticity is key.

Types of Social Media Apps

Each type of social media application has its own unique audience and purpose. For example, Facebook attracts older users for social networking, while Snapchat targets younger users and is designed for photo sharing.

Social Networks | Examples: Facebook, LinkedIn

Considered traditional or legacy apps, social networks allow people to interact in a multitude of ways, through profiles, activity feeds, posts, photo and video content, groups, and more. Sites like these are some of the most successful because they've continuously adapted over time while continuing to grow. People use Facebook to connect with faraway family and friends, and they use LinkedIn to build and maintain their professional network.

Photo Sharing | Examples: Instagram, Snapchat

Apps like Instagram allow users to share photos with captions and filters, as well as infographics and direct messages. Instagram launched at a time when mobile phone cameras became better, and Snapchat created something entirely new with ephemeral content that disappears within 24 hours. In recent years, Instagram and other photo apps have incorporated more video to compete with apps like TikTok.

Video Sharing | Examples: YouTube, TikTok

Some video-sharing apps are better for longer videos, like YouTube and Vimeo, while others focus on short-form content, like TikTok. YouTube is one of the oldest video-sharing platforms, making it extremely well known. It's accessed by more than 2.5 billion people each month. TikTok's success comes from its interactive capabilities and its #ForYou page algorithm that keeps users scrolling for hours. The app has more than 3 billion downloads.

Content Sharing | Examples: Pinterest, SlideShare

Content-sharing apps provide a place for users to share multiple types of content, but they typically lean toward informational content in visual formats. Pinterest serves as a virtual pinboard where users add links, images, and photos from around the internet. They can also upload their own content, like how-tos, recipes, and infographics. People gravitate toward Pinterest for planning and organizing their thoughts, like gathering ideas for a birthday party, inspiration for a kitchen renovation, or tips on childcare. SlideShare users upload professional presentation decks, making it a strong contender for business use.

Blogging/Microblogging | Examples: Medium, Twitter

Blogs and microblogs are primarily text-based but include photos, videos, and links as well. On sites like Medium, users can publish articles or blogs with media embedded. Many professionals use the site to share long-form thought leadership content. Twitter, an early social media app, provides a place for users to have conversations, publicly or privately, in media attachments and a maximum of 280 characters.

Post/Forum | Examples: Reddit, Quora

Forum-type sites serve as a place to have conversations and connect mainly through text posts and comments. Reddit, founded in 2005, allows users to join communities where moderators help guide discussions on a number of topics, from current events to personal finance to favorite television shows. Question-and-answer platform Quora draws many professionals who have the know-how to answer questions and share knowledge about their areas of expertise.

Messaging | Examples: WhatsApp, Discord

Similar to text or SMS messages, social media messaging services are mostly used for sending text messages but also let users share photos, videos, gifs, audio, and more. WhatsApp grew in popularity because it offered a way for users to message and call each other for free, including internationally. Discord drew a large audience of gamers and gave them a space to create online gaming communities.

The Social Media App Build Process

When you align the process of building your own social media app with the phases of project management, the roadmap to launching your product includes five phases.

Phase 1: Project Initiation

Document the big picture for your app idea. What niche will your social media app fill, or what problem will it solve for its users? You can do this with a product vision board or a product requirement document. These techniques help you outline your business goals, stay focused on your vision, and determine the target audience for your app.
Start your go-to-market strategy. Develop an ideal customer profile and research your competitors. Begin thinking about how you'll position and market your app and what tactics might be best to promote it.

Phase 2: Project Planning

Make key development decisions. Will you develop your social media app for iOS or Android? What development framework and tech stack will you use? What features do you want your app to have? Answering these questions will help you determine how long it will take to create your app and what resources you'll need. (We'll look at these decisions in depth later.)
Choose your project management methodology. Waterfall, Kanban, Agile, or something else? Each method has pros and cons. Consider the outcomes you'd like to achieve with your social media app.
Create your project timeline and budget. You'll be able to estimate timing and costs based on your selections to this point. For example, if you've opted to build an app for Apple iOS, you can include a line item in your budget for the developer membership fee.

Phase 3: Project Execution

Create an admin panel. The admin panel will serve as your operations headquarters to control your app. It's a space where you can view user activations, ban users who violate your terms of service, oversee app features, and more.
Design your mobile app. Create a wireframe for your app to map out its functionality using best practices for UI/UX design. Consistency in the design and simple, intuitive navigation help keep users coming back.
Complete app development and quality assurance. Hand your project plans and design to the team who will develop your social media app. Complete QA testing to address bugs. Social media users will jump to another platform when one isn't working correctly.

Phase 4: Project Monitoring and Controlling

Track project timeline, progress, and costs. Maintain documentation on the progress of your app throughout, so you will have a thorough history of what went smoothly and what didn't. You can use that information going forward when making updates and adding features to your social media app.
Adjust for unexpected delays, issues, and cost overruns. Modify your timeline and budget to accommodate changes to your product roadmap. There may be outside influences, like new legislation, or trends among competitors that prompt the adjustments.

Phase 5: Project Closing

Launch your app and wrap up the project. Release your social media app to the world and work on increasing downloads and building an active user base. Wrap up your project by making sure all related expenses are settled. Then archive all project plans, budgets, and documents.
Gather feedback from the team and users. Feedback helps you improve your social media app. You can also use this information on future projects or updates to your app.

Considerations for the Application Development Phase

The decisions you make during the development phase will impact which technologies you use — from the backend development to the frontend user experience to which app store it's available through.

Pick Your Platform

Startups typically develop their app for either Apple iOS or Android first — not both. The choice will depend on factors like your timeline and target market. While Android usage leads the way globally, iOS usage leads in countries like Japan and the United States.

Pin Down Your Development Framework

Selecting the type of development framework to use will depend on your team's skills and your budget.

SaaS — Many SaaS companies can help you build an app quickly with pre-built features. These services are DIY, so customization options will be limited.
CMS — Community management software allows you to create online community spaces that you can control. While they can be useful for brands that already have a loyal following, they lack customization capabilities.
Open-source — Open-source software is pre-built software that offers a lot of flexibility. You can even find no-code or low-code options, and some software is free.
Develop from the ground up — To develop an app from scratch, you'll need to hire skilled developers or an app development company. This is an expensive option but gives you the ability to develop a custom app to suit your specific needs.

Choose Your Tech Stack

Your platform choice and development framework will help you narrow down the rest of your technology stack for your app. If you're not using an agency to create your app, closely evaluate your options for APIs, plugins, and chat SDKs. These can be used to connect to app features like sign-in, chat, geolocation services, and more.

The right external tech solutions can help you increase efficiencies and cut down your app development cost and timeline. Consider the limitations of the technology, cost, and scalability. Technology that's limited or can't scale to the size of your user base will result in a bad user experience and harm retention.

You'll also need a database, such as MySQL or Amazon RDS, and a storage solution, like Amazon AWS or Google App Engine. Capabilities, cost, and scalability are equally important for your database and storage.

Select Your App Features

Start with the basic social media app features. Every social media network includes user profiles and activity feeds. Then consider what must-have features and advanced features will attract your target market and boost user engagement.

User profiles are an area where users can provide basic information about themselves, their company, or their organization.
Newsfeeds, or activity feeds, provide a list of actions that other users have taken and are updated in real time.
Content will depend on the type of social media network you create and can include text, images, video, or external links.
In-app content creation tools help users create content from scratch without having to leave the app.
Augmented reality within social media lets users apply filters or effects to images.
Search bar functionality on social media helps users find each other and find publicly viewable content about specific topics.
Social SEO allows users to use SEO strategies to help make their profiles more visible in search engines.
Chat/messaging services allow users of your app to connect privately and in small groups.
Groups/discussion forums provide a place for users to connect with others on special interest topics.
Push notifications send alerts to your users to let them know there is a new update or message.
Security and safety features, such as multi-factor authentication, keep users' information secure.

How to Monetize Your App

Part of creating a successful app is finding revenue streams to tap into. The monetization strategies you focus on will depend on the type of site you're creating and your target users.

Advertising

Digital advertising is standard for social media apps at this point. Even small advertising budgets can make a big difference for a small business on social media, as long as the algorithms for advertising are effective at pushing the ads to the right social media users.

A self-serve platform makes it easy for business users to create and place their own ads. You'll also want to offer meaningful ad metrics, like reach, impressions, and engagement rate.

In-app Purchases

In-app purchases might include subscriptions, special filters, exclusive features, points or tokens for games, or rewards or gifts that you can give content creators.

Usage of in-app purchases can be very specific to the type of app you offer. On the video app TikTok, users can purchase gifts to give to other content creators. On Instagram, users can subscribe to exclusive content created by users who've opted to monetize their accounts. The app, in turn, takes a percentage of the money from the transaction or charges the monetized user a monthly fee.

Subscriptions

The subscription model is one that's in an evolving state as apps continue to look for ways to earn revenue. Some apps, such as Raya, are exclusive, invite-only social networks. Tumblr offers a subscription to use the platform ad-free. On LinkedIn, you can subscribe as a premium member, which gives you access to view additional job applicant information, direct messaging features, and learning courses.

In November 2022, after Musk's purchase of Twitter, he launched an account verification subscription service that allowed anyone to buy a verified account with a blue checkmark. The service was rolled back within 48 hours after users created fake accounts that looked authentic and caused chaos so extreme that it impacted the stock markets. For $8, someone purchased a verified account and spoofed the pharmaceutical company Eli Lilly. After the spoof account tweeted that insulin would be free, Eli Lilly's stock fell.

Twitter's idea led other social media networks to explore similar strategies. In February 2023, Facebook announced a verification program that verifies a user's identity and offers customer support access, protection from impersonators, and other features.

Partnerships

Partner programs can be a lucrative way to encourage people to join your app and continuously create content. YouTube's Partner Program is available to users with at least 1,000 subscribers and 4,000 hours of videos they uploaded being watched. YouTube pays qualifying users who opt in a percentage of the ad sales for their videos.

You can also partner with prominent, highly active social media users and brands to create promotional content for your app. Promotions from influencers help them grow their follower count and give you visibility with new users. Just be sure that both you and your partners follow all FTC disclosure requirements.

FAQs: How To Build a Social Media App

How much does it cost to create a social media app?

Cost can vary from a few thousand dollars using out-of-the-box/premade software and plugins to half a million or more for a custom-built platform with sophisticated proprietary technology. The discovery phase will help you learn which options align with your budget and best fit your vision.

How long does it take to build a social media app?

For a simple platform, it can take just a few weeks to build out your app, while a more complicated app with a lot of features can take six months or more.

Is it hard to create a social media app?

While there are simple ways of creating social media apps, making them a success can be difficult. Your app needs to solve a problem for its users that makes it compelling to use and come back to. It needs to function well and be user-friendly, which takes the right team to make happen.

Is having your own social network profitable?

It can be. The key to creating a social media network that's profitable is gaining enough active users. The higher the number of users you have, the easier it is to charge business users for advertising and partnerships.

What's the safest social media app?

This depends on what you mean by safest. If you're referring to privacy on social media, apps like Signal and MeWe have a heavy focus on protecting users' privacy. Signal, a messaging app similar to WhatsApp, has end-to-end encryption. MeWe has a strong privacy policy and no advertising. When considering safety from the perspective of online harassment and abuse, sites like Twitter and Instagram allow you to create private accounts and easily block other users.

Can I make an app with no experience?

Absolutely. You should understand the basics of the process and of the social media market, but the development process doesn't require experience.

Do I need to know coding to create an app?

You can use services that allow you to build apps without coding by choosing features that are already built. Or, you can build a team that has the knowledge to build the app for you or even hire a company that specializes in social networking app development.

Which social media app makes the most money?

Facebook earns the most revenue of social media apps, with $116.6 billion in revenue in 2022.

Social Media App Development for Startups

There are many paths you can take to build your social media app, but you don't have to learn how to code your app design from scratch or hire a development team. You can integrate important core features like chat and activity feeds easily with Stream. With this basic functionality in place, you can focus your efforts on the best ways to build the rest of your app and make it a success.

Best Visual AI Agents in 2026: Real-Time & Multimodal Tools

Sarah Lindauer — Tue, 03 Mar 2026 23:03:09 +0000

Chatbot integration in popular software has become so widespread that it no longer offers a meaningful competitive edge. The real challenge now is moving beyond simple text interfaces to build products that can perceive the world as it is and carry out meaningful tasks.

Visual AI agents give this edge by combining computer vision with agentic reasoning to perform tasks with little to no input from the user.

This guide will go deeper into what AI agents are, some of the top picks, and supporting architecture, as well as how to choose the best visual AI agent(s) for your organization.

What Are Visual AI Agents?

Visual AI agents are intelligent and autonomous systems that can plan, reason, and make decisions using visual information from photos, videos, and live feeds. What sets visual agents apart from existing computer vision systems, such as traditional visual search systems, is their ability to act on contextual information.

Their core capabilities center around functions like object detection, spatial reasoning, multimodal understanding, and image-to-action.

By merging computer vision with agentic AI, these systems can act on visual context with minimal explicit instruction. One common consumer-facing use case is hands-free, conversational interactions with built-in AI assistants in smart glasses.

Here are some broad categories of visual AI agents to illustrate other uses:

Robotics/perception agents control objects in real-world settings like autonomous vehicle navigation, real-time object recognition and manipulation, and surveillance and escalation.
Creative visual agents generate and modify visual content. Examples include digital design assistants, automatic media post-production, and style transfer and editing.
Analytical agents extract information from visual input to make decisions. They’re used for medical imaging analysis, retail shelf monitoring and footfall tracking, and sports coaching.

The 4 Best Visual AI Agents

Many of the most powerful AI models are multimodal, allowing them to accept inputs in different forms, such as visual data, text data, and files. Some of our picks for the best visual AI agents aren't purpose-built to ingest purely visual data, but they happen to excel at it.

With that out of the way, let’s look at some of the best visual AI agents on the market.

Amazon Bedrock Agents

Bedrock Agents can be configured to work as visual agents by using foundation models with an orchestration layer that translates visual data into tool calls. It can ingest video and photos using Kinesis, M3U8, and S3.

Agents can be built using the AWS console. They’re deployed and maintained using AgentCore. Agents access actions through action groups that contain executable functions or MCP-enabled tools.

The capabilities of these functions range from simple notifications on event triggers to controlling IoT devices and sending API requests.

Pros

AWS Integration: Any compatible AWS service can be layered with Bedrock Agents, resulting in a high degree of extensibility.
Traceability: Every step taken by an agent produces a trace. These traces outline the reasoning of the agents, the inputs given, the functions used, and the output received.
Universal UI Controls: Supports direct computer use to imitate human interaction with software that doesn't allow agentic actions.

Cons

Ecosystem Lock-In: Bedrock Agents ties you tightly to AWS, making it difficult to migrate agent workflows to other platforms without a total rebuild.
Enterprise Pricing: Because it is meant largely for enterprise use, the cost to run Amazon Bedrock can be out of reach for smaller organizations and startups.

Google Gemini

Google Gemini can be used as a visual agent, as it combines multimodal perception along with reasoning to act on what it sees.

It uses Vision-Language-Action capabilities to translate visual data directly into low-level commands (like motor movements or mouse drags), while also being capable of high-level orchestration. By natively calling functions and tools, the agent can see a video (such as a specific error on a screen or a product defect in a livestream) and execute logic to fix it.

To use Gemini as a visual agent, use the Observer-Think-Act loop using the Gemini API or Multimodal Live API.

For static images or recorded video, the media is sent along with a tool definition, which results in a function call. For live feeds, the agent processes frames in real-time to trigger immediate actions while maintaining context through “Thought Signatures” that preserve its train of thought across sessions.

Pros

Universal UI Controls: Navigates and controls any visual interface without official APIs or HTML scraping.
3D Spatial Awareness: Gemini can output 3D bounding boxes and trajectories, allowing it to work well with AR/XR and robotics.
Bidirectional Streaming: The Multimodal Live API allows the model to see a video stream and trigger function calls, like trigger_alert() or log_data(), as events unfold in real-time.

Cons

Resource Intensive: High-resolution video and frequent screenshot loops consume tokens rapidly, seeing as it isn't priced specifically for ingesting visual input.
Provider Overload: The popularity of Google Gemini leads to occasional processing overload, which can break autonomous loops mid-task.

AskUI Vision Agent

AskUI Vision Agent is a specialized GUI-focused visual agent that works at the operating system level. Unlike more general-purpose models, AskUI is purpose-built to perceive mobile and desktop screens to interact with them exactly as a human would by taking control of input devices.

AskUI treats the entire device screen as a live coordinate system. It employs a computer-use architecture, where it takes a screenshot, identifies UI elements visually (like buttons, text fields, and icons), and maps those elements to physical actions.

Developers can integrate this agent by using the AskUI Python SDK or Typescript library. The first step is to create a “Controller” that bridges the AI to your OS. After that, intent-based commands are written (like agent.click(“Login”)), and the agent handles the rest.

Pros

Universal UI Controls: Due to its computer-use functionality, AskUI can access software that does not support agentic communication.
Low Cost of Operation: The SDK/CLI of AskUI is completely free, and it can use natively-hosted LLMs to avoid API fees.
Local-First Execution: Because the controller runs locally on your machine, it can automate highly secure, offline, or air-gapped environments where cloud-based agents might be restricted.

Cons

Visual Fragility: Since the only input is captured screenshots, UI changes like refreshes and unexpected pop-ups can break coordinate mapping.
Low Flexibility: AskUI is strictly meant for UI operations, so it can’t perform agentic functions with camera feeds or other visual input.

NVIDIA Metropolis

NVIDIA Metropolis is an enterprise-grade vision AI application platform designed to build and scale visual agents across edge and cloud environments, including devices like cameras and robots.

Metropolis is a full-stack engineering ecosystem for physical spaces. It provides the specialized SDKs, microservices, and blueprints needed to turn video feeds into agentic actions in industries like manufacturing and retail, as well as in smart city deployments.

Metropolis connects high-level vision language models (VLMs) with low-level sensor data. It uses models to analyze video at very high fidelity, with the NVIDIA Cosmos reasoning model reaching over 96% accuracy in a wafer map defect classification test.

Unlike standard LLMs and vision models that work one frame at a time, Metropolis uses tools like Multi-Camera Tracking to follow an object across 3D space, maintaining the state of the agent’s task as the subject moves.

Metropolis uses a “Microservice Pipeline” with the following components:

Ingestion (Video Storage Toolkit): Manages live RTSP streams from multiple cameras.
Inference (DeepStream/NIM): Runs the visual models on NVIDIA GPUs to extract real-time insights.
Agentic Logic (NVIDIA AI Blueprints): Provides reference code for Video Search and Summarization. This allows agents to answer natural language queries and perform multi-step planning.
Edge Computing: Agents deploy onto NVIDIA Jetson hardware, allowing the AI to act locally even if the internet goes down.

Pros

Edge-to-Cloud Flexibility: Can run entirely on-site (on Jetson Orin) for execution with zero network latency or scale to the cloud for massive video archives.
Digital Twin Training: Uses NVIDIA Omniverse to train visual agents in a virtual world before deploying them to the real world.
High Throughput: Optimized specifically for NVIDIA hardware, which claims to be able to process video 30 times faster than real-time analysis.

Cons

Specialization: An efficient implementation requires specialization in NVIDIA’s accelerated interface stack and hardware-aware AI deployment.
Hardware Lock-In: Requires specialized NVIDIA GPUs to run the software stack, like A100s, H100s, and Jetsons.

Comparison Table

Platform	Description	Ideal Use Case
Amazon Bedrock Agents	AWS-managed custom agents that translate visual data into tool and API actions using action groups and foundation models.	Enterprise workflows that combine with AWS services and automation.
Google Gemini	Multimodal AI that reasons over images and video to directly execute actions.	General-purpose visual reasoning, UI control, and live visual monitoring.
AskUI Vision Agent	OS-level visual agent that automates software by interacting with screens like a human.	Desktop/mobile UI automation where APIs are unavailable.
NVIDIA Metropolis	Full-stack vision AI agent platform for analyzing live camera feeds and physical environments.	Smart cities, factories, retail analytics, and large-scale camera networks.

Infrastructure Powering Visual Agents

While out-of-the-box agents are powerful, many specialized use cases require custom-built solutions. This involves assembling a stack of supporting technologies to bridge the gap between vision and execution.

Vision-Language-Action Foundation Models

The reasoning engine for modern visual agents is ‌vision-language-action models. These models are specifically trained to give outputs as actions instead of text or speech responses.

Models like InternVL3 and NVIDIA’s Cosmos-based GR00T are trained to ground their reasoning in spatial coordinates, allowing them to point to options directly from visual feeds. These models enable agents to understand complex instructions like “turn off the machine when the light turns red” and translate them into actions.

Multi-Agent Orchestration Tools

Complex visual tasks often require agent teams, consisting of specialized agents rather than a single monolithic model. Orchestration frameworks, like LangGraph, CrewAI, and Microsoft AutoGen, manage these collaborations, where one agent might focus on high-speed object detection (perception) while another handles long-term planning (reasoning).

These tools ensure that state is maintained across tasks, allowing agents to remember a visual context even as the camera view changes or the task evolves over time.

Real-Time Streaming Infrastructure

To function in the real world, visual agents require a live paradigm that employs bidirectional streaming. Frameworks like Vision Agents make this practical by leveraging low-latency edge transport layers (such as Stream’s global edge network) and real-time video/audio pipelines to enable continuous ingestion of visual data.

Similarly, StreamingVLM architectures enable agents to process unbounded video feeds by using specialized memory caches. This infrastructure makes agents situationally aware, treating live video as a continuous, unified context rather than a series of disconnected snapshots.

Robotics & Edge Control Platforms

Visual agents might be embedded into individual devices for certain tasks, but they can also run on control platforms for complex deployments that involve several robots or edge devices. For example, a centralized agent can use a warehouse’s cameras to optimize pallet placements before sending pathing commands to delivery robots via the platform.

Compatibility and capabilities will vary by platform, but three popular open-source choices are:

AWS IoT Greengrass: An AWS service for edge devices that can use Bedrock Agents for scenarios like agricultural fleet control.
NVIDIA Isaac: A robotics development platform that tightly integrates with Metropolis for digital twin training with Isaac Sim.
Viam: A robotics and edge control platform that is a little more complicated for agent setup but is hardware-agnostic, costs nothing to start, and has premade modules for integrating with Gemini, ChatGPT Vision, and more.

How to Evaluate the Best Visual AI Agent

Choosing the right agent requires an understanding of its performance metrics, operational costs, and safety protocols.

Let’s look at some of the most important metrics to evaluate a visual AI agent.

Model Flexibility

Visual AI agents are often powered by multiple models across different tasks. General-purpose models are good at open-ended reasoning and scene understanding, while specialized models often outperform them in latency-sensitive or narrowly defined tasks.

Model flexibility refers to an agent’s ability to route different stages of perception and reasoning to the most appropriate model, rather than forcing all workloads through one monolithic architecture. This is especially important in streaming environments, as it allows agents to choose between latency, reasoning depth, cost, and time constraints dynamically.

Latency vs Intelligence

Visual agents often need to make split-second decisions, but this comes at the cost of a lower reasoning depth. Low-latency agents are essential for physical tasks like robotics or security monitoring, where sub-second responses are required. The quicker models have fewer parameters (1B-11B), and they usually run on the system's edge.

High-intelligence agents take their time while making decisions, which comes in handy for tasks like complex GUI navigation or medical image analysis. These typically rely on larger, cloud-hosted models that can take several seconds to think through a visual scene.

Cost Tradeoffs

To evaluate the total cost of ownership, it's a good idea to compare the per-action LLM API costs of cloud providers against the infrastructure overhead of self-hosted models. Many organizations use a tiered cost model, which uses a smaller, cheaper model for routine monitoring tasks, and escalation to an expensive model occurs only when a visual anomaly is detected.

Human-in-the-Loop Workflows

For high-stakes decisions, such as authorizing a financial transaction or approving a medical diagnosis, a visual AI agent should support human-in-the-loop checkpoints. Some agents use confidence gating, which asks for human guidance if the model’s confidence score falls below a certain threshold.

FAQs

1. Which AI Has the Best Agents?

It’s impossible to definitively say which AI has the best agents overall for two reasons:

Performance Varies by Task: A given agent may excel in some areas but be outperformed in others. AskUI Vision Agents is one of the best for workflow automations, but it’d be a poor choice for a shopping agent.
Frequent Updates and Upgrades: AI companies are constantly improving their products, so the agent that scores the highest on a benchmark in March may lose to an updated competitor model in June.

2. What Exactly Is Visual AI?

Visual AI is the use of computer vision in AI systems, which allows models to understand information present in images and videos.

3. Is Siri an AI Agent?

Apple’s Siri can be described as an AI agent as it can perform tasks and make decisions based on commands. However, Siri doesn't make proactive automated decisions without instruction from the user.

4. What Are Level 3 AI Agents?

Level 3 AI agents use LLM reasoning and orchestration frameworks to make decisions and perform multi-step tasks without human intervention.

5. What Is the Difference Between LLMs and AI Agents?

LLMs are AI systems that can understand and respond in natural language. AI agents use LLMs along with tools, reasoning, and knowledge-base lookups to perform actions based on events or natural language requests.

What Visual AI Agent Should Your Organization Use?

While deciding, it’s important to remember that the right visual AI agent depends on your organization’s technical requirements rather than on general popularity. A team building a real-time golf coach will have different priorities than one working on a manufacturing quality control system.

Here are our recommended use cases for the visual AI agents mentioned in this guide. You should use:

Amazon Bedrock Agents for highly customizable enterprise workflows that have deep integration with existing AWS services and automated tool-calling.
Google Gemini if you need a versatile, general-purpose multimodal agent capable of sophisticated reasoning over live video and direct UI control.
AskUI Vision Agent for cross-platform desktop or mobile workflow automation, especially when you need to interact with software that lacks accessible APIs.
NVIDIA Metropolis for tasks involving large-scale camera networks where performance and reliability are essential.

The 8 Best Platforms To Build Voice AI Agents

Sarah Lindauer — Tue, 03 Mar 2026 21:12:38 +0000

Voice assistants like Siri and Alexa are great for non-trivial everyday personal assistive tasks. However, they are limited in providing accurate answers to complex questions, real-time information, handling turns, and user interruptions.

Try asking Siri about the best things to do with kids in a particular city or location. It won't provide an accurate answer because it can’t access web search tools. On devices supporting Apple Intelligence, asking the same question will be handed off to ChatGPT.

As in-app conversational app features, voice agents are here to solve these limitations.

The sections below will help you discover how to build AI voice agents and the best creation platforms. Although it is not required for this article, you can set up a local Node.js server and run our demo iOS/iPadOS voice agent in SwiftUI. After setting up your local Node server, you can also test the conversational agent for other platforms by following these step-by-step tutorials:

What Is an AI Voice Agent?

A voice agent is a conversational AI assistant capable of taking user instructions and responding with a human-like voice in real-time using a local or cloud-based LLM.

Like text-generation agents, voice-based ones use LLMs to output audio responses. The best way to think of it is to consider ChatGPT's voice mode, as illustrated in the above image. With the tap of a button and selecting your preferred voice, you can easily speak to ChatGPT for real-time responses.

In the following sections, we’ll look at the top platforms for building a ChatGPT-like voice mode experience.

Why Build a Voice Agent?

Like text-based AI agents, the support for MCP in voice applications helps agents retrieve accurate, real-time information from services such as Perplexity and Exa. With MCP, you can build an agent to manage tasks through Slack and Linear using voice interactions. You can also connect voice agents to MCP tools for custom workflows.

When creating your voice AI app, support for diverse accents may be needed. Luckily, major platforms, like the OpenAI Agents SDK, provide a library of voices to choose from. Voice-based agents can be used across many domains, including sales, marketing, customer support, small businesses, and enterprises.

Video AI: An excellent use case of voice agents in a video AI setting is Gemini Live. It uses your phone's camera to see and understand objects around you and provide answers through speech interactions. Gemini Live also allows users to screen share their phone device's screen to ask questions about the content on the selected screen.

Gemini voice mode

Sales leads: Use a voice agent to follow up and contact potential customers for inbound sales in enterprises and small businesses.
Customer support and call center: Voice agents can receive customer complaints and help fix issues.
Personal assistant: Like Gemini Live, you can create a voice system to help you understand your surroundings and what you’re looking at. Another trending application area is computer and browser use. You can integrate a voice system with an AI browser agent to automate online booking and appointment scheduling.
Social platform: Build voice agents to interact with people in social communities by giving real-time voice responses to users' queries.
Gaming: Build character dialog and interactive narration systems with voice agents for online gaming platforms.
Telecare: Use an AI voice to interact with patients and collect their information online in telehealth scenarios.

The Top 8 Platforms To Build Voice-Enabled Apps

Building a voice AI app can be daunting. Several aspects include backend infrastructure, audio quality, latency, and more. For these reasons, you can rely on frameworks, SDKs, and APIs to create AI solutions with voice as one of the main in-app features.

Most of these voice agent-building platforms offer a Python-first approach. TypeScript-based options are now catching up.

Let's look at the leading solutions and how to build quickly with them.

1. Stream Python AI SDK: Integrate In-App Voice AI

Stream allows developers to build real-time audio apps in React, Swift, Android, Flutter, JavaScript, and Python. With minimal effort, the recently released Python AI SDK helps developers create complex voice AI services, such as meeting assistants and bots for video conferencing.

You can use the Python SDK for transcribing, voice activity detection (VAD), converting speech to text, and vice versa. Aside from these features, you can integrate and extend your Stream-powered voice AI app with other leading platforms like:

The SDK's foundation is built with WebRTC and the OpenAI Real-Time API to endure low-latency communication.

Get Started with Stream Python AI SDK

Creating your first voice app with the Stream's Python AI SDK requires a few steps:

Set up a Python environment and install the necessary dependencies. You can configure your Python virtual environment with this command.

python3 -m venv venv && source venv/bin/activate

You may also set it up using uv and add a .env file to your project’s root directory and fill it up with the following:

STREAM_API_KEY=your-stream-api-key
STREAM_API_SECRET=your-stream-api-secret
STREAM_BASE_URL=https://pronto.getstream.io/
OPENAI_API_KEY=sk-your-openai-api-key

Sign up for a Stream dashboard account, create an app, and grab its API key and secret to substitute the above placeholders. With the STREAM_BASE_URL, you can create and join a Stream video call in your browser.

Next, run the following commands to install the Python AI SDK.

pip install --pre "getstream[plugins]"

# or using uv
uv add "getstream[plugins]" --prerelease=allow

Initialize a Stream client and user, and create a call.

from dotenv import load_dotenv
from getstream import Stream
from getstream.models import UserRequest
from uuid import uuid4
import webbrowser
from urllib.parse import urlencode

# Load environment variables
load_dotenv()

# Initialize Stream client from ENV
client = Stream.from_env()

# Generates a new user ID and creates a new user
user_id = f"user-{uuid4()}"
client.upsert_users(UserRequest(id=user_id, name="My User"))

# We can use this later to join the call
user_token = client.create_token(user_id, expiration=3600)

# Generate a user ID for the OpenAI bot that is added later
bot_user_id = f"openai-realtime-speech-to-speech-bot-{uuid4()}"
client.upsert_users(UserRequest(id=bot_user_id, name="OpenAI Realtime Speech to Speech Bot"))

# Create a call with a new generated ID
call_id = str(uuid4())
call = client.video.call("default", call_id)
call.get_or_create(data={"created_by_id": bot_user_id})

Create an OpenAI speech-to-speech pipeline.

The following code snippet allows you to create a speech-to-speech pipeline by initializing the OpenAIRealtime class of the SDK and launching the call with a web browser:

sts_bot = OpenAIRealtime(
    model="gpt-4o-realtime-preview",
    instructions="You are a friendly assistant; reply in a concise manner.",
    voice="alloy",
)

try:
    # Connect OpenAI bot
    async with await sts_bot.connect(call, agent_user_id=bot_user_id) as connection:

        # Sends a message to OpenAI from the user side
        await sts_bot.send_user_message("Give a very short greeting to the user.")

except Exception as e:
    # Handle exception
finally:
    # Delete users when done
    client.delete_users([user_id, bot_user_id])

base_url = f"{os.getenv('EXAMPLE_BASE_URL')}/join/"

# The token is the user token we generated from the client before.
params = {"api_key": client.api_key, "token": user_token, "skip_lobby": "true"}

url = f"{base_url}{call_id}?{urlencode(params)}"

try:
    webbrowser.open(url)
except Exception as e:
    print(f"Failed to open browser: {e}")
    print(f"Please manually open this URL: {url}")

When you put the code snippets together in the above steps and test out the combined code, you should see an output similar to this:

Stream voice agent

2. OpenAI: Create Voice Apps

OpenAI allows developers to integrate voice services with their apps in two ways. With its Python Agents SDK and TypeScript SDK, you can easily add voice agents to any AI application.

The SDK supports male and female TTSVoices such as Alloy, Ash, Coral, Echo, Fable, Onyx, Nova, Sage, and Shimmer. With its voice agent pipeline, you can transcribe an audio input into text, run a workflow for sequential text responses, and transform the text-based output into streaming audio.

Another way to build audio/speech experiences with OpenAI is to use its Realtime API. Using its WebRTC or WebSockets backend, developers can build multi-modal experiences supporting realtime text and speech generation, transcription, function calling, and more.

Some of the unique features of building voice agents with OpenAI include the following:

Tools: Give your voice applications access to external services to execute actions.
Agent monitoring: Provide guardrails and rules for voice agents to follow.
Agent handoff: Create multiple agents who can assign tasks to others.
Audio handling: It uses WebRTC to handle audio input/output by default.
Session management: It allows configuring and customizing real-time sessions.

Get Started with OpenAI JS SDK

With the sample code below, you can start building voice apps with the OpenAI JS SDK. Check out voice agents quickstart to learn more.

import { RealtimeAgent, RealtimeSession } from '@openai/agents/realtime';

const agent = new RealtimeAgent({
  name: 'Assistant',
  instructions: 'You are a helpful assistant.',
});

const session = new RealtimeSession(agent);

// Automatically connects your microphone and audio output
// in the browser via WebRTC.
await session.connect({
  apiKey: '<client-api-key>',
});

3. ElevenLabs: Build Conversational Voice Agents

ElevenLabs is one of the leading platforms for building conversational AI applications. It provides developers and enterprises with all the building blocks for integrating low-latency voice agents with any service.

The example below demonstrates a realistic text-to-speech interaction.

ElevenLabs interactive voice demo

With this platform, you can access categories of AI models for voice cloning, isolation, swapping, voice design, and making sound effects. Combinations of these models can be used to create and deploy interactive audio services.

For example, its latest model (Eleven V3) at the time of writing this article is an excellent choice for implementing realistic and expressive in-app text-to-speech. With its support for different categories of speech models, you have several options for video, audio, gaming, telehealth, and marketplace applications.

Get Started with ElevenLabs

ElevenLabs provides easy-to-use SDKs for Python and TypeScript developers. The sample code below is all you need to make your first API call to the Python voice AI SDK to create your text-to-speech app.

from dotenv import load_dotenv
from elevenlabs.client import ElevenLabs
from elevenlabs import play
import os

load_dotenv()

elevenlabs = ElevenLabs(
  api_key=os.getenv("ELEVENLABS_API_KEY"),
)

audio = elevenlabs.text_to_speech.convert(
    text="The first move is what sets everything in motion.",
    voice_id="JBFqnCBsd6RMkjVDRZzb",
    model_id="eleven_multilingual_v2",
    output_format="mp3_44100_128",
)

play(audio)

Check out the developer quickstart to learn more about the TypeScript version of the SDK.

4. Deepgram: Build Voice AI Solutions

Deepgram is a voice AI application creation platform. Developers can use its API to build audio apps with text-to-speech, speech-to-speech, and speech-to-text models. To experience and see how Deepgram works, visit the URL above and try the interactive, real-time voice demo on the home page.

With Deepgram, you can try different models and APIs to build intelligent audio apps for use cases in customer service, telemedicine, sales, service ordering, etc. The quickest way to start building your app with Deepgram is to try its API playground.

Get Started with Deepgram

One advantage of using Deepgram is that its APIs are available for Python, JavaScript, C#, and Go. To build your first voice agent with any of the Deepgram SDKs, configure your environment and install the platform-specific SDK. The example commands below are for Python.

To begin with your preferred platform for implementing a voice agent, head to the Getting Started guides.

mkdir deepgram-agent-demo
cd deepgram-agent-demo
touch index.py
export DEEPGRAM_API_KEY="your_Deepgram_API_key_here"
@TODO other Python commands

# Install the SDK
pip install deepgram-sdk

Deepgram interactive voice demo

5. Vapi: Voice AI Agents For Developers

The Vapi platform helps developers build and deploy voice agents and AI products in Python, React, and TypeScript. It provides two ways to make intelligent voice apps. It's assistant's option allows you to create simple conversational services that may require a single system prompt for the underlying model's operations.

Example use cases of Vapi Assistants include simple question and answer systems and chatbots. If an agentic system has a complex logic or involves a multi-step process, you can use the workflow feature of Vapi to build your agents. Applications in this category are suitable for appointment scheduling and service ordering.

Vapi is an excellent choice for developers and enterprises developing voice AI products for call operations involving actual phone numbers. You can integrate it with several applications and model providers, such as Salesforce, Notion, Google Calendar, Slack, OpenAI, Anthropic, Gemini, and more.

Multilingual support: The API supports multilingual operations. This means your app's users can speak to agents in English, Spanish, and 100+ other supported languages.
External tools: Easily add external tools to allow your voice agent to perform accurate actions.
Automated testing: Use simulated AI voices to create test suites for production-ready agents.
Plugin any model: You can bring your favorite text-to-speech, speech-to-text, and speech-to-speech models from any major AI service provider.

Get Started with Vapi

Making your first voice AI app with Vapi or integrating it with an existing app is simple. The React sample code below can get you started.

import Vapi from "@vapi-ai/web";
import { useState, useEffect } from "react";

export const vapi = new Vapi("YOUR_PUBLIC_API_KEY"); // Get your public api key from the dashboard

function VapiAssistant() {
  const [callStatus, setCallStatus] = useState("inactive");
  const start = async () => {
    setCallStatus("loading");
    const response = vapi.start("YOUR_ASSISTANT_ID"); // Get your assistant id from the dashboard
  };
  const stop = () => {
    setCallStatus("loading");
    vapi.stop();
  };
  useEffect(() => {
    vapi.on("call-start", () => setCallStatus("active"));
    vapi.on("call-end", () => setCallStatus('inactive'));
    return () => vapi.removeAllListeners();
  }, [])
  return (
    <div>
      {callStatus === "inactive" ? (<button onClick={start}>Start</button>) : null}
      {callStatus === "loading" ? <i>Loading...</i> : null}
      {callStatus === "active" ? (<button onClick={stop}>Stop</button>) : null}
    </div>
  );
}

Visit vapi.ai to try making voice agents for other platforms.

6. Play.ai: Build Real-Time Intelligent Voice Apps

PlayAI is a platform for making intelligent voice apps for the web and mobile. The platform allows engineers to create voice agents for healthcare, real estate, gaming, food delivery, EdTech, and more.

To see how the service works, check out the interactive voice chat demo from the above URL or try the PlayNote web app, which allows you to turn JPEG, PDF, EPUB, CSV, and several other files into a human-like-sounding audio format. Like other platforms, PlayAI has a library of AI voices for experimenting with your apps.

The PlayAI's playground/sandbox provides a starting point for experimenting, testing speech generation, and building audio experiences.

Get Started with PlayAI

You can use the PlayAI text-to-speech API in Bash, Python, JavaScript, Go, Dart, and Swift. First, set your API credentials on your machine.


# macOS (zsh)
echo 'export PLAYAI_KEY="your_api_key_here"' >> ~/.zshrc
echo 'export PLAYAI_USER_ID="your_user_id_here"' >> ~/.zshrc
source ~/.zshrc

# Windows
setx PLAYAI_KEY "your_api_key_here"
setx PLAYAI_USER_ID "your_user_id_here"

Then, make your first API call to create an audio from a text prompt using this Curl script:

curl -X POST 'https://api.play.ai/api/v1/tts/stream' \
  -H "Authorization: Bearer $PLAYAI_KEY" \
  -H "Content-Type: application/json" \
  -H "X-USER-ID: $PLAYAI_USER_ID" \
  -d '{
    "model": "PlayDialog",
    "text": "Hello! This is my first text-to-speech audio using PlayAI!",
    "voice": "s3://voice-cloning-zero-shot/baf1ef41-36b6-428c-9bdf-50ba54682bd8/original/manifest.json",
    "outputFormat": "wav"
  }' \

Running the above in your Terminal should output an audio file called hello.wav.

7. Pipecat: Build Voice AI Apps

Pipecat is one of the most widely used open-source frameworks for building conversational AI applications. The framework allows developers to create complex dialog systems, enterprise-grade customer support agents, multimodal interactions (video, voice, and images), and video meeting assistants.

Check out this X post to see a practical demo of Pipecat in action.

The client SDKs for Web, iOS, Android, and C++ allow you to build low-latency conversational apps with several AI services, tools, and underlying backend technologies such as WebRTC and WebSockets.

Get Started with Pipecat

You can start running Pipecat on a local machine by configuring your environment, installing the module, and switching to the cloud once your voice application is ready for production.

# Install the module
pip install pipecat-ai

# Set up your environment
cp dot-env.template .env

Refer to the Pipecat’s GitHub repo for more code samples and detailed instructions on building conversational agents with the framework.

8. Cartesia: Create Realistic AI Voices

Cartesia is a developer-first platform for voice AI. The Cartesia API makes incorporating high-quality voices into any product easier. It also provides seamless support for extending voice agents with other platforms like LiveKit, Vapi, and Pipecat. You can build your speech applications in 15+ languages and deploy them anywhere and on any device.

To learn more about how Cartesia works, you can check out its Sonic text-to-speech and Ink-Whisper speech-to-text models.

Get Started with Cartesia

Depending on your machine, there are a few installation requirements for using the Cartesia API.

# macOS
brew install ffmpeg

# Debian/Ubuntu
sudo apt install ffmpeg

# Fedora
dnf install ffmpeg

# Arch Linux
sudo pacman -S ffmpeg

After you install any of the above for your computer, you can now make an API call to generate your first speech from text using cURL, Python, or JavaScript/TypeScript.

curl -N -X POST "https://api.cartesia.ai/tts/bytes" \
        -H "Cartesia-Version: 2024-11-13" \
        -H "X-API-Key: YOUR_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{"transcript": "Welcome to Cartesia Sonic!", "model_id": "sonic-2", "voice": {"mode":"id", "id": "694f9389-aac1-45b6-b726-9d9369183238"}, "output_format":{"container":"wav", "encoding":"pcm_f32le", "sample_rate":44100}}' > sonic-2.wav

To go further, you can try Cartesia's Python and TypeScript SDKs.

Other Notable Voice AI Platforms

There are other excellent services like LiveKit, Kokoro TTS, and Moonshine for creating voice applications. You can also use platforms like Unmute.sh and OpenAI.fm to experiment and test with a library of natural and realistic AI voices.

To make voice apps for specific use cases, such as phone call operations, you can use services like Bland AI, Retell AI, and Synthflow AI. If you want to try open-source TTS models, check out Chatterbox TTS and the Speech Synthesis category on Hugging Face Spaces.

The Future of Voice Agents

This article covered eight of the best platforms you can use today to make speech AI products.

Although all the platforms covered in this article provide natural-sounding human-like voices, many have high latencies when interacting with agents. Some do not handle interruptions and noisy conditions properly. For example, many platforms will struggle to understand a user's voice, especially if a baby or a kid is playing and talking in the background.

As this AI field keeps improving regularly, future speech models, APIs, and SDKs will enhance their interruption capabilities, noise detection, and ensure low-latency speech-to-speech, speech-to-text, and text-to-speech interactions. Many of these platforms are currently closed source, but as the voice AI landscape evolves rapidly, open-source alternatives will continue to emerge.

Official Statement: Distinction Between GetStream.io and GetStream.live

Sarah Lindauer — Tue, 27 Jan 2026 22:40:28 +0000

We are issuing this advisory to clarify confusion regarding our brand, GetStream.io (Stream), and an unrelated third-party website known as "GetStream.live."

Stream is a legitimate technology company providing Chat and Activity Feed APIs for developers. We are NOT affiliated, associated, or connected in any way with "GetStream.live," a site primarily known for unauthorized sports streaming.

Important Security Warning for Users

If you arrived here searching for free sports streaming, please be aware that you have reached the wrong destination.

We strongly advise caution if you intend to visit unauthorized streaming websites ending in .live or similar TLDs. Cybersecurity researchers frequently associate illegal streaming sites with high risks of malware, browser hijackers, and phishing scams targeting personal data.

GetStream.io maintains strict security protocols and is a trusted service provider for over a billion end-users globally. We do not host, stream, or provide access to copyrighted sports content.

If you would like to watch your events online, we suggest a legal site like ESPN, NBA, or Go3.

What is Stream?

We build the world's most scalable Chat, Video, and Activity Feed infrastructure for applications.

For Developers: If you are looking to integrate in-app chat or social feeds into your product, you are in the right place.
For Sports Fans: If you are looking for live sports, we recommend using authorized, legal streaming platforms to ensure your device security.

Frequently Asked Questions

Is GetStream.io the same as GetStream.live?

No. GetStream.io is a US-based SaaS technology company founded in 2014. GetStream.live is an entirely unrelated entity.

Why does GetStream.live appear in search results when I search for your brand?

Due to the similarity in naming, search engines may occasionally conflate the two distinct entities. We are publishing advisories like this one to help clarify the distinction for users and search algorithms.

Is the .live associated with Getstreaming.TV?

No, the .live site is not associated with the site you are on right now (.io), or Getstreaming.TV

Am I really at risk of having malware installed?

Yes. The .live site is using sports streams created illegally. These streams are not from official sources and can contain any malicious files the creator chooses to include.

I've read reviews that the .live site is a legit place to watch sports, is that not correct?

If you are referring to these reviews of the .live site those should definitely not be trusted.