DEV Community: Ecohash

Best models to run on RTX Pro 6000 96GB

Ecohash — Sun, 01 Nov 2026 11:55:46 +0000

The right model depends on the task, not on the GPU. A 96 GB RTX Pro 6000 holds a 20B to 35B open model with room for context and batching, so you get to pick by the job instead of by a memory ceiling. For coding and agents that means Qwen3 Coder 30B; for low-cost chat, gpt-oss 20B; for retrieval, an embedding model with a reranker; for voice, Whisper and Kokoro. All of them run through the same OpenAI-compatible API at https://api.ecohash.com/v1, so you start on the shared API and move to a dedicated endpoint or a workspace later without touching your code.

Pick by task, not by memory. Coding: qwen3-coder-30b-a3b-instruct. Chat: gpt-oss-20b. Retrieval: jina-embeddings-v3 + bge-reranker-v2-m3. Voice: whisper-large-v3-turbo + kokoro-82m.

Model by task

Coding, agents, tool use · Model on EcoHash: Qwen3 Coder 30B · Model ID: qwen3-coder-30b-a3b-instruct
General chat, low cost · Model on EcoHash: gpt-oss 20B · Model ID: gpt-oss-20b
General chat, higher capability · Model on EcoHash: GLM 5.2, DeepSeek V4 Flash · Model ID: GLM-5.2, DeepSeek-V4-Flash
Embeddings for RAG · Model on EcoHash: Jina v3, Qwen3 Embedding · Model ID: jina-embeddings-v3, qwen3-embedding-8b
Reranking for RAG · Model on EcoHash: BGE reranker v2 m3 · Model ID: bge-reranker-v2-m3
Speech to text · Model on EcoHash: Whisper Large V3 Turbo · Model ID: whisper-large-v3-turbo
Text to speech · Model on EcoHash: Kokoro 82M · Model ID: kokoro-82m

The higher-capability chat models run through the hosted API, not on a single 96 GB card. Here is what the single-card models measure on EcoHash, end-to-end, in July 2026:

Coding · Model: qwen3-coder-30b-a3b-instruct · Measured on EcoHash: 30 ms TTFT, ~110 tok/s, 32k context
General chat · Model: gpt-oss-20b · Measured on EcoHash: 170 ms TTFT, ~125 tok/s
General chat · Model: llama-3.1-8b-instruct · Measured on EcoHash: 205 ms TTFT, ~143 tok/s
Speech to text · Model: whisper-large-v3-turbo · Measured on EcoHash: WER 4.37, ~0.5 s per 10 s of audio
Text to speech · Model: kokoro-82m · Measured on EcoHash: first audio ~120 ms

Against other providers of the same open models, EcoHash serves them at full BF16 precision and lands competitively on price and speed (llama-3.1-8b-instruct, EcoHash in purple):

Full data and charts: ecohash-benchmarks.

Facts

GPU: NVIDIA RTX Pro 6000 Blackwell Server Edition, 96 GB GDDR7
Single-card sweet spot: 20B to 35B open models
API base URL: https://api.ecohash.com/v1
Access: Shared API, dedicated endpoint, GPU workspace
Price: RTX Pro 6000 at $1.89 per GPU-hour (/pricing)

Who this is for

Developers choosing an open model for a coding, chat, retrieval, or voice feature.
Teams deciding what to run on a 96 GB card versus what to call through a hosted API.

What you can decide

Which model to start with for your task.
Whether it fits on a single RTX Pro 6000 or is better used through the hosted API.
When to move from the shared API to a dedicated endpoint or a workspace.

How to choose

Four questions get you most of the way:

What is the task? Code, chat, retrieval, or speech. Start here, not with the hardware.
Does it fit in 96 GB at your context length and batch size? A 20B to 35B model fits with room to spare, models near 70B need quantization, and 235B and up are better through the hosted API.
What are your latency and throughput needs? Steady, high-volume traffic is the reason to reserve a dedicated endpoint.
Do you need to self-host at all? For most teams the shared API is enough, and the GPU is EcoHash's problem, not yours.

Coding

Qwen3 Coder 30B (qwen3-coder-30b-a3b-instruct) is the default for code generation, refactoring, and tool-using agents. It is a Mixture-of-Experts model with about 30B total parameters, it fits comfortably on one 96 GB card, and on EcoHash it serves a 32,768-token context at about 110 tokens/sec single-stream with a 30 ms time to first token. See /models/qwen3-coder-30b-a3b-instruct.

For lighter or cheaper high-volume coding, gpt-oss 20B (gpt-oss-20b) is the smaller option. The three deployment paths for Qwen3 Coder are covered in Qwen3 Coder 30B on RTX Pro 6000.

General chat

Start with gpt-oss 20B (gpt-oss-20b) for low-cost chat and assistants. It fits on one card, has a long context, and reaches about 8,900 output tokens/sec under load.

For stronger general capability, EcoHash also serves larger models such as GLM 5.2 (GLM-5.2) and DeepSeek V4 Flash (DeepSeek-V4-Flash), and very large Mixture-of-Experts models like Qwen3-235B-A22B (Qwen3-235B-A22B). Those are not single-card deployments. Use them through the shared API or a dedicated endpoint and let EcoHash handle the hardware.

Retrieval (RAG)

A retrieval pipeline needs two models: an embedding model to index and search text, and a reranker to reorder the top results.

Embeddings: jina-embeddings-v3, jina-embeddings-v4, or the Qwen3 Embedding family (qwen3-embedding-8b, qwen3-embedding-4b, qwen3-embedding-0.6b).
Reranking: bge-reranker-v2-m3.

All of them run through the same key, so one service can embed, rerank, and generate the answer without stitching vendors together. See the RAG use case.

Voice

A voice agent uses three model types: speech to text, a chat model, and text to speech.

Speech to text: whisper-large-v3-turbo or qwen3-asr-1-7b.
Text to speech: kokoro-82m or qwen3-tts.

These speech models are small and do not need an RTX Pro 6000. They run through the API and move onto GPU capacity only when throughput grows. The full build is in Build a voice agent with Whisper, Kokoro, and an OpenAI-compatible API; see also /use-cases/voice-speech, /models/kokoro-82m, and /models/whisper-large-v3-turbo.

Calling these models

The API is OpenAI-compatible. A coding call:

from openai import OpenAI

client = OpenAI(base_url="https://api.ecohash.com/v1", api_key="eco_...")

resp = client.chat.completions.create(
    model="qwen3-coder-30b-a3b-instruct",
    messages=[
        {"role": "user", "content": "Refactor this to be iterative:\n\ndef fib(n):\n    return n if n < 2 else fib(n-1) + fib(n-2)"}
    ],
)
print(resp.choices[0].message.content)

An embeddings call for a RAG index:

emb = client.embeddings.create(
    model="jina-embeddings-v3",
    input=["EcoHash serves open models through an OpenAI-compatible API."],
)
print(len(emb.data[0].embedding))

Shared API, dedicated endpoint, or workspace

Shared API: no GPU to manage, pay per token, good for building and variable traffic. See /inference.
Dedicated endpoint: reserved capacity for one model, steadier throughput and more predictable latency. See /dedicated-inference.
GPU workspace: an hourly RTX Pro 6000 for experiments, fine-tuning, and batch jobs. See /gpu-compute.

Limitations

Model fit depends on quantization, context length, and batch size. The ranges here are starting points to validate, not guarantees.
The higher-capability chat models are for the hosted API, not single-card serving.
Real performance depends on your setup. Reproduce any figure with its method and date.
Model IDs change as the catalog changes. Check /models before you build.

FAQ

What is the best model to run on an RTX Pro 6000 96GB? It depends on the task. Coding: qwen3-coder-30b-a3b-instruct. General chat: gpt-oss-20b. Retrieval: jina-embeddings-v3 with bge-reranker-v2-m3. Speech: whisper-large-v3-turbo with kokoro-82m.

What is the best coding model for the RTX Pro 6000? Qwen3 Coder 30B (qwen3-coder-30b-a3b-instruct). It fits on one 96 GB card and serves a 32,768-token context on EcoHash.

What is a good low-cost general chat model? gpt-oss 20B (gpt-oss-20b). It fits on one card and has a long context.

Can I run a 235B model on one RTX Pro 6000? No. Use a model like Qwen3-235B-A22B through the hosted API instead.

What models do I need for RAG? An embedding model and a reranker, for example jina-embeddings-v3 and bge-reranker-v2-m3, plus a chat model to write the answer.

Do speech models need an RTX Pro 6000? No. Whisper and Kokoro run through the API, and move onto GPU capacity only when throughput demands it.

How many models can I use with one API key? All of them. Every model shares one key and one base URL, https://api.ecohash.com/v1.

How much does this cost? API usage is billed per token; a GPU workspace runs at $1.89 per GPU-hour. See /pricing for current rates.

Next steps

Browse all models: /models
See the RTX Pro 6000 page: /gpu-compute
Reserve capacity: /dedicated-inference

LTX 2.5 on One RTX PRO 6000: A 10-Second 1080p Video With Audio in 90 Seconds

Ecohash — Sat, 31 Oct 2026 16:08:53 +0000

On October 27, 2026 we launched the public LTX 2.5 Video Generation template on one NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB). We had Claude Opus 5.5 do the launch through the EcoHash GPU MCP server and write the prompts. Then we timed 24 renders across four output sizes and three clip lengths. Every render included a generated soundtrack.

Disclosure: EcoHash operates the platform and the template tested here. This is our own measurement on a standard on-demand instance from the public Console catalogue, not an independent review.

Results at a glance

Measurement (one RTX PRO 6000, warm server, 24 fps, audio on)	Result
Launch call → `/health` ready	3 min 10 s (instance running at 1 min 33 s; the rest is loading 72.9 GiB of weights)
First request after start (2 s, 768×512)	6.0 s
10 s clip at 1080p (1920×1088)	90.0 s (9.0× real time)
10 s clip at 720p (1280×704)	33.6 s (3.4× real time)
10 s clip at 480p (768×512)	14.8 s (1.5× real time)
10 s 1080p clips per GPU-hour, sequential	39
Same seed, same request	Byte-identical mp4 on every repeat

How long does LTX 2.5 take per clip on an RTX PRO 6000?

Mean wall time per clip, two runs each, measured from the client. The figure includes the two-stage diffusion, VAE decode, audio muxing and returning the file.

Output	Resolution	Clip	Runs	Wall time (mean)	× real time	Clips / GPU-hour
480p 3:2	768×512	2 s (49 frames)	2	5.1 s	2.5×	700
480p 3:2	768×512	5 s (121 frames)	2	8.9 s	1.8×	405
480p 3:2	768×512	10 s (241 frames)	2	14.8 s	1.5×	243
720p 16:9	1280×704	2 s (49 frames)	2	8.4 s	4.1×	427
720p 16:9	1280×704	5 s (121 frames)	2	16.9 s	3.4×	213
720p 16:9	1280×704	10 s (241 frames)	2	33.6 s	3.4×	107
1080p 16:9	1920×1088	2 s (49 frames)	2	16.8 s	8.2×	214
1080p 16:9	1920×1088	5 s (121 frames)	2	40.5 s	8.0×	88
1080p 16:9	1920×1088	10 s (241 frames)	2	90.0 s	9.0×	39
1080p 9:16	1088×1920	2 s (49 frames)	2	16.7 s	8.2×	215
1080p 9:16	1088×1920	5 s (121 frames)	2	41.8 s	8.3×	86
1080p 9:16	1088×1920	10 s (241 frames)	2	118–194 s*	11.8×	30

At 1080p, render time grows roughly in proportion to clip length: 16.8 s for 2 s of video, 40.5 s for 5 s and 90.0 s for 10 s. For 2 s and 5 s clips, portrait 1088×1920 matched landscape within 1.3 s. At 10 s, the faster portrait run took 118 s, 28 s slower than landscape.

* The two portrait 10 s runs took 118 s and 194 s. During the slower one, the GPU logged about 75 s at near-zero utilization while our SSH tunnel stopped answering /health. That points to transfer, not generation. The rate columns therefore use the 118 s run.

In the other eleven cells, two runs of the same request differed by at most 1.0 s. In all twelve, the two output files were byte-identical.

How fast does the LTX 2.5 template start?

Event	Time after `launch_template`
Instance `running`, SSH ready	1 min 33 s
`/health` returns `{"status":"ready"}`	≤ 3 min 10 s
First 2-second test clip returned	+ 6.0 s

You don't download or install anything. The template mounts the LTX 2.5 Diffusers weights read-only from shared storage and starts a FastAPI server on port 8000.

What does a clip cost on a rented RTX PRO 6000?

A GPU instance is billed per hour while it runs, not per clip, so the useful number is clips per GPU-hour. Multiply by the current RTX PRO 6000 hourly rate on the RTX PRO 6000 page to get a cost per clip.

Output	10 s clips per GPU-hour (sequential)
480p	243
720p	107
1080p	39

Two things move the real number:

Idle time. The instance bills whether or not it is rendering. Our session was billed for 4 h 41 min, but the GPU log shows it was busy (≥ 10% utilization) for only 23 minutes. The rest was idle time while our client was disconnected. Terminate as soon as you are done.
Retries. We kept 6 of 8 shots on the first take. Budget for at least one extra take per shot you intend to keep.

How do you run LTX 2.5 with Claude and the EcoHash GPU MCP?

Connect the EcoHash GPU MCP server to Claude (setup guide).
Launch. Ask Claude to start the "LTX 2.5 Video Generation" template. It calls list_templates, then quote_template, waits for your confirmation, then calls launch_template. Pass your SSH public key if you want a shell.
Wait for get_instance to report running, then poll GET /health until it returns ready.
Prompt. Have Claude write one prompt per shot in five parts: camera, subject, one action with a visible result, an ending, and the sound.
Render. Call POST /v1/video/generations with prompt, width, height, num_frames, fps and seed. The response holds the mp4 with audio as base64.
Terminate with terminate_instance once your files are saved locally, because the scratch disk is deleted with the instance.

A minimal request:

curl -s http://localhost:8000/v1/video/generations \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Doorbell camera footage at night on Halloween ... Audio: crinkling candy wrappers, distant crickets. No music, no speech.",
       "width":1920,"height":1088,"num_frames":241,"fps":24,"seed":20261027}' \
  | python3 -c "import sys,json,base64;open('out.mp4','wb').write(base64.b64decode(json.load(sys.stdin)['data'][0]['b64_mp4']))"

Width and height are rounded to multiples of 32, so 1080 becomes 1088. num_frames is rounded to 8k+1, and 241 frames at 24 fps is about 10 seconds. The server runs the distilled two-stage schedule (8 + 3 sigmas, unguided). It accepts num_inference_steps and guidance_scale but ignores them.

Which prompt structure worked best?

We gave Claude a one-line brief: a Halloween-week variant of the "caught on a doorbell camera" format, with eight 16:9 shots. We asked for each prompt in five parts:

[camera / footage type], [time + place]. [one subject doing one thing].
[one action with a visible result]. [ending: freeze or look into the lens].
Audio: [2–3 concrete sounds]. No music, no speech.

Claude then checked every prompt against four questions before we rendered anything:

Is something happening in the first second?
Is there only one action?
Is there an ending?
Can you hear the sound?

The camera phrase, such as "doorbell security camera footage" or "handheld phone video", did more for realism than any quality adjective.

What did LTX 2.5 get wrong?

We rendered eight 1080p shots, at 5 s or 8 s each, from Claude's prompts. On the first take we kept six. The misses are worth knowing before you write prompts:

Objects vanish instead of falling. We asked for a black cat to push a jack-o'-lantern off a railing. The cat never touched it, and the pumpkin simply disappeared.
Readable text is gibberish. A hand-lettered sign beside a candy bowl came out as invented words.
Words are taken literally. "A great horned owl" produced an owl with actual horns.
Counts drift. We asked for five costumed raccoons and got four.

What worked well: fisheye "doorbell camera" framing, night-vision grain, animal fur and motion, and sound that matched the action, such as wrappers crinkling and a plastic bowl clattering.

Does the network path change the timing?

Yes, so measure where you render. Called from inside the instance (localhost:8000), an 8 s 1080p shot took 63.7–64.5 s and a 5 s 1080p shot took 36.3–36.9 s. Through our SSH tunnel, an earlier 8 s shot took 329 s, almost all of it transfer. The platform's public service URL returned 502 for requests that ran longer than about a minute. For long renders, run the request on the instance and copy the files back afterwards.

Test environment

GPU: 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB, driver 580.105.08
Template: EcoHash public template "LTX 2.5 Video Generation" (image vllm/vllm-omni:v0.27.0rc1), Atlanta region
Pipeline: Diffusers LTX2Pipeline, distilled DiT, two-stage 8 + 3 sigma schedule, guidance 1.0, audio on
Timing matrix: sequential requests through an SSH tunnel, client-side wall clock. Opening shots: sequential requests from inside the instance.
Prompt for the timing matrix: a fixed Halloween porch scene, seed 20261027, 24 fps
Date: October 27, 2026

Limitations

This is one GPU, one prompt for the timing matrix and two runs per cell. Other prompts may vary slightly in time; content does not change the step count.
We did not test concurrent requests. The server holds a GPU lock and queues them.
We did not test image-to-video timing, longer clips beyond 241 frames, or other runtimes.
Clip quality is judged by eye. We report what we kept, not a quality score.
LTX 2.5 is distributed under the LTX-2 Community License. Check its terms for your use case before you publish commercially.

FAQ

Can one RTX PRO 6000 render 1080p LTX 2.5 video? Yes. A 10-second 1920×1088 clip with audio rendered in 90.0 s on a single 96 GB RTX PRO 6000. We used no CPU offload and no quantization flags.

Does LTX 2.5 generate sound? Yes. Every render in this test came back with an AAC soundtrack generated alongside the frames. Describe the sounds in an Audio: sentence in the prompt.

Is the output reproducible? In our test it was. Repeating a request with the same seed produced byte-identical files in every one of the 12 size and length pairs.

How long until the template is usable? In our test, /health reported ready 3 min 10 s after the launch call. The first short render returned 6 s later.

Do I need to download the model? No. The template mounts the weights from shared storage. You pay only for the GPU time you run.

Disclosure: I work on EcoHash. This post was originally published on the EcoHash blog.

Run DeepSeek V4 Flash Vision with an EcoHash GPU template

Ecohash — Fri, 23 Oct 2026 05:40:01 +0000

n EcoHash platform template saves the configuration you would otherwise assemble yourself: the container image, GPU count, startup command and service port. The DeepSeek GPU template also mounts publisher-provided model weights from shared storage, so you do not need to download another copy before starting the server.

This walkthrough uses deepseek-v4-flash-8gpu-instance. Check that its resource kind reads GPU Instance. A similarly named Model Instance template creates a different resource, with different access and lifecycle routes.

The template detail page shows the configuration and a quote for your account. Prices and available capacity can change.

Start in the console

Open Compute → Templates → Platform, then select the DeepSeek GPU template.
Review its resources and README. The published configuration uses eight RTX PRO 6000 GPUs, SGLang and service port 8000. Keep the preset image and startup command for your first launch.
Review the region, capacity and pricing breakdown. Shared model weights can determine the region; the current DeepSeek launch form selects Atlanta and locks that field. A quote does not reserve capacity.
Click Launch to open the prefilled GPU-instance form. Add your SSH public key if you want shell access, and review the duration and storage settings.
Confirm the launch only after checking the credit hold. Open the new instance's detail page to follow its status and find its access information.

The template opens the GPU launch form. Reviewing this form does not start an instance; its final Launch button does.

The container can be running while the inference engine is still loading. Wait for the model server to answer before sending a generation request. The template README provides a startup estimate; it is not a guaranteed readiness time.

Keep the template's shared model weights and HTTP port. Add persistent storage separately for results you need to retain.

Use the API

You need curl, jq and an EcoHash API key. Read-only Compute access permits template discovery and quoting; launching and terminating GPU instances requires Full Compute access. Keep the key in an environment variable. See API-key permissions.

Read the template and review its quote

Set ECOHASH_API_KEY in your shell before running these commands. They read the template and request a quote; they do not launch an instance.

set -euo pipefail
API="https://api.ecohash.com"
TEMPLATE_ID=51
: "${ECOHASH_API_KEY:?Set ECOHASH_API_KEY first}"

curl -fsS "$API/compute-templates/$TEMPLATE_ID" \
  -H "Authorization: Bearer $ECOHASH_API_KEY" -o template.json
jq -e '.template.target_kind == "gpu_instance"' template.json >/dev/null

curl -fsS -X POST "$API/compute-templates/quote" \
  -H "Authorization: Bearer $ECOHASH_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"template_id\":$TEMPLATE_ID,\"hours\":1}" -o quote.json
jq '{lines,upfront_hold_usd,recurring_daily_usd,capacity_ok,warnings}' quote.json
jq -e '.capacity_ok == true and (.region_id|length>0) and (.gpu_type|length>0)' \
  quote.json >/dev/null

The target_kind check prevents a Model Instance template from being sent to the GPU-instance endpoint. If the quote reports unavailable capacity or a warning you cannot resolve, stop before creating the instance.

Prepare the launch request

Copy the template's spec, then add its ID and the quoted region and GPU type. This example sets a one-hour duration and includes your SSH public key. Replace the public-key file path if yours differs. Never use your private-key file.

jq --slurpfile q quote.json \
  --rawfile ssh "$HOME/.ssh/id_ed25519.pub" \
  '.spec + {
    template_id: .template.id,
    name: "deepseek-template-demo",
    region_id: $q[0].region_id,
    gpu_type: $q[0].gpu_type,
    estimated_duration_hours: 1,
    ssh_enabled: true,
    ssh_public_key: ($ssh|rtrimstr("\n"))
  }' template.json > launch.json
cat launch.json

The next POST creates billable compute. Review launch.json, the quote and any warnings first. Sending only template_id is insufficient; the request must include the template's launch configuration.

curl -fsS -X POST "$API/gpu-instances" \
  -H "Authorization: Bearer $ECOHASH_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @launch.json

Keep the returned instance ID. Use it to check status and retrieve the connection details:

INSTANCE_ID="<id returned by the launch request>"
curl -fsS "$API/gpu-instances/$INSTANCE_ID" \
  -H "Authorization: Bearer $ECOHASH_API_KEY"

Send a request to the model server

Once the server is ready, list its models and use the returned model ID. The public service URL forwards requests to the container's HTTP server on port 8000 and requires authentication.

SERVICE="$API/gpu-instances/$INSTANCE_ID/service/v1"
MODEL="$(curl -fsS "$SERVICE/models" \
  -H "Authorization: Bearer $ECOHASH_API_KEY" | jq -er '.data[0].id')"

curl -fsS "$SERVICE/chat/completions" \
  -H "Authorization: Bearer $ECOHASH_API_KEY" \
  -H "Content-Type: application/json" \
  --data "$(jq -n --arg model "$MODEL" '{
    model: $model,
    messages: [{role:"user",content:"Explain tensor parallelism in two sentences."}],
    max_tokens: 1024
  }')"

A successful chat request returns an OpenAI-compatible response. If the model-list request fails while the engine is initializing, check the instance status and startup logs rather than sending repeated generation requests. For your own server configuration, the application must listen on 0.0.0.0 at the exposed port.

For a Model Instance, the documented calling convention instead uses https://api.ecohash.com/v1/chat/completions with model="<name>:<instance_id>". Do not use that routing convention for this GPU-instance walkthrough. See calling a model endpoint.

Finish the evaluation

Save your results to persistent storage before terminating the GPU instance. Its local /workspace scratch data is temporary.

curl -fsS -X POST "$API/gpu-instances/$INSTANCE_ID/terminate" \
  -H "Authorization: Bearer $ECOHASH_API_KEY"

This termination route is for GPU instances. Manage a Model Instance through its own lifecycle controls; terminating one resource type is not a substitute for stopping the other.

A reservation-backed quote may show no additional hourly compute charge because capacity is covered by a reservation agreement. This does not mean the service is free. Review your account's quote and terms; separately retained storage may continue to incur charges.

Open platform templates · GPU-instance API reference

How to launch an 8-GPU RTX PRO 6000 instance on EcoHash

Ecohash — Thu, 22 Oct 2026 03:48:35 +0000

An eight-GPU instance in EcoHash's Atlanta region gives you a whole node: eight RTX PRO 6000 GPUs with 96 GB of VRAM each, 200 vCPUs, 1,352 GB of RAM, and up to 5,260 GiB of temporary local scratch space. You can start with an EcoHash container image or bring the stack you already use.

This walkthrough covers the choices that matter before you launch, how to connect afterward, and where to put files you need to keep. Open GPU Instances in the console to follow along. You will need an account with enough credit for the configuration you select.

Choose Atlanta and all eight GPUs

Click Launch Instance, give the instance a name, and select atl for Atlanta. Under GPUs per Instance, choose 8x.

The configuration summary now shows the whole node dedicated to this instance. The eight GPUs provide 768 GB of total VRAM, with 96 GB on each GPU. That is memory distributed across eight devices; your serving or training software must support using multiple GPUs to take advantage of it.

Selecting eight GPUs allocates the whole node. Check the resource summary for your chosen region.

Pick a container for the work you want to run

The EcoHash Images tab offers several starting points. Their names include the installed software versions, so check the dropdown rather than relying on a tag from an old tutorial.

Use the inference image for vLLM, Jupyter for notebooks, or a base image for your own stack.

GPU Inference: vLLM configured for the RTX PRO 6000. This is the starting point for serving an LLM. It is a large image, so the first launch on a node may spend several minutes downloading it.
GPU Base: a CUDA and PyTorch environment for installing your own software. Choose the CUDA version your workload needs.
CUDA Bare: a CUDA 13.0 image without a framework, for building your environment from a smaller starting point.
GPU Jupyter: JupyterLab and an ML environment for browser-based notebook work.

The optional Startup Command runs when the instance starts. Use it to launch your application or model server. If you leave it blank, the image's default command applies. Make sure that command keeps the container running.

Bringing your own image

Open Images under Registry and click Add Image. Enter a display name and the full image reference, including its tag. For example, registry.example.com/team/eval:v1.0 is the format; replace it with your real registry and image.

A public image needs only its reference. Private images also need credentials with pull access.

For a private registry, provide its username and a read/pull access token. The form displays guidance for the registry you enter. AWS ECR login tokens expire after 12 hours and cannot be saved for this workflow; use a public ECR repository or arrange for EcoHash's account to have pull access.

Before saving the image, check these requirements:

Architecture: build for linux/amd64. On an Apple Silicon Mac, include --platform linux/amd64 in your docker buildx build command.
CUDA: use CUDA 13.0 or earlier for the configuration described here.
GPU support: your framework and CUDA extensions must support Blackwell sm_120. Check the specific build you are using, especially if you compile extensions yourself.
SSH: EcoHash configures SSH automatically for Debian, Ubuntu, and Alpine-based images. Include OpenSSH yourself for other base distributions.

Save the image, return to the launch form, and select it under Custom / Registry.

Set up SSH and an HTTP endpoint

Enable SSH access and add your public SSH key in the launch form. Keys are applied when the instance starts, so do this before launching. Keep the matching private key on your own computer.

If you want to call a model server or another HTTP application, enable Expose a public service URL and enter the port your application listens on inside the container, such as 8000.

The endpoint forwards to the HTTP port inside your container.

Your application must listen on 0.0.0.0, not only on 127.0.0.1. A process bound to localhost can run normally inside the container while remaining unreachable through the public URL.

Choose the runtime and attach storage

Select No end time if you want the instance to run until you terminate it. The form explains the credit hold and hourly deductions. This setting does not keep an instance running after the account runs out of credit.

Alternatively, choose a fixed estimated duration if you want an automatic end time. Read the billing summary before launching; it changes with the number of GPUs and the duration you select.

Attach persistent storage before starting work you need to keep.

A cloud drive attaches to one instance at a time. A shared filesystem can be attached to several instances. Both are retained after the compute instance ends. Choose a cloud drive for files used by this instance, or a shared filesystem when several instances need the same datasets, weights, or results.

Treat /workspace as temporary. Its local scratch data is erased when the instance ends, including when it stops because your account balance runs out. Write results you need to keep to an attached cloud drive or shared filesystem.

Review the GPU count, image, SSH keys, service port, duration, and storage attachments. Then click Launch. Allow extra time for the first download of a large container image.

Connect and check your application

Once the instance is running, its detail page shows the SSH command, the public URL if you enabled one, and a web terminal you can open in the browser. Copy the SSH command from that page rather than constructing the hostname or port yourself.

In the terminal, run:

nvidia-smi

For this configuration, the output should list eight RTX PRO 6000 GPUs. Check that your application can see all eight before starting a long job.

The public URL requires your console login or an EcoHash API key. To call it from a script, send the key in an Authorization: Bearer header. Paths after /service pass through to your application: /service/v1/models reaches the container as /v1/models.

For a vLLM server exposing that route, copy the instance's public URL and test it with:

export ECOHASH_SERVICE_URL="https://api.ecohash.com/gpu-instances/INSTANCE_ID/service"
# Set ECOHASH_API_KEY in your environment before running this request.
curl --fail-with-body "${ECOHASH_SERVICE_URL%/}/v1/models" \
  -H "Authorization: Bearer ${ECOHASH_API_KEY}"

Replace INSTANCE_ID with your instance ID, or use the full URL copied from the console. A ready vLLM server should return a model list. For another application, use one of its own HTTP routes instead.

If the request fails, check that the application has finished starting, listens on 0.0.0.0, and uses the same port you entered in the console. Also confirm that your request includes a valid API key.

Before ending the session, confirm that your outputs are on persistent storage. Terminate the instance when you are finished to stop compute charges.

Open GPU Instances to configure a node, or read more about launching an instance and container images.

Qwen-Image-2.1 on RTX PRO 6000: 15-Image Quality and Performance Test

Ecohash — Fri, 16 Oct 2026 03:53:35 +0000

Qwen-Image-2.1 combines text-to-image generation, image editing, multilingual typography, native 2K output, and RGBA transparency in one model. We wanted to answer a practical question for researchers considering a 96 GB workstation-class GPU: what quality and speed can one NVIDIA RTX PRO 6000 Blackwell Server Edition actually deliver?

On September 24, 2026, we ran 15 fixed-seed prompts through the official Diffusers pipeline. The set covered English and bilingual typography, portrait and product photography, architecture, landscapes, illustration, a transparent asset, and one native 2048 × 2048 render.

Representative outputs from the full 15-image run are included below. We did not rerun prompts to select more favorable outputs.

License note: This article documents an independent, non-commercial technical evaluation. Qwen-Image-2.1 is distributed under the Qwen Research License Agreement, which limits the model materials to non-commercial research or evaluation unless a separate commercial license is obtained. This article does not distribute model weights, provide a hosted Qwen-Image-2.1 service, or grant commercial-use rights.

Results at a glance

Approximately 1 megapixel

14 samples
BF16, 40 steps, batch size 1
9.378 seconds per image on average
36.801 GiB average peak allocated GPU memory

Native 2048 × 2048

1 sample
BF16, 40 steps, batch size 1
52.021 seconds
56.516 GiB peak allocated GPU memory
15/15 requested samples completed without an out-of-memory or CUDA error.
The 14 approximately 1 MP samples ranged from 9.271 to 9.524 seconds, with a 9.375-second median.
Warm-cache pipeline loading took 4.427 seconds. Initial model download time was excluded.
Total recorded generation time for all 15 images was 183.317 seconds.
At the measured sequential rate, 1 MP generation corresponds to roughly 384 images per GPU-hour before orchestration, file I/O, safety review, retries, or model-loading overhead.
The transparent sample contains a real alpha channel, rather than a simulated solid-color background.

These figures describe one configuration, not a universal model benchmark. We did not test batching, reduced-step schedules, quantization, CPU offload, or alternative inference runtimes.

Test environment

GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU memory reported by PyTorch: 95.0 GiB
Compute capability: 12.0
NVIDIA driver: 580.105.08
PyTorch: 2.11.0+cu128
PyTorch CUDA runtime: 12.8
Pipeline: Diffusers QwenImage21Pipeline
Precision: BF16, without quantization or CPU offload
Inference steps: 40
Batch size: 1
Test date: September 24, 2026

All images used deterministic recorded seeds. Peak-memory figures are PyTorch peak allocated memory for each generation, not total board power, reserved memory, or whole-system memory consumption.

Typography: the strongest part of this run

Four prompts explicitly required rendered text. In this single fixed-seed run, all four produced the requested words correctly. That is an encouraging result, but it is four observed samples, not a 100% text-accuracy claim. Three representative typography results are shown below.

English neon sign

“MAKE IDEAS VISIBLE” is correctly spelled and follows the perspective of the storefront rather than appearing as a flat overlay. Reflections, mist, glass, and mixed cyan-and-amber light make the result feel spatially integrated.

Educational infographic

The title and all four labels—“CLOUD,” “RAIN,” “SOIL,” and “RIVER”—are correct. The visual logic is less reliable: the cloud panel already shows rain, so the first two stages are not as distinct as the labels imply. This is a useful reminder that correct typography does not guarantee correct instructional content.

Bilingual storefront

This was the most demanding text sample: two languages, two lines, a fascia sign, interior shelves, reflective glass, and softly blurred pedestrians. Both requested lines rendered correctly in the requested order.

Photographic output

Editorial portrait

The fictional subject has natural-looking skin texture, warm window light, believable fabric, and visible clay dust. Hands and fingers remained plausible in this sample, though anatomy should always be reviewed at full resolution.

Product photography

The model separated matte polymer, woven grille, and copper-like metal convincingly. The floating composition and controlled shadow are useful for concept work. Exact product representation would still require reference conditioning and human verification.

Food editorial

The noodles, chili oil, sesame, scallions, ceramic bowl, and a subtle trace of steam are clearly differentiated. The prompt requested a top-down view, but the output is closer to a 45-degree overhead angle—an example of attractive output that does not fully obey the camera instruction.

E-commerce layout

The reusable bottle appears on the right with generous negative space on the left, matching the requested layout. Small condensation droplets and the soft shadow are handled cleanly without introducing a brand or logo.

Architecture and landscape

Reading room built around a tree

Curved timber walls, glazing, bookshelves, sunlight, and the central tree read as a coherent space. As with most generated architecture, junctions and construction details deserve close inspection before the image is used as more than a concept.

Alpine landscape after a storm

The narrow river catches the requested shaft of light, atmospheric depth remains readable across several ridgelines, and tiny hikers provide scale without dominating the scene.

Illustration and reusable assets

Storybook illustration

The result preserves the prompt’s gouache-and-colored-pencil character across the robot, luminous flowers, warm windows, and dusk sky. The character is fictional and was explicitly requested as a non-franchise design.

Transparent RGBA sticker

This PNG has a valid alpha channel with values from 0 to 255. Of 1,048,576 pixels, 989,473 are not fully opaque. That verifies actual transparency rather than a white or checkerboard background painted into the image. Some viewers may display transparent RGB values against a colored matte; that does not change the alpha data.

Vertical editorial composition

The 9:16 result reserves open sky for copy while keeping the fictional cliffside railway, turquoise water, and sunrise readable at social-media proportions.

3D application icon

The centered orthographic view, glass-like blue tile, ivory hourglass, and cyan inner glow form a clean fictional icon without text or third-party marks.

Native 2K output

The final prompt generated a 2048 × 2048 image directly, without a separate upscaler. It deliberately asked for a detail-heavy fictional observatory with dozens of small windows, railings, weathered copper, concrete, waves, rocks, and a distant horizon.

At normal viewing size, the result is coherent and richly textured. At 100% zoom, water staining, window frames, and material transitions remain useful, although a few railing paths and small structural junctions are not physically convincing.

The 2K render required 52.021 seconds and 56.516 GiB of peak allocated GPU memory. Compared with the approximately 1 MP group, it used about 1.54× the memory and 5.55× the time for roughly four times the pixels. Native high-resolution diffusion is therefore practical on this 96 GB card, but it is substantially more expensive than first generating at approximately 1 MP.

What the RTX PRO 6000 changes

The main advantage in this test was not merely that the model ran. It ran in BF16 without quantization, model sharding, or CPU offload, while retaining substantial memory headroom:

Approximately 1 MP generation peaked near 36.8 GiB, leaving roughly 58 GiB relative to the 95.0 GiB reported by PyTorch.
Native 2048 × 2048 generation peaked at 56.5 GiB, leaving roughly 38.5 GiB.
Runtime variation across the 14 similarly sized samples was under three percent from fastest to slowest.

That headroom may be useful for experimentation with concurrency, larger conditioning inputs, prompt-rewriting components, or alternative runtimes. This test did not measure those scenarios, so it should not be read as a batching or multi-user capacity claim.

For researchers optimizing total throughput, the result also suggests a practical workflow: generate candidates near 1 MP, review them, and reserve native 2K generation for selected outputs. That conclusion follows from the measured latency difference; it is not a statement about every pipeline or quality target.

Where the model still needs human review

Text: Four text prompts succeeded in this run, but longer copy, smaller type, unusual fonts, and denser layouts need broader testing.
Instruction following: The food image missed the requested top-down camera angle.
Semantic accuracy: The infographic rendered its labels correctly but did not clearly separate the first two stages.
Fine structure: Railings, window grids, hands, and other repeated small geometry should be checked at 100% zoom.
Production claims: Attractive output is not evidence that a depicted product, building, process, or person is accurate.

Conclusion

In this fixed-seed evaluation, Qwen-Image-2.1 produced a broad range of visually useful outputs on a single RTX PRO 6000. Approximately 1 MP images completed consistently in about 9.4 seconds at 40 steps, while native 2K completed in about 52 seconds. The card’s 96 GB memory class allowed the official BF16 Diffusers path to run without memory-saving compromises in the tested configuration.

The strongest observed results were multilingual typography, photographic material rendering, composition with deliberate copy space, and real RGBA transparency. The main limitations were familiar but important: semantic mistakes can survive correct typography, camera instructions may be missed, and tiny repeated structures still require human review.

For anyone evaluating this hardware-model combination, the practical takeaway is straightforward: one RTX PRO 6000 provides ample memory for Qwen-Image-2.1 research at both approximately 1 MP and native 2K, with predictable single-image latency in this configuration.

License, rights, and attribution

This evaluation was conducted independently by EcoHash for research and technical assessment. EcoHash is not affiliated with, endorsed by, or sponsored by Qwen, Alibaba, or NVIDIA in connection with this article.

Qwen-Image-2.1 is distributed under the Qwen Research License Agreement. The license permits use of the model materials for non-commercial research or evaluation and requires a separate license for commercial use. Readers must review the current upstream license and obtain any permissions required for their intended use. The official model repository is available on Hugging Face, and the official project repository is available on GitHub.

The example images in this article were generated from original prompts for this evaluation. They use fictional people, fictional places, unbranded objects, and non-franchise characters. AI-generated output is not automatically cleared of third-party copyright, trademark, publicity, privacy, or other rights; the applicable risk depends on the prompt, reference inputs, output, jurisdiction, and intended use. Users remain responsible for reviewing outputs and securing any necessary rights.

“Qwen,” “Alibaba,” “NVIDIA,” and “RTX” are names or marks of their respective owners and are used here only for factual identification. Product specifications and software licenses may change; verify current upstream documentation before relying on this evaluation.

How to deploy and tune Qwen3.8-27B on one RTX Pro 6000

Ecohash — Thu, 15 Oct 2026 04:08:40 +0000

Qwen3.8-27B is 27B dense, published in FP8 under Apache-2.0, and the weights plus runtime overhead come to about 29 GiB. A single RTX Pro 6000 holds it with a large KV cache still to spare, on one process, with no tensor parallelism to configure. This post is the serving config EcoHash runs in production, the settings whose obvious alternative fails, and the one flag that took a single card from 999.9 to 1954.0 output tokens/sec.

Everything needed to run it well is in the download, including the draft head that doubles what one card serves.

Why this model suits self-hosting

At FP8 the checkpoint is 28.75 GiB on disk, and vLLM reports 29.34 GiB of weights and non-torch overhead once loaded. On a 96 GB card that leaves 53.8 GiB for KV cache, which is 767,317 tokens. Nothing has to be split across devices and nothing runs close to the edge.

Apache-2.0 means no usage terms to read. Context is 262,144 tokens natively, extensible to 1M. The official model card reports 89.2 on GPQA Diamond, 90.3 on LiveCodeBench v6, 61.7 on SWE-bench Pro, and 73.0 on Terminal Bench 2.1. Artificial Analysis puts it at 52 on their Intelligence Index, level with GPT-5.6 Luna and one point behind GLM-5.2 (753B) and DeepSeek V4 Pro (1.7T). Downloads passed three million within days of the 14 August 2026 release.

The checkpoint also carries its own multi-token-prediction draft head, 22 tensors and 0.44 GiB, in the same download. Speculative decoding normally means training or sourcing a second model; here it is a flag.

Plan for one quirk. Artificial Analysis needed 160 million output tokens to run this model through their index, against a 43 million median for comparable open-weight models, and Simon Willison found the same thing: it is excellent, but it defaults to wildly overthinking things. Because it reasons at length by default, decode throughput rather than prefill is what repays tuning.

The serving config

This is what runs in production, on vLLM 0.27.1 with driver 580.105.08 and CUDA 13.0.

vllm serve /path/to/Qwen3.8-27B-FP8 \
  --served-model-name qwen3.8-27b \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --safetensors-load-strategy prefetch \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Five of those lines are there because the obvious alternative failed.

--dtype bfloat16, not float16. FP8 checkpoints dispatch to kernels that assert a bf16 output dtype. With float16 the engine loads every weight, reports success, then crash-loops at warmup. It cost a failed rollout on a sibling model before anyone connected the two.
No --quantization flag. The checkpoint is already FP8 (e4m3, weight block size 128x128), so vLLM reads quantization_config from config.json and loads straight through. Passing the flag anyway makes it quantize at runtime on every start: 2105 seconds of weight loading against 17 seconds pre-quantized.
--safetensors-load-strategy prefetch if the weights sit on network storage. vLLM does not always recognise a network filesystem, and sequential loading falls off a cliff past the eighth shard, from 94 seconds to 370 seconds per shard. This checkpoint is sharded per layer, so there are 66 of them.
--tool-call-parser qwen3_xml, not hermes. Qwen3 emits its native XML tool format rather than Hermes JSON. With the wrong parser you get the raw <tool_call> text and tool_calls: null, and every agent integration breaks quietly.
--max-num-seqs 128. This is a dense 27B, so every token touches every weight and the batching knee arrives earlier than it does on an A3B MoE, where 160 is the better setting.

--max-model-len 32768 is a deliberate cap rather than a limit of the model. It keeps the KV cache bounded when the card is shared. Raise it after checking the memory numbers, not before.

The flag worth tuning for

The draft head proposes several tokens, the model verifies them in one pass, and the accepted ones come out together. num_speculative_tokens, K below, is how many it proposes. Deeper drafts win more per step and get rejected more often, so K has an optimum that moves with the workload.

K=3 wins on all three workloads, so that is the setting. Returns are already flattening on chat, where K=2 to K=3 buys 6%, and the acceptance rate is what separates the workloads: predictable, low-entropy output drafts well, open-ended prose does not.

The effect holds as the card fills up. The dotted line is the 30 ms per-token budget that defines a usable interactive response.

Per-token latency drops from 22 ms to 13 ms on a single stream, and the number of concurrent requests one card holds inside that budget goes from 32 to 64. Aggregate output at the ceiling goes from 999.9 to 1954.0 tokens/sec. Measured in August 2026 on one card at 150-token replies; longer generations shift the numbers, and the full sweep is in the write-up.

The cost is cache. The draft head brings an attention layer that needs KV of its own, so capacity falls from 767,317 tokens to 592,164, a 23% cut. At 48 to 64 concurrent short requests that is not the binding constraint, but it would become one before pushing concurrency much higher.

The full write-up has the draft-depth sweep, acceptance rates by position, the whole concurrency table, the memory accounting, and the quality checks: Qwen3.8-27B with MTP speculative decoding. The raw CSVs are in the same repository.

Use the API instead

All of the above is already running. Calling qwen3.8-27b on EcoHash gets the tuned deployment with nothing to provision: 179 ms to first token, about 78 tokens/sec on a single stream, $0.30 per 1M input tokens and $0.60 per 1M output.

Peer numbers are as published by OpenRouter in August 2026. Time to a finished answer is the first token plus the rest of the stream, so a 500-token answer is roughly one step of an agent loop, and the whole field except one provider needs at least 1.7 times as long as EcoHash to produce it.

Blended at three parts input to one part output, that is $0.375 per 1M tokens against $1.05 to $1.21 elsewhere. Almost all of the gap is the output side, $0.60 against $3.00 to $3.40, and output is the side a reasoning model spends on. An agent making 400 requests an hour at 800 input and 2,000 output tokens each costs $0.58 an hour here and $2.70 at $0.45 / $3.20.

The API is OpenAI-compatible, so only the base URL and the model name change.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ecohash.com/v1",
    api_key="eco_...",  # create a key at console.ecohash.com
)

resp = client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[
        {"role": "user", "content": "Refactor this function and explain the change."},
    ],
    stream=True,
)

for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Tool calling and structured outputs work, and images go in the same messages array. The shared API serves a 32k context rather than the checkpoint's 262k, which is what keeps the KV cache bounded on a shared card.

Or run it yourself on the same card

If the config above is what you want but the hardware is not, an RTX Pro 6000 workspace rents by the second at $1.89 per GPU-hour as of August 2026, with root access and a browser terminal. Pull the FP8 checkpoint, paste the vllm serve line, and the same deployment is yours to change: the full 262k context, your own --max-num-seqs, no shared-tenancy cap.

Which is the argument for self-hosting this particular model. One card covers it, and the tuning that moves the numbers is a flag you can read in a config file.

FAQ

Is this the full model or a quantized version? FP8, which is how the Qwen team publishes it. The precision is stated because not every provider states theirs.

Do I need to do anything to get the speculative decoding speedup on the API? No. It is on for every request to qwen3.8-27b.

Will it fit on a 48 GB card? The weights alone are about 29 GiB, so it loads, but the KV cache left over is small enough that concurrency and context both suffer. The 96 GB card is what makes the deployment comfortable rather than merely possible.

Does MTP change the output? Verification means the accepted tokens are the ones the model would have produced anyway. Answers matched on every checkable question in our quality pass; the write-up explains why a byte-for-byte comparison is not the right test.

Does it accept images? Yes. It is a vision-language model and takes images in the standard OpenAI content format.

Next steps

Call the model: /models
Rent the card and run the config above: /gpu-compute
Read the benchmark: Qwen3.8-27B with MTP speculative decoding
Reserve steady capacity: /dedicated-inference

Run your Retell AI agent on an open model: the custom LLM bridge

Ecohash — Wed, 14 Oct 2026 10:44:40 +0000

Retell AI prices the brain of your voice agent by the minute: the LLM line on its pricing page runs from $0.045 to $0.16 per minute for frontier models, on top of the voice engine. Retell also has a custom LLM slot that skips that line entirely: it opens a WebSocket to your server during the call, streams the live transcript, and speaks whatever text you stream back. That slot is a clean fit for an open model on EcoHash. A small bridge server, about 90 lines, receives Retell's events and answers with streaming completions from llama-3.1-8b-instruct, which costs a fraction of a cent per call minute in tokens. This post walks through the protocol, the bridge, a local test that does not need a phone call, and the honest math.

Same agent, cheaper brain. Retell's custom LLM slot plus a 90-line bridge puts an open model behind your voice agent for well under a cent per minute in tokens.

What you are wiring together

Retell slot: Custom LLM, WebSocket; Retell connects to your server per call
Protocol: Retell sends transcript events; you stream back response events
Bridge: integrations/retell in ecohash-examples, about 90 lines
Default model: llama-3.1-8b-instruct, $0.10 in / $0.10 out per 1M tokens
Measured on EcoHash: 205 ms TTFT (p95), ~143 tokens/sec single stream
What stays on Retell: Voice engine, STT, TTS, telephony (their built-in providers)
API base URL: https://api.ecohash.com/v1

How the pieces fit

Retell does not call a chat completions URL directly. When a call starts, Retell opens a WebSocket to your server and the two sides exchange JSON events:

Your server sends a config event and a first sentence to speak.
When the caller finishes a turn, Retell sends response_required with the live transcript and a response_id.
The bridge forwards the transcript to EcoHash /v1/chat/completions with stream: true.
Each token delta goes back to Retell as a response event; a final content_complete closes the turn. If the caller interrupts, a newer response_id arrives and the stale reply is dropped.

The bridge

The full server is in integrations/retell/server.py. The core of it:

@app.websocket("/llm-websocket/{call_id}")
async def llm_websocket(ws: WebSocket, call_id: str):
    await ws.accept()
    await ws.send_json({"response_type": "config",
                        "config": {"auto_reconnect": True, "call_details": False}})
    await ws.send_json({"response_type": "response", "response_id": 0,
                        "content": BEGIN_SENTENCE, "content_complete": True})

    async for event in ws.iter_json():
        if event.get("interaction_type") in ("response_required", "reminder_required"):
            stream = await client.chat.completions.create(
                model="llama-3.1-8b-instruct",
                messages=to_messages(event["transcript"]),
                stream=True,
            )
            async for chunk in stream:
                delta = chunk.choices[0].delta.content or ""
                if delta:
                    await ws.send_json({"response_type": "response",
                                        "response_id": event["response_id"],
                                        "content": delta, "content_complete": False})
            await ws.send_json({"response_type": "response",
                                "response_id": event["response_id"],
                                "content": "", "content_complete": True})

Run it and expose it publicly (ngrok is fine for a first test):

pip install fastapi uvicorn openai
export ECOHASH_API_KEY=eco_...   # create one at console.ecohash.com
uvicorn server:app --host 0.0.0.0 --port 8000

The key comes from the EcoHash console: create a free account, open API Keys, create one, and copy the eco_ value into ECOHASH_API_KEY.

Point the agent at it

In the Retell dashboard, click Create an Agent, open Other options, and pick Custom LLM:

Then set the Custom LLM URL to your bridge:

wss://your-host.example.com/llm-websocket

Retell appends the call id and connects when a call starts. Note the cost estimate in the agent header: it now covers voice and telephony only, with no LLM line in it, which is the point of this post. Everything else about the agent, voice, transcriber, phone number, stays exactly as it was. Retell's own custom LLM setup guide covers the dashboard walkthrough in detail; this bridge is the part their guide leaves as an exercise, wired to an open model.

Test it without making a call

The bridge speaks a documented protocol, so you can fake Retell from your terminal with a WebSocket client:

import asyncio, json, websockets

async def main():
    async with websockets.connect("ws://localhost:8000/llm-websocket/test-1") as ws:
        print(json.loads(await ws.recv()))          # config
        print(json.loads(await ws.recv())["content"])  # first sentence
        await ws.send(json.dumps({
            "interaction_type": "response_required", "response_id": 1,
            "transcript": [{"role": "user", "content": "Do you deliver on weekends?"}]}))
        reply = ""
        while True:
            ev = json.loads(await ws.recv())
            reply += ev.get("content", "")
            if ev.get("content_complete"):
                break
        print(reply)

asyncio.run(main())

We ran this exact loop while writing the post: the bridge answered the config handshake, spoke its first sentence, and streamed a coherent 18-chunk reply from llama-3.1-8b-instruct, with ping and interrupt handling working as documented.

What it costs

With Retell's built-in models, the LLM is billed per minute. With a custom LLM, that line drops off Retell's bill and you pay your own provider per token instead. List prices, checked 2026-07-23:

GPT 5.5 via Retell: $0.16
GPT 5.4 or Claude 4.6 Sonnet via Retell: $0.08
GPT 4.1 via Retell: $0.045
llama-3.1-8b-instruct on EcoHash via bridge: ~$0.0003 in tokens

The EcoHash figure assumes a typical support call burns about 2,000 input and 250 output tokens per minute at $0.10 per 1M each; even a context-heavy agent at ten times that usage stays under a cent. Retell's other lines, the voice engine at $0.055 per minute plus TTS and telephony, are unchanged, so the saving applies to the LLM line only. You also take on hosting a small server, which is the real trade.

The trade-offs

The custom LLM slot covers the brain only. Retell's STT and TTS stay on its built-in providers, so you cannot plug EcoHash's Kokoro in here. For that pattern, Vapi has an open TTS slot: see Give your Vapi voice agent a cheaper voice.
You are hosting a WebSocket server on the call path. Keep it close to Retell's region and return the first token fast; llama-3.1-8b-instruct measures 205 ms TTFT (p95) on EcoHash, and your hosting adds on top. Measure before promising numbers.
Voice replies should be short. Keep the system prompt strict about one-sentence answers or the agent will monologue.
The cost table uses list prices as of the date above and stated token assumptions. Recheck both before building a business case.

FAQ

Does Retell support an OpenAI-compatible base URL directly? No. The custom LLM slot is a WebSocket protocol, not a base URL field, which is why you run a small bridge. The bridge in this post is about 90 lines and the protocol is documented.

Which models can the bridge use? Any chat model on EcoHash. llama-3.1-8b-instruct is a good default for voice; gpt-oss-20b adds capability at $0.05 in / $0.18 out per 1M tokens. See /models.

What happens when the caller interrupts? Retell sends a new response_required with a higher response_id. The bridge tracks the latest id and drops any reply still streaming for an older one.

Can I add tool calls or RAG? Yes, that is the point of owning the bridge: run retrieval, call tools, or apply guardrails between the transcript and the model. The same key also serves /v1/embeddings and a reranker for RAG.

Does this change Retell's voice quality or latency budget? Voice stays on Retell's engine. Your contribution to latency is bridge hosting plus model TTFT, so host near Retell and pick a fast model.

How do I transcribe or analyze calls afterwards? Use EcoHash whisper-large-v3-turbo on the recordings and a chat model for summaries, on the same API key. See Build a voice agent with Whisper, Kokoro, and an OpenAI-compatible API.

Next steps

Create a free EcoHash API key: console.ecohash.com
Bridge source: integrations/retell
Browse chat models and prices: /models
The Vapi version of this pattern, for the voice side: /blog/vapi-custom-tts-kokoro
LLM benchmarks: ecohash-benchmarks

Give your Vapi voice agent a cheaper voice: Kokoro TTS via custom-voice

Ecohash — Mon, 12 Oct 2026 08:16:59 +0000

Vapi lets you swap the voice of an agent for any TTS you want through its custom-voice provider: it POSTs the text to your server and plays back whatever PCM you return. That makes it a clean slot for Kokoro on EcoHash, an open 82M-parameter TTS that costs about $0.006 per minute of generated speech and returns first audio in roughly 120 ms. The whole integration is one small adapter server, about 60 lines: it receives Vapi's voice-request, calls EcoHash's OpenAI-compatible /v1/audio/speech, resamples to the rate Vapi asks for, and sends the raw PCM back. This post walks through the adapter, the assistant config, a local test that does not need a phone call, and the per-minute math against ElevenLabs.

Swap the voice, keep the agent. A 60-line adapter puts Kokoro behind Vapi at about $0.006 per minute of speech, roughly 7x below ElevenLabs Flash list price.

Who this is for

Teams running voice agents on Vapi who want to cut the TTS line item.
Developers who want an open TTS model behind their agent instead of a proprietary voice.
Anyone comparing per-minute costs across TTS providers before scaling outbound calls.

What you can build

A Vapi assistant that speaks with Kokoro at a fraction of the usual TTS cost.
The same adapter pattern for any OpenAI-compatible TTS endpoint.
A post-call pipeline that also transcribes recordings with whisper-large-v3-turbo on the same API key.

Facts

TTS model: kokoro-82m, open weights, served on EcoHash
Price: $1 per 1M audio tokens, about $0.006 per minute of speech
First audio: ~120 ms measured end-to-end on EcoHash
Vapi slot: custom-voice provider, HTTP POST, raw PCM response
Sample rates: Vapi requests 8000, 16000, 22050, or 24000 Hz; Kokoro is 24 kHz native
Adapter: integrations/vapi in ecohash-examples
API base URL: https://api.ecohash.com/v1

How the pieces fit

Vapi does not call EcoHash directly, because its custom-voice contract is not the OpenAI speech API. When the agent needs to speak, Vapi POSTs a JSON voice-request with the text and a target sample rate, and expects headerless 16-bit mono PCM back at exactly that rate. So you run a small adapter in the middle:

Vapi POSTs {"message": {"type": "voice-request", "text": ..., "sampleRate": ...}} to your server.
The adapter calls EcoHash /v1/audio/speech with model: kokoro-82m and response_format: pcm.
Kokoro returns 24 kHz PCM; the adapter resamples if Vapi asked for a different rate.
The adapter answers with application/octet-stream PCM, and the caller hears Kokoro.

The adapter

The full server is in integrations/vapi/server.py. The core of it:

@app.post("/synthesize")
async def synthesize(request: Request) -> Response:
    if VAPI_SECRET and request.headers.get("x-vapi-secret") != VAPI_SECRET:
        raise HTTPException(status_code=401)

    msg = (await request.json()).get("message", {})
    text = msg.get("text", "")
    sample_rate = int(msg.get("sampleRate", 24000))

    async with httpx.AsyncClient(timeout=30) as client:
        r = await client.post(
            "https://api.ecohash.com/v1/audio/speech",
            headers={"Authorization": f"Bearer {ECOHASH_KEY}"},
            json={"model": "kokoro-82m", "voice": "af_heart",
                  "input": text, "response_format": "pcm"},
        )

    pcm = resample(r.content, 24000, sample_rate)
    return Response(content=pcm, media_type="application/octet-stream")

Run it and expose it publicly (ngrok is fine for a first test):

pip install fastapi uvicorn httpx numpy
export ECOHASH_API_KEY=eco_...          # create one at console.ecohash.com
export VAPI_SECRET=your-webhook-secret
uvicorn server:app --host 0.0.0.0 --port 8000

The key comes from the EcoHash console: open API Keys, create one, and copy the eco_ value into ECOHASH_API_KEY.

Point the assistant at it

In your assistant config, set the voice to the custom-voice provider with your adapter's URL:

{
  "voice": {
    "provider": "custom-voice",
    "server": {
      "url": "https://your-host.example.com/synthesize",
      "secret": "your-webhook-secret",
      "timeoutSeconds": 30
    }
  }
}

If you prefer the dashboard, it is the same three fields under Assistant → Voice: set the provider to Custom voice and paste your adapter's /synthesize URL.

Vapi sends the secret back in the x-vapi-secret header on every request, and the adapter rejects anything that does not match.

Test it without making a call

You can verify the whole path from your terminal by sending the same request Vapi would:

curl -s http://localhost:8000/synthesize \
  -H "Content-Type: application/json" \
  -H "x-vapi-secret: your-webhook-secret" \
  -d '{"message":{"type":"voice-request","text":"Your order number seven two nine is confirmed.","sampleRate":16000}}' \
  -o reply.pcm

ffplay -f s16le -ar 16000 -ch_layout mono reply.pcm

We ran this exact loop while writing the post, at 8, 16, and 24 kHz, and fed the resampled PCM back into whisper-large-v3-turbo on EcoHash. The transcription came back word for word, so the resampling holds up at telephony rates.

What it costs

Kokoro on EcoHash is billed at $1 per 1M audio tokens, and a second of generated speech is 100 tokens. One minute of speech is 6,000 tokens, or $0.006. List prices for the common alternatives, checked 2026-07-13:

Kokoro on EcoHash · List price: $1 / 1M audio tokens · Rough cost per audio minute: ~$0.006
ElevenLabs Flash/Turbo (API) · List price: $0.05 / 1k characters · Rough cost per audio minute: ~$0.04
ElevenLabs Multilingual (API) · List price: $0.10 / 1k characters · Rough cost per audio minute: ~$0.08

The per-minute figures for ElevenLabs assume roughly 800 spoken characters per minute; your text and pacing will move them. Even so, the gap is around 7x to 13x, which adds up quickly on high-volume outbound calls. Voice quality is a judgment call: Kokoro is a small open model, so test it on your own scripts before switching production traffic.

Bonus: run the brain on EcoHash too

Vapi also supports a custom OpenAI-compatible LLM. In the dashboard, open your assistant's Model section, choose Custom LLM, and set the endpoint to https://api.ecohash.com/v1 with your EcoHash key as the API key and a model such as llama-3.1-8b-instruct. That puts the LLM and the voice on one provider and one bill; keep replies short in the system prompt so turns stay snappy.

Limitations

Live call transcription stays on Vapi's built-in transcribers. Vapi's custom transcriber slot expects a streaming WebSocket, and EcoHash's transcription endpoint is batch. Use EcoHash whisper-large-v3-turbo for post-call transcription instead.
Latency depends on where you host the adapter. Kokoro's first audio is ~120 ms measured end-to-end on EcoHash; keep the adapter close to your region and the total stays comfortable for calls, but measure your own setup before promising numbers.
The cost table uses list prices on their published pages as of the date above. Recheck before you build a business case on them.
Model IDs and voices change. Confirm kokoro-82m and the voice list on /models/kokoro-82m before you build.

FAQ

Can Vapi use any TTS provider? Yes, through the custom-voice provider. Vapi POSTs the text to your server and plays whatever PCM you return, so anything with a synthesis API works behind a small adapter.

Why do I need an adapter at all? Vapi's custom-voice contract is its own JSON request and raw PCM response, not the OpenAI speech API. The adapter translates between the two and resamples the audio.

What sample rate should I return? Exactly what Vapi asks for in sampleRate. The adapter in this post resamples Kokoro's 24 kHz output to 8000, 16000, or 22050 Hz when needed.

How much does Kokoro cost on EcoHash? $1 per 1M audio tokens, which is about $0.006 per minute of generated speech. See /pricing.

Can I also do speech-to-text on EcoHash for Vapi calls? Not live: Vapi's custom transcriber needs streaming input. For recordings and post-call analysis, whisper-large-v3-turbo on the same key works well.

Which voices does Kokoro have? Several; this post uses af_heart. See /models/kokoro-82m.

Next steps

Adapter source: integrations/vapi
Kokoro on EcoHash: /models/kokoro-82m
Voice and speech on EcoHash: /use-cases/voice-speech
Speech benchmarks: ecohash-benchmarks

Build a voice agent with Whisper, Kokoro, and an OpenAI-compatible API

Ecohash — Sat, 10 Oct 2026 07:03:48 +0000

A voice agent turns speech into speech: it transcribes what the user says, sends the text to a language model, and speaks the reply back. On EcoHash you build all three stages through one OpenAI-compatible API and one key, so there are no three vendors and no three billing accounts to stitch together. Whisper (whisper-large-v3-turbo) does speech to text, a chat model such as llama-3.1-8b-instruct writes the reply, and Kokoro (kokoro-82m) turns it into audio. None of it needs a GPU of your own, since the models are served for you. This post walks through the pipeline, the three API calls, and when an RTX Pro 6000 workspace is worth it for heavier voice work.

Three API calls, one key. Whisper hears, a chat model thinks, Kokoro speaks. About $0.006 per minute of generated speech.

Who this is for

Developers building a voice assistant, an IVR replacement, or a spoken interface for an app.
Teams that want speech to text, an LLM, and text to speech from one provider rather than three.
Anyone prototyping a voice feature before deciding on dedicated capacity.

What you can build

A back-and-forth voice assistant that listens, thinks, and speaks.
A transcription-plus-reply feature inside an existing app.
A spoken front end for a chatbot you already run on EcoHash.

How a voice agent works

The loop has three stages, and each maps to one EcoHash endpoint:

Speech to text. The user's audio goes to Whisper, which returns text. This is the transcription endpoint.
LLM. The text goes to a chat model, which returns a reply. This is the chat completions endpoint.
Text to speech. The reply goes to Kokoro, which returns audio you play back. This is the speech endpoint.

Then you repeat for the next turn. The whole loop uses one key and one base URL, so the three stages share auth, billing, and SDK.

The models

Speech to text · Model: Whisper Large V3 Turbo · Model ID: whisper-large-v3-turbo
Speech to text (alt) · Model: Qwen3 ASR · Model ID: qwen3-asr-1-7b
LLM · Model: Llama 3.1 8B Instruct · Model ID: llama-3.1-8b-instruct
Text to speech · Model: Kokoro 82M · Model ID: kokoro-82m
Text to speech (alt) · Model: Qwen3 TTS · Model ID: qwen3-tts

Use a small or mid-size chat model for the LLM stage so replies come back fast. A coding-specialized model is the wrong pick for general conversation.

Here is what the speech models measure on a single RTX Pro 6000, end-to-end, in July 2026. Speech to text:

qwen3-asr-1-7b · WER %: 3.28 · Peak RTFx: 360 · Price: input $0.05/1M
whisper-large-v3 · WER %: 3.64 · Peak RTFx: 45 · Price: $0.006/min
whisper-large-v3-turbo · WER %: 4.37 · Peak RTFx: 59 · Price: $0.006/min

Text to speech:

kokoro-82m · TTFA: 120 ms · Price: $1/1M
qwen3-tts · TTFA: 2621 ms · Price: $2/1M

On the HF Open ASR Leaderboard, the models EcoHash serves land among the accurate ones (purple = served on EcoHash):

Full data and method: ecohash-benchmarks.

The three API calls

The API is OpenAI-compatible, so the OpenAI SDK works once you set the base URL and key. One client covers all three stages.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ecohash.com/v1",
    api_key="eco_...",  # create a key at console.ecohash.com
)

# 1. Speech to text
with open("input.wav", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3-turbo",
        file=f,
    ).text

# 2. LLM reply
reply = client.chat.completions.create(
    model="llama-3.1-8b-instruct",
    messages=[
        {"role": "system", "content": "You are a concise voice assistant. Keep replies short."},
        {"role": "user", "content": transcript},
    ],
).choices[0].message.content

# 3. Text to speech
with client.audio.speech.with_streaming_response.create(
    model="kokoro-82m",
    voice="af_heart",
    input=reply,
    response_format="wav",
) as speech:
    speech.stream_to_file("reply.wav")

print("You said:", transcript)
print("Assistant:", reply)

That is a full single-turn voice agent. For multi-turn, keep the message history and run the loop again for each new audio input.

Running it without a GPU

You do not need one. All three models are served through the API, so you can build and run a voice agent with no hardware to provision, and Kokoro in particular is small enough that it never needs an RTX Pro 6000.

A workspace helps later, not at the start: when you want to co-locate a heavier speech-to-text, LLM, and text-to-speech pipeline on one card, push higher throughput, or run your own serving stack. That is a scaling choice. See /gpu-compute.

Limitations

This post does not promise real-time or a specific latency. End-to-end voice latency depends on network, the models you pick, whether you stream, and your own pipeline. Measure it on your setup and cite the method and date for any number you publish.
Whisper and Kokoro do not require an RTX Pro 6000. They run through the API and move onto GPU capacity only when throughput demands it.
The example is request-response. A production voice agent adds streaming audio, voice activity detection, turn-taking, and barge-in, which are your application's job.
Model IDs and voices change. Confirm them on /models, /models/whisper-large-v3-turbo, and /models/kokoro-82m before you build.

FAQ

What is a voice agent? A system that listens, thinks, and speaks: speech to text, then an LLM, then text to speech. The user talks, and the agent replies in audio.

Which models do I need? Three: a speech-to-text model like whisper-large-v3-turbo, a chat model like llama-3.1-8b-instruct, and a text-to-speech model like kokoro-82m.

Can I do all of it with one API key? Yes. On EcoHash the transcription, chat, and speech endpoints share one base URL and one key, so there is no three-vendor integration.

Do I need an RTX Pro 6000 to run Kokoro or Whisper? No. Both run through the API. A workspace is optional, for co-locating a heavier pipeline or higher throughput.

Is it fast enough for real-time conversation? That depends on your network, model choices, streaming, and pipeline. This post does not publish latency numbers; measure your own setup before relying on a target.

Which voices does Kokoro support? Several; the example uses af_heart. See /models/kokoro-82m.

How much does a voice agent cost? The LLM stage is billed per token, and the audio stages by what they process. See /pricing for current rates.

Which chat model should I use for the LLM stage? A small or mid-size general model keeps replies quick. See Best models to run on RTX Pro 6000 96GB.

Next steps

See voice and speech on EcoHash: /use-cases/voice-speech
Speech-to-text model: /models/whisper-large-v3-turbo
Text-to-speech model: /models/kokoro-82m
Browse all models: /models

Qwen3 Coder 30B on RTX Pro 6000: API, dedicated endpoint, or GPU workspace?

Ecohash — Tue, 29 Sep 2026 03:48:15 +0000

Three ways to run Qwen3 Coder 30B (qwen3-coder-30b-a3b-instruct), and the right one comes down to your traffic and how much control you want. The shared OpenAI-compatible API gets you a first call in minutes and bills per token, which fits prototypes and bursty traffic. A dedicated endpoint reserves capacity for steadier throughput and predictable latency in production, still with no GPU to manage. An hourly RTX Pro 6000 workspace gives you the whole card for fine-tuning, batch jobs, or your own serving stack. The model and the API contract are the same across all three, so moving up later does not mean a rewrite.

Same model, three ways to run it. Start on the per-token API, reserve a dedicated endpoint for production, rent the card itself for fine-tuning.

Facts


Model	Qwen3 Coder 30B (`qwen3-coder-30b-a3b-instruct`)
Type	Mixture-of-Experts, about 30B total parameters
Context served	32,768 tokens
API price	input $0.10 / output $0.30 per 1M tokens (pricing)
GPU workspace	RTX Pro 6000 at $1.89/GPU-hour
API base URL	`https://api.ecohash.com/v1`
Hugging Face	`Qwen/Qwen3-Coder-30B-A3B-Instruct`

On a single RTX Pro 6000, end-to-end, we measure it at 30 ms to first token and 9 ms per output token, about 110 tokens per second on a single stream (July 2026). A responsive coding assistant lives or dies on time to first token, and in a same-model provider comparison (llama-3.1-8b-instruct across providers, EcoHash in purple) it comes in lowest in the field:

Full data, method, and charts: ecohash-benchmarks.

Who this is for

Developers adding Qwen3 Coder to a coding assistant, an agent, or a code-review tool.
Teams deciding whether to call a shared API, reserve capacity, or run their own GPU.
Anyone moving a Qwen3 Coder prototype toward steady production traffic.

What you can decide

Which of the three paths fits your current stage.
What changes and what stays the same when you move between them.
When it is worth reserving a dedicated endpoint or renting a workspace.

The shared API

The shared API serves Qwen3 Coder through an OpenAI-compatible endpoint at https://api.ecohash.com/v1. You send a request, get a completion, and pay per token. There is no GPU to provision and no capacity to plan, and because traffic runs on shared infrastructure, latency moves with overall load.

This is where most projects should start, and it suits bursty traffic well since you pay for what you use rather than for reserved capacity. See inference.

The dedicated endpoint

A dedicated endpoint reserves capacity for Qwen3 Coder so your traffic is not competing with anyone else's. You still call the same API and still manage no GPU. What you gain is steadier throughput and more predictable latency, which starts to matter once a coding feature is in production and people expect consistent response times.

Reach for it when your traffic is steady and high enough that predictability is worth reserving capacity for. See dedicated inference.

The GPU workspace

A GPU workspace is an RTX Pro 6000 you rent by the hour at $1.89/GPU-hour. You get the whole card and control of the environment, so you can run a custom serving stack such as vLLM or SGLang, fine-tune with LoRA or QLoRA, or run batch jobs. Qwen3 Coder 30B fits on one 96 GB card with room for context and batching.

Reach for it when the API and dedicated endpoints do not give you enough control, or when the work is not request-response serving at all, like fine-tuning or offline batch generation. See GPU compute and fine-tuning.

How the three compare

	Shared API	Dedicated endpoint	GPU workspace
What it is	Call the model over the API	Reserved capacity for the model	Rent the RTX Pro 6000 by the hour
You manage	Nothing	Nothing	The environment and serving stack
How you pay	Per token	Reserved capacity	Per GPU-hour ($1.89)
Latency	Varies with shared load	Steadier, more predictable	The full card is yours
Scaling	Elastic	Reserved throughput	One card, or more on request
Best for	Prototypes, spiky traffic	Steady production traffic	Fine-tuning, batch, custom serving
Where	inference	dedicated inference	GPU compute

The model ID and the API contract hold across the first two, and the same model runs in the workspace, so moving between them does not force a rewrite.

Calling Qwen3 Coder

The API is OpenAI-compatible, so existing OpenAI SDK code works once you change the base URL and key. The same call works on the shared API or a dedicated endpoint.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ecohash.com/v1",
    api_key="eco_...",  # create a key at console.ecohash.com
)

resp = client.chat.completions.create(
    model="qwen3-coder-30b-a3b-instruct",
    messages=[
        {"role": "user", "content": "Write a Python function that parses an ISO 8601 timestamp into a datetime."}
    ],
)
print(resp.choices[0].message.content)

curl https://api.ecohash.com/v1/chat/completions \
  -H "Authorization: Bearer $ECOHASH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-coder-30b-a3b-instruct",
    "messages": [{"role": "user", "content": "Write a Python function that parses an ISO 8601 timestamp into a datetime."}]
  }'

When to move up

Prototype and early traffic: shared API. You are still changing prompts and shipping features, and per-token billing keeps cost tied to use.
Consistent production traffic: dedicated endpoint. Once users notice latency and volume is steady, reserved capacity buys predictability without infrastructure work.
Full control or non-serving work: GPU workspace. Fine-tuning, batch generation, and custom serving need the card itself.

You do not choose once. Plenty of teams run the API in development, a dedicated endpoint in production, and spin up a workspace only when they fine-tune.

Limitations

The context served here is 32,768 tokens. Do not assume a larger window unless it is confirmed for this deployment.
Performance depends on quantization, batch size, and context length. Reproduce any figure with its method and date.
Dedicated endpoint and workspace pricing depend on configuration. Check pricing and dedicated inference.
Model IDs change as the catalog changes. Confirm qwen3-coder-30b-a3b-instruct on the model catalog before you build.

FAQ

Can I run Qwen3 Coder 30B on an RTX Pro 6000? Yes. It is a Mixture-of-Experts model with about 30B total parameters and fits on one 96 GB RTX Pro 6000 with room for context and batching.

What is the difference between the API and a dedicated endpoint? Both use the same OpenAI-compatible API and neither asks you to manage a GPU. The shared API bills per token on shared infrastructure; a dedicated endpoint reserves capacity for steadier throughput and more predictable latency.

When should I rent a GPU workspace instead? When you need the whole card: fine-tuning with LoRA or QLoRA, batch generation, or a custom serving stack such as vLLM or SGLang.

What context length does Qwen3 Coder have here? 32,768 tokens as served.

How much does Qwen3 Coder cost? Through the API, $0.10 per 1M input tokens and $0.30 per 1M output tokens. A GPU workspace is $1.89/GPU-hour. See pricing for current rates.

Do I have to change my code to switch paths? No. The model ID and API contract stay the same across the shared API and a dedicated endpoint, and the same model runs in a workspace.

Which models pair well with Qwen3 Coder for a coding product? See Best models to run on RTX Pro 6000 96GB for embeddings, rerankers, and chat models that share the same key.

Next steps

See the model page: Qwen3 Coder 30B
Start on the shared API: ecohash.com/inference
Reserve a dedicated endpoint: ecohash.com/dedicated-inference
Rent an RTX Pro 6000: ecohash.com/gpu-compute

What makes RTX Pro 6000 96GB a good GPU for AI inference?

Ecohash — Wed, 23 Sep 2026 01:47:33 +0000

The RTX Pro 6000 Blackwell Server Edition gives you 96 GB of GDDR7 on one card, and for inference that capacity is the whole point. 96 GB holds a 20B to 35B open model with room left for KV cache and larger batches, so you serve it on a single GPU instead of splitting it across several. In the size range most teams actually deploy, that keeps both latency and operations simpler. On EcoHash the same card backs three ways to use it, a shared OpenAI-compatible API, a dedicated endpoint, and an hourly GPU workspace, and you move between them without changing the model or your code. It is a strong fit for mid-sized open models, and it is not the card for the very largest ones.

96 GB on one card. A 20B to 35B open model, its KV cache, and real batch sizes fit on a single GPU at $1.89 per hour.

Who this is for

Developers checking whether a single 96 GB card can host the open model they want to serve.
Teams weighing a shared model API against dedicated or self-managed GPU capacity.
Anyone sizing an RTX Pro 6000 for 20B to 35B models before spending on it.

What you can decide

Whether your model fits on one card, or needs quantization or more GPUs.
Which path fits your stage: shared API, dedicated endpoint, or workspace.
When a model is large enough that a different GPU is the better call.

Facts


Hardware	NVIDIA RTX Pro 6000 Blackwell Server Edition
VRAM	96 GB GDDR7
Architecture	Blackwell generation
Interconnect	PCIe (single-GPU and multi-instance inference)
Best-fit models	20B to 35B open models, plus selected quantized larger models
EcoHash GPU price	$1.89 per GPU-hour (pricing)
Access on EcoHash	Shared API, dedicated endpoint, GPU workspace
API base URL	`https://api.ecohash.com/v1`

Why 96 GB matters

VRAM sets the ceiling on what you can serve. The weights have to fit, and so does the KV cache, which grows with batch size and context length. Once the weights eat most of the card, you are left with small batches, short context, or a model split across GPUs.

96 GB changes that math for mid-sized models. A 30B-class model in a memory-efficient format still leaves room for concurrent requests and long prompts on one card, so you skip tensor-parallel serving and the latency and operational overhead that come with it. A 7B or 13B model has even more headroom, which you spend on bigger batches and longer context rather than leaving idle.

Blackwell, on a single card

The RTX Pro 6000 is a Blackwell-generation card on PCIe. EcoHash runs it as single-GPU and multi-instance capacity rather than an NVLink cluster, which suits inference: one card serves one model, or hosts a few smaller ones side by side.

Blackwell also handles the low-precision formats modern inference leans on, such as FP8, which lets a larger model fit in the same memory and can lift throughput. How much depends on the model and the serving stack, so treat the 96 GB as a fixed fact and treat quantization gains as something to measure on your own workload.

How it compares to other inference GPUs

These are published hardware specs, not our own benchmarks, and they are here for context.

GPU	VRAM	Interconnect	Typical best fit
RTX Pro 6000 (Blackwell)	96 GB GDDR7	PCIe	Single-GPU serving of 20B to 35B models; headroom on smaller ones
RTX 6000 Ada (previous gen)	48 GB GDDR6	PCIe	Smaller models, tighter memory budgets
L40S	48 GB GDDR6	PCIe	Cost-focused serving of smaller models
H100	80 GB HBM3	NVLink	High-throughput and multi-GPU serving of large models
H200	141 GB HBM3e	NVLink	Very large models and long-context workloads

The H100 and H200 use HBM with higher bandwidth and NVLink, which favors large models and multi-GPU tensor parallelism. The RTX Pro 6000 trades some of that bandwidth for a large, cost-effective single-card budget, and it is at its best when the model fits on one card.

What fits on 96 GB

Model size	Fit on one 96 GB card	Notes
7B to 13B	Comfortable	Room for large batches and long context
20B to 35B	The sweet spot	For example `qwen3-coder-30b-a3b-instruct`, `gpt-oss-20b`
Around 70B	Case by case	Usually needs quantization; validate first
235B and above	Not the target	Use an H200 or a multi-GPU or partner path

Model IDs are current EcoHash catalog IDs. See the model catalog for the full list.

Measured numbers

Numbers below are measured on a single RTX Pro 6000, end-to-end through the API, in July 2026. Full data, method, and charts are in ecohash-benchmarks.

Text generation, single stream:

Model	TTFT (p95)	Per-token	Single-stream tok/s
`llama-3.1-8b-instruct`	205 ms	7 ms	~143
`gpt-oss-20b`	170 ms	8 ms	~125
`qwen3-coder-30b-a3b-instruct`	30 ms	9 ms	~111

Under concurrency the card sustains about 11,500 output tok/s on llama-3.1-8b-instruct. With 8k-token prompts, prefill reaches roughly 170k to 220k tok/s, which is why long-context and RAG are the most cost-effective way to use it. Against other providers of the same model, its time to first token is the lowest in the field:

Image generation at 1024×1024:

Model	Steps	Latency	Images/min
`flux2-klein`	4	1.2 s	52
`z-image-turbo`	8	3.2 s	20
`qwen-image`	50	13.4 s	4.6

Three ways to use it

Shared API. Call open models through the OpenAI-compatible API at https://api.ecohash.com/v1, with no GPU to manage. Good for building and for traffic that comes in bursts. See inference.
Dedicated endpoint. Reserve capacity for one model when production traffic needs steadier throughput and predictable latency. See dedicated inference.
GPU workspace. Rent the card by the hour for experiments, fine-tuning, and batch jobs. See GPU compute.

You can start on the API and move to reserved capacity later without touching the model or the code.

Calling a model

The API is OpenAI-compatible, so existing OpenAI SDK code works once you point the base URL and key at EcoHash.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ecohash.com/v1",
    api_key="eco_...",  # create a key at console.ecohash.com
)

resp = client.chat.completions.create(
    model="qwen3-coder-30b-a3b-instruct",
    messages=[
        {"role": "user", "content": "Write a Python function that reverses a linked list."}
    ],
)
print(resp.choices[0].message.content)

curl https://api.ecohash.com/v1/chat/completions \
  -H "Authorization: Bearer $ECOHASH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-coder-30b-a3b-instruct",
    "messages": [{"role": "user", "content": "Write a Python function that reverses a linked list."}]
  }'

Validating the fit

Before you commit a model to this card:

Check that the model plus KV cache fits in 96 GB at your context length and batch size.
Pick a quantization format and compare its output against full precision on your own prompts.
Choose a serving stack such as vLLM or SGLang and pin the version.
Measure latency and throughput on your own traffic, and record the model revision, quantization, context length, framework, and driver so the numbers reproduce.

Where it falls short

96 GB is a single-card budget. Models above roughly 70B usually need quantization or more than one GPU.
Models at 235B and above are not the target. Use an H200 or a multi-GPU or partner path.
We run the card as PCIe capacity, not an NVLink cluster, so workloads that lean on wide tensor parallelism see less benefit.
Real numbers depend on the model, quantization, batch size, and context length. Treat any performance figure as something to reproduce with its method and date.
Small and voice models do not need this card. They run through the API and move onto GPU capacity only when throughput calls for it.

FAQ

Is the RTX Pro 6000 good for AI inference? Yes, for open models in the 20B to 35B range and for smaller models at higher batch sizes. Its 96 GB lets one card serve a mid-sized model with room for context and concurrency.

How much VRAM does the RTX Pro 6000 have? 96 GB of GDDR7, on the Blackwell Server Edition that EcoHash deploys.

How does it differ from an H100 for inference? The H100 has 80 GB of HBM with higher bandwidth and NVLink, which suits large models and multi-GPU serving. The RTX Pro 6000 has 96 GB of GDDR7 over PCIe, which suits single-GPU serving of 20B to 35B models at a lower hourly cost.

What models fit on a single RTX Pro 6000 96GB? Open models up to about 35B, including qwen3-coder-30b-a3b-instruct and gpt-oss-20b. Models near 70B usually need quantization.

Can it run a 70B model? Sometimes, with quantization and reduced context or batch size. Validate the specific model and settings before relying on it.

Does it support FP8 and other low-precision formats? Blackwell supports the low-precision formats used in inference, including FP8. Confirm what your serving stack uses and check output quality before depending on it.

How much does it cost on EcoHash? $1.89 per GPU-hour. See pricing for the current rate.

Do voice or small models need it? No. Models like Kokoro and Whisper run through the API, and move onto GPU capacity only when throughput demands it.

Next steps

Launch a GPU workspace: ecohash.com/gpu-compute
Browse open models: ecohash.com/models
Reserve steady capacity: ecohash.com/dedicated-inference