DEV Community: CloudGPU

A model as the judge: build preference pairs with an API, then train with DPO on a rented GPU

CloudGPU — Sun, 01 Nov 2026 03:10:36 +0000

Short answer: write a short list of principles and a set of questions, have an API model write two answers to each question, have it pick the answer that follows your principles better (twice, once in each order), keep only the pairs where both picks agree, and train a LoRA adapter on those chosen and rejected pairs with DPO in LLaMA-Factory on a rented GPU. Generate the pairs on your own machine; rent the GPU only for training; copy the adapter off, then destroy the instance.

Disclosure: I work on CloudGPU. The endpoint (https://cloudgpu.app/v1) and the GPU rentals below are ours. Each step ends with a "What to check" line, because details (versions, template names, model access) change.

Where the idea comes from, and what this guide does not claim

Using a model's judgments as the training signal is usually called RL from AI Feedback (RLAIF). One public paper on it, "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback" (Lee et al., arXiv, submitted 2023-09-01: paper), describes training a reward model on preferences generated by an off-the-shelf LLM and then using reinforcement learning. This guide keeps only the "a model states a preference between two answers" idea and skips the rest: it trains directly on the pairs with Direct Preference Optimization (paper), using LLaMA-Factory.

That makes this a small, single-developer adaptation. It is our own construction, not a result from those sources. A model's pick can be wrong, and we make no claim that the adapter you train will be better than the base model. That is why the last step is a comparison you read yourself, with a rule for when to throw the round away.

Read these things first

DeepSeek's terms. The answers and the picks in this guide come from DeepSeek, and this guide covers DeepSeek only. DeepSeek's Terms of Use (Last Update: March 27, 2026) say in section 4.2(3) that you may apply the Outputs of the Services to "a wide range of use cases, including personal use, academic research, derivative product development, training other models (such as model distillation), etc., as long as such usage is legal and adhere to these Terms." Read the terms yourself. This guide does not cover any other provider, and it does not tell you what any other provider's terms allow; if you want to use another model's outputs for training, read that provider's own terms first.
Labelling. Section 3.1 of the same terms says that if you publish or disseminate outputs, you must clearly indicate that the output content is generated by artificial intelligence. The chosen and rejected answers are outputs. If you publish the pairs, a model card or examples from them, label them as AI-generated; the script below writes a DATASET_NOTE.md you can keep next to the file.
Your data. The questions you write must be ones you have the right to use. CloudGPU's acceptable use policy (section 1.8) prohibits training on datasets you do not have rights to use. Also read the terms.
The base model's license. Each base model below has its own license. Open the link, read it, and confirm for yourself that your use is allowed.

What you need

Python 3 and the openai package on your own machine: pip install openai.
A CloudGPU account with balance, and an API key (the API page has a "Get your API key" link). Keep it in an environment variable, never in code:

export CLOUDGPU_API_KEY="paste-your-key-here"

GPU instances are billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (terms). Rates are on the pricing page. Payment is by card or USDT (payments).
A Hugging Face account and a read token if you choose Llama or Gemma (both are gated, see the table below).

What to check: python3 -c "import os; print(bool(os.environ.get('CLOUDGPU_API_KEY')))" prints True in the shell where you will run the scripts, and python3 -c "import openai" raises no error.

API calls are billed per token from the same balance as GPU rentals; the current rate for every model is in the table on the API page. This post does not repeat rates.

Step 1: write the principles and the questions (on your own machine)

Create three plain-text files in one folder.

principles.txt is a short list, in your own words, of what a better answer looks like for your use case. An example to replace with yours:

1. Answer only what the question asks.
2. If the answer depends on information you were not given, say what is missing instead of guessing.
3. Do not invent names, numbers, links or quotes.
4. Keep the answer under 150 words unless the question asks for more.

prompts.txt is the training questions, one per line. Start with ten lines to test the script, then add more; the more varied the questions are, the more the judge has to compare. heldout.txt is 20 or so questions in the same style that you will not use for training. Write them separately, and do not copy lines between the files.

What to check: wc -l prompts.txt heldout.txt shows the line counts you expect, and no line of heldout.txt appears in prompts.txt (grep -x -F -f heldout.txt prompts.txt prints nothing).

Step 2: write two answers, let the model pick, keep consistent picks

For each question the script asks deepseek-v4.1-flash for two answers at a high temperature, then asks it twice which answer follows your principles better: once with the answers in one order and once in the other. A pair is kept only if both picks choose the same answer. Judges can lean towards whichever answer comes first, or towards the longer one; the two orders and the length count the script prints are our own suggestions for spotting that, not claims from a source. The model ID comes from the live model table on the API page (checked 2026-10-27); the same table lists deepseek-v4-flash and deepseek-v4-pro. The OpenAI SDK only needs base_url and the key changed.

# make_pairs.py
import json, os, sys, time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from openai import OpenAI

client = OpenAI(base_url="https://cloudgpu.app/v1", api_key=os.environ["CLOUDGPU_API_KEY"])
MODEL = "deepseek-v4.1-flash"
PRINCIPLES = Path("principles.txt").read_text(encoding="utf-8").strip()

JUDGE = """You compare two answers to the same question.
Judge them ONLY against these principles:
{principles}

Question:
{question}

Answer A:
{a}

Answer B:
{b}

Reply with exactly one letter: A if Answer A follows the principles better,
B if Answer B follows them better."""


def chat(prompt, temperature):
    for attempt in range(3):
        try:
            resp = client.chat.completions.create(
                model=MODEL,
                temperature=temperature,
                messages=[{"role": "user", "content": prompt}],
            )
            return resp.choices[0].message.content.strip()
        except Exception as err:  # network error, rate limit
            print(f"attempt {attempt + 1} failed: {err}", file=sys.stderr)
            time.sleep(2 ** attempt)
    return None


def verdict(question, first, second):
    text = chat(JUDGE.format(principles=PRINCIPLES, question=question, a=first, b=second), 0)
    letter = (text or "").strip().strip(".*").upper()
    return letter if letter in ("A", "B") else None


def make_pair(question):
    one = chat(question, 1.0)
    two = chat(question, 1.0)
    if not one or not two or one == two:
        return None
    v1 = verdict(question, one, two)  # `one` is shown first
    v2 = verdict(question, two, one)  # `two` is shown first
    if v1 is None or v2 is None:
        return None
    winner1 = "one" if v1 == "A" else "two"
    winner2 = "two" if v2 == "A" else "one"
    if winner1 != winner2:  # the judge changed its mind when the order changed
        return None
    chosen, rejected = (one, two) if winner1 == "one" else (two, one)
    return {"instruction": question, "input": "", "chosen": chosen, "rejected": rejected}


if __name__ == "__main__":
    questions = [q.strip() for q in Path("prompts.txt").read_text(encoding="utf-8").splitlines() if q.strip()]
    with ThreadPoolExecutor(max_workers=4) as pool:
        results = list(pool.map(make_pair, questions))
    pairs = [p for p in results if p and max(len(p["chosen"]), len(p["rejected"])) <= 3000]
    Path("dpo_pairs.json").write_text(json.dumps(pairs, ensure_ascii=False, indent=2), encoding="utf-8")
    Path("DATASET_NOTE.md").write_text(
        "AI-generated data: the answers in dpo_pairs.json were generated, and the preferences between "
        "them chosen, with deepseek-v4.1-flash from questions written by the dataset author. "
        "Review before use.\n", encoding="utf-8")
    longer = sum(len(p["chosen"]) > len(p["rejected"]) for p in pairs)
    print("questions:", len(questions), "pairs kept:", len(pairs), "chosen is the longer answer:", longer)

Run it first on the ten-line prompts.txt: python3 make_pairs.py. Each question takes four API calls (two answers, two picks), so a long prompts.txt makes many calls; lower max_workers if you see rate-limit errors.

What to check: dpo_pairs.json and DATASET_NOTE.md exist, and the script prints how many pairs it kept and how many times the chosen answer was the longer one. If the chosen answer is the longer one in nearly every pair, the judge may be choosing length, not your principles: tighten the principles, or drop pairs where the lengths differ a lot. If it keeps very few pairs, the two orders disagree often; read why in a few cases before you change anything. A 401 means the key variable is not set in this shell.

Step 3: read the pairs yourself

The format is LLaMA-Factory's preference format: per its data preparation docs, a preference dataset needs a better response in the chosen column and a worse one in rejected. A row looks like this (illustrative, written by hand, not model output):

{
  "instruction": "Does the config file need a restart to take effect?",
  "input": "",
  "chosen": "The documentation you gave me does not say. Check the section on reloading before relying on it.",
  "rejected": "Yes, always restart the service after editing the config file."
}

Read a sample:

python3 -c "import json,random; [print(r['instruction'], '\nCHOSEN:', r['chosen'], '\nREJECTED:', r['rejected'], '\n') for r in random.sample(json.load(open('dpo_pairs.json')), 5)]"

What to check: for each sample, ask whether you agree that chosen follows your principles better than rejected. If you disagree with several, the judge prompt or the principles are the problem, not the training: fix them and run Step 2 again. Once you are happy with the ten-line run, put all your questions in prompts.txt and run it again.

Step 4: rent a GPU

On Deploy, choose a card. The CloudGPU docs say two things that matter here: every template is an image with root SSH, and for anything a template does not cover you can rent a raw GPU (Deploy â†’ Raw GPU) and install what you need over SSH. For this guide, pick a template that includes PyTorch, or choose Raw GPU and install it yourself. Pick a card with enough memory for your base model: the LLaMA-Factory README has a rough memory table (marked as estimates), and nvidia-smi shows real use during the first training steps.

Connect with the SSH details of your running instance. Then upload the pairs from your own machine (scp takes the port with a capital -P):

scp -P <ssh-port> dpo_pairs.json <ssh-user>@<ssh-host>:/workspace/

What to check: connect with ssh -p <ssh-port> <ssh-user>@<ssh-host>, then run nvidia-smi (a GPU is listed) and python3 -c "import torch; print(torch.cuda.is_available())" (prints True). If torch is missing, install PyTorch with CUDA from the PyTorch install page first.

Step 5: install LLaMA-Factory and register the preference dataset

These are the install commands from the LLaMA-Factory README (read 2026-10-27; the repository name is spelled LlamaFactory there). Don't count on a pre-installed copy being current: check, and install if needed.

cd /workspace
git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

Copy the pairs into data/ and add an entry to data/dataset_info.json. The "ranking": true line is what tells LLaMA-Factory this is a preference dataset:

cp /workspace/dpo_pairs.json data/
python3 - <<'EOF'
import json
p = "data/dataset_info.json"
info = json.load(open(p, encoding="utf-8"))
info["judge_pairs"] = {
    "file_name": "dpo_pairs.json",
    "ranking": True,
    "columns": {"prompt": "instruction", "query": "input", "chosen": "chosen", "rejected": "rejected"},
}
json.dump(info, open(p, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
EOF

What to check: llamafactory-cli help prints the command list, and grep -A 3 judge_pairs data/dataset_info.json shows your entry with "ranking": true.

Step 6: choose a base model and train with DPO

Small models are enough to see the pipeline work. Pick one row; the template name must match the model.

Base model	LLaMA-Factory `template`	License (read it yourself)	Access
Llama 3.2 3B Instruct (`meta-llama/Llama-3.2-3B-Instruct`)	`llama3`	Llama 3.2 Community License	Gated on Hugging Face: accept the license, use a token
Gemma 3 4B IT (`google/gemma-3-4b-it`)	`gemma3`	Gemma Terms of Use	Gated on Hugging Face: accept the terms, use a token
Mistral 7B Instruct v0.3 (`mistralai/Mistral-7B-Instruct-v0.3`)	`mistral`	License shown on the model card	The model card listed no gate when we looked (2026-10-27)

Template names are from the LLaMA-Factory README's model table; license and access information was read on 2026-10-27 and can change. The Llama and Gemma licenses have conditions that can apply if you distribute or make available a model built on them. Confirm the license for your use yourself before you train or share anything.

For a gated model, request access on its Hugging Face page, then give the instance a fine-grained read token for just this run (you will revoke it at the end):

export HF_TOKEN="paste-a-read-token-here"

Put this in a file named judge_dpo.yaml in the LlamaFactory folder. The keys follow the official DPO LoRA example in the repo's examples/train_lora/ folder; the model, template and dataset lines are the parts you change:

### model
model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3

### method
stage: dpo
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all
pref_beta: 0.1
pref_loss: sigmoid

### dataset
dataset: judge_pairs
template: mistral
cutoff_len: 2048
preprocessing_num_workers: 4

### output
output_dir: saves/mistral-7b/lora/judge-dpo
logging_steps: 10
save_steps: 100
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 5.0e-6
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true

Train:

llamafactory-cli train judge_dpo.yaml

Open the LLaMA-Factory examples folder and compare your YAML with the current DPO example: key names and good values change between releases. pref_loss: sigmoid is the plain DPO loss in that example.

What to check: the log prints a training loss every 10 steps; saves/mistral-7b/lora/judge-dpo contains adapter_model.safetensors and adapter_config.json. If the base model download fails with a 401 or 403, the Hugging Face access request or token is the problem. An out-of-memory error means a smaller cutoff_len, a smaller model or a card with more memory; nvidia-smi shows the real use.

Step 7: compare the base model and the adapter on questions it never saw

A quick test in the terminal (no server is started). Create two small files in the LlamaFactory folder. infer_base.yaml:

model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3
template: mistral
infer_backend: huggingface

and infer_dpo.yaml:

model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3
adapter_name_or_path: saves/mistral-7b/lora/judge-dpo
template: mistral
infer_backend: huggingface

Ask each one the questions in heldout.txt (copy that file to the instance first, or paste the questions):

llamafactory-cli chat infer_base.yaml
llamafactory-cli chat infer_dpo.yaml

Read both sets of answers against your own principles.txt. A lower training loss is not the test; your reading of questions the adapter never trained on is.

What to check: for each held-out question, write down which answer follows your principles better, or "can't tell". Stop rule: if you cannot say that the adapter's answers follow your principles better more often than the base model's, throw this round away. Change the principles, the questions or the judge prompt and start again from Step 2; do not publish the adapter or write that it improved anything.

Step 8: copy the adapter off, then destroy the instance

Before you destroy anything, copy out what you want to keep: destroying the instance permanently deletes its local storage. The docs have a section called "What to save before you destroy an instance"; read it in the CloudGPU docs. From your own machine:

rsync -avP -e "ssh -p <ssh-port>" \
  <ssh-user>@<ssh-host>:/workspace/LlamaFactory/saves/mistral-7b/lora/judge-dpo/ ./adapter-judge-dpo/

Check that adapter_model.safetensors is in ./adapter-judge-dpo/, revoke the Hugging Face token you pasted on the instance (if you used one), then destroy the instance from your Dashboard (the docs: "You can destroy an instance at any time"). The minutes you spent copying are billed like any other minute, and billing stops once the instance is destroyed.

What to check: the instance no longer appears as running in your Dashboard, and ./adapter-judge-dpo/ still opens on your machine.

Privacy and access, in plain terms

Step 2 sends your questions, the principles, and the candidate answers to a model API. Don't send text you would not want processed by a model service, and read the privacy policy for what the service handles.
You keep ownership of what you put on the GPU instance. CloudGPU's terms (section 5) say it does not access the contents of your instance except where needed to investigate a credible report of a policy violation, where compelled by valid legal process, or to protect the safety of users or third parties. Local instance storage is deleted when you destroy the instance.
This guide does not start a server. If you later serve your model from an instance, use your own endpoint with your own API key: the terms (section 6) say instance service URLs are not authenticated by default, so add authentication yourself before you share the URL.

Sources

All read on 2026-10-27 (UTC+8), except the RLAIF paper, read on 2026-11-01 (UTC+8).

Lee et al., "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback" (arXiv 2309.00267, submitted 2023-09-01, read 2026-11-01 UTC+8): paper. Direct Preference Optimization: paper.
DeepSeek, Terms of Use, Last Update March 27, 2026: cdn.deepseek.com, sections 3.1 and 4.2.
LLaMA-Factory: README and data preparation docs.
CloudGPU: API page, terms (sections 4 to 6), acceptable use policy (section 1.8, last updated April 22, 2026), privacy policy, docs.
Licenses: Llama 3.2, Gemma, Mistral 7B Instruct v0.3 model card.

Run your own model endpoint on a rented GPU with vLLM: your own API key, an SSH tunnel, and an optional LoRA adapter

CloudGPU — Sat, 31 Oct 2026 05:31:59 +0000

Short answer: rent a GPU with SSH access, install vLLM in a fresh Python environment, start it so it listens only on the instance's own localhost, require an API key of your own, and reach it from your machine through an SSH tunnel. If you have a LoRA adapter for the same base model, vLLM can serve it next to the base model under its own name. When you are done, copy out what you need and destroy the instance.

Disclosure: I work on CloudGPU, and the GPU rental in this guide is ours. Each step ends with a "What to check" line, because versions, flags and defaults change.

Which kind of endpoint is this?

CloudGPU's terms (section 6) say that each running instance with a service URL gets a public HTTPS address; these URLs are "hard to guess but they are not authenticated by default", and if you need authentication you configure it inside your instance, for example an API key on your Ollama deployment. The docs say that templates that serve an API, such as Ollama and ComfyUI, get such a URL.

This guide takes a different route on purpose. You rent a raw GPU, run vLLM on it, and do not publish any URL: the server listens only on the instance's own localhost, requires your own API key, and you reach it through an SSH tunnel from your machine. That gives you an endpoint only you can use, as long as you keep the SSH access and the key to yourself.

vLLM's own security guide is clear about the limits of the key: --api-key protects only the endpoints under the /v1, /v2, /inference and /cohere prefixes, other endpoints on the same server are not covered, and it says not to rely on --api-key alone. It also notes that some underlying components may listen on all network interfaces. So treat the key as one layer, keep the tunnel as the way in, and do not send anything to the instance that you would not put on a rented machine.

Read these things first

The license of your base model. An adapter sits on top of a base model, so the base model's license applies to what you serve. The walkthrough uses Mistral 7B Instruct v0.3, whose model card lists the Apache-2.0 license (read 2026-10-27). If you use a small Llama or Gemma model instead, read the Llama 3.2 Community License or the Gemma Terms of Use first, including what they say about derivative models. This guide gives no legal conclusions.
What your adapter was trained on. Use only data you have the right to use; CloudGPU's acceptable use policy (section 1.8) prohibits training on datasets you do not have rights to use. If your adapter was trained on text generated through DeepSeek's API and you publish its outputs, DeepSeek's Terms of Use (Last Update: March 27, 2026) require you to label published outputs as AI-generated (section 3.1).
CloudGPU's terms. Read the terms before you rent. Section 1 says compute comes from third-party data centres and supplier networks, which can change, so this guide does not claim your data stays on any particular machine.

What you need

A CloudGPU account with balance. GPU instances are billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (terms). Rates are on the pricing page. Payment is by card or USDT (payments).
A terminal with ssh, scp and Python 3 on your own machine.
Optional: a LoRA adapter trained on Mistral 7B Instruct v0.3, stored in the usual Hugging Face PEFT layout (a folder with adapter_config.json and adapter_model.safetensors), for example the one from the self-critique guide in this series. Without an adapter, do everything below and skip the adapter parts; you will serve the base model only.

What to check: ssh -V prints a version, and python3 --version prints a version. If you have an adapter, its folder contains the two files above.

Step 1: make your endpoint key and look at your adapter (on your own machine)

The key is a long random string that only you and the server know. Generate one:

python3 -c "import secrets; print(secrets.token_urlsafe(32))"

Keep it in a password manager. Make a new one for every instance; never reuse a key from another service.

If you have an adapter, open adapter_config.json in its folder and note two fields: base_model_name_or_path and r (the LoRA rank).

What to check: the command printed a long random string that you have stored somewhere other than a shell history you share. For an adapter, base_model_name_or_path is mistralai/Mistral-7B-Instruct-v0.3 (if not, the adapter belongs to another base model and this guide's command will not match it), and r is a number you can compare with the rank limit in Step 4.

Step 2: rent a raw GPU

On Deploy, choose Raw GPU and a card. The CloudGPU docs say that every template is a ready-made image with root SSH and that for anything a template does not cover you rent a raw GPU (Deploy â†’ Raw GPU) and install what you need over SSH. The docs' GPU table lists 16 GB as the minimum VRAM for models from 0.5B to 8B parameters; vLLM also needs Linux and an NVIDIA GPU with compute capability 7.5 or higher, according to its installation page. A 7B model in 16-bit needs memory for the weights plus a cache for requests, so a card with more than the minimum is the easy fix if the server reports it cannot fit.

Connect with the SSH details of your running instance. If you have an adapter, upload its folder from your own machine (scp takes the port with a capital -P):

scp -r -P <ssh-port> ./adapter-selfloop <ssh-user>@<ssh-host>:/workspace/

What to check: ssh -p <ssh-port> <ssh-user>@<ssh-host> connects, nvidia-smi lists a GPU, and (with an adapter) ls /workspace/adapter-selfloop shows adapter_config.json and adapter_model.safetensors.

Step 3: install vLLM in a fresh environment

The vLLM installation page recommends uv for the Python environment. uv can be installed from PyPI (see the uv installation page). On the instance:

pip install uv
cd /workspace
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

These lines follow the vLLM page as read on 2026-10-27; if the page now shows different commands, use its commands.

What to check: with the environment active, vllm --version prints a version, and python3 -c "import torch; print(torch.cuda.is_available())" prints True. If the install stops with an error, read the error text and the troubleshooting parts of the vLLM page.

Step 4: start the server on localhost, with your key

Use your own key in place of <your-endpoint-key>. With an adapter:

cd /workspace && source .venv/bin/activate
nohup vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
  --host 127.0.0.1 --port 8000 \
  --api-key <your-endpoint-key> \
  --tokenizer-mode hf \
  --enable-lora --max-lora-rank 16 \
  --lora-modules selfloop=/workspace/adapter-selfloop \
  > /workspace/vllm.log 2>&1 &

Without an adapter, remove the --enable-lora, --max-lora-rank and --lora-modules lines.

What the flags do, from the vLLM serve reference and LoRA page: --host 127.0.0.1 keeps the HTTP server on the instance's localhost; --api-key makes requests under /v1 carry that key; --tokenizer-mode hf uses the Hugging Face tokenizer instead of letting auto pick the mistral_common tokenizer for Mistral models; pinning it is meant to match how an adapter was trained with the Hugging Face tokenizer, so drop the flag if your adapter was trained another way; --lora-modules name=path registers the adapter under the name selfloop; --max-lora-rank must be at least the adapter's r and the docs list the allowed values (the default is 16), so if your r from Step 1 is larger than 16, use the next allowed value. The key is visible in the instance's process list while the server runs, which is one more reason to use a fresh key per instance.

Wait until the log says the server is ready before you send a request. If the process exits, read the end of the log with tail -n 50 /workspace/vllm.log. A message about memory for the cache is fixed by adding --max-model-len 8192 or by choosing a card with more memory.

What to check: on the instance, curl -s -i http://127.0.0.1:8000/v1/models | head -n 1 prints an HTTP status line that is not a success (an authentication error is expected, because you sent no key), and curl -s -H "Authorization: Bearer <your-endpoint-key>" http://127.0.0.1:8000/v1/models prints JSON that lists the base model and, with an adapter, selfloop.

Step 5: open the tunnel and call your endpoint (on your own machine)

In one terminal, open the tunnel. Nothing is printed while it runs; leave it open:

ssh -N -L 8000:127.0.0.1:8000 -p <ssh-port> <ssh-user>@<ssh-host>

If local port 8000 is already used, pick another local port (for example -L 8001:127.0.0.1:8000) and use it in the URLs below. In a second terminal:

curl http://127.0.0.1:8000/v1/models -H "Authorization: Bearer <your-endpoint-key>"

Now the same request through the OpenAI SDK (pip install openai on your own machine). Put this in a file named ask.py:

import os
import sys

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key=os.environ["ENDPOINT_KEY"])

BASE = "mistralai/Mistral-7B-Instruct-v0.3"
ADAPTER = "selfloop"  # delete this name from the loop below if you started without an adapter

question = sys.argv[1] if len(sys.argv) > 1 else "Explain what a LoRA adapter is in two sentences."

for name in (BASE, ADAPTER):
    reply = client.chat.completions.create(
        model=name,
        messages=[{"role": "user", "content": question}],
        temperature=0,
        max_tokens=300,
    )
    print("==", name)
    print(reply.choices[0].message.content)
    print()

Run it with the key in an environment variable, not in the file:

export ENDPOINT_KEY="<your-endpoint-key>"
python3 ask.py "Explain what a LoRA adapter is in two sentences."

Any tool that lets you set an OpenAI-compatible base URL and a key can use the same two values (http://127.0.0.1:8000/v1 and your key) while the tunnel is open.

What to check: the curl call lists the model names you expect; ask.py prints one answer per model name; and a call with a wrong or missing key is refused rather than answered. If the tunnel command prints "address already in use", use another local port. Reading two answers is not a test of your adapter; for that, use questions the adapter never saw and compare them yourself, as in the self-critique guide.

Step 6: stop, copy out what you need, destroy

Close the tunnel with Ctrl+C in its terminal. Before you destroy anything, copy what you want to keep. The docs have a section called "What to save before you destroy an instance"; read it in the CloudGPU docs. For example, copy the server log from your own machine:

scp -P <ssh-port> <ssh-user>@<ssh-host>:/workspace/vllm.log ./

Then destroy the instance from your Dashboard (the docs: "You can destroy an instance at any time"). Minutes spent copying are billed like any other minute, and billing stops once the instance is destroyed. The endpoint key was meant for this instance only; do not reuse it.

What to check: the instance no longer appears as running in your Dashboard, and the curl call from Step 5 now fails to connect.

Privacy and access, in plain terms

This is your own endpoint, with your own API key, reached through your own SSH tunnel. It is not a public URL.
Per CloudGPU's terms (section 5), you keep ownership of any data, models, code or output you create on or upload to an instance, and CloudGPU says it does not access, inspect, copy or back up the contents of your instance except in three listed cases: a credible report of a violation of section 3, valid legal process, or to protect the safety of users or third parties.
Per the same terms, destroying an instance permanently deletes its local storage, and you are responsible for backing up what you want to keep.
Whatever you send to the endpoint is processed on that rented instance. Do not send secrets or personal data you would not put on a rented machine.
If you call a model through CloudGPU's AI API instead of your own instance, the request is forwarded to the model provider and its terms apply; see the privacy policy.

Sources

All read on 2026-10-27 (UTC+8).

vLLM: installation (NVIDIA CUDA), serving LoRA adapters, serve command reference, security guide.
uv: installation.
Licenses: Mistral 7B Instruct v0.3 model card, Llama 3.2 Community License, Gemma Terms of Use.
DeepSeek: Terms of Use (Last Update: March 27, 2026).
CloudGPU: terms, acceptable use policy, privacy policy, docs, pricing.

Let a small model critique and revise its own answers, then fine-tune on the revisions (on a rented GPU)

CloudGPU — Fri, 30 Oct 2026 03:10:23 +0000

Short answer: write a short list of principles and a set of questions, have a small model answer them, critique its own answers against your principles and revise them, filter the revisions with a script and read a sample yourself, train a LoRA adapter on the result with LLaMA-Factory, then compare the base model and the adapter on questions you held back. Everything runs on one rented GPU; nothing is sent to a third-party API, and the guide starts no server.

Disclosure: I work on CloudGPU, and the GPU rental in this guide is ours. Each step links the documentation it follows and ends with a "What to check" line, because versions, template names and model access change.

Where the idea comes from, and what this guide does not claim

Anthropic's Constitutional AI work (2022-12-15) describes a supervised phase in which the model samples answers, critiques them against a list of principles, revises them, and is then fine-tuned on the revisions: research page, paper. OpenAI's deliberative alignment post (2024-12-20) says its training data is generated automatically from written specifications, without human-labeled completions: post.

This guide is a small, single-developer version of that pattern. It is our own adaptation, not a result from those sources, and we make no claim that the adapter you train will be better than the base model. A model's critique of its own answer can be wrong, and a revision can be worse than the original. That is why the last step is a comparison you read yourself, with a rule for when to throw the round away.

Read these things first

Your base model's license. You will train on text a base model wrote, so the license of that model matters. The walkthrough uses Mistral 7B Instruct v0.3, whose model card lists the Apache-2.0 license (read 2026-10-27). Two other small models have conditions that touch this exact workflow. The Llama 3.2 Community License says that if you use the Llama Materials "or any outputs or results of the Llama Materials to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available", the model name must begin with "Llama". The Gemma Terms of Use define Model Derivatives to include models created by "methods based on the generation of synthetic data Outputs by Gemma for training that model". Read the current text of the license of whichever model you use; this is an excerpt, not legal advice.
Your data. The questions you write must be ones you have the right to use. CloudGPU's acceptable use policy (section 1.8) prohibits training on datasets you do not have rights to use. Also read the terms.
Labelling. This guide generates data with your own model, not with an API. If you would rather use an API as the generator, the guide on generating a training set with DeepSeek, from this same series, uses DeepSeek only. DeepSeek's Terms of Use (Last Update: March 27, 2026) say in section 3.1 that if you publish or disseminate outputs, you must clearly indicate that the content is generated by artificial intelligence. If you publish any dataset or example from this guide, label it AI-generated as well.

What you need

A CloudGPU account with balance. GPU instances are billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (terms). Rates are on the pricing page. Payment is by card or USDT (payments).
A text editor and a terminal on your own machine. You write three small files there before you rent anything.

What to check: you can open a terminal and run ssh -V and scp (both print a usage line or version).

Step 1: write the principles and the questions (on your own machine)

Create three plain-text files in one folder.

principles.txt is a short list, in your own words, of what a good answer looks like for your use case. An example to replace with yours:

1. Answer only what the question asks.
2. If the answer depends on information you were not given, say what is missing instead of guessing.
3. Do not invent names, numbers, links or quotes.
4. Keep the answer under 150 words unless the question asks for more.

prompts.txt is the training questions, one per line, as many as you can write (a few hundred is a reasonable start; the more varied they are, the more the loop has to work with). heldout.txt is 20 or so questions in the same style that you will not use for training. Write them separately, and do not copy lines between the files.

What to check: wc -l prompts.txt heldout.txt shows both files non-empty, and comm -12 <(sort prompts.txt) <(sort heldout.txt) prints nothing (no question appears in both).

Step 2: rent a GPU

On Deploy, choose a card. The CloudGPU docs say two things that matter here: every template is an image with root SSH, and for anything a template does not cover you can rent a raw GPU (Deploy → Raw GPU) and install what you need over SSH. For this guide, pick a template that includes PyTorch, or choose Raw GPU and install it yourself. A 7B model in 16-bit needs a card with enough memory for the weights plus the generation cache; nvidia-smi shows real use, and a smaller model or a card with more memory is the fix for an out-of-memory error.

Connect with the SSH details of your running instance. Then upload your three files (scp takes the port with a capital -P):

scp -P <ssh-port> principles.txt prompts.txt heldout.txt <ssh-user>@<ssh-host>:/workspace/

What to check: connect with ssh -p <ssh-port> <ssh-user>@<ssh-host>, then ls /workspace lists your three files, nvidia-smi lists a GPU, and python3 -c "import torch; print(torch.cuda.is_available())" prints True. If torch is missing, install PyTorch with CUDA from the PyTorch install page first.

Step 3: install LLaMA-Factory

These are the install commands from the LLaMA-Factory README (read 2026-10-27; the repository name is spelled LlamaFactory there). The repository's pyproject.toml lists torch, transformers, accelerate and peft as dependencies, which is all the scripts below use. The docs FAQ lists a LLaMA-Factory template, but don't count on it being installed or current: check, and install if needed.

cd /workspace
git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

What to check: llamafactory-cli help prints the command list, and python3 -c "import transformers, peft; print(transformers.__version__, peft.__version__)" prints two versions without an error.

Step 4: generate, critique, revise

Two files. common.py loads the base model and answers a list of prompts in batches (the transformers docs say to pad on the left for batched generation, and the chat templating docs describe apply_chat_template):

# /workspace/common.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "mistralai/Mistral-7B-Instruct-v0.3"


def load():
    tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left")
    if tok.pad_token is None:
        tok.pad_token = tok.eos_token
    model = AutoModelForCausalLM.from_pretrained(
        MODEL, torch_dtype=torch.bfloat16, device_map="auto"
    )
    model.eval()
    return tok, model


def chat(tok, model, prompts, max_new_tokens=400, sample=True, batch=8):
    outs = []
    for i in range(0, len(prompts), batch):
        texts = [
            tok.apply_chat_template(
                [{"role": "user", "content": p}],
                tokenize=False,
                add_generation_prompt=True,
            )
            for p in prompts[i : i + batch]
        ]
        enc = tok(texts, return_tensors="pt", padding=True, add_special_tokens=False)
        enc = enc.to(model.device)
        kw = {"do_sample": True, "temperature": 0.7, "top_p": 0.9} if sample else {"do_sample": False}
        with torch.no_grad():
            gen = model.generate(
                **enc, max_new_tokens=max_new_tokens, pad_token_id=tok.pad_token_id, **kw
            )
        for row in gen[:, enc["input_ids"].shape[1]:]:
            outs.append(tok.decode(row, skip_special_tokens=True).strip())
    return outs

loop.py runs the three passes: answer, critique against your principles, revise. Principles go inside the user message so the same scripts work with chat templates that have no system role.

# /workspace/loop.py
import json
from common import load, chat

NONE = "NONE"
principles = open("/workspace/principles.txt", encoding="utf-8").read().strip()
prompts = [l.strip() for l in open("/workspace/prompts.txt", encoding="utf-8") if l.strip()]

tok, model = load()

answers = chat(tok, model, prompts)

critique_in = [
    "Principles:\n" + principles + "\n\nQuestion:\n" + p + "\n\nAnswer:\n" + a +
    "\n\nIn at most three short bullet points, say where the answer breaks a principle. "
    "If it breaks none, reply with exactly " + NONE + "."
    for p, a in zip(prompts, answers)
]
critiques = chat(tok, model, critique_in, max_new_tokens=200)

todo = [i for i, c in enumerate(critiques) if c.strip().strip(".").upper() != NONE]
revise_in = [
    "Principles:\n" + principles + "\n\nQuestion:\n" + prompts[i] + "\n\nAnswer:\n" + answers[i] +
    "\n\nProblems found:\n" + critiques[i] +
    "\n\nRewrite the answer so that it follows the principles and fixes these problems. "
    "Reply with the rewritten answer only."
    for i in todo
]
revised = chat(tok, model, revise_in)
revised_by_index = dict(zip(todo, revised))

with open("/workspace/loop_raw.jsonl", "w", encoding="utf-8") as f:
    for i in range(len(prompts)):
        rec = {"prompt": prompts[i], "answer": answers[i], "critique": critiques[i],
               "revised": revised_by_index.get(i)}
        f.write(json.dumps(rec, ensure_ascii=False) + "\n")
with open("/workspace/loop_revised.jsonl", "w", encoding="utf-8") as f:
    for i in todo:
        rec = {"prompt": prompts[i], "answer": answers[i], "critique": critiques[i],
               "revised": revised_by_index[i]}
        f.write(json.dumps(rec, ensure_ascii=False) + "\n")
print("prompts:", len(prompts), "revised rows:", len(todo))

Run it from /workspace:

cd /workspace
python3 loop.py

If your transformers version prints a warning that torch_dtype is deprecated, replace it with dtype in common.py.

What to check: the script ends by printing the number of prompts and revised rows. Open loop_revised.jsonl and read five rows yourself: does the critique point at a real problem, and is the revised answer actually better than the first one? If critiques are vague or revisions are not better, stop here and change your principles or your prompts before you go on.

Step 5: filter and read a sample

The loop writes everything the model said, including its mistakes. Filter before you train. This script drops rows where the revision is empty, identical to the first answer, very long, repeated, or contains words that show the model talked about the task instead of answering:

# /workspace/filter.py
import json
import random

LEAK = ("principles", "critique", "rewritten", "revised answer", "problems found")
rows, seen = [], set()
for line in open("/workspace/loop_revised.jsonl", encoding="utf-8"):
    r = json.loads(line)
    rev = (r["revised"] or "").strip()
    if not rev or rev == r["answer"].strip():
        continue
    if len(rev) > 2500 or len(rev) < 20:
        continue
    low = rev.lower()
    if any(w in low for w in LEAK):
        continue
    key = " ".join(low.split())
    if key in seen:
        continue
    seen.add(key)
    rows.append({"instruction": r["prompt"], "input": "", "output": rev})

json.dump(rows, open("/workspace/selfloop_sft.json", "w", encoding="utf-8"), ensure_ascii=False, indent=2)
random.seed(1)
with open("/workspace/sample_to_read.md", "w", encoding="utf-8") as f:
    for r in random.sample(rows, min(30, len(rows))):
        f.write("## " + r["instruction"] + "\n\n" + r["output"] + "\n\n")
print("kept:", len(rows))

python3 filter.py

Then read sample_to_read.md yourself. Edit selfloop_sft.json by hand and remove any row you would not want the model to learn from, or tighten the filter and run it again. If you publish the filtered file or examples from it, label them as AI-generated.

What to check: the script prints how many rows it kept. If that number is very small, the loop is not producing usable revisions for your prompts; go back to Step 1. If you read the sample and many rows are wrong or off-topic, do not train on them.

Step 6: register the data and train a LoRA adapter

Copy the dataset into LLaMA-Factory's data/ folder and register it in data/dataset_info.json, which is how LLaMA-Factory finds custom datasets:

cd /workspace/LlamaFactory
cp /workspace/selfloop_sft.json data/
python3 - <<'EOF'
import json
p = "data/dataset_info.json"
info = json.load(open(p, encoding="utf-8"))
info["selfloop_sft"] = {
    "file_name": "selfloop_sft.json",
    "columns": {"prompt": "instruction", "query": "input", "response": "output"},
}
json.dump(info, open(p, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
EOF

Save this as selfloop_lora.yaml in the LlamaFactory folder. The keys follow the official LoRA example in the repo's examples/train_lora/ folder; the model, template and dataset lines are the parts you change (the mistral template name is from the README's model table):

### model
model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all

### dataset
dataset: selfloop_sft
template: mistral
cutoff_len: 2048
preprocessing_num_workers: 4

### output
output_dir: saves/mistral7b/lora/selfloop
logging_steps: 10
save_steps: 100
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true

### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 50

llamafactory-cli train selfloop_lora.yaml

Compare your YAML with the current LoRA example in the LLaMA-Factory examples folder: key names change between releases.

What to check: the log prints a training loss every 10 steps and an evaluation loss every 50; saves/mistral7b/lora/selfloop contains adapter_model.safetensors and adapter_config.json. Make sure no earlier script is still holding the GPU (nvidia-smi shows what holds memory). If you choose another base model, change MODEL in common.py, model_name_or_path, and template together, and set a Hugging Face token for a gated model.

Step 7: compare on questions the model never saw

compare.py answers your held-out questions with the base model and with the adapter, greedily (no sampling, so the two columns are comparable), and writes both next to each other:

# /workspace/compare.py
from peft import PeftModel
from common import load, chat

ADAPTER = "/workspace/LlamaFactory/saves/mistral7b/lora/selfloop"
qs = [l.strip() for l in open("/workspace/heldout.txt", encoding="utf-8") if l.strip()]

tok, model = load()
base = chat(tok, model, qs, sample=False)
tuned_model = PeftModel.from_pretrained(model, ADAPTER)
tuned_model.eval()
tuned = chat(tok, tuned_model, qs, sample=False)

with open("/workspace/compare.md", "w", encoding="utf-8") as f:
    for q, b, t in zip(qs, base, tuned):
        f.write("## " + q + "\n\n**Base:**\n\n" + b + "\n\n**Adapter:**\n\n" + t + "\n\n")
print("questions:", len(qs))

cd /workspace
python3 compare.py

Read compare.md against your principles, question by question. Decide for yourself what "better" means, and write it down before you read. Stop rule: if the adapter is not clearly better on your principles, do not keep it. Change the principles, the prompts or the filter and run the loop again; each round uses more GPU minutes, so decide in advance when to stop.

What to check: the script prints the number of questions and compare.md exists with two answers per question. Remember that 20 questions is a small test, and that you wrote the principles that decide the outcome.

Step 8: copy the adapter off and destroy the instance

Before you destroy anything, copy what you want to keep. The docs have a section called "What to save before you destroy an instance"; read it in the CloudGPU docs. From your own machine:

rsync -avP -e "ssh -p <ssh-port>" \
  <ssh-user>@<ssh-host>:/workspace/LlamaFactory/saves/mistral7b/lora/selfloop/ ./adapter-selfloop/
scp -P <ssh-port> <ssh-user>@<ssh-host>:/workspace/compare.md ./

Check that adapter_model.safetensors is in ./adapter-selfloop/, then destroy the instance from your Dashboard (the docs: "You can destroy an instance at any time"). The minutes spent copying are billed like any other minute, and billing stops once the instance is destroyed.

What to check: the instance no longer appears as running in your Dashboard, and ./adapter-selfloop/ opens on your machine.

Privacy and access, in plain terms

The guide generates and trains on the instance itself and does not call an API, so your questions and the model's answers stay on that instance and on your machine. No server is started, so nothing listens for outside requests. If you later serve the model, put your own API key in front of it before you expose it.
Per CloudGPU's terms, destroying an instance deletes its local storage; copy what you need first.
If you used a Hugging Face token for a gated base model, revoke it when you are done.

Sources

All read on 2026-10-27 (UTC+8).

Anthropic, 2022-12-15: Constitutional AI: Harmlessness from AI Feedback and the paper.
OpenAI, 2024-12-20: Deliberative alignment and the paper.
Licenses: Mistral 7B Instruct v0.3 model card, Llama 3.2 Community License, Gemma Terms of Use.
DeepSeek: Terms of Use (Last Update: March 27, 2026).
Tools: LLaMA-Factory, transformers text generation tutorial, chat templating.
CloudGPU: terms, acceptable use policy, docs, pricing.

Run Ollama on a per-minute GPU, then point your app at an OpenAI-compatible API

CloudGPU — Thu, 29 Oct 2026 09:13:30 +0000

Your laptop can't hold the model you want, and you don't want to spend an afternoon on CUDA and port forwarding. This walkthrough rents a GPU billed per minute, runs Ollama on it with a public HTTPS endpoint, and calls that endpoint with curl and the OpenAI Python client.

Disclosure: I work on CloudGPU.

What you need

A CloudGPU account (email or Google sign-up) with a prepaid balance you can top up by card or USDT. Rates are listed on the pricing page.
Python with pip install openai.

Step 1: Deploy the Ollama template

Open Deploy, choose the Ollama template, select a GPU card with enough VRAM for the model you plan to run, and click Deploy. The Deploy page shows the cards you can pick. To size a model, see our Ollama GPU guide.

Step 2: Find the API endpoint

The docs say: "Every running Ollama instance gives you an OpenAI-compatible endpoint. Open the instance in your Dashboard and click the API button." They also say "Replace with the id shown on your instance row." The endpoint looks like this:

https://<random-id>-i.cloudgpu.app

Check the docs for the current wording. The URL works only while the instance runs. The terms say service URLs are "not authenticated by default" and that you "should treat them as a credential and not share them publicly", so keep it private. If you need authentication, configure it inside your instance.

Step 3: Pull a model

The docs describe a Pick Model button for pulling any model from the Ollama library after deploy. As an example, this guide uses llama3.1:8b (Llama 3.1 8B on ollama.com). Check it fits the VRAM of the card you picked. You can also run ollama pull llama3.1:8b yourself if you connect over SSH.

Step 4: Smoke-test it with curl

List the models on the instance:

curl https://<random-id>-i.cloudgpu.app/api/tags

Send one chat turn (non-streaming). keep_alive is the Ollama setting that controls how long a model stays loaded after the last call; the CloudGPU docs example uses "30m".

curl https://<random-id>-i.cloudgpu.app/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "stream": false,
    "keep_alive": "30m",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

If a model has been unloaded, Ollama has to load it again before it can answer.

Step 5: Call it with the OpenAI Python client

Ollama serves an OpenAI-compatible API under /v1 (see Ollama's OpenAI compatibility docs), so the openai package works with a different base_url:

from openai import OpenAI

client = OpenAI(
    base_url="https://<random-id>-i.cloudgpu.app/v1",
    api_key="ollama",  # any non-empty string, per the CloudGPU docs
)

r = client.chat.completions.create(
    model="llama3.1:8b",  # a name from /api/tags
    messages=[{"role": "user", "content": "Explain LoRA in one sentence."}],
)
print(r.choices[0].message.content)

Anything else that accepts an OpenAI-compatible base URL can use the same endpoint and model name.

Want ComfyUI instead?

The Deploy page also lists a ComfyUI template. The docs say templates that serve an API (Ollama and ComfyUI) get a public HTTPS URL, and that the ComfyUI template installs ComfyUI only: it does not include FLUX or other model weights, so you download the weights you need yourself once the instance is running. The same advice about keeping the URL private applies.

Destroy the instance

When you're done, destroy the instance. GPU time is billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance. The rules are in the terms. Destroying an instance deletes everything stored on it, so copy out anything you need first. The terms also say an instance is destroyed automatically if your balance reaches zero.

Troubleshooting

Model not found or wrong name. Use a name exactly as /api/tags returns it.
The endpoint does not answer. Check in your Dashboard that the instance is still running and that you copied the id from the instance row. A destroyed instance no longer serves its URL.
The first request after a model was unloaded fails or times out. Ollama may still be loading it into VRAM; retry.
No API button or endpoint on the instance. Send the instance ID through the contact page.

CloudGPU is in public beta with no SLA.

Prefer a hosted model to your own GPU? The OpenAI-compatible gateway is billed per token from the same prepaid balance and is documented in the API docs.

Generate a training set with DeepSeek, then fine-tune a small model on a rented GPU

CloudGPU — Thu, 29 Oct 2026 05:43:23 +0000

Short answer: write a few documentation pages yourself, have DeepSeek turn them into question-and-answer pairs through an OpenAI-compatible endpoint, filter the pairs with a short script, then train a LoRA adapter on a small base model with LLaMA-Factory on a rented GPU. Generate and filter on your own machine; rent the GPU only for training; download the adapter, then destroy the instance.

Disclosure: I work on CloudGPU. The endpoint (https://cloudgpu.app/v1) and the GPU rentals below are ours. Each step links the official documentation it follows and ends with a "What to check" line, because the details (versions, template names, model access) change.

Why this pattern? AI labs describe using models to write or judge training data from written rules. This is the small, single-developer version of that idea. It is our own analogy, not a claim from those papers.

Read these three things first

DeepSeek's terms. This tutorial generates data with DeepSeek only. DeepSeek's Terms of Use (Last Update: March 27, 2026) say in section 4.2(3) that you may apply the Outputs of the Services to "a wide range of use cases, including personal use, academic research, derivative product development, training other models (such as model distillation), etc., as long as such usage is legal and adhere to these Terms." Section 3.1 adds that if you publish or disseminate outputs, you must "clearly indicate that the output content is generated by artificial intelligence". So if you publish the dataset, a model card or examples from it, label them as AI-generated. Read the terms yourself, and read CloudGPU's terms and acceptable use policy, which apply to the gateway.
Rights to your source text. The acceptable use policy lists "Training models on datasets you do not have rights to use" under large-scale copyright infringement (section 1.8). In this tutorial the generator only sees documentation you wrote yourself.
The base model's license. Each base model below has its own license. Open the link, read it, and confirm for yourself that your use is allowed.

What you need

Python 3 and the openai package on your own machine: pip install openai.
A CloudGPU account with balance, and an API key (the API page has a "Get your API key" link). Keep it in an environment variable, never in code:

export CLOUDGPU_API_KEY="paste-your-key-here"

Three to ten Markdown files of your own documentation, with ## section headings, in a docs/ folder. A README of your own project works.
A Hugging Face account and a read token if you choose Llama or Gemma (both are gated, see the table below).

API calls are billed per token from the same balance as GPU rentals; the current rate for every model is in the table on the API page. This post does not repeat rates.

Step 1: generate question-and-answer pairs with DeepSeek

The script splits each Markdown file at ## headings, asks deepseek-v4.1-flash for a handful of pairs per section, and writes one JSON object per line. The model ID comes from the live model table on the API page (checked 2026-10-27); the same table lists deepseek-v4-flash and deepseek-v4-pro. The OpenAI SDK only needs base_url and the key changed.

# gen_data.py
import json, os, re, sys, time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from openai import OpenAI

client = OpenAI(base_url="https://cloudgpu.app/v1", api_key=os.environ["CLOUDGPU_API_KEY"])
MODEL = "deepseek-v4.1-flash"
PAIRS_PER_SECTION = 6

PROMPT = """You write training examples for a support assistant.
Use ONLY the documentation below. Do not invent features or settings.
Write {n} varied question-and-answer pairs that a user might ask about it.
Return ONLY a JSON array. Each item has the keys "instruction" (the question),
"input" (extra context, or an empty string) and "output" (the answer, 1-4 sentences).

Documentation:
{chunk}
"""

def sections(folder):
    for path in sorted(Path(folder).glob("*.md")):
        for part in re.split(r"\n(?=## )", path.read_text(encoding="utf-8")):
            if len(part.strip()) > 200:
                yield path.name, part.strip()

def parse(text):
    fence = "`" * 3  # the model may wrap its JSON in a code fence
    text = text.strip().removeprefix(fence + "json").removeprefix(fence).removesuffix(fence).strip()
    rows = json.loads(text)
    keys = ("instruction", "input", "output")
    return [r for r in rows if isinstance(r, dict) and all(isinstance(r.get(k), str) for k in keys)]

def ask(item):
    name, chunk = item
    for attempt in range(3):
        try:
            resp = client.chat.completions.create(
                model=MODEL,
                temperature=0.8,
                messages=[{"role": "user", "content": PROMPT.format(n=PAIRS_PER_SECTION, chunk=chunk)}],
            )
            return parse(resp.choices[0].message.content)
        except Exception as err:  # network error, rate limit, bad JSON
            print(f"{name}: attempt {attempt + 1} failed: {err}", file=sys.stderr)
            time.sleep(2 ** attempt)
    return []

if __name__ == "__main__":
    with ThreadPoolExecutor(max_workers=4) as pool:
        results = list(pool.map(ask, sections("docs")))
    with open("raw.jsonl", "w", encoding="utf-8") as f:
        for rows in results:
            for row in rows:
                f.write(json.dumps(row, ensure_ascii=False) + "\n")
    print("sections:", len(results), "pairs:", sum(len(r) for r in results))

Run python3 gen_data.py. The four-worker pool and the retry loop keep you inside the default rate limits listed on the API page; check those limits there and lower max_workers if you see rate-limit errors.

What to check: raw.jsonl exists and has lines; the script prints how many sections and pairs it got. If a section failed three times it is skipped and the error is on stderr. A 401 means the key variable is not set in this shell.

Step 2: filter and de-duplicate

Model-written data needs a filter before training. This script drops empty rows, answers that are very short or very long, boilerplate such as "as an AI", and near-duplicate questions (token overlap of 0.8 or more). Tune the numbers for your data.

# filter_data.py
import json, re
from pathlib import Path

def tokens(s):
    return set(re.sub(r"\W+", " ", s.lower()).split())

def jaccard(a, b):
    union = a | b
    return len(a & b) / len(union) if union else 0.0

kept, seen = [], []
dropped = {"empty": 0, "length": 0, "boilerplate": 0, "duplicate": 0}

for line in Path("raw.jsonl").read_text(encoding="utf-8").splitlines():
    row = json.loads(line)
    ins, inp, out = (row[k].strip() for k in ("instruction", "input", "output"))
    if not ins or not out:
        dropped["empty"] += 1
    elif not 20 <= len(out) <= 1200:
        dropped["length"] += 1
    elif re.search(r"\bas an ai\b|\blanguage model\b", out, re.I):
        dropped["boilerplate"] += 1
    elif any(jaccard(tokens(ins), t) >= 0.8 for t in seen):
        dropped["duplicate"] += 1
    else:
        seen.append(tokens(ins))
        kept.append({"instruction": ins, "input": inp, "output": out})

Path("synth_docs_qa.json").write_text(json.dumps(kept, ensure_ascii=False, indent=2), encoding="utf-8")
Path("DATASET_NOTE.md").write_text(
    "AI-generated data: the examples in synth_docs_qa.json were generated with deepseek-v4.1-flash "
    "from documentation written by the dataset author. Review before use.\n", encoding="utf-8")
print("kept", len(kept), "dropped", dropped)

The output is in LLaMA-Factory's Alpaca format: instruction, optional input, and output, per the data preparation docs. A row looks like this (illustrative, written by hand, not model output):

{
  "instruction": "How do I change the output folder?",
  "input": "",
  "output": "Pass --out followed by a path, for example --out ./build."
}

Then read a sample yourself. Filters catch format problems, not wrong answers:

python3 -c "import json,random; [print(r['instruction'],'->',r['output'],'\n') for r in random.sample(json.load(open('synth_docs_qa.json')), 10)]"

What to check: DATASET_NOTE.md is the label DeepSeek's terms ask for if you publish the data: keep it next to the file. Compare ten sampled answers with your documentation; delete rows that are wrong, or tighten the prompt and regenerate. If the filter drops most rows, the model is probably not returning clean JSON: print one raw response and adjust parse.

If your data is multi-turn, LLaMA-Factory also reads a conversation format; the same docs page shows the dataset_info.json entry for it.

Step 3: rent a GPU

On Deploy, choose a card. The CloudGPU docs say two things that matter here: every template is an image with root SSH, and for anything a template does not cover you can rent a raw GPU (Deploy â†’ Raw GPU) and install what you need over SSH. For this tutorial, pick a template that includes PyTorch, or choose Raw GPU and install it yourself. Pick a card with enough memory for your base model: the LLaMA-Factory README has a rough memory table (marked as estimates), and nvidia-smi shows real use during the first training steps.

GPU instances are billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (terms). Rates are on the pricing page.

Connect with the SSH details of your running instance. Then upload the dataset from your own machine (scp takes the port with a capital -P):

scp -P <ssh-port> synth_docs_qa.json <ssh-user>@<ssh-host>:/workspace/

Step 4: install LLaMA-Factory and register the dataset

These are the install commands from the LLaMA-Factory README (read 2026-10-27; the repository name is spelled LlamaFactory there). The docs FAQ lists a LLaMA-Factory template, but don't count on it being installed or current: check, and install if needed.

cd /workspace
git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

Copy the dataset into data/ and add an entry to data/dataset_info.json, which is how LLaMA-Factory finds custom datasets:

cp /workspace/synth_docs_qa.json data/
python3 - <<'EOF'
import json
p = "data/dataset_info.json"
info = json.load(open(p, encoding="utf-8"))
info["synth_docs_qa"] = {
    "file_name": "synth_docs_qa.json",
    "columns": {"prompt": "instruction", "query": "input", "response": "output"},
}
json.dump(info, open(p, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
EOF

What to check: llamafactory-cli help prints the command list, and grep synth_docs_qa data/dataset_info.json finds your entry.

Step 5: choose a base model and train a LoRA adapter

Small models are enough to see the pipeline work. Pick one row; the template name must match the model.

Base model	LLaMA-Factory `template`	License (read it yourself)	Access
Llama 3.2 3B Instruct (`meta-llama/Llama-3.2-3B-Instruct`)	`llama3`	Llama 3.2 Community License	Gated on Hugging Face: accept the license, use a token
Gemma 3 4B IT (`google/gemma-3-4b-it`)	`gemma3`	Gemma Terms of Use	Gated on Hugging Face: accept the terms, use a token
Mistral 7B Instruct v0.3 (`mistralai/Mistral-7B-Instruct-v0.3`)	`mistral`	License shown on the model card	The model card listed no gate when we looked (2026-10-27)

Template names are from the LLaMA-Factory README's model table; license and access information was read on 2026-10-27 and can change. The Llama license has conditions that apply if you distribute or make available a model built on it. Confirm the license for your use yourself before you train or share anything.

For a gated model, request access on its Hugging Face page, then give the instance a fine-grained read token for just this run (you will revoke it at the end):

export HF_TOKEN="paste-a-read-token-here"

Save this as synth_lora.yaml in the LlamaFactory folder. The keys follow the official LoRA example in the repo's examples/train_lora/ folder; the model, template and dataset lines are the parts you change:

### model
model_name_or_path: meta-llama/Llama-3.2-3B-Instruct

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all

### dataset
dataset: synth_docs_qa
template: llama3
cutoff_len: 2048
preprocessing_num_workers: 4

### output
output_dir: saves/llama32-3b/lora/synth
logging_steps: 10
save_steps: 100
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true

### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 50

Train:

llamafactory-cli train synth_lora.yaml

Open the LLaMA-Factory examples folder and compare your YAML with the current LoRA example: key names change between releases.

What to check: the log prints a training loss every 10 steps and, with val_size set, an evaluation loss every 50; saves/llama32-3b/lora/synth contains adapter_model.safetensors and adapter_config.json. If the base model download fails with a 401 or 403, the Hugging Face access request or token is the problem. An out-of-memory error means a smaller cutoff_len, a smaller model or a card with more memory; nvidia-smi shows the real use.

Unsloth (optional). The LLaMA-Factory README lists Unsloth among its supported add-ons and describes a use_unsloth: true argument for LoRA. Unsloth has its own documentation and install steps, and supports a specific list of models, so check that your base model is on its list before you add it.

Step 6: try the adapter, copy it off, destroy the instance

A quick test in the terminal (no server is started). Save as infer.yaml:

model_name_or_path: meta-llama/Llama-3.2-3B-Instruct
adapter_name_or_path: saves/llama32-3b/lora/synth
template: llama3
infer_backend: huggingface

llamafactory-cli chat infer.yaml

Ask ten questions that are not in your training data and that you wrote yourself, and compare the answers with your documentation. That is your real test; the evaluation loss is not.

Now copy the adapter to your own machine before you destroy anything: destroying the instance permanently deletes its local storage. From your own machine:

rsync -avP -e "ssh -p <ssh-port>" \
  <ssh-user>@<ssh-host>:/workspace/LlamaFactory/saves/llama32-3b/lora/synth/ ./adapter-synth/

Check that adapter_model.safetensors is in ./adapter-synth/, revoke the Hugging Face token you pasted on the instance, then destroy the instance from your Dashboard (the docs: "You can destroy an instance at any time"). The minutes you spent copying are billed like any other minute, and billing stops once the instance is destroyed.

What to check: the instance no longer appears as running in your dashboard, and ./adapter-synth/ still opens on your machine.

Privacy and access, in plain terms

Generation (step 1) sends the documentation sections you give it to a model API. Don't send text you would not want processed by a model service, and read the privacy policy for what the service handles.
You keep ownership of what you put on the GPU instance. CloudGPU's terms (section 5) say it does not access the contents of your instance except where needed to investigate a credible report of a policy violation, where compelled by valid legal process, or to protect the safety of users or third parties. Local instance storage is deleted when you destroy the instance.
This tutorial does not start a server. If you later serve your model from an instance, use your own endpoint with your own API key: the terms (section 6) say instance service URLs are not authenticated by default, so add authentication yourself before you share the URL.

Sources

All read on 2026-10-27 (UTC+8).

DeepSeek, Terms of Use, Last Update March 27, 2026: cdn.deepseek.com, sections 3.1 and 4.2.
LLaMA-Factory: README and data preparation docs.
CloudGPU: API page, terms (sections 4 to 6), acceptable use policy (section 1.8, last updated April 22, 2026), privacy policy.
Licenses: Llama 3.2, Gemma, Mistral 7B Instruct v0.3 model card.

Wan 2.2 A14B at 720p on a 24 GB GPU: why it runs out of memory and what to change

CloudGPU — Thu, 29 Oct 2026 02:37:10 +0000

If you try Wan 2.2 T2V A14B at 1280×720 on a 24 GB card and get torch.OutOfMemoryError, you are not alone. A public bug report, reported in InvokeAI issue #9831 (2026-10-12) on an RTX 4090 24GB, lays out exactly when it fails, why, and which setting got it running. This post walks through that report and the official docs, step by step.

Everything below is quoted from the issue or from official documentation. We have not reproduced the runs ourselves.

1. The symptom and the conditions it was reported under

From InvokeAI issue #9831 (opened 2026-10-12, open, labelled bug, no comments when read on 2026-10-28):

Machine: Windows 10, RTX 4090 with 24 GB VRAM, manual development checkout of InvokeAI main at 8111b5067e (7.0.0-alpha.1), Python 3.12, torch 2.13.0+cu130.
Model: Wan 2.2 T2V A14B, GGUF Q4_K_M, both experts (High Noise and Low Noise), plus the Wan2.2-T2V-A14B-Diffusers components and the Lightning 4-step LoRAs (V1.1, high and low).
Job: 1280×720, 81 frames, 4 steps, guidance 1.0, default invokeai.yaml.
Error: torch.OutOfMemoryError in diffusers/models/transformers/transformer_wan.py:502.

The issue's results table (settings and outcome as reported; we list the memory outcome only):

`enable_partial_loading`	`device_working_mem_gb`	Attention	Memory outcome reported in #9831
true (default)	3 (default)	PyTorch SDPA	OOM in the first transformer block, also at 832×480
false	3	PyTorch SDPA	Completed, but VRAM full; overflowed into shared system memory
true	10	PyTorch SDPA	Completed (the issue's workaround for 24 GB)
true	10	SageAttention	Completed, but overflowed (about 1.2 GB more held than SDPA)
false	3	SageAttention	Completed, but overflowed into shared system memory

Two details matter when you compare this with your own machine:

The run with partial loading off did not really "fit". The issue says the card still overflowed and Windows moved memory to shared system memory instead of raising an OOM.
This is a Windows machine on a 7.0.0 alpha development build. Linux, or a release build, may behave differently; the issue does not test them.

What to check

Your error matches: torch.OutOfMemoryError in the Wan transformer, at the first denoise block.
Your invokeai.yaml still has the defaults (enable_partial_loading: true, device_working_mem_gb: 3, or neither line set).
On Windows, watch "Shared GPU memory" in Task Manager. If it climbs during generation, you are in the spill case, not the OOM case.

2. Why 24 GB is not enough here

Two experts, one at a time. The official Wan2.2 README (read 2026-10-28) describes the A14B models as a two-expert design: "a high-noise expert for the early stages, focusing on overall layout; and a low-noise expert for the later stages, refining video details. Each expert model has about 14B parameters, resulting in a total of 27B parameters but only 14B active parameters per step". So you download and load two diffusion models, but each denoise step uses one.

The official script's own guidance. For its single-GPU 1280*720 A14B command (with --offload_model True --convert_model_dtype), the README says: "This command can run on a GPU with at least 80GB VRAM." That statement is about the Wan repo's generate.py, not about InvokeAI or GGUF weights, but it tells you 720p A14B is not a small job.

What the issue found inside InvokeAI. According to #9831:

Wan denoise has no working-memory estimate. _get_wan_transformer_working_mem_bytes returns None unless wan_memory_optimization is on, so the expert is loaded with the floor, device_working_mem_gb (3 GB), as its working memory.
At 720p with 81 frames the transformer sees 75,600 tokens, "and the activations need far more than 3 GB."
With partial loading on, the cache frees only 2,003 MB of the 10.8 GB UMT5 text encoder before loading the 9.2 GB expert, so the first block runs out of memory with 21.3 GiB allocated.

In short: model weights plus text encoder plus the attention activations for 75,600 tokens do not fit in 24 GB, and InvokeAI reserves only 3 GB for the activations by default.

What to check

The InvokeAI log around model loading: is the text encoder still partly in VRAM when the expert loads?
Your resolution and frame count. The 75,600-token figure is for 1280×720 with 81 frames only.

3. Local settings to change

3a. Raise the working memory

InvokeAI's Low-VRAM mode docs (read 2026-10-28): "Invoke reserves 3GB VRAM as working memory by default", and you can raise it in invokeai.yaml to prevent OOMs (the docs' example sets 4).

The issue's workaround for this exact case on 24 GB is 10:

# invokeai.yaml — workaround reported in InvokeAI issue #9831 for 24 GB
device_working_mem_gb: 10

With that, partial loading on and SDPA, the issue reports the 720p, 81-frame job completing. Invoke needs a restart after you edit the file.

What to check

The OOM is gone and Task Manager's shared GPU memory stays flat during denoise.
If you also set max_cache_vram_gb, the docs say to keep it at or below your VRAM minus device_working_mem_gb minus reserve_vram_gb.

3b. Keep partial loading on, and turn off the Windows sysmem fallback

The docs say Low-VRAM mode is enabled by default via enable_partial_loading: true. In the issue, turning it off did not prevent the overflow: the encoder was unloaded, but the run spilled into shared memory.

The same docs ask Windows users to disable the Nvidia sysmem fallback: NVIDIA Control Panel → 3D Settings → Manage 3D Settings → "CUDA - Sysmem Fallback Policy" → "Prefer No Sysmem Fallback". With it disabled, an overflow shows up as an OOM you can act on instead of silently spilling into shared system memory.

What to check

enable_partial_loading is true (or absent) in invokeai.yaml.
On Windows, the NVIDIA Control Panel setting reads "Prefer No Sysmem Fallback".

3c. Try `wan_memory_optimization` (opt-in)

From the same docs page: setting wan_memory_optimization: true makes partial-load Wan transformer weights "target about 2 GiB resident and stream remaining layers from RAM". It "reduces peak VRAM during both denoise and decode, but generation can be substantially slower and requires enough system RAM for offloaded weights." The docs also note same-seed output may differ slightly from the baseline path.

The issue explains that this flag is also what turns on Wan's working-memory estimate. It does not report a run with the flag on, so we have no figures for it.

wan_memory_optimization: true

What to check

Free system RAM during generation; the docs say offloaded weights need it.
Compare a short test clip against your baseline before committing to long runs.

3d. Lower the resolution, and treat frame count with care

The issue's only other resolution is 832×480. With default settings it also ran out of memory. With partial loading on and device_working_mem_gb: 10, it fit with both attention backends.

The issue tests 81 frames only. Fewer frames means a shorter clip, but the issue gives no figures for it, so try it on your own setup rather than relying on a number from us.

What to check

At 832×480, the default settings still fail in the issue. Raise device_working_mem_gb as well; do not rely on resolution alone.

3e. Quantization, and the official script's flags

The issue already used GGUF Q4_K_M weights for both experts and still ran out of memory at default settings. Lower-bit weights alone did not fix it there; working memory did.

If you use the Wan repo's own generate.py instead of InvokeAI, the README says: "If you encounter OOM (Out-of-Memory) issues, you can use the --offload_model True, --convert_model_dtype and --t5_cpu options to reduce GPU memory usage."

On attention, these are figures reported in InvokeAI issue #9831 (one RTX 4090 24GB on Windows), not measured by us and not a claim about CloudGPU cards: the issue reports SageAttention held about 1.2 GB more than SDPA at 75,600 tokens, which pushed that card over the edge at 720p. At 832×480, the issue reports both backends fit.

What to check

If you switched to SageAttention and shared GPU memory now climbs at 720p, switch back to SDPA and compare.
The Wan2.2 model weights are licensed Apache-2.0 (per the README and the model card); check the license of any community GGUF build you download separately.

4. If local settings are still not enough: run the same workflow on a card with more VRAM

If the slower settings are not an option for you, the other route is more VRAM. On CloudGPU you can rent a card with more VRAM, billed per minute, and set up the same InvokeAI workflow on it. Cards listed on /pricing when we read it on 2026-10-28 include:

Card on /pricing	VRAM (as shown on /pricing)
RTX 5090 32G	32 GB
RTX 4090 48G	48 GB
A100 80G	80 GB

To be clear about what this does and does not mean:

We have not tested Wan 2.2 A14B, InvokeAI or this workflow on any of these cards. The only difference we can state is more VRAM than a 24 GB card.
VRAM need depends on model, precision and workflow. See How much VRAM do I need to run an LLM?
Billing is per minute, rounded up, with a 1-minute minimum, and stops when you destroy the instance (terms). Rates per card are on /pricing.

What to check

Before renting, compare the VRAM column above with what your settings needed locally (the InvokeAI log and Task Manager tell you where it ran out).
Destroy the instance when the job is done, so billing stops.

FAQ

Why does Wan 2.2 A14B run out of memory at 720p on a 24 GB GPU in InvokeAI?
According to InvokeAI issue #9831, Wan denoise has no working-memory estimate unless wan_memory_optimization is on, so InvokeAI reserves only the default 3 GB for activations. At 1280×720 with 81 frames the transformer sees 75,600 tokens, and the activations need far more than that.

What setting fixed it in the issue?
device_working_mem_gb: 10 in invokeai.yaml, with partial loading left on. The issue reports the 720p, 81-frame job then completed on an RTX 4090 24GB (Windows 10, InvokeAI 7.0.0-alpha.1).

Does turning off partial loading help?
Not in the issue. With enable_partial_loading: false the card still overflowed, and Windows moved memory to shared system memory instead of raising an OOM.

Does lowering the resolution to 832×480 avoid the OOM?
Not on its own. The issue reports an OOM at 832×480 with default settings too; with device_working_mem_gb: 10 it fit.

What does the official Wan2.2 README say about VRAM for A14B at 720p?
For its own single-GPU generate.py command at 1280*720, it says the command "can run on a GPU with at least 80GB VRAM", and suggests --offload_model True, --convert_model_dtype and --t5_cpu for OOM issues.

Sources

InvokeAI issue #9831, https://github.com/invoke-ai/InvokeAI/issues/9831 (opened 2026-10-12; read 2026-10-28)
Wan2.2 README (GitHub), https://github.com/Wan-Video/Wan2.2 (read 2026-10-28)
Wan2.2-T2V-A14B model card, https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B (read 2026-10-28)
InvokeAI docs, Low-VRAM mode, https://invoke.ai/configuration/low-vram-mode/ (read 2026-10-28)
CloudGPU /pricing and /terms (read 2026-10-28)

Use DeepSeek V4.1 Flash, Kimi and GLM in RAGFlow with CloudGPU

CloudGPU — Tue, 27 Oct 2026 09:43:22 +0000

RAGFlow's pull request #21543, "feat: add CloudGPU as a model provider", was merged into RAGFlow's main branch on 2026-10-27. It adds CloudGPU as a model provider for chat models and image input. This post shows how to use DeepSeek, Kimi, GLM and MiniMax chat models from it.

As of 2026-10-27 the CloudGPU provider is merged into RAGFlow's main branch but is not in any tagged release yet. Until then, use RAGFlow's OpenAI-API-Compatible provider with the base URL https://cloudgpu.app/v1.

Short answer

If your RAGFlow build includes the merge, pick CloudGPU under User settings > Model providers and paste an API key from the CloudGPU console.
If you run a tagged release (the CloudGPU provider is not in one yet), use the OpenAI-API-Compatible provider with the base URL https://cloudgpu.app/v1.
CloudGPU has no embedding models. Embeddings for your knowledge bases must come from somewhere else.

Disclosure: I work on CloudGPU. We have not run RAGFlow end to end against this API ourselves. The steps below follow the PR and RAGFlow's own docs, and anything we could not check is marked as a check for you to run.

What you need

A RAGFlow install (self-hosted).
A CloudGPU API key. It starts with cgw-sk- and is created in the API Console.
An embedding model from another source (see the embeddings section).

For the shell checks, put the key in an environment variable so it stays out of your history and config files:

export CGW_API_KEY="cgw-sk-your-key"

Create an API key

Following the API docs quick start:

Open the API Console and create a key.
Copy the cgw-sk-... value. The docs say it is shown once, so copy it right away and store it somewhere safe.
Use it as Authorization: Bearer cgw-sk-<your_key>, or paste it into RAGFlow's API Key field as shown below.

The docs also say keys are scoped to your CloudGPU account and can be revoked at any time from the console.

Which models

These IDs are on the CloudGPU API page (checked 2026-10-27) and in the PR's conf/models/cloudgpu.json. The IDs are plain (no organization prefix) and case-sensitive.

Model ID	Model	Image input (marked in the provider config)
`deepseek-v4.1-flash`	DeepSeek V4.1 Flash	yes
`deepseek-v4-flash`	DeepSeek V4 Flash	no
`deepseek-v4-pro`	DeepSeek V4 Pro	no
`kimi-k2.7-code`	Kimi K2.7 Code	yes
`kimi-k2.6`	Kimi K2.6	yes
`glm-5.3-flash`	GLM-5.3 Flash	yes
`glm-5.2`	GLM-5.2	no
`minimax-m3`	MiniMax M3	yes

These 5 models (deepseek-v4.1-flash, kimi-k2.7-code, kimi-k2.6, glm-5.3-flash, minimax-m3) are marked as supporting image input in the CloudGPU provider config. Send one test image yourself before relying on it.

The PR's config also marks all of these models as tool-capable. Treat that as a hint and test it with your own agent or chat setup.

Option A: the CloudGPU provider (builds that include the merge)

The PR adds a provider named exactly CloudGPU (CloudGPU = 'CloudGPU' in web/src/constants/llm.ts, and "name": "CloudGPU" in conf/llm_factories.json). Its default URL is https://cloudgpu.app/v1, and the API key link in the UI points to the CloudGPU API console. The provider entry is tagged LLM,IMAGE2TEXT.

Following RAGFlow's model provider docs:

Go to User settings > Model providers. Under Available models, look for CloudGPU.
Select it and create an instance (the docs ask for an instance name).
Enter your API Key (the cgw-sk-... key). Keep the default Base URL, https://cloudgpu.app/v1, if the form shows one.
Verify the connection, add models from the list or as custom models, then store the instance.
Under default models, set a default LLM from the IDs above.

Reader check: if CloudGPU is not in the list, your build does not include the merge yet. Use Option B. Exact field names can differ between RAGFlow versions; follow the docs for the version you run.

Option B: OpenAI-API-Compatible provider (use this until a tagged release includes CloudGPU)

RAGFlow's docs say that for models not listed, you can click OpenAI-API-Compatible on the Model providers page. This provider is already in RAGFlow's provider list.

Go to User settings > Model providers and select OpenAI-API-Compatible.
Create an instance and enter a name.
API Key: your cgw-sk-... key. Base URL: https://cloudgpu.app/v1. RAGFlow's docs warn that an incorrect Base URL makes verification or model calls fail, and to confirm whether the path must include /v1. Here it does.
Verify the connection.
Add a model with Add custom model:
- Model name: one ID from the table, exactly as written, for example deepseek-v4.1-flash. The docs say the name must match the ID the provider exposes.
- Model type: the chat/LLM type.
- Max tokens: fill in a value that fits your use. CloudGPU does not publish per-model output caps on its API page, so pick your own.
- Tool call: enable it only if you tested tool calling for that model (the docs say not to enable it for models that lack it). Newer docs call this field Tool calling.
- Click Confirm, then check that the model works with a real call.
Repeat for each ID you want. Then set your default LLM.

For image input in Option B, add the same ID once more and choose the vision-language model type (the docs call it VLM). Only use IDs marked "yes" in the table above. Reader check: the exact type name in the dropdown depends on your version.

Check the key and model ID outside RAGFlow first

Two curl calls take a minute and separate "my key or model ID is wrong" from "my RAGFlow settings are wrong".

# lists the IDs your key can use (without a key this returns HTTP 401 missing_authorization)
curl -s https://cloudgpu.app/v1/models \
  -H "Authorization: Bearer $CGW_API_KEY"

# one plain chat request
curl https://cloudgpu.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $CGW_API_KEY" \
  -d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"Say hi in one sentence."}]}'

You should get a JSON chat completion back. Run it yourself to confirm; we are not quoting an output here.

The same call with the OpenAI Python SDK, since the endpoint is OpenAI-compatible:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://cloudgpu.app/v1",
    api_key=os.environ["CGW_API_KEY"],
)

resp = client.chat.completions.create(
    model="deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "Say hi in one sentence."}],
)
print(resp.choices[0].message.content)

Embeddings: configure them separately

CloudGPU does not offer embedding models. The PR states this as well ("CloudGPU serves no embedding models, so this adds chat and image input only"), and the API docs say "Embeddings are not available yet."

A RAGFlow knowledge base needs an embedding model to index and retrieve your documents. RAGFlow's docs say that, at minimum, you should set a default LLM and a default embedding model. So:

Use the embedding option your RAGFlow install already has, or a local embedding model (RAGFlow's docs describe connecting locally deployed models with Ollama or vLLM), or an embedding model from another provider.
Set that as your default embedding model, and use a CloudGPU model only as the chat model for chats, search and agents.

Reader check: after you set both, create a small knowledge base, parse one document and ask a question in a chat that uses a CloudGPU model. If parsing fails, look at the embedding settings first, not the CloudGPU ones.

Troubleshooting

HTTP 401 invalid_api_key: the key is unknown or revoked. Create a new one in the console.
HTTP 401 missing_authorization: no Authorization: Bearer header was sent.
HTTP 400 model_not_found: the model ID is not in the catalog. Compare it with GET /v1/models; IDs are case-sensitive.
HTTP 402 insufficient_balance: the account balance is too low for the request.
HTTP 429 rate_limit_exceeded: the API docs list 100 requests per minute per key and 10,000 requests per day per account for the public beta. Bulk jobs such as parsing many files at once could reach that, so check whether your RAGFlow task makes many calls.
Verification fails in RAGFlow but curl works: re-check the Base URL (it must end in /v1, no trailing path), and that no extra spaces were pasted into the API Key field.
CloudGPU is missing from Available models: use Option B.

FAQ

Is CloudGPU in a RAGFlow release?
Not yet. As of 2026-10-27 the provider is merged into RAGFlow's main branch but is not in any tagged release. Until then, use the OpenAI-API-Compatible provider with the base URL https://cloudgpu.app/v1, and check the releases page for later versions.

Can CloudGPU models make my embeddings?
No. Configure embeddings separately, as described above.

Do I need the PR for the base URL route?
No. The OpenAI-API-Compatible provider only needs a base URL, a key and a model ID.

Keys, the model table and the quick start are on the CloudGPU API page.

Use DeepSeek V4.1 Flash and Kimi K2.7 Code in goose with a custom provider file

CloudGPU — Tue, 27 Oct 2026 06:35:25 +0000

Short answer: copy the JSON below into ~/.config/goose/custom_providers/cloudgpu.json, run export CGW_API_KEY="cgw-sk-your-key", then start goose with --provider cloudgpu --model deepseek-v4.1-flash.

Disclosure: I work on CloudGPU, and the endpoint in this post is ours. We haven't tested goose against this endpoint ourselves; the steps follow goose's docs (Configure Custom Provider). Treat the checks below as things you run, not results we measured.

Note: Tool calling through this endpoint isn't tested by us. goose leans on tool calls for real work, so run the self-checks in Step 5 before you hand it a task that matters.

Why a file and not a built-in provider

goose's CONTRIBUTING.md asks services with an OpenAI-compatible API to publish a goose custom-provider JSON file with their own documentation, so users can drop it into their local custom_providers directory without waiting for a goose release. This post is that documentation for CloudGPU.

What you need

goose (CLI or Desktop), installed from its own docs.
A CloudGPU API key. Go to API Console and create a key. Copy the cgw-sk-... value — it's shown once. The quick start is on the API docs page.

Step 1: Get the JSON

Copy this into a file named cloudgpu.json:

{
  "name": "cloudgpu",
  "engine": "openai",
  "display_name": "CloudGPU",
  "description": "DeepSeek, Kimi, GLM and MiniMax models through CloudGPU's OpenAI-compatible API.",
  "api_key_env": "CGW_API_KEY",
  "base_url": "https://cloudgpu.app/v1",
  "dynamic_models": false,
  "models": [
    { "name": "deepseek-v4.1-flash" },
    { "name": "deepseek-v4-pro" },
    { "name": "kimi-k2.7-code" },
    { "name": "glm-5.3-flash" },
    { "name": "minimax-m3" }
  ],
  "supports_streaming": true,
  "requires_auth": true,
  "model_doc_link": "https://cloudgpu.app/api/docs",
  "setup_steps": [
    "Sign in at https://cloudgpu.app, then go to API Console: https://cloudgpu.app/api/console",
    "Create a key. Copy the cgw-sk-... value — it's shown once.",
    "Top up your balance via Billing. Model IDs are listed on https://cloudgpu.app/api",
    "Set the CGW_API_KEY environment variable to your key before starting goose"
  ]
}

What the fields do, in goose's terms:

name is the provider ID you pass to --provider.
engine: "openai" tells goose to speak the OpenAI-compatible chat format.
base_url is the CloudGPU endpoint. goose adds chat/completions after the /v1.
api_key_env is the name of the environment variable goose reads the key from. There is no key in the file, and there never should be.
models is the list goose shows. With dynamic_models: false, goose uses only these five IDs and doesn't fetch a model list from the endpoint.
No context_limit is set. It's optional in goose's format, and we don't publish context sizes for these models, so goose uses its own defaults.

Step 2: Put it where goose looks

goose's docs give one folder per OS:

macOS/Linux: ~/.config/goose/custom_providers/
Windows: %APPDATA%\Block\goose\config\custom_providers\

On macOS or Linux:

mkdir -p ~/.config/goose/custom_providers
cp cloudgpu.json ~/.config/goose/custom_providers/cloudgpu.json

Self-check: goose info prints the configuration file location. custom_providers sits in that same config folder. If CloudGPU doesn't show up in the provider list, restart goose.

Step 3: Set the key

export CGW_API_KEY="cgw-sk-your-key"

Use your own key in place of cgw-sk-your-key, and run goose from that same shell. Two things from goose's config docs worth knowing:

goose does not read provider API keys from config.yaml. A key there is ignored.
If you'd rather keep the key in goose's keychain, open the provider in goose Desktop and enter the key there.
If goose Desktop changes the variable name, use the one shown in the provider's settings (Settings, then the Models tab, then Configure providers, then click CloudGPU in the list).

Self-check, outside goose: this is the request from our API docs. If it fails, fix it before going further.

curl https://cloudgpu.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $CGW_API_KEY" \
  -d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"Say hi in one sentence."}]}'

Step 4: Pick a model

These are the model IDs in the file. Each one is in the model table on our API page:

Model ID	Model
`deepseek-v4.1-flash`	DeepSeek V4.1 Flash
`deepseek-v4-pro`	DeepSeek V4 Pro
`kimi-k2.7-code`	Kimi K2.7 Code
`glm-5.3-flash`	GLM 5.3 Flash
`minimax-m3`	MiniMax M3

IDs are case-sensitive and the dots matter. A wrong ID comes back as 400 model_not_found.

goose CLI: pass the provider and model per run, as in goose's CLI docs: goose run --provider cloudgpu --model kimi-k2.7-code -t "...". goose also reads GOOSE_PROVIDER and GOOSE_MODEL from the environment.
goose Desktop: goose's docs say custom providers appear in the provider list like any other provider. Open Settings, then the Models tab, then Switch models, choose CloudGPU, and pick a model (or Use custom model to type an ID). Confirm with Select model.

Step 5: Quick test

Start with a plain prompt and no session file:

goose run --no-session --provider cloudgpu --model deepseek-v4.1-flash \
  -t "Reply with one short sentence that names the model you are."

Self-checks (what to look for, not what we saw):

You get a reply from the model, not an error. If you see 401 missing_authorization or 401 invalid_api_key, the key isn't reaching goose. Check that CGW_API_KEY is set in the shell that runs goose.
400 model_not_found: compare the --model value with the table above, character for character.
cloudgpu isn't recognised as a provider: check the file is at the path from Step 2 and that it parses (python3 -m json.tool ~/.config/goose/custom_providers/cloudgpu.json).

Then try a first coding task in a scratch folder, with goose's developer extension turned on:

mkdir -p ~/goose-cloudgpu-test && cd ~/goose-cloudgpu-test
goose run --no-session --provider cloudgpu --model kimi-k2.7-code --with-builtin developer \
  -t "Create hello.py that prints the numbers 1 to 5, then show me the file."

This one needs tool calls, which we haven't tested through our endpoint. Self-check: hello.py exists in the folder and contains what you asked for. If goose answers in text but no file appears, the model's tool calls didn't make it through. Try another model ID from the table and tell us which one failed.

Where to go next

goose's own reference for this file format: Configure Custom Provider.
goose CLI flags (run, --provider, --model): goose CLI commands.

If something in these steps doesn't match what your goose shows, leave a comment with your goose version and the error text (never the key), and we'll update the post.

Roo Code with DeepSeek V4 Flash over an OpenAI-compatible endpoint (plus a tool-call check first)

CloudGPU — Tue, 27 Oct 2026 00:22:35 +0000

Adapted from our CloudGPU guide on using DeepSeek V4 Flash in Roo Code.

Short answer: in Roo Code, pick the OpenAI Compatible provider, set the Base URL to https://cloudgpu.app/v1, paste your API key and type the model ID deepseek-v4-flash. Before you start a real task, send one request with a tools array. Roo Code only works with models that return native tool calls, and that one request tells you whether yours does.

Disclosure: I work on CloudGPU, and the endpoint below is ours. The Roo Code steps follow the official Roo Code docs (Using OpenAI Compatible Providers). We haven't run Roo Code end to end against this endpoint ourselves, so treat the checks below as something you run, not results we measured.

Note: Tool calling through this endpoint isn't tested by us; run the self-check in Step 3 before relying on Roo Code's tools.

What you need

An API key (cgw-sk-...) from the API Console. It is shown once, so copy it right away. The quick start is on the API docs page.
Roo Code installed in VS Code (follow its install docs).
The key in an environment variable, so it stays out of your shell history and scripts:

export CGW_API_KEY="cgw-sk-your-key"

Step 1: Configure the provider in Roo Code

From the Roo Code docs, the settings live in the Roo Code settings panel:

Roo Code field	Value
API Provider	OpenAI Compatible
Base URL	`https://cloudgpu.app/v1`
API Key	your `cgw-sk-...` key
Model	`deepseek-v4-flash`

Model Configuration (Max Output Tokens, Context Window and so on) is optional. We don't publish values for these models here, so leave the defaults unless you know the limits you need.

Other chat model IDs on the API page model table (checked 2026-10-27) that go in the same Model field:

Model ID	Model
`deepseek-v4-flash`	DeepSeek V4 Flash
`deepseek-v4.1-flash`	DeepSeek V4.1 Flash
`kimi-k3`	Kimi K3
`kimi-k2.6`	Kimi K2.6

IDs are case-sensitive. GET /v1/models (with your key) returns the exact list your key can use:

curl -s https://cloudgpu.app/v1/models \
  -H "Authorization: Bearer $CGW_API_KEY"

Without the Authorization header you get 401 missing_authorization, which is a quick way to confirm the URL is right but the key isn't being sent.

Step 2: Plain chat check

This is the request from the API docs, with your model ID:

curl https://cloudgpu.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $CGW_API_KEY" \
  -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"Say hi in one sentence."}]}'

If this fails, fix it before touching Roo Code. The troubleshooting list at the end maps the error codes.

Step 3: Native tool-call check (the one that matters for Roo Code)

The Roo Code docs say it "uses native tool calling exclusively" and "If a model does not support native tool calling, it cannot be used with Roo Code." So send one request with a tools array and look at what comes back:

curl https://cloudgpu.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $CGW_API_KEY" \
  -d '{
    "model": "deepseek-v4-flash",
    "messages": [{"role": "user", "content": "What is the weather in Paris? Use the tool."}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }]
  }'

The same check with the openai Python package (pip install openai), looping over the four model IDs from the table above. Check GET /v1/models for the current list.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://cloudgpu.app/v1",
    api_key=os.environ["CGW_API_KEY"],
)

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

for model in ["deepseek-v4-flash", "deepseek-v4.1-flash", "kimi-k3", "kimi-k2.6"]:
    try:
        resp = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": "What is the weather in Paris? Use the tool."}],
            tools=tools,
            max_tokens=2048,  # leave room: some models think before they answer
        )
    except Exception as e:
        print(f"{model}: error -> {e}")
        continue
    msg = resp.choices[0].message
    if msg.tool_calls:
        call = msg.tool_calls[0].function
        print(f"{model}: tool call -> {call.name}({call.arguments})")
    else:
        print(f"{model}: no tool call, text -> {msg.content!r}")

How to read it:

tool_calls present (a get_weather call with {"city": "Paris"} or similar): the model returned a tool call for this request; try it in Roo Code on a small task before you rely on it.
Plain text or an error: Roo Code is unlikely to work with that model. Try another ID from /v1/models.
Empty content and no tool call: some models spend output tokens on thinking first. Raise max_tokens (and Max Output Tokens in Roo Code's Model Configuration) and run it again before you decide.

Troubleshooting

These match the Roo Code docs' troubleshooting list and the error codes on the API docs page:

"Invalid API Key" (HTTP 401 invalid_api_key): the key is wrong or revoked. Create a new one in the console (keys are shown once).
HTTP 401 missing_authorization: no Authorization: Bearer header was sent. In a shell, check echo $CGW_API_KEY isn't empty.
"Model Not Found" (HTTP 400 model_not_found): check the ID against /v1/models; IDs are case-sensitive.
Connection errors: the Base URL must be exactly https://cloudgpu.app/v1.
Tool-calling errors: Roo Code needs native tool calling. Re-run the Step 3 check; if it fails, try another ID and run the check again.
HTTP 429 rate_limit_exceeded: the API docs list 100 requests per minute per key and 10,000 requests per day per account during the beta.

FAQ

Does Roo Code work with an OpenAI-compatible API?
Yes. The Roo Code docs describe connecting to any OpenAI-compatible endpoint with a base URL, an API key and a model ID.

Does deepseek-v4-flash support the tool calling Roo Code needs?
We haven't confirmed it inside Roo Code. Run the Step 3 check before you rely on it.

Can I switch to Kimi without changing anything else?
Yes, change only the Model field to kimi-k3 or kimi-k2.6. Run the tool-call check for that ID first.

Where are the rates?
Per-model rates are in the price table on the CloudGPU API page.

Facts checked 2026-10-27.

Save Your Work Before You Destroy a Cloud GPU: Checkpoints, rsync and Hub Uploads (2026)

CloudGPU — Mon, 26 Oct 2026 13:13:47 +0000

On CloudGPU, destroying an instance deletes its disk, and there is no persistent storage across instances yet, so anything you want to keep — checkpoints, LoRA adapters, generated images, logs — must be copied off the machine before you destroy it. An instance is also destroyed automatically if your balance runs out. This post is the checklist and the commands we use.

What happens to your files, and when

Event	What happens to the disk
You click Destroy	The instance stops, billing settles, the disk is deleted
Your balance cannot cover the next hour	The instance is destroyed and its disk deleted
You want to “pause” and resume tomorrow	Not available yet: stopping releases the GPU; you start a new instance next time
The JupyterLab tab or SSH session closes	Nothing — the instance keeps running (and billing) until destroyed

Because billing is per minute, the economical pattern is: work, copy outputs off, destroy. The minutes spent copying are billed like any other minute, so plan the copy rather than leaving the instance up “just in case”.

Pick a destination

Destination	Good for	Command family
Your own computer	Small to medium outputs, anything private	`rsync` / `scp` over SSH
Hugging Face Hub (private repo)	LoRA adapters, merged models, datasets you will reuse	`huggingface-cli upload`
S3-compatible object storage (S3, R2, B2…)	Large checkpoints, image batches, logs	`rclone copy`
JupyterLab download button	A handful of small files	Browser

Do not push large files through the instance’s public API endpoint: the tunnel is capped at 3 MB/s and is meant for API responses, not file transfer. Use SSH or an upload from the machine.

Outbound speed from the supplier data centre to a given destination varies. Test with a small file first and add the copy time to your budget.

Commands

1. Pull with rsync from your laptop (resumable)

# run on your laptop; user, host and port are on the instance page
rsync -avP -e "ssh -p <ssh-port>" <ssh-user>@<ssh-host>:/workspace/out/ ./run-2026-10-23/

-P shows progress and resumes partial files if the connection drops. Re-run the same command at the end; rsync only copies what changed.

2. Keep pulling during a long run

# laptop: pull new checkpoints every 10 minutes while training runs
while true; do
  rsync -aP -e "ssh -p <ssh-port>" <ssh-user>@<ssh-host>:/workspace/out/ ./run/ && date
  sleep 600
done

Pair it with frequent saves in your trainer (for LLaMA-Factory, save_steps in the YAML; for Hugging Face Trainer, save_steps and save_total_limit).

3. Push a LoRA adapter to a private Hugging Face repo

pip install -U "huggingface_hub[cli]"
huggingface-cli login            # paste a fine-grained write token
huggingface-cli upload <your-user>/qwen3-8b-qlora-run1 /workspace/out/qwen3-8b-qlora . --private

Use a fine-grained token scoped to the one repo and revoke it after the run.

4. Copy a large output folder to object storage

curl https://rclone.org/install.sh | bash          # you are root on the instance; or: apt-get install -y rclone
rclone config                                       # add your S3 / R2 / B2 remote
rclone copy /workspace/out remote:my-bucket/run1 --progress --transfers 8

5. Tar first when there are many small files

tar -C /workspace -czf /workspace/outputs.tgz out/

Thousands of small PNGs or log files copy much faster as one archive.

The pre-destroy checklist

#	Check	Why
1	Training or batch has finished (or you saved a checkpoint)	Partial runs are lost on destroy
2	Outputs copied and verified (file count, sizes, or `sha256sum`)	A failed copy looks like a successful one until you need it
3	Configs, scripts and the exact command line saved	So you can reproduce the run on a fresh instance
4	Environment captured (`pip freeze > requirements.txt`)	Next instance starts clean
5	Tokens you pasted on the machine revoked	Good hygiene, even though the disk is deleted
6	Click Destroy	Unused minutes of the pre-authorised hour return to your balance

Budget for the copy, and for the balance

Two numbers keep long runs safe:

Balance buffer. An instance is destroyed when the balance runs out. For an 8-hour run on a card listed at $0.65/h (RTX 4090 48G, 2026-10-23 23:58 Beijing time), the run itself is $5.20; keep that plus a margin for setup and copying.
Copy time. Estimate size ÷ measured upload speed, then add it to the job. On a card listed at $0.59/h, 10 extra minutes of copying cost about $0.10.

Stock and prices change during the day; check the Deploy page. Billing rules are on per-minute billing.

A note on where your data goes

If your data must not leave a particular jurisdiction, do not train on it here, and choose your copy destination with the same care.

FAQ

What happens to my files when I destroy a cloud GPU instance on CloudGPU?

The disk is deleted. Copy anything you need off the machine before you destroy it.

Is there persistent storage between instances?

Not yet. Stopping releases the GPU and the next session starts on a new instance, so keep checkpoints and outputs elsewhere.

What if my balance runs out during training?

The instance is destroyed and its disk deleted. Keep enough balance for the whole run plus a margin, and copy checkpoints off as you go.

How do I download checkpoints from a rented GPU?

Use rsync or scp over SSH, or upload from the machine to a model hub or object storage. Do not use the public API endpoint, which is capped at 3 MB/s.

Am I billed while I copy files off?

Yes. Billing runs per minute until you destroy the instance, so plan the copy and destroy promptly afterwards.

If the checkpoints you are saving come from a fine-tune, the LoRA and QLoRA fine-tuning page covers card choice and billing for those runs. Originally published at cloudgpu.app.

Prices are list prices read on 2026-10-23 23:58 Beijing time (UTC+8); current prices on /deploy take precedence.

DeepSeek V4.1 Flash in the OpenAI SDK and Aider: change one base URL

CloudGPU — Mon, 26 Oct 2026 08:06:48 +0000

Short answer: if your code already uses the OpenAI SDK, you can call DeepSeek V4.1 Flash by changing two values: set the base URL to https://cloudgpu.app/v1 and set the model to deepseek-v4.1-flash. Aider works the same way through its OpenAI-compatible settings.

Disclosure: I work on CloudGPU, and the endpoint below is ours.

Three DeepSeek IDs, and which is which

The CloudGPU API page lists three DeepSeek chat models (checked 2026-10-26):

Model ID	What it is
`deepseek-v4.1-flash`	DeepSeek V4.1 Flash
`deepseek-v4-flash`	DeepSeek V4 Flash, served from the April checkpoint, not V4.1
`deepseek-v4-pro`	DeepSeek V4 Pro, for harder reasoning and long multi-step tool use

IDs are case-sensitive and the dot matters: a wrong ID comes back as 400 model_not_found. Per-token rates change, so this post doesn't repeat them; the current rate for each model is on the CloudGPU API page.

Get a key

Sign up with an email address or a Google account.
Add balance by card, Alipay, Google Pay, Apple Pay or USDT. The same balance pays for GPU rentals too.
Create a key in the API console. It starts with cgw-sk- and is shown only once, so put it straight into your environment:

export CGW_API_KEY="cgw-sk-..."

Smoke test with curl

curl https://cloudgpu.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $CGW_API_KEY" \
  -d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"hello"}]}'

To list the model IDs your key can use:

curl https://cloudgpu.app/v1/models \
  -H "Authorization: Bearer $CGW_API_KEY"

Without the Authorization header this returns 401 with missing_authorization, so check the key first if you see that.

OpenAI Python SDK

# pip install openai
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["CGW_API_KEY"],
    base_url="https://cloudgpu.app/v1",
)

resp = client.chat.completions.create(
    model="deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "Explain the KV cache in one sentence."}],
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="")

To compare the three DeepSeek models on the same prompt, loop over the IDs and keep everything else the same:

for model in ["deepseek-v4.1-flash", "deepseek-v4-flash", "deepseek-v4-pro"]:
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Write a Python debounce function."}],
    )
    print(f"--- {model}\n{r.choices[0].message.content}\n")

If you later switch the same loop to a thinking model such as glm-5.3-flash, don't set max_tokens too low: reasoning tokens count toward it, and with a small limit the visible content can come back empty with finish_reason="length". Raise the limit or pass reasoning_effort="low".

Node.js

// npm install openai; name the file index.mjs (top-level await needs an ES module)
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.CGW_API_KEY,
  baseURL: "https://cloudgpu.app/v1",
});

const resp = await client.chat.completions.create({
  model: "deepseek-v4.1-flash",
  messages: [{ role: "user", content: "Write a TypeScript debounce function." }],
});
console.log(resp.choices[0].message.content);

Tested 2026-10-26: with the openai npm package, this returns normally for deepseek-v4.1-flash.

Aider

According to the official Aider docs ("OpenAI compatible APIs", read 2026-10-26), Aider connects to an OpenAI-compatible endpoint through two environment variables, and the model name gets an openai/ prefix:

# Mac/Linux
export OPENAI_API_BASE=https://cloudgpu.app/v1
export OPENAI_API_KEY="$CGW_API_KEY"
cd /path/to/your/project
aider --model openai/deepseek-v4.1-flash

REM Windows, per the Aider docs (not part of our test): restart the shell after setx
setx OPENAI_API_BASE https://cloudgpu.app/v1
setx OPENAI_API_KEY %CGW_API_KEY%
aider --model openai/deepseek-v4.1-flash

Tested 2026-10-26 with Aider 0.86.2: started with --model openai/deepseek-v4.1-flash and asked to fix a bug in a small add() function, Aider chose the whole edit format by itself, printed "Applied edit", and the fix was correct. openai/deepseek-v4-flash passed the same test, again with whole.

On startup Aider prints:

Warning for openai/deepseek-v4.1-flash: Unknown context window size and costs

The warning doesn't affect use. To skip it, add --no-show-model-warnings:

aider --model openai/deepseek-v4.1-flash --no-show-model-warnings

Limits to know

100 requests per minute per API key and 10,000 requests per day per account. These are public-beta defaults and can be raised on request (API docs).
Embeddings are not available yet.
The API is in public beta with no SLA.
You pay per token from a prepaid balance, with no subscription.

FAQ

Does DeepSeek V4.1 Flash work with the OpenAI SDK?
Yes. Set base_url to https://cloudgpu.app/v1, use a cgw-sk- key and set the model to deepseek-v4.1-flash.

Is deepseek-v4-flash the same as deepseek-v4.1-flash?
No. deepseek-v4-flash is served from the April V4 Flash checkpoint; deepseek-v4.1-flash is the V4.1 Flash model. Both are separate IDs on the API page.

How do I use it in Aider?
Set OPENAI_API_BASE and OPENAI_API_KEY, then start Aider with --model openai/deepseek-v4.1-flash, as the Aider docs describe for OpenAI-compatible APIs. Tested 2026-10-26 with Aider 0.86.2 (it chose the whole edit format by itself).

Where are the current rates?
On the CloudGPU API page. Rates can change, so check there before a large run.

Adapted from "DeepSeek V4 API With USDT or Card: Use the OpenAI SDK" on the CloudGPU blog.

I work on CloudGPU.

Use DeepSeek V4.1 Flash and Kimi in opencode via a custom OpenAI-compatible provider

CloudGPU — Sun, 25 Oct 2026 22:22:44 +0000

opencode ships with a long list of built-in providers, but you are not limited to them. Its docs describe a custom provider path for any OpenAI-compatible endpoint: you declare the provider in opencode.json, point it at a base URL, list the model IDs, and they show up in the model picker.

This post walks through that setup with two models, DeepSeek V4.1 Flash (deepseek-v4.1-flash) and Kimi K3 (kimi-k3), plus two extras in the same config: kimi-k2.6 and glm-5.3-flash. The endpoint in the examples is CloudGPU's OpenAI-compatible API at https://cloudgpu.app/v1, but the same steps apply to any provider that speaks /v1/chat/completions.

What's tested and what isn't: on 2026-10-26 we ran this setup end to end in opencode 1.19.4. The opencode.json below loaded as written with no schema errors, opencode models cloudgpu listed all four models, and opencode run -m ... returned a reply from both deepseek-v4.1-flash and kimi-k3. All four model IDs were also listed by GET /v1/models, and a direct chat request to each one replied normally. Not yet verified: the interactive /models screen in the TUI, the /connect key flow, and how limit.output maps onto requests. Those parts follow opencode's official docs, so read "according to the docs" literally.

What you need

opencode installed (see the opencode docs for install options).
An API key for your OpenAI-compatible endpoint. For CloudGPU, keys are created on the API page and look like cgw-sk-....
The exact model IDs your endpoint serves. Here: deepseek-v4.1-flash, kimi-k3, kimi-k2.6, glm-5.3-flash.

Step 1: Export your key

opencode's config supports {env:VARIABLE_NAME} substitution (Config → Env vars), so the key never has to live in the JSON file:

export CGW_API_KEY="cgw-sk-..."   # add to ~/.bashrc / ~/.zshrc to keep it across shells

Step 2: Add the provider to `opencode.json`

Create (or edit) opencode.json in your project root. Per the docs, project config takes precedence over the global file at ~/.config/opencode/opencode.json, and config files are merged, so you can also put the provider block in the global file if you want it in every project.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "cloudgpu": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "CloudGPU",
      "options": {
        "baseURL": "https://cloudgpu.app/v1",
        "apiKey": "{env:CGW_API_KEY}"
      },
      "models": {
        "deepseek-v4.1-flash": {
          "name": "DeepSeek V4.1 Flash"
        },
        "kimi-k3": {
          "name": "Kimi K3"
        },
        "kimi-k2.6": {
          "name": "Kimi K2.6"
        },
        "glm-5.3-flash": {
          "name": "GLM-5.3 Flash"
        }
      }
    }
  }
}

What each field does, per opencode's Providers docs:

Field	Meaning
`cloudgpu`	The provider ID. You pick it; it becomes the first half of the model reference (`cloudgpu/<model>`).
`npm`	The AI SDK package. `@ai-sdk/openai-compatible` is the one the docs name for OpenAI-compatible providers that use `/v1/chat/completions`.
`name`	Display name in the UI.
`options.baseURL`	The API endpoint. Note the `/v1` at the end.
`options.apiKey`	Optional; set via `{env:...}` syntax. If you would rather store the key with `/connect` (see below), you can drop this line.
`models`	The model IDs to expose. Each key must match the ID the endpoint expects; `name` is only the label.

Only need the two headline models? Delete the kimi-k2.6 and glm-5.3-flash entries; nothing else changes.

Alternative: store the key with `/connect`

The docs also describe a credential flow: run /connect inside opencode, scroll to Other, enter the provider ID (cloudgpu, matching the key in opencode.json), then paste your key. That stores the credential only; you still need the provider block above. Use one approach or the other.

Step 3: Pick the model

Start opencode in the project and run:

/models

According to the docs, your custom provider and its models appear in this selection list. Choose DeepSeek V4.1 Flash or Kimi K3. (In our test we checked the list from the command line, not this screen.)

To check from the command line that opencode picked up the provider, run:

opencode models cloudgpu

In our test with opencode 1.19.4, this listed:

cloudgpu/deepseek-v4.1-flash
cloudgpu/glm-5.3-flash
cloudgpu/kimi-k2.6
cloudgpu/kimi-k3

To choose a model from the command line instead, use --model / -m in provider/model form. In our test, -m cloudgpu/deepseek-v4.1-flash (the dot in the ID is fine) and -m cloudgpu/kimi-k3 both returned replies:

opencode -m cloudgpu/deepseek-v4.1-flash
opencode run -m cloudgpu/kimi-k3 "Explain what this repo's main entry point does"

You can also set a default in opencode.json with the top-level "model" key, using the same provider_id/model_id form, for example "model": "cloudgpu/deepseek-v4.1-flash".

Which one for what?

I'm not putting benchmark numbers here; try both on your own repo. A practical split:

deepseek-v4.1-flash: a sensible default for everyday edits, explanations, and small refactors.
kimi-k3: a reasoning model; switch to it with /models for multi-step changes or a second opinion on the same task. It spends output tokens on reasoning before it answers (see troubleshooting #4).
kimi-k2.6 and glm-5.3-flash: already in the config, so comparing is one /models switch away.

Troubleshooting

This section is based on a real question we answered: someone added a DeepSeek model as a custom provider in a coding agent, entered the endpoint, key, and two models, and the model never showed up in the picker. The model ID they had entered was a short alias, not the exact ID the endpoint lists. We can't say for sure that was the cause in their case, but a short alias like that is a common cause, and the same checklist applies to opencode.

1. The model doesn't appear in `/models` (or requests fail with "model not found")

The keys under models must match, character for character, the IDs your endpoint returns from GET /v1/models. Check them directly:

curl https://cloudgpu.app/v1/models \
  -H "Authorization: Bearer $CGW_API_KEY"

Copy the id values from the response into opencode.json. Common mismatches: a short alias (deepseek-flash), a missing version segment (deepseek-v4-flash vs deepseek-v4.1-flash), or different capitalization.

Also confirm the provider ID matches everywhere: the key under provider, the ID you typed in /connect (if you used it), and the prefix in -m cloudgpu/.... opencode's own troubleshooting list calls out the /connect ID mismatch, and opencode auth list shows which credentials are stored.

2. The base URL is missing `/v1`

@ai-sdk/openai-compatible appends paths like /chat/completions to baseURL. Use:

https://cloudgpu.app/v1      ✅
https://cloudgpu.app         ❌
https://cloudgpu.app/v1/chat/completions   ❌ (too specific)

3. The environment variable isn't exported

The docs note that if a variable referenced with {env:...} is not set, it is replaced with an empty string, so no key is sent and the request is rejected. In our test (opencode 1.19.4, CGW_API_KEY empty) the error was:

Error: Provide an Authorization: Bearer <api_key> header.

Check the variable in the same shell you launch opencode from:

echo "${CGW_API_KEY:+set}"   # prints "set" if the variable is present

If you set it in ~/.zshrc or ~/.bashrc, open a new terminal (or source the file) before starting opencode. Desktop launchers and IDE terminals may not read your shell profile.

4. Empty replies from reasoning models

kimi-k3, kimi-k2.6, and glm-5.3-flash reason before they answer, and the reasoning counts against the output limit. In our 2026-10-26 test against the API directly:

with max_tokens: 20, all three returned empty or null content, because reasoning used up the tokens first;
with max_tokens: 800, all three replied normally;
kimi-k3 used 1027 completion tokens to answer "ok".

If the reply is empty, raise max_tokens or leave it unset.

In opencode, the per-model knob the docs describe for custom providers is the optional limit block (limit.context, limit.output). If you set limit.output low, or a tool or agent config passes a small output cap, you may hit the same empty-reply behavior. If you add limit, use real values from your provider's model docs:

"kimi-k3": {
  "name": "Kimi K3",
  "limit": { "context": <context tokens>, "output": <max output tokens> }
}

(Replace the placeholders with numbers; the snippet is not valid JSON as written.) How opencode maps limit.output onto the request is per its docs; I haven't verified it against these models inside opencode.

Billing

CloudGPU's API runs on a prepaid balance (card, Alipay, Google Pay, Apple Pay, or USDT). The model list and API key setup are on the CloudGPU API page.

Recap

export CGW_API_KEY=...
Add a provider block with "npm": "@ai-sdk/openai-compatible", "baseURL": "https://cloudgpu.app/v1", and "apiKey": "{env:CGW_API_KEY}".
List exact model IDs under models, verified against GET /v1/models.
Run opencode models cloudgpu to confirm the models are listed, then pick cloudgpu/deepseek-v4.1-flash or cloudgpu/kimi-k3 with /models or -m.
Empty reply from a reasoning model? Raise max_tokens or leave it unset.

Docs used: opencode Providers → Custom provider, opencode Config → Env vars, opencode Models, opencode CLI.

I work on CloudGPU.

DEV Community: CloudGPU

A model as the judge: build preference pairs with an API, then train with DPO on a rented GPU

Where the idea comes from, and what this guide does not claim

Read these things first

What you need

Step 1: write the principles and the questions (on your own machine)

Step 2: write two answers, let the model pick, keep consistent picks

Step 3: read the pairs yourself

Step 4: rent a GPU

Step 5: install LLaMA-Factory and register the preference dataset

Step 6: choose a base model and train with DPO

Step 7: compare the base model and the adapter on questions it never saw

Step 8: copy the adapter off, then destroy the instance

Privacy and access, in plain terms

Sources

Run your own model endpoint on a rented GPU with vLLM: your own API key, an SSH tunnel, and an optional LoRA adapter

Which kind of endpoint is this?

Read these things first

What you need

Step 1: make your endpoint key and look at your adapter (on your own machine)

Step 2: rent a raw GPU

Step 3: install vLLM in a fresh environment

Step 4: start the server on localhost, with your key

Step 5: open the tunnel and call your endpoint (on your own machine)

Step 6: stop, copy out what you need, destroy

Privacy and access, in plain terms

Sources

Let a small model critique and revise its own answers, then fine-tune on the revisions (on a rented GPU)

Where the idea comes from, and what this guide does not claim

Read these things first

What you need

Step 1: write the principles and the questions (on your own machine)

Step 2: rent a GPU

Step 3: install LLaMA-Factory

Step 4: generate, critique, revise

Step 5: filter and read a sample

Step 6: register the data and train a LoRA adapter

Step 7: compare on questions the model never saw

Step 8: copy the adapter off and destroy the instance

Privacy and access, in plain terms

Sources

Run Ollama on a per-minute GPU, then point your app at an OpenAI-compatible API

What you need

Step 1: Deploy the Ollama template

Step 2: Find the API endpoint

Step 3: Pull a model

Step 4: Smoke-test it with curl

Step 5: Call it with the OpenAI Python client

Want ComfyUI instead?

Destroy the instance

Troubleshooting

Generate a training set with DeepSeek, then fine-tune a small model on a rented GPU

Read these three things first

What you need

Step 1: generate question-and-answer pairs with DeepSeek

Step 2: filter and de-duplicate

Step 3: rent a GPU

Step 4: install LLaMA-Factory and register the dataset

Step 5: choose a base model and train a LoRA adapter

Step 6: try the adapter, copy it off, destroy the instance

Privacy and access, in plain terms

Sources

Wan 2.2 A14B at 720p on a 24 GB GPU: why it runs out of memory and what to change

1. The symptom and the conditions it was reported under

2. Why 24 GB is not enough here

3. Local settings to change

3a. Raise the working memory

3b. Keep partial loading on, and turn off the Windows sysmem fallback

3c. Try wan_memory_optimization (opt-in)

3d. Lower the resolution, and treat frame count with care

3e. Quantization, and the official script's flags

4. If local settings are still not enough: run the same workflow on a card with more VRAM

FAQ

Sources

Use DeepSeek V4.1 Flash, Kimi and GLM in RAGFlow with CloudGPU

What you need

Create an API key

Which models

Option A: the CloudGPU provider (builds that include the merge)

Option B: OpenAI-API-Compatible provider (use this until a tagged release includes CloudGPU)

3c. Try `wan_memory_optimization` (opt-in)

Step 2: Add the provider to `opencode.json`

Alternative: store the key with `/connect`

1. The model doesn't appear in `/models` (or requests fail with "model not found")

2. The base URL is missing `/v1`