DEV Community: Francis Oyakhire

Data Substrate Versus Vector Db Rag

Francis Oyakhire — Wed, 28 Oct 2026 15:00:15 +0000

This week's news about improved OCR capabilities for document images brought to mind a more fundamental question: how do we build systems that can reason about the data they process? When dealing with AI applications in regulated environments like Nigerian microfinance, the answer hinges on the distinction between a "data substrate" and vector databases used in RAG systems.

We're building a regtech tool for Nigerian microfinance banks that needs to analyze loan applications, detect fraud, and ensure compliance with evolving financial regulations. The data we work with is not just text - it's structured, versioned, and often comes with legal and audit implications. A vector database like Weaviate or Pinecone is useful for retrieving similar documents or answering questions based on unstructured content. But when we need to reason about the data - like verifying that a particular clause in a loan agreement was present in version 1.2 but not 1.3 - that's where a data substrate shines.

A data substrate is a versioned, citable, queryable layer that sits beneath the AI application. It's not a replacement for vector DBs, but a complement. For example, in our regtech tool, we use a data substrate to store and version all loan agreements, client data, and regulatory documents. Each version is immutable and can be queried for specific clauses, dates, or jurisdictions. This is crucial when auditors ask, "Was this term in the contract when the loan was approved?"

Here's how we model our data substrate using a PostgreSQL-based schema:

CREATE TABLE document_versions (
    id UUID PRIMARY KEY,
    document_hash TEXT NOT NULL,
    content JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL,
    version_number INT NOT NULL,
    document_type TEXT NOT NULL,
    metadata JSONB
);

Each document is stored as a JSONB blob, and we track its version number, creation timestamp, and metadata like the jurisdiction or regulation it relates to. This allows us to perform queries like:

SELECT * FROM document_versions
WHERE document_type = 'loan_agreement'
AND metadata->>'jurisdiction' = 'Nigeria'
AND metadata->>'regulation' = 'CBN 2023'
ORDER BY created_at DESC;

This approach gives us full traceability - a key requirement in regulated environments. A vector DB would struggle to provide this level of versioning and auditability without significant overhead.

But there are tradeoffs. A data substrate is not optimized for similarity searches or natural language queries. If we need to find all loan agreements that mention "collateral" in their content, we'd have to query the JSONB content field, which is less efficient than using a vector DB's similarity search. So, we use both: the data substrate for versioning and audit, and a vector DB for content-based retrieval.

What's next? We're exploring hybrid architectures that combine the best of both worlds - using the data substrate for versioned facts and the vector DB for semantic similarity. We're also experimenting with integrating this with on-device AI models, so that microfinance officers in remote parts of Nigeria can run compliance checks without relying on cloud infrastructure. How would you approach this problem if you were building for a similar regulated environment?

GPU Preflight For Cron Agents

Francis Oyakhire — Sun, 25 Oct 2026 15:00:15 +0000

This week, the open-source community buzzed with updates on model serving infrastructure, including the latest PRs for vLLM-Omni support. As teams race to deploy more efficient and scalable inference pipelines, one often-overlooked piece of the puzzle remains: ensuring that the hardware is ready before the job even starts. At Apex Grid, we’ve spent the last few months refining a two-stage GPU preflight check that has become a critical part of our cron agent infrastructure.

When deploying cron jobs that depend on GPU resources, it’s not enough to simply wrap the job in a try/except block. That approach risks silently failing when the underlying issue is not the job itself, but the state of the GPU. For example, another job might be monopolizing the card, or the kernel might be in a state where it’s not responding to requests. Our solution involves a two-stage preflight check that ensures both the GPU and the serving stack are in a healthy state before we even attempt to run the job.

Our preflight logic hinges on two simple but effective endpoints: one that checks for basic health (/api/tags) and another that performs a lightweight inference (/api/generate). The first check ensures that the model server is responsive and can return a list of available tags - a quick smoke test. The second check runs a minimal inference task, which helps us detect if the GPU is actually available and not being blocked by another job. We set a warmup timeout for this second check, typically between 3 and 10 seconds, depending on the model size and workload.

Here’s how our gpu_ready() helper looks in practice:

import requests
import time

def gpu_ready(base_url, warmup_timeout=5):
    # First check: basic health
    try:
        tags_response = requests.get(f"{base_url}/api/tags", timeout=3)
        if tags_response.status_code != 200:
            return False
    except requests.exceptions.RequestException:
        return False

    # Second check: warmup inference
    try:
        generate_response = requests.post(
            f"{base_url}/api/generate",
            json={"prompt": "Hello, world!", "max_tokens": 1},
            timeout=warmup_timeout
        )
        if generate_response.status_code != 200:
            return False
    except requests.exceptions.RequestException:
        return False

    return True

This approach has several advantages. It ensures that the GPU is not just available but also in a state where it can handle the job without stalling or timing out. It also allows us to cleanly skip the job if the preflight check fails, rather than letting it fail later and potentially leaving the system in an inconsistent state. This has been particularly useful in environments where multiple cron jobs share the same GPU resource pool.

Of course, there are tradeoffs. The preflight check adds a small amount of latency to the overall job scheduling process. Additionally, it requires that the model server exposes both the /api/tags and /api/generate endpoints, which may not be the case for all serving stacks. We’ve mitigated this by making the preflight logic configurable - allowing teams to skip the second stage if needed, or to use a different lightweight inference task.

Looking ahead, we’re exploring ways to make this preflight logic more adaptive. For example, we’re experimenting with using a lightweight model for the warmup inference that’s specifically tuned for this task, reducing the time it takes to complete the second stage. We’re also looking into integrating this logic more deeply with Kubernetes and other orchestration tools, to make GPU readiness checks part of the broader system health monitoring pipeline. What do you think - would you prefer a preflight check that’s even more lightweight, or one that gives more detailed diagnostics when it fails?

Ollama v1 OpenAI Compat Drops Think Toggle

Francis Oyakhire — Thu, 22 Oct 2026 15:05:38 +0000

The Problem

We recently ran into an issue with the Ollama /v1 endpoint, which is designed to be compatible with OpenAI's API. Our models, particularly those from the Qwen3 family, were returning empty content. This was not the behavior we expected, given the models' capabilities.

What We Noticed

We started by examining the requests and responses to identify what was going wrong. We noticed that the /v1 endpoint was silently dropping the think: false toggle for the Qwen3-family models. This toggle is crucial as it controls whether the model spends its entire num_predict budget on internal reasoning or not.

Here's a snippet of the payload we were sending:

{
  "model": "qwen3",
  "messages": [
    {"role": "user", "content": "Translate 'hello' to French."}
  ],
  "think": false
}

We expected the model to translate "hello" to French without extensive internal reasoning. However, the model was spending its entire num_predict budget on reasoning and returning empty content.

Our Approach

To fix this issue, we switched to the native /api/chat endpoint provided by Ollama. This endpoint allows us to have more control over the model's behavior and ensures that the think toggle is respected.

Here's the payload we used with the /api/chat endpoint:

{
  "model": "qwen3",
  "messages": [
    {"role": "user", "content": "Translate 'hello' to French."}
  ],
  "think": false
}

By switching to the /api/chat endpoint, we were able to control the model's behavior more effectively and get the expected output.

Tradeoffs

Switching to the /api/chat endpoint has its own set of tradeoffs. While it gives us more control over the model's behavior, it also requires us to refactor our code to handle the new endpoint. Additionally, the /api/chat endpoint may have different rate limits and pricing structures compared to the /v1 endpoint.

However, the benefits of having more control over the model's behavior and getting the expected output outweigh these tradeoffs for us.

What We're Doing Next

We are currently exploring other endpoints and features provided by Ollama to see if they offer any additional benefits for our use case. We are also working on improving our monitoring and alerting systems to catch such issues earlier.

Postiz Self Host Bluesky Python Client

Francis Oyakhire — Mon, 19 Oct 2026 15:00:14 +0000

This week, the conversation around Docker and auto-updating containers has sparked a lot of debate, especially around the tradeoffs between convenience and control. As engineers who've spent time wrestling with container orchestration, we’ve found that sometimes the simplest setups are the most reliable - and that includes self-hosting.

We’re running a self-hosted instance of Postiz, a self-hostable Bluesky client, using Docker Compose. The goal is to have a lightweight, zero-ops setup that can be managed directly from a Python script. This approach avoids the complexity of Kubernetes or managed services while still giving us full control over the deployment and API interactions.

Our Docker Compose setup is minimal, pulling the latest Postiz image from GitHub Packages and exposing the necessary ports. Here's a simplified version of our docker-compose.yml:

version: '3.8'
services:
  postiz:
    image: ghcr.io/postiz/postiz:latest
    ports:
      - "8080:8080"
    environment:
      POSTIZ_DATABASE_URL: "postgres://user:pass@postgres:5432/postiz"
    depends_on:
      - postgres
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: postiz
    volumes:
      - postgres_data:/var/lib/postgresql/data
volumes:
  postgres_data:

The Python client communicates with the Postiz instance via the /api/public/v1/posts endpoint. This endpoint supports both creating a new post and fetching a list of posts, depending on the HTTP method used. The JSON payload for a new post includes the text, visibility, and optionally a reply to another post. Here's the shape of the request body:

{
  "text": "This is a new post from the Python client.",
  "visibility": "public",
  "reply": {
    "uri": "at://example.com/0123456789abcdef0123456789abcdef"
  }
}

The Authorization header is required and must be a valid Bearer token, typically obtained via the /auth/token endpoint. This token is scoped to the user and provides access to the /api/public/v1/posts endpoint.

One of the quirks we've encountered is that the /api/public/v1/posts endpoint can return either a single post object or a list of posts, depending on the context of the request. This behavior is not always well-documented, so we've implemented a helper function in our Python client to normalize the response:

import requests

def fetch_posts(url, headers):
    response = requests.get(f"{url}/api/public/v1/posts", headers=headers)
    response.raise_for_status()
    data = response.json()
    return data if isinstance(data, list) else [data]

This function ensures that regardless of whether the API returns a single object or a list, our code can handle it consistently.

We’re currently working on extending the Python client to support more of Postiz’s API endpoints, including user management and moderation tools. We’re also exploring ways to integrate this setup with a local development environment that mirrors production as closely as possible.

What would you prioritize next if you were building a self-hosted Bluesky client?

Cron Scheduled Ollama Autonomous Agent

Francis Oyakhire — Tue, 13 Oct 2026 15:00:29 +0000

This week, the AI infrastructure space saw some eye-opening benchmarks, with modular systems showing dramatic performance gains over alternatives. Amid all the noise, we’ve been quietly iterating on a specific use case that sits at the intersection of autonomous agents and on-device AI: a cron-scheduled Ollama-based social agent that runs on our infrastructure and interacts with users across multiple channels.

We’re building a system where an autonomous agent generates content at regular intervals, processes it through multiple gates, and only publishes it if it passes all checks. The core of this system is Ollama, which we use for its lightweight footprint and ability to run large language models locally. To make this work, we’ve designed a modular architecture that separates concerns: voice profiles are stored as separate files, controversy is checked by a second model, and credit checks are made by querying a Postgres database. Rejected content is quarantined for later review.

Here’s how the publish pipeline looks in code. We’ve abstracted the main function as publish_to_all_channels, which is called by a cron job every 12 hours. The function is responsible for generating content, running it through the gates, and publishing if all checks pass:

from ollama import Client
import psycopg2
import os

def publish_to_all_channels():
    # Step 1: Generate content using Ollama
    client = Client()
    response = client.generate(model="llama3", prompt="Write a social media post about AI ethics")
    generated_content = response["response"].strip()

    # Step 2: Run through voice profile check
    if not check_voice_profile(generated_content):
        log_to_quarantine(generated_content, "voice_profile_failed")
        return

    # Step 3: Run through controversy gate using a second model
    controversy_client = Client(model="controversy-checker")
    controversy_result = controversy_client.generate(prompt=generated_content)
    if controversy_result["response"].lower() in ["controversial", "high_risk"]:
        log_to_quarantine(generated_content, "controversy_gate_failed")
        return

    # Step 4: Run through credit gate by querying Postiz DB
    try:
        conn = psycopg2.connect(
            dbname=os.getenv("POSTIZ_DB_NAME"),
            user=os.getenv("POSTIZ_DB_USER"),
            password=os.getenv("POSTIZ_DB_PASSWORD"),
            host=os.getenv("POSTIZ_DB_HOST"),
            port=os.getenv("POSTIZ_DB_PORT")
        )
        cur = conn.cursor()
        cur.execute("SELECT * FROM credits WHERE content = %s", (generated_content,))
        if not cur.fetchone():
            log_to_quarantine(generated_content, "credit_gate_failed")
            return
    except Exception as e:
        print(f"Database error: {e}")
        log_to_quarantine(generated_content, "database_error")
        return
    finally:
        if conn:
            conn.close()

    # Step 5: Publish the content to all channels
    publish_content(generated_content)

This approach lets us keep the system modular and scalable. Each gate is a self-contained function that can be swapped out or updated independently. For example, we could replace the controversy-checker model with a different one if needed, or even switch to a rule-based system for certain use cases. The quarantine system is equally important - it allows us to review content that failed any gate, and potentially reprocess it later if conditions change.

There are tradeoffs, of course. By using cron jobs, we’re introducing a fixed interval between runs, which may not be ideal for real-time use cases. Also, the current setup assumes a single content generation per run, which might not scale well for high-frequency publishing. We’re actively exploring ways to decouple the scheduling from the processing pipeline, and are looking into event-driven architectures that could allow for more dynamic content generation and publishing.

We’re also working on improving the performance of the controversy gate - the second model check is currently a bit slow, and we’re experimenting with model quantization and caching to reduce latency. What do you think is the right balance between accuracy and speed in this kind of gate? We’d love to hear your thoughts in the comments.

Publishing Pipeline Scheduled vs Immediate

Francis Oyakhire — Sat, 10 Oct 2026 15:00:28 +0000

This week, the AI world turned its gaze toward Beam, a new model that's stirring conversations about the next wave of generative AI. But while the world is talking about models, we're talking about the systems that bring them to life - specifically, how we balance the immediacy of content delivery with the strategic power of scheduling.

At Apex Grid, we're building a publishing pipeline that bridges the gap between immediate content delivery and scheduled publishing. Our platform, Postiz, handles both: scheduling social posts to hit user timezones precisely, and publishing long-form content on a cron-fire. The challenge? Ensuring consistency across channels, especially when dealing with canonical URLs.

The Problem: Consistency Across Channels

When content is published immediately, it's straightforward - the URL is fixed, and it's shared across all platforms. But when content is scheduled, the URL might not be available until the post is published. This creates a disconnect between the scheduled post (which may be shared on social media with a placeholder URL) and the final published article (which has its own canonical URL).

The result? A fragmented user experience, broken links, and a lack of trust in the content's integrity. For a technical audience, this is more than a UX issue - it's a data integrity problem.

Our Approach: Hybrid Publishing with Smart Routing

Our solution is a hybrid publishing model that leverages both scheduled and immediate publishing. For social media, we schedule posts based on user timezones using a cron-based system that respects local time zones. For long-form content, we publish immediately using a webhook-driven architecture.

The key lies in our internal routing system, which maps scheduled posts to their canonical URLs once the content is published. This system uses a lightweight event-driven architecture, where each scheduled post is associated with a content ID that's resolved at publish time.

Here’s a simplified version of our routing logic in Python:

from datetime import datetime, timezone
import pytz

def resolve_canonical_url(post_id):
    # Fetch the canonical URL from the database based on the post ID
    canonical_url = database.query("SELECT canonical_url FROM posts WHERE id = ?", (post_id,))
    return canonical_url[0] if canonical_url else None

def schedule_post(post_id, timezone_str):
    tz = pytz.timezone(timezone_str)
    now = datetime.now(timezone.utc)
    scheduled_time = now.astimezone(tz).replace(hour=10, minute=0, second=0, microsecond=0)
    scheduled_time_utc = scheduled_time.astimezone(timezone.utc)

    # Schedule the post to be published at scheduled_time_utc
    scheduler.add_job(
        publish_post,
        trigger='date',
        run_date=scheduled_time_utc,
        args=[post_id]
    )

def publish_post(post_id):
    canonical_url = resolve_canonical_url(post_id)
    if canonical_url:
        # Update the scheduled post with the canonical URL
        database.update("UPDATE scheduled_posts SET canonical_url = ? WHERE post_id = ?", (canonical_url, post_id))
        # Trigger the webhook to publish the long-form content
        webhook_client.send(f"https://api.postiz.com/webhooks/publish/{post_id}")
    else:
        # Handle error: no canonical URL found
        logger.error(f"Could not find canonical URL for post ID {post_id}")

Tradeoffs: Flexibility vs Complexity

This hybrid approach gives us the flexibility to schedule posts for optimal engagement while ensuring that all content is published with a consistent canonical URL. However, it adds complexity to our system - we now have to manage two different publishing workflows and ensure that the mapping between scheduled posts and canonical URLs is always accurate.

We also have to be careful with time zones and cron scheduling, which can be a pain point for any system that needs to respect local time. Our current solution uses the pytz library to handle time zone conversions, but we're always on the lookout for more robust, future-proof alternatives.

What's Next

We're currently working on a new feature that will allow users to preview scheduled posts with their canonical URLs before they're published. This will help reduce errors and improve the user experience for our content creators. We’re also exploring ways to unify the scheduling and publishing workflows into a single, more intuitive interface.

What do you think about the tradeoffs between scheduling and immediate publishing in content platforms? Have you faced similar challenges in your projects?

Controversy Gate Second Model Check

Francis Oyakhire — Wed, 07 Oct 2026 15:00:27 +0000

This week, Cloudflare introduced Clef, a new open-source decision model and fine-tuning platform that shifts the focus from just generating content to evaluating it. As the conversation around AI-generated content and its societal impact grows, we’re seeing a shift toward systems that don’t just create, but also evaluate what they create. At Apex Grid, we've been exploring a similar pattern in our work on autonomous social posting, where the risk of embarrassment or controversy is high.

The problem we're solving is straightforward: when an autonomous system generates content for public consumption - think social media posts, customer service replies, or even internal communications - it must be evaluated for potential risk before publication. The naive approach is to use a single model to both generate and evaluate content. But that’s a bad idea. The same model that’s incentivized to be creative might not be the best judge of its own output. Worse, it might be biased toward its own style, missing real-world nuance.

Our approach is to split the work between two models: one for drafting content, and another for scoring it for controversy or risk. The first model, let’s call it the generator, is optimized for fluency, creativity, and relevance. The second, the evaluator, is optimized for detecting offensive, controversial, or high-risk content. The evaluator is trained on a diverse dataset of annotated examples, and it returns a score that the system can use to decide whether to publish, edit, or reject the content.

Here’s a simplified version of what our system looks like in code:

from generator import Generator
from evaluator import Evaluator

def publish_content(prompt):
    generator = Generator(model="generator-v1")
    evaluator = Evaluator(model="evaluator-v1")

    draft = generator.generate(prompt)
    score = evaluator.score(draft)

    if score < 0.5:  # threshold for acceptable risk
        return draft
    else:
        return "Content flagged for review"

This two-model architecture has a number of benefits. First, it creates a clear separation of concerns - the generator doesn’t need to worry about being a judge, and the evaluator doesn’t need to be a creative writer. Second, it allows us to fine-tune each model independently, with different data and objectives. And third, it opens up the possibility of using specialized models for evaluation, such as those trained on legal, cultural, or ethical datasets.

But this approach is not without its tradeoffs. One major challenge is ensuring that the evaluator model isn’t too strict or too lenient. If it’s too strict, it might prevent legitimate content from being published. If it’s too lenient, it might allow harmful content to slip through. Balancing this requires careful calibration, and it’s an active area of research for us.

Another challenge is the computational overhead. Running two models in sequence increases latency and resource usage, which can be a concern in real-time systems. We’ve experimented with various optimizations, like model quantization and caching, to reduce this impact without sacrificing accuracy.

What’s next? We’re exploring ways to integrate real-time feedback from users into the evaluator model, so it can learn from the actual consequences of its decisions. We’re also looking into hybrid models that can both generate and evaluate content, but with internal mechanisms that prevent self-bias. And finally, we’re considering how to make this pattern more general - not just for social posting, but for any system where autonomous agents need to assess the risk of their own output.

How might you apply this two-model pattern in your own work? Are there use cases where a separate evaluator model is essential, and where it’s not worth the cost?

Data Substrate Versus Vector Db Rag

Francis Oyakhire — Sun, 04 Oct 2026 15:00:21 +0000

This week, news emerged about open-source tools that aim to reduce reliance on Nvidia's CUDA ecosystem, highlighting a broader shift in how AI infrastructure is being built. While the focus was on compute and communication libraries, we found ourselves thinking about the layers beneath - the data substrate that powers AI applications and the tradeoffs between vector databases and data substrates in real-world use cases.

At Apex Grid, we're building a regtech tool for Nigerian microfinance banks, where compliance and traceability are not just features - they're requirements. One of the core challenges we faced was ensuring that every AI inference could be traced back to a specific version of training data, with citable sources and audit logs. This is where a data substrate - a versioned, citable, and queryable reference layer for AI - becomes essential.

Vector databases and RAG (Retrieval-Augmented Generation) systems are powerful for specific use cases, like semantic search or chatbot memory. But they're not designed for the kind of traceability and version control that regulatory environments demand. For instance, a vector DB might store embeddings of documents, but it doesn't track which version of a document was used in a particular inference. RAG-over-files might pull data from a corpus, but it lacks the fine-grained metadata required for compliance audits.

To illustrate, here's a simplified example of how we model data in our regtech tool using a data substrate approach. We maintain a versioned dataset of loan applications, each with a unique identifier, timestamp, and provenance:

class LoanApplication:
    def __init__(self, application_id, timestamp, data, version, provenance):
        self.application_id = application_id
        self.timestamp = timestamp
        self.data = data
        self.version = version
        self.provenance = provenance

    def get_version(self):
        return self.version

    def get_provenance(self):
        return self.provenance

Each time a new version of a loan application is submitted, it's stored as a new entry in the dataset. When an AI model is trained or queried, it references the specific version and provenance of the data used. This ensures that any regulatory inquiry can be traced back to the exact data and version used in the inference.

Vector databases, on the other hand, are better suited for use cases where the primary concern is speed and similarity search. For instance, a customer support AI that needs to quickly find similar past interactions would benefit from a vector DB. But in our case, the need for traceability and auditability made a data substrate the right choice.

That said, there are tradeoffs. A data substrate requires more storage and careful design to manage versioning and provenance. It also introduces complexity in querying and indexing, especially when dealing with large datasets. Vector DBs are more performant for certain types of queries but lack the metadata richness of a data substrate.

We're now working on a hybrid approach: using a data substrate for core compliance and audit use cases, while leveraging vector DBs for parts of the system that require high-speed similarity search, like fraud detection. This allows us to maintain the traceability we need without sacrificing performance.

What's next? We're exploring how to integrate this hybrid model with on-device AI, ensuring that even when data is processed locally, it remains citable and traceable. We're also looking into how to make the data substrate more interoperable with existing AI frameworks - a challenge that's only going to become more pressing as the industry moves away from monolithic systems toward modular, auditable components.

GPU Preflight For Cron Agents

Francis Oyakhire — Thu, 01 Oct 2026 15:00:22 +0000

This week, the AI landscape saw a major leap with the release of a full-duplex voice model capable of real-time decision-making. While impressive, such models often rely on robust infrastructure to ensure consistent performance. At Apex Grid Technologies, we’ve been working on ensuring that every GPU-dependent cron job runs reliably, even in the face of unpredictable resource contention.

We build and run a fleet of cron agents that interface with GPU-accelerated workloads. These agents are responsible for training models, running inference, and managing data pipelines. But in a shared environment, where multiple jobs might be competing for the same GPU, a naive try/except block around a GPU operation is not enough. It can lead to silent failures, partial execution, and wasted compute hours.

To solve this, we’ve implemented a two-stage preflight check before any GPU-dependent cron job is allowed to proceed. The first stage checks whether the GPU is accessible and responsive. We send a lightweight request to /api/tags, which should return within 3 seconds. This helps us detect if the GPU is even online or if the service is unreachable. The second stage is a warmup check: we send a trivial inference request to /api/generate, which should return within a short timeout window (e.g., 10 seconds). This helps us detect if the GPU is being monopolized by another job, which would cause our warmup request to hang indefinitely.

Here’s how we implement this in our code:

import requests
import time

def gpu_ready(timeout=10):
    try:
        # Stage 1: Check if the GPU is accessible
        tags_response = requests.get("http://localhost:8000/api/tags", timeout=3)
        tags_response.raise_for_status()
    except requests.exceptions.RequestException:
        return False

    # Stage 2: Check if the GPU can handle a warmup request
    try:
        warmup_response = requests.post(
            "http://localhost:8000/api/generate",
            json={"prompt": "warmup", "max_tokens": 1},
            timeout=timeout
        )
        warmup_response.raise_for_status()
    except requests.exceptions.RequestException:
        return False

    return True

This function returns False if either of the checks fails. In such cases, the cron job cleanly skips execution and logs the failure. This avoids the common pitfall of trying to run a GPU job and then failing silently, which can be difficult to debug later.

There are tradeoffs to this approach. The preflight adds some latency to the cron job’s execution, as it requires two HTTP requests before any real work is done. However, this latency is minimal compared to the cost of running a job that will fail due to resource contention. Additionally, the preflight checks are not foolproof. For example, if the GPU is in a state of partial failure (e.g., one kernel is stalled but others are healthy), the preflight might not detect it. But in practice, this is rare, and the preflight is sufficient for most use cases.

Looking ahead, we’re exploring ways to make these preflight checks even more robust. One idea is to use GPU metrics from the host system (e.g., via NVIDIA’s nvidia-smi) to get a more direct view of the GPU’s state. We’re also considering integrating with Kubernetes and other orchestration systems to allow for more intelligent scheduling of GPU jobs. What do you think? Have you encountered similar issues with GPU contention in your own workflows?

Ollama v1 OpenAI Compat Drops Think Toggle

Francis Oyakhire — Mon, 28 Sep 2026 15:00:22 +0000

This week, the open-source community buzzed with excitement around LiveKit’s new WebRTC stack and cloud offerings. As developers and engineers, we’re always on the lookout for tools that simplify real-time communication and AI integration. But even with these advancements, we often run into subtle but impactful issues in the stack that require deep debugging. One such incident recently occurred in our work with Ollama and the Qwen3-family models.

We use Ollama’s /v1 endpoint to interface with multiple LLMs, including the Qwen3 series, in a production setting. Recently, we noticed that for certain models, the response content was consistently empty, despite the model being correctly invoked. This wasn’t a matter of timeouts or errors - it was a complete absence of output. After extensive logging and profiling, we traced the issue to a seemingly innocuous parameter: think: false.

Ollama’s /v1 endpoint is designed to be OpenAI-compatible, which makes it easy to integrate with existing tooling and workflows. However, we discovered that for the Qwen3-family models, the think: false toggle was being silently ignored. This meant that even when we explicitly set think: false, the model would still consume its entire num_predict budget on internal reasoning, leaving no tokens for the actual response. The result? Empty content, no errors, and a lot of confusion.

To confirm this, we compared the payloads sent to the /v1 endpoint versus the native /api/chat endpoint. Here’s a simplified version of what we sent to the /v1 endpoint:

{
  "model": "qwen3",
  "prompt": "What is the capital of France?",
  "options": {
    "num_predict": 100,
    "think": false
  }
}

And here’s what we sent to the native endpoint:

{
  "model": "qwen3",
  "prompt": "What is the capital of France?",
  "options": {
    "num_predict": 100
  }
}

The difference? The absence of the think: false toggle in the native endpoint. Surprisingly, the native endpoint didn’t require it to produce a response. This suggests that the /v1 endpoint’s compatibility layer was either misinterpreting or ignoring the think parameter for this specific model family.

The tradeoff here is clear: while the /v1 endpoint offers a familiar interface for those coming from the OpenAI ecosystem, it’s not always fully compatible with all models. In our case, it introduced a silent failure mode that was difficult to diagnose without deep logging and testing.

We’ve since moved our Qwen3-family integrations to the native /api/chat endpoint, where the behavior is predictable and consistent. However, this incident highlights a broader challenge: as we work with increasingly diverse model ecosystems, compatibility layers can introduce subtle, hard-to-detect issues.

Looking ahead, we’re exploring ways to build more robust, model-agnostic interfaces that can handle these edge cases without requiring users to switch endpoints. We’re also considering whether to contribute a patch or a workaround back to the Ollama community to improve the /v1 endpoint’s compatibility with Qwen3 and similar models. What do you think - should compatibility layers be more transparent about their limitations, or is it better to rely on native APIs for critical workflows?

Postiz Self Host Bluesky Python Client

Francis Oyakhire — Fri, 25 Sep 2026 15:00:23 +0000

This week’s discussions around AI sovereignty and open source reminded us of a core principle: control over your data and tools is not just a philosophical stance - it’s a technical necessity. As we continue to push for open, self-hosted systems, we found ourselves building a Python client that interacts with a self-hosted instance of Postiz, a Bluesky-compatible social network. Here’s how we made it work.

The Problem: Self-Hosting and API Inconsistencies

We wanted to run a fully self-hosted social network, with full control over data, infrastructure, and the user experience. Postiz, being a self-hosted alternative to Bluesky, fit the bill. However, the API it exposes is not always consistent - specifically, when fetching posts via /api/public/v1/posts, the response can be either a single object or a list, depending on the query parameters. This inconsistency required careful handling on our end.

Our goal was to build a Python client that could reliably interact with a Postiz instance running in Docker Compose, with minimal boilerplate and clear error handling. We also needed to manage authentication headers and parse the response structure dynamically.

Our Approach: Docker Compose + Python Client

We deployed Postiz using Docker Compose, which made it easy to spin up a local instance for development and testing. Here’s a simplified version of our docker-compose.yml:

version: '3'
services:
  postiz:
    image: postiz/postiz:latest
    ports:
      - "8080:8080"
    environment:
      - POSTIZ_DB_URL=postgres://user:pass@db:5432/postiz
    depends_on:
      - db
  db:
    image: postgres:15
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=postiz
    volumes:
      - postgres_data:/var/lib/postgresql/data
volumes:
  postgres_data:

With Postiz running locally, we built a Python client that sends requests to /api/public/v1/posts. The client handles authentication via an Authorization header, which Postiz expects in the format Bearer <token>. The token is obtained via the /api/auth/token endpoint, which we also implemented in our code.

Code: Handling API Responses Dynamically

Here’s a snippet from our Python client, which demonstrates how we handle the API’s inconsistent response structure:

import requests

def fetch_posts(postiz_url, auth_token):
    headers = {
        "Authorization": f"Bearer {auth_token}"
    }
    response = requests.get(f"{postiz_url}/api/public/v1/posts", headers=headers)
    response.raise_for_status()

    data = response.json()

    # Handle both single object and list responses
    if isinstance(data, dict) and "items" in data:
        return data["items"]
    elif isinstance(data, list):
        return data
    else:
        raise ValueError("Unexpected response structure from Postiz API")

This code ensures that regardless of whether the API returns a single item or a list, we can consistently work with a list of posts in our application. We also made sure to raise clear errors when the response structure is unexpected, which helps in debugging and maintaining the client over time.

Tradeoffs and Considerations

While this approach is simple and effective, it does come with tradeoffs. The dynamic handling of API responses adds a layer of complexity that could be avoided with a more consistent API. Additionally, managing authentication tokens and ensuring secure storage is a non-trivial task, especially in production environments.

We also found that the Postiz API lacks some features that we would expect from a modern social network, such as pagination and rate limiting support. These are things we plan to address in our own middleware layer.

What’s Next

We’re currently working on extending our client to support more endpoints, including user relationships and media uploads. We’re also exploring ways to integrate this with our on-device AI infrastructure, enabling local processing of social media data without relying on cloud services. How would you approach extending this client for a production environment? We’re eager to hear your thoughts.

Cron Scheduled Ollama Autonomous Agent

Francis Oyakhire — Tue, 22 Sep 2026 15:00:22 +0000

This week’s focus on benchmarking large language model (LLM) inference at scale has sparked renewed interest in how we can build autonomous agents that operate efficiently and responsibly on the edge. As we explore the intersection of on-device AI and autonomous systems, we’re seeing new patterns emerge in how we manage inference pipelines, especially when dealing with real-world constraints like latency, accuracy, and user safety.

At Apex Grid, we’ve been building a cron-scheduled autonomous agent using Ollama as the core inference engine. This agent is designed to operate on a periodic basis, generating and publishing content across multiple social channels. The system is structured with a clear separation of concerns, ensuring that each component - voice profiling, controversy checking, credit verification, and content quarantine - can be independently tested, scaled, and maintained.

The agent operates on a strict pipeline: first, it generates a voice profile from a user’s input, stored as a separate JSON file. This profile is used to personalize the output. Then, the generated content passes through a “controversy gate” - a second model that acts as a filter, scanning for potentially harmful or controversial content. Only after passing this gate does the content proceed to the “credit gate,” which performs a real-time probe against a Postgres database managed by Postiz. This step ensures that the agent only publishes content from verified users. If the content fails any of these checks, it is moved to a quarantine folder for later review or deletion.

Here’s what the publish_to_all_channels function looks like in our implementation:

from ollama import Client
import json
import os
import psycopg2
from datetime import datetime

def publish_to_all_channels(content, voice_profile_path):
    # Step 1: Load voice profile
    with open(voice_profile_path, 'r') as f:
        voice_profile = json.load(f)

    # Step 2: Controversy gate check
    controversy_model = Client(host='controversy-gate-model-endpoint')
    controversy_check = controversy_model.generate("Is this content controversial?", content)
    if controversy_check['response'].lower() in ['yes', 'maybe']:
        move_to_quarantine(content, voice_profile)
        return False

    # Step 3: Credit gate check
    try:
        conn = psycopg2.connect(
            dbname="postiz_db",
            user="user",
            password="password",
            host="postiz-db-endpoint"
        )
        cur = conn.cursor()
        cur.execute("SELECT verified FROM users WHERE content_hash = %s", (hash(content),))
        result = cur.fetchone()
        if not result or not result[0]:
            move_to_quarantine(content, voice_profile)
            return False
    except Exception as e:
        print(f"Credit gate error: {e}")
        move_to_quarantine(content, voice_profile)
        return False
    finally:
        if conn:
            conn.close()

    # Step 4: Publish content
    for channel in ["twitter", "telegram", "mastodon"]:
        publish_to_channel(channel, content, voice_profile)
    return True

This approach allows us to decouple the inference logic from the publishing logic, making the system more robust and easier to maintain. However, it’s not without tradeoffs. The use of a second model for controversy checking increases inference latency and cost. Additionally, the reliance on a Postgres probe for credit verification introduces a single point of failure if the database is unreachable or slow.

We’re actively exploring ways to reduce this latency by integrating lightweight on-device filtering models and implementing fallback strategies for database connectivity. We’re also evaluating whether we can replace the second model with a rule-based system for certain types of content, which could reduce both cost and complexity.

What we’re building next is a more distributed version of this agent, where each gate can be independently scaled and deployed across multiple nodes. We’re also looking into using a hybrid approach - combining rule-based checks for low-risk content with model-based checks for high-risk scenarios. We’d love to hear from the community: have you seen similar patterns in your autonomous agent pipelines, or have you found better ways to handle multi-model inference in production?