DEV Community: NEXT4I DEV

Semantic Caching and Live Internet Search: Tools That Let AI Cache Answers to Save Money, and Pull Data From the Web When It Needs To

NEXT4I DEV — Fri, 30 Oct 2026 18:31:43 +0000

Say you open the logs of your chat system and they are full of questions like these:

"How do I change my password?"
"Forgot my password, what do I do?"
"Where do I reset my password?"

All three mean the same thing. But the system sent each one to the AI, had it write a fresh answer three times, and paid for the tokens three times. Multiply that by thousands or tens of thousands of users, and the API bill gets... sigh.

At the same time, there is another kind of question the AI cannot answer from memory at all: "What's the gold price today?", "Is it going to rain today?", or "What promotions are on this month?" These answers change every day, and the data a model was trained on stops at a certain date, its knowledge cutoff. Anything new after that date, the model simply does not know.

This article covers two techniques, one for each side. Semantic caching lets an AI remember answers to questions that keep coming back with the same meaning, so you stop paying for them again. Live internet search lets the AI pull fresh data from the web, or a search engine, when a question needs it. And we will look at how to decide which questions should be "remembered" and which should be "looked up again."

The short version

Semantic caching keeps past answers. When a new question is close enough in meaning, it sends the old answer straight back, without calling the AI again and again.
If around 90% or more of your questions can be answered from the cache, you might save about 10x on tokens. The research I found reached roughly 60 to 70%, which works out to about 3x cheaper.
A cache has three main risks: a wrong answer because two questions looked like the same question, an outdated answer, and someone else's answer.
Live internet search lets an AI look up and answer current questions with sources. But each call costs more, because you also pay for the search API, and answers are slower because the system has to search and summarize first. Web pages it finds can also contain harmful instructions.
The two work together, as long as you have a step that sorts questions first: which ones can use the cache, which ones need a fresh search, and which ones must never share an answer with other users.

Think of your AI as a call-center agent with a notebook

A call-center agent who has been on the job for a while knows customers ask the same things over and over, so they write the answers down in a notebook. When someone calls and asks "I lost my card, what do I do?", they open the notebook and answer right away, without digging through the policy manual again. The idea here is pretty much the same.

For example:

The customer asks	What a good agent does	In an AI system
"I lost my card. What do I do?"	Opens the notebook and answers	Semantic cache
"What's the exchange rate today?"	Checks whether today's rate is already in the notebook. If not, looks at the live screen, then writes it down again	Live internet search + cache (per day)
"What's my outstanding balance?"	Never answers from another customer's notes; opens this customer's account	Fetch from the real system with permissions; no cache shared with other users
"I lost my credit card" vs. "I lost my debit card..."	Listens carefully first, because the steps may differ	Watch out for wrong matches (false hits)

What a cache is, and why it pays off with AI

A cache is a place where you keep results you have already worked out, so next time you can grab them instead of doing the work again. Websites and apps have used this technique for a long time.

With AI it pays off even more, because every time a model writes an answer, you pay per token and wait several seconds. Every word you send in and every word that comes back costs money, and output tokens cost more than input. If you want the background, I wrote an earlier article on tokens.

Three kinds of cache in the AI world

Exact-match cache: the whole question is the key, so it only hits when every character matches. "How do I change my password" and "How do I change my password?" count as two different questions.
Prompt caching from model providers: providers such as Anthropic and OpenAI remember the beginning of a prompt you send repeatedly, such as a long system prompt or the same set of documents, and charge less for that part. On many models, the part read from cache costs about 0.1x the normal input price. The catch: it has to match character for character from the start, and the model still writes a new answer every time, so you pay full price for output.
Semantic cache: you keep "question and answer" pairs on your own application side and compare questions by meaning. If the meaning is close enough, you send the old answer back and skip the model call.

Type	Matches on	Saves	Limitation
Exact-match cache	Identical text	The whole AI call	Type it slightly differently and it misses
Prompt caching	Identical prompt prefix	Input cost for the repeated part, and input processing time	You still pay for output and still wait for the model
Semantic cache	Similar meaning	The whole AI call	May answer in the wrong context; needs care with stale data and data permissions

When you see "10x cheaper," part of it usually comes from that 0.1x price on prompt caching. But that discount only covers repeated input. It is not 10x off the whole bill.

How semantic caching works

Most setups have four steps.

Turn the question into numbers that represent its meaning (an embedding). This is the same method as vector search. I covered it in my earlier article on hybrid search.
Search the cache for the old question closest in meaning.
Compare it against a similarity threshold. If it is closer than the threshold, that is a cache hit, and the old answer goes straight back.
If it does not reach the threshold, it is a miss. The question goes to the AI as usual, and the new question and answer pair is saved for next time.

The clearest win: on a hit, you pay nothing for the model and you do not wait for it, so users get their answer much faster. Turning the question into an embedding and searching the cache cost far less than having the AI answer.

So how much can you actually save?

To get a feel for it, let's do some simple math. If we ignore the cost of embeddings and the cache system itself, your AI cost comes down to the questions that miss.

Share of questions that hit	Still sent to the AI	Roughly cheaper by
30%	70%	1.4x
50%	50%	2x
69%	31%	3.2x
90%	10%	10x

To get 10x, about 90% of requests need to hit. That means almost every question has to repeat the meaning of one you have seen before, like a bot that answers FAQs.

The GPT Semantic Cache paper (Regmi and Pun, 2024) tested repetitive, customer-service-style questions. It got cache hit rates of about 61.6% to 68.8%, cut API calls by up to 68.8%, and over 97% of the answers served from cache were correct, by the authors' own measurement. Run that through the table above and it works out to roughly 3x cheaper. (More in the references at the end.)

Even 3x is a very good deal.

The risks of a semantic cache you need to know first

1. A wrong answer because two questions looked like the same question (false hit)

"How do I cancel my credit card?" and "How do I cancel my debit card?" differ by one word. Their embeddings may be very close, but the answers may be completely different. The same goes for sentences that differ only slightly, by a number, a date, or even a different context.

Redis's documentation says it plainly: semantic caching does not guarantee that a hit is always correct. The vCache paper (Schroeder et al., 2025) makes the same point from another angle: using one fixed similarity threshold for every question does not guarantee correctness either.

To lower the risk, start with a strict threshold, then loosen it step by step, and spot-check the answers that hit the cache along the way. Test with real questions in the languages your users actually write in, because every embedding model understands each language a bit differently.

2. An outdated answer (stale answer)

Some things have already changed, a promotion for example, but the cache keeps serving the old answer. The fix is to give each answer an expiry time (TTL) that matches how often the source data changes. Redis gives the example that real-time data like stock prices or weather should expire within 15 to 30 minutes, while FAQs that rarely change can be kept for days or weeks. And when a source document gets edited, clear the related cache entries right away.

3. Someone else's answer

This one is the most serious. If a customer asks "What's my outstanding balance?" and the answer gets saved into a shared cache, the next customer who asks something similar could get the first customer's answer. That is bad. Redis gives a multi-tenant example too: Company A's policy could end up answering Company B if the cache is not separated.

There is also research (Song et al., 2024) showing that by watching for responses that come back unusually fast, because they hit the cache, someone may be able to guess what other people asked. So a cache shared across users is a privacy issue as well, not only an accuracy issue.

The fix is to separate the cache by user, organization, or permission level, and to not cache answers built from personal data at all unless you really need to.

4. Questions where users want a new answer

Requests like "Help me come up with a caption" or "Give me name ideas for my shop" should get a fresh answer each time. Serving a repeat from the cache feels weird. GPTCache itself skips the cache more often when the temperature is set high.

Same question, same context

The same question asked at a different time or in a different context may need a different answer. So the cache key should be tied to its context, for example:

Model version
System prompt version
Language
The asker's permissions
Version of the documents used to answer (if you run a RAG system)

If any one of these changes, the old answer may no longer be up to date.

Live internet search: going out to find real data

Even with a RAG system that can search your company's documents (I wrote an article explaining RAG, if you want to go back to it), some questions still need the latest data: news, prices, newly issued regulations, or software versions. That information is neither inside the model nor in your internal documents.

A typical live internet search flow looks like this:

Decide whether this question needs a search
Turn the question into a search query, then call the search
Read the search results or the content of the relevant pages
Pick the passages that answer the question
Answer with sources

There are mostly two ways to do it: 1. use the web search tool your model provider builds in, or 2. use a separate search API service and wire up the steps yourself.

The limits of live search

Cost: search APIs charge per call, and results are long, so the tokens sent into the model are many times higher than for a normal question.
Speed: you wait for the search results, then load them into the model.
Source quality: results may come from unreliable sites, or the information may contradict itself.
Web pages may carry harmful instructions: the text on a web page is data we do not control.

To lower the risk, you can restrict domains for topics that need accuracy, for example only searching official sites or a whitelist of trusted ones, and show sources users can check. Cap the number of searches per question. And do not give an AI that reads the web permission to do important things at the same time, because it may end up running or doing something unsafe.

Sort first: cache it or search again?

Put a router in front that sorts the questions first.

Question type	Example	Best approach	Answer lifetime (TTL)
Data that rarely changes	"What is RAG?", "How do I change my password?"	Semantic cache	Long, days or weeks, and cleared when the documents change
Data that changes over time	"Gold price today", "Latest news on..."	Live search, then cache the search result briefly	Minutes, with the data's timestamp shown
Personal or permission-bound data	"What's my outstanding balance?"	Fetch from the real system with permissions	No shared cache, or a per-user cache
Design or creative work	"Help me come up with a caption"	Always let the AI answer fresh	No cache

Takeaway: cache what rarely changes, search what keeps updating, and be careful with caches shared between users

Semantic caching really can lower costs and make AI answer faster. But caching has to be checked carefully and tightly. You need to weigh the benefit against the risk for each kind of question. If the AI gives a wrong answer without having any idea it is wrong, the savings may not be worth it compared with simply calling the LLM or the API again.

Live internet search lets an AI look things up to answer current questions, but it costs both money and time. And the key point is that we cannot control anything about content that comes from the web, so we need a way to handle that too.

Once you have a solid step that sorts questions into "cache" or "search again," your AI starts working like a great call-center agent: quick to answer, remembers what rarely changes, checks or looks up new information when it makes sense, answers accurately, does not leak data, stays safe, and saves money. If you can get all of that, that is the complete system everyone wants.

References

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Originally published at:
https://www.next4i.com/dev-notes/en/semantic-caching-and-live-internet-search-dev-en

1.3M Changes in 10 Days !!! "N-Flow Agentic Inception": The NEXT4I Agentic Workflow

NEXT4I DEV — Fri, 23 Oct 2026 10:11:18 +0000

The NEXT4I Micro-batching Architecture for Enterprise Software Implementation

"I want to share the story and the stats from running an AI Agent workflow that I built and kept improving until it went into real use on an internal project. I gave it a cool name, "N-Flow Agentic Inception." It is NEXT4I's own internal AI workflow, built on the Micro-batching concept and running at Enterprise scale, with a workload of about 1,200 tasks. Right now I have the AI churning out work at about 5-7 tasks per hour.
It has turned me from 'a dev who sits and writes code' into 'a dev who runs an AI factory, checking that the production line is still healthy and reading the reports from a factory that runs 24 hours a day.'"

For several months I went through a lot of trial and error building a workflow with AI Agents to develop software as a Long running Implementation, using the Micro-batching Concept. The goal was to have AI Agents work continuously for 24 hours++, keep the cost worthwhile, stay on requirement without drifting off course, work across services within the scope of the work, test themselves, and produce evidence. I'll keep sharing all of this bit by bit until the work is done. Based on the stats, the AI estimates the whole run should take about 20 days.

Stats from real runs, last month (wave 1):

Speed & Scale: After a sweeping round of fixes (I'll write about the big failure in a later article), it started running: 30 Tickets in 4 days (not running all day). Tickets hit problems often. A batch could only get through 3-5 tasks before it couldn't go any further, and I had to keep going back to fix tickets.
Low-Accuracy: Tickets failed the Quality Gate very often. Some failed for real, and some failures came from the Quality Gate itself checking things that didn't make sense, such as expecting code coverage when there were no Unit Tests yet, or Ticket Dependencies that Blocked each other so nothing could move.
Trackable: The system committed code to Git on its own, in an orderly way. In the first batch, the models closing tasks were gpt-5.6-sol (main), gpt-5.6-luna and gpt-6-astra.
The stats weren't interesting yet, and I had to fix things so often that multi-hour continuous runs were nowhere near. So I decided to tear the workflow down and rebuild it again.

Stats from real runs, 7 days ago (wave 2):

Speed & Scale: Speed kept scaling up from the very slow first wave. It now does 3.5-7 tickets per hour and has cleared about 11.7% of the Epic (146 of 1,243 tickets).
Run-to-run variation: This wave ran about 4-5 times. Some services were slower, or the loop stopped often, dropping speed to 2-4 tickets per hour. Continuous runs lasted about 3 hours, reaching 6 hours on some runs.
100% Accuracy: Among the tickets that completed, every one passed the Quality Gate on the first try.
Trackable: The system still commits code to Git and clearly records which Model finished each Task (e.g. [sonnet], [opus]).

Stats from real runs after another Workflow Refactor (wave 3):

Speed & Scale: This wave does 5-7 tickets per hour, and 43.4% of all the work is now cleared.
Running 100% on Local Claude Code, it closed 109 Tasks in the first round, using 28% of the Weekly Quota on Claude Max 5x, with a 99% Cache hit rate and 54% Context.
Then, while I was at the weekly quota limit, I used the remaining 197 USD of Free Claude Cloud credit while waiting for the weekly quota reset, and closed another 82 tickets, averaging 2.4 USD per Task. (Most of them were done by sonnet, 78 tasks, and opus picked up 4 tasks.)

Project Estimate

A project at this scale might normally need a Dev team of about 12 people (Backend 8-9, Frontend 2-3, DevOps 1-2) running for about 12 months. 1,243 tasks in total, about 3,500 Man-days (estimated by Claude Sonnet 5.5). At the current speed, the system projects it will finish in about 20 days (dev time only).

What is N-Flow Agentic Inception?

The project is enterprise-scale software that NEXT4I is building for itself. The work crosses several services and has been broken down into 1,243 tasks. (I use task, ticket and work order interchangeably here.)

The core idea is micro-batching. Instead of handing the AI one huge job, you cut the work into small pieces with a clear scope and run them in small batches. Each piece has to pass a check before the next one starts.

Here is how it actually runs:

Step 0, human: I write the requirements and refine them together with AI, using Claude, GPT-SOL and Gemini as advisors. This stage took about five to six months in total.
Step 1, planner AI: turns the requirements into tasks. Each task has a goal, things it must not do, guardrails, the tests that will verify it, and a complexity level. A set of unit tests then checks that the work orders themselves are complete.
Step 2, worker AI: picks up a work order. The system chooses a model based on the task's complexity. Some tasks are implementation work that produce code; others are test work that produce evidence.
Step 3, Quality Gate: checks each result. Only work that passes gets committed to Git, tagged with the model that finished it, such as [sonnet] or [opus].
Step 4, when it fails: retry, or escalate to a stronger model. There are human-in-the-loop checkpoints at important states so I can step in and review.

Requirements (human + AI advisors)
        │
        ▼
Planner AI ──► 1,243 work orders ──► unit tests check the work orders
        │
        ▼
Worker AI (model chosen by complexity)
        │
        ▼
Quality Gate ──► pass ──► commit + model tag ──► next task
        │
        └──► fail ──► retry / escalate to stronger model / human review

What happens when the AI gets something wrong?

As for failures, here is a stretch of log I really like:

Ticket 2: a script hit sed filter broke. Instead of stopping, the system re-checked the state, saw the work was actually done, and let it through.
Tickets 3 and 4: came back as DONE_WITH_CONCERNS. The system treated that as "not passed" and escalated from Sonnet to Opus automatically.
Opus on Ticket 3: did not start rewriting code. It concluded "no code change needed, just mark it DONE," citing earlier MCP tickets that had hit the same situation as precedent.

In factory terms, line workers (Sonnet) do most of the work, and a QC lead (Opus) only gets called in when something looks off. That matches Wave 3, where Opus handled just 4 of 82 tickets.

What I learned from the earlier rounds

The classic problem with having AI write code at scale is that we "give instructions that are too broad." The AI guesses wrong (Hallucinates), burns through Tokens, and we end up fixing things ourselves. On top of that, the cost was very high: I was using the LLM API with tools looping way too much, tokens ballooned, and the cost spiraled. It took a long time to tune, and I took a lot of pain from it myself.

So I rebuilt it around the Micro-batching concept and called it "N-Flow Agentic Inception" (AI assigns work to AI, and AI checks AI's work). The process is:

Planner AI: Takes the Requirements I designed and breaks them down into tasks with a clear Scope.
Worker AI: Takes the "work order" and runs it. The system automatically picks the Model that fits the complexity of the task, writes code and Tests until it passes the Quality Gate, then Commits the work. These AIs handle both implement tickets, where the output is code, and test tickets, where the output is evidence.

As a result, I monitor the AI's tasks less and less. After all the tuning and fixing, it has become fairly stable. I used to check the AI's progress Report every 1-3 hours, and sometimes a run would get through 50-80 tasks and then hit a problem. Now I can run long stretches of 100 tasks++, which take about 24 hours in total, and during the day I only look in once in a while. One day uses about 25-30% of the weekly quota, so I've now had to upgrade to Max 20x to be able to run the full week, since my Free Cloud credit is used up. And the stat from the latest 24 hours of runs: not a single Task has had to Escalate back to me to fix.

Summary

Breaking my back on Task Decomposition to close the Garbage In, Garbage Out gap means the system no longer bottlenecks on us. Our job is to finish the requirements, make the work clear, and then let the AI do the long runs.

My first goal for this architecture was to save cost and speed up the project. By now I think it's getting good enough that I dare to leave it running, get up from the computer, and spend my time on other things without worrying, while my NEXT4I Factory keeps churning out code like an automated production line.

As for how things turn out from here until the work is done, keep following along on this channel.

Welcome to the era of Agentic Workflow. 🚀

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Originally published at:
https://www.next4i.com/journey/en/next4i-agentic-workflow-01-en

Hybrid Search Explained: Why Vector Search Still Needs Keywords

NEXT4I DEV — Thu, 15 Oct 2026 08:55:07 +0000

“Which air purifier models support the AF-240B replacement filter?”

That question needs evidence about AF-240B, not another filter with a similar name. A result can be about the right topic and still be about the wrong part.

Now consider: “How do I get my air purifier working properly again?” The answer might sit under “Replacing the filter,” without using the wording in the question.

These are different retrieval problems. Hybrid search combines semantic and keyword retrieval so the system has a better chance of finding both the intended meaning and the specific terms that matter. It does not make every result correct by default.

The short version

Vector search helps with paraphrases and related meaning.
Keyword search helps with terms, names, and identifiers present in the content.
Hybrid search combines their candidate lists and ranks the results.
Exact identifiers need explicit matching rules, not just relevance scores.
Permissions, versions, chunk quality, and evaluation remain separate responsibilities.

What vector search does well

An embedding model turns text into a vector: a list of numbers representing learned features of the text. Vector search compares the question's representation with stored content representations using a chosen similarity measure.

Suppose a handbook says:

“Employees may work from their residence up to two days per week.”

A user asks:

“How many days can I work from home?”

A suitable embedding model may connect those phrases even though the wording differs. This is useful when people do not know the vocabulary used in a document.

“May” matters here. Language coverage, model quality, and the indexed content all affect the result.

Embedding generation is also separate from answer generation. Vector retrieval does not inherently ask a generative LLM to read the whole collection for every query. Document vectors are commonly prepared ahead of time, then compared with a query vector at search time.

Where similar meaning becomes a problem

Identifiers are not interchangeable

Compare AF-240B with AF-2408, or X100 with X100 Pro.

Their surrounding descriptions could be almost identical. Yet the products may have different compatibility requirements. An embedding model can learn patterns involving identifiers, but vector similarity is not an exact-match guarantee.

The same issue applies to invoice numbers, serial numbers, API names, error codes, contract clauses, dates, and software versions such as 2.10 versus 2.1.

If a value must match, I would make that requirement explicit rather than leave it to semantic ranking.

The right passage may not reach the model

Systems usually pass only a limited set of results to the answering model. A relevant passage outside that cutoff is missing evidence, even if it exists in the collection.

Chunk boundaries, embedding quality, query wording, language, and candidate depth affect which passages survive. A stronger answer model cannot reliably compensate for evidence it never receives.

What keyword search adds

Keyword, or lexical, search works with terms found in the indexed text. Search engines commonly use an inverted index and ranking methods such as BM25. BM25 considers factors including term frequency, how uncommon a term is across the collection, and document length.

This gives a query term like AF-240B a direct role in retrieval. But lexical search is not automatically exact matching.

A text analyzer may split punctuation, lowercase text, or turn an identifier into several tokens. If a complete product code must match, use a field and query type that preserve and enforce the intended value. Define any normalization rules deliberately.

Keyword search has its own weakness: users may describe a need with words that never appear in the document. “Time off to look after my child” might need a passage labeled “Family care leave.” Semantic retrieval can help bridge that wording difference.

Neither method replaces the other in every collection.

A practical hybrid retrieval flow

At a conceptual level:

Question + user context + document constraints
                    |
           Authorized search scope
                    |
         +----------+----------+
         |                     |
    Vector search         Keyword / BM25
         |                     |
         +----------+----------+
                    |
        Fuse rankings and deduplicate IDs
                    |
           Optional reranking
                    |
  Validate access, versions, and evidence coverage
                    |
      Supply permitted evidence to the LLM

This is an explanatory design, not NEXT4I's internal architecture. Access control must cover every content-fetching path, including any external reranker. Hiding an answer afterward is not a substitute for keeping unauthorized text out of model context.

For the filter question, vector retrieval might find maintenance guidance while lexical retrieval finds passages containing the part code. The combined result still needs evidence of model compatibility. Merely mentioning the code is not enough.

Combining results without being misled by scores

BM25 and vector similarity scores do not share a universal scale. Adding them directly can let the numerically larger scale dominate, without proving that its results are more relevant.

Normalize and weight scores

One approach transforms each source's scores into a chosen scale, then combines them with weights. This offers control, but score distributions can vary across questions and collections.

Increasing the lexical weight for code-heavy queries may be useful. It still is not equivalent to enforcing a mandatory identifier constraint.

Fuse ranks with RRF

Reciprocal Rank Fusion uses positions in ranked lists rather than comparing their raw scores:

RRF(document) = sum of 1 / (c + rank(document))

Ranks start at 1. A document absent from a list receives no contribution from that list. The constant c controls how sharply rank positions affect the contribution; it is separate from the number of candidates retrieved.

A document ranked well by multiple retrievers can rise in the fused list. RRF does not check truth, policy applicability, or access rights. It is also different from PageRank, which uses a web link graph.

Reranking is another option: a model evaluates the relationship between the question and candidate passages. It can help reorder close results, at the cost of additional work and latency. It cannot recover a passage that never entered the candidate set.

A brief note on vector precision

Precision affects resource use, but it is not the main reason to use hybrid retrieval.

A float32 value uses four bytes and has roughly seven significant decimal digits of precision, not seven fixed places after the decimal point. At the same dimensionality, float16 and bfloat16 use two bytes per value. Quantizing values to int8 uses one byte per value.

Those ratios describe vector values, not the total database footprint. Indexes, metadata, quantization parameters, and retained original vectors also take space. The two 16-bit formats have different precision and range characteristics.

Quality loss and speed gains depend on the model, collection, hardware, and search configuration. I would not attach a universal accuracy-loss percentage to quantization. Check whether important evidence remains retrievable at the actual cutoff you use.

Evaluate the whole evidence path

Compare vector-only, keyword-only, and hybrid retrieval using the same labeled questions. Keep separate results for paraphrases, exact identifiers, policy conditions, and other query types.

Recall@k: What fraction of known relevant evidence appears in the first k results?
Hit@k: Does at least one relevant item appear? This is not the same as recall when multiple items matter.
MRR or nDCG: Is useful evidence near the top, and does the ordering reflect relevance?
Evidence completeness: Are the required conditions and exceptions included?
Answer correctness and grounding: Does the answer follow from the retrieved text, with supporting citations?
Latency and cost: Does the improvement justify the extra retrieval and ranking work?
Access and version handling: Does forbidden or obsolete content stay out?

Include questions with no answer in the collection and questions whose supporting documents the user cannot access. Returning nothing can be the correct outcome.

Better retrieval metrics do not automatically establish better final answers. Test both.

The goal is evidence, not a smarter-looking stack

Vector search helps when users and documents use different words. Keyword search helps when specific terms matter. Hybrid search gives the system two ways to find candidates, then asks you to combine them responsibly.

I would start with questions the system currently gets wrong and trace where their evidence disappears: retrieval, fusion, or context assembly. That is a more useful starting point than changing the answer model without inspecting what it can read.

Chunking in RAG: How to Split Documents Without Losing the Conditions

NEXT4I DEV — Wed, 07 Oct 2026 18:27:50 +0000

Your AI assistant has the employee benefits handbook. You ask: “Can someone who is still on probation claim the eyewear benefit?”

It confidently answers: “Yes, up to 3,000 baht per year.” But the handbook says the benefit is only for permanent employees who have completed probation. The limit reached the model. The eligibility condition did not.

That failure can start before generation, at the document boundary. Chunking in RAG is not about making text as small as possible. It is about creating searchable pieces without separating facts from the conditions that make them true.

The policy and amounts here are fictional examples, not NEXT4I or customer policies.

TL;DR

A chunk is a piece of source content used for retrieval. It is not inherently a vector.
Large chunks can mix unrelated topics. Small chunks can lose subjects, conditions, and exceptions.
Start with document structure, cap oversized sections, and test with real questions.
Overlap and parent-child retrieval can help with missing context. They cannot repair incorrectly extracted text.
Check retrieved evidence, context completeness, and answer correctness separately.

What is chunking, and why not use the entire file?

Chunking divides a document into pieces that can be indexed and retrieved. A simplified pipeline looks like this:

Document -> Extract and organize text -> Create chunks
         -> Store text and source references -> Build an index

Question -> Retrieve relevant chunks -> Assemble context -> LLM answer with sources

Retrieval may use keyword search, vector search, or both. Not every chunk needs an embedding. When vector search is used, an embedding model creates a numerical representation of the content.

The searchable unit also does not have to be the final reading unit. A system can retrieve a short passage, then supply its surrounding section to the model.

A single chunk containing a 100-page handbook mixes many topics. That can make specific passages harder to retrieve and exceed model input limits. Sending unrelated material also costs tokens and can distract from the evidence that matters.

Whole-document context can work for short documents. Tiny chunks are not automatically better: an amount without its eligibility rule is incomplete evidence.

One boundary can change the answer

Consider this fictional passage:

Section 8.1: Prescription eyewear benefit
The company reimburses up to 3,000 baht per year.
Only permanent employees who have completed probation are eligible.

A length-based split might produce:

Chunk A: The company reimburses up to 3,000 baht per year.
Chunk B: Only permanent employees who have completed probation are eligible.

Without the heading, both pieces lose their subject. B never names the benefit, and vector search does not guarantee recovery of that missing relationship.

If only A reaches the LLM, it has evidence about the amount, not enough evidence about eligibility. A prompt telling it not to invent information may help it acknowledge uncertainty. It cannot restore a sentence that was never supplied.

Keeping the heading, amount, and condition together helps make evidence usable. It does not guarantee a correct answer.

The main approaches and their trade-offs

These approaches can be combined. Splitting by heading and then dividing oversized sections is often more useful than committing to one method everywhere.

1. Fixed-size splitting

Choose a length, such as 500 tokens, and divide the text into consecutive pieces. It is straightforward, but the boundary may cut through a sentence, table, or exception.

Use the actual model tokenizer when token limits matter. Character counts are not interchangeable with token counts, and a Thai word is not necessarily one token.

2. Fixed-size splitting with overlap

Overlap repeats some text from one chunk in the next, giving material near the boundary a chance to stay together.

It cannot fix distant conditions or scrambled reading order. Thai OCR with unreliable spacing still needs boundary checks.

More overlap means more indexed content and duplication. Near-identical results can occupy retrieval slots that should contain other evidence. Deduplicate or merge overlapping context before sending it to the LLM where appropriate.

3. Recursive splitting

A recursive splitter tries separators in a chosen order, such as paragraphs, sentences, then finer boundaries, until pieces fit the size limit.

It avoids unnecessary cuts through sentences, but does not understand meaning. A paragraph can cover several topics; damaged OCR can remove boundaries.

4. Structure-aware splitting

Use headings, lists, tables, and code blocks to guide boundaries. A subsection can carry its parent heading so a short passage still has a clear subject.

Tables need special care. An isolated “3,000” is not enough. Preserve the row name, column name, and unit. In our fictional example, “Prescription eyewear: annual reimbursement limit of 3,000 baht” is clearer. Any eligibility note must accompany it too.

Code and ordered requirements have similar dependencies. If an oversized block must be split, preserve its relationships and necessary conditions.

5. Semantic chunking

Semantic methods estimate where nearby sentences or paragraphs change topic, often using embeddings or a model.

They can help when structure is unclear, but add processing cost and tuning. Results depend on the model and source quality. If headings already separate topics well, I would test that simpler structure first.

6. Parent-child retrieval and contextual chunking

These solve different problems alongside the splitting methods above.

Parent-child retrieval indexes small child passages for search, then expands a match to a larger parent section for reading. Maintain the links, control context size, and check access permissions for the expanded parent too.

Contextual chunking adds useful context before indexing, such as a document title, section name, or short source-grounded explanation. If an LLM generates that explanation, check that it has not invented conditions. Keep original text available for citations instead of treating generated context as source evidence.

Propositional or agentic approaches may use an LLM to extract standalone statements or choose content groupings. Implementations vary. The shared trade-offs are extra cost and the risk that rewriting drops a qualification. Compare them with a non-LLM baseline and retain the original source.

Match the approach to the document

Document	A reasonable first approach	Check carefully
Markdown or structured manuals	Split by heading, subdivide oversized sections	Parent headings, tables, footnotes
Plain text with inconsistent structure	Recursive splitting, limited overlap	Conditions at boundaries
Contracts or rules requiring several paragraphs	Retrieve small passages, expand or combine evidence	Clause numbers, versions, permissions
Scanned or multi-column PDFs	Repair extraction and reading order first	OCR errors, scrambled columns, broken tables

These are starting points, not substitutes for inspecting actual documents.

The Thai tourism PDF that changed my priorities

While experimenting with a RAG pipeline, I used a beautifully designed tourism PDF about Thailand. The extracted text was much less attractive: Thai vowels and tone marks were misplaced, while colored backgrounds, watermarks, and illustrations interfered with reading.

For some pages, I tried black-and-white conversion to make the letters clearer. That could also lose visual context. I compared outputs with help from multiple models and had a person review spelling again.

I described this in my earlier PDF and Markdown write-up, in Thai. It is one experience, not evidence that every PDF needs the same workflow.

The lesson was simple: changing chunk size cannot bring back words that extraction has already lost.

Markdown makes structure easier to identify, not evaluation unnecessary. Sections can still be oversized, and table rows need headers.

How large should chunks and overlap be?

There is no universal setting. 300-500 tokens with 10-20% overlap can be an initial experiment for paragraph-based text. Those numbers are not a benchmark result from my system or a standard to apply everywhere.

Before tuning, check four things:

Meaning: Does each piece identify the subject and preserve necessary conditions?
Limits: Use the actual tokenizer and budget for instructions, the question, all retrieved context, and the answer.
Duplication: Does overlap add useful evidence or just repeat it?
Question type: A lookup for an amount and a question about eligibility may require different evidence.

Start with structure and a size cap. Add overlap or parent expansion when tests reveal missing context, not simply because more context feels safer.

How to tell whether chunking works

Build questions with known supporting documents, sections, or pages. Include boundary-sensitive questions, amount lookups, and questions the documents cannot answer.

Compare two or three approaches on the same sources and questions with comparable context budgets. Then inspect three layers:

Retrieval: Are the labeled supporting passages found? Recall@k measures retrieval of predefined relevant evidence in the top k results. Separately check whether the evidence available is sufficient for the question.
Context: Are conditions and exceptions present? Do duplicates crowd out other evidence? Are table headers and units intact?
Answer: Does it follow the evidence? Does each citation support the associated claim? Can the system acknowledge insufficient information?

Store metadata such as document_id, section, page/offset, and version so passages can be traced and stale index entries managed.

For organizational data, enforce permissions before content reaches the model, including any expanded parent context. Hiding the final answer is not a substitute for controlling what the model receives.

If source text is wrong, fix ingestion. If evidence exists but is not found, inspect chunking and retrieval. If complete evidence reaches the model but the answer is wrong, inspect context assembly and generation.

The goal is usable evidence

A RAG system generally supplies selected evidence, not the entire library, for each question. Chunk boundaries help determine what the model gets to see, though they are not the only factor in answer quality.

My starting point is accurate source text, structure-aware boundaries, sensible size limits, source references, and tests with real questions. Add complexity where those tests show a need.

The best chunk is not necessarily the smallest one. It is a piece the system can find and use without losing what makes the answer valid.

If you are building RAG, share which document types cause missing conditions and where in the pipeline you catch them.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Originally published at:
https://www.next4i.com/dev-notes/en/chunking-in-rag-dev-en

From Ctrl + F to AI-Powered Search

NEXT4I DEV — Wed, 30 Sep 2026 18:11:58 +0000

How Do Computers Find Content and Answer Questions from Files?

(A Deep Dive into Search & Retrieval Fundamentals: From Ctrl+F to AI Semantic Search)

In the daily work of modern tech teams and knowledge workers, there is a classic problem that never really goes away:

"We know this information exists somewhere, but we have no idea which file, which page, or which line it is on."

In the past, we relied on Ctrl + F to search through documents one file at a time.
Later, search servers and file search tools introduced indexing, allowing us to find file names and keywords much faster across storage systems.
Today, in the era of Generative AI, we can simply ask a question in a chat interface: "Summarize the vendor contract for me: what is the late delivery penalty?" And within seconds, the system digs through a 200-page document and extracts the exact numbers and contractual conditions.

Behind the scenes, though, how does a computer actually "search" and "understand" the content inside our files?

Why does traditional keyword search frequently miss critical answers? And why do modern search engines seem to understand what we mean, even when our queries share zero words with the source document?

This article walks through the engineering foundations of Search & Retrieval, from brute-force byte scans to vector embeddings and AI-powered synthesis. It provides the essential mental model you need before diving deeper into Retrieval-Augmented Generation (RAG).

1. The Fundamental Gap: Humans Read Meaning, Computers Read Bytes

Before discussing search algorithms, we need to understand the fundamental limitation of computers:

Humans: When we read the phrase "an employee is unwell", we instantly connect it with "sick leave", "medical certificate", "hospital", or "health insurance benefits".
Computers: At a baseline level, the phrase "sick leave" is nothing more than an array of UTF-8 bytes. The computer does not inherently know that this sequence of bytes relates to physical health, time off work, or payroll deductions.

This gap created two distinct eras of information retrieval:

Lexical Search: Cares strictly about character matching: "Are the words spelled the same way?"
Semantic Search: Cares about conceptual intent: "Are the meaning and context aligned?"

2. Era 1: Exact Character Matching (Lexical / Exact Keyword Search)

This is where search technology started. In production environments, it primarily takes two forms:

2.1 Linear Scan (`Ctrl + F` or `grep`)

How it works: When a user searches for loan interest, the system opens the file from the very first byte and iterates through every line sequentially until it reaches the end.
Complexity: $O(N)$, where $N$ is the total number of characters or bytes in the dataset.
Advantages:
- Real-time accuracy: if you edit a file, the next search reflects the update immediately without an indexing delay.
- Zero storage overhead: no dedicated search database or pre-computed structures are needed.
Disadvantages:
- Does not scale: If you have 10,000 files with hundreds of millions of words, opening and scanning every file sequentially will quickly exhaust CPU cycles and disk I/O.

2.2 Inverted Index (The Backbone of Search Engines)

To eliminate the bottleneck of linear scans, computer scientists developed the Inverted Index, which powers engines like Elasticsearch, Apache Lucene, and OpenSearch:

How it works: Instead of waiting for a query before opening files, the system analyzes and indexes documents ahead of time (Index Time). The indexing pipeline reads the documents, splits text into tokens (Tokenization), removes irrelevant filler words (Stopwords), and records a lookup table: "Which files and positions contain this specific word?", much like the index at the back of a textbook.

[ Source Documents ]
File_A: "Employee is entitled to sick leave for 30 days"
File_B: "Submitting sick leave requires a medical certificate"

[ Inverted Index Table ]
Term                 │ Posting List (Occurrences)
─────────────────────┼─────────────────────────────────────────────
certificate          │ File_B (pos 7)
days                 │ File_A (pos 8)
employee             │ File_A (pos 1)
leave                │ File_A (pos 5), File_B (pos 3)
medical              │ File_B (pos 6)
sick                 │ File_A (pos 4), File_B (pos 2)

At query time (Query Time): When a user searches for "sick leave", the system does not rescan File A or File B from scratch. It directly looks up the table and returns matching documents in sub-millisecond time ($O(1)$ or $O(\log M)$, where $M$ is the vocabulary size of the index).

Limitations of Lexical Search: Why Exact Matching Falls Short

While inverted indexes are fast and power production systems through algorithms like BM25, exact character matching suffers from three inherent language barriers:

The Synonym Problem: If a policy document says "Rules for absence while undergoing medical treatment" but the user searches for "how to take sick leave", the system returns zero results, even though the content is a 100% match in meaning.
The Polysemy Problem (Word Ambiguity): The same word can carry completely different meanings depending on context. For example, searching for "crane" might return documents about construction machinery, a bird species, or an industrial pulley system without distinction.
Typos & Natural Questions: A single typo can break a match. Furthermore, when users ask conversational questions like "If someone gets burned in the cafeteria kitchen, who handles the medical claim?", tokenizers break the query into fragmented words, which distorts keyword relevance scores.

3. Era 2: Searching by Meaning (Semantic / Vector Search)

To overcome the limitations of spelling, modern AI leverages deep learning to create Text Embeddings and Vector Search.

In early hands-on experiments, when seeding test data with different wording, paraphrased sentences, or even cross-language queries, running a top-k similarity search using embedding models like BAAI/bge-m3 consistently retrieved documents with matching intent, despite having zero exact word overlap.

3.1 What is Text Embedding? (Turning Language into Coordinates)

For those interested in exploring how text embeddings power Retrieval-Augmented Generation, check out NEXT4I's Guide on RAG.

An embedding model (such as OpenAI's text-embedding-3, Cohere Embed, or open-source models like BAAI/bge) takes text passages as input and maps them into a high-dimensional dense vector, such as 768 or 1,536 dimensions.

The core principle:

"Passages with similar meaning or shared context end up close together as clusters in this multi-dimensional vector space."

                     [ Semantic Cluster: Illness & Healthcare ]
                     • "Employee is feeling sick"
                     • "Medical leave guidelines"
                     • "Healthcare expense claims"

  [ Semantic Cluster: Finance ]              [ Semantic Cluster: Time Off ]
  • "Q3 balance sheet review"                • "Submitting annual leave"
  • "Withholding tax deduction"              • "Official public holidays"

3.2 Measuring Proximity Mathematically (Similarity Metrics)

Once texts are converted into vectors, a computer determines conceptual similarity through geometry, most commonly via Cosine Similarity:

$$\text{Cosine Similarity}(\vec{A}, \vec{B}) = \frac{\vec{A} \cdot \vec{B}}{|\vec{A}| |\vec{B}|}$$

Value close to 1.0: The vectors point in nearly identical directions, indicating closely aligned conceptual meaning.
Value close to 0: The vectors are orthogonal, indicating little to no contextual correlation.

Example:

Query: "Caught the flu, who should I notify?" $\rightarrow$ Vector $\vec{Q}$
Document A: "Procedure for submitting doctor's note to team lead" $\rightarrow$ Vector $\vec{D_1}$ (Cosine Similarity = 0.88)
Document B: "Setting up email passwords for sales department" $\rightarrow$ Vector $\vec{D_2}$ (Cosine Similarity = 0.12)

The search system immediately retrieves Document A because its cosine similarity is exceptionally high, even though Document A contains neither the word "flu" nor "notify".

4. Why Not Convert an Entire 100-Page File into a Single Vector?

A frequent question is: "If vector search is so powerful, why not embed an entire 100-page PDF into one vector and be done with it?"

Doing so fails in production for three key engineering reasons:

Information Dilution: If you take an employee handbook covering procurement, vacation policies, legal compliance, IT security, and annual bonuses, and compress all 100 pages into a single vector, that coordinate becomes a vague mathematical average of all topics. When a user asks a specific question, the combined vector will sit far away from the query. It is like blending dozens of different fruits into one container: you end up with a diluted mixture where individual flavors are unrecognizable.
Context Window Limits of Embedding Models: Embedding models have token input limits, such as 512, 2,048, or 8,192 tokens. They cannot ingest megabytes of raw text in a single pass.
Precision for Answer Generation: When retrieving evidence, we need to pass only the specific, relevant passages into the language model. Feeding 100 pages of irrelevant noise increases latency, cost, and hallucination risks.

The Solution: The Art of Chunking (Chunking Strategy)

Before indexing documents, we split files into smaller passages (chunks):

Typically 300 to 500 words per chunk.
A slight overlap (e.g., 50 words) to prevent context from breaking across chunk boundaries.
Each chunk maintains a focused conceptual topic and receives its own unique vector representation.

5. From Finding Content to Answering Questions: The 3-Step Pipeline

Search and retrieval systems only identify where the evidence lives. In real-world workflows, users rarely want ten raw links or disconnected paragraphs: they want a concise, reliable, and actionable answer.

This is where Retrieval connects with Generative AI (LLMs) through a three-step pipeline:

[ 1. User Asks a Question ]
  "Can an employee who is still on probation claim eyeglasses reimbursement?"
       │
       ▼
[ 2. Step 1: RETRIEVAL (Locate Relevant Chunks) ]
  - The system converts the user's question into a query vector.
  - Performs an ANN search across a Vector Database or Inverted Index.
  - Retrieves the top-k chunks with the highest relevance scores.
       │
       ▼
[ 3. Step 2: CONTEXT AUGMENTATION (Assemble Prompt for AI) ]
  - Combines the user query with the retrieved evidence passages:

    ┌────────────────────────────────────────────────────────┐
    │ SYSTEM: Answer the question using ONLY the provided    │
    │ context. Do not extrapolate. If the context does not   │
    │ contain the answer, reply with "Information not found".│
    │                                                        │
    │ CONTEXT:                                               │
    │ [Chunk #42]: Section 8.1 Eyeglasses Benefit            │
    │ The company provides an annual allowance of $100 for   │
    │ prescription glasses. This benefit is reserved exclusively│
    │ for permanent full-time employees who have successfully│
    │ passed their probationary period...                    │
    │                                                        │
    │ QUESTION: Can an employee on probation claim glasses?  │
    └────────────────────────────────────────────────────────┘
       │
       ▼
[ 4. Step 3: GENERATION & REASONING (Synthesize the Answer) ]
  - The LLM reads only the verified context provided.
  - Formulates a concise answer with source citation:

    👉 "No, employees on probation are not eligible. The optical
        allowance ($100/year) is reserved exclusively for permanent
        employees who have completed probation (Ref: Section 8.1)."

6. Comparison: 4 Paradigms of Document Retrieval

Here is a side-by-side comparison of how search methods have evolved:

Dimension	Linear Scan (`Ctrl+F` / `grep`)	Inverted Index (BM25 / Search Engines)	Semantic Vector Search (Embeddings)	Retrieval + LLM (RAG Paradigm)
Core Mechanism	Scans and compares characters line by line	Looks up pre-compiled word dictionary and postings	Measures geometric distance between dense vectors	Retrieves matching evidence chunks, then asks AI to synthesize
Search Speed	Slows down linearly with data size ($O(N)$)	Sub-millisecond lookup	Fast using Approximate Nearest Neighbor (ANN/HNSW)	Moderate (depends on LLM inference time)
Intent Understanding	None	None (tracks term frequency and statistics)	Understands semantic concepts and context	Deep contextual reasoning with synthesis
Handling Synonyms & Typos	Fails completely	Limited (Fuzzy matching, lemmatization)	Excellent	Best-in-class
Output Format	Line positions or character highlights	Document IDs and line offsets	Relevant text chunks	Direct synthesized answer with references

7. Conclusion: What Should Real-World Systems Use?

When building production document intelligence systems, we do not have to pick one method and discard the rest:

If you rely solely on Keyword Search, you will miss documents whenever user terminology diverges from source wording.
If you rely solely on Vector Search, you may struggle with exact identifiers such as SKU numbers (NK-9920-X), tax IDs, version strings, or domain-specific jargon absent from pre-trained embeddings.

This is why modern enterprise search architectures adopt Hybrid Search, combining lexical keyword indexes with vector embeddings, running the candidates through a Re-ranking stage, and passing the most relevant evidence to an LLM.

At NEXT4I, these search principles guide how we explore and architect document intelligence solutions, ensuring retrieval systems deliver speed, accuracy, and provable evidence. Stay tuned for future deep dives as we continue building and sharing our engineering journey.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Originally published at:
https://next4i.com/dev-notes/en/search-and-retrieval-fundamentals-dev-en

RAG Architecture Beyond the Demo: Retrieval, Thai Chunking, and Production Boundaries

NEXT4I DEV — Wed, 23 Sep 2026 15:37:19 +0000

Connecting an LLM to an application is the easy part of a RAG demo.

The difficult part starts when the source is a real document instead of a clean string in an array.

I learned this while feeding a Thai tourism PDF into a document pipeline. (You can read more about that story here: "Why Markdown Is the Ultimate AI-Native File Format A War Story from Building NEXT4I" https://www.next4i.com/dev-notes/en/markdown-the-ultimate-ai-native-dev-en).

Direct text extraction produced fragmented characters and misplaced vowels or tone marks. Rendering pages as images preserved the layout but introduced colorful backgrounds and watermarks. Converting them to black and white made some text clearer while removing useful context from charts and photographs. Different models produced different interpretations, so the outputs still required synthesis and human review.

Think of it like OCR processing for an ID card selfie: even with modern vision models, apps still often need human input fields to verify or correct the data. Document ingestion is just step one of RAG, and this is where the real world begins.

That experience changed how I look at Retrieval-Augmented Generation. RAG is not only a prompt pattern. It is a data and retrieval system with an LLM at the end.

This article unpacks that system through:

separate ingestion and query pipelines
document extraction and chunking
embeddings and retrieval
dense, sparse, and hybrid search
access control in the retrieval path
evaluation boundaries
a conceptual implementation in Python

The code below is generic and sanitized. It demonstrates mechanics, not a production-ready application. Provider names, model IDs, SDK calls, and executable integration still need verification against the versions you use.

The three limits RAG is trying to address

When an application uses an LLM out of the box, three limits appear quickly.

Knowledge boundaries

The model does not automatically know a new event, a private company policy, or a document created yesterday.

Context and cost boundaries

Passing an entire document collection in every request is usually impractical. A large context also does not guarantee that the model will use every passage equally well.

Verifiability

A generated answer does not prove itself. If a user cannot trace a statement back to a permitted source, confidence is not evidence.

RAG addresses these limits by finding a small set of relevant passages and placing them in the context before generation. It can improve grounding and traceability, but it does not guarantee either accuracy or security.

RAG is two pipelines, not one request

A useful mental model separates offline ingestion from online querying.

[Ingestion pipeline]

Raw documents
    |
    v
Extract and clean
    |
    v
Split into chunks
    |
    v
Create embeddings and metadata
    |
    v
Store in a searchable index


[Query pipeline]

User question
    |
    v
Apply identity and permission scope
    |
    v
Retrieve candidate chunks
    |
    v
Filter and optionally rerank
    |
    v
Build prompt with citations
    |
    v
Generate answer
    |
    v
Return answer and inspectable sources

The pipelines are decoupled for a reason. Documents change on their own schedule. User queries arrive on another. Extraction failures should not be rediscovered during every chat request, and permission changes should not require retraining a model.

Ingestion quality becomes retrieval quality

The phrase “garbage in, garbage out” is almost too familiar, but it describes RAG accurately.

A document can look correct to a human and still become poor retrieval material after extraction. Typical problems include:

repeated headers and footers
tables flattened into an unreadable order
text stored as positioned glyphs rather than paragraphs
watermarks mixed with body text
diagrams whose meaning exists only in the visual layout
scanned pages without a reliable text layer
old and current versions indexed together

The ingestion pipeline needs to preserve more than plain text. Useful metadata can include a stable document ID, version, page, section heading, timestamps, source location, tenant, and permission attributes.

That metadata supports citations, updates, deletion, filtering, and review. Without it, a retrieved paragraph becomes an orphan.

Thai documents make naive chunking easier to break

Many chunking examples assume that spaces and punctuation provide reliable boundaries. Thai text does not always behave that way.

Ambiguous boundaries

Consider the Thai text:

ตากลมนั่งมองตากลม

Depending on segmentation and context, ตากลม can relate to round eyes, while ตากลม can also be read through a boundary involving exposure to wind. A bad split changes the meaning represented by the chunk and therefore affects retrieval.

Long dependencies

A policy sentence may introduce the subject and action early, then place the condition or consequence much later. A fixed-size cut can leave one chunk with the violation and another with the disciplinary action. Neither passage is complete enough to answer the question reliably.

Periods that are not sentence endings

Thai abbreviations, titles, legal terms, locations, and time expressions can contain periods:

ศ.ดร.สมชาย ... พ.ร.บ. ... อ.เมือง จ.เชียงใหม่ ... 09.00 น.

A generic splitter that treats every period as an end of sentence can create tiny, meaningless chunks.

Possible approaches from the original engineering notes include:

semantic chunking around topic changes
Thai-aware tokenization, for example with PyThaiNLP, plus a domain dictionary
overlap around chunk boundaries
hierarchy-aware parent and child chunks

The correct strategy and parameters are corpus-specific. Any fixed chunk size or overlap percentage should be evaluated against real questions rather than copied as a universal setting.

Chunking strategies and their trade-offs

Fixed size with overlap

This is simple and predictable. It also cuts across structure when the chosen length does not match the document.

Recursive splitting

This method tries larger structural separators first, such as sections and paragraphs, then falls back to smaller boundaries. It works better when the extracted structure is trustworthy.

Semantic chunking

This approach compares nearby passages and creates a new chunk when the topic shifts. It can preserve meaning better, but it adds model dependency, threshold tuning, and processing cost.

Parent-child chunking

Small child chunks can support precise retrieval while a larger parent section supplies enough context to answer. The trade-off is more complex indexing, deduplication, and prompt assembly.

Chunking should be treated as a retrieval decision, not a formatting task. The unit you retrieve determines the evidence the model can see.

Embeddings are coordinates, not facts

An embedding converts content into a numeric vector. Content with related meaning may appear near each other in that vector space, depending on the model and data.

That makes semantic retrieval possible, but an embedding does not verify truth. It also does not know that an older policy is invalid unless version and filtering logic provide that boundary.

Conceptual implementation in Python (In-Memory Retrieval & Query Pipeline)

To illustrate how a query pipeline works in one place, the following Python example simulates cosine similarity calculation, user permission and tenant filtering prior to retrieval, top-K ranking, and prompt augmentation for the LLM:

import math

def cosine_similarity(vec_a, vec_b):
    """Compute cosine similarity between two numeric vectors."""
    if len(vec_a) != len(vec_b) or len(vec_a) == 0:
        raise ValueError("Vectors must have the same non-zero length")

    dot_product = sum(a * b for a, b in zip(vec_a, vec_b))
    norm_a = math.sqrt(sum(a * a for a in vec_a))
    norm_b = math.sqrt(sum(b * b for b in vec_b))

    if norm_a == 0 or norm_b == 0:
        raise ValueError("Cosine similarity is undefined for a zero vector")

    return dot_product / (norm_a * norm_b)

def answer_with_rag(question, user, knowledge_base, top_k=2, create_embedding=None, generate_answer=None):
    """Demonstrate an end-to-end RAG query flow with permission scoping."""
    # 1. Convert user question into an embedding vector
    query_vector = create_embedding(question)

    # 2. Apply identity and permission scope before retrieval
    # Critical: The model must never receive context the user is not allowed to see
    authorized_chunks = [
        chunk for chunk in knowledge_base
        if chunk["tenant_id"] == user["tenant_id"] and user["role"] in chunk["allowed_roles"]
    ]

    # 3. Retrieval: calculate similarity and rank Top-K
    ranked_chunks = [
        {**chunk, "score": cosine_similarity(query_vector, chunk["embedding"])}
        for chunk in authorized_chunks
    ]
    ranked_chunks.sort(key=lambda item: item["score"], reverse=True)
    selected_chunks = ranked_chunks[:top_k]

    # 4. Augmentation: build context with verifiable source citations
    context_text = "\n\n".join([f"[ID: {c['id']}] {c['content']}" for c in selected_chunks])

    # 5. Generation: ask the model to ground its response in the provided context
    instruction = "Answer only from the supplied context. If the context is insufficient, state that clearly."
    answer = generate_answer(
        instruction=instruction,
        question=question,
        context=context_text
    )

    return {
        "answer": answer,
        "sources": [{"id": c["id"], "score": round(c["score"], 4)} for c in selected_chunks]
    }

The core takeaway is that permission filtering must happen before passages become model context. In production enterprise systems, this is handled through row-level security (RLS), policy engines, document ACLs, or metadata filtering on a vector store rather than an in-memory loop.

Dense search is not enough for every query

Dense vector search is useful for semantic similarity. It can connect related wording such as synonyms and paraphrases.

Exact identifiers are a different problem. Product SKUs, serial numbers, legal section numbers, error codes, and names can be better served by sparse keyword retrieval.

That creates three common choices:

Dense retrieval: strong for semantic similarity
Sparse retrieval such as BM25: strong for exact terms
Hybrid retrieval: combines candidates from both systems, then fuses or reranks them

Reciprocal Rank Fusion and cross-encoder reranking are possible tools in that pipeline. They are not automatic improvements for every corpus. Evaluate them using your own question set and latency boundary.

Five production boundaries worth testing

1. Language and embedding fit

Do not assume an embedding model that looks good on English examples will behave the same way on Thai policies, abbreviations, and domain terms. Compare models against labeled queries from the actual corpus.

2. Extraction and version quality

Track failed pages, missing sections, table quality, repeated boilerplate, and document versions. Retrieval cannot recover content that ingestion lost.

3. Context pollution

More chunks are not always better. Irrelevant context can distract the model and increase cost. Evaluate the complete answer path rather than maximizing topK by intuition.

4. Security and data isolation

Permission-aware retrieval needs to be enforced outside the model. Test cross-tenant access, role changes, deleted permissions, cached results, and citation links.

5. Evaluation

The original notes use three useful questions:

Context relevance: Did retrieval find evidence related to the question?
Groundedness or faithfulness: Does the answer stay within the supplied evidence?
Answer relevance: Does it answer what the user asked?

Add operational checks that match the system, such as extraction failures, stale indexes, permission denials, and citation integrity. Do not invent target scores. Establish them from your risk and user requirements.

The main engineering lesson

RAG connects language-model capability with external knowledge, but the connection is only as useful as the pipeline around it.

A local demo can hide document quality, authorization, stale versions, and evaluation because the sample chunks are already clean. Production exposes all of them.

For me, the work starts before prompt engineering:

prepare and version the knowledge
choose chunk boundaries that preserve meaning
retrieve with semantic and exact-match needs in mind
apply permissions before context construction
preserve citations
evaluate retrieval and generation separately

The LLM writes the final response. Most of the evidence it can use has already been decided by then.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Originally published at:
https://www.next4i.com/dev-notes/en/what-is-rag-dev-en

How to Write an AI Agent Skill File That Reduces Guesswork

NEXT4I DEV — Mon, 14 Sep 2026 16:58:08 +0000

A prompt can tell an agent what to do once. A skill file can document how a team handles that class of work repeatedly.

For developers, the distinction matters. An agent may have repository access and the right tools, yet still has to make decisions about scope, references, validation, and failure handling. If those decisions are undefined, a technically valid result can still be the wrong result.

This guide builds a small, generic skill for reviewing API changes. The point is not to prescribe one universal Agent Skills format. It is to show how to turn vague intent into an executable workflow.

Skill discovery, instruction loading, and tool use vary by model, agent runtime, system instructions, and available tools. Treat the structure below as a portable design pattern, then adapt it to your platform.

The problem with a reasonable instruction

Consider this prompt:

Review this API and make sure it follows best practices.

It sounds clear, but the agent still has to decide:

Which API contract is authoritative?
Does the review include authentication and permissions?
Should it modify code or only produce a report?
Which tests should it run?
What should it do when information is missing?
What does a completed review contain?

Different models may fill those gaps differently. The result can be reasonable without matching the workflow you intended.

Prompt versus skill

I use this distinction:

A prompt says, “Do this task now.”
A skill says, “This is how we handle this type of task.”

A useful skill does not make a model magically smarter. It makes the operating boundaries visible.

At minimum, it should define:

Trigger: when the skill applies
Goal: what outcome it should produce
Scope: what is included and excluded
Workflow: steps the agent can follow
References: information to read under specific conditions
Constraints: actions and claims that are prohibited
Uncertainty handling: what to do when evidence is missing
Completion criteria: how to validate the result

A practical directory structure

Start with one file while the workflow is small. Split it when references, examples, or reusable scripts become large enough to distract from the main instructions.

api-review-skill/
├── SKILL.md
├── references/
│   ├── api-contract.md
│   ├── security-rules.md
│   ├── output-examples.md
│   └── troubleshooting.md
├── scripts/
│   └── validate.sh
└── assets/
    └── review-template.md

The exact file names and supported directories depend on the platform and runtime. More importantly, models may not select or load these files in the same way. The main file should therefore explain why and when each reference matters.

A generic `SKILL.md` example

---
name: api-change-review
description: "Review API changes for contract compatibility,"
  security boundaries, and required validation. Use when an
  endpoint, request schema, response schema, authentication,
  or permission behavior changes.
---

# API Change Review

## Goal

Produce a review report that identifies contract-breaking changes,
security-sensitive changes, missing validation, and unresolved assumptions.

## Scope

Review the proposed change and related tests.
Do not modify implementation files unless the user requests edits.

## Workflow

1. Read the changed files and the API contract.
2. Identify changes to endpoints, fields, status codes, and behavior.
3. If authentication, secrets, or permissions are involved,
   read the security rules.
4. Compare the tests with the changed behavior.
5. Produce the required report.
6. Run the validation command when the environment supports it.

## Reference routing

- Read `references/api-contract.md` for endpoint or schema changes.
- Read `references/security-rules.md` for authentication,
  secrets, or permission changes.
- Read `references/output-examples.md` only when the report format
  is unclear.
- Read `references/troubleshooting.md` when validation fails.

## Constraints

- Do not invent endpoints, fields, metrics, or test results.
- Do not expose credentials, customer data, or internal URLs.
- Do not present an unimplemented requirement as released behavior.
- Flag assumptions that require human review.

## Required output

1. Summary
2. Contract-breaking changes
3. Security-sensitive changes
4. Missing or affected tests
5. Assumptions requiring review
6. Validation performed

## Completion criteria

- Every changed behavior is mapped to the contract.
- Security-sensitive changes are explicitly identified.
- Validation results state what was and was not run.
- Unresolved assumptions are visible.

This example is intentionally generic. It demonstrates an instruction contract, not a claim about one platform's required syntax.

Before and after

Before

Review this API and make sure it follows best practices.

After

Review the API change using the api-change-review skill.

Use `references/api-contract.md` as the contract source.
If the change touches authentication, secrets, or permissions,
also apply `references/security-rules.md`.

Produce the required review report. Do not modify implementation files.
State which validation commands were run and flag unresolved assumptions.

The second instruction is not better because it is longer. It is better because fewer operational decisions are left undefined.

Turn the index into a routing table

A file list tells an agent what exists. A routing table explains when a file becomes relevant.

Reference	Read when	Skip when
`api-contract.md`	An endpoint, schema, status code, or behavior changes	The task does not involve an API contract
`security-rules.md`	Authentication, secrets, roles, or permissions are involved	The task has no security-sensitive behavior
`output-examples.md`	The required report shape is unclear	The output contract is already explicit
`troubleshooting.md`	Validation cannot run or fails	Validation succeeds normally

This table is guidance, not an enforcement mechanism. Actual reference selection still depends on the model, runtime, context strategy, and tools.

Single file or multiple files?

Keep the skill in one file when:

it has one short workflow
all constraints fit without hiding the main steps
examples are small
no reusable scripts are needed

Split the skill when:

references are long or domain-specific
several workflows share the same rules
examples make the main file difficult to scan
validation scripts are reusable
sensitive rules require separate ownership or review

The main file should remain enough to answer two questions: What should the agent do, and where should it look next?

Treat skills like software

A skill can contain instructions, references, and executable code. Review an external skill before using it, especially when it can access files, call a network service, or run scripts.

Check for:

file access outside the expected scope
network and API destinations
hardcoded credentials
destructive file operations
hidden instructions that attempt to bypass system rules
scripts whose behavior has not been reviewed

For organizational use, sandboxing and coexistence tests may also be appropriate. Different models can respond to the same instruction or tool surface differently.

Validation checklist

- [ ] The description states when the skill should trigger.
- [ ] The goal and scope are explicit.
- [ ] The workflow contains observable steps.
- [ ] Each reference has a condition for when to read it.
- [ ] Prohibited actions and claims are listed.
- [ ] Missing information has a defined handling rule.
- [ ] The output contract is reusable and testable.
- [ ] Completion criteria can be checked.
- [ ] No secrets or internal URLs are embedded.
- [ ] Executable scripts have been reviewed.
- [ ] The skill was tested on tasks that should trigger it.
- [ ] The skill was tested on tasks that should not trigger it.

Final takeaway

Before handing a skill to an agent, read it as if you were a developer joining the project today:

Can I complete the work from this information? What would I still have to guess? If I must make a decision, do I know which direction to take?

A good skill does not remove reasoning. It removes avoidable guessing.

Read the canonical article on NEXT4I: https://go.next4i.com/next4i-devto-en

Originally published at:
https://www.next4i.com/dev-notes/en/skill-file-with-index-dev-en

What Is Token & LLM Cost Optimization

NEXT4I DEV — Mon, 07 Sep 2026 14:35:04 +0000

What Is a Token in LLMs? A Developer's Guide to Cost Optimization and Architecture

Understanding how LLM tokens work is the difference between an AI feature that costs $50/month and one that runs up a $5,000 AWS/OpenAI bill.

Here is the engineering breakdown of tokenization algorithms, input vs. output pricing asymmetries, multilingual overhead, model pricing dynamics, and six production strategies implemented at NEXT4I to cut inference costs by up to 50-80%.

#Buildinpublic, #ModelAI, #LLM, #AIArchitecture, #TokenOptimization, #AIDeveloper

1. What Is a Token Under the Hood?

LLMs do not process raw text or strings. They consume Tokens—subword representations mapped to high-dimensional embedding vectors via algorithms like Byte-Pair Encoding (BPE).

"Hello world" -> ["Hello", " world"] (2 tokens)
"Unstoppable" -> ["Un", "stoppable"] (2 tokens)

The Multilingual Token Penalty

Because vocabulary dictionaries are predominantly trained on English corpora, languages without whitespace delimitation (such as Thai) suffer severe subword fragmentation:

English: 1 Token ≈ 0.75 words (~4 characters).
Thai: 1 Word ≈ 3 to 8 Tokens (frequently split into byte-level representations).

A prompt written in Thai can cost up to 6x more in raw token usage and introduce noticeable latency compared to its English equivalent.

2. Formatting Mechanics: Linebreaks & The "Emoji Tax"

Every character in your prompt payload carries a token cost:

Newlines (\n) and Numbered Markdown:
- Consumes 1 token per newline.
- Verdict: Highly recommended. Clear formatting provides structural anchors for transformer attention heads, significantly reducing hallucination.
The Emoji Tax:
- Emojis are complex multibyte Unicode sequences, often consuming 2 to 6+ tokens each.
- Avoid embedding decorative emojis in static system prompts that execute millions of times.

3. Input vs. Output Tokens: Why Output Costs 3x–5x More

API rate cards price Output Tokens significantly higher than Input Tokens. Why?

Input Tokens (Parallelized Compute): Processed simultaneously across GPU tensor cores in a single matrix multiplication pass (Prefill / Encoding). It is fast and hardware-efficient.
Output Tokens (Sequential Autoregression): Generated one token at a time (Autoregressive Decoding). The model generates token N, appends it back to the context history, and re-computes attention for token N+1. This locks GPU resources over the entire generation cycle.

4. Why Model Pricing Varies by 100x (Dense vs. MoE Architecture)

You may have noticed that API pricing spans from $0.30 to $50.00+ per 1 million tokens across models:

DeepSeek-V4: $0.50 – $1.70 / 1M Tokens (Massive architecture, exceptionally cost-efficient)
Gemini 3.7 Flash: $0.30 – $1.80 / 1M Tokens (High-efficiency edge tier)
Claude Sonnet 5: $2.00 – $10.00 / 1M Tokens (Mid-tier balanced powerhouse)
OpenAI GPT-5.6 Terra: $2.00 – $12.00 / 1M Tokens (Enterprise mid-tier)
Claude Opus 5: $5.00 – $25.00 / 1M Tokens (Premium reasoning class)
Claude Fable 5: $10.00 – $50.00 / 1M Tokens (Mythos-class model from Anthropic)
OpenAI GPT-5.6 Sol: $4.00 – $30.00 / 1M Tokens (OpenAI's flagship)

Note: Model pricing per token fluctuates frequently across providers and should be used strictly for relative comparative analysis.

This pricing variance stems from three fundamental drivers:

1. Architectural Design: Dense Models vs. Mixture-of-Experts (MoE)

Dense Models

Mechanics: Every incoming vector/token is processed through 100% of the model's parameters, from the initial layer to the final output layer.
Analogy: Like a company where every single employee must sit in every meeting and vote on every decision—regardless of whether the task is simple arithmetic or legal compliance.
Trade-offs: Highly compute-intensive (massive FLOPs per token), resulting in higher per-token inference costs. However, memory management and GPU scheduling remain straightforward.

Mixture-of-Experts (MoE)

Mechanics: An intelligent gating mechanism called a Router (or Gating Network) acts as a dispatcher, evaluating incoming tokens and dynamically routing them to specialized sub-networks (Experts).
Execution Flow:
1. The Router analyzes each token and activates only a sparse subset of experts (e.g., selecting 2 out of 8 or 64 total experts).
2. Compute flows exclusively through the parameters of the chosen experts.
Analogy: Like a well-structured organization with an executive dispatcher. A calculus question is routed strictly to the math experts without distracting the linguistics team.
Advantages: Enables total model capacity (Total Parameters) to scale massively while keeping active compute per token (Active Parameters / FLOPs) extremely low. This allows lightning-fast generation and drastically lower API prices (e.g., DeepSeek, Mixtral).
Trade-offs: Requires massive VRAM/RAM pools to keep all expert weights loaded in memory simultaneously.

The MoE Achilles' Heel: When Routers Fail

While MoE unlocks unmatched cost efficiency, its performance is tightly bound to routing stability:

Loss of Nuance & Context Disruption:
- Natural language is rich with subtle subtext. If a router misinterprets a token and dispatches it to the wrong expert, nuanced meaning collapses. The output may stay grammatically intact but lose analytical depth or fail to address the core prompt intent.
Router Collapse & Expert Imbalance:
- Routers naturally develop bias toward a handful of frequently trained experts, causing severe load imbalance. The favored experts hit computational bottlenecks while neglected experts become Dead Parameters, defeating the entire purpose of modular specialization.
Cascading Errors:
- LLMs process representations layer by layer. If an early-layer router misroutes a token, downstream layers receive corrupted intermediate activations, amplifying routing errors across subsequent layers.

🛠️ Engineering Safeguards Used by Frontier Labs:

Auxiliary Load Balancing Loss: Incorporating penalty penalties into the loss function during pre-training to enforce uniform token distribution across all expert sub-networks.
Capacity Factor Enforcement: Setting strict token buffer caps per expert. Once an expert's capacity threshold is reached, excess tokens are spilled over to secondary experts to prevent execution bottlenecks.

2. Reasoning Overhead (Thinking / Chain-of-Thought Tokens)

Reasoning-focused models (like Claude Opus, OpenAI GPT Sol) generate thousands of internal, hidden Chain-of-Thought tokens before emitting their first visible output token. Providers meter and bill for every single background reasoning step.

3. Hardware Sovereignty & Custom Silicon

Hyperscalers operating proprietary custom silicon (such as Google’s TPU clusters for Gemini) achieve significantly lower baseline operating costs than providers renting general-purpose Nvidia H100/H200 GPU clusters.

5. 6 Production Strategies to Cut Token Costs by 80%

Here are some of the production-level strategies we use in building NEXT4I:

1. Prompt Engineering for Token Efficiency

Eliminate fluff and instructions that don't add semantic value. Use concise formats like YAML or Markdown instead of verbose JSON schemas.

2. Leverage Prompt Caching

Major LLM providers (Anthropic, OpenAI, DeepSeek, Google) offer prompt caching. Placing static context (system instructions, tool definitions, schemas) at the prompt root allows providers to cache the KV-cache, reducing input costs by 75%–90%.

3. Intelligent Model Cascading (Model Routing)

Never route every query to flagship models. Route simple classification, data extraction, and formatting tasks to lightweight models or budget-friendly models (Gemini Flash, DeepSeek), escalating only complex reasoning tasks to larger models (Claude Sonnet, GPT Terra, GPT Sol, Claude Opus / Fable).

4. Sliding Context Windows & Conversation Pruning

Chat histories grow quadratically ($O(n^2)$) if sent in their entirety on every turn. Maintain a rolling sliding window of the last 5–10 messages, or summarize older context into a single concise paragraph.

5. Pre-Retrieval RAG Filtering

In RAG pipelines, do not inject full documents into the context window. Use embedding similarity and rerankers to select top-k (3 to 5) chunks, applying semantic deduplication before prompt construction.

6. Multilingual Translation Layer

For bulk data extraction or batch processing on non-Latin languages, translating text to English with a lightweight model prior to deep inference on flagship models can reduce total token usage and improve execution latency.

Key Takeaways

+-------------------+---------------------------------------------------+
| Metric            | Engineering Reality                               |
+-------------------+---------------------------------------------------+
| Token Ratio (EN)  | ~1 Token ≈ 0.75 words (~4 characters)             |
| Token Ratio (TH)  | ~1 Word ≈ 3–8 Tokens (byte-level inflation)       |
| Cost Ratio        | Output is 3x–5x more expensive than Input         |
| Pricing Deltas    | Dense vs MoE, TPU/ASIC custom silicon, CoT tokens |
| Core Optimizers   | Caching + Routing + Windowing + RAG Filtering     |
+-------------------+---------------------------------------------------+

Treating tokens as finite compute bandwidth ensures your AI infrastructure remains fast, scalable, and economically sustainable.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Originally published at:
https://www.next4i.com/dev-notes/en/what-is-token-dev-en

The RAG War Story: How I Learned to Stop Worrying and Love Markdown

NEXT4I DEV — Tue, 01 Sep 2026 11:23:36 +0000

TLDR; When building an AI knowledge retrieval pipeline that extracts text from documents, I discovered that PDF is the worst format for AI and Markdown is the best. Here's the multi-step pipeline I had to build just to handle PDFs, why it was necessary, and the generic pattern you can steal to handle document ingestion in your own RAG systems.

The Problem: PDFs Are Pixel-Perfect Hell for AI Parsers

I was building a RAG (Retrieval-Augmented Generation) pipeline for NEXT4I the kind of system that reads your documents first, then answers questions from them. Standard stuff: document ingestion → chunking → embedding → vector search → LLM answer generation.

I chose a beautiful Thai tourism PDF as my test document. Professional design, complex Thai typography, images, tables, charts the works. Real-world document, real-world pain.

Here's what the naive approach looked like:

PDF File → PDF Parser → Extracted Text → Chunk → Embed → Search

And here's what actually worked:

PDF File
  ├─→ PDF Parser → Raw Text (broken Thai, missing punctuation)
  ├─→ Page Renderer → Full-Color Images
  │     └─→ B&W Converter → High-Contrast Images
  ├─→ AI Vision Model (color images) → Image Descriptions
  ├─→ AI Vision Model (B&W images) → Text Extraction
  └─→ Cross-Validation Layer
        ├─→ Multi-Model Synthesis
        ├─→ Spell-Check Model (critical for Thai)
        └─→ Human Review
              └─→ Final Structured Text → Chunk → Embed → Search

Why the complexity? Because PDF is fundamentally a presentation format, not a data format. When you extract text from a PDF, you're not reading structured data you're reverse-engineering a rendered page layout. For languages with complex typography like Thai (where vowels can appear above, below, left, or right of consonants, and tone marks float above), this is especially brutal.

The Generic Pattern: Multi-Path Document Ingestion with Cross-Validation

If you're building any system that ingests arbitrary documents, you'll inevitably hit the PDF wall. Here's the reusable pattern I settled on:

Architecture

                    ┌──────────────┐
                    │   Document   │
                    │   Ingest     │
                    └──────┬───────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │  Direct  │ │  Image   │ │  Image   │
        │  Text    │ │  (Color) │ │  (B&W)   │
        │ Extract  │ │  Render  │ │  Render  │
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │            │            │
             ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │  Text    │ │  Vision  │ │  Vision  │
        │  Output  │ │  Model   │ │  Model   │
        │          │ │  (Desc)  │ │  (OCR)   │
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │            │            │
             └────────────┼────────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Cross-Validate │
                 │  & Synthesize   │
                 │  (Multi-Model)  │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Spell-Check    │
                 │  & Normalize    │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Human Review   │
                 │  (Optional)     │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Structured     │
                 │  Output → Embed │
                 └─────────────────┘

The key insight: no single extraction path is reliable enough on its own. You need multiple independent paths producing results, then a synthesis layer that cross-validates. Think of it like sensor fusion each path is a noisy sensor, and the truth emerges from the overlap.

Why Spell-Check is Non-Negotiable for Non-English Languages

For English, you might get away without a dedicated spell-check pass. For Thai where a single misplaced tone mark changes the entire word you absolutely cannot. OCR and vision models hallucinate characters constantly on decorated fonts or text-over-image backgrounds. A dedicated language model fine-tuned for spell correction is the difference between "usable" and "garbage."

The Real Takeaway: Markdown is AI-Native. Everything Else Is Legacy.

After building this entire pipeline, I had a moment of clarity. If that same document had been authored in Markdown:

## Top Destinations

| Province | Highlight | Best Season |
|----------|-----------|-------------|
| Krabi    | Islands   | Nov–Apr     |
| Chiang Mai | Mountains | Nov–Feb   |

See the [full itinerary](#itinerary) for details.

mermaid
graph TD
A[Arrive Bangkok] --> B[Fly to Krabi]
B --> C[Island Hopping]
C --> D[Return]

markdown

...the entire pipeline collapses to: read the file → chunk → embed → search. That's it.

No OCR. No vision models. No B&W conversion. No multi-path cross-validation. No spell-check model. No human review for format-induced errors.

Markdown is structured, plain-text, and both human-readable and machine-parseable by default. It's the only format where:

Headings are unambiguously # / ## not inferred from font size
Tables are | column | row | syntax not pixel grids
Diagrams are Mermaid text not flattened raster images
Code is fenced not monospaced-font heuristics

"And here is how the human user experiences it:"

Top Destinations

Province	Highlight	Best Season
Krabi	Islands	Nov–Apr
Chiang Mai	Mountains	Nov–Feb

See the full itinerary for details.

graph TD
    A[Arrive Bangkok] --> B[Fly to Krabi]
    B --> C[Island Hopping]
    C --> D[Return]

"In reality, we can't always control the documents we ingest, and we can't just ignore them because they might contain critical data. But if we were to start from scratch, Markdown is definitely the go-to choice."

How This Shapes Our Architecture at NEXT4I

At NEXT4I, we treat Markdown as a first-class format throughout our stack. When building AI knowledge retrieval systems for everyday users and organizations, we encourage Markdown as the source of truth and handle PDFs as a necessary-but-painful compatibility layer.

The design principle is simple: AI Integration by Design. Make AI a first-class citizen of your content architecture, not something you bolt on later and hope it works. The format you choose today determines the ceiling of your AI capabilities tomorrow.

Thanks for reading all the way to the end, I'll keep working on more articles like this.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Originally published at:

next4i.com

What Is LLM Actually Doing? A Fellow Engineer's Take on Vectors, Next-Token Prediction, and Fail-back Routing

NEXT4I DEV — Tue, 01 Sep 2026 05:30:51 +0000

Why treating an LLM as a probability engine, not a brain, changes how you architect around it. The reasoning behind NEXT4I's AI layer.

#LLM #BuildinPublic #SystemArchitecture #AI #Model AI #AI Router #AI Stable

I used to wonder why the model confidently gives you a wrong number. It's not a bug in the traditional sense, it's the model doing exactly what it's built to do: predicting the next token from probability, with zero actual arithmetic happening underneath.

TLDR; An LLM (large language model) converts text into vectors (numeric coordinates in a high-dimensional meaning-space) and generates output via next-token prediction, sampled with parameters like top-k and temperature. Because it's fundamentally a probability engine and not a calculator or a database, I designed NEXT4I with automatic model fail-back routing and task-based model selection instead of trusting any single model as a source of truth.

The Mental Model: Vectors, Not Meaning

Every token gets embedded into a vector, often with hundreds or thousands of dimensions. Semantically similar tokens end up close together in that space. That's why semantic search (vector-based retrieval) can match "large flying animal consumes insects" to "big bird eats worms" even with zero shared keywords, unlike old-school lexical search (TF-IDF/BM25) which needs literal term overlap.

GPUs handle this well because they're already wired for massive parallel floating-point math (the same math used to shade millions of pixels per frame), so throwing billions of similarly-directed vectors at a GPU is a natural fit, not a coincidence.

Generation Is Sampling, Not Retrieval

Given a prompt, the model doesn't look up an answer, it samples one token at a time from a probability distribution. Two knobs matter in practice:

top_k: 2        # only sample from the top-2 most likely next tokens
temperature: 0.2  # low = deterministic/precise, high = creative/varied

Low temperature + low top_k gives you consistent, "boring" output, good for structured extraction. High temperature gives you variety, good for brainstorming, bad for anything requiring precision.

Why LLMs Hallucinate on Arithmetic

There's no calculator inside the model. It doesn't evaluate x * y, it predicts digits that are statistically plausible given the prompt, one token at a time. It gets 2 * 2 right because that pattern is everywhere in training data. It confidently botches large multiplication because it's still just sampling digits, not computing. This is exactly why production systems now delegate real math to a tool call (a Python sandbox, a calculator function) instead of trusting raw model output.

Core Value: A Generic Model-Tier Fail-back Router

Here's the simple pattern, stripped of any specific business logic, a reusable fail-back wrapper for any set of same-tier model clients:

package modelrouter

import (
    "context"
    "errors"
    "fmt"
)

// ModelClient is any backend that can answer a prompt.
type ModelClient interface {
    Name() string
    Complete(ctx context.Context, prompt string) (string, error)
}

// TieredRouter tries each client in a tier in order until one succeeds.
type TieredRouter struct {
    tier []ModelClient
}

func NewTieredRouter(clients ...ModelClient) *TieredRouter {
    return &TieredRouter{tier: clients}
}

// Complete attempts each model in the tier, fail-back on error.
func (r *TieredRouter) Complete(ctx context.Context, prompt string) (string, error) {
    var errs []error
    for _, client := range r.tier {
        resp, err := client.Complete(ctx, prompt)
        if err == nil {
            return resp, nil
        }
        errs = append(errs, fmt.Errorf("%s: %w", client.Name(), err))
    }
    return "", errors.Join(errs...)
}

This is intentionally boring: try the next model in the same tier on failure, return the first success. No retries with backoff yet, no circuit breaker, just the core fail-back idea. In NEXT4I's actual implementation, tiers are populated dynamically and health state feeds back into ordering, but that logic is abstracted here on purpose, the generic version above is what's actually useful to share.

Core Value: Task-Based Routing, the Simple Version

A minimal router that inspects task complexity before picking a tier:

package modelrouter

type Complexity int

const (
    Simple Complexity = iota
    Complex
)

func ClassifyAndRoute(task string, simpleTier, complexTier *TieredRouter) *TieredRouter {
    if estimateComplexity(task) == Simple {
        return simpleTier
    }
    return complexTier
}

func estimateComplexity(task string) Complexity {
    if len(task) < 100 {
        return Simple
    }
    return Complex
}

estimateComplexity can be as crude as a length/keyword heuristic or as sophisticated as a small classifier model, the point is the routing decision happens before the expensive call, not after.

Trade-offs I Made

Fail-back within a tier, not across tiers. Swapping a cheap model in for an expensive one silently would change output quality without anyone noticing. Tiers exist specifically to avoid that.
No cross-request state in the router. Keeps it stateless and trivially horizontally scalable, at the cost of not learning from past failures within a single request lifecycle.
Routing is a toggle, not a mandate. Users can pin a specific model when they need deterministic behavior from one exact provider, the router only kicks in by default.

If there's one thing worth taking away: treat the model as a probability engine you can't fully trust, and let the system around it, fail-back, routing, tool calls for math, carry the reliability burden instead.

Thanks for reading all the way to the end, I'll keep working on more articles like this.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Originally published at:
https://next4i.com/dev-notes/en/what-is-llm-dev-en

How to Build a "Second Brain" with Obsidian That Your AI Agent Can Read, Without Building a Custom RAG Pipeline

NEXT4I DEV — Mon, 17 Aug 2026 11:40:46 +0000

Ever run into this? You wrote a detailed technical spec three months ago, and today someone asks "how did we design this module again?" You end up spending 20 minutes hitting Cmd+F across Google Docs, Trello, and README.md files scattered across different repos, or sometimes there's no documentation at all, so you have to dig through the code and reverse-engineer it by eye.

TLDR; I built a knowledge-base system for NEXT4I using Obsidian (plain Markdown) together with Git and a VS Code AI agent, without writing a single line of custom integration. The core of the system is choosing tools that already "speak the same language" from day one (plain text, open formats), so AI can read our knowledge base practically for free, no RAG pipeline required.

Pain Point: Scattered Knowledge and Requirements Make Search Hard and Nothing Stays in Sync

Before this system, all of NEXT4I's knowledge was scattered across many places: notebooks, Apple Notes, Google Docs, Google Sheets, Trello, README.md files, or, even worse, sometimes there was no documentation at all, just buried in the code, spread across multiple repos each written in a different language, frontend and backend alike.

This isn't just an inconvenience. It makes search genuinely hard, sometimes it's Cmd+F and pray. Worse: my AI coding agent could read the entire codebase, but it had zero visibility into the reasoning behind that code, because that reasoning was scattered somewhere the AI couldn't reach.

Design Constraints: 3 Non-Negotiables

Before picking a tool, I set 3 rules:

It has to be affordable, or free if possible.
It has to be accessible online anytime, from my phone, and still work offline.
Plain text, zero lock-in. If the tool disappears tomorrow, the files must remain immediately readable and usable.
Local-first, Git-friendly. It has to be a normal folder I can git init and track right away.
AI-readable without building extra infrastructure. My AI agent already lives in VS Code, so the knowledge base has to sit inside that workspace without me building an extra pipeline.

Obsidian passed everything, because at its core, an Obsidian vault is just a folder of .md files.

Architecture: Vault Structure

vault/
├── Ideas/          # Raw concepts, brainstorming
├── Manifesto/      # Vision, mission, core policies
├── Principles/     # Design rules, engineering guidelines
├── Infrastructure/ # Deployment topology, IaC specs
├── Platform/       # Domain model, API contracts
├── Script/         # Utility scripts, automation, runbooks
├── Skill/          # Patterns, checklists, reusable knowledge
└── Appendix/       # Domain glossary, citations

Every file is a plain .md. Links use [[wiki-link]] syntax. Metadata lives in YAML frontmatter, and Graph View renders the relationships as a visible dependency graph.

Version control is just git init inside the vault folder, then committing every change with a rationale. git log -- "Infrastructure/sharding-strategy.md" shows the full decision history for that topic. Push it to a private GitHub repo and you get backup, an audit trail, and branching for major revisions, all for free.

Key Insight: AI Access Without Building Anything

This is the part that changed everything.

Obsidian vault = a folder of .md files.
VS Code = opens any folder as a workspace.
AI coding agent (running as a VS Code extension) = reads every file in that workspace.

So the "integration" here is: open the vault folder in VS Code.

That's it. No API, no embedding pipeline, no vector database, no chunking strategy. Just plain Markdown files that the AI agent reads natively.

Prompt Patterns I Actually Use

Contextual search + reasoning:

Search all documents that mention our sharding strategy.
Summarize every trade-off we've considered
and tell me which approach we ultimately chose and why.

Gap analysis:

Look at everything in the Infrastructure/ folder
and tell me which architectural decisions are still undocumented,
compared against the template in Skill/.

Drafting from conventions, not from a blank page:

Using the patterns in Skill/go-backend/ and the domain model in Platform/core/,
draft a design doc for a new message consumer
following our established conventions.

Impact analysis via link traversal:

If I change the authentication rule in Principles/auth.md,
trace every file in Platform/ and Skill/ that links to it via [[links]]
and tell me what needs updating.

The AI reads across multiple files, follows [[wiki-links]], understands the relationships, and synthesizes an answer, without me writing a single line of integration code.

Philosophy: Seamless Integration by Design

The pattern here isn't "integrate 3 tools." It's choosing components that already speak the same language.

Obsidian chose Markdown, the most universally readable format in computing, over a proprietary database. Git works with any plain text. AI agents already know how to read files in a VS Code workspace.

Bonus: Obsidian Canvas as a Visual Layer

Obsidian's Canvas is an infinite whiteboard, place document cards, text, media, then draw connections between them.

I use Canvas for:

System architecture sketches: each service as a card, data flow arrows, real specs embedded right on the board
Decision trees: "if we pick X, then Y and Z are affected," with linked evidence
Plan strategy & flow: for planning work, sequencing, and various NEXT4I workflows

Canvas files are Markdown too under the hood (JSON-like structure), so they're Git-versioned and AI-readable as well. Drop a .canvas file into your VS Code workspace and ask the AI to analyze it for circular dependencies or single points of failure.

What I Learned

What makes this system work isn't the technology, it's what I didn't build. No middleware, no proprietary pipeline, no vendor lock-in.

The discipline of plain text + Git + open formats is a feature, not a limitation.

If you're a dev or a small team drowning in scattered documents, try this before jumping to a heavyweight knowledge-management platform. Plain Markdown with a good folder structure will take you further than you'd expect.

Thanks for reading all the way to the end, I'll keep working on more articles like this.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Alternatively, you can register to join NEXT4I the AI-Native Ecosystem I am currently building at: https://go.next4i.com/cta

tags:
#buildinpublic
#secondbrain
#next4i
#obsidian

Originally published at:
https://www.next4i.com/dev-notes/en/obsidian-second-brain-dev-en

How I Built an AI-Readable Second Brain with Obsidian, Git, and a VS Code AI Agent

NEXT4I DEV — Fri, 07 Aug 2026 07:55:45 +0000

When you're a solo founder and lead architect, your knowledge base is your most valuable asset. Lose the thread on why a decision was made, and you spend hours — sometimes days — reconstructing context that you already figured out once.

I want to share the exact setup I use at NEXT4I to turn a folder of Markdown files into a fully searchable, version-controlled, AI-readable knowledge system. No proprietary SaaS, no vendor lock-in, no custom integration work.

This is the Key Highlight of this post: a genuinely useful, generic pattern you can apply to your own projects today. The NEXT4I-specific business logic stays abstracted (per our security rules), but the pattern itself is 100% reusable.

1. The Architecture: Three Layers, Zero Magic

┌─────────────────────────────────────────┐
│           AI Agent (VS Code)            │
│   Reads, Searches, Summarizes, Drafts   │
└──────────────────┬──────────────────────┘
                   │ reads plain .md files
┌──────────────────▼──────────────────────┐
│       Git-tracked Obsidian Vault        │
│  ├── Idea/           (brainstorms)      │
│  ├── Infrastructure/ (architecture docs)│
│  ├── Platform/       (product specs)    │
│  ├── Script/         (automation)       │
│  └── Skill/          (reusable limits)  │
└──────────────────┬──────────────────────┘
                   │ committed & pushed
┌──────────────────▼──────────────────────┐
│         GitHub (remote backup)          │
└─────────────────────────────────────────┘

Layer 1 — The Vault (Obsidian): A folder of interconnected .md files. The key insight is that Obsidian uses plain Markdown with [[wiki-links]] for connections — no database, no proprietary format.

Layer 2 — Version Control (Git): Every vault is a git repo. Every change to any document has a commit message, a timestamp, and a diff. You can git log --oneline -- Idea/ to see the evolution of a concept.

Layer 3 — AI Agent (VS Code Extension): Because the vault is just a file tree of .md files, any AI coding agent that can read a codebase can also read your knowledge base. Point the agent at the vault folder, and it has full context.

2. The Setup: Step-by-Step

Step 1: Create the Vault


mkdir next4i-knowledge

cd next4i-knowledge

mkdir Idea Infrastructure Platform Script Skill

git init

Open this folder in Obsidian: Open folder as vault.

Step 2: Link Everything

Inside a note, link to another note with [[Note Name]]. Obsidian auto-suggests as you type. Over time, this builds a graph you can visualize with Cmd/Ctrl + G.

Pro tip: Create a _INDEX.md in each folder that links to the most important notes. This becomes a human-readable table of contents AND a search anchor for the AI.

Step 3: Add Git Discipline


git add -A && git commit -m "infra: initial sharding strategy decision"

git remote add origin git@github.com:your-org/knowledge-vault.git

git push -u origin main

Treat commit messages like code. Use prefixes: idea:, infra:, platform:, script:, skill:. This makes git log --oneline --grep="infra:" instantly useful.

Step 4: Open in VS Code and Activate the AI


code /path/to/vault

With an AI agent extension active (Copilot, Cline, Cody, etc.), try prompts like:

"Summarize the key architectural decisions in the Infrastructure folder."
"Find any contradiction between documents in /Platform/ and /Infrastructure/."
"Draft a new document in /Idea/ based on the sharding notes in /Infrastructure/."

The agent reads the files as context, just like it would for code.

3. The Design Pattern: Folder-Convention-as-API

Here's the key pattern: your folder structure IS your API.

By keeping a consistent vault structure, both humans and AI know where to look:

Folder	Contains	AI Use Case
`Idea/`	Raw, unstructured thinking	Generate summaries, find related concepts
`Infrastructure/`	System topology, deployment, config	Validate consistency, trace dependencies
`Platform/`	Feature specs, user flows	Draft task tickets, check requirement coverage
`Script/`	Automation, one-liners	Explain what a script does, suggest improvements
`Skill/`	Reusable patterns, checklists	Retrieve relevant patterns for new tasks

This is essentially a convention-based RAG (Retrieval-Augmented Generation) setup without any vector database, embedding pipeline, or chunking strategy. The "chunking" is the natural boundary of each .md file. The "retrieval" is the AI agent's file-reading capability.

4. Why This Beats a Wiki

Wiki / Confluence	This Setup (Obsidian + Git)
Vendor lock-in	Plain `.md` files, portable anywhere
Search is siloed within the tool	VS Code AI searches across the whole vault
No version control (or poor built-in)	Full version control (`git blame`, `git diff`, `git log`)
Hard to automate	Scriptable — `grep`, `sed`, and AI prompts all work
AI needs API integration	AI reads files natively, zero setup required

5. What I Learned

The biggest surprise: The AI agent became better at finding connections in my own notes than I was. It doesn't have recency bias. It doesn't forget what I wrote 8 months ago. It reads everything with equal attention.

The biggest lesson: AI-native doesn't mean "add an AI button." It means design your systems — including your thinking systems — so that AI can participate as a first-class citizen without special plumbing.

I'm building NEXT4I as an AI-native ecosystem from the ground up. If you're interested in following a solo founder's engineering journey — or want early access — join here:
Subscribe NEXT4I or want early access — join here

DEV Community: NEXT4I DEV

Semantic Caching and Live Internet Search: Tools That Let AI Cache Answers to Save Money, and Pull Data From the Web When It Needs To

The short version

Think of your AI as a call-center agent with a notebook

What a cache is, and why it pays off with AI

Three kinds of cache in the AI world

How semantic caching works

So how much can you actually save?

The risks of a semantic cache you need to know first

1. A wrong answer because two questions looked like the same question (false hit)

2. An outdated answer (stale answer)

3. Someone else's answer

4. Questions where users want a new answer

Same question, same context

Live internet search: going out to find real data

The limits of live search

Sort first: cache it or search again?

Takeaway: cache what rarely changes, search what keeps updating, and be careful with caches shared between users

References

1.3M Changes in 10 Days !!! "N-Flow Agentic Inception": The NEXT4I Agentic Workflow

The NEXT4I Micro-batching Architecture for Enterprise Software Implementation

Stats from real runs, last month (wave 1):

Stats from real runs, 7 days ago (wave 2):

Stats from real runs after another Workflow Refactor (wave 3):

Project Estimate

What is N-Flow Agentic Inception?

What happens when the AI gets something wrong?

What I learned from the earlier rounds

Summary

Hybrid Search Explained: Why Vector Search Still Needs Keywords

The short version

What vector search does well

Where similar meaning becomes a problem

Identifiers are not interchangeable

Related policies can have different answers

The right passage may not reach the model

What keyword search adds

A practical hybrid retrieval flow

Combining results without being misled by scores

Normalize and weight scores

Fuse ranks with RRF

A brief note on vector precision

Evaluate the whole evidence path

The goal is evidence, not a smarter-looking stack

Further reading

Chunking in RAG: How to Split Documents Without Losing the Conditions

TL;DR

What is chunking, and why not use the entire file?

One boundary can change the answer

The main approaches and their trade-offs

1. Fixed-size splitting

2. Fixed-size splitting with overlap

3. Recursive splitting

4. Structure-aware splitting

5. Semantic chunking

6. Parent-child retrieval and contextual chunking

Match the approach to the document

The Thai tourism PDF that changed my priorities

How large should chunks and overlap be?

How to tell whether chunking works

The goal is usable evidence

From Ctrl + F to AI-Powered Search

How Do Computers Find Content and Answer Questions from Files?

1. The Fundamental Gap: Humans Read Meaning, Computers Read Bytes

2. Era 1: Exact Character Matching (Lexical / Exact Keyword Search)

2.1 Linear Scan (Ctrl + F or grep)

2.2 Inverted Index (The Backbone of Search Engines)

Limitations of Lexical Search: Why Exact Matching Falls Short

3. Era 2: Searching by Meaning (Semantic / Vector Search)

3.1 What is Text Embedding? (Turning Language into Coordinates)

3.2 Measuring Proximity Mathematically (Similarity Metrics)

4. Why Not Convert an Entire 100-Page File into a Single Vector?

The Solution: The Art of Chunking (Chunking Strategy)

5. From Finding Content to Answering Questions: The 3-Step Pipeline

6. Comparison: 4 Paradigms of Document Retrieval

7. Conclusion: What Should Real-World Systems Use?

RAG Architecture Beyond the Demo: Retrieval, Thai Chunking, and Production Boundaries

The three limits RAG is trying to address

Knowledge boundaries

Context and cost boundaries

2.1 Linear Scan (`Ctrl + F` or `grep`)

A generic `SKILL.md` example