DEV Community: Pingredsai

LocalRAG Desktop: a Docker-free local document Q&A app for Windows

Pingredsai — Wed, 21 Oct 2026 01:52:12 +0000

Most local RAG demos still start with the same obstacle:

cp .env.example .env
docker compose up -d
docker exec localrag-ollama ollama pull ...

That is fine for developers. It is not fine for a lawyer, accountant, or small-business owner who just wants to search their own documents.

So I rebuilt the user-facing setup around a different question:

Can a Windows user run a private document Q&A system without installing Docker, WSL, or Python?

This beta is my answer.

What it does

LocalRAG Desktop is a local document knowledge base:

upload text-based PDF, Word, TXT, or Markdown files
parse and chunk documents locally
create embeddings with a local Chinese embedding model
retrieve relevant passages from Qdrant
generate an answer with a local Qwen model
show the source file and chunk used for the answer

Everything stays on the local machine. The services bind to 127.0.0.1 only.

What changed in this beta

The earlier version was a Docker Compose stack. The Windows beta now uses:

native Windows Qdrant
native llama.cpp CPU runtime
Qwen2.5 1.5B for generation
bge-small-zh-v1.5 for embeddings
a packaged FastAPI backend
a native Windows launcher

The user flow is now:

Download ZIP
→ verify SHA256
→ extract to a data drive
→ double-click the launcher
→ wait for the local services
→ use the local web UI

The launcher performs environment checks, starts the services, waits for health checks, and opens the browser.

Why I did not just wrap Docker

A wrapper can hide Docker commands, but it cannot remove the Docker installation, WSL2 requirement, virtualization settings, admin permissions, and large image downloads.

For a technical user, Docker is convenient. For a non-technical user, it is a second product they have to install before they can try the first one.

The goal here was to remove that second product entirely. A real one-click deployment is not a script that hides Docker commands. It is a setup where the user never needs Docker in the first place.

Architecture

Windows launcher
├── Qdrant native
├── llama.cpp generation server
├── llama.cpp embedding server
├── packaged FastAPI backend
├── local web UI
└── local data directory

Security-related choices:

services bind to 127.0.0.1
the local API uses a random session token for non-health endpoints
CORS is restricted to local origins
diagnostics exclude documents, models, vector data, and access tokens
no default telemetry

What I verified

On the development machine:

native Qdrant and llama.cpp started successfully
a sample document was indexed into 3 chunks
a policy question returned the correct answer with a source
streaming answers worked
the native stack continued working after Docker containers were stopped
the packaged backend ran without system Python
no-token API requests returned 401

This is still a beta. It has not yet been tested on a wide matrix of clean Windows machines.

Download and try it

Release page:

https://github.com/Pingredsai/local-rag/releases/tag/v0.3.0-beta.1

Direct ZIP:

https://github.com/Pingredsai/local-rag/releases/download/v0.3.0-beta.1/LocalRAG-Beta-v0.3.zip

SHA256:

f35d424fedd98e9f8b143e4a72c6dfd2ee350c03c3338582449c2c479a287035

Current limitations

Windows 10 22H2 / Windows 11 x64 only
8GB RAM recommended for the included 1.5B model
text-based PDFs only; OCR is not included
CPU inference in this beta
single knowledge base
no multi-user permissions yet
the beta is not code signed yet, so Windows may show an “Unknown publisher” or SmartScreen warning

The ZIP contains a Windows prompt guide and a diagnostic tool. Please do not disable Defender, SmartScreen, or Application Control to run it.

Feedback I am looking for

I am mainly interested in:

installation/startup failures
unclear Windows prompts
first-answer latency
document types that fail to parse
whether source references are clear enough
whether the setup feels simple enough for a non-technical user

If you test it, please include your Windows version, RAM, the step where you got stuck, and the diagnostic ZIP generated by the launcher.

Contact: siquhjh@163.com

I will keep the technical limitations and test results honest as the project evolves.

I Built a Self-Hosted RAG Stack That Runs Without Cloud AI APIs

Pingredsai — Fri, 16 Oct 2026 02:58:56 +0000

Many teams want AI to answer questions from internal documents, but they cannot send those documents to a cloud service.

Legal teams have confidential case files. Clinics have patient data. Finance teams have client working papers. Manufacturers have proprietary process documentation. Some organizations operate on isolated networks.

For these teams, a cloud-first RAG stack is not an option.

I built LocalRAG to verify a simple idea:

A document question-answering system should be able to run inside a controlled environment, use local models, and return source references for every answer.

The project is open source and the current v0.1.0 release has been verified end to end.

Repository: https://github.com/Pingredsai/local-rag
Release: https://github.com/Pingredsai/local-rag/releases/tag/v0.1.0

What LocalRAG does

LocalRAG allows a user to upload documents and ask questions against them.

The verified flow is:

Upload a PDF, DOCX, TXT, or Markdown document.
Parse and split the document into chunks.
Generate embeddings with a local bge-m3 model.
Store vectors in Qdrant.
Embed the user's question.
Retrieve the most relevant chunks.
Generate an answer with a local LLM.
Return the answer with source references.

The application does not call a cloud AI API at runtime.

Initial installation still requires network access to pull Docker images and Ollama models. After that, the stack can run inside a local or air-gapped environment.

Architecture

LocalRAG uses three main containers:

Browser UI
    |
    v
FastAPI Backend
  - document parsing
  - chunking
  - RAG orchestration
  - REST API and static frontend
    |
    +--> Qdrant
    |      vector storage and search
    |
    +--> Ollama
           bge-m3 embeddings
           local LLM generation

Why these components?

Layer	Choice	Reason
Backend	FastAPI	Easy to inspect, modify, and deploy
Vector database	Qdrant	One container, clean API, persistent storage
Embedding runtime	Ollama + bge-m3	Local, multilingual, avoids a separate Python model stack
Generation runtime	Ollama + Qwen2.5	Easy model switching from 1.5B to larger models
Frontend	Plain HTML/CSS/JS	No frontend build step
Deployment	Docker Compose	Reproducible and easy to run locally

I deliberately did not build a new model runtime, vector database, or agent engine.

The interesting engineering problem is not reinventing Ollama. It is making the complete workflow reproducible, observable, and usable in a private deployment.

Verified test

I tested the current release with:

Docker Desktop 4.95.0
Ollama 0.11.3
Qdrant
bge-m3 for embeddings
qwen2.5:3b for generation

The test document was a fictional employee handbook.

The system:

Indexed it into three chunks.
Received the question: “How many days of annual leave do full-time employees receive?”
Retrieved the relevant policy text.
Returned “15 days.”
Displayed the source document and similarity score in the UI.

A health check returns:

{
  "status": "ok",
  "qdrant": "ok",
  "ollama": "ok",
  "llm_model": "qwen2.5:3b",
  "embed_model": "bge-m3"
}

The repository includes screenshots and a 42-second demo video.

Running it

git clone https://github.com/Pingredsai/local-rag.git
cd local-rag
cp .env.example .env
docker compose up -d

docker exec localrag-ollama ollama pull qwen2.5:3b
docker exec localrag-ollama ollama pull bge-m3

Open:

http://localhost:8080

Four problems I ran into

1. The official Ollama image was too large

The latest Docker image is convenient, but the unpacked image can approach 9 GB. In a slow network environment, that is a serious deployment problem.

I pinned the project to a smaller compatible Ollama release and documented the deployment tradeoff.

This is a reminder that runtime size and download behavior are part of the product experience, not just infrastructure details.

2. A slow Docker registry is not the same as a broken Docker installation

Docker Hub was extremely slow in my environment. The solution was a combination of a mirror, resumable pulls, and local model import in some cases.

I did not hard-code a third-party mirror into the project. Instead, the deployment notes describe the approach so operators can replace it.

3. The original streaming response did not show sources

The first implementation returned text from the streaming endpoint, while sources were only available from the non-streaming endpoint.

That made the UI look like a normal chatbot. The source references, which are essential in a business system, were invisible.

I changed the streaming endpoint to return source metadata in a response header. The frontend reads the header, streams the answer, and renders the source cards when the answer completes.

4. A 1.5B model was not reliable enough for the demo

The smaller model worked on short context but produced inconsistent responses on slightly longer documents.

I switched the demo to qwen2.5:3b and tightened the prompt and generation options.

Local AI is not automatically better just because it is smaller. Model size must match the hardware, use case, and required answer quality.

Current limitations

The first version is intentionally narrow.

It does not include:

OCR for scanned PDFs
Multi-tenant permissions
Multiple isolated knowledge bases
Audit logging
A reranker
Visual agent orchestration

The goal of v0.1.0 is to make the core path dependable:

upload -> index -> retrieve -> answer -> show sources

I would rather state those boundaries clearly than imply that the project is already a production system.

What I plan to improve next

The next work is focused on the existing engineering path:

Improve PDF and DOCX parsing
Evaluate and refine chunking strategies
Improve retrieval quality and testing
Polish streaming answers and source rendering
Add clearer checks for ports, models, and data directories
Document backup, restore, and upgrade paths
Continue reducing deployment friction

Before adding more modules, I want the core path to be dependable:

upload -> index -> retrieve -> answer -> show sources

Feedback and contributions

LocalRAG is an open-source project under active development.

If you work with local inference, RAG, or private deployment, feedback is welcome through GitHub issues and pull requests.

Repository:

https://github.com/Pingredsai/local-rag

The most useful reports are reproducible bugs, deployment issues, retrieval quality observations, and documentation improvements.

Is Your Phone's LLM 37x Too Slow? Check This One Compiler Flag

Pingredsai — Mon, 12 Oct 2026 02:14:50 +0000

Is Your Phone's LLM 37x Too Slow? Check This One Compiler Flag

Part 3 of a series.

Part 1: found the missing -march (2.5x faster).

Part 2: ruled out scheduling and frequency scaling (negative results).

Part 3: found the real culprit — -O0.

1. The result first

If you build llama.cpp for Android with a Gradle debug variant, you get an -O0 binary. It is 37x slower than Release.

Metric	Debug (-O0)	Release (-O3)	Improvement
Generation	0.5 tok/s	18.5 tok/s	37x
First token	1863 ms	52 ms	36x
Prefill, 43 tokens	42 s	0.556 s	75x
Model load	24 s	3.9 s	6x

Device: Pixel 4 (Snapdragon 855, 2019). Model: Qwen2.5-1.5B-Instruct Q4_K_M.
Nothing changed except two build config lines.

2. How I missed something this basic

What I had already checked (parts 1 and 2):

Check	Verdict
Missing `-march`	✅ Real, fixed — 2.5x on prefill
Threads on little cores	❌ Ruled out
OpenMP not threading	❌ Ruled out
mmap pages evicted	❌ Ruled out
CPU frequency throttling	⚠️ Real, but only ~20%
Optimisation level	🔴 Never checked

I spent days on how the CPU was running, and never checked whether the compiled code was optimised at all.

Worse: I had disassembled the binary and confirmed sdot (the ARM dotprod instruction) was present, and concluded "the build is fine".

sdot only tells you the instruction set. It says nothing about the optimisation level. At -O0 the compiler still emits sdot — it just does nothing else.

3. What turned it around

A reader on DEV left this comment:

Build type. A Gradle debug variant hands CMake CMAKE_BUILD_TYPE=Debug unless you override it, and ggml at -O0 with asserts on is 20-50x slower than release.

sdot showing up in the disassembly does not tell you the optimisation level.

Add -DCMAKE_BUILD_TYPE=Release to that arguments list, or benchmark the release variant.

I would bet the build type.

He called it exactly.

4. Verifying takes one command

Look at what the build system actually passes to the compiler:

find app/.cxx -name build.ninja

grep -o '\-O[0-3s]' build.ninja | sort -u

My result:

(no output)

Not a single -O flag.

And:

CMAKE_BUILD_TYPE:STRING=Debug

So ggml, llama.cpp and my own JNI layer were all compiled with zero optimisation.

5. The fix

Change 1: force optimisation in `CMakeLists.txt`

This must come before add_subdirectory(llama.cpp), otherwise it won't affect ggml.

set(CMAKE_C_FLAGS_DEBUG   "-O3 -g" CACHE STRING "" FORCE)
set(CMAKE_CXX_FLAGS_DEBUG "-O3 -g" CACHE STRING "" FORCE)

set(CMAKE_C_FLAGS_RELWITHDEBINFO   "-O3 -g -DNDEBUG" CACHE STRING "" FORCE)
set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "-O3 -g -DNDEBUG" CACHE STRING "" FORCE)

Change 2: tell Gradle to use Release

externalNativeBuild {
    cmake {
        arguments += listOf(
            "-DUSE_LLAMA_CPP=ON",
            "-DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16",
            "-DCMAKE_BUILD_TYPE=Release"    // key
        )
    }
}

Both, because AGP can override what you pass via Gradle, while FORCE in CMakeLists wins.

6. Verify the fix actually landed

Don't trust the flag you passed. Check what happened.

Check 1: build.ninja

grep -c '\-O3' build.ninja
grep '\-DNDEBUG' build.ninja

Mine:

-O3 appears 82 times
-DNDEBUG present

Check 2: binary size (the fastest smell test)

Library	Debug (-O0)	Release (-O3)
`libggml-cpu.so`	3.92 MB	0.95 MB
`libllama.so`	7427 KB	3404 KB

A large drop in size means optimisation actually happened — inlining, dead code elimination and constant folding all shrink the binary.

This is my favourite quick check: you don't need to disassemble anything, just look at the file size.

7. Why it's this dramatic

What -O0 turns off:

Optimisation	-O0	-O3
Function inlining	❌	✅ on hot paths
Loop unrolling	❌	✅
Auto-vectorisation	❌	✅
Register allocation	conservative	aggressive
Dead code elimination	❌	✅
Asserts	kept	removed (`-DNDEBUG`)

For LLM inference the killers are auto-vectorisation and inlining.

Quantised matrix multiplication — 99% of the compute — is made of tight loops. At -O0 those loops are not vectorised, so every element is computed and fetched individually. Tens of times slower is expected.

That explains every anomaly we saw:

CPU at full clock ✅
threads on big cores ✅
dotprod instructions present ✅
utilization: 1% ❌

The CPU was running full speed on unoptimised element-by-element code.

8. How to avoid this

If you're building llama.cpp for Android

# after building, check the optimisation flags
find app/.cxx -name build.ninja -exec grep -c '\-O3' {} \;

# or just look at the binary size
ls -lh app/build/intermediates/cxx/*/obj/arm64-v8a/*.so

libggml-cpu.so around 1 MB → optimised. Around 3-4 MB → probably -O0.

A general debugging order

1. Optimisation level (-O0/-O1/-O2/-O3)     <- start here
2. Architecture flags (-march / -mcpu)
3. Thread count and affinity
4. Memory and IO
5. The algorithm itself

I worked through step 5 before coming back to step 1.

9. Lessons

1. The more basic something is, the easier it is to skip.

We checked CPU architecture, thread scheduling, frequency, bandwidth — and never checked whether the code was optimised. Because "basic things don't look like the problem".

2. "The instruction exists" ≠ "the build is right".

Instruction set and optimisation level are different axes.

3. Binary size is an underrated signal.

A big size change is the simplest evidence that optimisation happened, and it's much faster than reading disassembly.

4. One good comment can beat days of solo debugging.

That reader spotted it in a sentence. And he could only do that because I published the negative results — if I'd only written about the successes, he wouldn't have known where I was stuck.

10. Where it stands now

18.5 tok/s, fully offline, on a 2019 phone.

A 20-character sentence finishes in about a second
First token in 52 ms — essentially no perceptible delay
1.3 GB RSS

For a five-year-old phone, that's genuinely usable.

Part 1: Why Your Phone Runs LLMs 80x Slower Than It Should

Part 2: Threads on Big Cores, CPU at Full Clock — Still Slow

Part 3: this post
Code: https://github.com/Pingredsai/local-ai-android

On-Device Inference Debugging (Part 2): Threads on Big Cores, CPU at Full Clock — Still Slow

Pingredsai — Sat, 10 Oct 2026 11:02:50 +0000

On-Device Inference Debugging (Part 2): Threads on Big Cores, CPU at Full Clock — Still Slow

Part 1: Why Your Phone Runs LLMs 80x Slower Than It Should
This part focuses on thread scheduling and CPU frequency scaling.

1. Where part 1 left off

In part 1 I benchmarked Qwen2.5-1.5B on a Pixel 4 (Snapdragon 855) and measured 0.5 tok/s — 60-120x off the theoretical limits for that SoC.

That investigation found one real bug: llama.cpp cross-compilation passes no -march by default, so quantized matmul falls back to a slow path. Fixing it cut prefill from 114 s to 45 s.

But throughput was still 0.5 tok/s. The biggest hole was filled; the problem remained.

2. A reader points in a new direction

After publishing on DEV, a reader offered two traps that look like a slow model but aren't:

Scheduling: a llama.cpp server whose child processes were parked on efficiency cores under load — a 3-second transcription became 37 seconds. Same binary, same flags; the only difference was who launched it.

Storage: running off a USB stick, the whole model is re-read at every start — 2.6 GB on a 34 MB/s drive means over a minute before the first token.

And the line that stung:

The number that looks like "the model is slow" is almost never the model.

Both point the same way: maybe the problem isn't what is being computed, but where and how fast.

3. Know your silicon first: 1+3+4

for c in /sys/devices/system/cpu/cpu[0-9]*; do
  echo "$(basename $c) $(cat $c/cpufreq/cpuinfo_max_freq)"
done

Cores	Max freq	Type
cpu0–3	1.79 GHz	A55 little
cpu4–6	2.42 GHz	A76 mid
cpu7	2.84 GHz	A76 big

Classic 1+3+4. If inference threads land on cpu0–3, you lose 3-4x immediately.

4. Test 1 — which core are the threads actually on?

Method: read /proc, no tooling required

Field 39 of /proc/<pid>/task/<tid>/stat is the CPU the thread is currently running on.

pid=$(pidof com.example.localai)

for t in /proc/$pid/task/*; do
  tid=$(basename $t)
  comm=$(cat $t/comm)
  cpu=$(awk '{print $39}' $t/stat)
  ticks=$(awk '{print $14+$15}' $t/stat)
  echo "$tid $comm cpu=$cpu ticks=$ticks"
done

Sample repeatedly — a single snapshot tells you nothing.

Result: all on big cores. Scheduling is not the problem.

main thread 9556:  cpu5 -> cpu5 -> cpu5 -> cpu3 -> cpu5 -> cpu4
19524 openmp_worker  cpu6
19525 openmp_worker  cpu7   <- big core
19526 openmp_worker  cpu5

All four inference threads stayed on cpu4–7. None touched the little cores.

Also confirmed OpenMP is genuinely working:

19524 openmp_worker   5607 ticks
19525 openmp_worker   5606 ticks
19526 openmp_worker   5602 ticks
9556  DefaultDispatch 12132 ticks (main)

n_threads=4 → 3 workers + 1 main thread. Exactly as configured.

Verdict: "threads parked on efficiency cores" does not happen on this device. Ruled out.

5. Test 2 — is the CPU being throttled?

The symptom: frequency swinging wildly

cpu4:  710 ~ 1920 MHz   (max 2419)   <- as low as 29%
cpu5: 1612 ~ 2419 MHz   (max 2419)
cpu6:  710 ~ 2419 MHz   (max 2419)
cpu7: 1612 ~ 2016 MHz   (max 2841)   <- never above 71%

First rule out thermals

adb shell dumpsys battery | grep temperature
# temperature: 339  ->  33.9 C

33.9 °C. Not thermal.

The culprit is schedutil, Android's default governor. It scales frequency based on utilization estimates — and LLM inference is bursty: small work units, back to back. The estimate lags, the frequency hunts, and you get exactly this oscillation.

The decisive experiment: pin everything

# root required; SELinux Enforcing blocks a direct write, so chmod first
for c in 0 1 2 3 4 5 6 7; do
  g=/sys/devices/system/cpu/cpu$c/cpufreq/scaling_governor
  chmod 666 "$g"
  echo performance > "$g"
done

Confirmed pinned: cpu0-3 @1785, cpu4-6 @2419, cpu7 @2841 MHz.

Result: only 20% faster

Metric	schedutil	performance	Delta
Generation	0.5 tok/s	0.6 tok/s	+20%
First token	1863 ms	1619 ms	−13%

20%. Throttling is real, but it is not the dominant factor.

6. Updated elimination table

Hypothesis	Verdict
Missing ARM optimization (`-march`)	✅ Real, fixed — 2.5x on prefill
Threads scheduled onto little cores	❌ Ruled out
OpenMP not actually threading	❌ Ruled out
mmap pages evicted	❌ Ruled out (`LLAMA_LOAD_MODE_MLOCK` changed nothing)
CPU frequency throttling	⚠️ Real, but only ~20%

7. What's left

CPU at full clock (2419/2841 MHz)
threads on big cores (cpu4-7)
dotprod instructions present (verified by disassembly)
        |
        v
Utilization of peak compute capacity: ~1%

The arithmetic:

per token: 1.5B params x 2 = 3 GFLOPs
measured: 1.67 s per token
effective: 1.8 GFLOPS
peak: ~182 GFLOPS (4x A76 at full clock)
utilization: 1%

And memory:

per token reads all weights = 1.06 GB
1.06 GB / 1.67 s = 635 MB/s
available bandwidth: 17-34 GB/s
utilization: 2-4%

Neither the CPU nor the memory bus is anywhere near saturated. It's just slow.

The remaining explanation points inside ggml's quantized kernels on this platform — memory access patterns, cache behaviour, or OpenMP barrier overhead on small work units.

I can't fix that from configuration. It stays on the list as the next thing to dig into.

8. Why this post is about method, not answers

The investigation produced no final answer — but the process is reusable.

Zero-dependency diagnostics

awk '{print $39}' /proc/<pid>/task/<tid>/stat      # current CPU per thread
awk '{print $14+$15}' /proc/<pid>/task/<tid>/stat  # cumulative CPU time per thread
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq   # live frequency
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor   # governor

Works on any Android device. No extra tooling.

Negative results are results

Ruling out A, B and C is progress. It shrinks the search space from "anything" to "inside ggml's kernels".

If I only published the successful fix, you'd think this was a clean path. Real engineering is mostly wrong hypotheses and the value is eliminating them quickly.

Compute the ceiling first

Most people say "my model is slow" without quantifying it. Work out the memory-bandwidth ceiling and the compute ceiling, then measure. Knowing the gap is 60-120x tells you the problem isn't "needs tuning" — it's "something isn't running at all".

9. Restore your environment

Item	Restored to
CPU governor	`schedutil`
SELinux	`Enforcing`
stay-awake flag	`false`
screen timeout	60000 ms
temp scripts	deleted

Don't leave side effects on a test device — especially SELinux.

10. Next

Next up: ggml quantized kernel efficiency on ARM.

Which code path does Q4_K_M take?
What's the cache miss rate?
How much does the OpenMP barrier cost per generated token?

If you've dug into this, I'd love to hear it.

Part 1: Why Your Phone Runs LLMs 80x Slower Than It Should
Part 2: this post
Code: https://github.com/Pingredsai/local-ai-android

Why Your Phone Runs LLMs 80x Slower Than It Should (And What I Found)

Pingredsai — Fri, 09 Oct 2026 11:45:01 +0000

Why Your Phone Runs LLMs 80x Slower Than It Should (A Debugging Log)

Tags: on-device inference / llama.cpp / Android / performance
Status: draft

I ran Qwen2.5-1.5B-Instruct Q4_K_M fully offline on a Google Pixel 4 (Snapdragon 855, 2019).

Metric	Measured
Generation	0.5 tok/s
First token	1.9 s
Peak RSS	1.3-2.0 GB
CPU	402% (4 threads saturated)

Theoretical limits say this device should do 30-60 tok/s.
It is 60-120x slower. And the CPU is already maxed out.

Here is the full debugging log — including three mistakes I made.

1. Do the math first

Memory bandwidth bound:

model size      1.06 GB
SD855 bandwidth ~34 GB/s
one token reads all weights once
-> theoretical ceiling ~32 tok/s

Compute bound:

1.5B params x 2 FLOPs = 3 GFLOPs per token
SD855 CPU peak ~182 GFLOPS
-> theoretical ceiling ~60 tok/s

Both ceilings land in the 30-60 tok/s range. Measured: 0.5.

Knowing the gap (60-120x) tells you what to suspect.

2. Suspect 1: thread count? (ruled out)

adb shell top -b -n 1 -o PID,%CPU,%MEM,ARGS

PID    %CPU   %MEM   ARGS
17834  402%   24.3%  com.example.localai

402% — four threads fully saturated. Not a threading problem. Ruled out.

3. Suspect 2: missing ARM optimization? (HIT)

This is the valuable one.

llama.cpp does its heavy lifting in ggml. From ggml/src/ggml-cpu/CMakeLists.txt:

if (GGML_NATIVE)
    # probe host CPU, add -mcpu=native
    ...
else()
    if (GGML_CPU_ARM_ARCH)
        list(APPEND ARCH_FLAGS -march=${GGML_CPU_ARM_ARCH})   # <-- key
    elseif(GGML_CPU_ALL_VARIANTS)
        ...
    # neither set -> NO -march at all
endif()

When cross-compiling for Android:

GGML_NATIVE is OFF (can't probe the target)
GGML_CPU_ARM_ARCH defaults to empty
Result: no -march is passed at all

Compiler falls back to the baseline ARMv8-A — no dotprod, no fp16.

Quantized matrix multiplication (99% of LLM compute) degrades to a slow path.

Fix:

arguments += listOf(
    "-DUSE_LLAMA_CPP=ON",
    "-DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16"
)

Result: prefill went from 114 s to 45 s — 2.5x faster.

Better. But not enough.

Verify the optimization actually landed

Don't trust the flag. Disassemble:

llvm-objdump -d libggml-cpu.so | grep sdot

18f558: 4e829420    sdot   v0.4s, v1.16b, v2.16b

sdot is there. dotprod is active. Ruled out.

Still 0.5 tok/s.

4. Suspect 3: mmap pages getting evicted? (ruled out)

Android reclaims mmap'ed pages under memory pressure. Then every weight read falls back to flash:

flash read ~1.5 GB/s
1.06 GB / 1.5 GB/s ~ 700 ms/token

Right order of magnitude for the 2 s/token we saw.

So I forced the model to stay resident:

mparams.load_mode = LLAMA_LOAD_MODE_MLOCK;

Result: still 0.5 tok/s. Ruled out.

5. Three mistakes I made (worth avoiding)

Mistake 1: assuming `llama_batch_get_one` pins position to 0

I claimed: "llama_batch_get_one() sets pos to 0, so every token recomputes the whole sequence."

Then I hand-rolled llama_batch_init() with explicit positions.

What actually happened:

Reading the header: llama_batch_get_one's pos is nullptr — llama.cpp assigns positions automatically. Not 0.
My hand-rolled version hung the prefill (43 tokens, never returned).
Reverting to llama_batch_get_one fixed it.

Lesson: for perf issues, suspect compiler flags and memory access first — not a detail you "remember".

Mistake 2: using a removed API

llama_kv_cache_clear(g_ctx);   // removed in current llama.cpp

Now:

llama_memory_clear(llama_get_memory(g_ctx), true);

Mistake 3: guessing struct field names

mparams.use_mmap  = false;   // fields no longer exist
mparams.use_mlock = true;

Merged into one enum:

mparams.load_mode = LLAMA_LOAD_MODE_MLOCK;

Common lesson: llama.cpp's API moves fast. Always read include/llama.h. Never write from memory.

6. Where it landed

After ruling out three suspects:

3 GFLOPs per token
2 s per token measured
-> ~0.8% of peak CPU utilization

And it is not a bandwidth wall (~500 MB/s measured vs ~34 GB/s available).

So compute efficiency itself is the problem — beyond "wrong config". Could be Android CPU scheduling, OpenMP synchronization overhead, or platform-specific behavior in llama.cpp.

That is not a one-day fix. And it should not block shipping.

7. What I did about it

Ship it and state the number honestly — README says "Pixel 4: 0.5 tok/s"
Explain the limitation — old SoC + large model
Ask the community for more data — Snapdragon 8 Gen 2/3 results welcome
Move performance work to v2 instead of blocking v1

8. Transferable lessons

Lesson	Why
Check compiler flags first	`-march` / dotprod issues are the #1 cause of slow on-device inference
CPU saturated ≠ efficient	402% usage can mean you're running the slowest implementation
Compute theoretical limits	The gap tells you what to suspect
Verify with disassembly	Flags set ≠ instructions generated
Read the header, don't recall it	llama.cpp API churns fast
Set a stop-loss for the project	Perf work must not block shipping forever

Appendix: environment

Item	Value
Device	Google Pixel 4 (Snapdragon 855, 6GB, Android 13)
Model	Qwen2.5-1.5B-Instruct-Q4_K_M.gguf (1065 MB)
llama.cpp	commit 8a1a9b5 (2026-10-09)
NDK	30.0.16248370
CMake	4.1.2
Build flags	`-DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16`, `-DUSE_LLAMA_CPP=ON`
Runtime	`n_threads=4`, `n_ctx=2048`, `load_mode=MLOCK`

Source code

Full implementation (Kotlin + JNI + llama.cpp + Jetpack Compose), MIT licensed:

https://github.com/Pingredsai/local-ai-android

Running the same model on newer hardware? Send me your numbers — that is exactly what this project needs.

DEV Community: Pingredsai

LocalRAG Desktop: a Docker-free local document Q&A app for Windows

What it does

What changed in this beta

Why I did not just wrap Docker

Architecture

What I verified

Download and try it

Current limitations

Feedback I am looking for

I Built a Self-Hosted RAG Stack That Runs Without Cloud AI APIs

What LocalRAG does

Architecture

Why these components?

Verified test

Running it

Four problems I ran into

1. The official Ollama image was too large

2. A slow Docker registry is not the same as a broken Docker installation

3. The original streaming response did not show sources

4. A 1.5B model was not reliable enough for the demo

Current limitations

What I plan to improve next

Feedback and contributions

Is Your Phone's LLM 37x Too Slow? Check This One Compiler Flag

Is Your Phone's LLM 37x Too Slow? Check This One Compiler Flag

1. The result first

2. How I missed something this basic

3. What turned it around

4. Verifying takes one command

5. The fix

Change 1: force optimisation in CMakeLists.txt

Change 2: tell Gradle to use Release

6. Verify the fix actually landed

Check 1: build.ninja

Check 2: binary size (the fastest smell test)

7. Why it's this dramatic

8. How to avoid this

If you're building llama.cpp for Android

A general debugging order

9. Lessons

10. Where it stands now

On-Device Inference Debugging (Part 2): Threads on Big Cores, CPU at Full Clock — Still Slow

On-Device Inference Debugging (Part 2): Threads on Big Cores, CPU at Full Clock — Still Slow

1. Where part 1 left off

2. A reader points in a new direction

3. Know your silicon first: 1+3+4

4. Test 1 — which core are the threads actually on?

Method: read /proc, no tooling required

Result: all on big cores. Scheduling is not the problem.

5. Test 2 — is the CPU being throttled?

The symptom: frequency swinging wildly

First rule out thermals

The decisive experiment: pin everything

Result: only 20% faster

6. Updated elimination table

7. What's left

8. Why this post is about method, not answers

Zero-dependency diagnostics

Negative results are results

Compute the ceiling first

9. Restore your environment

10. Next

Why Your Phone Runs LLMs 80x Slower Than It Should (And What I Found)

Why Your Phone Runs LLMs 80x Slower Than It Should (A Debugging Log)

1. Do the math first

2. Suspect 1: thread count? (ruled out)

3. Suspect 2: missing ARM optimization? (HIT)

Verify the optimization actually landed

4. Suspect 3: mmap pages getting evicted? (ruled out)

5. Three mistakes I made (worth avoiding)

Mistake 1: assuming llama_batch_get_one pins position to 0

Mistake 2: using a removed API

Mistake 3: guessing struct field names

6. Where it landed

7. What I did about it

8. Transferable lessons

Appendix: environment

Source code

Change 1: force optimisation in `CMakeLists.txt`

Mistake 1: assuming `llama_batch_get_one` pins position to 0