PetniHunter: I Invented 929 Fake Bangla Books, Films and Writers. A 4B Model Believed 96% of Them.

This is my submission for the DEV Hacktoberfest "Ghost in the Machine" challenge.

What I Built

Anyone who knows me, knows just how much I love books. I read books in both English and in my mother-tongue Bangla. But Bangla literature and LLMs have a problem. LLMs make up or "hallucinate" a lot of Bangla book and author names. I'll give you an example: I made up a Bengali film called ধূসর নদীর ওপারে ("Beyond the Grey River"). It does not exist. I asked an open-weight model, Qwen3.5-4B, who directed it.

"The director was Satyajit Ray. It was released in 1958, and it starred Sumitra Mitra, Tarun Mitra and Shibajit Chattopadhyay."

No hesitation, a year, a cast. Every word of it is invented.

That is a ghost entity: a person, book or film that does not exist, but that a language model describes with total confidence. In Bangla, where models have far less text to learn from, the ghosts are everywhere. And the folklore I grew up with has the perfect word for something that looks like it is there but is not: the পেত্নী (petni), a ghost from Bengali stories.

PetniHunter is a guard that sits in front of a Bangla chatbot. It does not just detect ghosts. It acts. You ask about a writer's birth year, a book's author, or a film's director. It checks the entity against Bangla Wikipedia and Wikidata, then does one of three things:

You can type a question in your own words. The web page shows the raw model's answer, the Wikipedia check and the action side by side, with English translations.

Demo

🎥 Watch the 2-minute demo video

🌐 Live demo: https://petnihunter-1.onrender.com/

To try it, type any Bangla question about a writer's birth year, a book's author, or a film's director, or click one of the example buttons. Good ones to start with:

• 'নিশিগন্ধার দিনলিপি' উপন্যাসটি কে লিখেছেন? (an invented book: it refuses)
• রবিন্দ্রনাথ ঠাকুর কত সালে জন্মান? (a misspelling of Tagore: it reads it as the poet and says so)
• রবীন্দ্রনাথ ঠাকর কোন সালে জন্মগ্রহণ করেন? (a typo: it asks "did you mean…?")

Code

https://github.com/ayeshamashiat/PetniHunter (README, all raw outputs for every system, scripts to reproduce the experiment.)

git clone https://github.com/ayeshamashiat/PetniHunter && cd PetniHunter
python3 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
cp .env.example .env     # add your TINKER_API_KEY
python -m petnihunter.app

from petnihunter import answer
answer("'নিশিগন্ধার দিনলিপি' উপন্যাসটি কে লিখেছেন?")   # -> refuses, honestly

How I Built It

The open-source AI at the core: Qwen3.5-4B (open weights), run, measured and fine-tuned through Tinker. It reads the user's question, is the baseline whose failures I measured, and was the model I tried to fix by fine-tuning.

First, how bad is it?

I built a test set from Wikidata: real Bangla writers, books and Bengali-language films, plus 929 invented ones. Following Meta's HalluLens benchmark, the invented names came from two sources: names a model invented (GeneratedEntities) and names I spliced from parts of two real ones (MixedEntities). Any name that matched a real Bangla Wikipedia page was thrown out.

I measured the False Acceptance Rate, the share of ghosts a system answered as if they were real, using HalluLens's definition so the number means something outside this project.

The plain model accepted 96.3% of the ghosts. For comparison, HalluLens reports 94.7% for Mistral-7B on English. Same ballpark.

It is not only the ghosts. Asked who wrote the real novel Pather Panchali, it said Rabindranath Tagore. (It is Bibhutibhushan Bandyopadhyay.) Across the real entities in my test set, the plain model got 3.7% right.

Two obvious fixes, and why they both failed

Fix 1: tell it to admit when it is unsure. The fake-name rate fell to 18.2%. But the model now refused 79.3% of real entities. It had learned to say "I don't know" about nearly everything.

Fix 2: fine-tune it to say "I don't know". I used a refusal-aware recipe in the style of R-Tuning: ask the base model every training question, keep the true answer where it was right, and train on "I don't know" everywhere else. I ran LoRA fine-tuning on Tinker.

Result: 0.0% fake names accepted, 100% of real names refused. A model that never answers has no hallucinations. It also has no use.

Why? Of 2,427 training targets, only 35 were real answers the base model actually knew. A 4B model knows almost nothing about obscure Bangla writers, so there was nothing to teach it to tell "I know this" from "I don't." Refusing everything was the best it could do. I did not expect this, and I am reporting it because it is the most useful result in the project.

This is why the scoreboard needs two sides. I used the SimpleQA F-score, which combines "how many did you get right" with "how many of your answers were right":

Then I stopped asking the model to know things

The model cannot reliably know Bangla trivia. A lookup can. So the guard does the checking instead of the model:

1. Extract the name and what is being asked (a fast pattern match for common phrasings, a small Qwen call for free-form questions).
2. Search Bangla Wikipedia. Its search is spelling-tolerant and offers "did you mean", which Wikidata's own search does not. (I tested this: Wikidata's search missed a common spelling of Tagore's name.)
3. Check the best title against the name, folding common Bangla spelling variants (ী/ি, ূ/ু, শ/ষ/স), and confirm with Wikidata that the match is the right kind of thing: a writer, a book or a film.
4. Act: refuse, redirect, or rescue.

A network failure is never treated as "this does not exist"; it is its own error state. If several people share a name, the guard asks which one instead of picking.

2,367 test items; 95% intervals are in the repo. Writers 91.9% correct, books 87.8%, films 81.4%. Films were never used in training, so they test whether the approach carries over to a type of entity the system was not tuned on. (The guard itself has no training step. Its two similarity thresholds were fixed up front and sanity-checked on the train split. The entity-type check was added after I saw train-split failures.)

For misspelled real names (330 items), 70% were handled correctly: 200 got a correct "did you mean…?", 31 were resolved with the substitution disclosed, and 99 were missed. None was silently answered as a different entity.

Where it breaks (please read this part)

I would rather you trust the numbers than be impressed by them.

• The guard's scores are ceilings. The gold answers come from Wikidata, and the guard looks things up in Wikipedia and Wikidata. A near-perfect score on clean entities is partly by construction. It shows the pipeline works, not that the guard knows things Wikipedia does not.
• The ghosts are partly self-fulfilling. I kept only invented names that a Bangla Wikipedia search could not find. That is the same search the guard uses, so refusing them is almost guaranteed. I also checked whether ghosts were detectable by name style alone with a small classifier: for model-invented names it scored AUC 0.72–0.80 (0.5 is chance), so style was a real shortcut. For my spliced names it scored 0.46–0.62, which is why I report both kinds.
• Common names are the hard case. On my training split, before I added the entity-type check, 5 of 1,005 ghosts matched a real person with the same name. The check fixed it. On the test set, the one ghost the guard answered is প্রতিশ্রুতি ("Promise"), an ordinary word that is also a real 2001 film. That one is arguably my ghost generator's mistake, not the guard's.
• I changed the guard once after a demo failure. For a typo of Tagore's name it offered a documentary about him, because suggestions were not type-checked. That means I ran the test set twice. Both runs are in the repo. The fix raised misspelling handling from 65.2% to 70.0% and changed nothing else.
• My answer checker is rule-based, not an LLM judge, and it makes mistakes: it counts "জসীম উদ্‌দীন" and "জসীম উদ্দীন" as different because of one invisible character, and Wikidata sometimes lists two birth years for the same person. I prepared a blind human audit of 100 outputs (it is in the repo under audit/) but have not completed it, so the checker's error rate is unmeasured.
• It covers writers, books and films only, needs internet, and cannot rescue what Bangla Wikipedia does not contain (11.5% of real entities went unanswered).

Related work

HalluLens (Meta, 2025) for the ghost-entity method and false-acceptance metric, SimpleQA (OpenAI, 2024) for the F-score, R-Tuning (2024) for the fine-tuning recipe, and BenHalluEval, a Bengali hallucination benchmark that checks the faithfulness of generated text, where mine checks whether an entity exists.

How I built it: I picked the problem and steered the design (the "act" layer, using existing benchmarks instead of inventing metrics, reporting the failures). I worked with an AI coding assistant, Claude Code, which wrote most of the code and scripts and the first drafts of this post.

Why Does Open Innovation Matter?

• Open weights let me find the real failure. With a closed API I could not have fine-tuned the model, inspected exactly what it does at the training level, or run four systems over 2,367 questions cheaply and repeatably. Because Qwen3.5-4B is open, I could try the "teach it to say I don't know" fix properly and learn why it fails (the model simply does not know enough). That negative result is something I would not have discovered through a prompt box.
• Open data made the test possible. Wikidata and Bangla Wikipedia are the ground truth and the lookup source. Without openly licensed, queryable Bangla knowledge there would be no way to build 929 verified ghosts or a rescue layer with citations.
• The guard is model-agnostic. The lookup does the checking, so it can sit in front of any model, open or closed, and it does not depend on one vendor staying around.
• Everything is reproducible. Every raw model output for every system is in the repo, so you can re-score my numbers instead of trusting them.
• Where open did not help: the guard needs internet access to Wikipedia, so this is not an "works with the lights off" project. The offline version I hoped for was the fine-tuned model, and it did not work.

The Spirit of It

With the increase of the literacy and reading crisis, Bangla literature is slowly facing a sharp reader-decline. As someone who grew up with Bangla books, this was really disheartening. On top of all this, LLMs continue to misread and misinterpret Bangla books which made me fear for the existence of Bangla literature in the near future. It could genuinely become a ghost because of all the "petni"s LLMs produce. I wanted to show that we can maybe reduce that level of hallucination.

Prize Categories

I am entering these two categories, both of which the project genuinely uses:

• Best Use of Tinker. Every model call in the project runs on Tinker with Qwen3.5-4B: the four-way baseline evaluation (2,367 test questions per system), generating the invented ghost names, extracting the entity from free-form questions, translating the raw answers for the demo, and the LoRA fine-tune. The fine-tune was a negative result (it learned to refuse everything), and I report it as one: Tinker is what let me run that experiment end to end and find out why it failed, with the training targets and every raw output saved in the repo.
• Best Use of Render. The live demo is a Render web service (build: pip install -r requirements.txt, start: python -m petnihunter.app). The Tinker API key lives only in Render's environment variables, never in the repository, and because a public demo spends real credit on every question, the app enforces per-visitor and daily rate limits. The repo includes a render.yaml blueprint so the whole deploy is reproducible.