<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Aman Kumar</title>
    <description>The latest articles on DEV Community by Aman Kumar (@devidevilaldevil).</description>
    <link>https://dev.to/devidevilaldevil</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160411%2Fa42a8ea6-3d01-4fde-8cfb-1671cff062ed.png</url>
      <title>DEV Community: Aman Kumar</title>
      <link>https://dev.to/devidevilaldevil</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/devidevilaldevil"/>
    <language>en</language>
    <item>
      <title>I Built a Coroner for Abandoned GitHub Repos</title>
      <dc:creator>Aman Kumar</dc:creator>
      <pubDate>Sat, 31 Oct 2026 17:25:15 +0000</pubDate>
      <link>https://dev.to/devidevilaldevil/i-built-a-coroner-for-abandoned-github-repos-4dic</link>
      <guid>https://dev.to/devidevilaldevil/i-built-a-coroner-for-abandoned-github-repos-4dic</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the Hacktoberfest Open-Source AI Challenge, Week 4: Ghost in the Machine.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;Every developer has cloned an abandoned repo, run &lt;code&gt;npm install&lt;/code&gt;, and watched it explode. The real question is &lt;strong&gt;why&lt;/strong&gt;. Did the world move out from under it (a breaking dependency release), or was it already broken when the author walked away?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;RepoRequiem&lt;/strong&gt; is a tool that investigates a dead repository and answers that question with evidence, not vibes.&lt;/p&gt;

&lt;p&gt;It runs a five-stage pipeline:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Evidence Collector&lt;/strong&gt;: pulls commits, releases, issues, forks, and dependency manifests (from the GitHub API or an offline snapshot) into SQLite. Every fact gets a deterministic &lt;code&gt;evidence_id&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time-Machine Rebuilder&lt;/strong&gt;: builds the project in a Docker sandbox twice, once with dependencies as they were on the day the project died and once with today's. The result is one of &lt;code&gt;alive&lt;/code&gt;, &lt;code&gt;world_moved&lt;/code&gt;, &lt;code&gt;born_broken&lt;/code&gt;, or &lt;code&gt;cannot_determine&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Death Bisector&lt;/strong&gt;: for &lt;code&gt;world_moved&lt;/code&gt; repos, binary-searches dependency release dates between then and now to find the window where it broke, diffs the lockfiles, and isolates the suspect package.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Heir Finder&lt;/strong&gt;: scores forks on divergence, recency, stars, and independent authorship to find a living successor.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verified Resurrection Report&lt;/strong&gt;: a local Gemma model writes the report and a step-by-step revival checklist.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The Rule of Truth
&lt;/h2&gt;

&lt;p&gt;The part I care most about is that &lt;strong&gt;the LLM never decides facts.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Dates, versions, exit codes, and counts all come from deterministic tools and the SQLite evidence store. Gemma only explains them. Every claim it writes has to cite an evidence ID, and a &lt;code&gt;ReportValidator&lt;/code&gt; checks each citation and number after generation. Claims that fail validation are flagged or pruned. A small local model is useful here because it is fast and private, and the validator keeps it from making things up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Safety: Zero Host Execution
&lt;/h2&gt;

&lt;p&gt;Running code from abandoned repos is risky, so target code &lt;strong&gt;never runs on the host&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Install stage: network on, &lt;code&gt;--ignore-scripts&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Build/test stage: &lt;code&gt;--network none&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Non-root user, 1.5 CPU, 2 GB RAM, hard timeouts&lt;/li&gt;
&lt;li&gt;Container destroyed after every run&lt;/li&gt;
&lt;li&gt;If Docker isn't available, the answer is &lt;code&gt;cannot_determine&lt;/code&gt;, never a degraded guess&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/0AVzq_bxhs0" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  Try It
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Prerequisites:&lt;/strong&gt; Python 3.11+, Docker (for live rebuilds), and &lt;a href="https://ollama.com" rel="noopener noreferrer"&gt;Ollama&lt;/a&gt; with a Gemma model (&lt;code&gt;ollama pull gemma2:2b&lt;/code&gt;).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/AMAN240310/reproequiem.git
&lt;span class="nb"&gt;cd &lt;/span&gt;reproequiem
pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-e&lt;/span&gt; &lt;span class="nb"&gt;.&lt;/span&gt;

&lt;span class="c"&gt;# Offline mode (no network or GitHub token needed)&lt;/span&gt;
reporequiem inspect tests/fixtures/sample_offline_repo.json &lt;span class="nt"&gt;--offline&lt;/span&gt;

&lt;span class="c"&gt;# Live mode&lt;/span&gt;
reporequiem inspect https://github.com/owner/dead-repo &lt;span class="nt"&gt;--token&lt;/span&gt; &lt;span class="nv"&gt;$GITHUB_TOKEN&lt;/span&gt;

&lt;span class="c"&gt;# Dashboard at http://localhost:8080&lt;/span&gt;
reporequiem serve &lt;span class="nt"&gt;--port&lt;/span&gt; 8080
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Without Docker, rebuild verdicts come back as &lt;code&gt;cannot_determine&lt;/code&gt;, by design.&lt;/p&gt;
&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/AMAN240310" rel="noopener noreferrer"&gt;
        AMAN240310
      &lt;/a&gt; / &lt;a href="https://github.com/AMAN240310/reproequiem" rel="noopener noreferrer"&gt;
        reproequiem
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;RepoRequiem ⚰️✨&lt;/h1&gt;
&lt;/div&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;An instrument that measures why an abandoned repository died and whether it can live again.&lt;/strong&gt;&lt;br&gt;
Built for the &lt;strong&gt;Hacktoberfest Open-Source AI Challenge, Week 4 ("Ghost in the Machine")&lt;/strong&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;RepoRequiem investigates dead and abandoned software repositories. It answers the fundamental question: &lt;em&gt;Did this project break because the external world moved under its feet (package drift, API breakages), or was it already broken? Can it be revived?&lt;/em&gt;&lt;/p&gt;




&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;🔮 The Pipeline&lt;/h2&gt;
&lt;/div&gt;


&lt;ol&gt;

&lt;li&gt;

&lt;strong&gt;Stage 1: Evidence Collector&lt;/strong&gt; (&lt;code&gt;reporequiem.collect&lt;/code&gt;)

&lt;ul&gt;
&lt;li&gt;Queries GitHub REST API (or offline cached snapshot).&lt;/li&gt;
&lt;li&gt;Ingests commits, releases, issues, forks, dependency manifests, and determines last known active date $T$.&lt;/li&gt;
&lt;li&gt;Stores everything with deterministic &lt;code&gt;evidence_id&lt;/code&gt;s in SQLite (&lt;code&gt;store.py&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;

&lt;li&gt;

&lt;strong&gt;Stage 2: Time-Machine Rebuilder&lt;/strong&gt; (&lt;code&gt;reporequiem.rebuild&lt;/code&gt;)

&lt;ul&gt;
&lt;li&gt;Isolated Docker container execution: install with &lt;code&gt;--ignore-scripts&lt;/code&gt;, build/test with &lt;code&gt;--network none&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Builds at time $T$ (historical dependencies) vs. today.&lt;/li&gt;
&lt;li&gt;Determines verdict: &lt;code&gt;alive&lt;/code&gt;, &lt;code&gt;world_moved&lt;/code&gt;, &lt;code&gt;born_broken&lt;/code&gt;, or &lt;code&gt;cannot_determine&lt;/code&gt;…&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;

&lt;/ol&gt;&lt;/div&gt;
&lt;br&gt;
  &lt;/div&gt;
&lt;br&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/AMAN240310/reproequiem" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;br&gt;
&lt;/div&gt;
&lt;br&gt;


&lt;h2&gt;
  
  
  How I Used Gemma
&lt;/h2&gt;

&lt;p&gt;Gemma runs locally through Ollama (&lt;code&gt;gemma2:2b&lt;/code&gt; by default) as an explainer and summarizer. The prompt includes only evidence pulled from the store, the output must be JSON with evidence-ID citations, and the validator is the gatekeeper. The model writes the memorial ("Ofrenda") and the revival checklist, but not the verdict.&lt;/p&gt;

&lt;p&gt;## Why Does Open Innovation Matter?&lt;/p&gt;

&lt;p&gt;RepoRequiem runs Gemma locally through Ollama, so a repo's code, history, and forks never leave your machine. That matters because the tool analyzes untrusted code and sometimes private forks, and a closed API would mean sending that evidence to a server I don't control.&lt;/p&gt;

&lt;p&gt;Open weights also let me swap models without touching the pipeline. The "Rule of Truth" validator works with any model, so a small local one is enough. And the whole thing runs offline, with no API costs and no service that can disappear, which fits a tool built to bring abandoned projects back to life.&lt;/p&gt;

&lt;h2&gt;
  
  
  Honest Limitations
&lt;/h2&gt;

&lt;p&gt;I documented these in &lt;code&gt;LIMITS.md&lt;/code&gt; instead of hiding them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ecosystems:&lt;/strong&gt; only npm and Python for now; everything else returns &lt;code&gt;cannot_determine&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not every death is a dependency death:&lt;/strong&gt; dead APIs, missing credentials, and rotted system libraries also kill projects. Those cases should land in &lt;code&gt;born_broken&lt;/code&gt; or &lt;code&gt;cannot_determine&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The benchmark is a scoring harness over 7 hand-built fixtures&lt;/strong&gt;, not a measurement of real-world accuracy. The next step is running the pipeline on real dormant repos with known outcomes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Heir scoring&lt;/strong&gt; is a simple weighted formula, and I haven't validated the weights yet.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Best Use of Gemma&lt;/strong&gt;: Gemma runs locally via Ollama as the report explainer, with every claim checked against stored evidence.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What's Next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Run the full pipeline on a real dataset of dormant repos&lt;/li&gt;
&lt;li&gt;Pin historical Node/Python runtimes per era in the rebuilder&lt;/li&gt;
&lt;li&gt;Normalize the heir-scoring features&lt;/li&gt;
&lt;li&gt;Add more ecosystems (Rust, Go)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks for reading. If you have a dead repo you'd like to investigate, open an issue. 🕯️&lt;/p&gt;

</description>
      <category>hf26challenge</category>
      <category>devchallenge</category>
      <category>gemma</category>
      <category>python</category>
    </item>
    <item>
      <title>LectureAI: I Let Whisper and Gemma Attend Class for Me</title>
      <dc:creator>Aman Kumar</dc:creator>
      <pubDate>Sun, 25 Oct 2026 13:48:27 +0000</pubDate>
      <link>https://dev.to/devidevilaldevil/lectureai-i-let-whisper-and-gemma-attend-class-for-me-4n6f</link>
      <guid>https://dev.to/devidevilaldevil/lectureai-i-let-whisper-and-gemma-attend-class-for-me-4n6f</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-week3-2026-10-19"&gt;Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;LectureAI&lt;/strong&gt; turns a lecture recording into everything you need to revise it. Upload an MP3, WAV or M4A in Hindi, English or Hinglish, and it gives you:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a &lt;strong&gt;timestamped transcript&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;a &lt;strong&gt;summary&lt;/strong&gt; and a &lt;strong&gt;topic timeline&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;likely exam topics&lt;/strong&gt;, each linked to the moment it was taught&lt;/li&gt;
&lt;li&gt;structured &lt;strong&gt;notes&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;flashcards&lt;/strong&gt; with a revision queue&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;assignments and deadlines&lt;/strong&gt; the teacher mentions, collected on a calendar for you to confirm or dismiss&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It saves the hour you would spend replaying a recording, pausing and typing notes. It also catches the "submit this by Friday" line you missed while writing something else. The lazy student gets to actually listen in class.&lt;/p&gt;

&lt;p&gt;By default, everything runs on my own computer. The dashboard shows an "All local services ready" badge, and no audio or transcript is sent to the cloud. If you choose the optional ElevenLabs engine, that lecture's audio is uploaded for transcription.&lt;/p&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;🎥 Video demo: &lt;a href="https://www.youtube.com/watch?v=dIBrWXM4P5E" rel="noopener noreferrer"&gt;https://www.youtube.com/watch?v=dIBrWXM4P5E&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dashboard:&lt;/strong&gt; courses, lectures, flashcards due and open deadlines at a glance.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lecture view:&lt;/strong&gt; summary, topic timeline and likely exam topics sit next to an audio player, so every topic is one click away from the audio.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Assignments:&lt;/strong&gt; deadlines detected from lectures appear on a calendar with Open, Needs review, Done and Dismissed tabs. The AI proposes and you decide.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/AMAN240310/Lectur-to-everything" rel="noopener noreferrer"&gt;https://github.com/AMAN240310/Lectur-to-everything&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;p&gt;LectureAI is built on open-source models running locally, with an optional cloud engine:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Whisper&lt;/strong&gt; transcribes the audio, including Hindi and Hinglish, with timestamps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gemma, served through Ollama,&lt;/strong&gt; reads the transcript and produces the summary, topic timeline, exam topics, notes, flashcards and deadline candidates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ElevenLabs Scribe&lt;/strong&gt; is an optional cloud transcription engine. Whisper runs locally by default, and you can switch a lecture to ElevenLabs when you want faster or more accurate transcripts and are fine with the audio being uploaded.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The pipeline is: &lt;code&gt;upload → Whisper (or ElevenLabs) transcript → Gemma structured extraction → notes, flashcards, deadlines → dashboard&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A few design choices that mattered:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Timestamps everywhere.&lt;/strong&gt; Every topic and exam topic links back to the audio, so you can check the AI against what was actually said.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Human in the loop for deadlines.&lt;/strong&gt; Detected assignments go into a "Needs review" tab instead of silently landing on your calendar, because a wrong deadline is worse than a missing one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reprocess and processing log.&lt;/strong&gt; Any lecture can be re-run, and the log shows what happened at each step, so weak output is fixable instead of a dead end.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;A lecture recording is private: your teacher's words, your classmates' questions, your own voice. With a closed API, every recording is uploaded, billed per minute, and useless when the Wi-Fi drops. Open-weight models changed that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Privacy:&lt;/strong&gt; by default, nothing leaves the laptop.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost:&lt;/strong&gt; a whole semester of lectures costs nothing beyond electricity, which matters for students.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Language:&lt;/strong&gt; Whisper handles Hindi and Hinglish, which many note-taking tools handle poorly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Control:&lt;/strong&gt; I can swap model sizes, tune prompts and re-run processing as often as I like without worrying about a bill.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Best Use of Gemma:&lt;/strong&gt; Gemma, served locally through Ollama, is the brain of LectureAI. It turns a raw transcript into the summary, topic timeline, likely exam topics, notes, flashcards and deadline candidates, all on a laptop with no internet connection and no API bill.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Best Use of ElevenLabs:&lt;/strong&gt; LectureAI can transcribe lectures with ElevenLabs Scribe as an alternative to local Whisper. The default path stays fully local, and ElevenLabs is an opt-in engine for people who want cloud-grade transcription.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Lazy Payoff
&lt;/h2&gt;

&lt;p&gt;In the demo, a 7-minute lecture became a full summary, a topic timeline, seven likely exam topics and flashcards with no typing on my side. Across four lectures, LectureAI generated 41 flashcards ready to revise.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
      <category>opensource</category>
      <category>llm</category>
    </item>
    <item>
      <title>InvoicePilot: a private AI invoice sorter for freelancers that runs offline with Gemma 4</title>
      <dc:creator>Aman Kumar</dc:creator>
      <pubDate>Sun, 18 Oct 2026 19:38:36 +0000</pubDate>
      <link>https://dev.to/devidevilaldevil/invoicepilot-a-private-ai-invoice-sorter-for-freelancers-that-runs-offline-with-gemma-4-4epm</link>
      <guid>https://dev.to/devidevilaldevil/invoicepilot-a-private-ai-invoice-sorter-for-freelancers-that-runs-offline-with-gemma-4-4epm</guid>
      <description>&lt;p&gt;&lt;em&gt;This is my submission for the Hacktoberfest Open-Source AI Challenge: Week 2 (Side Hustle).&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;Every freelancer I know has the same quarterly ritual: dig through email attachments, download folders and photos of paper receipts, then retype everything into a spreadsheet.&lt;/p&gt;

&lt;p&gt;The "AI receipt scanners" that promise to fix this share two problems. They upload your client's invoices to someone else's API, which can break an NDA, and they add yet another monthly subscription.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I built
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;InvoicePilot&lt;/strong&gt; is a local-first invoice and receipt assistant. Drop in a PDF, PNG or JPG and it:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Reads the text, using the PDF text layer or Tesseract OCR for scans and photos&lt;/li&gt;
&lt;li&gt;Sends it to &lt;strong&gt;Gemma 4 E2B&lt;/strong&gt; running locally through Ollama, to extract vendor, dates, line items, tax and total&lt;/li&gt;
&lt;li&gt;Validates the result and flags anything suspicious&lt;/li&gt;
&lt;li&gt;Shows the original document next to the extracted fields so you can review it&lt;/li&gt;
&lt;li&gt;Exports approved records to CSV and charts spending by month, category and vendor&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Everything runs on localhost. No document text, price or client name leaves the machine.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Demo video:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/vYqXYCfynrI" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  Why open models matter here
&lt;/h2&gt;

&lt;p&gt;An invoice is one of the most sensitive documents a freelancer holds: client names, rates, bank details, tax IDs. Open weights change what's possible:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Privacy:&lt;/strong&gt; the model runs on my laptop, so none of that data goes to a server I don't control.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost:&lt;/strong&gt; there is no per-document fee and no subscription, which matters for a tool whose users watch every rupee or dollar.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Swappable:&lt;/strong&gt; the model is one line in &lt;code&gt;.env&lt;/code&gt; (&lt;code&gt;OLLAMA_MODEL&lt;/code&gt;). I run Gemma 4 E2B by default, and I can try Llama 3.2 or Qwen without touching any code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Offline:&lt;/strong&gt; after the one-time model download, it works with no internet connection.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A small model like Gemma 4 E2B is what makes this realistic on an ordinary laptop (mine has 16 GB RAM and an RTX 3050).&lt;/p&gt;

&lt;h2&gt;
  
  
  Small models make mistakes, so I don't trust them blindly
&lt;/h2&gt;

&lt;p&gt;Small on-device models can invent a zero tax line when a tax breakdown exists, confuse MM/DD with DD/MM dates, or return totals that don't add up. I treat the model's output as untrusted input:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Schema validation&lt;/strong&gt; (Zod) rejects malformed output&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One bounded repair attempt&lt;/strong&gt; sends the exact validation error back to the model. If that fails, the record goes to human review instead of crashing&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A deterministic math check:&lt;/strong&gt; the backend verifies subtotal + tax - discount = total itself, in integer minor units, and warns on any mismatch&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A human approves everything:&lt;/strong&gt; nothing reaches the CSV export until you click approve&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Who would pay
&lt;/h2&gt;

&lt;p&gt;Freelancers, consultants and small business owners who handle client documents under confidentiality. The open core stays free (MIT). A paid one-time desktop build with extras such as separate client books is the way I'd make money from it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Code (MIT): &lt;a href="https://github.com/AMAN240310/invoicepilot" rel="noopener noreferrer"&gt;https://github.com/AMAN240310/invoicepilot&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Stack: React + Vite, Express, SQLite, Tesseract.js, Ollama with Gemma 4 E2B.&lt;/p&gt;

&lt;p&gt;Setup: &lt;code&gt;ollama pull gemma4:e2b-it-qat&lt;/code&gt;, copy &lt;code&gt;.env.example&lt;/code&gt; to &lt;code&gt;server/.env&lt;/code&gt;, then run the server and the client.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
      <category>gemma</category>
      <category>opensource</category>
    </item>
    <item>
      <title>I Tested a Local Gemma Plant Guide. It Got the Sunflower and Missed the Lily</title>
      <dc:creator>Aman Kumar</dc:creator>
      <pubDate>Sat, 10 Oct 2026 14:29:56 +0000</pubDate>
      <link>https://dev.to/devidevilaldevil/i-tested-a-local-gemma-plant-guide-it-got-the-sunflower-and-missed-the-lily-59kd</link>
      <guid>https://dev.to/devidevilaldevil/i-tested-a-local-gemma-plant-guide-it-got-the-sunflower-and-missed-the-lily-59kd</guid>
      <description>&lt;p&gt;I wanted an excuse to look at plants instead of my phone. &lt;strong&gt;Wander&lt;/strong&gt; is a plant guide where the phone's job is small: take one photo, get one answer, ask one follow-up, and put the phone away.&lt;/p&gt;

&lt;p&gt;A Gemma model running on my own laptop names the plant and explains what it saw. Then I decide whether the answer is good enough to keep.&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/8mqOB4Ks19U" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  It started with a mistake
&lt;/h2&gt;

&lt;p&gt;My first test photo was a sunflower, and Gemma got it right. My second was a pink flower with six spotted petals, and Gemma told me, confidently, that it was a &lt;strong&gt;Fuchsia&lt;/strong&gt;. It was a lily.&lt;/p&gt;

&lt;p&gt;That shaped the app. A small local model is useful, but it can be wrong in a convincing voice. So Wander keeps the human in charge:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Nothing counts until you confirm it,&lt;/strong&gt; and you can edit the name first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every answer shows its doubt:&lt;/strong&gt; a confidence score, the features it used, and look-alikes it may be confusing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You can cross-examine it.&lt;/strong&gt; The Ask tab sends the same photo back and lets you challenge the answer, for example "Why not a lily?"&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The model is told never to call a plant safe,&lt;/strong&gt; and every result carries a warning not to eat or handle plants based on an AI guess.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it does
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Find:&lt;/strong&gt; photograph a plant and get a name, confidence, a short summary and visible features.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Map:&lt;/strong&gt; each confirmed find appears where you took the photo, or where you drop a pin.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Collection:&lt;/strong&gt; your species, catalogued like a herbarium, with every sighting.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Badges:&lt;/strong&gt; milestones like first find, five species and an early-morning sighting. These are plain rules in code, not AI.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why open-weight mattered
&lt;/h2&gt;

&lt;p&gt;Identification and follow-up questions run through Ollama on my own laptop (RTX 3050, 16 GB RAM) using &lt;code&gt;gemma4:e2b-it-qat&lt;/code&gt;, a 4.3 GB download. The photos are processed on my machine, there's no cost per photo, and the identification prompt, safety rules and chat behavior are all text I can change in my own code.&lt;/p&gt;

&lt;p&gt;It isn't fully offline. The photos and the AI model stay on my laptop, but the map background loads from OpenStreetMap, and sighting records are stored in MongoDB, which has to be reachable when the app runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  How it's built
&lt;/h2&gt;

&lt;p&gt;React and Tailwind on the front, an Express server for uploads, MongoDB and badges, and a small FastAPI service that makes Gemma answer in strict JSON. My phone connects to the laptop over the local network. Most of the code was written with AI coding assistants. I directed the design, ran every service and tested it on photos of plants.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A proper walk.&lt;/strong&gt; So far I've tested Wander on photos of plants on my laptop. The next step is taking it outside and counting how often it's right.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Offline map tiles.&lt;/strong&gt; The map background needs internet today, so I want to cache map tiles for the area I'll walk in.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A specialist plant model for naming.&lt;/strong&gt; Gemma sometimes names a flower wrongly with high confidence. I'd like to test a dedicated plant classifier for the naming step and keep Gemma for the explanations and questions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;GitHub: &lt;a href="https://github.com/AMAN240310/Wander" rel="noopener noreferrer"&gt;https://github.com/AMAN240310/Wander&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Prize category:&lt;/strong&gt; Best Use of Gemma&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
      <category>gemma</category>
      <category>opensource</category>
    </item>
    <item>
      <title>I built a hands-free slide remote for a friend who can never find the right page</title>
      <dc:creator>Aman Kumar</dc:creator>
      <pubDate>Sun, 04 Oct 2026 14:44:08 +0000</pubDate>
      <link>https://dev.to/devidevilaldevil/i-built-a-hands-free-slide-remote-for-a-friend-who-can-never-find-the-right-page-1298</link>
      <guid>https://dev.to/devidevilaldevil/i-built-a-hands-free-slide-remote-for-a-friend-who-can-never-find-the-right-page-1298</guid>
      <description>&lt;h1&gt;
  
  
  I Built a Presentation Remote That Uses Hand Gestures and Voice
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-weekend-2026-10-01"&gt;Hacktoberfest Weekend Challenge: Build for a Friend&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;My friend &lt;strong&gt;Dinesh Prajapath&lt;/strong&gt; presents frequently. His decks are often large PDFs, and halfway through a talk he can lose track of where something is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"The pricing part was around page 12... no, 14?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Scrolling through a PDF in front of an audience is stressful. Traditional clickers make sequential navigation easy, but jumping directly to a specific slide usually still requires knowing its position.&lt;/p&gt;

&lt;p&gt;So I built a &lt;strong&gt;presentation remote controlled using a laptop webcam and microphone&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The goal was simple:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Let the presenter navigate a presentation without touching the laptop.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What It Does
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Hand Gestures
&lt;/h3&gt;

&lt;p&gt;The webcam tracks the presenter's hand using MediaPipe.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Swipe right/left&lt;/strong&gt; → change slides&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Open palm&lt;/strong&gt; → blank the presentation screen&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fist&lt;/strong&gt; → jump to the first slide&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No special hardware or physical clicker is required.&lt;/p&gt;

&lt;h3&gt;
  
  
  Voice Navigation
&lt;/h3&gt;

&lt;p&gt;The presenter can hold a key and say something like:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Go to the pricing slide."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Instead of requiring the presenter to remember the page number, the system searches the content of the presentation and finds the most relevant slide.&lt;/p&gt;

&lt;p&gt;It can also handle commands such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;"page 7"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;"first slide"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;"last slide"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;"go to pricing"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;"show weather GPT"&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why Open Models Mattered
&lt;/h2&gt;

&lt;p&gt;Everything runs locally on the laptop:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;MediaPipe&lt;/strong&gt; → hand tracking&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Whisper&lt;/strong&gt; → speech-to-text&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gemma via Ollama&lt;/strong&gt; → semantic understanding and slide selection&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This was important for several reasons.&lt;/p&gt;

&lt;h3&gt;
  
  
  Privacy
&lt;/h3&gt;

&lt;p&gt;The presentation, webcam feed, and microphone audio are processed locally and are not sent to a cloud service.&lt;/p&gt;

&lt;p&gt;That matters when presenting slides that are confidential or haven't been published yet.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cost
&lt;/h3&gt;

&lt;p&gt;There is no per-request API cost.&lt;/p&gt;

&lt;p&gt;Once the required models are downloaded, there is no cloud inference bill for each presentation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Offline
&lt;/h3&gt;

&lt;p&gt;After the initial model downloads, the system can operate without an internet connection.&lt;/p&gt;

&lt;p&gt;That's useful in classrooms, seminar halls, and event venues where Wi-Fi can be unreliable.&lt;/p&gt;

&lt;h3&gt;
  
  
  Control
&lt;/h3&gt;

&lt;p&gt;I could change the models, prompts, matching logic, and decision-making process myself.&lt;/p&gt;

&lt;p&gt;This ended up being one of the biggest advantages of using an open/local approach.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Category
&lt;/h2&gt;

&lt;p&gt;I'm entering the &lt;strong&gt;Best Use of Gemma&lt;/strong&gt; category because Gemma runs locally through Ollama and is used for semantic slide selection when deterministic matching is insufficient.&lt;/p&gt;

&lt;p&gt;Instead of sending the presentation content to a cloud API, Gemma helps the system understand ambiguous voice commands and identify the most relevant slide locally.&lt;/p&gt;

&lt;p&gt;Gemma is not responsible for controlling the presentation directly. It acts as the final semantic decision-maker when deterministic matching cannot confidently choose a slide.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Problem With My First Approach
&lt;/h2&gt;

&lt;p&gt;My first version was much simpler.&lt;/p&gt;

&lt;p&gt;I extracted the text from every slide, gave all of it to &lt;strong&gt;Gemma 2B&lt;/strong&gt;, and asked it to return the page number matching the user's request.&lt;/p&gt;

&lt;p&gt;It didn't work reliably.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;"intelligent record system"&lt;/code&gt; → page 14 instead of page 5&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;"WeatherGPT"&lt;/code&gt; → page 15 instead of page 18&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One problem was that some slides contained numbers such as:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;code&gt;04 Land Records&lt;/code&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Those numbers were section labels printed on the slides, not the actual PDF page numbers.&lt;/p&gt;

&lt;p&gt;The small language model could confuse those numbers with the real slide positions.&lt;/p&gt;

&lt;p&gt;So instead of making the LLM responsible for everything, I changed the architecture.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Hybrid Approach
&lt;/h2&gt;

&lt;p&gt;The final system uses multiple levels of matching.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Direct Commands
&lt;/h3&gt;

&lt;p&gt;Simple commands don't need an LLM.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;page 7
first slide
last slide
next slide
previous slide
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These are handled directly by the application.&lt;/p&gt;

&lt;p&gt;This makes them faster and more reliable.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Keyword + Fuzzy Matching
&lt;/h3&gt;

&lt;p&gt;For content-based requests, the system first searches the extracted slide text.&lt;/p&gt;

&lt;p&gt;This handles obvious matches without involving the LLM.&lt;/p&gt;

&lt;p&gt;It also helps with small differences caused by speech recognition.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Weather GPT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;can still match:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WeatherGPT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  3. Gemma as a Final Decision Maker
&lt;/h3&gt;

&lt;p&gt;If the keyword matching produces several plausible slides, Gemma receives the remaining candidates and decides which one best matches the request.&lt;/p&gt;

&lt;p&gt;The normal path limits this to the &lt;strong&gt;top five candidates&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;If keyword matching cannot produce useful candidates, the system can fall back to semantic matching against the slide titles.&lt;/p&gt;

&lt;p&gt;Gemma can also return:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;none
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;when none of the candidates are relevant.&lt;/p&gt;

&lt;p&gt;In that case, the presentation does nothing instead of jumping to a potentially incorrect slide.&lt;/p&gt;

&lt;p&gt;That conservative behavior is intentional.&lt;/p&gt;

&lt;h2&gt;
  
  
  How Voice Recognition Works
&lt;/h2&gt;

&lt;p&gt;Whisper runs locally, so the microphone audio does not need to be uploaded to a speech API.&lt;/p&gt;

&lt;p&gt;I also give Whisper the presentation's slide titles as vocabulary hints.&lt;/p&gt;

&lt;p&gt;For example, if the deck contains terms such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Land Records
WeatherGPT
Intelligent Record System
Digital Governance
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;those terms can be provided as context to improve recognition.&lt;/p&gt;

&lt;p&gt;I also filter out Whisper segments that are likely to be silence or low-confidence/noise.&lt;/p&gt;

&lt;p&gt;This helps prevent silence or background noise from accidentally becoming a navigation command.&lt;/p&gt;

&lt;h2&gt;
  
  
  How It Works
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Gesture Pipeline
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Webcam
   ↓
MediaPipe Hand Landmarks
   ↓
Gesture Rules
   ↓
Keyboard / Presentation Action
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Voice Pipeline
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Microphone
   ↓
Whisper
   ↓
Command Detection
   ↓
Keyword + Fuzzy Matching
   ↓
Gemma / Ollama when needed
   ↓
Target Slide
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Overall Architecture
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                     ┌─────────────────────┐
                     │     Presentation    │
                     │       PDF/PPTX      │
                     └──────────┬──────────┘
                                │
                           Extract Text
                                │
                                ▼
                     ┌─────────────────────┐
                     │    Slide Indexing   │
                     └─────────────────────┘


 Webcam ──→ MediaPipe ──→ Gesture Rules ──→ Slide Action

 Microphone ──→ Whisper ──→ Keyword/Fuzzy Search
                              │
                              ▼
                       Candidate Slides
                              │
                              ▼
                         Gemma / Ollama
                              │
                              ▼
                         Target Slide
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What I Learned
&lt;/h2&gt;

&lt;p&gt;The biggest lesson from this project was that &lt;strong&gt;using an LLM for everything isn't necessarily the best architecture&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;My first instinct was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User Request
     ↓
    LLM
     ↓
Slide Number
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But the system became much more reliable when I divided the problem into smaller parts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Simple request
     ↓
Deterministic logic

Clear content match
     ↓
Keyword + fuzzy matching

Ambiguous request
     ↓
Small candidate set
     ↓
LLM decision
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This reduced the amount of reasoning the model had to perform and also made failures safer.&lt;/p&gt;

&lt;p&gt;The LLM became a &lt;strong&gt;decision-making component rather than the entire navigation system&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I Chose This Design
&lt;/h2&gt;

&lt;p&gt;A presentation remote has an unusual requirement:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Being wrong is worse than doing nothing.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If the system fails to find a slide, the presenter can try again.&lt;/p&gt;

&lt;p&gt;But if it confidently jumps to the wrong slide during a live presentation, it creates confusion.&lt;/p&gt;

&lt;p&gt;So I designed the system to prefer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;No action
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;over:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Probably the right slide
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;when confidence is low.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limitations
&lt;/h2&gt;

&lt;p&gt;The project is still a prototype, so there are several limitations.&lt;/p&gt;

&lt;h3&gt;
  
  
  Gesture Detection
&lt;/h3&gt;

&lt;p&gt;The gesture thresholds are tuned for a particular webcam setup.&lt;/p&gt;

&lt;p&gt;Lighting, camera position, hand visibility, and background can affect detection accuracy.&lt;/p&gt;

&lt;h3&gt;
  
  
  Speech Recognition
&lt;/h3&gt;

&lt;p&gt;Whisper can sometimes misrecognise accented or noisy speech.&lt;/p&gt;

&lt;p&gt;A larger model such as &lt;code&gt;small&lt;/code&gt; can improve recognition, but requires more computational resources.&lt;/p&gt;

&lt;h3&gt;
  
  
  Text-Based Navigation
&lt;/h3&gt;

&lt;p&gt;Voice navigation depends on text extracted from the presentation.&lt;/p&gt;

&lt;p&gt;Image-only or scanned PDFs may not contain useful extractable text.&lt;/p&gt;

&lt;p&gt;Slides with very little text are also harder to locate using voice commands.&lt;/p&gt;

&lt;h3&gt;
  
  
  Slide Synchronization
&lt;/h3&gt;

&lt;p&gt;The remote maintains its own slide position.&lt;/p&gt;

&lt;p&gt;If the presenter manually changes slides using the mouse or keyboard, the internal slide counter can potentially drift from the actual presentation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Local Model Resources
&lt;/h3&gt;

&lt;p&gt;Larger language models can improve semantic matching, but they require more RAM and CPU/GPU resources.&lt;/p&gt;

&lt;p&gt;The project therefore involves a trade-off between:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Model Size
    ↕
Accuracy
    ↕
Local Hardware Requirements
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What I'd Improve Next
&lt;/h2&gt;

&lt;p&gt;There are several directions I would like to explore:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Better automatic gesture calibration&lt;/li&gt;
&lt;li&gt;More robust hand tracking across different lighting conditions&lt;/li&gt;
&lt;li&gt;OCR support for image-based/scanned slides&lt;/li&gt;
&lt;li&gt;Better synchronization with browser-based PDF viewers&lt;/li&gt;
&lt;li&gt;More advanced semantic slide indexing&lt;/li&gt;
&lt;li&gt;Support for multiple presentation applications&lt;/li&gt;
&lt;li&gt;Confidence scores and visual feedback before changing slides&lt;/li&gt;
&lt;li&gt;More natural voice commands such as:

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;"Go back to the slide about pricing"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;"Show the architecture section"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;"Find the slide mentioning MongoDB"&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/72gDGmY8-7c" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;The complete source code is available on GitHub:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/AMAN240310/gesture-voice-presenter-remote" rel="noopener noreferrer"&gt;View the project on GitHub&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The project includes the gesture recognition, voice navigation, PDF/PPTX slide extraction, fuzzy matching, and local Gemma/Ollama integration.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final Thoughts
&lt;/h2&gt;

&lt;p&gt;I started this project because a friend kept running into a very practical problem while presenting.&lt;/p&gt;

&lt;p&gt;What looked like a simple "control the slides with gestures and voice" project turned into a useful lesson in system design.&lt;/p&gt;

&lt;p&gt;The most important improvement wasn't choosing a bigger model.&lt;/p&gt;

&lt;p&gt;It was &lt;strong&gt;reducing what the model had to solve&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Instead of asking an LLM to understand the entire presentation and return a page number, I combined deterministic rules, fuzzy search, and an LLM only where it actually added value.&lt;/p&gt;

&lt;p&gt;That made the system faster, more predictable, and safer to use during a live presentation.&lt;/p&gt;

&lt;p&gt;And most importantly, Dinesh no longer has to remember:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Was pricing on page 12 or 14?"&lt;/p&gt;
&lt;/blockquote&gt;

</description>
      <category>devchallenge</category>
      <category>weekendchallenge</category>
      <category>hf26challenge</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
