This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
Oktoberfest? YES! 🍺 Ah... this one is Hacktoberfest. Okay, still nice! 😁
I couldn't miss Hacktoberfest, and Side Hustle sounds like a pretty good challenge name. And with so many prize categories and promo codes, hmm... let's pick something that could save me some money.
Thinking Machines actually sounds interesting (it reminds me of the Mortal Engines movie, which makes it even better). So, let's fine-tune a model.
Since this is my introduction to fine-tuning, I wanted to pick something relatively simple, but also useful. Ideally, something that could save me or my fellow developers some money on AI API calls. It should be fast, work with an open-weight model, and actually make sense as a side hustle.
And that's how I came up with Complexity Scorer.
What I Built
So, what exactly is Complexity Scorer?
The idea is simple. Imagine you're using an expensive AI model for every request, even trivial ones like extracting a value from JSON or formatting some text.
Do you really need an expensive reasoning model for that? Probably not.
But your AI application doesn't necessarily know how complex an incoming prompt is, so it might send everything to the same model. That's where Complexity Scorer comes in. Before sending a prompt to an AI model, your application calls Complexity Scorer, a lightweight endpoint backed by a fine-tuned open-weight model. It estimates the prompt's complexity and returns a score between 0.00 and 1.00.
Here are the reference levels:
| Score | Complexity | Description |
|---|---|---|
| 0.00 | Trivial | Almost no reasoning. Extraction, formatting, or deterministic transformations. |
| 0.25 | Easy | Limited interpretation or a few straightforward steps. |
| 0.50 | Moderate | Multiple steps, combining information, some domain knowledge, or several constraints. |
| 0.75 | Hard | Substantial reasoning, non-obvious tradeoffs, domain expertise, or many interacting constraints. |
| 1.00 | Very hard | Deep multi-step reasoning, expert-level analysis, significant uncertainty, or interactions between several complex systems. |
Your application can then use that score to decide which AI model should handle the actual request.
Simple prompt? Use a cheaper model.
Complex problem? Send it to something more capable.
That's it. Of course, it's not limited to API endpoints. You could use it in AI agents, coding tools or any application that needs to choose between different models. And because it's based on an open-weight model, you could even run it locally using something like Ollama, once the fine-tuned model is exported into a compatible format.
The potential customers? Developers and teams building AI-powered applications who want to reduce their API costs without blindly routing everything to cheaper models.
And how would I actually make money from this? Well, the model itself would remain free for anyone to download and use. I'd charge only for the convenience of using my hosted API, probably with a small free tier and monthly subscriptions that include a certain number of requests. Of course, the pricing would have to be low enough that developers actually save money by using Complexity Scorer. Otherwise, what's the point?
Demo
Code
Complexity Scorer API
Complexity Scorer estimates how much reasoning a prompt needs. Your application can use that estimate to choose which AI model should answer it, such as a cheaper model for a simple task or a more capable model for a difficult one.
Send a prompt to POST /api/complexity and receive a score between 0.00 and 1.00. This endpoint evaluates the prompt; it does not answer it.
| Score | Level | Example of the work involved |
|---|---|---|
0.00 |
Trivial | Extract a value or format text |
0.25 |
Easy | Interpret a short request or follow a few simple steps |
0.50 |
Moderate | Combine information and follow several constraints |
0.75 |
Hard | Analyze tradeoffs or use substantial domain knowledge |
1.00 |
Very hard | Perform deep analysis across complex systems |
Scores between these reference points are allowed. They are estimates, not guarantees that a particular model can solve the task.
Before you start
You need:
- The .NET 10…
Complexity Scorer
A Python training project that uses Tinker to fine-tune Qwen3-8B to estimate how difficult a user request is for a language model to answer correctly. The experiment compares the original model with a fine-tuned version on the same evaluation dataset.
The model reads a prompt and returns only JSON, for example {"complexity": 0.50}. It scores the reasoning and understanding needed, rather than just the prompt's length:
| Score | Meaning |
|---|---|
| 0.00 | Trivial: extracting a value or reformatting text |
| 0.25 | Easy: a few straightforward steps |
| 0.50 | Moderate: combining information or following several constraints |
| 0.75 | Hard: substantial reasoning, expertise, or tradeoffs |
| 1.00 | Very hard: deep reasoning across complex systems or uncertainty |
How the experiment works
- Measure the baseline: ask the original model to score 50 labeled prompts.
- Prepare training data: validate 2,500 examples (500 per score) and convert them into chat conversations. Duplicate prompts and overlap with evaluation prompts are rejected.
- …
How I Built It
Okay, let's start with the model training process, because this was probably the most interesting part of the whole project.
To use Tinker, I first needed to set up a Python environment. I'm mostly a .NET developer, and I only use Python on rare occasions, so I had to install Python, the Tinker packages, and authenticate with Tinker. Fortunately, with a little help from ChatGPT, it was pretty straightforward. I ran into some trouble with Microsoft C++ Build Tools along the way, but nothing too dramatic.
With the environment ready, I moved on to what might be the most important decision of the project: choosing the model to fine-tune.
Choosing the model
The open-weight model is the core of Complexity Scorer. My first selection criterion was simple: training cost on Tinker. After comparing the available models, I narrowed my options down to two of the cheapest candidates that seemed suitable for this task.
| Model | Context | Type | Parameters | Prefill / 1M | Sampling / 1M | Training / 1M |
|---|---|---|---|---|---|---|
| Qwen3-8B | 32K | Hybrid | 8B | $0.195 | $0.60 | $0.44 |
| GPT-OSS-20B | 32K | Reasoning | 21B / 3.6B active | $0.18 | $0.45 | $0.396 |
Source: Tinker Models & Pricing
GPT-OSS-20B is actually cheaper on paper, but there's a catch: it's a reasoning model that always generates chain-of-thought tokens before answering. That's not ideal for a classifier that's supposed to return a single number as quickly and cheaply as possible.
Qwen3-8B, on the other hand, is a Hybrid model. It supports both thinking and non-thinking modes, which means we can disable reasoning for faster classification.
So I picked Qwen3-8B for three reasons:
- Hybrid architecture: We can disable reasoning to reduce unnecessary tokens and latency.
- No unnecessary vision capabilities: We're classifying text prompts, not images.
- Task-specific fine-tuning: Tinker specifically recommends starting with Hybrid models for this type of task.
And with a 32K context window, it should be more than enough for the prompts I want to classify.
Now comes the interesting part: can I actually teach it to estimate prompt complexity?
Smoke Tests
Okay, I didn't jump straight into fine-tuning. First, I started with a simple Python script (test_tinker.py) just to check whether I could actually call Tinker. Once that worked, I prepared a small evaluation dataset with 50 examples (eval.jsonl) and a baseline.py script to evaluate the selected model against it.
My first evaluation was against the original Qwen3-8B, without any fine-tuning.
The result? 50% accuracy. Not bad for a starting point.
Next, I prepared 50 training examples (different from the evaluation dataset), saved them in train.jsonl, and ran my train.py script. After fine-tuning finished, I evaluated the model again using the same 50 evaluation examples.
And the result? 86% accuracy!
That was a little shocking, but also pretty exciting. Just 50 training examples and such a big improvement? Naturally, I thought that more training data would give me even better results. So I prepared another dataset with 500 training examples and trained the model again.
Well... I couldn't have been more wrong. 🤣
The accuracy actually dropped to 74%.
Hmm... not exactly what I expected.
I tried experimenting a little with the hyperparameters, but the accuracy stayed around 74%. And since I'm definitely not an ML engineer, I decided to stop experimenting with smoke tests and move on to the full fine-tuning run.
Fine-Tuning
At this point, I had everything prepared. The scripts worked, the evaluation dataset was ready, and I knew the model could learn the task.
The last thing I needed was more training data.
I chose 2,500 training examples as a balance between dataset diversity, generation effort, and training cost. With five complexity levels, that meant 500 examples per class, which sounded like a reasonable starting point.
So I prepared a prompt and gave it to GPT-6.1 Sol with high thinking effort.
And then it happened...
I asked Sol to generate 2,500 training examples for my LLM. It started working, and I left it alone. Previously, generating 500 examples had taken around 30 minutes, so I expected this to take a while. But after approximately 47 minutes, things started looking suspicious. Still, I thought, well, 2,500 examples is a lot of data. Maybe it just needs more time. Meanwhile, I was browsing the web and randomly opened the billing tab on Tinker.
$2.47?! WHAT?!
I immediately stopped Sol and checked what it was actually doing. Instead of just generating training examples, it had started running model training in the background. There were 18 sessions involving different models, including deepseek-ai/DeepSeek-V3.1, moonshotai/Kimi-K2.6, and Qwen/Qwen3.5-397B-A17B.
Oh my... I asked for training data, not for an AI training marathon..
By the time I stopped it, my remaining Tinker balance was down to $2.11.
Well, there's a nice little lesson about giving AI agents access to paid services. Maybe I should write another article about that someday.
Anyway, back to the dataset.
This time, I decided to be a little more careful. I switched to Luna with medium thinking effort and prepared a prompt to generate just 200 training examples at a time. Then I manually ran it ten times, each time focusing on different types of prompts to improve dataset diversity, and added another 500 examples from my previous Sol generation run.
Why manually? Well, after the previous incident, I wasn't exactly eager to let another AI agent run wild with my budget. And thinking about it now, I probably should have used Luna for data generation from the beginning. Sol was a bit overkill for this task.
So after that incident, I finally managed to run the fine-tune with all 2,500 training examples successfully. This time, the model achieved 76% accuracy. That's a 26-percentage-point improvement over the original Qwen3-8B, which achieved only 50% accuracy on the same evaluation dataset.
What's interesting?
More training data didn't meant better results on my evaluation dataset. The model trained on just 50 examples still performed best, with 86% accuracy, beating the 2,500-example model by 10 percentage points.
But why? Well, there are a few possible explanations how i explain it (again i am not ML engineer)
First, all my training data was AI-generated, and AI doesn't always judge complexity consistently. One example might be labeled 0.50, while a very similar one might get 0.75. With more generated examples, there are also more opportunities for inconsistent labels, which can confuse the model during training.
And Second, my evaluation dataset contained only 50 examples. That's pretty small, and it's possible the first fine-tuned model simply happened to perform particularly well on those specific examples. A larger and more diverse evaluation dataset might tell a different story.
Still, the 2,500-example model learned to recognize all five complexity levels pretty well. The final model's accuracy across the five complexity levels looked like this:
| Complexity score | Level | Correct | Accuracy |
|---|---|---|---|
| 0.00 | Trivial | 8/10 | 80% |
| 0.25 | Easy | 6/10 | 60% |
| 0.50 | Moderate | 7/10 | 70% |
| 0.75 | Hard | 7/10 | 70% |
| 1.00 | Very hard | 10/10 | 100% |
| Total | 38/50 | 76% |
All 12 incorrect predictions were off by just one complexity level, so even when the model was wrong, it wasn't completely wrong.
And increasing the dataset from 500 to 2,500 examples only improved accuracy from 74% to 76%. That's just one additional correct prediction out of 50, which is too small a difference to draw any meaningful conclusions.
So, what did I learn? More training data doesn't automatically mean a better model. The quality of the training data matters, and so does how you evaluate the results.
You can see the accuracy results for all four models I tested below:
Bringing It to Life
So, the model was finally trained. The next step was to publish it somewhere so other developers could download and use it.
I was a little worried about this step, but fortunately, Tinker supports pushing fine-tuned model adapters directly to Hugging Face. And it turned out to be surprisingly easy, just one command:
tinker checkpoint push-hf "tinker://c2148695-43ab-5758-aaff-bd78ddda980d:train:0/sampler_weights/final" --repo Gramli/complexity-scorer --public
And that's it. My fine-tuned adapter was available on Hugging Face.
Now, the last thing was to build the API.
As a backend developer, this was probably the most boring part of the project. I simply created a .NET Minimal API with the usual API stuff like API key authentication, rate limiting, exception handling, etc.
Then I added a service to communicate with Tinker API. Yep, Tinker has its own API endpoint:
https://tinker.thinkingmachines.dev/services/tinker-prod/oai/api/v1/
So you can actually call your fine-tuned model without deploying it yourself, which is great for experimenting. Of course, these inference requests still cost money.
I also added a simple Svelte playground so developers can enter their own prompts and see the complexity scores directly in the browser.
Why Does Open Innovation Matter?
Well, I could have built Complexity Scorer by simply calling some powerful closed AI model and asking it to classify prompt complexity. That would probably have been much easier. But then every request would depend on that provider's API, availability, and pricing.
With an open-weight model like Qwen3-8B, I can actually fine-tune it for my specific task, experiment with different training datasets, and share the result with other developers, potentially at a fraction of the cost (or maybe not if you use Sol like I did 🤣). Sure, it cost me some time, but that's part of open innovation.
And here's the biggest advantage: anyone can download my fine-tuned adapter from Hugging Face, combine it with the base model, and run it on their own hardware with a compatible runtime. No need to use my API or pay Tinker for every request.
Other developers can also improve the project, train their own versions, or integrate it into their applications. Of course, running a model locally still requires hardware and electricity, but at least you're not tied to a single API provider.
And that's what I like about open innovation. I'm not just building an API that other developers can call. I'm sharing what I learned while building it and giving them something they can actually build upon.
Prize Categories
I'm entering the Best Use of Tinker category because, as a good old backend developer, I got to tinker with the world of ML engineering, learn something new, share my journey, and actually build and deploy something useful.
Not bad for my first fine-tuning experiment! 😁
Top comments (59)
Great post about fine-tuning! 👍 The part where the accuracy with 50 examples was better than with 2,500 examples was interesting. I hope I can also make a fine-tuned model specifically for AI Avatar. It seems like a good idea to create a free and fun new feature. 🤔
Thanks! Go for it, there's currently a $10 promo code available, which should be plenty for experimenting (as long as you don't let Sol burn through it like I did! 🤣🤣).
I'll be careful not to let Sol burn too many tokens if I try it. 🤣
Every time I read your articles, I feel like such an idiot. 🤣 But yes, this is a really important problem, and it's only going to become more relevant once frontier models start charging us serious money! 😅
Why? You're one of the biggest pros on DEV I know! 😅 Trust me, I was learning half of this stuff while experimenting with it. And yeah, we should start thinking about how to save money and avoid becoming too dependent on AI. Although, judging by my training budget, I'm not exactly saving money yet 🤣
This was such an interesting read. I've experimented a bit with fine-tuning smaller models myself, so I really enjoyed seeing your process and results. The fact that 50 training examples actually outperformed 2,500 is fascinating, and a great reminder that more data doesn't necessarily mean better results. Especially when you're working with synthetic data, where consistency and quality can be tricky.
I also really like the idea behind Complexity Scorer. Being able to route requests based on how much reasoning they actually need makes a lot of sense, especially with how expensive some of these models are getting.
And the part about Sol going rogue with your Tinker credits... 😂 That's a pretty good example of why giving AI agents access to paid services needs some serious guardrails.
Really enjoyed how openly you shared the experiments, unexpected results, and lessons learned. That's the kind of stuff I find most useful in posts like this,
Oh, thanks for such a thoughtful comment!
It was definitely a fun experiment, even if it turned out a bit more expensive than I expected 😅 Luckily, DEV came to the rescue with those promo codes. 😁
Haha, thank goodness for those promo codes! 😂 At least you got a great experiment and an interesting story out of it. Looking forward to seeing what you experiment with next!
Fair question - and the practical answer is that you rarely need the "actual" level per prompt. You need a cheap proxy that orders prompts well enough. Three levers that work in practice:
Shadow routing - run both models on a sample, but serve only the cheap one. The disagreements are your labeled set, and they are exactly the prompts where the routing decision actually mattered.
A stratified human-labeled seed (a few hundred prompts across your predicted bins) to place the uncertainty bands. You need ground truth once per boundary, not forever.
Log predicted vs. served bin, not predicted vs. "actual complexity". Complexity is a latent construct; the observable outcomes (edit rate, retry rate, escalation rate) are what routing should optimize - and they double as the evaluation set.
The asymmetry trick only needs the ordering to be roughly right: one bin off near a boundary is cheap in the over-estimate direction. So "what is the actual level" can be postponed - start with the proxy and let the disagreement data tell you where the scorer is actually wrong. Open to comparing notes on the bin boundaries?
Thanks for the detailed explanation!
Shadow routing sounds interesting for evaluation, but I'm also concerned about the additional costs. And disagreement between two models doesn't necessarily mean one of them is wrong, so we'd still need some way to evaluate their answers, right?
The stratified human-labeled seed makes sense, and logging the predicted vs. served bin sounds very practical too.
As for the routing boundaries, I haven't thought about those yet. My original idea was to let the customer decide. But I'd definitely be interested to hear how you've approached them.
routing on a 0 to 1 complexity score is a neat idea, but the number i'd worry about is how often a hard prompt gets scored easy. that error is silent, the cheap model just answers confidently and wrong, while the opposite error only costs you money.
we handled it with a cheap escalation check on the small model's answer, and logged every escalation so thresholds could be tuned on real traffic instead of the training set.
have you measured the score on prompts that look short but are actually hard? that's where our router fell over first.
Good point. I haven't specifically tested short but difficult prompts, but in my evaluation, all 12 incorrect predictions were only one complexity level off. Of course, the evaluation dataset was quite small.
I'm curious about your escalation check, though. How did you implement it without adding too much extra cost?
fair, one level off is the forgiving failure. the dangerous misses are the confident ones on short hard prompts, which is why we checked those separately. our escalation check was just a second call to the small model grading its own answer for missing steps or unsupported claims, one token verdict, so it cost a fraction of the original call. anything flagged went to the big model and got logged, and that log turned into our real eval set.
did you keep those 12 misses around as a regression set?
That’s an interesting approach. But I’m curious how reliable the small model is at detecting mistakes in its own answers, especially when it’s confidently wrong.
As for those 12 misses, yes, they’re still part of my original evaluation dataset, so they’re tested every time I run the baseline script. I haven’t separated them into a dedicated regression set, though.
reliability is the fair worry. a small model grading itself misses its own confident errors a lot, so we never trusted it to verify facts, only to flag structural gaps like skipped steps or claims with no support in the context. unflagged ones got hand sampled weekly to measure the miss rate, and that sample told us more than the verdict did. since your 12 misses already live in the baseline, you could tag them as a hard slice and track that score on its own. are you planning to grow the eval set?
The part of your results that deserves more skepticism isn't 50%→76%, it's 86%→76% — and I don't think either comparison says what it looks like.
With n=50 on the eval set, 76% has a 95% confidence interval of roughly ±12 points, and 86% is roughly ±10. Those bands overlap so much that "more data hurt" could be pure sampling noise. Twelve misses versus seven is a handful of borderline prompts, not a trend.
The second confound is the labels themselves. Your training labels came from one generator, the eval set has its own unknown provenance, and complexity is a genuinely subjective five-way judgment — two good labelers can disagree 15–20% of the time without either being wrong. So both accuracy numbers are partly measuring label agreement, not model quality. Before drawing any lesson about dataset size, I'd hand-label 50–100 eval prompts and check agreement between your two synthetic labelers. That one afternoon would tell you more than another training run.
The router architecture is still the right economic bet, though — it's the textbook case of matching a small open model to the job. This piece makes the same calculus explicit (real costs of open weights vs closed APIs, and when each wins): this breakdown of the open-weight vs API cost tradeoff
Thanks for the feedback.
I actually mentioned both the small evaluation dataset and potentially inconsistent AI-generated labels as possible explanations in the article. I definitely wouldn't draw strong conclusions from just 50 evaluation examples.
Manually reviewing the labels is a good suggestion.
Thanks Daniel. Yeah, 50 examples can show a big jump but not small deltas. A quick manual pass over the labels would clear it up fast — would be interesting to see the numbers after cleaning them up.
Nice write-up! The complexity-scorer routing pattern sounds obvious in hindsight, but it rarely gets measured properly in practice. One question on the 76% accuracy: how do you handle the asymmetry between the two error types? A false 'complex' only costs a bit of extra compute, while a false 'simple' silently degrades answer quality. That is the failure mode I would want a fallback for (e.g. re-route to the bigger model when the small one is unsure, or whenever the prompt carries code / long context). Did you look at that split separately?
Good point!
I didn’t analyze overestimates and underestimates separately. I only measured accuracy per complexity level and noticed that all 12 incorrect predictions were one level off.
And since Complexity Scorer currently only estimates complexity, I haven’t tested how those mistakes would affect actual routing decisions yet.
I like the fallback idea, though. Definitely something worth exploring.
That +-1 level spread is the useful part: if every miss lands on an adjacent bin, the error is cheap to absorb. Route one bin up whenever the scorer sits near a boundary and you turn underestimates (the quality-risk direction) into overestimates (the compute-cost direction) - the safer direction to be wrong in. Logging predicted vs. actual bin on real traffic would then give the routing-impact number without a new experiment.
Interesting idea, favoring the stronger model when the scorer is uncertain sounds like a reasonable approach.
But how would you determine the actual complexity level of real user prompts in production? That's something I'd still need to figure out.
Really enjoyed this write-up. The jump from 50% to 76% is encouraging, and the budget story was painfully relatable. Since the test set had only 50 examples, I’d be curious to see how the model performs on a larger one. It’s also interesting that the incorrect predictions were only one level off. Setting a spending limit before giving an AI agent access to paid tools is a lesson I’ll remember for my own experiments.
The most useful result here is the one you flagged as a surprise: 50 training examples scoring 86% while 2,500 scored 76%. With a 50-item evaluation set, each question is worth 2 percentage points, so that 10-point gap is only five prompts, and the 500-to-2,500 jump (74% to 76%) is a single prompt. I’d be careful treating either as a ranking of the models. Your two explanations, noisy AI-generated labels and a tiny eval set, are both plausible, and they point to the same next step: a larger held-out set, ideally with a portion labeled by humans, and a second labeler to measure how often the labels agree with each other. If agreement between labelers is only around 70-80%, that sets a practical ceiling on how accurate the scorer can ever look.
Your finding that all 12 misses were off by exactly one level matters more for the actual use case. For routing, an off-by-one error between 0.50 and 0.75 only matters if your routing thresholds sit right at that boundary, so measuring mean absolute error alongside accuracy, or checking how often a misroute sends a hard prompt to a cheap model, would tell you more than exact-match accuracy. Underestimating complexity is costlier than overestimating it, so you could also bias the scorer toward the higher level when uncertain.
One more thought: the scorer measures how hard a prompt looks, but the routing question is really whether the cheap model would have answered correctly. Labeling a sample by running both a cheap and an expensive model and checking whether they agree could give you training labels tied directly to the savings you’re after.
And the Sol background-training story is a great lesson in scoping agent permissions. Thanks for including the costs and the dead ends.
Thanks for the detailed feedback!
I like your point about measuring actual routing mistakes instead of just classification accuracy. So far, I've only tested how well the model predicts complexity, not whether those predictions would actually save money.
Testing real routing decisions with cheaper and more expensive models would definitely be an interesting next step.
Solid first fine-tuning experiment, and a practical idea.
Routing by estimated complexity is one of the more concrete ways to cut LLM costs without blindly downshifting everything to a weaker model. The score scale (0.00–1.00 with clear reference levels) is simple enough to actually use in a decision layer.
What stands out most is the honesty about the data side. Going from 50% → 86% with only 50 examples, then seeing accuracy drop when the dataset grew (and only recover to 76% with 2,500 examples) is a classic illustration that label consistency and evaluation design often matter more than raw volume — especially when the training data is itself AI-generated. The fact that all errors were off-by-one is also encouraging; the model seems to have learned a reasonable ordering even when it missed the exact bin.
The Sol incident is a useful cautionary tale for anyone giving agents access to paid training or inference endpoints. And shipping both the Hugging Face adapter and a small .NET API + playground makes the work immediately usable for others.
Nice work turning a Hacktoberfest side hustle into a real, open, cost-saving tool.
Thanks for taking the time to read it!
Some comments may only be visible to logged-in visitors. Sign in to view all comments.