Stop Using Your Smartest Models for Your Simplest Tasks
The Small Model With One Job
One of our engineers, Andreja Tonev, wanted to see what a very small language model could do.
He gave it one job: translate plain-English questions into Cypher queries for Memgraph.
He chose an open-source model with roughly one billion parameters and started fine-tuning it overnight on his laptop. Training used the laptop's GPU, but the resulting model could run on a CPU for inference. Later versions went through additional rounds of training.
Andreja built the training dataset backwards. First, he generated Cypher queries programmatically using the language's grammar. Then he asked another language model, DeepSeek, to come up with different ways a person might ask for each query.
An unusual approach to an examination, admittedly, but a perfectly reasonable way to create a training dataset.
The first experiment produced approximately 100,000 training examples, at an estimated generation cost of $20.
Andreja subsequently expanded the process. The later version involved hundreds of synthetic database schemas, roughly 69,500 verified queries, and 270,000 training rows. The LLM API calls used to generate schemas and requests for that dataset cost approximately $26.
On the narrow task he'd trained it for, the model worked well enough to surprise him. For some inputs, it generated queries in under a second, without needing a dedicated GPU. Andreja later added failed queries and database error messages to the training data, teaching the model to attempt corrections.
These aren't hand-written demonstrations of what the model ought to produce. They're queries generated by the fine-tuned model against databases it hadn't encountered during training. In both examples, the queries returned the expected results.
Andreja had assumed that doing anything particularly useful with a neural network required much more powerful hardware, particularly a GPU.
His laptop had other ideas.
And it left us wondering why we're so quick to reach for some of the world's most capable language models, even when the job at hand might require considerably less.
The Curious Habit of Using Our Smartest Models for Everything
A powerful general-purpose model is wonderfully convenient when you're building an AI application. Give it a task, provide some context, and hope that somewhere among its enormous collection of learned patterns lies the answer.
Often, there is.
Take natural-language database querying.
The ability to translate questions into database queries isn't new. Tools such as Memgraph's GraphChat already support natural-language-to-Cypher generation, error handling, and tool invocation.
Andreja was interested in how small and specialized the model performing the query-generation task could become.
There was also a practical reason for specializing it.
Cypher is used by several graph databases, most notably Neo4j and Memgraph, but their implementations differ in certain syntax, available procedures, and supported functionality.
As Andreja points out, general-purpose models have likely encountered far more Neo4j examples than Memgraph-specific ones during training. A model can therefore generate a query that looks perfectly reasonable but uses syntax or procedures that Memgraph doesn't support.
Fine-tuning on Memgraph-compatible queries could steer a smaller model toward the right conventions, without requiring extensive instructions every time.
Now consider a GraphRAG system. A user's question might trigger several operations, from identifying relevant information and retrieving it from a graph to assembling an answer.
Some questions require substantial reasoning. Others can be answered through relatively predictable queries against a known schema.
Yet it's tempting to send both through the same powerful general-purpose model because, well, the model can handle them.
This is a little like employing a brilliant mathematician to count the chairs in a meeting room.
The mathematician will probably do an excellent job. They may even develop an elegant proof that the chairs exist. But you might reasonably wonder whether the job called for quite so much mathematical talent.
A specialized model trained to generate database queries doesn't need to explain quantum mechanics, write convincing dialogue, or compose a sonnet about a missing semicolon.
It needs to understand a question, work with the available schema, and generate an appropriate query.
Andreja's experiment suggests that a much smaller model might be capable of doing that narrow job.
How reliably it can do so outside a controlled experiment is another matter.
Intelligence Comes With an Invoice
There are good reasons to use frontier models. Their flexibility can save enormous amounts of development work, especially when requirements are uncertain or questions vary widely.
But flexibility has an operating cost.
Hosted models may charge by token, introduce network dependencies, and add latency. A single user question can trigger several model calls before the application produces an answer.
When you're processing thousands or millions of similar requests, those costs add up.
Some organizations also need to consider where their data goes, which external services receive their requests, and how dependent they want their applications to be on external providers.
A small model running locally could be an attractive alternative for certain tasks. If it can generate reliable queries on ordinary hardware, you may be able to avoid external inference calls for that part of the workflow and retain more control over deployment and data handling.
Of course, running locally doesn't make inference free.
You still need hardware, maintenance, monitoring, and someone who knows what to do when the model develops an unfortunate relationship with your database schema.
Nor does a query returned in under a second on Andreja's laptop establish that his model is faster or cheaper than a hosted frontier model. We haven't made that comparison under comparable conditions.
And the cost of generating a query only tells part of the story.
A cheap query that returns the wrong answer isn't much of a bargain. Neither is a fast query that needs three retries, a larger model to repair it, and an engineer to explain why it was wrong.
What matters is how much it costs to get a correct, safely executed result, including the failures, retries, and work needed to keep the system running.
That calculation might favor a small specialized model. It might also be less flattering than the inference bill first suggests.
The Last 10% Is Where the Trouble Lives
Andreja put it rather well:
"Like everything with AI, it is very easy to get to 90% working and a nice demo. But that last 10% is probably going to be a challenge."
He wasn't reporting a measured 90% success rate. He was describing the distance between a promising experiment and something people can rely on.
The later evaluation results show why he's cautious. On held-out synthetic domains, the final model returned correct results for 94% of requests. It reached 97% on unseen merged graphs and 84% on held-out real databases. But on a separate public Text2Cypher benchmark involving three held-out databases, accuracy dropped to 41%.
These are different evaluation sets, so the percentages aren't directly comparable. The public benchmark also has its own limitations: approximately 20% of its reference results could not be reproduced.
Still, the variation is revealing. The model can perform remarkably well on certain unfamiliar databases while struggling considerably more with other questions. Getting a query to work under controlled conditions is only part of the problem.
He suspects the main limitation of his current model is the dataset rather than the model's capacity. The experiment hasn't yet established how far that holds as the task becomes more complicated.
The original dataset covered plain Cypher queries. What happens when a user needs a graph algorithm or a custom procedure?
You can't teach the model every procedure it might encounter. Different environments expose different capabilities, and those capabilities can change.
Ideally, the model would read the descriptions of available procedures, work out which ones are relevant, and use them correctly, even if those procedures never appeared in its training data.
A procedure might be essential to one query and entirely irrelevant to the next. The model has to know the difference.
Then there's query optimization.
Andreja generated the original queries programmatically but hadn't checked whether they were the most efficient versions. Teaching a model to produce efficient queries requires more than a collection of examples that happen to execute successfully.
And successful execution doesn't guarantee a correct answer.
A query can run perfectly well while answering the wrong question. The database won't necessarily complain. As far as it's concerned, it did exactly what it was told.
Databases are rather literal creatures.
Andreja has already experimented with feeding failed queries and database error messages back to his small model so it can attempt a correction.
A syntax error gives the model something concrete to work with. It can use the error message to generate another query.
But a query that executes successfully and returns a plausible, incorrect result may produce no error at all. The system still needs some way to determine whether it answered what the user intended.
Even much larger models misunderstand questions. And a model trained specifically to generate database queries has an additional problem: the user might not be asking about a database in the first place.
A model trained to generate queries may enthusiastically produce one even when nobody actually asked for a query.
It has one job, after all. The possibility that the job doesn't need doing is apparently a separate subject.
Teaching it to recognize requests outside its scope means more training examples, more evaluation, or additional safeguards.
Andreja estimates that covering the full range of functionality could require ten or even a hundred times as many examples as his original experiment, along with a more complicated process for generating them.
Those initial API costs were surprisingly low. But generating training examples is only one part of the expense. Training, evaluating, and maintaining the model all require additional resources, and expanding the dataset to cover more complex cases means more engineering work.
The savings from cheaper inference could eventually disappear into the cost of maintaining the specialized system.
For certain operations, a predefined tool might be more reliable than asking a language model to generate a query from scratch. If the operation is known, its inputs are well-defined, and its behavior can be tested, generating new query text every time may introduce complexity we don't need.
Some of the tasks we're trying to assign to smaller language models might be better handled without a language model at all.
The Right Amount of Clever
Andreja wanted to see what a small model could accomplish on his laptop. It managed more than he expected, but it also left him with a collection of engineering problems that won't disappear merely because the model is small.
A frontier model may still be the simpler, more economical choice for some applications, once development and maintenance costs are included. For others, a smaller specialist might do the job. And for certain operations, ordinary software may be more reliable than either.
We have plenty of reasons to keep developing increasingly capable language models. We could also spend a little more time finding out how much of that capability each task calls for.
Andreja has already taught his model to generate database queries in under a second.
Teaching it when not to generate one may take considerably longer.
