Small vs large language models: choosing the right size

Choose a small language model when the task is narrow, repeated many times and has an output you can check, such as sorting tickets, pulling fields out of invoices or rewriting text to a template. Choose a large model when the task is open-ended, needs several steps of reasoning or depends on broad knowledge the model must carry in its weights. The parameter count is a starting guess; the decision belongs to a test set built from your own traffic.

Teams skip that last part. They pick the biggest model for the prototype, because the demo works on the first try, and never revisit it. Six months later the same model that drafts contract summaries is also deciding whether an email is a refund request or a delivery question, at a price and latency that task never needed.

What "small" means, roughly

There is no official threshold. The most useful definition I have found is a practical one from an NVIDIA research paper, Small Language Models are the Future of Agentic AI (June 2025): an SLM is a model that "can fit onto a common consumer electronic device" and can answer one user's requests with latency low enough to be practical. The authors add that, as of 2025, that meant "most models below 10bn parameters in size".

I like it because it is about deployment, not prestige. A 3B and an 8B model both run on a single modest GPU or a laptop; a 70B model does not without heavy quantization or several cards. That hardware line is where cost and deployment options change sharply.

Small models have also become much better than their size suggests. Microsoft's Phi-3 report (April 2024) describes phi-3-mini, a 3.8 billion parameter model trained on 3.3 trillion tokens, which scores 69% on MMLU and 8.38 on MT-bench, "despite being small enough to be deployed on a phone". The report credits much of this to the training data: heavily filtered public web data plus synthetic data, rather than more parameters.

What you lose when you shrink a model

Parameters do two jobs. They store knowledge, and they encode skills such as following instructions, keeping to a format and reasoning over what is in the prompt. Shrinking a model hits the first job harder than the second.

The Phi-3 authors say this plainly about their own model: "The model simply does not have the capacity to store too much 'factual knowledge'", which shows up as weak results on TriviaQA. Their suggested fix is equally plain: augment it with a search engine.

That split is the most practical idea in this whole topic. If a task mainly needs skill applied to text you supply (classify this, extract that, rewrite in this tone), a small model has a fair chance. If a task needs the model to know things nobody put in the prompt (an obscure regulation, the history of a supplier dispute, a niche API), a small model will guess, and guessing confidently is worse than failing.

For company questions the answer is usually the same either way: put the facts in the prompt with retrieval. Our article on RAG covers how. Once retrieval supplies the facts, much of the large model's advantage on "knowing things" disappears, and the comparison becomes about reading and reasoning over the retrieved text. CloudOEngine's AI runtime includes RAG next to the LLM gateway and agents, so the facts a smaller model needs can come from your connected data instead of its weights.

What retrieval does not fix is reasoning depth. A question like "given these three contracts and last quarter's delivery log, which supplier is most likely to breach its SLA and why" needs the model to hold several documents in mind, compare them and argue a case. Small models tend to lose the thread on this kind of work. I would not try to save money there until a test proves otherwise.

The memory arithmetic

Size matters most when you host a model yourself, and the first constraint is memory. Hugging Face's guide to optimising LLMs gives a rule of thumb that holds up well: loading the weights of a model with X billion parameters takes roughly 4 × X GB in float32 and roughly 2 × X GB in bfloat16 or float16, the precision the guide says almost all models are now trained in.

Quantization reduces it further. The same guide quantizes a 15.5B coding model and measures peak memory falling from about 31 GB in bfloat16 to about 15 GB in 8-bit and about 9.5 GB in 4-bit, with little visible change in output for that example. It also warns that 4-bit results can differ from full precision in practice and that quantized inference is often slightly slower, not faster, because weights are dequantized on the fly. Phi-3's report gives the extreme case: quantized to 4 bits, phi-3-mini occupies about 1.8 GB and generates more than 12 tokens per second on an iPhone 14, fully offline.

Here is the weights-only arithmetic for a few common sizes. It is a sizing illustration from the rule of thumb, not a benchmark.

The table leaves out the key-value cache, and that is where long prompts bite. The cache stores attention keys and values for every token in the context, so it grows with prompt length and with the number of concurrent requests. In the Hugging Face guide's worked example, a single 16,000-token input on that 15.5B model would need around 15 GB of cache with standard attention, roughly half the size of the weights; attention variants such as multi-query and grouped-query attention, which many models now use, shrink it a lot. A model that fits comfortably for short chats can run out of memory once you feed it long retrieved documents, so size your hardware with your real prompt lengths, not the model card.

Cost and latency

If you call a hosted API, memory is the provider's problem, but the same physics shows up in the price list and in response times. Fewer parameters mean less compute per token.

The NVIDIA paper puts a number on it: serving a 7B SLM is "10–30× cheaper (in latency, energy consumption, and FLOPs) than a 70–175bn LLM". Treat that range as an order of magnitude, not a quote for your workload. It does explain why size is usually the biggest single lever on the cost of an AI feature. Our article on LLM cost control covers the other levers, such as caching and per-team budgets, and the routing patterns that send each request to the cheapest model that can handle it.

Latency deserves its own thought. A support agent waiting for a suggested reply feels the difference between 300 milliseconds and three seconds. Agents make this worse because one user request can trigger five or ten model calls in sequence, and every call pays the latency again. Using a small model for the routine steps inside an agent (choosing which tool to call, formatting arguments, checking a result against a schema) often does more for responsiveness than any infrastructure change.

Where the data is allowed to go

Size also decides where a model can run, which matters for privacy and regulation as much as for cost. Meta's Llama 3.2 announcement (September 2024) presents its 1B and 3B models as built for on-device use cases such as summarization, instruction following and rewriting, with a 128K-token context, so developers can build applications "where data never leaves the device".

A model you can run on a laptop, a phone or a single server in your own network gives you options a very large hosted model does not: processing customer records without sending them to an outside API, working in a factory with poor connectivity, or keeping a regulator happy about where personal data is processed. If any of those constraints apply to you, they may settle the size question before quality even comes up.

Small models can beat big ones, on one job

The most interesting results in this area come from specialisation. A general model has to be decent at everything; a model tuned for one task only has to be good at that task.

The Distilling step-by-step paper (Hsieh et al., 2023) is a clean example. The authors trained small T5 models on labels and on rationales generated by a large model. Their fine-tuned 770M-parameter model "outperforms the few-shot prompted 540B PaLM model using only 80% of available data" on a benchmark task, while standard fine-tuning of the same small model could not match PaLM even with all of the data. That is a model roughly 700 times smaller winning on one job.

Model makers use the same idea. Meta says its Llama 3.2 1B and 3B models were made by structured pruning of Llama 3.1 8B and then distillation, using outputs of the 8B and 70B models as targets during pre-training. The large model is still in the picture; it just works as a teacher instead of answering every request.

This is why I would not frame the question as "small or large" for an established product. The more useful framing is: which of my tasks are stable enough that a large model's outputs can train a small one to do them? The NVIDIA paper describes exactly this as a conversion process: log the agent's model calls, clean and anonymise them, cluster the prompts to find recurring tasks, pick candidate small models, fine-tune them (for example with LoRA), and keep retraining as usage changes. In their case studies of three open-source agent systems, they estimate that roughly 40% to 70% of the LLM queries could be handled by specialised SLMs. Those are the authors' estimates for those systems, not a measurement of yours.

The first step of that process only works if you have the logs. CloudOEngine traces every agent step and tool call and makes it inspectable, which gives you the record of real prompts and outputs a comparison like this needs to start from.

How I would choose for a new feature

For a feature that does not exist yet, I start large and then move down, because the large model tells you quickly whether the task is possible at all. Starting small risks blaming the idea for the model's limits.

1. Write the task down as narrowly as you can, including the exact output format. "Return one of these seven categories as JSON" is testable; "help with customer emails" is not.
2. Get it working with a large model and collect 100 to 300 real inputs with answers your team agrees are right. Our guide to evaluating LLM applications covers how to build that set without fooling yourself.
3. Agree on the quality bar with whoever owns the business outcome, before you look at any small-model results.
4. Run two or three small models against the same set, with the same retrieval and the same prompt adjusted only for format. Compare quality, latency at your real prompt length and cost per correct answer.
5. If a small model clears the bar, ship it with a check on its output (a schema, an allowed-values list, a confidence threshold) and send anything that fails the check to the large model.
6. If none clears the bar but they come close, that task is a candidate for fine-tuning or distillation once you have a few thousand logged examples.

Step 5 is where most of the savings and most of the safety come from. A small model that is right 95% of the time with a check that catches most of the other 5% is a better system than either model alone. And keep sampling live outputs after launch, because inputs drift and a model that cleared the bar in March may not in September.

When the small model's output triggers an action rather than a label, such as refunding an order or closing a ticket, add a person to the loop for the actions that matter. CloudOEngine's Autopilot runs triggered agents with approval rules, so a person approves an action in the app or through an email approve link before it runs, whichever model proposed it.

Mistakes worth avoiding

The common one is choosing by leaderboard. A model's MMLU score tells you about multiple-choice knowledge questions, not about whether it extracts GST numbers from your scanned invoices. Benchmarks are a way to shortlist candidates, nothing more.

The second is comparing models with different context. If the large model got the retrieved documents and the small one got a shortened prompt to save tokens, you measured the prompt, not the model.

The third is forgetting the long tail. A small model can score well on an average test set and still fall apart on the rare, weird input: the email in two languages, the invoice photographed at an angle, the ticket that is really a legal threat. Include those cases deliberately in the test set, and make sure the escalation path catches them in production.

The last is treating the decision as permanent. The right size for a task this year may be one size down next year, so keep model calls behind one interface and the swap stays a configuration change.

How CloudOEngine helps

CloudOEngine is one platform and one console for connecting data (databases, files and camera streams), running pipelines, and running AI through an AI runtime with an LLM gateway, agents with tool access and RAG, alongside Vision AI and live video. Everything shares one permission model and one audit trail.

For model sizing this matters in two ways. The LLM gateway sits in the same runtime as your agents, so model calls go through one place. And RAG over your connected data supplies the facts a smaller model cannot hold in its weights, which is often what makes a smaller model viable in the first place.

Agents run real tools, for example filing a ticket, and every step and tool call is traced and inspectable. Those traces show the real prompts, tool calls and outputs, which is what you compare models on.

Autopilot runs triggered agents with approval rules. A person approves an action, in the app or through an email approve link, before it runs, which keeps a human check on consequential actions regardless of which model proposed them.

Governance is included, not upsold: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets. CloudOEngine is built in Bengaluru, India.

Book a demo or see the AI runtime.

Frequently asked questions

What counts as a small language model?

There is no formal cut-off. One NVIDIA research paper defines an SLM as a model that fits on a common consumer device and answers one user's requests with practical latency, and says that as of 2025 that meant most models below 10 billion parameters.

When should I use a small language model instead of a large one?

Use a small model for narrow, repeated tasks with output you can check automatically, such as classification, field extraction, routing and short rewrites. Keep a large model for open-ended reasoning, messy long documents and questions that depend on broad world knowledge.

Are small language models less accurate?

On broad knowledge and hard reasoning, usually yes. On a narrow task they can match or beat much larger models once tuned: the Distilling step-by-step paper reports a 770M-parameter model outperforming the few-shot prompted 540B PaLM on a benchmark task.

How much GPU memory does a model need?

A common rule of thumb from Hugging Face is about 2 GB per billion parameters for the weights in bfloat16, so a 7B model needs roughly 14 GB before the key-value cache. 8-bit and 4-bit quantization cut that to about a half and a quarter.

Can a small model answer questions about our company's data?

Not from memory, because it has little room to store facts. Give it the facts at query time with retrieval (RAG); the Phi-3 report itself says its weak factual recall can be addressed by adding a search engine.

Sources

1. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (Abdin et al., 2024)
2. Small Language Models are the Future of Agentic AI (Belcak et al., NVIDIA, 2025)
3. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (Hsieh et al., 2023)
4. Optimizing LLMs for Speed and Memory (Hugging Face Transformers documentation)
5. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models (Meta, 25 September 2024)

#small language models #llm selection #model size #distillation #quantization #llmops