I tested every tiny local LLM worth running, and only three survived the cut

Running AI locally used to be something you either needed real hardware for or a lot of patience with slow tokens and suboptimal outputs. That's shifted a lot in the last year or so. The smaller models coming out of Google, Alibaba, and a few others are actually built for the hardware most people already own, so you don't need a home server rack or a 24GB GPU to have something useful running on your machine.

My hardware is modest by any measure, and yes, it can sometimes be a limitation. But I've been leaning into it because it forces me to pick models that fit rather than models that only impress on paper (a high advertised context window means nothing if your GPU can't support it). I've tried a lot of sub-4B models over the past few months, and most of them are fine as demos and not much else. The three below are the ones I actually kept, though, and one of them isn't even from this year.

Google's smallest edge models

Tiny footprint, but punches above its weight

Gemma 4 E2B is the smallest variant in Google's Gemma 4 family, which dropped at the end of March 2026. The "E" stands for effective parameters, and the trick is Per-Layer Embeddings, where each decoder layer gets its own small embedding table used as a lookup rather than full compute. So the model carries more weights than the 2B label suggests, but the active footprint stays at 2B-model levels. And this is how it manages to run so smoothly on limited hardware.

Gemma 4 also uses a hybrid attention mechanism that interleaves local sliding window attention with full global attention, so long context doesn't blow up memory usage. It's natively multimodal with text, image, and audio input, 128K context, 140+ languages, and function calling. Basically, all the features of a much bigger model, but squeezed into a small package.

It's safe to say that E2B is my daily driver on my phone and Chromebook, and I'd say the main reason is its personality. It's quite warm and expressive, meaning it gives you a similar experience to popular cloud-based AI like ChatGPT and Claude. The way it lays out its responses is closer to a cloud chatbot than anything else I've tried at this size; it even does that somewhat cringe thing of dropping emojis into the responses unprompted.

Beyond the personable chat style, it's also one of the stronger vision local models out there. I constantly use it for working through screenshots of design projects, and even send it photos on my phone of things I don't quite understand, such as an ingredient list on a cleaning product. Depending on your runner, you may or may not be able to utilize its audio features. LM Studio on my PC and PocketPal on my phone don't accept audio input, so I recommend using llama.cpp if you want to be able to upload audio files - it's highly accurate.

I still run it on my PC even though I've got room for 12B, because the breathing room lets me keep a million browser tabs and my design editor open at the same time. Chromebook is the funny one. I keep local AI on it for the times my internet cuts out. It's not fast, but it still works.

And I have to give a shoutout to E4B while I'm at it: it is the same architecture one tier up and is a bit better at reasoning and longer sessions, so if your hardware isn't super limited, it's probably the safer pick.

Qwen 3.5's small models bring flagship architecture to small devices

And they're not stripped-down versions of the bigger variants

The Qwen 3.5 small series dropped in March 2026 - four dense models at 0.8B, 2B, 4B, and 9B parameters, all released two weeks after Alibaba's flagship 397B MoE. What matters here is that the small models weren't stripped down from something bigger; they were built on the same Gated DeltaNet hybrid architecture as the flagship. That's a 3:1 ratio of linear attention to full softmax attention, which is how they keep the KV cache small at long contexts. And "long" here is genuinely long - 262K native context across the whole small lineup, which is wild on a 2B or 4B.

They're natively multimodal too, with text and image input trained in from the start rather than a vision encoder added on afterward. There are also 201 languages and dialects (up from 119 in Qwen 3), and tool calling was leaned into hard during training with reinforcement learning across million-agent environments. Overall, this is an insanely capable model family, which is why it's been my default reach since forever.

Qwen 3.5 9B was my first genuinely smooth local LLM on my PC, and it was a massive step up from wrestling gpt-oss 20B onto 8GB of VRAM before that. So when the small siblings came out, I already knew I wanted them on my phone (though they're also great options for limited PCs).

The 2B runs really smoothly on my iPhone 16 through PocketPal, and the 4B pushes the phone a bit harder but is still comfortable, plus it's noticeably sharper on reasoning and structured tasks than any other LLM I've run on my phone. The personality is different from Gemma - less warm, more grounded, more focused-assistant than friendly-chatbot. Which sounds like a downside but is actually what I want when I'm trying to get something done rather than chat.

It's also been my go-to for local design work. Qwen's tool calling is better than Gemma's at this size, so it slots into local vibe design tools cleanly, and the structured component code holds up. Vision technically works on both models, but I still reach for Gemma when the input is image-heavy and lean on Qwen more when the output needs structure.

And an older coding model from Qwen

As someone who doesn't code, I didn't expect to get use out of a coding model

Qwen 2.5 Coder 3B Instruct is the oldest model on this list by a mile, dropped in November 2024 by Alibaba's Qwen team as the small end of a series that scales up to 32B. Even now, it's one of the very few coding-tuned models I could actually get running on my 8GB VRAM without compromises.

At Q4_K_M the GGUF is around 1.9GB, so my RTX 3070 offloads the whole thing and still has plenty of room for a decent context window. It's built on the Qwen 2.5 base (already strong at instruction following and structured output), then trained further on 5.5 trillion tokens of source code, text-code grounding, and synthetic data. It covers 92 programming languages, including a lot of niche ones.

So why am I even bothering with a coding model when I don't code? I have a lot of files on my PC that are code or shaped like code at least, so I needed a model that can handle that type of work. This includes stuff like YAML frontmatter, JSON config files, Docker Compose files, text-to-SQL, and things along those lines.

A general model treats formatting rules as a suggestion because it's optimized to cater to the user as much as possible, prioritizing sounding helpful. But this one treats code and code-adjacent formats as its whole identity. So when I need to add a new MCP server to my Claude Desktop config or standardize the YAML across a batch of notes, I hand it over and get it back clean. The "something broke" workflow is the other one - paste in a Docker error or a Python stack trace, and it actually finds the line I need (but never would have found on my own).

Tiny doesn't mean compromised anymore

None of these three were on my radar as "the ones I'd keep" when I first tried them, and I did kind of consider them as less capable versions of their bigger siblings. But that's kind of the interesting part of local AI right now; the smaller end of the model spectrum has genuinely caught up. If you've been holding off because you assumed your hardware couldn't handle it, I'd give at least one of these a shot.