Nvidia isn't the only choice for local LLMs anymore, and my AMD test proves it

For years, AMD has historically lagged behind Nvidia for most AI-bound workloads. Nvidia has almost always been the more performant choice, even if this came at a premium, which still remains partly true to this day.

Thankfully, AMD seems to have (mostly) caught up with Nvidia and their Radeon Open Compute (ROCm) software stack, which is a surprisingly pleasant experience on Linux.

Recent hardware releases like the Ryzen AI series also helped, and to put this to the test, I took out my trusty ROG Flow Z13 — which has 32GB of shared memory and runs on an AMD Ryzen AI Max 390 chipset. It's both a boon and a curse, as the AI Max "shares" that pool of memory for both its VRAM and system RAM, which complicates things a bit.

Getting everything up and running on Arch Linux

And a look into the test system

Installing Ollama on Arch Linux was quite straightforward. All I had to do on my system was fire up a terminal and download the latest version of the AUR using Pacman.

Since this was an all-AMD system, I had to cherry-pick packages. The two that I chose were ollama-vulkan and olla-rocm, followed by enabling the service daemon system-wide.

It’s also possible to entirely install Ollama using a web installer script, but I intentionally chose not to do that. The reasoning behind it would be to opt in for centralized updates, instead of relying on Ollama for updates separately.

With Ollama set up, it was finally time to deploy some models:

• gemma2:9b
• mistral:7b
• phi4:14b
• deepseek-r1:8b
• deepseek-r1:14b
• llava:7b
• llava:13b

We also have to take into mind hardware limits. My particular model of the Flow Z13 uses the lower-specced AI Max 390 chipset, with a Radeon 8050S iGPU that has 8 fewer CUs than the flagship 8060S.

It also has “only” 32 gigabytes of shared memory between the system and iGPU, which will limit the number of models we can run. That being said, it is possible to allocate up to 24GB of VRAM from within the BIOS, which should be on par with most high-end consumer GPUs.

However, that’s only half the story — we still need to take memory bandwidth into account, which is roughly 256 GB/s for the AI Max. Compare this to the RX 9070 (640 GB/s), which is one of the absolute best offerings from AMD, and it’s quite obvious that this might not be all that impressive, at least at first glance.

Running the benchmarks

Some rather surprising results for ROCm versus Vulkan

For the benchmarking, I used the llm-benchmark suite, which automatically pulls in necessary dependencies.

Switching between Vulkan and ROCm backends, I ran a series of benchmarks that took a while to complete. The machine was pushed to its absolute limit, with max TDP in place.

From the graphs, it was quite surprising how close ROCm and Vulkan were in terms of performance. For the most part, they performed largely the same. However, I did run into “out of memory” errors on ROCm quite frequently. As such, I’m hesitant to recommend ROCm for the moment — stick to Vulkan for a more stable experience.

The 32 CU Radeon 8050S performs admirably because of its small form factor and lower TDP, going close to the Nvidia RTX 4060 Mobile, which is a much more power-hungry mid-range GPU. While the RX 4060 Mobile has the greater grunt, it will inevitably run into bottlenecks thanks to its measly 8 gigabytes of VRAM.

The Strix Halo, in contrast, has access to the full 32GB of RAM, which makes it an obvious choice for heavier LLMs. This is a pattern with AMD you’ll come to get used to.

A desktop RX 9070 XT, on the other hand, has the best use case of 95 tok/s on 7B models, which is roughly a 2x step-up over the 8050S. It also has far better value for money and memory bandwidth, which makes it an easy pick for most.

Still, when compared to Nvidia’s RTX 5090, all of AMD’s offerings fall laughably short of expectations. The 5090 is simply a much better card for local AI, but it’s also unreasonably expensive for most folks. AMD is just a much saner pick that also happens to be a lot lighter on your wallet, offering competitive performance for its price.

Performance is reasonably good, but value is where AMD truly shines

While I cannot deny that Nvidia currently offers the absolute best for LLMs, AMD has done well to catch up in recent years. Both ROCm and Vulkan have had significant improvements, and AMD is now a surprisingly competitive alternative.

There isn’t much reason to shell out extra for an Nvidia card — especially when we consider how severely gimped they tend to be with VRAM.