Where the AMD Work Started: Running 321B at Home With Project Maya

I never expected to run a 321B model at home. A month ago I had nothing shipping on AMD, and then Project Maya appeared one evening: a way to run GLM-5.3-Flash on consumer GPUs. I jumped on it that night. A few days later my ROCm port was merged upstream as experimental AMD support, and that port is where all of my AMD work since started.

Step back and consider what should be impossible about that sentence. A 321-billion-parameter model, answering questions in a browser window, on a desktop with one graphics card, with nothing leaving the machine. Models this size used to need a server room. Maya’s trick is refusing to hold the whole model at once: the hot experts sit in VRAM, the next tier in RAM, the rest on NVMe, and the engine moves them as the conversation needs them. Only about 18B parameters work on any given token, so most of the weights can sleep. The scandal is how well it works.

Finding

Project Maya (github.com/mw00/project-maya, MIT) runs GLM-5.3-Flash, a 321B mixture-of-experts model with about 18B parameters active per token and up to 1M context, on one or two consumer GPUs. The engine keeps the hot experts in VRAM, the next tier in RAM, and the rest on NVMe, moving them as the conversation needs them. Its Maya-S quant at 96.5 GB keeps 97.9 percent of the full FP8 model’s accuracy on zero-shot tasks, with the MTP speculative block intact.

What v1.0.11 had was one GPU, text only, on RX 7900 XT / XTX and R9700 / RX 9070, with Strix Halo compiling but untested. What v1.1.53 has:

• Strix Halo is a supported target. gfx1151 builds, runs, and has its own hipBLASLt tuning table plus automatic pool sizing from available memory. On a 128 GB box every expert fits in the pool, at about 99.9 percent hit rates. MTP drafting is on by default on gfx115x.
• Vision on AMD. GLM reads pictures on AMD GPUs after the encoder’s matmuls and attention were moved to float32 accumulation, verified on a Strix Halo box. The installer still writes a text-only config; vision is a manual config addition.
• Two GPUs on AMD. A layer split across two supported cards, which may be different architectures, with one hipBLASLt table per card and no peer-to-peer requirement. An opt-in expert helper lets a third device, including the APU, absorb misses inside its own pinned tier.
• Windows AMD support. ROCm wheels, first-setup detection of AMD-only PCs, and GPU monitoring from Windows counters.
• A project that outgrew GLM. DeepSeek, MiniMax, and Qwen-3.8-Flash engines, a smaller Maya-XS quant line, a community Metal engine for Apple silicon, experimental Instinct MI300X/MI350 support, a Turbo button, and a Models page and API for downloading, swapping, and deleting models without the command line.

My measurements remain the reference numbers in the AMD docs. Maya-S, 8K context, ROCm 7.2, 4K-token prompts, greedy:

The dGPU box has 192 GB of RAM, so most experts live in VRAM or pinned RAM. With less RAM, more comes from the SSD and decode drops. The port work was AI-assisted, openly.

Meaning

The month of releases confirms the original read. Prefill is dense matrix math and responds to tuning: engine-sized chunks, per-card hipBLASLt tables, and larger sub-batches took the 7900 XT from about 50 to about 415 tok/s, and Strix prompt work continued release after release. Decode is a memory logistics problem. Each token touches a different handful of experts, so speed follows tier residency, and the SSD-path optimization in v1.1.39 (about 22 percent faster decode when experts come from disk) is the same bottleneck being attacked from the other end.

The two-GPU row is the cleanest statement of the model. Splitting layers and drafting with MTP roughly doubles decode to 33 tok/s while prefill barely moves, because prefill was never waiting on capacity. And the unified-memory machines are the ones to watch: when every expert fits in one pool, the tier problem disappears and the question becomes pure compute, which is why the APU rows keep getting their own tuning passes.

Connection

This is the same lesson as the SageAttention work from September, seen from the other side. There, a 13 percent faster attention kernel projected to 2.5 percent off full generation because attention was a fifth of wall time. Here, the operation is rarely the problem and where the weights sleep is the problem. Both results point the same way: at this scale, placement beats arithmetic.

The second thing worth noting is the development velocity itself. Eighty-nine releases in a month, most of them integrating community measurements and pull requests with per-machine numbers attached, is a working model for performance software: the users own the hardware matrix no single author can cover, so every release converts someone’s evening of benchmarking into everyone’s faster defaults. My port was one instance of that loop. The Strix tuning tables from other contributors are a dozen more.

And the thread running through all of it, from that first night to now, is the surprise I have not gotten over: the giant model fits. Not comfortably, not fast, but it fits, and it answers, on hardware I already owned. Every piece of my AMD work since has been some version of asking how much of that gap is real and how much is software that has not caught up yet.

Key links: the AMD setup, measurements, and Strix notes in docs/AMD_MAYA.md, and the releases page for the full month of changelogs.