Thread should be renamed "LLMs General", and OpenAI should be slaughtered.
AMD announced their "Strix Halo" mega APUs at CES, which will appear on the market in the coming months. Gaming-oriented models are paired with a minimum of 32 GB of LPDDR5X, but "AI workstations" will come with up to 128 GB. AMD's software can configure 75% (up to 96 GB) as "VRAM", allowing it to easily run 70B size models.
One of the claims AMD made is that Strix Halo can outperform the RTX 4090 while using a fraction of the power:
AMD’s beastly ‘Strix Halo’ Ryzen AI Max+ debuts with radical new memory tech to feed RDNA 3.5 graphics and Zen 5 CPU cores (
archive)
AMD also shared some rather impressive results showing a Llama 70B Nemotron LLM AI model running on both the Ryzen AI Max+ 395 with 128GB of total system RAM (32GB for the CPU, 96GB allocated to the GPU) and a desktop Nvidia GeForce RTX 4090 with 24GB of VRAM (details of the setups in the slide below). AMD says the AI Max+ 395 delivers up to 2.2X the tokens/second performance of the desktop RTX 4090 card, but the company didn’t share time-to-first-token benchmarks.
Perhaps more importantly, AMD claims to do this at an 87% lower TDP than the 450W RTX 4090, with the AI Max+ running at a mere 55W. That implies that systems built on this platform will have exceptional power efficiency metrics in AI workloads.
Strix Halo's big iGPU is only around a 7600 XT. If this kind of result is typical, it's only because it has the equivalent of 96 GB of VRAM compared to a measly 24 GB for the 4090, so it can handle larger models better. To get more memory bandwidth, Strix Halo is configured with quad-channel (256-bit) LPDDR5X for around 256 GB/s. It also has 32 MiB of Infinity Cache, which may not help (it's good for gaming).
I haven't seen many AI benchmarks for the RTX 5090 yet, but if it's only around a +30-50% improvement, Strix Halo could outperform that too. But only when limited by VRAM. If a 30B model can fit in the 5090's 32 GB, it will smoke everything out there.
Nvidia has an answer for this: the
Project Digits "AI supercomputer". That is a mini PC with a 20-core ARM CPU and larger Blackwell GPU that includes... 128 GB of LPDDR5X. It's $3,000, but I expect 128 GB Strix Halo systems to cost around that much too. Tiny Attention Whore Corp
scorned the box.
If Strix Halo is successful, AMD could follow it up with a Zen 6 "Medusa Halo". Hopefully with even greater memory support, e.g. 192-256 GB. There's also nothing stopping AMD from adding 3D V-Cache (like a 9800X3D) because these parts use desktop CPU chiplets.
Also, take a look at this. It could be marketing bullshit, but it might actually be a weird trick to download more VRAM:
CNX Software:
Phison’s aiDAPTIV+ AI solution leverages SSDs to expand GPU memory for LLM training
On the other hand, the aiDAPTIV+middleware manages data flow between the GPU memory and the SSDs. It analyzes the AI model’s needs, identifies less active data segments, and transfers them to the SSDs, effectively creating a vast virtual memory pool for the GPU. This technique allows for the training of much larger models than would be possible with the GPUs onboard.
Worked fine for me with rapidsave.com
I use that site and don't get audio unless I pick the (default) highest resolution copy.
