And how come so much system resources are used for image generation anyway?
First of all, VRAM. A diffusion model has a limited amount of data it can have in it's dataset, and for it to be effective it needs a lot of it. Stable Diffusion 1.5 models were ~2GB, SDXL models are ~6.5GB, and Flux.1 dev is ~16GB. Each larger than another, having more data in the dataset and capable of more. The entire model has to be loaded into VRAM as running it off of system RAM or the drive would be tremendously slow.
For example, the full Deepseek R1 model that has 671 billion parameters, which is the real deal that you use on their website, is 404GB. You'd need a proper Nvidia server rack with GPU's connected via NVLink for a grand total of those 404GB of free VRAM to run it, and those would cost you as much as a brand new house. In comparison, the smaller model that has 32 billion parameters is only 20GB. It is way less detailed than the full model, but still fairly capable and you could run it on a single 3090 with 24GB of VRAM.
Second of all, computation power. Everything, from latent diffusion to LLM's, are simply heavy arithmetic operations on mathematical matrices. That's why the 20 series was much better at those workload than the 10 series, as it introduced dedicated cores for matrix computations. The Tensor cores. GPU's are perfect for this type of workload, as they have tens of thousands of cores that can do such calculations in parallel.
It's also the reason why Nvidia became such a market giant and asserted dominance in the space with CUDA. Jensen predicted this is where tech will go so very early on Nvidia created a framework that lets you utilize the same cores used for your video games to do such computations.
And of course, it's the reason for which VRAM is important, as the computations are done on the GPU, and the VRAM is the fastest memory connected to it, right on the board. Anything else would be bottlenecked by the CPU. Reading from an NVMe drive? Now you're adding the PCIe lanes to the equation. System RAM? That's the CPU. VRAM? No bottleneck, it's a direct connection, one that you need for good performance with a workload this heavy.
This video was probably already linked here but it explains how it all works. The concepts of how LLM's work also apply to latent diffusion models, since they're all
transformer models stemming from the 2017 research paper called "
Attention Is All You Need". All the current "AI" models began with this paper.
, on the next launch or page reload it can restore your last used settings,prompts )
