And a 3070 and a 3060 Ti. Although I prefer the 4060 Ti 16GB as I can run SDXL AND Chrome at the same time. It's rather tight on the 8GB cards.
Just for reference, I use around 10.5 gb of VRAM while training SDXL LoRAs with batch size 2, and gradient checkpointing turned on. It's local LLMs that are the worst VRAM hogs. Video as well probably, although I have not experimented much with that.
Depends what your expectations are. These NPUs are more about efficiency than performance. Which is not a criticism. But I just timed his "What is the meaning of life?" question to Llama 2 7B. Took just under two minutes to complete. Llama 2 7B here just now same question started responding immediately and completed in under 10 seconds. So really what is your use case? Frequent running of medium to small models for without draining power? Such as Copilot's needs which is why the 40TOPs NPU requirement is good for W11. Or something more substantial and probably in a desktop or rack where you don't much care about the extra power consumption.
Yeah I heard 12GB is the bare minimum these days, anyway you mean google chrome or some other chrome?
That's the problem with these SBCs and why Rpi reigns supreme: basically everything is made for the later, it's the standard. One old timer told me it was just like the days of the apple ][ when everything was made for that one computer (not sure if it was exactly the same situation but I get it).
I take that was on runpod? or locally with some GPU? which one?
Nah power isn't really an issue but being able to run locally and offline on a thin laptop, but again seems a chunky gaming one with a decent GPU is the only real solution.
Not really cool but more like interesting specially as I said before as an alternative to retrofit x86 computers
Locally. Just a Radeon 7900XT, so 20GB VRAM. But you don't need nearly that much for it.
I mean, you're comparing a $700-1000 GPU to a $150-180 computer, at least use a GPU in that price range (which would still need the rest of the computer to work).
No, I don't think I will thanks, because what the fuck point would I be making? That NPU x which is inadequate for this sort of work is marginally better or worse than NPU y that is also inadequate? My entire point was that the use case for these NPUs isn't really what you're focusing on. Their goal is to run small to medium models that are heavily quantized efficiently without impacting the CPU or GPU or causing a significant drain on the battery. And for that, they show good promise. Maybe they have a little utility for other niche applications such as being able to perform inference (slowly) on a Rasberry Pi. The comparison I made is the one that is appropriate to my point, which is to illustrate what they're intended for and what they're not.
I'm sure @The Ugly One will be along to say what a nightmare developing for AMD is shortly and I wont dispute it because he knows his stuff. But as an end user what I'll say is in the space of a year I've seen ROCm go from barely adequate to very performant for my use cases. Using it for Stable Diffusion I get pretty good performance and it's been stable also. If you want to see the pace of the progression, check this out: https://github.com/ROCm/ROCm/releases
Just Google Chrome. But really trying to do anything which uses any significant VRAM while SDXL images are processing I found would cause out of memory. Without doing anything except a terminal window my system with 8GB VRAM doesn't seem to have any issues, Linux.
It doesn't hurt to check what's running with nvidia-smi. That's how I discovered I was losing a few GB of VRAM to a memory leak in xdg-desktop-portal-gnome. They still have not fixed it, but at least now I know to do a 'systemctl restart' once in a while if it has gotten bad.
Developing AI for AMD or anything for AMD?
Can't chrome just use the iGPU and system RAM instead? what's your setup?
What iGPU?
The techniques used to train Stable Diffusion LoRAs and Checkmarks are being improved constantly worldwide by an international community of Coomers and furries. A team of programmers from a mid-tier cs university, doesn't stand a chance against the funding behind that much less the talent. This would have become apparent overnight if they had decided to Open Source Glaze. Something they could do under AGPL if they wanted to poison the commercial well.
The real problem, which is explained by Nicholas Carlini in his blog post, is that thousands of tech-illiterate artists are "glazing" their images, expecting it to do a damn thing. Their glazed stuff will get copied, Glaze will get defeated, and they're left in the same position or worse than they would have been if they had taken everything they could down and stopped uploading. Which is why they felt it was important to attack Glaze into the ground ASAP. Carlini compares the situation to the NSA storing encrypted traffic/files so that they can decrypt it all later.
Real talk when I was in the initial stages of training, I decided what artists to train based on if they had a watermark. Those things really fuck up LoRAs.
They do for now, won't forever. Especially if the watermark is identical and in the same place every time in a set of images, that should be trivial to detect and discard.
I knew what I was doing!
'Find a job you enjoy doing, and you will never have to work a day in your life.' scnr
I wrote all this in the AI Art Seething thread already, but thing is, this is what this kind of AI just excels at. It's one of it's strengths. If we can filter away the noise and see the image for what it is, such a model will be able to, too. This is fundamental. It's like there's no protection against movie piracy that will ever work a 100%, because as long as you can see the movie on your computer, it can by definition also be pirated.
