The Linux Thread - The Autist's OS of Choice

  • 🐕 I am attempting to get the site runnning as fast as possible. If you are experiencing slow page load times, please report it.
TheDataHorder said:
honesty, they kinda are the same but they are designed to accelerate different workloads.
gpus are more suited for graphical workloads while npu's are more suited for computational workloads.
Click to expand...
I know about say TPUs, that are possibly faster than some graphics cards for ML uses, but are NPUs faster than low/mid range or even high end cards for any use cases?

The office worker that isn’t going for 60 FPS DLSS gayming on their laptop, but would appreciate some copilot thing finding where some stupid control panel setting has been moved this week for the thousandth time.

I can see the business case of balking at some discrete GPU but being able to check a box that they’re doing a AI by buying a bunch of NPU-equipped laptops.
 
  • Lunacy
Reactions: Betonhaus
Ayana Otonashi said:
I've basically narrowed down my choices to Debian, Fedora or Arch lol

I feel like Fedora is nice and all but dnf feels slow as fuck, if anyone can help with that I might consider using Fedora full time lol

I was also thinking of using Endeavour but I feel like it'd be more satisfying to set up Arch.
Click to expand...
Debian if you want something that moves slow and is stable. Fedora if you want something that's very up to date while still having a complete system built-out for you. Arch if you want to do something custom like running a tiling window manager with minimal extra packages.
 
  • Informative
Reactions: Ayana Otonashi
Ok. I really like garuda. I just wiped everything else from my thinkpad. Gonna just keep the i3 version of that. And fedora on my other laptop, and I think I'm good.
 
Edict of Expulsion said:
but are NPUs faster than low/mid range or even high end cards for any use cases?
Click to expand...
probably depends on the type of workload. while ai is what they are mainly used for, I've heard of others getting video encoding working on npus but getting very slow processing of the video as a result.
i honesty can't think of any other workloads they might do better then a gpu. maybe protein folding could be one but as far as i know, no one's tried that yet.
 
  • Agree
Reactions: Edict of Expulsion
Susanna said:
docker
Click to expand...
I'd just use a bwrap chroot, but then again I'm old.

From what I saw so far, as soon as you walk of the beaten path the AMD AI stuff gets annoying, but quite possible that it works better now, I do not know.
--
That said interesting AI stuff that isn't image generation is still mostly a fools errand on current home hardware IMO. It's a lot cheaper and easier to just rent one or a few A6000s or a bunch of 80 gig A100s for a bit or just pay any of the countless API providers. The tiny LLMs are really difficult to get to do anything properly and I would not bother with them if you are not willing to finetune them for a specific task, and the big LLMs are out of reach if you don't want to buy a few high end graphics cards and a computer with a truckload of RAM.

Sure, A lot of stuff runs relatively well on high end macs because of the memory architecture, but that also is very slow. There's a lot of rather cheap providers now. I don't even wanna know how many decades you'd need to use a machine with 80 gigs of VRAM to break even with what cloud providers cost. (and for the best performing open weight LLMs, 80 gigs isn't even enough to run them properly) If you say privacy is a concern and you cannot and will not use "the cloud" then yes, I'm with you and won't even argue, but otherwise renting/API access at least for LLMs is the pragmatic approach. The tech just isn't there yet. I tried building a GPU server and I basically gave up. It's just so immensely uneconomical both in material and energy cost and yet still underwhelming vs. to what you can rent for a fiver a week that it sucked all the fun out of it. At least I can resell the hardware. Currently, investors are also shoveling money into anything with "AI" in it, so I wouldn't be surprised if a lot of these operate at a loss. Make use of it while it's still there.

Funny about the intel NPU - intel just released a very interesting paper about strong, lossless compression of AI models. at ~40 TOPS that NPU can run e.g. a (small) 7b LLM model at good speeds. It's probably related to micrsoft announcing that they want to put AI into windows, and microsoft recently also released a bunch of small models that are quite competent for their size and I could see being useful for very specific tasks if properly tuned. I sense another minimum hardware requirement for Windows incoming. Also probably an AI that will helpfully read all your documents, chat logs and internet activities and send summarizations to microsoft. Yikes.
 
NPUs aren’t big enough to run more complex AI stuff like language or image models (7b LLM is genuinely useless, I find the output barely legible), but they will fit things like upscalers, document classification, and image recognition. And they’re far more efficient at it than GPUs. I expect it’ll mostly be useful on laptops, where power constraints actually matter, than on desktops where spending a few extra watts to do these things on a GPU isn’t a big deal.

Upscaling, document classification/summarisation and image recognition are actually some of the more useful things we’re doing with AI. You’ll be able to search for “essay on wind mills” instead of anything more specific, or “all my photos of dogs”, or watch 480p streams scaled up to 4k to save loads of bandwidth, which is a big deal to yanks with their horrendous infrastructure and horrifying practice of data caps.
 
  • Like
Reactions: ⠠⠠⠅⠑⠋⠋⠁⠇⠎ ⠠⠠⠊⠎ ⠠⠠⠁ ⠠⠠⠋⠁⠛
AmpleApricots said:
Funny about the intel NPU - intel just released a very interesting paper about strong, lossless compression of AI models. at ~40 TOPS that NPU can run e.g. a (small) 7b LLM model at good speeds.
Click to expand...
Link to paper? And what constitutes good speeds, if they say token/second?

Baguette Child said:
I kind of want to try this, it looks so clean. How are you liking Trinity Desktop? Would you still recommend?
Click to expand...
Used it like five years ago since it happened to be in power PC macports and let it compile. Was crashy for that very specific use case.

I’d say unless you’re running on like hardware that really really is resource constrained, the requirements for modern KDE really aren’t that much.

Trinity is fine to try but daily driving is kind of lost its use case unless there’s something you need or really want that only 3.5 has.

Susanna said:
NPUs aren’t big enough to run more complex AI stuff like language or image models (7b LLM is genuinely useless, I find the output barely legible), but they will fit things like upscalers, document classification, and image recognition. And they’re far more efficient at it than GPUs. I expect it’ll mostly be useful on laptops, where power constraints actually matter, than on desktops where spending a few extra watts to do these things on a GPU isn’t a big deal.

Upscaling, document classification/summarisation and image recognition are actually some of the more useful things we’re doing with AI. You’ll be able to search for “essay on wind mills” instead of anything more specific, or “all my photos of dogs”, or watch 480p streams scaled up to 4k to save loads of bandwidth, which is a big deal to yanks with their horrendous infrastructure and horrifying practice of data caps.
Click to expand...
Yea my limited recent lazy NPU research confirms it’s really focused on energy efficiency above all else. For inference, image recognition are the big ones.

But, “far more efficient at it”, granted yes power efficiency, but I call bullshit on this marketing.

Modern GPUs have tensor cores, basically NPUs already. No one seems to be saying this and it’s very odd but NPUs are just tensor cores without the rest of the GPU, seems to me.

If you have to deploy a thousand or ten thousand and batteries might be involved or it’s in some cucked area where energy costs are really high (rip UK) then sure.

Otherwise for power users, discrete GPU seems to do more than NPUs. For now
 
Last edited:
  • Informative
Reactions: Baguette Child
Edict of Expulsion said:
Link to paper? And what constitutes good speeds, if they say token/second?
Click to expand...
They didn't say anything but it should manage a 7b model at 4 bit my guess somewhere in the area of 30 tokens/s. People got 7b and 13b LLMs running on Rockchip RK3588 NPUs and these have around 6 TOPS, slow yet still usable. The real bottleneck is memory capacity and bandwidth to begin with. Microsoft already talked months ago about copilot running on a 40 TOPS NPU. My guess is it will be optimized for simple queries and "using tools" (generating JSON with grammar rules) querying an external, much more sizable AI in the cloud when more difficult reasoning is required. People underestimate the small models, they suck at general reasoning and off-the-wall prompts but if they are tuned for some very specific functions (and tuned well) they can easily beat much bigger models at that specific function. Microsoft has some very capable AI people that released a ton of good shit in the last few months and I actually trust to pull that off. Stable Diffusion if it would run should fly on this thing and while a GPU would potentially do better, you won't be able to beat the GPU in power consumption. What would that thing eat? 10 Watts maybe while generating? Try that with a GPU, some of them consume more when idling. If I'm not off with my LLM guess and if it was all supported, your average SD generation would probably not take more than five seconds or so. I could see vidya taking advantage of such.

All links to paper etc. are here. I post AI news in that thread ocassionally. It can be hard to keep up.

speaking of microsoft: can't wait for the first malware that targets AI with malicious prompting.
 
Last edited:
  • Like
  • Informative
Reactions: Edict of Expulsion and ⠠⠠⠅⠑⠋⠋⠁⠇⠎ ⠠⠠⠊⠎ ⠠⠠⠁ ⠠⠠⠋⠁⠛
TheDataHorder said:
probably depends on the type of workload. while ai is what they are mainly used for, I've heard of others getting video encoding working on npus but getting very slow processing of the video as a result.
i honesty can't think of any other workloads they might do better then a gpu. maybe protein folding could be one but as far as i know, no one's tried that yet.
Click to expand...
One thing I thought would be interesting with that might be able to use those chips is if Video Games had a system where the NPCs are constantly interacting with each other and reacting to changes that you have made in the game like for example if there was a game that's in a Time loop and every single action every single move you do ends up changing things slightly which causes the NPCs to interact slightly different every time causing a chain reaction all the way through cuz I'll be like a lot of processing power needed and it's possible the npu might be capable of handling that I just thought that because of something interesting I suppose also for video games if they had a kind of a language model built into a kind of so that you had more natural communication with the AIS and if the AI seem to be able to respond more intelligently to your actions I think if they do that there's a chance that the NPCs might find a broken way to act in a certain ways that might make the game impossible to win.🎤🗨️➡️🔤
 
  • Like
Reactions: Vecr and TheDataHorder
Betonhaus said:
One thing I thought would be interesting with that might be able to use those chips is if Video Games had a system where the NPCs are constantly interacting with each other and reacting to changes that you have made in the game like for example if there was a game that's in a Time loop and every single action every single move you do ends up changing things slightly which causes the NPCs to interact slightly different every time causing a chain reaction all the way through cuz I'll be like a lot of processing power needed and it's possible the npu might be capable of handling that I just thought that because of something interesting I suppose also for video games if they had a kind of a language model built into a kind of so that you had more natural communication with the AIS and if the AI seem to be able to respond more intelligently to your actions I think if they do that there's a chance that the NPCs might find a broken way to act in a certain ways that might make the game impossible to win.🎤🗨️➡️🔤
Click to expand...
LLMs are wild. They are actually useful, so useful there are finally useful chatbots that it's an industry fad, but they're also uncontrollable and impossible to secure.
You could setup some kind of LLM system that would connect to something more traditional, to eliminate the chance of that broken state.

But video games even with just the way they're developed now have the T pose motorcycle riding through tunnels of cyberpunk, and the weird clipping. There is absolutely no chance in hell that an LLM that doesn't come from an extremely capable company that specifically focused on locking it down to only interact with the outside through traditional means would ever be well-executed by a game company.

One of the big outcomes of the fad of inserting LLMs everywhere in business is when they inevitably screw up.
Self-driving cars kill a person or two and either uber sells off their dept that does that, or companies cease operating in a state, so with an LLM it's going to be fascinating what kind of data gets leaked through them, or awful outcomes from giving an LLM enough access to or ability to change a system or settings.
 
  • Like
Reactions: Rusty Metal Skull Gun and Vecr
Edict of Expulsion said:
LLMs are wild. They are actually useful, so useful there are finally useful chatbots that it's an industry fad, but they're also uncontrollable and impossible to secure.
You could setup some kind of LLM system that would connect to something more traditional, to eliminate the chance of that broken state
Click to expand...
One of the biggest problems is that these people adding AI to their websites are often using prebuily LLMs and adding it to their site with a few preloading prompts. If you're willing to take the time to tailor the exact content that gets used in the LLM you can do things better.

However, I wasn't referring to a language AI but a behavioral one. So have the AI trained to find various pathfinding solutions to objectives that it will choose from. Maybe make the core of the game an insane world or one that the player drives insane and it'll work.
 
  • Like
Reactions: Edict of Expulsion
Betonhaus said:
If you're willing to take the time to tailor the exact content that gets used in the LLM you can do things better.
Click to expand...
I think LLMs for video-games can be a great step forward if you do this. If you only gather high quality writing(lol) data from established video game writers(lol) and you only use something trained on the specific dataset you need (e.g no reddit language in a medieval game) then you should be able to have something pretty cool.

I always found it very gay that in games where you're supposed to be this dude changing shit, you do something, get the reward and the world stays the same, or there's just some visual change with no actual game effects.

I like how in Gothic 1 NPCs had a schedule, and in a couple quests there was some rudimentary "if you disturb this guy's schedule something different might happen or a new path might open to you". I'd love that type of branching and choice for games without the dev having to hardcode every single possibility. This wouldn't fit in all games, but all of the "open world" games would benefit a lot from this, as their open world feels empty and bleak after you finish the dots on the map.

I'm sure this can lead to some retarded moments, maybe the quest giver decides to drink bleach all of a sudden, but those moments are some of the most fun in games.
 
  • Like
Reactions: Kiwi2Shoes and Rusty Metal Skull Gun
The problem with that is that LLMs are really difficult to make to keep track of things over a long term. Several memory systems exist of varying complexity and with varying theories behind them but they have all in common that they are kind of hit-and-miss. Then you can never truly predict what an LLM will focus on or decide in any given context and that can lead to bizarre problems in something as reglemented as a game. While LLMs have some unique reasoning capabilities that are super useful in natural language problems, I just don't see an LLM solution performing better at problem solving in a closed system like a videogame in most cases vs. more conventional approaches like state machines and such. More unpredictable and potentially more lifelike though? Absolutely.

What an LLM could do really well is creative additions. I am convinced you could get an LLM to write custom generated quests, NPCs, dialogs and event chains (e.g. for games like crusader kings and stellaris) and they'd actually be somewhat interesting. (although you'd have to "fact-check" them in code somehow) They could add a ton of flavor to open world games this way. Somehow people don't want to see that capability in LLMs. I blame science fiction. It's been telling us for a good 80 years that when AI comes to be, it'll be this super-logical and intelligent being who breathes math with perfect recall that flat out won't understand the most simple social cues and won't even be capable of emulating them in theory, while LLMs are more scatter-brained creative, charismatic empaths with ADD who can't do basic math in their head but are already better at theory-of-mind than some humans. I genuinely think people struggle with this disconnect.

That might change one day but I don't think LLMs in their current form will be very useful for general computer tasks for that reason. (vs. conventional approaches) If Microsoft would overhaul their UI nightmare they would have done more for the accessibility of their OS than slapping another layer on top of it with copilot. I think the real purpose is data-gathering for machine learning anyways, which to be fair, might actually help to get to that classical scifi AI. LLMs, in their current form, will never be it, though.
 
Last edited:
Goddamn why is it always gstreamer6.9-plugins-retardedbullshit that breaks upgrades? Who is letting this garbage into the repositories?
 
AmpleApricots said:
They could add a ton of flavor to open world games this way. Somehow people don't want to see that capability in LLMs. I blame science fiction. It's been telling us for a good 80 years that when AI comes to be, it'll be this super-logical and intelligent being who breathes math with perfect recall that flat out won't understand the most simple social cues and won't even be capable of emulating them in theory, while LLMs are more scatter-brained creative, charismatic empaths with ADD who can't do basic math in their head but are already better at theory-of-mind than some humans. I genuinely think people struggle with this disconnect.
Click to expand...
Star Trek covered this one a few times with the holodeck. Everyone thinks of Data, the emotionless mathematical genius who can calculated at tremendous speed when they think of AI in Trek, but they never remember the time when the holodeck was given a command to "make an opponent capable of defeating Data" and just decided to spit out a fully self-aware holographic lifeform. Or every time the holodeck generated lifelike responses to arbitrary, unpredictable events and commands, or the way it can just gin up almost everything you're asking for with a couple of sentences of instruction. Star Trek's holodeck really does seem to be a prediction of AI prompting, including all the messy "do it again but better this time" business.
 
  • Like
  • Agree
  • Informative
Reactions: Foghotten, ellroy, Antithesmith and 11 others
  • Like
  • Informative
  • Winner
Reactions: Grin Machine, Kiwi2Shoes, Wooo and 5 others
TheDataHorder said:

kde has opened up submissions for what goals they should focus on for the next few years.
i have put in a task to continue X11 support https://phabricator.kde.org/T17393 but i'll require some of you boys to sign up and add your voice.
why do this?
to make the wayland troons seeth. why else?
Click to expand...
uhhhhh how do i create an account? and for some reason my old one doesn't work (plus it's a Ring 2 identity which means people here can figure out my real name based on other info I've revealed on the farms)
 
  • Like
  • Dislike
Reactions: Kiwi2Shoes, jeff7989 and TheDataHorder
You must log in or register to reply here.
Back