The Linux Thread - The Autist's OS of Choice

[std::string]

Hallucinating is a big problem in general and lethal in combination with "devs" that trust the AI output blindly. (AI hallucinates software packages and devs download them – even if potentially poisoned with malware)

Holy shit this is the funniest attack vector. Even better than typosquatting.

I bet you could use ChatGPT to generate thousands of JS tutorials using your malicious package, toss them on the internet and wait for it to be picked up into ChatGPT's corpus. Actually if you took a package with a dumb name, copied its functionality and gave it a descriptive name stating its function I think ChatGPT would start preferring it due to how it makes inferences.

Have fun Pajeets

 

[Miller]

https://www.phoronix.com/news/Microsoft-Linux-More-Inclusive, https://archive.is/9iSPX

Thanks Microsoft, very cool

"Caroline" started it with his CoC and now Microsoft has taken over the task of pozzing Linux.

 

[Overly Serious]

Wasn't Javascript/Node/NPM one of the previous supply chain hacks?

Dev corrupts NPM libs 'colors' and 'faker' breaking thousands of apps​

https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/

One of which was... colorful console messages.

I guess it didn't do anything malicious, just stopped thousands of apps from working, which was probably a win.

Time to break out this oldie but goodie.

I bet you could use ChatGPT to generate thousands of JS tutorials using your malicious package, toss them on the internet and wait for it to be picked up into ChatGPT's corpus. Actually if you took a package with a dumb name, copied its functionality and gave it a descriptive name stating its function I think ChatGPT would start preferring it due to how it makes inferences.

Whelp. That's me not sleeping tonight. No, not because I do this (I rarely touch code professionally these days anyway) but because the world is now full of developers who do.

 

[HahaYes]

but because the world is now full of developers who do.

People will download the most bootlegged versions of gpt they can find just to shit out absolutely useless pieces of work that don't function. Then they wonder why they're told off and made to actually put in effort for their wagebux instead of relying on soybots to do it for them.

 

[Trans Fat 41g]

I had to make sure this wasn't an April Fools joke. Again.

It would be the funniest fucking thing if KDE usurped GNOME as the Workstation flagship for Fedora.

 

[Betonhaus]

I had to make sure this wasn't an April Fools joke. Again.

It would be the funniest fucking thing if KDE usurped GNOME as the Workstation flagship for Fedora.

Be awesome if it did. Fuck Gnome.

 

[std::string]

I had to make sure this wasn't an April Fools joke. Again.

It would be the funniest fucking thing if KDE usurped GNOME as the Workstation flagship for Fedora.

Fedora willfully blazes ahead with KDE shit even if it's not ready for primetime, so...I don't think it would be a good flagship distro.

 

[AnOminous]

It becomes a problem if you don't really understand it's output. I'm not sure the hallucinating can really be fixed in the current iteration of AI, it's really the nature of the thing, even the nature of the thing that the hallucinations are so believable.

This has certainly occurred in the realm of law when lawyers, or really lolyers, used AI to draft legal documents.

For instance: https://www.reuters.com/legal/new-y...ng-fake-chatgpt-cases-legal-brief-2023-06-22/ (archive)

The problem is the AI can output properly formatted citations that are even Bluebook, but they don't actually exist and the actual legal reasoning is total hallucination. That's because the AI is not in fact actually intelligent but is good at formatting things it has seen a lot of so can imitate something. It's only when you actually check the citations or run the code that you realize "hey these don't even exist" or "hey this code just deleted my home directory."

Incidentally this is really stupid because even pre-AI, services like WestLaw or LexisNexis have had services where if you have more money than sense, or just more money than God, you can have your briefs Shepardized automatically and it actually checks that they exist.

Code seems like an even more dangerous place to use an AI prone to hallucinations because it could have direct real world impacts, especially when code is controlling physical objects in actual reality. It can in law too but usually someone is going to crack a book and say "what the fuck?"

 

[Betonhaus]

This has certainly occurred in the realm of law when lawyers, or really lolyers, used AI to draft legal documents.

For instance: https://www.reuters.com/legal/new-y...ng-fake-chatgpt-cases-legal-brief-2023-06-22/ (archive)

The problem is the AI can output properly formatted citations that are even Bluebook, but they don't actually exist and the actual legal reasoning is total hallucination. That's because the AI is not in fact actually intelligent but is good at formatting things it has seen a lot of so can imitate something. It's only when you actually check the citations or run the code that you realize "hey these don't even exist" or "hey this code just deleted my home directory."

Incidentally this is really stupid because even pre-AI, services like WestLaw or LexisNexis have had services where if you have more money than sense, or just more money than God, you can have your briefs Shepardized automatically and it actually checks that they exist.

Code seems like an even more dangerous place to use an AI prone to hallucinations because it could have direct real world impacts, especially when code is controlling physical objects in actual reality. It can in law too but usually someone is going to crack a book and say "what the fuck?"

AI is very effective at fooling upper management, because upper management usually has the same surface level understanding of the work.

 

[HahaYes]

AI is very effective at fooling upper management, because upper management usually has the same surface level understanding of the work.

High level management also don't tend to have a use case beyond "oooo look new shiny thing lets implement it into the business in every way under the sun" without thinking twice to look good for stakeholders. Then they're genuinely surprised when it comes back to bite them in the ass, and ends up costing them far more than keeping a competent team on the payroll would have.

 

[Rusty Metal Skull Gun]

This has certainly occurred in the realm of law when lawyers, or really lolyers, used AI to draft legal documents.

For instance: https://www.reuters.com/legal/new-y...ng-fake-chatgpt-cases-legal-brief-2023-06-22/ (archive)

The problem is the AI can output properly formatted citations that are even Bluebook, but they don't actually exist and the actual legal reasoning is total hallucination. That's because the AI is not in fact actually intelligent but is good at formatting things it has seen a lot of so can imitate something. It's only when you actually check the citations or run the code that you realize "hey these don't even exist" or "hey this code just deleted my home directory."

Incidentally this is really stupid because even pre-AI, services like WestLaw or LexisNexis have had services where if you have more money than sense, or just more money than God, you can have your briefs Shepardized automatically and it actually checks that they exist.

Code seems like an even more dangerous place to use an AI prone to hallucinations because it could have direct real world impacts, especially when code is controlling physical objects in actual reality. It can in law too but usually someone is going to crack a book and say "what the fuck?"

I can't find the story now, but someone asked ChatGPT what the most cited scholarly article was. It came back with an economics paper written by a famous economist, with a plausible title on his subject of interest, in a plausible journal in a plausible year. Only, of course, the article never existed.

I do suspect AI will prove to be counter-productive in a whole load of areas, including coding. It's prone not just to nonsense, but to plausible-sounding nonsense, the worst kind. It's a bullshit generator.

 

[geckogoy]

It's a bullshit generator.

Its weird because when it knows it does seem to usually regurgitate the answer. If it doesn't know, it makes the best-scoring guess it can think of, so in other words high grade BS yes.

 

[Creative Username]

Its weird because when it knows it does seem to usually regurgitate the answer. If it doesn't know, it makes the best-scoring guess it can think of, so in other words high grade BS yes.

The thing is that it doesn't really know anything, being a very large mathematical function that predicts the next word, given previous words. Sometimes it's right, but it's only on accident because the training algorithm doesn't care at all about facts or anything related to the real world, it only cares about the next word and how it relates to the previous words. It's just a very powerful and advanced version of the predictive words that appear on the top of a phone keyboard, and it will be very hard to integrate any form of highly reliable rigidity or formal correctness into the way it "thinks". Not that it can't be done, I don't know that, I just know that it's very hard.

 

[geckogoy]

it doesn't really know anything, being a very large mathematical function that predicts the next word, given previous words

I'm pretty sure it doesn't work this way. The models themselves always say that, but I think thats just based on what its training data says about past models, or possibly that is what its told to say to protect trade secrets. At any rate its certainly not self aware and does not understand its own workings, just what it was told.

I think its got some kind of huge self referential table of how words relate to eachother and actually can comprehend and re-articular one sentence into another sentence or answer questions purely based on that table. IE it knows that apples are red, and apples are a fruit, so if you ask for red fruit it can infer that apples are red fruit and say 'apple' without having any idea what any of those things mean.

 

[WhiteNight]

"Caroline" started it with his CoC and now Microsoft has taken over the task of pozzing Linux.

It's worse than Caroline. It's "Coraline" https://where.coraline.codes/.

 

[AmpleApricots]

It's good at recognizing and repeating patterns in certain ways to make them fit to similar patterns it "knows". That's why the hallucinations, at least with the better models, are always really, really convincing and not complete nonsense, they are very close to what you would actually expect because they fall into the same patterns to what it actually was trained on, even if they are "hallucinated" because the information in it's entire accuracy meant to complete it's "context" does not exist or isn't reproduced in the expected way. In the way this happens, it's actually really working as designed. There is no such thing as truth or even "linear knowledge" for an LLM. If you spend some time with LLMs and try your hand at training and tuning them you kinda develop a feeling for a "tell" where you can tell where it gets fuzzy and less accurate and it just starts piecing things together, but this can vary a lot by model and even iterations of the same model and is certainly not something the average end user can figure out. The process has no concept of question, answer, the user or it's own "persona". It's just taking guesses at the most likely continuation of a chain of tokens. This sounds like phone autocomplete so far, yet (!) is not that simple. Even the creators of the models don't really exactly know how the training primed them to really get to their "replies" and the whole process of training a model is basically black magic. (which leads to funny side effects like gpt-instruct being a very apt chess player without that ever being focus in any way, stable diffusion apparently having some sort of 3D "imagination", and multimodal models being better at describing places in text, or the strange thing where larger models - and this disproportionately affects larger models - sometimes literally go braindead during training and you will have to restart it from an earlier point) Yes, these models are artifically created, yet in their function a complete black box. OpenAI does not know what really makes GPT4 hallucinate a law paper that is in the training, while accurately solving a logic puzzle in another place. Nobody does. There are ways via sampling to manipulate the token likelihoods after one mathematical model or other, and this is even some more voodoo on top which effects are subjective on the best of days but that's pretty much all you can do. This makes it sound like a cheap parlor trick (which it gets called by some people) and makes you think that nobody should ever rely on anything something like that generates, but you also have to consider that the group of neurons which process the input from the receptors from the light that forms this text you're reading in your skull don't inheriently "know" what language or writing is either (or even what will happen to these impulses past them) but they are still an important part of the process of you understanding what I am writing and damage to or chemically altered states in your brain for example could easily cause your impression of this text to vastly shift and there are examples of such effects. Intelligence isn't also something like room temperature superconductors, cold fusion or time travel. (Our understanding of) Intelligence has been empirically proven and observed.

This was a lot of words to basically just say that I would not disregard this technology in general for anything, as it is simply too early to. We are in the C64 stage of things. Cool computer in the 80s, gave you a glimpse of what is and will be possible if you cared to look, but ultimately useless for many things and doing them the old fashioned way was quicker and more reliable and it also just was one of many different approaches to the concept of a personal computer of which the vast majority failed. There's still an immense ground to cover in this field and a lot of things that were suggested (in human written, good papers) haven't even been tried yet, in many cases because of the lack of both competent manpower and resources. Many people back then also said that owning a computer is a fad and will never be a real thing for most people outside of niche interests. I know because I had this conversation many, many times.

 

[Creative Username]

I'm pretty sure it doesn't work this way. The models themselves always say that, but I think thats just based on its training data of past models, or possibly that is what its told to say to protect trade secrets. At any rate its certainly not self aware and does not know how it, itself works outside of what it was told.

I think its got some kind of huge self referential table of how words relate to eachother and actually can comprehend and re-articular one sentence into another sentence or answer questions purely based on that table. IE it knows that apples are red, and apples are a fruit, so if you ask for red fruit it can infer that apples are red fruit and say 'apple' without having any idea what any of those things mean.

GPT-2 worked like that, and current models use the same basic method of operation as it did, with a lot of architectural tricks performed to make it scale better and have better perplexity scores. It just predicts the next word given a sequence of words. "Perplexity" is even a measure of how likely the next word is given a sequence of previous words, and LLMs optimize mainly for that, AFAIK.

Your apple example is possibly because of the fact that it has been basically trained on the entire Internet, which is a massive corpus, and it has inevitably scooped up some chunks of text saying that apples are red fruits, and it also probably scooped up a couple of questions asking for examples of <adjective> <noun>s, where it spits out another noun, and maybe somewhere in the training process the LLM developers used a knowledge base to fabricate millions of questions like this, and (somehow, maybe there has been interesting research about this but I don't know about it) it can relate the literal petabytes of text into the answer to that question.

@AmpleApricots just made a massive text wall, but he knows more about this than me. Give his post more weight than mine.

 

[Susanna]

literal petabytes of text

The sum of all written text in human history probably would fit in under 500 terabytes. The reason we talk about the Internet in terms of petabytes and exabytes is because it contains a lot of bulky information, such as pictures, music, and video. Pure text, which is what LLMs operate on, is actually quite dense.

 

[AmpleApricots]

listen to AmpleApricots he knows better

It's Magic, lol

I'm very sorry it really did sound that way but a lot really is poorly understood. To give an example for people that don't closely follow: In the last few months people in the open source sphere have figured out that a) you can basically "cut away" huge parts of a models "brain" with little effect, vastly lowering resource requirements b) you can stich several "brains" together in various ways and the resulting model sometimes is smarter than the models it's made of as a result. That's where we are at here. That's also why it really is too early to tell where it will go.

 

[HahaYes]

Many people back then also said that owning a computer is a fad and will never be a real thing for most people outside of niche interests. I know because I had this conversation many, many times.

At this stage it reminds me a lot of the initial introduction of cloud and all the skepticism that it would hold any weight. Look how that turned out. Not saying AI is about to mirror it but with things like co-pilot being shilled so hard across enterprise it's definitely going to stick around for a while as it develops into something more than "haha look at funny bot that can sometimes help with stuff".