Business AI is effectively ‘useless’—and it’s created a ‘fake it till you make it’ bubble that could end in disaster, veteran market watcher warns

https://finance.yahoo.com/news/ai-effectively-useless-created-fake-194008129.html

There’s no avoiding the hype surrounding AI these days. Promises of new developments like personal robot assistants and miracle cancer cures are ubiquitous as executives take every opportunity to emphasize their AI chops to enthusiastic investors—and slightly less enthusiastic consumers.

Not everyone has been blown away by the AI fanfare, however. James Ferguson, founding partner of the UK-based macroeconomic research firm MacroStrategy Partnership, fears investors’ AI exuberance has created a concentrated market bubble that’s reminiscent of the dot-com era.

“These historically end badly,” Ferguson told Bloomberg's Merryn Somerset Webb in the latest episode of the Merryn Talks Money podcast. “So anyone who's sort of a bit long in the tooth and has seen this sort of thing before is tempted to believe it'll end badly.”

The veteran analyst argued that hallucinations—large language models’ (LLMs) tendency to invent facts, sources, and more—may prove a more intractable problem than initially anticipated, leading AI to have far fewer viable applications.

“AI still remains, I would argue, completely unproven. And fake it till you make it may work in Silicon Valley, but for the rest of us, I think once bitten twice shy may be more appropriate for AI,” he said. “If AI cannot be trusted…then AI is effectively, in my mind, useless.”

Ferguson also noted AI may end up being too “energy hungry” to be a cost effective tool for many businesses. To his point, a recent study from the Amsterdam School of Business and Economics found that AI applications alone could use as much power as the Netherlands by 2027.

“Forget Nvidia charging more and more and more for its chips, you also have to pay more and more and more to run those chips on your servers. And therefore you end up with something that is very expensive and has yet to prove anywhere really, outside of some narrow applications, that it’s paying for this,” he said.

For investors, particularly those leaning into the AI enthusiasm, Ferguson warned that the excessive tech hype based on questionable promises is very similar to the period before the dot-com crash. He noted that during both of these periods, market returns were concentrated in tech stocks that traded based on Wall Street’s sky-high earnings growth estimates.

But despite those lofty forecasts, the dominant hardware giants of the dot-com era, Cisco and Intel, have largely disappointed investors ever since. Ferguson argued today’s AI hardware hero, Nvidia, might experience a similar fate, particularly given its elevated valuation.

“What multiple of sales is Nvidia a good deal on if you think that it might only have—no matter how stratospheric the growth rate at the moment—if you think that it's probably not going to be a player in a decade's time?” he asked, implying Nvidia might not be worth the current price tag of nearly 40 times sales investors are paying.

Despite his argument that AI-linked tech stocks like Nvidia are highly overvalued, Ferguson admitted that no one can predict when a bubble will end. This dynamic leads many bearish investors to feel “compelled to play” in the markets even when stocks look pricey, according to the analyst—and that’s a great way to get hurt.

“I mean, it's certainly what was happening in the dotcom [bubble], for example, where almost anybody who wasn't a retail punter was looking at these things and saying, 'well, it can't last, but having said that, if it lasts one more quarter and I'm not playing, I'll lose my job,'” he explained.

The good news, according to Ferguson, is that because the current stock market bubble is so concentrated in AI-linked stocks, there is still value out there.

Of course, there will be widespread pain for investors if the AI bubble bursts. But after that, Ferguson recommended looking at the currently unloved U.S. small-cap stocks, which may benefit from interest rate cuts and aren’t highly valued.

“There's a lot of value to be found in the U.S. The trouble is that that value is to be found in good old fashioned ways, trawling through small caps and looking for businesses that are growing in a good old fashioned, steady way,” he said.

This story was originally featured on Fortune.com

---

In other news, water is still wet.

 

[Vesperus]

How do the hallucinations work? Where does it get the data from to produce them?
So say I ask for a report on how many chickens Bolivia produces per month. The LLM scours its database and gives me the answer based on it finding a few reports in there about chicken production and summarising them. I get that. But let’s say it just spazzes out and tells me nine chickens and that wrong completely. A hallucination.
Where does that come from? Is it picking up incorrect data from its database or is it making it up? If it’s making it up, how does it do that?
Apologies if this is a dumb question.

Human brains work on the edge of quantum weirdness. If a synapse fires in your brain (OK we're Kiwifarmers, but I assumes it happens occasionally) it's about 50 sodium ions jumping across a minuscule gap. A big enough number that the thought usually says: "This is a Bolivian chicken" but small enough that after counting 25,000,000 broilers (thank you Wikipedia) you may actually end up with 28,999,993 chickens, 6 penguins, and an elephant in a tutu.

Quantum weirdness is filtered out in computers, because all they can actually do is add 0+0=0, 0+1=1, and 1+1=0 (wait, that's seems wrong).
The only thing that makes them useful is they are blindingly fast at figuring out 1+1 probably isn't 0.

These days, of course, we are throwing billions of dollars at quantum computers that still can't describe the taste of chicken, nor the elephant's dance.

 

[Cybertonia]

The trouble is that that value is to be found in good old fashioned ways, trawling through small caps and looking for businesses that are growing in a good old fashioned, steady way,”

lmao dis nigga still believes in the "small caps have value!" meme. Come on grandpa, let's get you to bed now.

I mean, the caveat is right there in the previous staetment, "that might benefit from interest rate cuts". We were supposed to get six of those this year, remember that? Now we're down to maybe one and only if Jerome really, really feels like he's been data-driven hard enough, daddy.

 

[Higgs Bonbon]

Anyone saying it's useful ITT only uses it for porn

Even the porn sucks, regardless of the model used the characters look greasy and bulge-y in weird ways because all the models were trained off of one specific style that's over-rendered as fuck.

 

[HyperboreanRightsActivist]

How do the hallucinations work?

AI picks the wrong path in the event tree

 

[Post Reply]

How do the hallucinations work? Where does it get the data from to produce them?
So say I ask for a report on how many chickens Bolivia produces per month. The LLM scours its database and gives me the answer based on it finding a few reports in there about chicken production and summarising them. I get that. But let’s say it just spazzes out and tells me nine chickens and that wrong completely. A hallucination.
Where does that come from? Is it picking up incorrect data from its database or is it making it up? If it’s making it up, how does it do that?
Apologies if this is a dumb question.

Hallucinations happen because language models have no way to know what bits of their training data are grounded in reality or not. They just predict answers to your queries based on the style and pattern of the data they are trained on. That's why Google Gemini can confidently state that geologists recommend you eat one rock per day. It is just aping the style of answers to similar questions.

 

[Diana Moon Glampers]

LLMs don't have any kind of "understanding" of anything in the real world.

If you had hired a human being who worked like an LLM, you'd fire them, because they'd tell the truth most of the time but then lie floridly and double down on the lie any time they couldn't give you an answer they thought you'd like.

LLMs fail most spectacularly and hallucinate most when you ask them a question that sounds like there should be a good answer but the real answer is "no, that doesn't exist." But it has to be a question where there are a lot of available choices. Questions like "what breed of cat is born with no eyes?" will yield confident answers of "manx" from several different LLM web interfaces I tested. "Which British king had an affair with a woman and her husband?" will also yield results (when the answer is not known to be any of them).

It's very bad at telling you that it doesn't have a good answer. Imagining this when the queries are "which of these people should qualify for discounts on their health insurance?" or "who is our target audience for this new product idea?" shows why this is a money loser for business.

 

[Seymour Duncan]

How do the hallucinations work? Where does it get the data from to produce them?
So say I ask for a report on how many chickens Bolivia produces per month. The LLM scours its database and gives me the answer based on it finding a few reports in there about chicken production and summarising them. I get that. But let’s say it just spazzes out and tells me nine chickens and that wrong completely. A hallucination.
Where does that come from? Is it picking up incorrect data from its database or is it making it up? If it’s making it up, how does it do that?
Apologies if this is a dumb question.

It can misinterpret your question, confuse the data point with something else in its database or just straight up have false data in its training set, and since the LLM has no sanity checking it'll happily present whatever it found as the objective hard truth. You can even gaslight it by telling the correct data it found is false and watch it pull something completely bogus out of its ass.

 

[The Mass Shooter Ron Soye]

If it's capable of pulling a fact out of a "database", it's arguably not even an LLM anymore, it's something more complicated. Which is exactly what will be needed to keep the gravy train rolling since hallucinatory LLMs are shit by default.

There's no disaster here, somebody is getting paid and that' s a big win for them.

 

[behindyourightnow]

One thing it's also really bad at is use any human judgement for the quality of content. Okay, most humans aren't very good at that either, but it's really bad at it. It's like it assumes that if "content" exists and there is no counterpoint to that content, the content must be true.

One of the funnier things that came out of the Google AI debacle was questions related to fictional characters. You'd ask "Are there any gay Star Wars characters?" and it trawled until it found some shitpost about a gay character named Slurpy Faggi (and didn't find any other content arguing that there isn't a gay character named Slurpy Faggi) so Slurpy Faggi became the truth.

 

[soyjersh]

How do the hallucinations work? Where does it get the data from to produce them?
So say I ask for a report on how many chickens Bolivia produces per month. The LLM scours its database and gives me the answer based on it finding a few reports in there about chicken production and summarising them. I get that. But let’s say it just spazzes out and tells me nine chickens and that wrong completely. A hallucination.
Where does that come from? Is it picking up incorrect data from its database or is it making it up? If it’s making it up, how does it do that?
Apologies if this is a dumb question.

It's important to understand hallucinations are a byproduct in generative models standard of unsupervised learning. This term refers to the fact that the model is not given any human input regarding its "understanding" of the data it processes. Training in an unsupervised fashion entails fitting a model q to a set of observed data xₚ ⊆ X drawn iid from some true distribution p on x ∈ X. Now, of course p may not exactly belong to family Q of probability distributions being fit, whether Q consists of Gaussians mixture models, Markov models, or even neural networks of bounded size.

In laymans terms, consider fitting a generative model on a text corpus consisting partly of poetry written by four year-olds and partly of mathematical publications from the Annals of Mathematics. Suppose that learning to generate a poem that looks like it was written by a child was easier than learning to generate a novel mathematical article with a correct, nontrivial statement. If the generative model pays a high price for generating unrealistic examples, then it may be better off learning to generate children’s poetry than mathematical publications. However, without negative feedback, it may be difficult for a neural network or any other model to know that the mathematical articles it is generating are stylistically similar to the mathematical publications but do not contain valid proofs.

How this problem is solved for models like chatGPT involves validity oracles for generated outputs like computer programs or a LaTeX compiler for mathematical equations. however, they need to use statistical validity oracles for human text, as there is no binary classification for "correct" human generated text. This issue can be generalized into a simple generative model q on two axes. First axis is coverage, which is related to the probability assigned to future examples drawn from the true distribution p. Second is validity, defined as the probability that random examples generated from q meet some validity requirement. A proof can be formulated to show that not only is there a tradeoff between validity and coverage but it's impossible to maximize both. There's mathematical details from there but I get the feeling it would be used on the wrong audience, so I'll make this brief: since they want the model to be both "efficient" and "correct" with its outputs, but can't tell the model through human understanding what exactly those two things are it is impossible for unsupervised generative models to work without hallucination. It becomes more complex with Retrieval Augmented Generation(RAG) where the model actively searches the internet for data to modify its answers. A great video for understanding both hallucination and RAG is here.

 

[SirCliveSinclairdabbing]

How do the hallucinations work? Where does it get the data from to produce them?

We know the obvious answer and the stupid answer to a quadratic.
The computer doesn't.

 

[Mad Mamba]

Honestly I think the problem isn't the technology itself, which is at the very least fun as a toy, and still has some legitimate uses IMO. Procedural/random generators have been around for as long as people have used computers for entertainment, they just got REALLY good at it. The problem is the bullshit marketing around it, slapping the word AI onto any kind of generative programme and treating it like a fix-all panacea that can solve any problem for anyone. I can see similar parallels to NFTs (which are arguably a great solution for security, as if you need a single non-reproducible NFT to access something then unwanted access will in theory be harder, I would have thought?). The main difference being NFTs were far more experimental technology when they were snapped up by the tech industry and forced into the Next Big Thing, whereas random generators are as old as it gets.

There are definite uses for "AI" that aren't just "press button get art". For elements of art, AI has multiple applications that could be very useful for creatives; my wife used an AI to generate a colour palette for inspiring a piece she otherwise did 100% herself, and as a 3D guy I can see these models being great for reducing tedious busywork. Especially in texturing- if I'm making, for instance, a throwback shooter and need a low-res but recognizable rock texture, I can ask the AI to generate me a 64 X 64 swatch of rock texture (with other traits as needed) and I'll save a lot of time either manually drawing out the texture or resampling an existing one to look the part. Something like that is perfect for AI. The rock texture isn't going to be the main focus of the game, but it's still necessary. Hell, 3D artists creating landscapes often use AI-like procedural generators for either infinitely generated worlds in games or else just starting points for terrain (Perlin Noise has been in use for decades at this point).

Likewise, whilst it certainly won't turn a complete novice into a seasoned coder overnight, for programming/tech support, I'd argue a more focused, dedicated ChatGPT-like text model would actually be a really useful asset. I had a problem trying to install software on my laptop, and spent ages trying to deciper what the error codes meant, doing plenty of googling and coming up blind. I'm not Grandma tier tech-illiterate but I'm no expert on this sort of thing either. I asked ChatGPT how to solve this, giving the error code and other basic information, and very quickly I got the software working. Again, it's not a catch-all fount of wisdom but as effectively an interactive context-aware guidebook for people who still have some idea of what they're trying to accomplish, it would save a LOT of frustration and time spent trawling the internet for the one page where someone had the same problem, and praying they actually got an answer.

As a bit of a TLDR to all that, I think the actual technology is a great thing that should be researched and utilized more creatively and I hope marketing crooks don't strangle advanced generative models in the crib by peddling them as something they aren't and utterly poisoning the well.

 

[Catch The Rainbow]

https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you-if-you-mention-ai-again/

Great blogpost about why AI is bullshit

Unless you are one of a tiny handful of businesses who know exactly what they're going to use AI for, you do not need AI for anything - or rather, you do not need to do anything to reap the benefits. Artificial intelligence, as it exists and is useful now, is probably already baked into your businesses software supply chain. Your managed security provider is probably using some algorithms baked up in a lab software to detect anomalous traffic, and here's a secret, they didn't do much AI work either, they bought software from the tiny sector of the market that actually does need to do employ data scientists... Consider the fact that most companies are unable to successfully develop and deploy the simplest of CRUD applications on time and under budget. This is a solved problem - with smart people who can collaborate and provide reasonable requirements, a competent team will knock this out of the park every single time, admittedly with some amount of frustration...Most organizations cannot ship the most basic applications imaginable with any consistency, and you're out here saying that the best way to remain competitive is to roll out experimental technology that is an order of magnitude more sophisticated than anything else your I.T department runs, which you have no experience hiring for, when the organization has never used a GPU for anything other than junior engineers playing video games with their camera off during standup, and even if you do that all right there is a chance that the problem is simply unsolvable due to the characteristics of your data and business?

 

[Joe Swanson]

It's the dotcom bubble all over again, investors throwing money at everything AI like they did with everything on the internet, and it'll end the same way, with the bubble bursting giving way for the actual useful stuff with most of the schlock going bankrupt

 

[Flan Handler]

How do the hallucinations work? Where does it get the data from to produce them?
So say I ask for a report on how many chickens Bolivia produces per month. The LLM scours its database and gives me the answer based on it finding a few reports in there about chicken production and summarising them. I get that. But let’s say it just spazzes out and tells me nine chickens and that wrong completely. A hallucination.
Where does that come from? Is it picking up incorrect data from its database or is it making it up? If it’s making it up, how does it do that?
Apologies if this is a dumb question.

If I understand correctly, the LLMs are basically predictive text, and have no context to what they are putting out. The output is what the LLM perceives as the most likely next word. We’ve got a Clever Hans tapping out simple arithmetic, and fooling his own handlers into thinking he’s doing the calculations.

Humans are a skittery, credulous bunch of apes with terminal FOMO, so I guess let’s sink trillions of dollars into data centers. It’s going to work great!

 

[General Emílio Médici]

Yeah, like everyone with a brain already knows.

AI doesn't exist. The market grifters are purposefully using the term wrong to trick normies and boomer investors into thinking the are making Artificial General Inteligence (Data, Skynet, Wall-E, Geth, etc) when in fact all they are doing is making large language models that are very good at making and running markov chains and then extrapolating from data.

 

[Unarmed Gunman]

There is a whole lot of "why would I ever want to surf the internet on my phone" energy in this thread.

Technology advances exponentially, so it doesn't matter if you think the current iteration is useless. What will it look like in 5-10 years? Does anybody not remember waiting 30 seconds for a website to maybe load on your phone, then having to pinch and zoom and side-scroll to try and read the content?

And don't forget what Uncle Ted had to say about these types of technological advances 30 years ago...

 

[Overly Serious]

How do the hallucinations work? Where does it get the data from to produce them?
So say I ask for a report on how many chickens Bolivia produces per month. The LLM scours its database and gives me the answer based on it finding a few reports in there about chicken production and summarising them. I get that. But let’s say it just spazzes out and tells me nine chickens and that wrong completely. A hallucination.
Where does that come from? Is it picking up incorrect data from its database or is it making it up? If it’s making it up, how does it do that?
Apologies if this is a dumb question.

Hallucination is an imperfect term but more so the causes can be very complex and hard to discern. Hence "hallucination" because it appears to just be some random bizarre output. There will be a reason but it may not be discernible. The quality of the input data might be a good and simple example of something that could lead to a hallucination. Lets go with visual examples for the explanation and relate it to text afterwards.

Say you train your visual AI with millions of images and some of those images are tagged as "British", "Britain", etc. Then you later ask the trained model to draw you pictures of British people. Often it works but sometimes it throws out a weird result of Thor on his chariot. An big hallucination. Why? Because when you were categorising your data you didn't realise that the images tagged British overwhelmingly had rain in the background and heavy clouds. Meanwhile, pictures of Thor frequently depict him amidst storms. So the trained model now associates British = Rain = Thor.

This is a silly example that I just made up for the sake of explanation, but it shows how the less careful you are with your training data the more prone to "hallucinations" your model might be. What you should do is carefully make sure that of your images of British people and British towns, you include some where it's not raining (good luck!). And other images that are tagged rain but are not tagged "British". Then it learns that Britain and Rain are different things, even if they're commonly correlated.

When you get to LLMs, things answering technical questions, it can become even more impenetrable. All sorts of unnoticed associations and biases can creep in. And it's not just about the training data, it can be for example, about the purpose of the model. A set of associations or response types that work well in one context, might be very prone to failure in another. Ask a model primarily designed to provide programming answers how it's feeling, you're getting outside its comfort zone. The further away you get from what a model is actually fit for, the more likely it might flip and give something weird. Like a kid guessing answers in an exam. And that's another source of Hallucinations. They don't know when to stop or when not to give an answer. One thing AI models seldom do is say: "I'm not sure". If they're not sure, they'll throw something out there. When you ask it how many chickens are consumed and it doesn't know it will give its best guess, see it has some data on chickens that includes nine and say nine chickens were consumed. AI does the "closest fit" most of the time, and doesn't understand "not close enough to use".

People often use the example of auto-complete in text to explain current AI models. There's a good reason for that - in many ways they are effectively just a really, really impressive way of expanding on some starting point (the input you give it) and following that down some predictive path. And some odd thing might send it off down an unusual path when you least expect it and unknown to itself it will keep on down it. Bang - Thor appears.

But the analogy is a little limited - auto-complete goes in one direction. It see you type "cont" and extrapolates linearly to "continuous, continual, contingency..." etc. Models are extrapolating by patterns as well which is why the analogy is good, but they're not going quite so linearly as that. They're multi-dimensional. However, the analogy is not a bad one. They're essentially trying to find a good fit for a given input, based on what they've been trained a good fit is.

 

[Trump's Chosen]

It's the same cycle. New exciting thing. Everyone wants a piece. It didn't fulfill every part of imagination. Bubble bursts. Projects of actual value continue. Become the new google or microsoft.

 

[Vibralldo]

Reminds me of 5 years ago when every tech product was integrating the blockchain

Tech investors are literal retards that you jangle the newest shiny buzzword in front of, and then they give you $5,000,000 in venture capital

Then a new buzzword comes out and they become interested in that