Hallucinations

AI suffers from hallucinations: it gives you a wrong answer in full confidence. The wrongness isn't the interesting part, since everything is wrong sometimes. The confidence AI has is the key difference, because intelligence is largely based on uncertainty.

Think about where that importance of uncertainty comes from. The world is uncertain and the things around us change constantly, which drives intelligence to think broad and general instead of memorising. You can't predict the future and you don't know what to expect, so intelligence builds in safety measures. And you can't know everything, which is what drives it to keep improving.

Take uncertainty out of the real world and none of that pressure would exists. In a world like that intelligence wouldn't have evolved like it did in our world, because our survival was depending on it.

Uncertainty

To build real intelligence, we need a measure of uncertainty. Call it the mathematics of uncertainty.

Self-driving cars show why it matters. A lot of unexpected things happen on a road, and the car needs to know when something is uncertain so it can slow down instead of making a wrong decision.

Which way will the bike turn? You can't predict that, and no amount of extra data fixes it, because the information doesn't exist yet. It's like flipping a coin and is non-deterministic, but it will happen.

A horse running in a hailstorm is a different problem: the world is uncertain, and this car has simply never been trained on it.

Humans have a sense for both. Something moves in the grass, and immediately you're asking whether it's a bird or an animal. That's probabilistic inference. A handful of possibilities, each with odds, reweighted whenever extra context arrives. It's night, so probably not a bird. Maybe a hedgehog, maybe a weasel. It's winter and the hedgehog is asleep, so it should be a weasel, right?

What comes out of that isn't one answer. It's a list with weights or probabilities on it.

Correctness versus confidence

Image recognition makes the distinction between correctness and confidence visible. A neural network gives you a label and a number next to it, and we have spent years optimising only the label. That label is only correctness: does it match the image.

The number is a different thing, and we barely optimise it at all. Take a photo of a school bus, change a few pixels in a way nobody can see, and the neural network says cheetah, 99% certain.

The 99% is the problem, not the cheetah. The model moved from certain to certain without ever exposing "I don't know what this is", because that number was never a reward. Training rewarded it for picking the right label, and never once checked whether 99% meant it was right 99% of the time.

This is the textbook example of hallucination with confidence, no uncertainty involved at all.

Historical perspectives

Expert rule-based systems couldn't learn from data, so every gap had to be filled in by hand. Neural networks are more flexible and can learn from data, and probability came along in the same package: the output of a network is a distribution, and training it is a statistical exercise. Neural networks dragged in theories from the statistical field, and they were also the driving force behind introducing cognitive science into it.

So why didn't we do this properly from the start? Representing a distribution over every possible hypothesis was computationally impossible, and it needed machines nobody had. Ghahramani's undergraduate work in 1989 ran on a supercomputer with 65,000 processors, which is slower than the phone in his pocket today.

Bayesian thinking/rules

It all starts with prior beliefs: a certain expectation, expressed as probability.

A murder mystery is a good example. You have a prior probability on each suspect, then the murder weapon turns up and every suspect gets a new probability based on that fact, then an alibi collapses and you do it again. That models what learning actually is, raising and lowering probability with findings, and it matches how our brains work.

So how good are current systems at this? Do the test: ask the AI something obscure and it will fake certainty, and you can trick it. Push back with "I think you're wrong", with no argument behind it, and it folds.

It struggles because it was never trained for this. It was trained on a big amount of data, where confident statements are rewarded and "I don't know" had no meaning.

There's a real tension underneath, though. Creative versus factual, hallucination versus correct: being able to write a made-up story requires exactly this hallucination, while naming the year of a treaty does not. Same mechanism, opposite outcome.

Uncertainty in the real world

Weather forecasts are the example of uncertainty done right. Nobody says it rains at 15:00, they say 70% change of rain around 15:00 close to Leuven.

GenCast predicts 15 days ahead and it's very fast, using a diffusion model that goes from noise to result. Being fast is what lets you run it often, with incremental improvements hour by hour.

These systems are better because they added the concept of uncertainty. A broken sensor or a sudden change in wind direction bends the forecast instead of breaking it.

If we do manage to expose the uncertainty of an answer, representing it becomes the next challenge, and that depends heavily on the AI scepticism of the person receiving it.

Future of AGI

There are roughly two camps. One says more data, more compute, more scale and you're ready. This is what I previously called brute-force AGI. The other camp says we need new architecture, which is the newer approach to the challenge, and the argument for it is a list of things current models do badly.

  • Continuous learning. Animals and humans train constantly and improve, while current models train for months and then it stops, and a new model starts from scratch again.
  • Catastrophic forgetting. Overwriting existing data and conclusions is a symptom of current neural networks.
  • Power consumption. Organic brains are insanely efficient, so more energy efficiency is certainly needed.
  • Data efficiency. They need 100 books to get what we get out of a few.
  • Speed. Neural networks are just slow, and that shouldn't only be solved with more hardware.

We need to add human characteristics like honesty and doubt, and that means finetuning the definition of what we mean by an artificial intelligent system.

Reference

Inspired by the podcast [The mathematics of AI uncertainty](https://www.youtube.com/watch?v=tBjgCj_dGZM).

Google DeepMind: The Podcast, hosted by Hannah Fry. On this episode Zoubin Ghahramani, who has spent the last 30 years pioneering a type of intelligence built on the mathematics of uncertainty.