The Efficient Market Hypothesis can deal with fat tails

Perhaps it’s just me that has been getting tangled up in my reading, but it seems to me that there’s confusion out there between two different assumptions underlying financial models.

The first assumption is that when prices of a stock (or whatever) change, they go up and down randomly, with the size and direction of the change plucked from a Gaussian distribution, i.e. a bell curve.

The second assumption is the Efficient Market Hypothesis — that there really isn’t any way to “beat” the market with expert knowledge or deep understanding of industry or the economy or whatever, because everybody knows all that stuff and it’s already incorporated into the price of the stock (or whatever).

Part of what’s confusing is that both of these positions are critiqued by many of the same people. Benoit Mandelbrot, for one. He spent the 1960s taking on the first assumption in a series of papers where he built the case that scientists were too much in love with the Gaussian distribution. He insisted that many things instead looked like their changes were plucked from Pareto distributions. These are fat-tailed compared to the bell curve — the things at the extreme are more likely to happen than the Gaussian model would predict.

Probability_density_function_of_Pareto_distribution.svg.png

This means that if you’ve built your theory on how to manage risk on a Gaussian model, you’re going to systematically underestimate the true risk. In particular, you won’t plan carefully enough for true disasters if you really start believing in your model.

(Of course, you might keep the Gaussian assumption for your metric while also keeping in mind your model’s limitation. Somehow, that doesn’t always seem to happen when these models are deployed in organizations. I mean I don’t have first hand knowledge of that previous sentence, but it sure seems that way from what I’ve read.)

But if the Gaussian models are wrong, does that implicate the Efficient Market Hypothesis? Some people write that way, but I don’t think it does. And the reason why is because the guy who came up with EMH — Eugene Fama — was a big hype man for Mandelbrot.

The Efficient Market Hypothesis appears in a big paper by Fama — “The Behavior of Stock-Market Prices.”  Here’s a note from the first page:

Many of the ideas in this paper arose out of the work of Benoit Mandelbrot oithe IBM Watson Research Center. I have profited not only from the written work of Dr. Mandelbrot but also from many invaluable discussion sessions.

He’s not kidding. A huge cornerstone of the paper is basically a retread of Mandelbrot’s work developing the theory of Pareto distributions, contra the Gaussian assumption. “The Gaussian hypothesis was not seriously questioned until recently when the work of Benoit Mandelbrot first began to appear,” he writes.

If I understand things correctly, the value of Mandelbrot’s work is that it allows Fama to claim that stock-market prices truly are a random walk. Whereas there are discrepancies if you assume this random walk is plucking change from a Gaussian distribution, Fama bolsters the “random walk” hypothesis using Mandelbrot’s work.

(By the way, a random walk using one of the distributions Mandelbrot supported is called a Levy Flight. Levy was Mandelbrot’s advisor. A great site to learn more about Levy Flights is this one — thanks to Mike Lawler for telling me about it. I tried to replicate a Levy Flight in P5 here, if you want to tinker with the code.)

So one of Mandelbrot’s main intellectual influences was on the originator of the Efficient Market Hypothesis. Clearly a belief that stocks are a random walk can coexist with a belief that the distribution guiding that walk is non-Gaussian.

Just as clearly, though, Mandelbrot would not agree with Fama, who represents a kind of establishment, conventional wisdom position in the financial world. There is a story here that I’m missing, one that I would very much like to know. Mandelbrot has a reputation as surly, with an enormous ego. (One joke goes that a ‘mandelbrot’ can be used as a unit of ego.) So when Mandelbrot critiques EMH and conventional finance models (like Markowitz’s theory of risk management) … there must be a story of falling out, tension with Fama. Right?

In any event, there is no necessary connection between those two assumptions, and in discussing the sort of thinking that results in financial catastrophe they really should be treated as two very separate things.