Mathematical and Scientific Crises

[This post is wildly speculative. File under ‘playing with ideas.’]

On the continuing and fascinating saga of the ABC conjecture and Mochizuki’s befuddling proof:

Five years ago, Cathy O’Neil laid out a perfectly cogent case for why the (at that point recent) claims by Shinichi Mochizuki should not (yet) be regarded as constituting a proof of the ABC conjecture. I have nothing further to add on the sociological aspects of mathematics discussed in that post, but I just wanted to report on how the situation looks to professional number theorists today. The answer? It is a complete disaster.

From Andrew Gelman’s post on the ongoing replication crisis facing psychology and many, many other fields:

At this point, certain earlier work was seen to fit into this larger pattern, that certain methodological flaws in standard statistical practice were not merely isolated mistakes or even patterns of mistakes, but that they could be doing serious damage to the scientific process. Some relevant documents here are John Ioannidis’s 2005 paper, “Why most published research findings are false,” and Nicholas Christakis’s and James Fowler’s paper from 2007 claiming that obesity is contagious. Ioannidis’s paper is now a classic, but when it came out I don’t think most of us thought through its larger implications; the paper by Christakis and Fowler is no longer being taken seriously but back in the day it was a big deal. My point is, these events from 2005 and 1007 fit into our storyline but were not fully recognized as such at the time. It was Bem, perhaps, who kicked us all into the realization that bad work could be the rule, not the exception.

Two entirely different scientific controversies occurring in entirely different fields and of an entirely different nature. They have nothing in common. Right?

Right? I don’t know if there’s anything here, especially as a non-expert in this (and any) field.

(Sigh. I do love teaching, but sometimes I wish I had a knowledge base and expertise that I could really rest on. All this stuff of controversy makes me wish that I could really speak with authority on any of this fascinating stuff. A different lament for a different time…)

OK, so let’s toss one more scientific controversy into the mix: Yann LeCun’s post about AlphaZero-style machine learning, and the opaqueness of AI’s solutions to human observers:

The main message was, in essence, that the current practice in machine learning is akin to “alchemy” (his word). It’s insulting, yes. But never mind that: It’s wrong! Ali complained about the lack of (theoretical) understanding of many methods that are currently used in ML, particularly in deep learning. Understanding (theoretical or otherwise) is a good thing. It’s the very purpose of many of us in the NIPS community.

But another important goal is inventing new methods, new techniques, and yes, new tricks. In the history of science and technology, the engineering artifacts have almost always preceded the theoretical understanding: the lens and the telescope preceded optics theory, the steam engine preceded thermodynamics, the airplane preceded flight aerodynamics, radio and data communication preceded information theory, the computer preceded computer science. Why? Because theorists will spontaneously study “simple” phenomena, and will not be enticed to study a complex one until there a practical importance to it.

And maybe this is the tension that unites all three of these scientific controversies. We aren’t living in an moment marked by intense industrial or even informational explosion. Rather we are living in an age of immense theoretical constructions that outstrip any single person’s ability to understand them. On the corporate side this includes the algorithms of Google or Facebook. The controversy over machine learning is of the value of algorithms whose methods are unknown to the operators — e.g. facial recognition software whose methods for identifying faces would itself need to be studied in order to be comprehended.

Now, the crisis facing psychology and other sciences isn’t exactly new and its causes include non-open data practices as well as publication bias and a commitment to conceptual replication. (More here.) But it seems that a lot of scientists were treating statistics itself like a black box. You collect some data, and then toss it into the statistics pot and, boom, out comes an effect. Science was using statistics as a big, mysterious theoretical edifice, and now we’re grappling with the aftermath.

And maybe this is a way to think about the Mochizuki controversy. True, the edifice he’s created is impossible to penetrate and basically no one understands it. But that puts it in good company, along with machine learning engines and the statistical packages that psychologists run their data through. Maybe there was a time when science was about understanding, but now we’re trying to take science beyond mere understanding, and struggling to find the rules of engagement.