Encore!

Another school year: done.

I love the dismantling that happens after the kids leave. For months this place has been made just so, and it takes just moments for the entire accumulation process to be set in reverse. Posters are first in the line of fire, but sooner or later everything is headed either to a drawer or the trash. The recycling bins were stuffed with homework this afternoon.

I’ve never been involved with anything theatrical (ok fine once) but I imagine it’s similar to what it feels like to strike the set the day after everything closes. Everything of significance has got to go. It’s all trash now, but just a few hours before the whole thing was whatever it is you call trash’s opposite. This stuff was indisposable by mutual assent. Now: nah.

And what that made me think of was the artifice of this whole enterprise. School is such a weird performance of the strangest kind of pretend — they call me ‘mister’! People sometimes point to the artificiality of schools as a critique: this isn’t what learning looks like in its natural state. And of course that’s exactly right, it’s not. It’s all fake. And that guy up on the stage — did you know he’s not really a wizard?

Teaching is weird, it’s fake. Teaching is not medicine. Medicine is someone is sick and you can help them, the most natural thing in the world. Driving a cab is not weird. People want to get to x, they pay you to take them there. Natural. Getting paid to cook people food is a reasonable transaction, it is not a weird. Lawyering, on the other hand: super weird. And as long as we’re on the topic: basketball player, musician, researcher, middle manager, writer, actor, teacher, weirdos all.

So it’s summer and the show is over and all but, come autumn, let’s do it again! I’ve got a school I love working at and I’m ready to play pretend for another year. Year nine is over.

Don’t ask “does it work?”

From Larry Cuban:

Do Core Knowledge Programs Work?

As for many school reforms over the past century, answering the “effectiveness” question–does it work?–is no easy task. The first major issue is answering the question of whether Core Knowledge was fully implemented in classrooms. If not completely implemented, then judging outcomes become suspect. Many of the early studies of Core Knowledge in schools were mixed, some showing higher test scores and some showing no positive effects (see herehere, here, and here). The Core Knowledge Foundation has a list of studies that they assert show positive outcomes. What is so often missing from research on reforms such as Core Knowledge are descriptions of the contextual conditions in which the reform is located and researchers saying clearly: under what conditions does this program prove effective? That is too often missing including the research on Core Knowledge schools.

The primary job of education research shouldn’t be to figure out what works, or to put it another way we shouldn’t expect the have a yes/no answer to that question. How does it work? When does it work? When it didn’t work, why didn’t it work? When it’s not used with fidelity, why wasn’t it used with fidelity? Education is not served well by the way research on program efficacy seems to frequently be done.

Mathematics that makes itself

Can something be true, just because you say it?

One example might be a promise. If you promise somebody that you’ll feed their cats…well, all of the sudden there is a promise there. The act of promising creates a promise. All of the sudden, there it is. It makes itself.

Anyway, maybe mathematics can sometimes pull off a trick like that. In 2003, MacKenzie and Millo argued that this is precisely what happened in financial markets with the Black-Scholes formula, a highly successful mathematical model used to find “correct” prices for a stock option:

Option pricing theory—a “crown jewel” of neoclassical economics—succeeded empirically not because it discovered preexisting price patterns but because markets changed in ways that made its assumptions more accurate and because the theory was used in arbitrage.

In other words, the use of the formula itself made the formula more reliable. It was a self-fulfilling mathematical model, a piece of mathematics that reshaped the world to conform to its assumptions. Wow.

(I found this interesting blog post that dives a bit deeper into the logic of a self-fulfilling equilibrium.)

If this feels eerie, it’s only because we’re forgetting how strange and self-referential the notion of predicting the markets really is: markets are hard to predict because they are predictions. This is a way that finance and economics is fundamentally unlike the natural sciences. In finance there is always the possibility that the scientist will influence the subject.

Black, Scholes, and Merton’s model did not describe an already existing world: when first formulated, its assumptions were quite unrealistic, and empirical prices differed systematically from the model. Gradually, though, the financial markets changed in a way that fitted the model. In part, this was the result of technological improvements to price dissemination and transaction processing. In part, it was the general liberalizing effect of free market economics. In part, however, it was the effect of option pricing theory itself. Pricing models came to shape the very way participants thought and talked about options, in particular via the key, entirely model‐dependent, notion of “implied volatility.” The use of the BSM model in arbitrage—particularly in “spreading”—had the effect of reducing discrepancies between empirical prices and the model, especially in the econometrically crucial matter of the flat‐line relationship between implied volatility and strike price.

To be clear, Ed Throp used option pricing to make a killing before the markets were influenced by Black-Scholes. So it’s not like the formula created its own reality entirely. The claim can only be one of degrees — that the model became more reliable, that the markets grew more like what the model predicted. I am unable to evaluate the evidence on its own and haven’t dived deeper into any of this literature but, huh, it makes you think doesn’t it?

It reminds me of Ben Blum-Smith’s excellent post about voting theory, where he suggests that mathematicians have at times gotten lost in their models and believed in them too strongly, more because of their mathematical properties than for any of their use in application. But what if — only at times, and only by degrees — your mathematical model could be its own fulfillment by changing the world to more closely accord to its predictions? Wouldn’t that be something.

On Benoit Mandelbrot’s “New Methods in Statistical Economics” [Part 1]

I’ve been reading Mandelbrot’s 1963 paper “New Methods in Statistical Economics.” It’s one of his many papers from the 1960s where he argues that people should pay more attention to non-Normal distributions (see my post). This is my attempt to explain the first two sections of his piece.

(My main purpose is to clarify things for myself here, so if you have questions or can explain issues with my exposition I would VERY VERY appreciate hearing from you!)

I come to the paper with two interests. First, Mandelbrot loudly argued that the price fluctuations of many commodities and securities is best thought of as nonGaussian. This is a paper where he makes that argument. That’s cool because finance is cool and important and interesting.

But the other reason is because just a few years after this, Mandelbrot became Mr. Fractal and published “How Long is the Coast of Britain?” This was the piece that popularized the notion of fractal dimension (something Mandelbrot calls a Trojan Horse for the way it represented a safe, neutral topic that served as a vector for his dimensional ideas).

But Mandelbrot also said that parts of this paper — which really seems to have nothing to do with fractals — represent the seeds of his geometric ideas. So, for example, he says this in the appendix of the reprinted edition of this 1963 paper:

The many footnotes in the original, except one, were easily integrated in the text. But Footnote 4 did not fit, and it cried out to be emphasized, because it was an early allusion to the theme of self-similarity that came to dominate my life and led to fractals. This footnote 4 read as follows:

“The various criteria of invariance used by physicists are somewhat different in principle from those I propose in economics. For example, the principle of relativity was not introduced to explain a complicated empirical relation, such as scaling. I am indebted to Harrison White for suggesting that I should stress the nuances between my methods and those of physics.”

So what does that have to do with fractals? It’s a subtle thing! Let’s dive in to the paper to see where he’s coming from. I’ll quote (sometimes at length) and then comment below the text.

Mandelbrot:

The approach I use to study the scaling distribution arose from physics. It occurred to me that, before attempting to explain an empirical regularity, it would be a good idea to make sure that this empirical identity is “robust” enough to be actually observed. In other words, one must first examine carefully the conditions under which empirical observation is actually practiced. The scholar observes in order to describe but the entrepreneur observes in order to act. Both know that most economic quantities can hardly ever be observed directly and are usually altered by manipulations.

Here is a thought experiment involving biology, not physics, but I think it illustrates the point. Suppose that you have a theory about trees: that the distribution of tree heights follows a nice bell curve. 

You are a lazy scientist that doesn’t want to measure anything, let’s say. Anyway, the world is large and you didn’t get funding so you can’t travel. Fortunately, a lot of other people have already done measuring! You discover this by googling.

But suddenly you run into a problem. Sure, some people have directly measured the heights of trees. But other people measured the total height of forests. That stinks! Sure, you can divide the forests by the number of trees to get an average tree height to put into your tree data. But that’s going to be messy.

So it’s a challenge to deal with data coming from many different sources. And if the data needed to explore your theory only could come from measuring every tree (and if you really can’t do that) then you probably should work on a different problem.

In most practical problems, very little can be done about this difficulty, and one must be content with whatever approximation of the desired data is available. But the analytical formulas that express economic relationships cannot generally be expected to remain unaffected when the data are distorted by the transformations to which we shall turn momentarily. As a result, a relationship will be discovered more rapidly, and established with greater precision, if it “happens” to be invariant with respect to certain observational transformations. A relationship that is noninvariant will be discovered later and remain less firmly established. Three transformations are fundamental to varying extents.

So in natural science, OK, it’s a problem. But in economic problems Mandelbrot is saying it is a MAJOR problem.

Suppose you think that stock prices tend to move along randomly, with the changes plucked from a Log-normal distribution, which looks like this:

LogNormal(median=3,stddev=2).png

OK, so you start looking for data. And some of your data comes from daily prices, some from weekly prices, other from yearly price variations. But there’s a problem: there’s no simple way to describe the relationship between the daily and the weekly data. You might want to simply add up a bunch of the daily data to compare it with the weekly data (or to take the weekly data and divide it by 5).

Well, you can’t. The sum of a bunch of log-normal distributions is not another log-normal distribution. So if your theory is true and the log-normal distribution is what guides the stock market’s random price changes, things are weird. If I understand correctly (not at all sure that I do) then there are two reasons why this is weird. First, it means that your attempt to compare different sources of data is likely to be a mess, as there is no easy way to compare and combine the different sources. Second…shouldn’t the daily and weekly prices show the same distribution? Wouldn’t it be weird if daily and weekly prices were governed by different distributions?

There is something exceptionally nice about the idea that small small variation is reproduced at higher scaled. People come in all shapes and sizes; the deepest levels of physical reality are governed by molecules randomly humming and bumping into each other. The idea that these scales are connected — that what we see is the cumulative result of the way the smallest things are — is highly attractive in both nature and economics. I think that this is a potential connection to the idea of fractals and self-similarity.

So, what are the ways that data might need to be stable when it’s transformed? Mandelbrot names three:

  1. Stable after being aggregated
  2. Stable after being mixed
  3. Stable when you only pay attention to the extremes

The first source of stability is the most important. Here’s an example Mandelbrot gives:

The distributions of aggregate incomes are better known than the distributions of each kind of income taken separately.

So if we are interested in the total distribution of income, we might be looking at a collection of different categories of income. Honestly, I can only guess what these categories might be. People sometimes talk about three forms of income (active, portfolio, passive) so maybe he means that? Or maybe he means that you have income distributions for each of the 50 US states, and you want to aggregate that into a national income distribution?

Anyway:

There is actually nothing new in my emphasis on invariance under aggregations. It is indeed well known that the sum of two independent Gaussian variables is itself Gaussian, which helps use Gaussian “error terms” in linear models. However, the common belief that only the Gaussian is invariant under aggregation is correct only if random variables with infinite population moments are excluded, which I shall not do (see Section V). Moreover, the Gaussian distribution is not invariant under our next two observational transformations.

This is indeed a very nice thing about the Gaussian distribution! You add them together, you get another one. Lots of little measurement mistakes add up to a Gaussian distribution of final measurements. (Lots of little differences at the cellular level lead to big differences in people.)

Mandelbrot is saying, you’d want this to remain true in your economic models. It would make the problems tractable to research; the prices and things really ought to add in this way too. And the vast majority of distributions (those “analytic equations” cited above) do NOT have this property. But Mandelbrot is going to argue in favor of an especially favorite family of distributions that do have this additive property — along with the invariance under the other two kinds of transformations.

These are the distributions that are called “Stable Distributions,” and include the Pareto distributions and the Gaussian distributions as members.

Levy_distributionPDF.png

More on the math of transforming various statistical distributions in Part 2.

Two classroom things

#1

Ask anybody: I’m not some sort of god teacher in the classroom. But the other day a teacher complimented me on my classroom management, in particular my ability to hold attention without using my voice. And I was like, yeah! That’s something I do very much on purpose.

I forget if I came up with this on my own or (more likely) picked it up from something I read, but I try very very hard not to interrupt the flow of the class with a redirection to a kid. None of that “I’ll wait” or “settle down” or “I need all eyes up here.” Not if I can help it, at least.

The thought is that the content of the class shalt not be interrupted, and I undermine this premise when I use my voice to interrupt. Plus, it’s just confusing to kids who are following along to hear this other thing, and then to dive back in. PLUS, it directs everyone’s attention to some kid and their shenanigans. PLUS PLUS, it creates a little pocket of energy in the room for a few moments and that can get filled with nonsense or accidental attention-grabbing things.

So I use looks, pauses, taps on the shoulder, etc. I know (I think) a lot of other teachers do this stuff, but I do it too and maybe it’s worth sharing.

#2

Very frequently there are kids that aren’t feeling confident and you can sort of feel them withdraw from the classroom. Very frequently there is an implicit contract these kids have with their teachers: I will be silent and confused, but you will not mind because I will not cause you an ounce of trouble.

Forget that! Cause me trouble.

So there’s this thing that sometimes happens in my classrooms. It’s a bit awkward, but I think it works out OK, most of the time. Which is that sometimes I’ll see a kid who is settling into that withdrawal, and so I’ll cold call on them.

Putting a kid on the spot in front of the class — especially one who is having a rough time in class — is a tricky proposition, and my purpose is not to embarrass a kid for not engaging. It’s just that they’ve got this contract, and they’re trying to figure out if I’m on board or not. But I’m not — I’m going to invite you in to whatever our class is doing, and I’m going to keep on doing that.

“Sarah, do you agree that 40 divided by 4 is 10? Why does that make sense, do you think?”

I’d be curious to hear the recording of that conversation. In that moment I felt myself trying very hard to balance two contradictory stances: a jokey, non-threatening, supportive teacher who is very persistently insisting that you share your thinking.

I also felt myself very much caught up in another contradiction, which was not signaling to Sarah (or the rest of the class) that I was “dumbing things down” for her, while also trying to find a mathematically interesting way for her to engage. This was also a high-wire act, even if it only lasted a few seconds. In these moments one of my moves is to just produce more and more on my own, but to be insistent that the student take on why.

“So from skip-counting we know that ten 4s makes 40. Why would that help us solve 40 divided by 4?”

Sarah ended up saying something nice, which was that 4 x 10 made 40 and that this gave you the answer to the division problem. (Did I mention yet that this is in my 3rd Grade class? I should’ve.)

Having survived this ordeal, I saw her withdrawing again, not looking at the board, not hearing other people, the sorts of things that raise all of my alarms as a teacher.

So I did something unexpected, which is for the next problem I cold-called on Sarah again. And we did that same thing, the same conversation. DEFINITELY a gamble — this is a risky teaching move. Looking at her face, I started getting very nervous…I thought tears were a distinct possibility. But what could I do? I had to keep going. So I turned back to that last problem.

“You had said 4 x 10 makes 40 to help with the last problem. So how can we think about 48 divided by 4?” 

She made it through that conversation. Thank god for her friend, Kya, who gave her a big high-five after that conversation wrapped up. Sarah put on a big smile.

This was a big risk, but at least this time I feel that it paid off. Sarah is pretty good at skip-counting, and she was able to use that to solve some problems on her own without any prompting from me. I was able to give her a high-five also, to compliment her on thinking to do that all on her own.

And though certainly it could have gone bad, the whole point of this to me was that, no, I do not accept the terms of that contract.

The Efficient Market Hypothesis can deal with fat tails

Perhaps it’s just me that has been getting tangled up in my reading, but it seems to me that there’s confusion out there between two different assumptions underlying financial models.

The first assumption is that when prices of a stock (or whatever) change, they go up and down randomly, with the size and direction of the change plucked from a Gaussian distribution, i.e. a bell curve.

The second assumption is the Efficient Market Hypothesis — that there really isn’t any way to “beat” the market with expert knowledge or deep understanding of industry or the economy or whatever, because everybody knows all that stuff and it’s already incorporated into the price of the stock (or whatever).

Part of what’s confusing is that both of these positions are critiqued by many of the same people. Benoit Mandelbrot, for one. He spent the 1960s taking on the first assumption in a series of papers where he built the case that scientists were too much in love with the Gaussian distribution. He insisted that many things instead looked like their changes were plucked from Pareto distributions. These are fat-tailed compared to the bell curve — the things at the extreme are more likely to happen than the Gaussian model would predict.

Probability_density_function_of_Pareto_distribution.svg.png

This means that if you’ve built your theory on how to manage risk on a Gaussian model, you’re going to systematically underestimate the true risk. In particular, you won’t plan carefully enough for true disasters if you really start believing in your model.

(Of course, you might keep the Gaussian assumption for your metric while also keeping in mind your model’s limitation. Somehow, that doesn’t always seem to happen when these models are deployed in organizations. I mean I don’t have first hand knowledge of that previous sentence, but it sure seems that way from what I’ve read.)

But if the Gaussian models are wrong, does that implicate the Efficient Market Hypothesis? Some people write that way, but I don’t think it does. And the reason why is because the guy who came up with EMH — Eugene Fama — was a big hype man for Mandelbrot.

The Efficient Market Hypothesis appears in a big paper by Fama — “The Behavior of Stock-Market Prices.”  Here’s a note from the first page:

Many of the ideas in this paper arose out of the work of Benoit Mandelbrot oithe IBM Watson Research Center. I have profited not only from the written work of Dr. Mandelbrot but also from many invaluable discussion sessions.

He’s not kidding. A huge cornerstone of the paper is basically a retread of Mandelbrot’s work developing the theory of Pareto distributions, contra the Gaussian assumption. “The Gaussian hypothesis was not seriously questioned until recently when the work of Benoit Mandelbrot first began to appear,” he writes.

If I understand things correctly, the value of Mandelbrot’s work is that it allows Fama to claim that stock-market prices truly are a random walk. Whereas there are discrepancies if you assume this random walk is plucking change from a Gaussian distribution, Fama bolsters the “random walk” hypothesis using Mandelbrot’s work.

(By the way, a random walk using one of the distributions Mandelbrot supported is called a Levy Flight. Levy was Mandelbrot’s advisor. A great site to learn more about Levy Flights is this one — thanks to Mike Lawler for telling me about it. I tried to replicate a Levy Flight in P5 here, if you want to tinker with the code.)

So one of Mandelbrot’s main intellectual influences was on the originator of the Efficient Market Hypothesis. Clearly a belief that stocks are a random walk can coexist with a belief that the distribution guiding that walk is non-Gaussian.

Just as clearly, though, Mandelbrot would not agree with Fama, who represents a kind of establishment, conventional wisdom position in the financial world. There is a story here that I’m missing, one that I would very much like to know. Mandelbrot has a reputation as surly, with an enormous ego. (One joke goes that a ‘mandelbrot’ can be used as a unit of ego.) So when Mandelbrot critiques EMH and conventional finance models (like Markowitz’s theory of risk management) … there must be a story of falling out, tension with Fama. Right?

In any event, there is no necessary connection between those two assumptions, and in discussing the sort of thinking that results in financial catastrophe they really should be treated as two very separate things.

It all is weird to me

My experience of being a Jew is experiencing a lot of things and not having the slightest clue how to make sense of them in the context of history and contemporary life.

Like that one time I went to Oklahoma. I was walking a few miles on Shabbat and wearing my yarmulke. A car pulls up beside me and starts honking, pulls down their window, shouts “WE SUPPORT ISRAEL!” and moves on. What do I do with that?

Here’s something that happened in camp last summer. A bunch of kids were eating pizza. The pizza was not kosher, so I did not eat it. I was hanging out in a classroom where some kids were eating, including B.

“Did you eat pizza?” B asks.

“Nah,” I say.

“Why not?”

“Because it’s not kosher pizza, and I only ever eat kosher food or pizza from kosher places.”

(Which is not a good explanation, but it’s sort of complicated and not every kid wants a whole long thing.)

“Where are you from?” he asked.

“I grew up outside of Chicago and live in NY.”

This was followed by a longish, kind of awkward pause.

“But where are your parents from?”

“Boston and Philadelphia.” No nod of satisfaction from B. “Is that what you were asking?”

“I mean ethnically, where are you from?”

“So…I’m Jewish, but Jews come from all over the place.”

“Oh ok. So you’re from…”

“Mostly Russia. But, yeah, mostly like Boston.”

“…where is your family from?” I asked.

“Philippines. Do you think we can get seconds?”

So, what was that? Is there something wrong with that interaction? Clearly he is puzzled by my whole deal. And, he’s a kid. There’s nothing wrong with curiosity, and he asked questions in good faith. I was happy to have the conversation, even though it underlined my differences.

And what about the time, when I was new to my school, that a friendly conversation with the maintenance guy turned to, like, a list of all the stuff he knew about Jews? And I wasn’t exactly sure what he was saying about the Jews and JFK, but he was saying it in a friendly way? So while I got weird vibes it was probably meant well?

I have a mental list of weird interactions with people in my adult life that I honestly am confounded by. Like that person at the grocery store a few weeks ago. It was Thursday night at our local place. She was, like, “you people shop late!” And I said, “what?” She leaned in a whispered: “There’s a lot of Jews here.” And I said “It’s because tomorrow is Shabbat.” And she was like “I think it’s because you work so hard.” But then she talked about how much we had in common because we both had tyrants: El Jefe and Hitler. So that was a nice conversation!

So I don’t know how to make sense of all this stuff. I see some frum Jews write about anti-Semitism online, but mostly it’s from (in my opinion) mistake assumption that anti-Semitism is exactly like contemporary racism in its nature and effects. The fact that white supremacists hate both Jews and Black people doesn’t mean that our experiences are identical or ought to be written about in identical ways. Some Jewish writers come from a place of angst that anti-Semitism isn’t treated like racism.

I can’t describe my Jewish experience with anything but confusion. I think the weirdness of it is an essential part of walking around with a yarmulke, but also of Jewish life in general. It’s just very hard to know what matters and how much. So when a person is shot in a synagogue and it doesn’t dominate the news…is that appropriate? Proportionate? And does it matter? I have no way to think about this, except that it’s confusing and it probably should be.

Q&A with Eugene Fama on the Bell Curve in finance, posted without commentary

It would be very enlightening if you would comment on the Nassim Nicholas Taleb (“The Black Swan”) attack on the use of Gaussian (normal bell curve) mathematics as the foundation of finance. As you may know, Taleb is a fan of Mandelbrot, whose mathematics account for fat tails. He argues that the bell curve doesn’t reflect reality. He is also quite critical of academics who teach modern portfolio theory because it is based on the assumption that returns are normally distributed. Doesn’t all this imply that academics should start doing reality-based research?

EFF: Half of my 1964 Ph.D. thesis is tests of market efficiency, and the other half is a detailed examination of the distribution of stock returns. Mandelbrot is right. The distribution is fat-tailed relative to the normal distribution. In other words, extreme returns occur much more often than would be expected if returns were normal.

There was lots of interest in this issue for about ten years. Then academics lost interest. The reason is that most of what we do in terms of portfolio theory and models of risk and expected return works for Mandelbrot’s stable distribution class, as well as for the normal distribution (which is in fact a member of the stable class). For passive investors, none of this matters, beyond being aware that outlier returns are more common than would be expected if return distributions were normal.

For other applications, however, the difference can be critical. Risk management by financial institutions is a good example. For example, portfolio insurance, which was the rage in the early 1980s, bombed in the crash of October 1987, because this was an event that was inconceivable in their normality based return model. The normality assumption is also likely to be a serious problem in various kinds of derivatives, where lots of the price is due to the probability of extreme events. For example, news story accounts suggest that AIG blew up because its risk model for credit default swaps did not properly account for outlier events.

KRF: I agree with Gene, but want to make another point that he is appropriately reluctant to make. Taleb is generally correct about the importance of outliers, but he gets carried away in his criticism of academic research. There are lots of academics who are well aware of this issue and consider it seriously when doing empirical research. Those of us who used Gene’s textbook in our first finance course have been concerned with this fat-tail problem our whole careers. Most of the empirical studies in finance use simple and robust techniques that do not make precise distributional assumptions, and Gene can take much of the credit for this as well, whether through his feedback in seminars, suggestions on written work, comments in referee reports, or the advice he has given his many Ph.D. students over the years.

The possibility of extreme outcomes is certainly important for things like risk management, option pricing, and many complicated “arbitrage” strategies. Investors should also recognize the potential effect of outliers when assessing the distribution of future returns on their portfolios. None of this implies, however, that the existence of outliers undermines modern portfolio theory or asset pricing theory. And the central implications of modern portfolio theory and asset pricing—the benefits of diversification and the trade-off between risk and return—remain valid under any reasonable distribution of returns.

Source

Did Common Core work?

Chalkbeat’s Matt Barnum does such a wonderful job reporting on edu research. Here he is, taking on a fascinating study that attempted to understand what effect the US’s Common Core standards had. Surprising results!:

How do you a study a policy as far-reaching as the Common Core, particularly one that was introduced alongside a host of other school reforms?

It’s not easy, but Song and her colleagues reasoned that some states were more affected by the switch to “college and career ready standards,” which meant Common Core in almost all cases. So they categorized states by the “rigor” of their previous standards and how similar those standards were to the Common Core.

They divided states into those more affected by the switch (because their prior standards were deemed less rigorous or less similar to the Common Core) and those less affected. Then they compared how each group’s scores changed on fourth and eighth grade NAEP tests between 2010 and 2017.

Common Core didn’t seem to help students’ scores, and over time the standards may have had an increasingly negative effect, according to the study, which has not been formally peer-reviewed.

Other researchers consulted by Chalkbeat, including Laura Hamilton of the RAND Corporation, say the study’s approach is a credible one.

“I’m not ready to conclude that the adoption of rigorous content standards is bad for student learning,” said Hamilton. “But I don’t look at this and think this looks totally wrong. It definitely looks plausible.”

Still, the approach has limitations. Most important is that the study is comparing two groups of states that adopted the standards — so if Common Core universally helped or hurt the states that adopted it, this study would miss that effect.

More, on an unpublished report:

Joshua Bleiberg, a doctoral student at Vanderbilt University, is also studying the impact of Common Core on NAEP scores. His study starts examining student scores when new standards hit classrooms, not when states decided to formally adopt the Common Core. He also excluded states that ultimately dropped the standards.

That all seems to lead to different results. In preliminary findings shared with Chalkbeat, Bleiberg finds that the Common Core had small positive effects on NAEP scores through 2013. His study has not been released publicly, so it can’t be fully examined.

“This is not going to be the type of thing that is going to turn around the whole ship really quickly,” Bleiberg said of the standards. “I would think about [the effects] as quite small.”

I don’t know what to make of all of this. I’m very worried that my internal compass has recently been getting out of whack, that I’m getting too pessimistic about any possible major change. At the same time I’m increasingly happy to talk about fundamental issues with the way things are done. (For example, that math is frequently not serving kids particularly well in high school.) But — back to the pessimism — there isn’t much to do about.

I guess I try to keep both of those ideas in mind at once, and to work in the tension between them.

Addendum: Though! Check out another piece of Matt Barnum reporting, about how reducing pollution increased test scores.

The situation seems to be that ed reform is less effective than you think it’ll be, but that improvements to quality of life tend to have more educational implications than you’d think. (The main driver of US educational improvements? GDP.)

My ed reform platform: capital improvements to schools, air conditioning in every school, better food for kids, cut pollution.