An extremely brief summary of what I’ve learned about math anxiety and timed tests over the past few days

• There are many studies that find math anxiety impacts how well a kid does in math. This includes performance on timed tests.
• There are pretty much no studies that attempt to find evidence that timed tests contribute to math anxiety. (See this thread for the full conversation.)
• There are a handful of studies (two that I saw) that surveyed teacher candidates and basically asked them what makes them anxious about math. Along with some other things like word problems, timed tests are implicated.

• A few people made the argument that “timed tests cause math anxiety” is an untestable hypothesis because it’s unethical. A few researchers chimed in: not untestable in practice, with caveats. (Researchers love caveats.)

• A few people wanted to know where the evidence was that timed tests don’t cause math anxiety. But that would necessarily involve the same sorts of studies that don’t yet exist; the studies that find a causal connection are precisely the ones that would be useful for showing there is no connection. Anyway, I wasn’t saying either of those things. I don’t have an opinion about the relationship between timed tests and math anxiety.
• Well, OK, I have a few opinions.
• A lot of people told me stories about the stress caused by timed tests. I hear you! Research isn’t the only thing that matters. We should keep telling our stories — about our children, experiences as students, what we’ve learned as teachers. True, it would be wise to hold off on the biggest and strongest proclamations (“WE KNOW THAT TIMED TESTS CAUSE MATH ANXIETY”) but just because something hasn’t been validated by research doesn’t mean that it’s not true.
• But I’m suspicious of much of what YouCubed produces precisely because they present everything as a research result, an absolute law of Brain Science. When you look closer, the research results aren’t there — which isn’t to say that I, Michael Pershan, know that timed tests don’t contribute to math anxiety. Just that there’s a difference between what one thinks and what the research says.

Do Core Knowledge Programs Work?

As for many school reforms over the past century, answering the “effectiveness” question–does it work?–is no easy task. The first major issue is answering the question of whether Core Knowledge was fully implemented in classrooms. If not completely implemented, then judging outcomes become suspect. Many of the early studies of Core Knowledge in schools were mixed, some showing higher test scores and some showing no positive effects (see herehere, here, and here). The Core Knowledge Foundation has a list of studies that they assert show positive outcomes. What is so often missing from research on reforms such as Core Knowledge are descriptions of the contextual conditions in which the reform is located and researchers saying clearly: under what conditions does this program prove effective? That is too often missing including the research on Core Knowledge schools.

The primary job of education research shouldn’t be to figure out what works, or to put it another way we shouldn’t expect the have a yes/no answer to that question. How does it work? When does it work? When it didn’t work, why didn’t it work? When it’s not used with fidelity, why wasn’t it used with fidelity? Education is not served well by the way research on program efficacy seems to frequently be done.

Did Common Core work?

Chalkbeat’s Matt Barnum does such a wonderful job reporting on edu research. Here he is, taking on a fascinating study that attempted to understand what effect the US’s Common Core standards had. Surprising results!:

How do you a study a policy as far-reaching as the Common Core, particularly one that was introduced alongside a host of other school reforms?

It’s not easy, but Song and her colleagues reasoned that some states were more affected by the switch to “college and career ready standards,” which meant Common Core in almost all cases. So they categorized states by the “rigor” of their previous standards and how similar those standards were to the Common Core.

They divided states into those more affected by the switch (because their prior standards were deemed less rigorous or less similar to the Common Core) and those less affected. Then they compared how each group’s scores changed on fourth and eighth grade NAEP tests between 2010 and 2017.

Common Core didn’t seem to help students’ scores, and over time the standards may have had an increasingly negative effect, according to the study, which has not been formally peer-reviewed.

Other researchers consulted by Chalkbeat, including Laura Hamilton of the RAND Corporation, say the study’s approach is a credible one.

“I’m not ready to conclude that the adoption of rigorous content standards is bad for student learning,” said Hamilton. “But I don’t look at this and think this looks totally wrong. It definitely looks plausible.”

Still, the approach has limitations. Most important is that the study is comparing two groups of states that adopted the standards — so if Common Core universally helped or hurt the states that adopted it, this study would miss that effect.

More, on an unpublished report:

Joshua Bleiberg, a doctoral student at Vanderbilt University, is also studying the impact of Common Core on NAEP scores. His study starts examining student scores when new standards hit classrooms, not when states decided to formally adopt the Common Core. He also excluded states that ultimately dropped the standards.

That all seems to lead to different results. In preliminary findings shared with Chalkbeat, Bleiberg finds that the Common Core had small positive effects on NAEP scores through 2013. His study has not been released publicly, so it can’t be fully examined.

“This is not going to be the type of thing that is going to turn around the whole ship really quickly,” Bleiberg said of the standards. “I would think about [the effects] as quite small.”

I don’t know what to make of all of this. I’m very worried that my internal compass has recently been getting out of whack, that I’m getting too pessimistic about any possible major change. At the same time I’m increasingly happy to talk about fundamental issues with the way things are done. (For example, that math is frequently not serving kids particularly well in high school.) But — back to the pessimism — there isn’t much to do about.

I guess I try to keep both of those ideas in mind at once, and to work in the tension between them.

Addendum: Though! Check out another piece of Matt Barnum reporting, about how reducing pollution increased test scores.

The situation seems to be that ed reform is less effective than you think it’ll be, but that improvements to quality of life tend to have more educational implications than you’d think. (The main driver of US educational improvements? GDP.)

My ed reform platform: capital improvements to schools, air conditioning in every school, better food for kids, cut pollution.

Modernism in Mathematics

Jeremy Gray makes the case (in here) that modernism applies to mathematics. His modernism consists largely of a move away from representations and towards formal approaches.

So on Lebesgue’s theory of the integral in 1903:

“The axioms specify what the integral is intended to do. They do not start from an idea that the integral is about, say area, or any other primitive concept. It is necessary to show that there is a model of these axioms, but once that is done it is at least possible to prove properties of the integral directly from the axioms and without reference to any model of them. The axioms are sometimes said to define their object implicitly, or to create it. There is no reference to a primitive concept available via abstraction from the natural world.”

And on Kronecker and Riemann:

“Neither man suggested that objects cannot be studied via their representations, but both believed that one must be vigilant to ensure that one establishes properties of the objects themselves and not the properties of merely this or that representation, and to this end it was best to avoid explicit representations whenever possible.”

I didn’t know about the Hausdorff paradox, which feels a lot like Godel. Gray’s summary: “on any plausible definition of the measure of a set there must be non-measurable sets.”

Borel ended up critiquing the use of the axiom of choice to call the paradox into question, but this was another step (apparently) in pushing people to accept that definitions of area are inherently imperfect — pushing us further away from meaning and belief in the representations.

Another interesting point from Gray: you know that thing about the unreasonable effectiveness of math? That wouldn’t have made any sense in the 19th or 18th centuries because math was coextensive with science. Like, there’s nothing surprising at all about the connection between math and the world back then, because math was an attempt to describe the world.

I’m interesting to read more, but I’m feeling as if a question has been answered. Whether we call it modernism or not, this is the time in the history of math when the connection between mathematics and the empirical world was made problematic. If we’re looking for the origins of the idea that math is “useless,” it’s going to be in this movement in mathematics between 1880 and 1920.

Things that I’d like to read: on modernity and mathematics

The world has changed immensely over the past several hundred years. Mathematics has too. Are these changes all related?

Plato’s Ghost looks like a good place to start.

Plato’s Ghost is the first book to examine the development of mathematics from 1880 to 1920 as a modernist transformation similar to those in art, literature, and music. Jeremy Gray traces the growth of mathematical modernism from its roots in problem solving and theory to its interactions with physics, philosophy, theology, psychology, and ideas about real and artificial languages. He shows how mathematics was popularized, and explains how mathematical modernism not only gave expression to the work of mathematicians and the professional image they sought to create for themselves, but how modernism also introduced deeper and ultimately unanswerable questions.

Building on Gray’s work is this presentation by Susumu Hayashi, which introduces (to me at least) the notion of “mathematical secularization.”

I also came across The Great Rift.

In their search for truth, contemporary religious believers and modern scientific investigators hold many values in common. But in their approaches, they express two fundamentally different conceptions of how to understand and represent the world. Michael E. Hobart looks for the origin of this difference in the work of Renaissance thinkers who invented a revolutionary mathematical system—relational numeracy. By creating meaning through numbers and abstract symbols rather than words, relational numeracy allowed inquisitive minds to vault beyond the constraints of language and explore the natural world with a fresh interpretive vision.

The focus is on early modernity and the shift to algebra, which is an earlier phenomenon than modernism. But maybe it’s part of the same story?

Also in the category of “is this related? maybe??,” there is a working group of philosophers that call themselves the Mathematics, Mysticism and Secularization working group.

There are all these -isms that I learned about in philosophy of math: empiricism, logicism, formalism, fictionalism (wiki). That’s part of this story too.

What I’m attracted to is the idea that math is as much a part of culture as anything else. Over the last few centuries Western society has gotten less and less comfortable with the abstract, invisible realm of religion and spirits. Wouldn’t that have an impact on how that culture thinks about that other invisible, abstract realm of mathematics?

People used to think of $x^2$ as referring just to a square’s area, but then it was emptied of that meaning. Is the break of algebra from geometry something like the break of philosophy from theology?

People used to think that mathematics was a search for ultimate truths, not just conditional ones. Are we living in a mathematically relativistic world?

Mathematicians sometimes talk — with pride! — of the uselessness of their work. Is that the end result of the sorts of processes described by these authors?

I have no clue, and I have no idea when I’ll be able to read those books. But the questions seem interesting and confusing.

Motivation

My interest in any particular piece of mathematical content varies. There are some things that I think are just absolutely fascinating, others…nah. So I think if I was, at bottom, motivated by a love of mathematics then my teaching would ultimately suffer for it.

Working with children, the future looms: I want these kids to have skills, to pass the tests, to get into schools, to get through college, to find meaningful work, to be able to see the world differently in the (hopefully) long, long time they’ll spend outside of schools. But when you take a look at what is genuinely preparatory out of what we’re supposed to teach, it’s hardly everything. People say that school math is mostly useless, and I do see what they mean.

So the answer for me is in the present, and I don’t worry too much about everything else that’s going on. The situation is that four or five times a day, a bunch of people get into a room and are supposed to study mathematics. I honestly don’t know why they’re supposed to study mathematics, but that’s just it: they are.

They are in this room, and I am also in this room and am supposed to help them learn it. If I quit tomorrow, someone else would do it, but I haven’t quit and I am in the room.

So given all this, the question for me is always, how can I do my job without making anybody feel dumb or miserable? And that gets me pretty far.

Why should mindset and purpose interventions work equally well?

This figure is from a 2015 paper, Mind-Set Interventions Are a Scalable Treatment for Academic Underachievement, and it comes out of the PERTS group, which generally does great work (as far as I as an outsider can tell).

There’s something fascinating about this study. I think, very quietly, their work represents a conceptual shift in research on mindset. The move is away from mindset and toward interventions as the main object of study.

I have Carol Dweck’s Mindset book, and it’s pretty clear that for her mindset is supposed to be a uniquely powerful force in our motivational psychology. It is the key. There really are two types of people: people who view intelligence as fixed or malleable, and this is a major factor in your motivation and subsequent success in a variety of arenas.

But check out this 2015 paper and check out that figure — there are two interventions that they tested, and only one of them has anything to do with mindset. First, the typical implicit theory of intelligence intervention:

Growth-mind-set interventions convey that intelligence can grow when students work hard on challenging tasks—and thus that struggle is an opportunity for growth, not a sign that a student is incapable of learning.

But then there’s the sense of purpose intervention which has nothing to do with the malleability of anything:

Sense-of-purpose interventions encourage students to reflect on how working hard and learning in school can help them accomplish meaningful goals beyond the self, such as contributing to their community or being examples for other people.

The theory that supports this intervention is entirely unrelated to growth mindset theory. It takes no position on whether someone thinks of human attributes as essentially fixed or malleable. If you thought that growth mindset was a hugely impactful factor that governs motivation, there’s no reason at all why you’d think a sense of purpose intervention works.

(There’s a reeaaaal cool move when the authors call both of these “academic mindset interventions” in that paper.)

And the results of this study found that both of these types of interventions worked about as well as each other. And their benefits didn’t seem to combine, which is also interesting, because why wouldn’t they, if they’re separate motivational concerns?

One possibility: people tend to be demotivated because of theory of intelligence or because of absence of purpose, but not by both. Another possibility is that demotivated people tend to be equally motivated by either intervention.

(I imagine there’s a lot of ways to sort this out with the data they’ve already collected. Which intervention works better for students assessed as having a fixed mindset?)

The second possibility — that both interventions work equally well for at-risk students — would represent a really interesting possibility, which is that the theory behind the mindset intervention doesn’t matter a ton. What if all this under the hood theory doesn’t matter a great deal? What if motivational interventions and their design is the thing worth studying, and the basic theory underlying them doesn’t matter a great deal?

If it’s true, this would make a great deal of sense to me. Dweck’s mindset theory would have not predicted that you could get the same results with an intervention like sense of purpose that uses an entirely different mechanism. (People who underwent the purpose intervention didn’t have changed beliefs about intelligence — they checked.) Mindset was supposed to be the big thing. The fact that it’s being considered as part of a menu of motivational interventions along with purpose seems significant. We’ve already moved most of the way away from seeing it as a uniquely powerful theory for explaining motivation.

And maybe the authors are saying as much in their paper. After all, it seems that now a mindset researcher doesn’t study “mindsets” at all but “mindset interventions,” which is a totally different thing.

I eagerly await something that will help clarify things. Speaking of, does anybody have a copy of this preprint? I wish I’d held on to it before it was taken down. (Update: oh, I think this is it. If so seems like sense of purpose interventions weren’t in play.)

The intellectual work that teachers can do but researchers probably can’t

[I’ve written versions of this post many times before. Here here here. Don’t read those, this version is probably better.]

Tomorrow night, I’m going to teach teachers about teaching. I think a legitimate question is, on what grounds am I claiming to know anything about teaching at all?

To be sure, I am pretty confident that I know something very important about my own teaching — in my school, with my students, in my courses, given my personality, etc. I observe my own classrooms (imperfectly) every day. The cumulative evidence of all that observation makes me pretty (not fully) sure that I’ve figured something out.

But the tricky thing about teaching is that this stuff often doesn’t translate to other situations. Just because something works in my classroom (according to me) doesn’t mean that it’ll work in vastly different contexts. To get really specific about this for a second, I teach students who are among the wealthiest children in America. This reality impacts my school in a bajillion ways. Who says that my dumb ideas about feedback will mean anything to the other teachers in my department. Teachers in other schools, and especially high-poverty schools? Forget about it.

(To be fair to myself for a second: I haven’t only taught in my school.)

The point is that there are obvious reasons to doubt that the things I think I know are really truths of teaching. This is even true if we move past the particular practices that I advocate and get behind the thinking and values that support those practices. I think I have a useful way of thinking about teaching, or I think I’ve identified some value that is important for the student experience. Who says that this is anything but my own thinking?

This is the natural place that research on teaching enters the conversation. Whatever you want to say about research, it’s not about my classroom. In general, it’s about forming generalizations in a way that improves upon (e.g.) my ability to make stuff up about my teaching.*

There are lots of interesting edge cases to consider, but I think the generalization about generalizations stands. Researchers might write cases grounded in particulars or engage in a teaching experiment, but the point of those is to contribute to the formation of generalizations that are broadly useful.

This is getting pretty abstract so let me just get to the point: could researchers ever respect the generalizations that teachers make about teaching as knowledge that stands on par with their own?

The usual way of talking about teacher/researcher parity is to say that researchers excel at making generalizations, while teachers contribute crucial local knowledge. And it’s totally true that teachers do have local knowledge.

But does this really create parity between teachers and researchers? The whole point of broadly useful knowledge about teaching is that it goes beyond local knowledge — it makes a generalization. If what teachers can contribute is local knowledge, then I think we’re just saying that teachers are at best a source of data to the researcher. The teacher inputs local knowledge, the researcher generates broadly useful generalizations.

It’s true that there’s no reason to inherently value general vs. local knowledge, so in a certain sense there can be parity between teachers/researchers. But at the end of the day, what’s broadly useful are generalizations, and teacher knowledge can’t really compete with what researchers contribute.

Or…can we?

I want to speculate a bit about some different ways of sorting out the relationship between what researchers and teachers can contribute. To start, I want to ignore local knowledge for a second and talk about how teachers contribute to generalizations, i.e. researcher turf.

I have a few ideas here, and they’re very rough, so bear with me.

First, researchers are institutionalized while teachers are necessarily amateurs at producing generalizations. The relationship between teachers/researchers can then be folded into the general relationship between amateurs and experts. And, of course, we need experts. But the ecosystem isn’t healthy if it’s entirely populated by experts.

Amateurs play a lot of important roles, even when it comes to forming generalizations that are broadly useful. Here are a few that I’ve read about (too lazy to cite right now):

• Amateurs can disrespect the boundaries of fields or sub-professions and put together ideas that from an institutional perspective are incongruous
• Amateurism is in general lower stakes/lower payout than being an expert. If I’m an amateur and my ideas are wrong or useless, my career isn’t on the line. So there’s a way in which amateurs can attend to riskier ideas, or work on lines of thought that are perceived to be less rich in reward or are in general undervalued.
• Amateurs play an important role in teaching and spreading expert generalizations, but in doing so amateurs often simplify or otherwise improve the results of experts in significant ways.

But this way of framing things — teachers as amateurs, researchers as experts — doesn’t really leave room for teachers to ever get institutional respect from experts as generators of generalizations about teaching, and two further points on that:

• this is probably true
• this is much more exciting to me than institutional respect

It’s not good for my \$\$\$, but I am really quite fine accepting the role of an amateur in all this. It’s exciting to try to smash fields together and to not be beholden to conventional wisdom in the field. I can chase ideas about teaching, throw them out there for others and see what resonates for others. We make up the rules as we go. It’s fun!

That’s the spirit in which I’m going to teach this class tomorrow night. I’ve got this stuff I’ve figured out about teaching. I don’t want to make myself sound like a tin-hat Alex Jones-type, but I do think that what I’ve learned about teaching goes against a certain conventional, institutional, expert way of thinking. And it is the result of mashing up a bunch of things — trial-and-error in the classroom, reading research, experimenting by giving presentations to teachers. And if it’s not broadly useful as a generalization about teaching? Hey, that’s OK too. There’s very little at stake here.

So, I’m not an expert, and neither are you: maybe these ideas are useful to you? Let’s find out. That’s the way I approach this stuff right now, as an amateur.

What is retrieval practice when you’re learning math?

I’ve never really carefully read the retrieval practice literature, but I think it gets confusing when people talk about retrieval practice when talking about math skills, as opposed to mathematical facts.

Here is the description from @poojaagarwal‘s website, committed to promoting retrieval practice among practitioners:

Retrieval practice is a strategy in which calling information to mind subsequently enhances and boosts learning. Deliberately recalling information forces us to pull our knowledge “out” and examine what we know. For instance, I might have thought that I knew who the fourth U.S. President was, but I can’t be sure unless I try to come up with the answer myself (it was James Madison).

But how does this apply to math skills? Can trying a problem (i.e. practicing the skill) ever count as retrieval practice? Does it make sense to use the metaphor of ‘calling information to mind’ to describe what’s going with skills practice?

I think not. But I also am finding retrieval practice useful in my lesson planning. There is a great deal of knowledge that is useful for students to know when they’re learning something new. This sort of knowledge is the sort of thing that I’d like my students to know (i.e. retrieve from memory), more than I’d like them to derive.

Often, at the beginning of class, the first thing I ask my students to do is to remember some facts that they may (or may not yet) know from memory. Some constraints:

• I don’t ask students to solve a problem and call it retrieval practice — that’s skills practice, not retrieval practice, and tickles other parts of the mind.
• I only ask students questions that I think they could remember, even if it might be difficult to recall these things. Ideally, these would be things that either students could derive if they can’t recall them.
• Because stuff from the last few days of class can often get forgotten really quickly, I often use these prompts to strengthen the memory of what we’ve recently done. (The prompt “Summarize what we did yesterday” is surprisingly difficult!)

Here are some prompts I’ve recently used with students:

“Draw a pair of ramps that are pretty close to being of equal steepness.”

“Write an equation of a quadratic, describe what it would look like.”

“What happens when you use the tan button on the calculator? Give some examples.”

“Write several pairs of decimals, and write the number that is between them.”

The truest ‘retrieval practice’ of these is the one about the tan button. Next in line is the one about the equation of the quadratic, since I’m prompting kids to remember what the features of the graph are (though it’s also skills practice). What made me think about these as retrieval practice is that they were all calling back on the previous day’s class.

Here are some purer examples of retrieval practice prompts in math:

“What’s the Pythagorean Theorem?”

(If a specific procedure is supposed to be known for converting a decimal into a fraction:) “How do you convert a decimal into a fraction?”

etc.

As I’m messing around in graph theory, I’m noticing that there are a lot of things that would be useful to remember — particular proofs that could serve as paradigms, constraints (in the form of inequalities) on possible planar or non-planar graphs, theorems, specific graphs that are useful examples, etc. If I had a teacher of graph theory, I’d want that teacher to prompt me to remember these things so that I could have more of them available as resources when I’m trying to learn something new or do some creative proving or problem solving.

(I should probably bust out some flash cards at some point…)

As an aside, I think that retrieval practice is sometimes mixed up with spaced practice, but I think these are different things. Spaced practice might be a better fit for what people are describing when they talk about intentionally building time-separated practice of skills into their courses and assignments. I think this requires a different sort of finesse than retrieval practice, though, as the problem with spaced practice is making sure students have something productive to do if they’ve actually forgotten the material.