Every once in a while people send me books. That’s very nice of those people, I always appreciate it, it’s a cool way to stay in touch with what’s going on. But I feel like I should acknowledge receipt in a clearer way. So, here are three books that I recently received and have started to take a look at, along with something nice I’ll say about each of them.

Tools for Teachers, Oliver Lovell: Oliver’s a great guy and he runs a really nice podcast about education research. He had me on to talk about worked examples back when my own book came out, and it was just a very deep conversation, much deeper than most conversations about pedagogy ever tend to be.

“This book summarises the most useful techniquest, strategies, and mental models from sixty in-depth conversations.” One cool thing about the book is that Ollie creates a giant diagram synthesizing all the things he’s learned. It’s an attempt to show how it all hangs together. Impressive and ambitious! It also features an appendix titled “Michael Pershan’s Worked Example Routine,” and I’m proud to be part of Oliver’s teaching journey.

Painless Statistics, Patrick Honner: Thankfully, I don’t have to prepare students for the Regents exams any more. When I did, books like Patrick’s were crucial for the kids. We need high quality study materials out there in the world, and I feel secure knowing that Patrick — who also writes for Quanta Magazine — is writing these things.

The book ends with a fun chapter titled Statistical Literacy that features excellent examples, including a COVID vaccine trial, a certain “large city” considering a congestion pricing plan (god help nyc), and a brief but clear explanation of p-hacking. There are practice problems, solutions, the whole shebang.

Figuring Out Fluency In Mathematics, Jennifer M. Bay-Williams, John J. SanGiovanni: These types of elementary mathematics texts often don’t have much to say about worked examples, but this one does! “Even though the focus is on someone else’s work, worked examples help students notice their own misconceptions or errors, especially when tasks include self-expalantion prompts. The notion that the work comes from a stranger alleviates anxiety and embarrassment that can come about when sharing with classmates. Worked examples offer the opportunity to inject an idea that hasn’t come about during instruction or other practice opportunities. Like any practice resource, they are flexible and adaptable.”

I also noticed some nice fluency games. I always need more nice fluency games.

The hard thing about talking about college admissions in the United States is that the facts don’t matter and the system is insane. The system is a patchwork of private and public colleges, some that admit basically everybody and others that reject 96% of applicants. Private colleges get to determine their own admissions criteria, money makes a big difference, and if you are extremely good at football you get to go wherever you want. There is a magazine nobody reads that ranks schools and everyone — especially the colleges — cares deeply about it. On top of all this, nobody can agree on what the point of college admissions is, anyway, so it’s unclear what we’re even trying to optimize for.

It’s so confusing to me. Who should get into Princeton? I don’t know, who gets to work for Goldman Sachs? Anyway, the vast majority of schools are nothing like Princeton. So: who gets to go to UC or SUNY? I don’t know, it all seems to depend on what you think these schools are supposed to be doing. Are they trying to reward excellence or help whoever they can? What are we doing here?

That said, the two most important factors for college admissions in the US are high school grades and standardized test scores. Some people think this is bad, and that test scores shouldn’t be used. Other people think grades are bad and should not be used. Most people — including the testing companies themselves — seem to think you should use both, because they’re both a little bit bad and a little bit useful.

In her book “Who Gets In?” Rebecca Zwick compares different admissions criteria, attempting to clarify the choices institutions face about selection. Zwick is a researcher who was a professor at UC Santa Barbara and worked for Educational Testing Services, an enormous testing company. You will not be shocked to learn that she likes tests. (She is not a fan of test-optional policies, for instance.) And though I find her book frustrating, as I think she handles the social and moral issues in a narrow and unenlightening way, she has a nice little chapter where she compares the pros/cons of grades and standardized tests. I want to summarize that chapter in this post.

• Your grades in high school do a good job predicting your grades in college. “High school grades have typically been found to be a somewhat better predictor than test scores,” Zwick writes. This is widely seen as significant for admissions decisions, a sign that a kid can succeed at your institution. It’s worth thinking hard about whether that makes sense, but lots of schools think it does.
• Grades are less correlated with socioeconomic status than test scores. Zwick and a colleague studied this and found less of a correlation than other people do. (They looked at the correlation between SES and scores/grades within a school, whereas most people just combine all the data.) Zwick emphasizes that this reduces the correlation, but not all the way: “When we calculated the student-level correlations within high schools and then took the avedrage, we found that the association with SES ranged from .20 to .25 for SAT scores, from .12 to .18 for high school GPA, and from .15 to .21 for class rank.”
• Grades measure more than just knowledge. “Grades reflect ‘scholastic engagement,’ which includes characteristics we might call industriousness, conscientiousness, and perseverence.” If you want that information, grades reflect it.
• Grades measure the knowledge that schools are attempting to teach. The test and the school may not agree on what students are expected to know. Quoting another ETS researcher Warren Willingham: “Individual assessment by the teacher can be based more closely on the specific material that each student has studied and what each has learned that is possible with a standards-based test.”
• Grades better predict who will get further credentials after college. “Perhaps the most interesting finding concerns the percentages of students who completed a postbaccalaureate credential by 2013, nine years after high school graduation. The vast majority of the credentials received were either master’s degrees or “professional doctorates,” which included degrees in law, medicine, and dentistry.”

But what are the problems with grades, according to Zwick?

• Grades collapse towards the top. Too many people have 4.0s to make it useful for elite admissions. “In 2015 an astonishing 37,200 applicants to UC Berkeley had high school GPAs of at least 4.0,” she notes. “They are not very useful for making distinctions at the high end of the scale.”
• Grades aren’t standardized and some schools grade harder than others. “High school grades can lead to systematic prediction errors because of the differences in grading standards across schools. A student with a high school GPA of 3.0 from a top-notch high school may be well prepared for college, whereas a student with the same GPA from an inferior school may not be.” I don’t like the way she talks about superior/inferior schools, but I understand what she’s talking about. It is possible for adjust GPAs to compensate for this, and some schools do that.
• Grades are biased by extraneous factors. “Studies and information observations involving college students have suggested that grades can be influenced by the student’s physical attractiveness and charm. Some research on younger students has found that boys receive lower grades than girls because of classroom misbehavior. A 2014 study of high school students found that young women and students with limited English proficiency tended to earn higher grades than other students, after taking test scores and school characteristics into consideration, whie low-income students earned lower grades,” Zwick writes. Just to editorialize for a moment: grades are obviously biased, I don’t dispute that, but Zwick’s argument here is fairly sketchy. Besides, how bad can the bias be if grades do better than tests at predicting college performance? But as a person who has given grades, yes, they are influenced by all sorts of things.
• Grades are less precise. “One way to think about this is in terms of signal-to-noise ratio…Because they are affected by many nonacademic factors, including the teacher’s stringency and the student’s attitudes and classroom behavior, grades are relatively noisy measures.”

Each of these flaws with grades is itself a point in favor of tests:

• Tests are more precise and are designed to discriminate between students, even at the high-end of achievement. “[Tests] can help to distinguish among the many applicants to elite colleges who share a GPA of 4.0 or above.”
• The test are standardized. “Test scores can also provide a means of evaluating home-schooled and international candidates.”
• Tests are not biased by non-academic factors. “An advantage of admissions tests is their ability to provide efficient comparisons of students whose high school grades are influenced by the schools they attend, the classes they chose, their rapport with their teachers, and many other extraneous factors.”
• Tests can identify talented students who have not succeeded in school. “As test proponents like to point out, admissions tests can sometimes serve to identify talented students with weak academic backgrounds — those who used to be called “diamonds in the rough” before that label came to be seen as disparaging.” (Editorializing: I was not aware that was a disparaging thing to say.)
• When tests and grades are combined they do a better job of predicing college grades. Though high school GPA is better at predicting college performance than test scores, using both is better still at predicting college grades. This is why testing companies recommend using both grades and scores to make admissions decisions.

So, what to make of all this? If the thing you care about is how well a student will perform in college, I think there’s a very clear case that both high school grades and tests are imperfect measures of that. Grades, on their own, are better than tests, but grades also have real drawbacks. Tests, however, are also flawed. Put them together and you get something with complementary flaws and benefits.

Of course, maybe you don’t care about predicting college grades! Maybe you just want reward people who did well in school, and you don’t care if grades incorporate behavior. Or maybe you just think that college is stupid and college grades don’t mean anything either, so you don’t care about that. Maybe what you value is a precise standardized competition for a scarce resource and so you like standardized tests because that’s what they are. Maybe you care about social mobility so you want to err on the side of accepting poor students into college, no matter their grades or tests. None of the above can decide between these options, the facts don’t matter when there are all these values to hash out.

But my point, the reason I think this matters, is to remember that none of this is certain or clear. Grades can be biased, yes, I think we all know this. And yet it’s not at all clear that tests do a better job. Even the ETS employee says so. And I think everybody involved in education should remember that we are working with flawed tools that do an imperfect job of measuring the things that most of us care about — and it’s not even clear what it is that we care about! What can you do? That’s education for you. It’s not a science. So don’t fall in love with the tools, acknowledge that they’re flawed, and then use them anyway if they’re the best that we have to work with.