A new option to study education options

June 21, 2018

Nirav Mehta, Assistant Professor, Department of Economics

Story by Rob Rombouts

Nirav Mehta, Assistant Professor in the Department of Economics, is interested in studying the effectiveness of different approaches to education, but ran into difficulty with the standard approach to studying the topic.

“Experiments can answer a very narrow set of questions very well, but they are expensive,” said Mehta. “In themselves, experiments generally only tell you about the effects of specific interventions.”

To address this, Mehta developed and estimated economic models to consider education policy. Mehta applied the approach to two issues: charter school policy, and the impact of ability tracking on students.

In the first case, Mehta considered whether flexibility of choice in charter schools is good for students.

“The modal existing way to figure out whether an outcome is better or not is to try to get random assignment,” said Mehta. “However, random assignment to charter schools doesn’t exist in the real world, which forces researchers pursuing this empirical approach to study oversubscribed charter schools, which sometimes admit students by lottery. However, lottery-based studies are based on a deep conceptual flaw. You can only measure the effects of charter schools which people want to attend.”

Due to this, Mehta said studies of charter schools based on experiments do not have external validity. “The studies do not tell you how all charter schools are doing; they only tell you the effect of charter schools that are over-subscribed,” said Mehta.

In his approach, Mehta focused on developing frameworks to allow wider extrapolation than the most common empirical approach. The framework he developed allowed him to measure the average effect of charter schools over all students who attended charter schools, over a counterfactual where there was no charter school.

Through the model, Mehta estimated that children who attend charter schools on average perform 10 to 12 per cent of a standard deviation better than students in public school.

However, this average effect masks considerable variation in how students are affected.

“There is huge variation in charter schools, so some do worse,” said Mehta. “If you only measured charter schools in high demand, it would overestimate the effectiveness of charter schools.”

In his ability tracking study, Mehta was considering whether dividing students based on ability level helps or hurts overall performance. When students are grouped by reading ability, students with lower-level reading skills do not interact with students with higher-level reading skills.

Experiments on the topic often did not consider additional inputs parents may provide if peer interactions change due to ability tracking. Moreover, experiments can only reveal treatment effects of particular tracking configurations — those chosen by the experimenters.

Mehta’s paper was the first to take into account equilibrium interactions, or changes parents might make based on different groups. The model he and his co-author developed showed that ability tracking increased inequality in the classroom.

“Ability tracking gives resources to better-off parents, or parents of better readers,” said Mehta, “and takes resources away from poorer parents.”

“There are many questions that experiments can answer for themselves, but without further structure, you can’t understand the many crucial policy implications,” said Mehta.