Great Buys, Good Buys, and Bad Buys by Joel Samoff

In this NORRAG Highlights, Joel Samoff, Research Associate at the Faculty of Education, University of Johannesburg looks at the World Bank’s business of providing policy advice to poor countries and specifically investigates the WB’s Global Education Evidence Advisory Panel (GEEAP) first report, Cost-Effective Approaches to Improve Global Learning and highlight three problematic “threads” within its methodology.

Great Buys, Good Buys, and Bad Buys. The World Bank’s and UK Aid’s new advisory panel on education evidence offers policy and programmatic advice to education ministries in poor countries. Advice based on evidence is a good thing. What we find in this recent report, however, is not a well grounded review of relevance evidence but a frayed garment knit with three weak threads.

For many years the World Bank has claimed that its main role is to provide policy advice, not to lend money. Of course, it does both. With the money, and often without, comes the advice.

Major education funding and technical assistance agencies created the Building Evidence in Education (BE2) working group in 2012. Within that umbrella the World Bank recently created the Global Education Evidence Advisory Panel (GEEAP), researchers charged to review studies and “help policy makers make sense of the evidence.” GEEAP’s first report, Cost-Effective Approaches to Improve Global Learning (October 2020) reviews research and provides action advice.

The report’s phraseology is catchy. Great Buys are cost-effective practices well supported by evidence. Good Buys are a good idea, but the evidence is weaker. Bad buys are practices that do not work and are certainly not cost-effective. A clever way to present things. Shopping is familiar in our consumer world. But the wrapping obscures what is inside the package.

GEEAP and its report reflect the union of three threads, each problematic in several ways. First, we find, once again, the search for what are usually termed “best practices,” now in research as well as in school. Second, what is considered to be high quality scholarship turns out to be quasi-experiments, with a preference for randomized controlled trials. And third, education is treated as a commodity, to be bought and sold in a (very imperfect) market place.

Think about “best practices.” The common approach is to see which education activities are associated with high exam scores (we will return to that logic shortly) and then recommend that they be adopted elsewhere. Or everywhere. Yet, however reasonable that search for best practices seems, implementing it puts education quality at risk. There are several sub-threads here.

One concerns the oft-repeated claims that knowledge is universal and that education standards can readily be generalized. Yet, at their roots learning objectives and learning practices are locally contingent and negotiated. Communities regularly decide what they want their members to learn and how that is to be done. They may be and generally are influenced by what happens elsewhere. But most often they are also responsive to the community’s (or its powerful constituents’) perceived needs and interests. In everyday language, there are not and cannot be universal best practices. It is communities, not experts, who decide what is best.

The notion that best practices are universal is problematic in another respect. Except in the most authoritarian settings, learning objectives are negotiated. Sometimes that is a very public and inclusive process. Other times that negotiation is shrouded and obscures the power relationships. The search for best practices assumes that guiding an education system is largely a technical process best left to the relevant experts. That assumption may contribute to the legitimacy and public acceptance of decisions. In practice, of course, making public policy is never technical. Those who specify objectives, curriculum, and pedagogy are attentive to interests, alliances, and power. Except in the most authoritarian societies, what is regarded as high quality education is continually re-defined and re-specified. In that sense, what is “best” or “good” or “poor” changes periodically and will continue to change. Ultimately, “best” is always ephemeral.

The notion of universal best practices also confuses where learning happens. Whether the focus is cognitive achievement, mastery of specified skills, or the development of desired values and behaviors, learning is at its core an interactive, relational, face-to-face process. System level observations drawn from distant experiences do not translate easily into local face-to-face contact practices. Where that translation process is ignored (most common) or managed poorly, learning objectives are less likely to be met. Ultimately, “best” is necessarily local.

Of course we can learn by observing what others do. But there are not and cannot be universal best practices, detached from their context. Plants that flower on a cool hillside wilt on a dry sandy plain.

A second problematic thread in the GEEAP approach is the notion that the strongest evidence is produced by quasi-experiments, efforts to reproduce in the field the model of what is done in a carefully managed laboratory. We all understand the experiment that has students see the steam that emerges when heat is applied to water. The experimenter reminds us that other important influences must be controlled. Heating water in a pressure cooker leaves it still water. For education research, “quasi” because fully controlled experiments are generally not possible in schools and classrooms. And there we have the standard model that is to be applied to determine what works in education.

Applying that model has several components. One is a largely linear notion of causality that has clear and distinct independent and dependent variables and that controls other influences. That approach marginalizes alternative research strategies.

The application of that narrow notion of causality is also problematic. How are we to know what works in education? Well, we are to see what leads to (that is, “causes”) better results. If students in a class with improved textbooks get higher examination scores than a comparable group of students—recall, this is a quasi-experiment—with the old textbooks, then we can conclude that better textbooks are worth the cost. Notice heroic assumptions in this explanation. If the classes are comparable, then we can ignore all other influences on examination scores. If the school system uses that examination, we can assume it is a reasonable measure of learning. All other measures of learning can be ignored. Even more dramatic, we can label as learning what may be students’ ability to recall statements by their teacher or sentences in the textbook. The focus on results pushes to the side, or perhaps out the window, social and emotional learning (ability to cooperate and collaborate and to plan, becoming self-confident and self-reliant) and preparation for citizenship. And more. Thus, the application of a narrow notion of cause leads to using a very limited notion of information recall as a proxy for learning. That legitimizes ignoring much of the learning process.

In this narrow notion of how to understand cause, large scale sample surveys are preferred over studies that rely on detailed observation, ethnographic investigation, and following learners through and after school. Other research strategies are rarely considered and regularly dismissed as not research at all, for example action research intended to support a social transformation objective (say, reducing discrimination against students from a racial, ethnic, or religious minority) and legal action to compel disclosure of critical information (say, details of school or district education spending).

Associated with that notion of causality is the presumption is that accuracy and reliability are achieved by counting. Ignored is observation that “Not everything that counts can be counted, and not everything that can be counted counts” (often attributed to Einstein, but author unknown, perhaps a collective insight). It is not quantification that is problematic but the uncritical adoption of the assumption that quantification increases reliability.

The era of big data has reinforced the claims that empirical review renders theory unnecessary and that objectivity can be achieved by “letting the data speak for itself.” However massive our data collection and however sophisticated our analytic tools, the assembled information never speaks for itself. Researchers always make choices about what to collect based on notions of what matters.

Combined, the narrow understanding of causality and the insistence on counting discard important tools for understanding education as process, not outcomes. So, GEEAP tells us what are the best buys by discounting much of the evidence.

For most learners, it is how education happens that carries them through their lives, not specific outcomes. Understanding education as a process requires attention to interactions and recognizes that causes can be effects and effects causes (rather than clear and distinct independent and dependent variables).

Where the concern is changes in outputs achieved by modifying inputs, there is no need to pay attention to the learning process, that is, to how, exactly, teachers and learners have changed their behavior as a result of improved textbooks or smaller classes, or how, exactly, instructional materials and class size have affected examination scores. In GEEAP’s approach, learning as process is not only messy, but a distraction. As well, we are led to think of quality as high exam marks, with no attention to equity. And no attention to what will ultimately be most important to students, learning how to learn.

The third problematic thread is the framing: education as a commodity. Why does that matter?

Without saying so, this approach summarily discards the idea of education as a right or as a public good. Instead, education is a commodity, to be understood through the lens of market, vendors, and purchasers. Note the sharp differences in those two understandings of education.

Education as a public good focuses on broad responsibility and accountability, on cooperation and collaboration, on widely shared funding, and on institutions and policies to manage the divergent interests that arise in society. Education as a commodity focuses on the price of access, on alternative education services on offer (assuming a link between quality and price), on the availability of information on alternatives and prices, and on individual interest, benefit, and resources.

Where education is a commodity, a school belongs not to a community but to a vendor, with private and public sellers to be treated the same. Accountability and redress are commercial, not political. The tools developed for markets and sales are deemed appropriate to make decisions about education policies and programs. Even where there is government oversight and regulation, both the provision of education and the uses of that provision are privatized.

The Buys phraseology is enticing but ultimately distracting and misleading. We all shop. We all face uncertainty when we try to decide which products will serve us best. We all look for products that do well at lower cost (“cost-effective”). But when we take education as a package to purchase, we lose sight of our role, and teachers’ and learners’ roles, in deciding about objectives and curriculum. Indeed, most marketed packages marginalize teachers and treat students as objects of interventions rather than learners with initiative, imagination, and responsibility. Embedded in the purchase metaphor are the ideas that private is better than public and that when the public schools are broken, rather than fixing them, we should shift to private provision. As we use the tools of business to manage education, we pay more attention to cost-effective than to learning-effective. What captures our attention are investments that will pay off, not schooling that will reduce inequalities or empower learners. We focus on the apparent quality of the package rather than how it is used. Without realizing it, we are paying for examination scores rather than learning.

So, GEEAP says, look for Best Buys. We will tell you which education products are effective and signal our high confidence with the Best Buy marker. Find them, purchase them, and put them to work.

So, education ministers should go to the big box store, find the education section, locate the teachers aisle, and purchase pre-service teacher education packages that are marked with five stars.

Following this path distracts us from things that matter. The right to education disappears entirely. Most important, great buys and market prices become how we see the world. Within that frame, education as a public good, subject to public decisions and public accountability, simply makes no sense. The more we focus on the results and their prices, the less we see how our thoughts are shaped and who decides what matters.

We must recognize the big bad buy—GEEAP and its report—when we see it, understand its defects, and resist the temptations of its catchy marketing.

About the author: Joel Samoff is Research Associate at Faculty of Education, University of Johannesburg. At Stanford University since 1980, he has been a faculty member at the Universities of California (Los Angeles; Santa Barbara), Michigan, and Zambia and has taught in Mexico, South Africa, Spain, Sweden, Tanzania, and Zimbabwe. His research interests are understanding how people organize themselves to transform their communities, and he studies the links among research, public policy, and foreign aid.