On Grading States

Did you care about your grades in school?  Maybe yes, maybe no.  Perhaps you thought that your own personal learning and experience was more important than some arbitrary number some outside entity assigned you.  Or perhaps you realized that regardless of how your grades were obtained, it mattered because others (parents, peers, universities) believed them to matter.

Recently a number of IR scholars have been pursuing a similar line of inquiry, but for states rather than schoolkids: do governments care about the grades that they get from outside observers?  There are, after all, an extraordinary number of international actors that periodically rank countries all around the world on a wide range of criteria, from how free they are, to how easy it is to do business in them, to how happy they are, to the degree they can be considered a “failed state,” or even on how well they provide their citizens with iodine.

Do these rankings matter at all?  Do governments change their behavior in response to receiving either good or bad grades?  An excellent piece recently published in the Washington Post by Duke University political scientist Judith Kelley provides a quick literature review of the emerging consensus on what is being called “Scorecard Diplomacy.”  I encourage anyone interested in this topic to read it in full, but the main bits are probably these:

Why should states, or anyone else, care about scorecards? First of all, they are easier to understand and digest than complicated policy reports. Instead of emphasizing detailed data, they sort countries into categories (e.g., countries that are succeeding vs. countries that are failing), or rank them with some score, showing which countries are at the top and at the bottom. These categories and rankings are framed to pressure the countries being ranked. For example, if your country is at the bottom of a well-respected scorecard for “Ease of Doing Business,” you might find that international businesses start to avoid investing in your economy.  […]  My recent book on the TIP report on human trafficking explains what I call the “cycle of scorecard diplomacy.” The TIP report doesn’t just rank countries. Producing the report involves U.S. diplomats on the ground engaging with governments year-round and orchestrating indirect pressure by media and civil society.  […]  Because countries are rated again and again, they have an incentive to improve their behavior in the hopes of boosting future grades. As a result, states pay more sustained attention to an issue than they would do if they were just shamed in an ad hoc way.  Beth Simmons and I have shown that countries criminalize human trafficking more quickly when they are included in the report, get worse grades or see their grades drop. My work on TIP shows that this is not just because countries fear being sanctioned. The stigma of the scorecard makes states change their behavior. Countries that criminalize trafficking also work harder on related efforts to fight the problem. In many countries, the TIP report has led states to set up new institutions, to train judges and police, to improve shelters, and to increase trafficking prosecutions and convictions. Thus scorecards can prompt real changes.

In addition to the evidence that Kelley, Simmons, and others have found, my own work going through U.S. diplomatic cables confirms that government officials around the world really do care about the results of external benchmarking.  For instance, in November 2009 the Tanzanian president told visiting American diplomats that following a poor outing in the World Bank’s Doing Business report that year, he had set up an inter-agency team to respond to the shortcomings in Tanzania’s business environment the report had called out.

Taking a step back, the bigger picture here is that International Relations as a discipline is increasingly cognizant of the fact that states are not the hyper-rational, soulless, emotionally-stunted creatures that many scholars depicted them as during the realist and rationalist heydays of the 1980s and 1990s.  States are disembodied, corporate actors, yes, but they are ultimately comprised of individual human beings and often respond in typically human ways to circumstances.  For instance, states can be shamed; states avoid actions that will cause them cognitive dissonance, even if they are in the state’s material interest.  States “puzzle about problems” and learn from one another.  In short, states are “social.”  So maybe it’s actually not that surprising that they do care about their grades.

If No One is Going to Read Something, Should You Still Write It?

Now that I’ve wrapped up my dissertation and am starting to think about how I might go about turning it into a book or some articles (or ideally both!), I’ve been giving some thought to the relationship between writing and reading.  I have many close friends in journalism, and for them the relationship is rather straightforward: you write in order to be read and, for the most part, the wider the readership, the better.

For academics in the humanities and social sciences… it’s complicated.  Yes, the ultimate goal is for our work to be read (and perhaps even have an impact on the wider world), but the intended audience is usually much more limited.  I’ll be lucky if my dissertation is read in whole or in part by more than a dozen people, and I’m sure many are familiar with the oft-heard claim that the median academic publication is never cited (for discussions, see here, here, and here).

But even if academic writing isn’t always widely disseminated, I think it has great value as a way of making the writer lay out his/her thoughts clearly and intelligibly.  Academic writing (and particularly a humanities/social sciences dissertation) can be a form of writing qua learning.  The messy realities of wrestling words and ideas into coherent shape on a page show up in an academic’s teaching, punditry, and activism for years down the line.

Which got me to thinking–are there other forms of writing which are not primarily intended to be read?  And the answer that leapt to mind are a lot of major governmental reports.  Sure, every now and then governmental organizations produce big reports that are widely read and discussed (and can even get on the NYT‘s bestseller list), but they are the exception, not the rule.  And even for the exceptionally well-publicized reports, like the recently released Chilcot report on the U.K’s involvement in the 2003 Iraq War, it’s doubtful that it’s primarily intended to be read, given that it’s 2.6 million words long, or about three times the length of the complete works of Shakespeare.

So then why are these reports written?  Why is all of this energy expended writing something that probably will not be read all that much or all that closely?  One possible answer is that bureaucratic writing may have a documentary function – the act of writing the report serves as evidence, independent of whether it’s read or not.  In other words, sometimes bureaucratic reports are written just because they have to be, in the same way that sometimes meetings are held just because they have to be, even if no one else is present in the room.

Maybe that’s why long, unreadable government reports get written, but critically-minded scholars like James Ferguson and Isaac Kamola have convincingly argued government texts almost always have a secondary role beyond simply fulfilling bureaucratic requirements.  Governmental and para-governmental reports are also always political interventions!  For instance, in my dissertation I discuss 14 high-status reports about East African regional integration that received funding from European sources between 2007 and 2014.  I note that while the reports typically present themselves as having technical, problem-solving aims, they also have the secondary political function of raising the profile of certain contested issues and forcing East African governments to cease foot-dragging and implement policy changes.  Sure, maybe they’re read only by a few dozen people, but that intended target audience will feel the message loud and clear.  We should not be fooled by the generic formats or the routine list of acronyms on the inside cover—intended or not, these reports are interjections into the political economy of East Africa.

Given this discussion, it’s particularly interesting that one organization which takes seriously the question of which of its reports actually get read is the World Bank (to its credit):

About 13 percent of [the WB’s] policy reports were downloaded at least 250 times while more than 31 percent of policy reports are never downloaded.  Almost 87 percent of policy reports were never cited.
If I can’t figure out how to turn my dissertation into a book or articles, it might nevertheless have a future as a World Bank policy report!