World Bank criticises World Bank Research!!

World Bank (or more specifically DEC, its Development Economics Group) publishes countless research studies, reports and papers, which influence development thinking and developmental policies of countries.

A couple of months back, in December 2006, it released one more of its research reports. Titled "Evaluation of World Bank Research: 1998-2005", this report has not received as much publicity as it deserved (though The Financial Times did characterize the 165 page report as “a scathing critique” of the Bank’s selective use of unscientific data to advocate policies and projects that are ideologically in vogue, such as pension reform, aid effectiveness, and poverty mapping).

This study, commissioned by World Bank, was conducted by a team of consisting of a panel of 24 established researchers (from places like London School of Ecnomics, MIT, Oxford, Universities of Columbia, Yale, Harvard, MIT, etc.), and was led by Profs Abhijit Banerjee (MIT), Angus Deaton (Princeton University), Nora Lustig (UNDP) and Ken Rogoff (Harvard University).

While the report is not all negative - in fact, it also lauds some of WB's reports and also empathises the constraints of doing research on complex topics - here are some excerpts which are quite revealing:

About the Quality of Research:

"...the panel had substantial criticisms of the way that this research was used to proselytize on behalf of Bank policy, often without taking a balanced view of the evidence, and without expressing appropriate skepticism. Internal research that was favorable to Bank positions was given great prominence, and unfavorable research ignored. There were similar criticisms of the Bank’s work on pensions, which produced a great deal that was useful, but where balance was lost in favor of advocacy. In these cases, we believe that there was a serious failure of the checks and balances that should separate advocacy and research."

"...evaluators generally found that Bank research was well-targeted towards important topics, but was often weak on execution and technique. While it is desirable for Bank technique to be behind the frontier, there has often been too large a gap. Some technically-flawed projects have run for years, and have been incorporated into country work without appropriate certification and review."

"Evaluators also noted that a high proportion of the citations in this group of papers are to other Bank papers, many of them unpublished. In some cases, where groups are almost entirely inward looking, the degree of self-reference rises almost to the level of parody."

"There is much selection of evidence, with obscure, sometimes unpublished, studies with the "right" message given prominence over better and often better-known studies that come to the "wrong" conclusion."

About Conditions under which Research is done:

"They (the WB resarchers) tend to be more academic than policy oriented, the technical execution tends to be weak, and although they nearly always contain policy conclusions, the conclusions are rarely well based on the preceding analysis. We suspect that the relentless pressure to give every paper a policy conclusion, whether or not it actually has one, is largely responsible, though it was not always clear that researchers understood the limitations of their work, let alone communicated it to their readers."

The report quotes from a 1997 paper - "The World Bank as "Intellectual Actor"" - by Nicholas Stern (before he became the Vice President for Research at WB) and Francisco Ferreira about the difficulties of doing research in World Bank:

"Researchers are not free to follow intellectual inspiration. They are under constraints of designated priorities and of an apparent need to be immediately useful to operations. Further there is the strong hierarchy and an atmosphere much more deferential then would be found in universities. Among researchers there is considerable concern with what superiors will think of conclusions reached, to the occasional detriment of whether an analysis is sound."

...and then goes on to say: "To this we would add that the superiors themselves are sometimes under pressure from the Bank Presidency and elsewhere not to say things that go directly against the broad policy line that the Bank is espousing."

About WB's annual World Development Reports:

"We are also concerned with quality control over the Bank’s large number of "flagship" publications, here taken to be the World Development Reports and the DEC and non-DEC major topic studies to which the term is applied. These reports are sometimes enormously influential (though we suspect that many just gather dust) and they are the vehicles where the line between the Banks’ advocacy role and its role in producing new research ideas becomes particularly blurred."

"Trade-offs tend to be eschewed in favor of ubiquitous "win-win" scenarios, so that, for example, growth and environmental improvement are never seen as in conflict, because poverty and pollution are social problems that each mark institutional failure, so that institutional repair can somehow lead to both being dealt with simultaneously. A more equal income distribution is seen as a generally good thing, but there is no discussion of the optimal tax literature that formalizes the necessary trade offs between equity and incentives. More generally, trade-offs between competing goals are downplayed relative to sometimes far-fetched complementarities. While there is something to be said for such an approach in forging the compromises that are required to make progress in policy formation, it hardly leads to intellectual clarity. The World Development Reports suffer from always trying to make everyone happy."

"The Equity and Development WDR is perhaps the leading example. It contains no stable concept of what it means by inequality; again, we suspect that is a response to the need to try to make everyone happy, even when they have mutually incompatible views. For example, the report never resolves the tension between "equality of outcomes" and "equality of opportunity"."

Sources:

Evaluation of World Bank Research: 1998-2005

Knowledge Bank-rupted: Evaluation says key World Bank research ‘not remotely reliable’