Economist-activist Jean Drèze has argued that economists are no better equipped to comment on development policy design than other social science researchers and other stakeholders, and that policymaking requires more than just evidence. In this post, Apurva Bamezai and M.R. Sharan explore the roots of development economists’ centrality in social policy design and locate it in the nature of the evidence they generate and the top-heavy policymaking paradigm in India. Further, focusing on evidence alone, they contend that a more multidisciplinary approach can prove beneficial, and evidence-generation can also be a by-product of increased citizen-State interactions.
Jean Drèze’s thought-provoking piece on the complex relationship between evidence, policy, and politics has been widely shared in policy research circles recently. One of the main theses in this article is that development economists (more precisely, quantitative social scientists in development – we will use ‘development economists’ as a general umbrella term to cover all of these, à la Drèze) are no better equipped to comment on development policy design than researchers from other social science disciplines and also other stakeholders. However, development economists continue to hold sway among bureaucrats and policymakers in government as well as big donor organisations in the international development space.
We explore the roots of development economists’ centrality in social policy design. We locate it in the coming together of two key factors: (a) the nature of the evidence they generate, and (b) the top-heavy policymaking paradigm in India.
Policymaking, as Drèze argues, requires more than just evidence. The apparent ubiquity of the phrase ‘evidence-based policy’ must not distract us from the recentness of the phenomenon. Much of policymaking continues to be in the old mould and is not evidence-based. Political preferences, personal ideology and beliefs, anecdotes, etc. continue to play key roles in determining policy.
However, focusing squarely on evidence alone, we make two broad points: one, that a more multidisciplinary approach, even under current constraints, can prove beneficial. Two, evidence-generation can also be a by-product of increased citizen-State interactions, a neglected aspect of policy design in India.
General vs. particular
If phenomena can be broadly classified as the ‘particular’ and the ‘general’, modern development economics has equipped itself with the tools for dealing with the latter.
Consider the following hypothetical policy problem: understanding the impact of a cash transfer scheme being implemented in Bihar at scale. What a well-designed study in economics allows us to state, often with a great degree of certainty, is the impact of cash transfers for the whole of Bihar on a range of quantifiable outcomes. Technically, this is simply the average effect – referred to as the average treatment effect (ATE) in the literature. This is, needless to say, extremely important from a policymaking perspective. Some studies, depending on sample sizes and design, may be able to tell us more: for example, the impact on households residing in Sitamarhi district, or the impact on women. Fewer still, can tell us even more, without compromising on precision – for instance, the impact on women in Sitamarhi. In practice, effects can differ across dimensions of caste, religion, class, and institutional setups, to name only a few. Furthermore, even the outcomes measured may be limited to those broad (albeit, again, important) outcomes amenable to precise capture in quantitative metrics – consumption expenditure, nutritional intake, or transaction costs. More amorphous outcomes like impact on local politics, caste alignments, or even the local economy may be harder to grasp. Thus, in the way it is practised, modern development economics is not equipped with the tools required to give us a very particular sense of the effect of a policy: identifying distributional consequences over complex outcomes requires a degree of nuance that is largely beyond the grasp of the average effect1.
Indian social sciences beyond economics have a long tradition of grasping complex social realities in small geographies. M.N. Srinivas pioneered ‘village studies’ and also the uniquely Indian discipline of social anthropology. Many of Srinivas’ key insights – the concepts of ‘vote banks’ or (the more contested) ‘sanskritisation’ – have surprising generalisability. These came from a deep engagement with a handful of villages, but now constitute accepted wisdom. Thus, while this line of research and work is equipped foremost to deal with the ‘particular, that is, to characterise phenomena occurring over smaller areas, it can, on occasion, attempt to theorise and shed light on issues occurring over wider geographies.
Centralised policymaking privileges ‘general’ over ‘particular’
In light of the above, what is the reason for the primacy accorded to quantitative data in informing (social) policies? It is our belief that the centralised policymaking regime in India prioritises the ‘general’ over the ‘particular’.
Independent India has had a history of centralised planning. The early planning processes adopted in India owed in no small measure to P.C. Mahalanobis, a statistician by training. While many aspects of social policy in India’s federal set-up are meant to be designed and implemented by state governments, much of it – including more recent social protection initiatives such as the Mahatma Gandhi National Rural Employment Guarantee Act (MNREGA) and the National Food Security Act (NFSA) – remains, for the most part, centrally conceived and designed. State-level development bureaucracies – and most certainly everyone below – are largely reduced to mere implementing agencies, focused on following rules set by agents above (Mangla 2015).
In a centralised policymaking set-up, it is natural that the ‘general’ will be privileged over the ‘particular’ for two reasons. One, policies are often designed for extremely large populations and this necessitates erring on the side of one-size-fits-all policy design. Two, the distance of the policymaker from the larger target population allows him/her the unfortunate luxury of being isolated from the consequences of the heterogeneity in average effects of the policy. Put differently, the voices of those adversely affected by the policy find it harder to reach this distant policymaker.
There is nothing natural about this centralised policymaking setup. Besides historical and political economy2 factors, it is also a consequence of a lack of resources and State capability at the local level (Dasgupta and Kapur 2017) to ideate, pilot, iterate, dynamically adapt, and generate evidence. This poses a tremendous burden on policymakers at higher levels, for the breadth of their policy activities is immense. Given the pressures most policymakers work under, the reduction of evidence to ‘numbers’ proves extremely seductive. It is much easier (and considerably quicker) to weigh policy alternatives if you can just put them on a simple balance. This is what other non-quantitative types of evidence do not allow for very easily.
The role of multidisciplinary evidence
This status quo, however, is not an ideal state of affairs. Taking Drèze’s point further, in the circumscribed set of cases where the ‘bridge between evidence and policy' is functional, evidence-gathering requires a marriage of sorts – bringing together qualitatively and quantitatively trained researchers, and civil society members. There are at least three ways in which qualitative researchers (this could also potentially include local civil society members) can significantly contribute to policy design.
First, in understanding distributional consequences: as we argue above, qualitative research brings nuance and texture to both the discussion of impact on groups and, also, on outcomes. If the primary objective of social policy is poverty alleviation, then understanding in depth the consequences of such policy on the poor and the vulnerable is paramount. It is precisely here that rigorous qualitative evidence can be most useful, allowing us to carefully untangle the effects of social policy on the most disadvantaged.
Second, mechanisms of impact: if quantitative evidence can give us a big-picture understanding of what works, then qualitative evidence is well-placed to help us understand how it works. Increasingly, quantitative economics has moved in this direction too, often looking for and building in ways to identify and measure mechanisms that lead from intervention to impact. Indeed, as Drèze points out, Duflo’s conceptualisation of “economists as plumbers” (Duflo 2017) points to this very trend. However, qualitative work still has a role to play: on the one hand, local knowledge and a deep understanding of the context remains important to grasp truths a one-hour survey may not be able to. Furthermore, policies are often complex and the levers are too many for each to be tinkered with and tested at scale.
Anthropologist Jefferey Witsoe’s careful work (2013) in Bihar serves as an example here. His primary objective is to study the impact of the policies – or lack thereof – of the controversial former Chief Minister, Lalu Yadav. On the face of it, the numbers tell a bleak story: growth stalled – and briefly reversed – in Bihar, while it soared in the rest of the country, propelled by the recently unleashed forces of liberalisation. Bureaucracies were severely understaffed; their morale suffered a significant blow. Crime thrived, particularly in urban areas – this period in the state is often referred to by sorry epithets such as ‘jungle raj’ and ‘goonda raj’.
Witsoe’s work allows us to peel layers underneath these broad facts and numbers. Using testimonies gathered from marginalised groups (backwards, Dalits, and Muslims) over years of fieldwork in a few villages in Bihar, he argues that Lalu Yadav’s political methods had complex distributional consequences. In particular, it significantly improved the relative well-being of the poor and the vulnerable. Moreover, he argues, this improvement was not merely in terms of income, but it refers to something more amorphous, for the nearly 15- year long tenure was marked by great social churning. Lalu Yadav’s actions, speeches and slogans – “ bhurabal hatao ” [remove the bhurabal; bhurabal is short for upper castes like Bhumihars, Rajputs, Brahmins, and Lalas], for instance – were marked by references and imagery that signalled the end of upper-caste domination and the rise of the backwards. Lalu Yadav, therefore, empowered the backwards and upturned generations of carefully nurtured caste hierarchies.
To understand the mechanisms that led to this impact, Witsoe uses data gleaned from months spent in a Block Office – the lowest nodal bureaucratic outpost prior to 2001 in Bihar. Lalu Yadav systematically weakened the upper-caste dominated bureaucratic State and, consequently, directly strengthened local politicians. Democracy conflicted with development, evocatively emphasised in another party slogan of the times: “vikas nahin, sammaan chahiye .” (“We want dignity, not development”). This contradiction – a feature (or bug) seen in many post-colonial democracies – gives rise to the patterns we see in aggregate: insignificant growth caused by a systematically weakened bureaucracy; a rise in criminal politicians resulting from empowered local representatives who operate in absence of a State machinery to enforce law and order; and immense social churning resulting from both the above processes.
Witsoe’s work is both insightful and atypical, given its sweeping nature. We are not arguing here that all qualitative work can unearth structural issues in the manner that his did. We only use his work as an instance3 – a particularly successful one at that – of how good qualitative work can shed light on (a) the distributional consequences, and (b) mechanisms of impact, of any development policy.
Third, qualitative work can help unpack anomalies in the data. Anomalies – or outliers – form classic examples of the ‘particular’ informing the ‘general’. Drawing an example from the discipline of neuroscience, Dr. V.S. Ramachandran has made surprising general (and sometimes controversial) claims about how the brain works using a smallish sample of patients suffering from something very particular – the rare disorder of “phantom limbs” or “synesthesia” (Ramachandran et al. 1998).
An analogous hypothetical example in social policy can be constructed: imagine that the policy of cash transfer works well on average, but fails spectacularly (predicted quantitative impact, although noisy, is highly negative) in three blocks in Samastipur district. A qualitative researcher with deep knowledge of the context could help shed light on the reasons for doing so: it could be something unsystematic or unchangeable – dumb luck, or a recent change in personnel, or the nature of the caste composition of the area. It could also be something fixable, like corruption. Perhaps, the local bureaucrats in those three blocks have ‘cracked’ the new cash transfer system, indulging in practices that reap immense benefits to themselves at the cost of the poor. These officers can be outliers, but there is a chance that the incident could also serve as a sign of things to come, as their peers learn from them: in either case (though especially in the latter scenario), this would call for tinkering with policy design or major policy change.
Citizens’ voice as evidence
We have discussed what may be considered rigorous forms of evidence – both quantitative and qualitative – and their value and place in policymaking. We propose that citizens’ experience and participation are forms of ‘evidence’ as well. For instance, the ‘particular’ experiences of some citizens with respect to mandatory Aadhaar4 authentication for accessing benefits such as the public distribution system (PDS) rations in some states are not in any way less valuable than abstract (and problematically calculated) metrics of ghost beneficiaries.
From a policymaker’s perspective, the issue with this form of evidence is to separate signal from noise5 and, in many cases, move from the ‘particular’ to the ‘general’. In other words, what mechanisms can we create – or employ – to systematise and aggregate such evidence?
Grievance redressal mechanisms form one potential channel6 A well-functioning grievance redressal mechanism – where citizens use the platform to consistently air and solve their issues with various arms of the State – can allow for the flow of a rich stream of systematised evidence. Viewed in the aggregate, these could form inputs for policy design. For instance, a disproportionately large number of complaints made in Darbhanga district regarding the PDS could signal something systematically wrong with procurement or distribution of grains. Grievances from female complainants could be distinct from those from male ones. Thus, a well-functioning MIS (management information system) on grievances could curate complaints, and from the policymakers’ perspective, help shape policy tweaks and design.
Social movements, civil society groups or even more traditional NGOs (non-governmental organisations) offer a second channel for curating and amplifying citizens’ voices – especially in flagging issues related to operations7. A recent example of policy change is the withdrawal of the Direct Benefit Transfer (DBT) scheme for food subsidy in Nagri block in Jharkhand after nearly a year of civil society action (organised by the Right to Food Campaign in this case). This was followed by a social audit conducted by the state government’s Rural Development Department. The civil society action nalso generated political pressure from opposition parties. Here was an example of citizen action, democratic pressures, and bureaucratic forces coming together to reconsider and reshape policy.
In general, social policy is almost never set in stone. Good policy design allows for dynamic changes and, crucially, incorporates listening to voices of those affected directly by it. While it may not be possible to always accommodate the choices or preferences of everyone, there is no gainsaying that this is a necessary condition to be met.
In sum, insofar as we have centralised policymaking and an over-burdened policy apparatus, the ‘general’ will be privileged over the ‘particular’ and development economists will continue to generate the kind of evidence that influences policy. However, this does not preclude the need for much greater State engagement with a wider set of researchers, cutting across disciplines in social sciences. This is particularly true for researching impact of policies on poor, vulnerable, and marginalised groups. Furthermore, policy design affects different groups differently and it is imperative that the State listens more (thereby also diversifying its evidence base) to citizens, civil society groups, social movements, and NGOs. Plugging gaps in policy design and implementation cannot be just a product of technocratic thinking and economic efficiency, especially in a democracy.
The authors are grateful to Dipa Sinha, Niharika Singh, and Nandini Gupta for their incisive comments.
- This is not to say that quantitative work in development cannot give us a sense of the particular: Drèze and Khera’s surveys often reveal new – and quite nuanced – trends in access to public services. This is partly because the inferences made are rooted in lived experiences in the field. The nascent sub-field of behavioural development economics also makes very particular claims. Similarly, a carefully grounded structural model can help make the leap from some general claims (reduced form estimates – from a randomised controlled trial (RCT), for instance) to a more nuanced understanding of mechanisms and distributional outcomes.
- Centralised policymaking allows for centralisation of credit (for its success; though not always blame for its failure), contributing in turn to potentially favourable electoral outcomes. To put it simply, decentralised policies are not suited to branding under a party or a political leader’s name and hence the returns to a successfully implemented policy are higher the higher one goes up the political or bureaucratic ladder.
- Indeed, this loose framing of Witsoe’s work as an ‘impact evaluation’ of the Lalu Yadav years is a device we use strictly for expositional purposes. It is not our claim that this was at all the original intention of his work.
- Aadhaar or Unique Identification number (UID) is a 12-digit individual identification number issued by the Unique Identification Authority of India (UIDAI) on behalf of the Government of India. It captures the biometric identity – 10 fingerprints, iris and photograph – of every resident, and is meant to serve as a proof of identity and address anywhere in India.
- We are not claiming that more rigorous forms of evidence are free from bias, only that the signal is clearer and knowledge more systematised.
- (i) Informal grievance redressal mechanisms, (ii) scheme-specific formal grievance redressal mechanisms, and increasingly, (iii) state-wide formal channels like the Bihar Right To Public Grievance Redressal Act (BPGRA) in Bihar do exist but have had variable levels of success. We are envisaging a mature version of the BPGRA.
- A free press plays a role in systematising evidence by conducting fair investigations, mediating between the citizens and the State, and informing public opinion.
- Dasgupta, A and D Kapur (2017), ‘The Political Economy of Bureaucratic Effectiveness: Evidence from Local Rural Development Officials in India’.
- Drèze, J (2018), ‘Evidence, policy, and politics’, Ideas for India, 3 August 2018.
- Duflo, Esther (2017), “Richard T. Ely Lecture: The Economist as Plumber”, American Economic Review, 107(5).
- Mangla, Akshay (2015), “Bureaucratic norms and state capacity in India: implementing primary education in the Himalayan region", Asian Survey, 55(5): 882-908. Available here.
- Ramachandran, VS, S Blakeslee and N Shah (1998), Phantoms in the brain: Probing the mysteries of the human mind, William Morrow, New York.
- in (2018), ‘Jharkhand withdraws direct benefit transfer scheme for food subsidy after problems in implementation’, 8 August 2018.
- Witsoe, J (2013), Democracy against development: Lower-caste politics and political modernity in postcolonial India, University of Chicago Press.