Poverty & Inequality

Balancing corruption and exclusion: A response

  • Blog Post Date 21 October, 2020
  • Perspectives
  • Print Page
Author Image

Karthik Muralidharan

University of California, San Diego

kamurali@ucsd.edu

Author Image

Paul Niehaus

University of California, San Diego

pniehaus@ucsd.edu

The role of Aadhaar in reducing corruption in PDS has been highly controversial. In a recent I4I article, Muralidharan et al. summarised the results of their study in Jharkhand, which confirmed the 'pain without gain' result of an earlier study by Drèze et al. of merely seeding the electronic point-of-sale machines with Aadhaar. However, their second main result that seeding Aadhaar in conjunction with reconciliation reduced leakage albeit with greater pain, was challenged by Drèze et al. in their rejoinder in I4I, who raised concerns regarding methodology and interpretation of the results. In this post, Muralidharan et al. defend their study’s methodology and address the questions raised by Drèze et al. It is a debate between two sets of top development economists on a crucial issue.

 

We thank Jean Drèze, Reetika Khera, and Anmol Somanchi (hereafter DKS) for engaging with our work (Muralidharan et al. 2020) on the impact of Aadhaar1-based biometric authentication (ABBA) on leakage and exclusion in the Public Distribution System (PDS) in Jharkhand. DKS do not dispute the majority of our findings, but do question two important results and (perhaps in part because of this) disagree with the way we frame and interpret the results as a whole. They offer an alternative interpretation.

We welcome this. We expect that different people will interpret and emphasise our results differently, and seek to report them transparently and in enough detail for scholars such as DKS to offer alternative perspectives. We also greatly value the on-the-ground, qualitative perspectives they provide to complement our quantitative findings.

At the same time, we believe that our public dissemination of the results – represents our study findings and our views on its policy implications faithfully. In this post, we articulate what we understand to be the main areas of agreement and disagreement, and situate that debate within our broader view of the role of evidence in policymaking.

Recapping the reform and the evaluation

First, some necessary context.

The reform we studied included two closely related components, which together were intended to reduce leakage in the PDS. These were introduced by the Government of Jharkhand (GoJH) in sequence (as well as by most other state governments, following the directive of the Government of India).

In the first phase, GoJH installed electronic point-of-sale (ePOS) machines in ration fair price shops (FPSs) and required beneficiaries to obtain an Aadhaar number, link (or “seed’’) it to their PDS account, and authenticate by scanning their fingerprints each time they transacted. We refer to this reform as “Aadhaar-based biometric authentication'', or ABBA.

In the second phase, GoJH began to adjust downwards (or “reconcile”) the amounts of grain it disbursed to each FPS each month to reflect the amount the FPS should still have in stock. This reform depended critically on the digital record of transactions generated by the earlier ABBA reform, transactions for which beneficiaries had ‘signed’ biometrically (in contrast with the status-quo approach of authentication based on physical ration cards and record-keeping on paper). Reconciliation contrasted with the status-quo approach, which was to disburse the full amount of grain needed to satisfy the entitlements of all beneficiaries in every month.

We evaluated the effects of these reforms using two related but distinct strategies. We evaluated the effects of ABBA using a randomised controlled trial (RCT) in which we worked with GoJH to randomise the order in which it introduced ABBA across 132 sub-districts in 10 districts. GoJH then launched reconciliation simultaneously in all areas two months after deploying ePOS devices in ‘control’ areas (where ABBA was introduced later) and 11 months after doing so in ‘treatment’ areas (where ABBA was introduced earlier). We therefore evaluated the effects of reconciliation using this natural experiment2 (an “event study” design) in a pre-specified econometric framework to identify trend breaks, and using unreconciled PDS commodities (sugar, salt, and kerosene) as a placebo group. For both evaluations, our sample is representative by design of 15.1 million individuals in 3.3 million beneficiary households in 17 of Jharkhand’s 24 districts, and representative on observables of the rest of the state.

DKS reproach us for implying in our popular writing that the reconciliation results are also experimental, while it is only the ABBA results that are from the RCT. To the extent that ‘implying’ suggests intent, this is untrue; our intent was not (and would never be) to mislead. As all researchers do, we make judgments about how much of the nuance of technical work to communicate in short-form summaries for non-technical audiences. The principle we seek to follow is to include the technical issues that, in our judgment, affect the substantive conclusions. In this case, we are as confident in the reliability of the evidence from the natural experiment (reconciliation) as from the randomised experiment (ABBA) and so did not draw the distinction. But doing so would evidently have avoided some confusion and suspicion of an intent to mislead, and so we accept and agree that we should have.

What impacts did ABBA per se have?

Here there is no disagreement between us and DKS. The results unambiguously show that there was no improvement in the beneficiary experience, and no reduction in leakage. There was evidence of some increased exclusion (2.4%), which was concentrated among households who had not seeded their PDS ration cards with Aadhaar numbers at the time of the onset of the reform. Finally, we find an increase in transaction costs for beneficiaries in accessing their benefits driven by a significant increase in the number of unsuccessful trips made to collect their rations (presumably because of authentication failures).

These findings are consistent with those reported from smaller-scale, prior studies by DKS and co-authors (Dreze et al. 2017), and corroborate their main messages with larger and more representative samples, as well as randomisation of programme rollout for more credible causal inference of programme impact.

It is not surprising that ABBA by itself had no impact on leakage, since GoJH did not adjust the amount of grain distributed to PDS dealers during the ABBA phase. It is only when the government started using the biometric transaction records enabled by ABBA to adjust the amount of grain distributed to PDS dealers (reconciliation) that we might reasonably hope to start seeing a reduction in leakage. Of course, this could also worsen exclusion. This is why it is necessary to evaluate the two reforms together and not in isolation.

It also highlights the importance of matched data between disbursals and receipts to get the full picture of what happened. Using only field-based studies (as DK and others often do) provides the beneficiary perspective but misses potential fiscal savings. Using only data on programme spending (like the government often does) captures fiscal savings but misses the extent to which these savings may come at the cost of beneficiaries. The matched data allows us to estimate both these quantities.

What impact did reconciliation have?

Empirically we do indeed find that there was a sharp reduction in both distributions and leakage when reconciliation began, as seen in Figure 3 in the paper, and that the reduction in amounts distributed was greater than in amounts received. At the same time, reconciliation also led to significant reductions in grains received by beneficiaries as well. No such reductions are seen for unreconciled commodities, which serve as a contemporaneous control or placebo group.

We take this (along with ancillary evidence which we discuss below) as convincing evidence that reconciliation reduced distributions and leakage on one hand, but also increased exclusion on the other. DKS are not convinced, primarily (as we understand) because this result is not identified using an RCT.

Here we find ourselves more in agreement with Drèze’s own prior writing pointing out that “evidence involves more than RCTs” (Drèze 2018), and that we should not privilege evidence from RCTs to the exclusion of evidence from other sources.

The natural experiment we examine here passes the credibility checks we would typically look for in work of this sort: a well-specified time of onset, a contemporaneous comparison group, and clear evidence of parallel trends prior to reconciliation. No reasons other than reconciliation, plausible or otherwise, have been proposed by DKS or anyone else to explain why disbursements and leakage would fall abruptly in July 2017 after having held quite steady through the first half of the year and before returning to trend after reconciliation was ended (and that this pattern would hold only for reconciled commodities and not for non-reconciled ones). Further, DKS do not appear to contest our finding of increased exclusion from reconciliation, which comes from the same method as that used to report the reduction in disbursement and leakage!

Our confidence in the finding that reconciliation reduced disbursements and leakage is further strengthened by two ancillary pieces of evidence which we discuss in paper. First, the impact of reconciliation (both in terms of reduction in grain distribution from the government, and reduction in grains obtained by beneficiaries) is significantly greater in the treatment group. Thus, there is experimental evidence that the changes that took place at the onset of reconciliation were larger in the treatment group. This is exactly what one would expect if reconciliation were causing those changes, since FPSs in the treated group had had ABBA for longer and thus, the ABBA records showed treated FPSs having a larger stock of undisbursed grain. This would lead to a larger reduction in grain sent to these shops by the government at the onset of reconciliation (contributing to both greater reductions in leakage and greater reduction in value received by beneficiaries in treated areas). No other explanation of this pattern has been proposed, and given that ABBA’s overall impacts were muted it is hard to see how it could otherwise have led to a differential change in quantities at the precise moment reconciliation began.

Second, prior to the onset of reconciliation we find that PDS dealers in the treatment group report a 72% lower expected future bribe price for a PDS license (a reduction of Rs. 58,400 compared to a control mean of Rs. 81,200). This is also an experimental result, which convinces us that dealers expected that their ability to siphon grains (leakage) would significantly decline after the anticipated beginning of reconciliation. Again, DKS do not present any other possible explanation for this striking result.

Put together, we disagree with DKS’s contention that the study results do not point to evidence of reduced leakage due to ABBA and reconciliation. We also disagree with their characterisation that our summary spins the results in a positive way given that we have consistently highlighted the increased exclusion both due to ABBA and due to reconciliation. If anything, several others have told us that we may have overstated the negative effects, by reporting heightened exclusion during a transition period as opposed to a steady state after initial implementation challenges (such as incomplete Aadhaar seeding) were resolved.

DKS also question our selection of figures to emphasise from this episode. In the paper we report how the reductions in grain disbursements broke down into reductions in leakage and in value received by beneficiaries both during the first month of reconciliation (66% and 34%) and averaged over the full four-month period (51% and 49%). We view the first-month numbers as more relevant for predicting the effects reconciliation may have in the longer run because GoJH adhered more closely to its reconciliation protocols during this month than subsequently (as pressure mounted to suspend it). For example, in the first month of reconciliation most dealers were recorded as holding positive stocks of grain, while in the second to fourth month large numbers were recorded as holding negative stocks, which is obviously not physically possible and implies a less punitive disbursement policy (see Appendix D, Figure D.1, Panel B).

That said, either set of numbers indicate that reconciliation generated both meaningful reductions in leakage and substantial harm to beneficiaries, which is our main point. None of our substantive conclusions would be different if we based them on the four-month numbers as opposed to the first-month numbers.

More broadly, our numeric findings should not be interpreted as representing an immutable set of trade-offs between leakage reduction and exclusion errors, since these may change over time. Rather, they highlight the possibility that even well-intentioned reforms supported by international development agencies and carried out by a democratically-elected government can generate substantial social costs with over two million legitimate (and disproportionately vulnerable) beneficiaries losing access to essential benefits at various points during the roll-out. Since estimates may change over time (as discussed further below), the most effective way to learn from our study for other such programmes is to focus on ‘principles’ as opposed to ‘point estimates’.

What impact would ongoing reconciliation have?

DKS also raise a distinct concern about the ‘applicability’ of the reconciliation results: even if it were true that reconciliation reduced leakage in Jharkhand for a few months, the fact that reconciliation itself was then halted four months later (due to its unpopularity) limits what we can learn from the episode about the likely effects of a sustained policy of reconciliation. This is even more so given that by the end of the four-month period enforcement of reconciliation was itself quite lax, that is, reconciliation had largely ended in practice before it ended in name.

There is no question that this is right to some degree, as we discuss in the paper. The key issues are not only of degree (how different might things be in the longer run?) but also of direction (do we have reasons to expect them to be better or worse?) DKS are concerned that the results could be worse after our study window. Other audiences have critiqued our interpretation for the opposite reason, arguing that our evaluation highlights negative results on exclusion because of ‘teething troubles’ that will be resolved over time. Who is right and who is wrong here is to some extent unknowable without further monitoring of beneficiaries' experiences as reconciliation continues – something we strongly advocate.

That said, our expectations and our policy recommendations are grounded in additional analysis we conduct in the paper. Specifically, we examine two alternative reasons that beneficiaries’ might be made worse-off by the onset of reconciliation. The less pessimistic interpretation is that beneficiaries suffered initially because GoJH held dealers accountable for up to 11 months of past activity when it rolled out reconciliation, as opposed to starting each with a ‘clean slate’ (or a zero opening balance). This effect led to a large initial reduction in grain disbursements, some of which was passed through to beneficiaries, but would not persist in the longer run. The more pessimistic interpretation is that reconciliation eroded beneficiaries bargaining power: dealers could now force them to ‘sign’ for grain they did not in fact receive by pointing out (correctly) that if they did not there would be no grain at all the following month. But the change in bargaining power could also help beneficiaries since dealers are less likely to send them back with no grain as they need them to biometrically sign for receipts.

To distinguish these interpretations we estimate the effects of a hypothetical ‘clean slate’ reconciliation (starting with an opening balance of zero) in which the former mechanism (the ‘opening balance’ effect) is removed while the latter (the ‘bargaining’ effect) remains. We do this exploiting the original RCT, which generated substantial experimental variation in the size of dealers’ opening balances at the onset of reconciliation. DKS’s reference to this as “an instrumental variable of doubtful validity” is puzzling since it was experimentally assigned, and is thus the textbook example of a credible instrument. The instrument is valid provided only that it did not affect outcomes during the reconciliation period other than through opening balances, and no reasons have been proposed by DKS or anyone else that it would. DKS’s critique of the exercise as using “noisy data” is similarly puzzling and not further elucidated. These are the same data we use in the rest of our analysis, and the precision of the estimates is similar to that of our other results.

Using this approach, we estimate that had GoJH rolled out reconciliation on a clean-slate basis, leakage would have fallen by less than it did in practice, but still significantly, and without significant harm to beneficiaries (the point estimate for value received by beneficiaries is positive). The data thus suggest that what caused harm to beneficiaries in Jharkhand in the summer of 2017 was not primarily the practice of reconciliation per se, but rather the ‘cumulative’ way in which it was introduced.

In other words, it was likely a mistake to hold dealers accountable for undisbursed grain for 11 months and expect that they had all of that in stock (when in practice, this stock may have been diverted or spoilt). However, our calculations based on the experimental instrumental variable suggest that reconciliation on a monthly basis starting from a zero opening balance could have reduced leakage without increasing exclusion on average.

What are the implications for using Aadhaar in the delivery of welfare programmes?

Reflecting on our results as well as their own extensive research and ground experiences, DKS suggest that ABBA and reconciliation as introduced in Jharkhand are not the best way to tackle leakage in the PDS. Rather, they suggest that a “simpler and more reliable technology like smartcards” could be used as the basis for reconciliation. We take this to refer to models like the one we previously evaluated in Andhra Pradesh in the context of MNREGA (Mahatma Gandhi National Rural Employment Guarantee Scheme) and pensions. In that evaluation we found that the introduction of offline biometric authentication for payments significantly reduced leakage while also improving beneficiaries’ experiences. In other words, there was in this case no tradeoff between reducing leakage and serving the needs of legitimate beneficiaries. (On top of this, the reform appears to have made MNREGA a more attractive option for low-income labourers and in doing so substantially raised rural wages, generating sizeable gains in their private-sector earnings.)

We certainly agree with the point that reconciliation does not require real-time online authentication, and is thus not a good reason to incur the associated costs and risks. That said, we do not think the technology of ABBA or connectivity requirements were the core issue in Jharkhand. We do not see evidence, for example, that the impacts of ABBA were more negative in places with worse data connectivity (Table A.11) or in places that were required to operate in online as opposed to offline mode (Table A.12).

Instead, our close experience evaluating the two programmes, leads us to believe that the contrast between the results in Jharkhand and those in Andhra Pradesh was driven less by differences in authentication technology, and more by deeper differences in political priorities on fiscal savings versus beneficiary experiences. The smartcards reform in Andhra Pradesh was introduced with an explicit goal of improving the beneficiary experience, and many consequential design choices followed from this. For example, the reform was accompanied by adding business correspondents to villages to improve the payments experience, and featured generous manual override provisions to minimise the risk of exclusion errors from beneficiaries not being able to authenticate. In contrast, ABBA and reconciliation were introduced in Jharkhand with the primary goal of achieving fiscal savings. Analogous override provisions were de facto not available, with the result that some genuine beneficiaries who had not been able to “seed” their ration cards lost access to their benefits.

Put together our studies highlight that the biometric authentication technology improved State capacity in both settings, because leakage did go down in both cases. But in Andhra Pradesh, the leakage reductions were passed on to beneficiaries, with no reduction in fiscal cost. In Jharkhand, the government saved money, but some of it was at the cost of beneficiaries. This is why our central message regarding the policy implications of our studies is to focus less on the technology of authentication, and more on the design details of how the technology is used, to ensure that technology is used to improve beneficiary experiences and empowerment (this is a point we have made in prior writing as well as government presentations).

Going forward, there are ways in which ABBA in the PDS could be used to empower beneficiaries. One is through increased portability of benefits both within and between states. If done well, this can improve beneficiary welfare both by giving them options across FPS dealers (which may help reduce corruption by creating competition across dealers for beneficiaries, particularly in urban areas), and by allowing tens of millions of migrant workers to access the PDS benefits even outside their home villages. Such portability requires online authentication and a technological ‘back end’ that knows in real time whether benefits have already been accessed somewhere else.

Another is by offering beneficiaries the choice between in-kind entitlements and a direct transfer of a fiscally equivalent amount into their bank accounts. Both DK and us have written in the past about the broader income versus in-kind debate, with DK typically quite skeptical about income transfers in lieu of in-kind benefits. Our view is that there is so much heterogeneity across the country and over time that the best way to proceed is to not impose one or the other but to empower beneficiaries to choose what works best for them. Again, this is a reform that requires online authentication with real-time back end tracking of withdrawals.

In short, ABBA expands state capacity in ways that could be used for beneficiary-friendly reforms that would not otherwise be possible. However, there is no guarantee that this will happen, and as the Jharkhand case aptly illustrates, rolling out ABBA and using it in pursuit of fiscal savings can cause real harm to legitimate beneficiaries. We think time and effort will be most productively devoted to pushing for uses of ABBA that are intended and designed to prioritise beneficiary welfare, and not just fiscal savings. We also advocate for continued real-time measurement and monitoring of beneficiary experiences to learn quickly if beneficiaries are suffering, as for example using outbound call centres. One of the lessons we draw from the fieldwork that DKS and their colleagues have conducted in Jharkhand and elsewhere is that more systematic versions of the feedback loop they provide are needed.

Role of evidence in policymaking?

In concluding, it is useful to reflect on the broader role of evidence in policy making, and in this we find ourselves in substantial agreement with prior writings of Drèze on this subject – especially in a previous I4I post (Drèze 2018), where he cautions that:

“economists need to be cautious and modest when it comes to giving policy advice, let alone getting actively involved in ‘policy design’. Their expertise and research can certainly contribute to more informed policy discussions and public debates…(but)…In the field of social policy, at least, I see no reason to privilege the advice of economists.”

He goes on to say that:

“None of this detracts from the value of RCTs (correctly understood), or from the case for evidence-based policy. If the idea is to bring more evidence to bear on public policy, there is much to be said for it. This endeavour, however, is likely to be all the more useful if we bear in mind that evidence involves more than RCTs, understanding more than evidence, and policy more than understanding.”

We are completely in agreement with this view. Indeed, evidence should be more than RCTs and include other credible empirical techniques for causal inference (as we do) as well as field experiences (as DKS add). Understanding the implications of our evidence requires paying attention to principles and not just point estimates, and translating such understanding into policy requires a broader engagement with the underlying goal of reforms (such as fiscal savings versus improving beneficiary experiences) being debated and validated through a broader democratic process.

More broadly, the sentiments in Drèze (2018) are fully consistent with our own policy writing in the past, where one of us has noted that: “Policy formulation needs to consider technical, administrative, ethical, as well as political factors and even the best technical studies can only provide inputs into one dimension of policy making.” (Muralidharan 2013)

We believe that we have provided a set of technically valid results on the impacts of an important reform for the design and delivery of welfare programmes in India. We have transparently presented both the substantial costs and the benefits (which are not visible on the ground without also accessing administrative data on grain distribution) of the reform, and have highlighted what we think are the most important broader principles for improving the design and delivery of welfare programmes in India. At the same time, we certainly do not believe that we should have the last word on this complex topic, and we welcome the engagement with the results and the alternative interpretations for policy presented by Drèze, Khera, and Somanchi.

Notes:

  1. Aadhaar or Unique Identification (UID) number is a 12-digit identification number linked to an individual’s biometrics (fingerprints, iris, and photographs) and issued to Indian residents by the Unique Identification Authority of India (UIDAI) on behalf of the Government of India.
  2. A natural experiment occurs when units of observation are exposed to treatment and control conditions using variation that is outside the control of the researchers. They typically involve comparing outcomes before and after a policy for those exposed to the policy relative to a contemporaneous control group that was not exposed to the policy, but where the exposure to the policy was not randomized by the researcher.

I4I is now on Telegram. Please click here to subscribe to our channel for quick updates on our content.

Further Reading

No comments yet
Join the conversation
Captcha Captcha Reload

Comments will be held for moderation. Your contact information will not be made public.

Related content

Sign up to our newsletter