Balancing corruption and exclusion: A rejoinder

The role of Aadhaar in reducing corruption in the PDS has been highly controversial. In a recent I4I article, Muralidharan et al. summarised the insights of their study of Aadhaar-based biometric authentication in Jharkhand. The study found that compulsory biometric authentication on its own had caused “pain without gain”, confirming similar findings of an earlier study by Drèze et al. However, the authors also argued that biometric authentication in conjunction with a follow-up reform, known as reconciliation, had reduced PDS leakages albeit with greater pain. In their rejoinder in I4I, Drèze et al argued that the authors’ summary of their findings is a little misleading, and that the short-lived reconciliation episode does not significantly alter the diagnosis of pain without gain. In this post, Muralidharan et al. defended their interpretation of the findings and presented their response to the questions raised by Drèze et al.

In a recent contribution to Ideas for India, Muralidharan, Niehaus, and Sukhtankar (hereafter MNS) present the findings of their study of the impact of Aadhaar-based biometric authentication (ABBA) on the public distribution system (PDS) in the state of Jharkhand.^1,2 We submit that this summary of the findings is misleading, as are related summaries that have appeared in Hindustan Times, Indian Express, Twitter, and other public forums.

Briefly, these summaries repeatedly create an impression that the study produced experimental evidence of a reduction in PDS leakages after ABBA was introduced, when it does nothing of the sort. We bring this up for two reasons. First, this is a telling illustration of the occasional “contrast between the care that goes into running an RCT [randomised controlled trial] and the carelessness that goes into advocating the use of its results” (Deaton 2020, p. 11). Second, a clear understanding of the findings is important for food policy in Jharkhand.

Before discussing our reservations, we acknowledge some important insights from their study. To appreciate them, a little background will help.

On the PDS

India’s PDS is probably the most extensive safety net in the world. It covers about 800 million people under the National Food Security Act (NFSA), and more under the ambit of state-specific schemes (Khera and Somanchi 2020). The value of the PDS has been amply demonstrated in the last few months, when subsidised food rations protected the bulk of India’s population from hunger even as a prolonged lockdown wrecked the economy.

Having said this, the PDS also has a long history of vulnerability to corruption. During the last 15 years or so, and especially after the NFSA came into force in 2013, many Indian states have tried to plug or at least reduce the leakages through various PDS reforms. Chhattisgarh’s success in this respect is particularly significant, and reasonably well documented.³ It influenced the NFSA, and inspired other states to undertake similar reforms, with varying but largely positive results.

Jharkhand is a particularly interesting case because it started off with astronomical levels of PDS corruption: leakages were estimated at 80-85% from four successive rounds of National Sample Survey (NSS) data between 2001-02 and 2007-08 (Khera 2011, Table 2). In the last few years, however, the PDS in Jharkhand has steadily improved. Today, most cardholders in Jharkhand are well aware of their entitlements, and while the practice of katauti (cuts extracted by PDS dealers from people’s monthly rations) continues, wholesale embezzlement has largely disappeared.

MNS’s study provides valuable confirmation of this recent progress. In the control areas of their RCT, leakages had reduced to 20% or so by early 2017, before the introduction of ABBA. There were very few fake ration cards in the system – at most 3%. And two thirds of the cardholders were satisfied with the quality of PDS grain.⁴ These are achievements that few observers would have thought possible 15 years ago.

It is worth noting, for future reference, that this progress was achieved through reforms that did not involve any obvious ‘trade-off’ between corruption and exclusion. On the contrary, many reforms have helped to reduce both. For instance, fostering better public awareness of PDS entitlements played a major role in reducing corruption and avoiding exclusion. This also applies to the expansion of PDS coverage, social audits, the computerisation of records and proactive transparency based on a user-friendly PDS portal.

Of course, the battle against corruption is far from over – even the reduced leakages, at 20% or so, are unacceptable. They are largely of two types: katauti, and what might be called ‘bogus transactions’ – cases where the cardholder gets nothing at all, for whatever reason, and the dealer siphons off their monthly ration by fudging the transaction records. The distinction is important because it points to the limitations of relying on ABBA for further improvement: ABBA is of little help in preventing cuts, as MNS correctly note. A deeper problem, however, is whether ABBA is an appropriate technology at all for the PDS in rural Jharkhand.

On the RCT

The RCT reported by MNS focuses on the imposition of ABBA on the PDS. The findings are consistent with those of our own study on the same subject (Drèze, Khalid, Khera and Somanchi 2017).⁵ We had come to the conclusion that this reform had caused “pain without gain”. On the one hand, it had little impact on leakages, partly because identity verification (the problem ABBA seeks to address) was not a serious problem in the first place. On the other, ABBA was a major source of exclusion and inconvenience, particularly for vulnerable groups such as widows, the elderly and manual workers. Some of them experienced abominable indignity and hardship as they struggled to navigate ABBA, often missing on their food rations.

The findings of the RCT, conducted by MNS in early 2017, are very similar: ABBA created exclusion problems and increased transaction costs, especially for vulnerable groups, without reducing corruption. We are glad, of course, that these earlier insights are now largely confirmed (and extended) using a different method – a “rigorous, independent, representative evaluation” as MNS modestly call it. Indeed, this is an impressive study, as one expects from this formidable trio.

The MNS survey data add an important dimension to the diagnosis of “pain without gain”. They confirm a conjecture we had made at the time of writing (in late 2017): the mass cancellation of rations cards not linked to Aadhaar in mid-2017 was a gross injustice, because most of these ration cards, far from being ‘fake’ as the Jharkhand government claimed, actually belonged to people who were alive and well.⁶ (It is possible that some of them were not eligible, but if so, they would have been a small minority – most households in rural Jharkhand are eligible, and the cancellation drive had little to do with eligibility.) MNS also find that cards not linked to Aadhaar tended to belong to poorer, less educated, and lower-caste households. The concerned cardholders were not informed, let alone told why their ration card had been cancelled. For some of them, such as the family of 11-year-old Santoshi Kumari who made national headlines after she succumbed to prolonged hunger on 28 September 2017, having a ration card was literally a matter of life and death.⁷

The authors themselves acknowledge that the RCT results “are consistent with the critique that ABBA per se caused at least some ‘pain without gain’” (Muralidharan et al. 2020b, p. 4). However, they argue that this diagnosis may be premature in the sense that ABBA had the merit of providing the foundation for a follow-up reform, “reconciliation” (the use of digital records to adjust foodgrain disbursements to PDS dealers so that they match actual distribution over time)⁸. This follow-up reform was introduced in July 2017 across Jharkhand, including the former control and treatment areas of the RCT. MNS studied the impact of reconciliation, not based on an RCT but by comparing the situation before and after reconciliation in both areas.⁹

On reconciliation

The reconciliation story (the reform, its impact, and MNS’ evaluation of it) is a little complicated – the reader is referred to the Appendix for details. The basic idea of reconciliation can be conveyed with a simple example. Suppose a PDS dealer receives 20 kgs of rice from the government every month, to be distributed to Savitri Devi. If Savitri does not get her rice (for whatever reason) in a particular month, this will show – hopefully – in the ABBA-generated transaction records. In that case, 20 kgs can be deducted from the dealer’s rice allocation in the following month, effectively preventing him (or so it seems) from siphoning off Savitri’s ration. In short, reconciliation helps to curb bogus transactions.

Reconciliation sounds like a good idea, especially in a PowerPoint presentation. In practice, it poses multiple challenges. Just to invoke one, again with an example, consider Olasi, a widow living alone for whom ABBA does not work, perhaps due to rough fingerprints. Before reconciliation, the dealer used to give her rice nevertheless – after all, he was not paying for it. Is he likely to continue giving rice to Olasi after reconciliation? You guessed it.

And what if Savitri’s dealer tries to cope with reconciliation by taking a 5-kg cut from Savitri’s monthly ration for a few months? We know that this sort of ‘adaptive corruption’ happened in Jharkhand post-reconciliation, not only blunting the reform but also leading to further exclusion in many cases (Drèze et al. 2017).

As these examples illustrate, reconciliation is not exactly a surefire policy. In particular, it requires an exacting level of preparedness – high performance of ABBA, complete transaction records, an efficient supply chain, cooperation from dealers, and more. We submit that the PDS in Jharkhand was nowhere near the required level of preparedness in July 2017. For good measure, as explained in the Appendix, reconciliation was introduced with retrospective effect – a kind of shock therapy.

The result was a predictable fiasco. Reconciliation led to chaos, further ‘pain’ for cardholders, discontent among dealers, complaints from both, and discontinuation of the reform within four months. In the first month of reconciliation, millions of people did not get any food rations at all. As the authors themselves put it:

“Overall the reconciliation policy was unpopular, drawing complaints from both dealers and beneficiaries and demands for waivers and exemptions. Effects on both leakage and beneficiaries were short-lived and largely attenuated within three months, after which the GoJH [Government of Jharkhand] temporarily rescinded the policy¹⁰.”

This ‘temporary’ rollback lasted almost a year: reconciliation was not reintroduced until August 2018, with unknown consequences since it happened beyond the study’s timeframe.

It is little consolation that, during the first few weeks of attempted reconciliation, the authors detect a reduction in leakages, in a specific sense discussed in the Appendix. This was, at best, a Pyrrhic victory over corruption, with plenty of casualties. The claim that reconciliation would work better under an alternative scenario (where it starts on a “clean slate” instead of being applied retrospectively) is very speculative, and takes us further and further away from the experimental approach advocated by the authors.

On spin

It is just possible that MNS are right in saying that ABBA could help to reduce corruption in the PDS, but interpreting this study as “evidence” that this has already happened is wishful thinking. Unfortunately, this heroic reading of their own results has severely biased the authors’ summary accounts of the study. We are referring not only to some academic presentations of the study (including their I4I article), but also, more importantly, to the way it has been presented to the media and the public.

For instance, in their Hindustan Times op-ed (very similar to their I4I article) where the findings are presented to the public, the authors assert that “leakage did go down”, without mentioning that the reconciliation attempt failed within four months.¹¹ Similarly, in a tweet advertising the study at the time of release (February 2020), Karthik Muralidharan summarises the findings as follows: “Requiring Aadhaar to obtain PDS benefits in Jharkhand led to considerable reduction in leakage, but also led to non-trivial increase in exclusion error and transaction costs for beneficiaries” (emphasis added). Tweets, of course, have little space for nuance, but this tweet is not just short of nuance, it simply does not correspond to the findings of the study. It is all the more misleading as it is accompanied by an image of the paper’s title page, with the prominent sub-title “experimental evidence from India”.

The I4I article itself is deeply flawed. Aside from repeating the claim that the study “showed that leakage did go down”, it creates an impression that this finding is based on experimental evidence, and draws the misleading “policy lesson” that the results “highlight the tradeoffs between lower corruption and higher exclusion”. In addition, as explained in the Appendix (footnote 16), the article quotes misleading figures on the impact of reconciliation.

To illustrate the critical role of interpretation in reporting the results of a study (RCT or other), we made up an alternative abstract of the MNS study:

This paper discusses some Aadhaar-enabled reforms of the Public Distribution System (PDS) in Jharkhand, based on a randomised controlled trial as well as non-experimental evidence. Consistent with earlier research, we find evidence that the PDS had significantly improved prior to these reforms, and that the imposition of Aadhaar-based biometric authentication (ABBA) on the PDS was counter-productive. Three successive disasters are documented. First, the imposition of ABBA on its own led to “pain without gain”: leakage did not decrease, exclusion increased, and so did inconvenience for PDS users. Second, the mass cancellation of ration cards not linked with Aadhaar led to further exclusion: most of the cancelled cards belonged to real – not ghost – households. Third, a chaotic attempt to “reconcile” foodgrain disbursements to PDS dealers with actual distribution, based on ABBA records, caused further exclusion, and had to be discontinued within four months. These results, read together with those of an earlier experiment of ours in Andhra Pradesh, lend some support to the view that smart cards are a better last-mile technology for the PDS than Aadhaar.

This abstract would be just as faithful to the findings as the authors’ own abstract, but it would give a very different message.

Looking beyond this particular study, there is a useful lesson here for RCTs in general. An RCT is not a guarantee of objectivity. However ‘rigorous’ the evidence may be, it still needs to be interpreted, summarised, and conveyed. It is quite easy for the evidence to get distorted or embellished in this communication process.

Appendix: The reconciliation fiasco

As mentioned in the text, reconciliation was introduced in July 2017 across Jharkhand, including the RCT’s former treatment and control area.¹² The idea of reconciliation is that, over time, the foodgrain quantities disbursed to PDS dealers should correspond to the quantities distributed by them to cardholders, not more not less. Reconciliation, of course, requires credible distribution records, something ABBA seemed to offer, unlike the earlier ‘ledger’ system (where distribution records were kept in manual ledgers by PDS dealers).¹³ During the short four-month period when reconciliation was enforced, the authors detect a decline in ‘leakages’, defined as the gap between (1) foodgrain disbursements to PDS dealers, and (2) foodgrain received from them by cardholders (the latter being estimated from a household survey, not distribution records).

It is important to understand what happened during these four months. Prior to the reconciliation period, foodgrain disbursements to PDS dealers were not based on distribution records but on the number of ration cards tagged to each ration shop. This means that, in the event where someone did not lift her foodgrain quota in a particular month (say, because of biometric failure or seasonal migration), the PDS dealer could easily siphon it off, with a little massage of distribution and stock ledgers. Post-ABBA, digital distribution records made it possible – or so it seemed – to track the dealers’ stocks. By the end of June 2017, the records indicated that the average dealer in treatment areas had more than 77 quintals of undistributed grain, roughly equivalent to a full month’s distribution.¹⁴ This was, incidentally, an embarrassing symptom of the fact that ABBA had failed to plug the leakages. For this or other reasons, the state government slammed the brakes: reconciliation was initiated with retrospective effect. This means, for instance, that a dealer who had saved a full month’s ration (according to the records) was told that the next month’s food ration would have to be distributed from his stocks – disbursement would be suspended for a month. By then, of course, many dealers had disposed of their undistributed stocks, wholly or partially.

Predictably enough, this sledgehammer move led to chaos. Some dealers simply told the cardholders that since they had not received any foodgrain for that month, they would be unable to distribute. Others managed the shortfall in disbursement (about 38%, on average, in treatment areas) partly by depriving the cardholders, and partly by using whatever undistributed stocks were still in their possession. Distribution declined, but the decline in distribution was less than the decline in disbursement, as some dealers did part with some of their undistributed stocks. There lies the suggested evidence of reduced leakages (defined as the gap between disbursement and distribution) during the reconciliation period.

This conclusion, however, is not very convincing. What happened is better described as a temporary recovery of past leakages than a sustainable reduction in leakages. Indeed, as the quote in the text makes clear, this reduction was “short-lived and largely attenuated within three months”.¹⁵

The event can be seen in a different way. Reconciliation was basically an attempt (initially at least) to transfer the cost of earlier leakages from the government to dealers and cardholders. Judging from the survey data, the axe fell more or less equally on both: over the four-month reconciliation period, 49% of the reduction in disbursements translated into lower benefits for cardholders. This involved “fairly dramatic reductions in benefits received”, as the authors aptly put it. Just to mention one dimension of the damage, in the first month of reconciliation (July 2017), “an additional 1.6 million people [13% of all beneficiaries] did not receive PDS benefits” in the treatment areas¹⁶.

Two of us happened to be in rural Jharkhand at that time, and we distinctly remember how the reconciliation move caused considerable confusion, bewilderment, and distress to PDS cardholders. In Latehar district, we personally visited some villages where people were at a loss to understand why they had all been deprived of a month’s food rations in August 2017. We also observed, at that time, how some PDS dealers tried to cope with the crisis by exploring ingenious forms of adaptive corruption¹⁷. As the authors note, the chaos and discontent were such that (1) reconciliation was very partially enforced, as the government kept granting exemptions under pressure (the damage would have been worse otherwise), and (2) the reconciliation policy had to be called off within four months.

Seen in this light, it is clear that the reconciliation attempt in 2017 was just another ABBA-related disaster, particularly in terms of further deprivation of entitlements. In short, far from qualifying the conclusion that ABBA caused “pain without gain” in Jharkhand (at least for the time being), the reconciliation story actually reinforces it.

So far, this account focuses on the former treatment areas of the RCT, which account for the bulk of the study areas (87 out of 132 blocks). The story in the control areas was much the same, except that the disruption there was less serious because the undistributed stocks, accumulated over just two months (ABBA was introduced there in May 2017), were smaller. MNS use the contrast between former control and treatment areas to argue that a “clean-slate” (instead of retrospective) reconciliation might have succeeded in reducing leakages without any adverse effects on the cardholders. This argument, however, is very speculative, relying as it does on a rather opaque extrapolation using noisy data and an instrumental variable of doubtful validity. Readers may or may not find the argument convincing (the tentative nature of this extrapolation is explicitly acknowledged in MNS 2020b), but it hardly qualifies as “evidence” that clean-slate reconciliation would work fine.

As mentioned in the text, reconciliation was reintroduced in August 2018, with unknown consequences. Perhaps it did achieve something at long last. But then, reconciliation does not require ABBA. What it requires is complete distribution records. As we have argued elsewhere, a simpler and more reliable technology like smartcards would probably serve that purpose better than ABBA (Drèze et al. 2017, Khera 2018)¹⁸. Nevertheless, ABBA is still being relentlessly promoted, indeed imposed, by the central government across the country. Could it be that it serves interests other than those of food security?

The authors are grateful to Diane Coffey, Angus Deaton, Sakina Dhorajiwala, Nazar Khalid, Sudha Narayanan, and Ankur Sarin for helpful comments. They also acknowledge the pleasure of extended correspondence, dialogue, and arguments with Karthik Muralidharan, Paul Niehaus, and Sandip Sukhtankar on the method and findings of the study discussed in this article.

Notes:

This article was published in collaboration with VoxDev.
Aadhaar or Unique Identification (UID) number is a 12-digit identification number linked to an individual’s biometrics (fingerprints, iris, and photographs) and issued to Indian residents by the Unique Identification Authority of India (UIDAI) on behalf of the Government of India.
See for example, Drèze and Khera (2017), and earlier work cited there.
MNS (2020b), Tables 3 and 4 and Figure A.1. Oddly, the authors do not present any data on leakages from their baseline survey, nor do they mention whether there was any ‘balance’ in this respect between control and treatment areas in the baseline. This is not a trivial matter, since a significant asymmetry in this respect would vitiate the RCT.
Our study was based on a June 2017 survey of 890 randomlyselected households spread over 32 villages in 8 districts of Jharkhand, and also built on a fair amount of prior experience (including earlier surveys) of the PDS in Jharkhand. The sample size was not very large, but we worked hard on data quality.
On 27 March 2017, the Chief Secretary of Jharkhand stated that “all the ration cards which have not been linked with Aadhar number will become null and void on 5th April... Nearly 3 lakh [300,000] ration cards have been declared invalid.” On 22 September 2017, the government of Jharkhand listed the mass cancellation of “fake ration cards” [sic] among its major achievements. It is not clear exactly how many ration cards were actually cancelled (the state government gave different figures at different times in 2017, ranging from 3 lakh to 11.6 lakh), but it is worth noting that the figure of 3 lakh given by the Chief Secretary on 27 March is consistent with MNS’s finding that 6% of ration cards were cancelled between October 2016 and May 2018 in the 10 study districts.
This entire episode illustrates the dangers of the ‘ultimatum method’ routinely used by the government to link social benefits with Aadhaar (Drèze 2018). To this day, the list of cancelled ration cards has not been released, despite repeated appeals to the state government.
According to Karthik Muralidharan (2020), “the point of ABBA was mainly to enable reconciliation”. Interestingly, however, reconciliation was not mentioned in MNS’s initial ‘pre-analysis plan’. The reconciliation study was based on a follow-up pre-analysis plan dated April 2018 (well after data collection had been completed).
Strictly speaking, the comparison is between the post-reconciliation situation and a counterfactual based on extrapolation from pre-reconciliation patterns.
The quote is from the February 2020 version of the authors’ NBER (National Bureau of Economic Research) Working Paper (pp. 4-5), cited in their I4I article. The same points are made in the April version (MNS 2020b). The two versions are more or less indistinguishable for our purposes, with one major qualification discussed in the Appendix.
An email widely circulated the next day (17 February) by Karthik Muralidharan to advertise this op-ed is even more misleading. It claims “substantial reductions in leakages in the PDS” based on “the first large-scale experimental evidence on the impact of introducing Aadhaar in the delivery of welfare programs in India”.
Following MNS, we use the term “treatment areas” and “control areas” to refer to these respective areas, but it is important to understand that, strictly speaking, they were former treatment and control areas: reconciliation applied in both areas, and, by July 2017, so did ABBA (the former “treatment”, extended to control areas two months prior to reconciliation).
Records maintained in manual ledgers (before ABBA) were hardly ever scrutinized. The imposition of ABBA was associated with the adoption of PoS machines to record last-mile transactions digitally.PoS machines contributed to improved record-keeping and facilitated greater scrutiny. Still, digital records can be compromised. For instance, some dealers in Jharkhand perform authentication and distribution on different days, defeating the purpose; sometimes they also tell a cardholder that her authentication has failed even when it has not. Transaction quantities (as opposed to the transaction count) are even easier to fudge – that is why katauti continues post-ABBA.
The average PDS dealer serviced about 300 ration cards in the study districts (MNS 2020b, Table 1). With standard entitlements of 5 kg per person per month, and assuming an average household size of 5, 300 ration cards would correspond to 75 quintals per month.
In fact, in the fourth month, according to MNS’ estimates, leakages were higher than pre-reconciliation trends, even as cardholder benefits remained lower (inferred from MNS 2020b, Table 11).
MNS 2020b (April version, p. 27). In the February version, it is incorrectly stated that the burden fell mainly on dealers (that was true for the first month of the reconciliation period), or in other words, that reduced disbursements translated more into reduced leakages than into reduced benefits. That incorrect statement, unfortunately, was repeated (also in February 2020) in the articles, interviews, and tweets mentioned earlier. Oddly, their I4I article made no attempt to correct the wrong impression that had been created in February; in fact, it continues to quote the incorrect figures, despite corrected figures being available at that time from the April version of the paper.
Examples are briefly discussed in the postscript to Drèze et al. (2017). Some of them involved tampering with ABBA-generated records (for example, by taking fingerprints without distributing grain). Interestingly, MNS’ survey data also suggest that, post-reconciliation, some transaction records were generated for cardholders who had actually received nothing (MNS 2020b, Figure C.2, Panel B).
Having said this, neither ABBA nor smartcards would, by themselves, detect the quantity cuts (katauti)extracted by many PDS dealers from the cardholders. These cuts, by definition, are ‘off the record’. Eliminating them requires a different approach, focused on empowering cardholders. Anecdotally (to use one of MNS’ favourite expressions), some of our friends from Chhattisgarh are baffled when they hear about the persistence of katauti in Jharkhand – the practice has virtually disappeared in Chhattisgarh (not an anecdote!), without any help from ABBA.