Why doesn’t anybody know if Swachh Bharat Mission is succeeding?

In 2014, the Prime Minister announced a goal of eliminating open defecation by 2019. In this article, Coffey and Spears, contend that now almost two-thirds of the way through the Swachh Bharat Mission, nobody knows whether it is succeeding because there is no credible, independent survey that can offer a useful estimate of the fraction of rural persons defecating in the open.

In 2014, the Prime Minister announced a goal of eliminating open defecation by 2019. Doing so would have required the slow pace of decline in open defecation to have accelerated, starting in 2014, by more than a multiple of 12. This would have been three times as fast as the fastest five-year decline in open defecation ever recorded in any country. (That record decline, which some experts believe is overestimated, goes to the 17-percentage-point decline in Ethiopia, a country with a much smaller population than India’s, and which still has not eliminated open defecation altogether.)

So, we should not be surprised that, now almost two-thirds of the way through the Swacch Bharat Mission (SBM), it appears very unlikely that open defecation will be eliminated by 2019. This was not a realistic goal in the first place.

But this does not necessarily imply that we should be disappointed. The diseases spread by open defecation in densely populated rural India are so threatening to the survival, health, and development of children that even a modest acceleration in decline of open defecation could represent a substantial improvement in well-being.

In fact, the best rural sanitation policy probably is not one that pretends an unachievable goal is achievable. A better policy would make serious plans that are both ambitious and realistic, strategising pragmatically to turn resources into feasible progress towards a hard problem.

Is that what we have? Has the SBM made meaningful progress towards accelerating the decline of open defecation?

Nobody knows.

SBM is not measuring open defecation

Nobody knows because there is no credible, independent survey that can offer a useful nationally representative estimate of the fraction of rural persons defecating in the open. Indeed, there is not even a cross-sectional estimate for a point in time after the start of the SBM, let alone a consistently measured data series to assess the pace of decline.

This is not to say that the government is not putting out numbers about sanitation. Indeed, data are collected, tabulated, and presented, but the resulting numbers do not teach us about the decline of open defecation. One reason is that the government’s monitoring system does not track open defecation - it tracks funds spent on latrine construction. For reasons we describe in our book, few people in villages want the pit latrines provided by the government, so in many cases funds spent on latrines do not result in functional latrines. Pits may never be dug and superstructures may turn into storage sheds or bathing areas.

Another reason why tracking funds spent on latrine construction tells us little about progress towards reducing open defecation is that even where funds translate into functional latrines, latrine ownership does not imply latrine use. In earlier research, we found high rates of open defecation among people who own government latrines (Coffey et al. 2014). Other studies have found the same (Barnard et al. 2013 , Clasen et al. 2014 , Coffey et al. 2017). If measuring open defecation is the goal, the government should be asking people carefully worded and respectfully posed survey questions about what they do.

Asking the right question

The government could launch a survey to measure open defecation. Indeed, many small-scale surveys done by independent research teams have measured open defecation in different parts of the country. What we have learned is that asking rural people where they defecate – and getting accurate answers – has its challenges. But it is nevertheless feasible, and not particularly costly.

A first step towards measuring open defecation is asking the right question. Our analysis of several sanitation surveys reveals some general principles on question wording. We recommend a question that has:

Balance: A good survey question explicitly permits both open defecation and latrine use as answers.
Disaggregation by person: A good survey question asks about each person individually, in the order they are listed on a household roster.
Specific, recent time frame: A good survey question refers to a short-term, specific time frame. People may feel more comfortable admitting to open defecation if saying ‘yes’ for now is not admitting to ‘yes’ for always.

An example of a question which incorporates balance, disaggregation by person, and a specific time frame is: “Yesterday, did Dean defecate in the open or did Dean use the latrine?”

The best approach to asking this question starts with introductory text that makes the respondent feel comfortable and normalises both possible answers. One example is: “I have been to several villages like this one, and I have seen that some people who have latrines use them, and some people who have latrines defecate in the open.” Then, as above, the surveyor asks about each person individually: “What about Dean - yesterday, did Dean defecate in the open or yesterday did Dean use the latrine? And Diane - yesterday, did Diane defecate in the open, or did she use the latrine?”

Unhurried, respectful interactions, and smaller sample sizes

Of course, simply writing a well-worded question on a survey form does not guarantee respondents will tell surveyors the truth about where they defecate. Respondents’ answers to survey questions depend in part on what they think the person asking the question wants to hear. Many people living in villages have some idea that the government would prefer that they use toilets. It is all too easy for the government’s representative – likely a well-dressed, urban man with papers and a clipboard – to give the impression that he expects rural people to use latrines.

The question wording that we described above is useful for slowing down an interaction between surveyor and respondent, and creating the impression that the surveyor actually wants to learn the truth. The opening text, which recognises that households that own latrines may have some people who use it and others who defecate in the open, makes it easier for a respondent to ‘admit’ what she thinks the surveyor does not want to hear.

At least as important as getting the question wording right is training surveyors to be patient and respectful in their interactions with respondents, and holding them accountable for learning the truth. Because of the training and monitoring required to ensure that unhurried and respectful interactions between surveyors and respondents happen consistently, it is harder for large survey teams to accurately measure open defecation than for small ones. Yet, the survey efforts that the SBM has commissioned have large samples sizes meant to be representative at the district level. Smaller sample sizes would be sufficient to measure SBM’s progress in the country as a whole and in the most important states.

It would be better to accept more sampling error as the price of accepting less non-sampling error. Sampling error is the error that comes from the random variation inherent in drawing a small sample. It is theoretically well understood and can be statistically quantified. Non-sampling error is the error that comes from poor measurement; its magnitude is anybody’s guess. A smaller survey with a more manageable sample size would have more sampling error but would permit the kind of training and monitoring of surveyors that would yield higher quality measurement.

Measuring SBM’s progress

None of these are deep problems. They would all be entirely surmountable, for a small fraction of the spending on the SBM, by a political decision to commit to measuring open defecation, whatever the numbers may reveal. Taking opportunities to prevent some of the (we estimate) hundreds of thousands of deaths each year due to poor sanitation is more important than the potential embarrassment of unsanitised statistics.

This post is adapted from a note in Coffey and Spears’ book, forthcoming in July 2017 winner of the 2017 Joseph W. Elder Prize in the Indian social sciences. Find out more about the book here