Data lies at the heart of research and policymaking in the social sector. But where does this data really come from? In this note, Prerna Mukharya (Founder, Outline India, Track your metrics), shares experiences from fieldwork in different parts of the country and presents insights into the complicated and challenging process of data collection.
One goes through pages of numbers each day, tucked away in our Whatsapp forwards, the op-eds we read and write, the reports we skim through, and the slides we view at conferences. Pause and consider the origin of those numbers.
These are numbers that were perhaps collected from public schools in Jharkhand, that were partly under water in the month of August, or an overcrowded public health centre in Odisha where the women line up with their newborns, to collect cash incentives due as incentives for institutional births. But who decides whether the numbers were collected from the right stakeholders, whether the enumerator hit the right option on their tablet while making the entry, whether they made it to the right household, whether they cared enough to understand the answer and probe further?
Doing fieldwork in India is akin to what marvel comics are made of: It is adventurous, unpredictable, and there is always a hero and an evil force.
Moving away from paper-based data collection has significant advantages but the enumerators’ subjectivity in recording answers during data collection, will continue to alter our perceptions and policies.
Based on field stories1, I share my insights into human behaviour, and how our policies must consider the counterfactual and think of all things alien, before sealing the deal. I hope to bring out how our culture, the everyday of events, news, political discussions, and online debate feed into the larger political narrative and affect data and hence policy outcomes.
Sensitive questions and timing
About two years ago, on one field mission we found ourselves in the remote reaches of Karnataka. We were investigating the effects of a certain government policy on healthcare expenses at the household level. Our questions were detailed. Our goal was to help the central government figure how it could widen the ambit of the policy from its current, below poverty line (BPL) stature to include subsets from the above poverty line (APL) groups. We asked about the money that the beneficiaries had spent on medicines in the last year and how their spending habits had changed as a result.
Owing to exposure to social media, and messaging from government and local NGO bodies, people are inclined to respond to surveys in a specific way, for example, state they are 18 if they are married (irrespective of their actual age), for they know marriage below the age of 18 is illegal for girls. On certain occasions, using design research or asking the same question phrased differently, can help the researcher arrive at a better understanding of what the respondent might believe or follow in practice.
During the survey, the respondents were not pleased about how far we would push them. We would ask the same question twice, with different wording, negatively loaded on certain occasions to irk out a response that would triangulate and ensure us of a reasonably accurate picture of the reality.
This was a painful process. For instance, do you (the reader) remember the amount of money you have spent on medical expenses in the last six months, much less, the last year? Would you as a woman or a man, be able to recount horrific details of a stillborn, or the death of a newborn? We had to ask such questions, across thousands of people. People would sometimes break down in the midst of the survey. We would take a break and then we had to be back, asking more questions.
People got angry on certain occasions. I remember one instance, it was the festival of Shivratri that day, an important one in the state. A day of religious festivities when people would stay home. We reached around 1 pm huddled up in two big cars – a team of five researchers, eight enumerators to pilot the survey tool. We assumed by then the religious activities for the day would stand closed or put on hold until later at night. We were wrong. This was an impoverished community, and alcohol was a part of the daily routine. On the occasion of Shivratri, it was apparently legit and expected that people would be inebriated by the afternoon, women folk included. Our arrival was not well-received and people lashed out. They thought we were unkind, and they assumed surveys would not help cure their sick or improve their lives. Much to our dismay, we had to leave the village.
We carry out large-scale data collection exercises, over hundreds of villages, households, and people. Sometimes our surveys are long and tedious and long, 60-90 minutes, and longer in certain cases. We may have to take anthropometric measurements or collect soil samples or play games with children or adults to understand behaviours or test cognition.
Most rural households welcome us in, but their livelihoods could be time-sensitive. A woman for instance must look after her husband, in-laws, send children to school, clean up, feed the cattle. The men may be putting in long hours in the field during sowing or harvesting season. School teachers may be on election duty. Timing is everything, when in the field. It affects the quality of responses, ranging from prompt and passionate early in the morning to dismissive and convoluted when the respondent is tired.
As luck would have it, the 2019 general elections were soon to be underway when we embarked on a field study on nutrition. Before the elections however, the model code of conduct (MCC) kicks in. The MCC allows the Election Commission to regulate the acts of parties and candidates alike, in the interest of fair elections. It ensures the ruling party does not get an unfair advantage and hence no new measures, policies, or acts must be rolled out at this time. This is also a politically charged time of the year and hence community work, surveys, or interventions can be misconstrued as acts to influence the voters in a certain direction. It might make certain sects happy and render others aggravated. A study focusing on food subsidies sponsored by the Bihar government and an international donor had to be put on hold for this very reason.
On a different occasion, for a study on gender, we were to explore various facets of childcare at the household level in Madhya Pradesh. We talked to the mothers extensively about division of financial resources among children, girl and boy. We discussed nutrition and daily calendars for each child based on gender. Our survey awarded pen sets to these households as a sign of gratitude for their time and patience. We assumed this to be a neutral and useful gift for everyone. Well, we were partly wrong. Firstly, word spreads like wildfire in villages that are hard to reach, for most people know each other. Visits from outsiders are rare and special occasions. The outsider is both revered and scrutinised. For the purposes of our study, households not part of the sample, were not given the gift, that is, the pen set. A day later our team of enumerators were met with a sub-group of unhappy individuals. One of these individuals was the nephew of the sarpanch (village head). To the villagers his station commanded subservience and our team was happy to comply. The piece of news that had reached him was as follows – we were a team from Delhi who were visiting to influence women and how they behaved with their husbands. More so, the gifts were intended to buy their vote. A few phone calls and two hours later, our lead supervisor was summoned to explain what was precisely the case. A pen set worth Rs. 18 was not to be doled out without the express permission of the village leaders.
The example above brings to light how power equations work in certain locales. When we must interview women, we ensure the family elders are not around. Women in certain communities do not enjoy freedom of expression and must adhere to certain tacit rules while sharing opinion. Their votes are not theirs to cast. Their role in financial decision-making is limited.
On most occasions, we find an open space within the household but away from the ears of other family members. Of course, this is not easy to achieve in numerous cases.
Take the case of the Swachch Bharat Mission (clean India campaign), wherein ‘saving the dignity’ of women was the jingle the authorities came up with. In households, where men enjoy absolute powers, women are considered as those who must be saved, rescued and in many cases are considered replaceable. When we tested for recall among households and communities, to figure whether the campaign was working – whether people understood why hygiene is crucial, why toilets must be built and used – we found that the campaign appealed to people who were more concerned about women exposing body parts in the open to defecate, over and above health risks. The recall for the jingle was high.
I will steer clear of the whether such messaging is appropriate or sets things right, but it seemed to be working on certain counts.
In July roughly two years ago, we found ourselves close to the Nepal and Bhutan border, not too far from Bengal. It was pleasant weather and the rain gods obliged. We were working with youths, boys and girls under 17 years of age. Many of these girls were young mothers; some of them had two children.
Our study focused on sexual and reproductive health among young girls and boys and we came across communities where no one knew about the concept of birth control. As our researchers conducted interviews and focus group discussions, our filters of age and number of respondents did not allow us to include everyone. We had a field staff of 15 to engage the onlookers and members of the community. However, one particular woman did not appreciate being left out. It was somehow assumed that we were there to enlist families for handing out of government subsidies. Those who were part of discussions would be at the receiving end and the rest would have to wait. This was not acceptable and a rumour soon spread. As we worked in an empty school compound, a mob of 200 people surrounded us. They were not happy. The village leader came to our rescue to assuage concerns. However, on the subject of financial incentives and subsidies, his power and might did not cut it. Forty minutes later, our team had to pull out. Much to our dismay, we eventually had to drop that village from our sample. No one emerged a winner here.
India’s problems are basic and solutions to those problems require scale, vision, and technology to achievable sizeable impact.
Data is the epicentre for research, the foundation of policymaking and the data collector that elegant juggler who must dictate how we collect information, how we figure out what is needed on the ground and to a large extent how we spend our funds. I urge you to consider the study context, field realities, and qualitative research before you run some quick numbers. In India, sometimes the ‘grey’ of policy outcomes is where our solutions lie.
- Certain names, locations, and timelines have been altered to protect intellectual property. All stories have been borrowed from Outline India’s field assignments and other works.