While citizen-led assessments of children’s learning have been applauded for building awareness around the issue, observers have pointed out that information does not automatically galvanise action. In this article, Rukmini Banerji, of the education NGO Pratham, contends that citizen-led assessments can lead to citizen-led action for improving learning by making the problem visible, engaging with people, and demonstrating how change can be brought about.

The recent publication of a report evaluating the family of citizen-led assessments has led to a number of blogs that are looking closely at the learnings from these efforts and trying to understand the implications of what has been achieved and what has not^1.

Citizen-led assessments of children’s learning

The flurry of interest is welcome. This family of assessments now stretches around the world from south Asia across eastern and western Africa and all the way to Veracruz in Mexico. India led the citizen-led assessment movement; initiated by Pratham 10 years ago, ASER (Annual Status of Education Report) was the first of its kind in the country and globally². ASER is a genuine model of South-South learning and cooperation and has been referred to as “one of the most internationally influential educational initiatives of the decade”.³

At the core of the work is a basic question: Enrolment levels are very high in many countries. Children are going to school, but are they learning? To search for answers, ordinary people fan out into villages and communities, to reach households and children. In the presence of their families and others, children read simple text and recognise numbers and do basic computations. Each child in a sampled household gets one-on-one attention. And over the heads of the children, between parents and neighbours there are conversations about the status of schooling and learning in the community. This mammoth exercise, covering more than a million children a year, happens annually across India, Pakistan, Kenya, Uganda, Tanzania, Mali, Senegal and Mexico (soon to be joined by Nigeria and maybe Bangladesh).

While applauding the role of the citizen-led assessments in building awareness around the issue of children’s learning, observers point out that these measurements have not been successful in raising children’s learning. They argue that information is not enough and that “reporting on woeful learning levels will not automatically galvanise action”.⁴

The Pratham model: Combining evidence-gathering and intervention

Well before ASER started in India, we knew that only providing information does not lead to action for improving learning. In 2004, J-PAL conducted a randomised controlled trial (RCT) experiment on a Pratham intervention in Jaunpur district in rural Uttar Pradesh. We learned several fundamental lessons from that work. First, availability of information led to extensive local participation in heated discussions, but no action (collective or individual) was taken either in the community or in the school to improve children’s learning. However, when live ‘demonstrations’ were provided about how to teach children, many villages and volunteers came forward to ‘act’. The lesson: along with seeing a problem and realising that there is a need to act, people seemed to need a clear demonstration of what can be done and how to do it. In a context where outcomes have been low and ‘stuck’, first-hand experience of how children’s learning could improve was essential to make people believe that change was possible. By the time ASER was launched in 2005, Pratham had evolved a frugal and fast method for enabling children to learn to read and to do basic maths. The method was effective with children who were seven or eight or older. Community volunteers could do it and teachers too, in any Indian language. We had confidence in the assessment and in the intervention as we had already implemented this approach on significant scale in India. Learning from our past experiences, we ensured that ASER – the evidence-gathering effort and the intervention (now called the Read India campaign) fed into one another.

Over time, Pratham has developed two models to improve children’s learning – one for communities and a similar one suited for government systems. In both, the ‘catalytic’ element incorporates elements of live demonstrations, participation and learning by doing. Recently, rigorous independent impact evaluations have been done on both models with very promising results.⁵ In the first model, a Pratham team member works with village volunteers to run “learning camps” with children in Grades 3 to 5. A learning camp is an intensive burst of activity that lasts 8-10 days and is repeated several times over the next few months. Using the ASER reading tool, children are assessed and then grouped by level rather than by grade and taught for a few hours a day. Each group uses appropriate methods and materials for instruction. As children progress, they move into the next group. The ASER tool is used to explain the goals of the intervention to parents and community members, and to track and review children’s progress. Within 100 hours, over 75% of locations where this has been tried have succeeded in getting more than 70% children able to read simple texts. (This has happened even where baselines were low and more than 85% children could not read sentences before the intervention began). In the last two years itself, the ASER method of assessment has been used as the first step to initiate local action to benefit over 1 million children in India alone. These large-scale demonstrations have helped many others see what is possible.

Pratham’s second model is one in which this “teaching-at-the-right-level” (TaRL) approach is used to catalyse action with government school systems. Seeing actual examples of how children are learning, several districts and states in India decided to move on this direct path from assessment to action. Using a variety of Pratham-government collaborations, close to 5 million government school children participated in such programmes in the last school year in India.

The links between ASER and Read India are direct; both initiatives are integrally linked to each other. Whether at the village level or at the state level, ASER helps people understand the problem and Read India enables village volunteers and teachers find a solution. To make parents and communities believe that children can learn, large-scale demonstrations are done so that people can see for themselves that significant and substantial change is possible in a relatively short period of time. To shake up a huge lethargic education system that is just about waking up to the importance of learning outcomes, a massive effort is needed to catalyse energy and commitment. Fuelling the changes at the local level and at the system level requires gradual building up of conviction and capability. For the last 10 years and more, Pratham has been very busy doing this on scale.

The power of visibility, engagement and demonstration

In the development world, there is a lot of talk around the need for ‘transparency’ and ‘accountability’. But the lens with which we viewed the situation in India when we started the ASER initiative was a bit different. Like many other countries, India had been focussed for many years on the ‘visible’ issues of access and enrolment. ASER, and other citizen-led assessments which followed in country after country, enabled common people to feel and ‘see’ a new problem – the challenge of children who are in school but not learning. If you do not know there is a problem, you do not need solutions. It is only when the problem is visible, when you can see it, that it becomes real and alive and demands a solution. At the ground level, it is by involving hundreds of ordinary people in uncovering the problem, by engaging with thousands of parents (many of whom are illiterate) that large-scale awareness began to grow. At this stage, it was visibility rather than transparency; engagement and demonstrations rather than accountability that seemed to be the key levers for moving learning into the centre of the stage.

At a higher level, many cogs are blocking the wheel. First, to date, outcomes have not been the focus of policy or practice. Moving learning from the shadows of the sidelines to the centre of the stage is no mean feat. Second, and perhaps more important, in India, and in many other countries, a culture of measurement is to be built. Historically, data and evidence have not been the basis on which decisions are taken. The capability to generate reliable information using systematic and rigorous methods is low; the capacity to analyse data is lower still. Weakest of all is the ability and the appetite for digesting evidence and translating it into meaningful action. It is in this kind of challenging scenario that citizen-led assessments have made substantial headway. They have succeeded in drawing national attention to a new problem based on information collected in a new way. It is unprecedented to have this ‘climate change’ in such a short period of time. Much much more needs to be done to shift mindsets and to develop mechanisms for evidence led decision-making and action.

The movement of citizen-led assessments has only just begun. But even early on this journey we are already beginning to see what is needed for change. We need simple tools that help ordinary people understand problems. Engaging leads to understanding. The tools need to lend themselves easily to action. The momentum generated by engagement can be translated into local action. It has been done in India. Pakistan also has linked its ASER to action. But large-scale demonstrations of how to bring about change and extensive hand-holding are needed. This is what leads to the conviction that change is possible. This is also what leads to the building of capabilities to make change happen.

Citizen-led assessments have a very strong potential to lead to citizen-led action for improving learning. Together, in the next few years, let us explore its power.

This article first appeared on the RISE Blog: https://www.rise.ox.ac.uk/content/building-movement-assessment-action

Notes:

1. Dana Schmidt of the Hewlett Foundation wrote about the citizen-led assessments (http://www.hewlett.org/blog/posts/how-ordinary-citizens-transformed-education-agenda). Also from the Hewlett Foundation, Pat Scheid reported from Senegal (http://www.hewlett.org/blog/posts/what-data-revolution-looks-ground). R4D’s (Results for Development) Nathaniel Heller wrote about the recent evaluation report of the citizen-led assessments (http://www.opengovpartnership.org/blog/nathaniel-heller/2015/07/02/why-citizen-led-assessments-educational-outcomes-matter#sthash.zrwYx99O.dpuf). ACER (Australian Council for Educational Research) in Australia talked about ASER (Annual Status of Education Report) in India (http://rd.acer.edu.au/article/citizen-led-educational-monitoring-shows-promise).
2. Pratham is a non-governmental organisation (NGO) working with children and youth in India. ASER stands for Annual Status of Education Report and is conducted by ASER Centre, the autonomous research and evaluation unit of Pratham.
3. Bangay, C (2015), ‘Why are citizen led learning assessments not having an impact on home soil – and how can we change that?’, World Education Blog, EFA/GMR, EFA Report, 2 June 2015.
4. Bangay, C (2015), ‘What role for Citizen Led Learning Assessments? – Moving beyond Measurement’, World Education Blog, EFA/GMR, EFA Report, 9 July 2015.
5. For the evaluation in Haryana, see: http://www.povertyactionlab.org/evaluation/improving-learning-outcomes-through-government-school-system-india; for the evaluation in Uttar Pradesh, see: http://www.povertyactionlab.org/evaluation/using-learning-camps-improve-basic-learning-outcomes-primary-school-children-india