Building resilient education systems: Evidence from five countries

With education disrupted due to the Covid-19 pandemic for over a billion children worldwide, this article emphasises the need to create an education system that is resilient and maintains continuity despite such shocks. It describes an intervention conducted across five developing countries to leverage existing, low-cost family-owned assets – mobile phones – and summarises its effect on children's learning outcomes and wellbeing. It reveals how the original proof-of-concept can be successfully scaled across various country contexts, without showing diminishing returns.

Education systems need to withstand frequent shocks, including conflict, outbreaks of disease, natural disasters, and climate events, all of which routinely close schools. In a recent paper (Angrist et al. 2023), we studied multiple scalable models of remote instruction for primary school children implemented during Covid-19, which disrupted education for over 1 billion school children worldwide. This study presents results from large-scale randomised trials evaluating the provision of education in emergency settings across five countries: India, Kenya, Nepal, Philippines, and Uganda.

Education in emergencies

More than 2 billion people live in countries affected by emergencies that frequently disrupt education. For example, monsoons and flooding in 2017 affected over 10,000 schools across Malaysia, India, Bangladesh, and Nepal, while Pakistan experienced school damage in 2010 and during a recent crisis in 2023, disrupting 34,000 schools. In the Indian state of Manipur over 4,500 schools were shut down for more than two months from May to July 2023 due to ethnic conflicts. Education systems need to be resilient and able to withstand these shocks to continue to provide education.

This study contributes to a nascent experimental literature on education in emergencies, as well as a growing literature on education at scale. Recent studies show the extent of the scaling challenge, with many social programmes that initially worked in a proof-of-concept phase no longer delivering impact when scaled up or delivered by governments (Mobarak 2022, List 2022). In contrast with much of the existing scale literature, the results of our study identify an approach that can work across contexts and within government systems. This study also contributes to the global education literature and approaches to address a global ‘learning crisis’, by identifying a highly cost-effective approach to improve learning.

The interventions

Interventions evaluated in this study took place in the context of Covid-19 school closures across India, Kenya, Nepal, the Philippines, and Uganda¹. In total, over 16,000 households were enrolled in this multi-country evaluation. In each household, a primary school student and a caregiver were identified to receive the intervention. Students were enrolled in grades 3 to 5, with one exception in Kenya where students were in grades 1 and 2. In India, students of grade 5 in Telangana Social Welfare Residential schools were enrolled. The trials took place between December 2020 and July 2022.

The evaluation included multiple programmes and delivery models, such as through NGO instructors and government teachers. The study leveraged mobile phones – a high-access, low cost and scalable way to reach students and their caregivers when school is out of session – to provide various educational interventions over the course of eight weeks. One treatment included educational aids, such as numeracy content provided weekly, as well as nudges to engage in educational activity sent via SMS. A second treatment added weekly one-on-one 20-minute phone call tutorials. Phone calls covered foundational numeracy and aimed to target instruction to student learning levels through light-touch, high-frequency assessments conducted on a weekly basis. Using the assessment, instructors led students through foundational numeracy practice problems to help them master addition, subtraction, multiplication, and division. Calls were made to caregivers who invited the child in their household to join the programme, and would then hand them the phone or use speakerphone for the tutorial session. The implementation ratio of students to teachers was approximately 20:1 for teachers implementing the programme full-time, and 5:1 for teachers implementing the programme part-time.

Data on outcomes were collected through endline surveys which took approximately 30 minutes to administer and included approximately 20 questions. These questions included a learning assessment, child well-being, parental engagement in educational activities, and parental perceptions of their child’s learning. A portion of the survey was conducted with the parent, and learning outcomes were collected by directly assessing the child over the phone. Endline surveys were conducted a few months after the programme ended.

This set of five trials sought to scale an approach first tested in Botswana during the early part of the Covid-19 pandemic in 2020 (Angrist et al. 2022). The original Botswana trial tested the impact of a phone-based tutoring programme that had been developed during the first few months of Covid-19, and found the programme to be cost-effective, resulting in a 0.12 standard deviation² improvement in learning outcomes.

Results of the five-country study

i) Increased learning in numeracy across contexts and delivery models: For the combined phone and SMS group, there was a 0.327 standard deviation increase in average numerical operation gains across all 5 countries. For the SMS group alone, we found smaller but still statistically significant improvements in learning, of 0.083 standard deviations. These effects are even larger in these multi-country studies than in the Botswana proof-of-concept study. This result contrasts with prior literature showing that proof-of-concept studies rarely scale successfully to new contexts (List 2022) or experience diminishing returns (Araujo et al. 2021). Rather than finding diminishing returns, the study found that results improve as the approach is adapted, scaled, and tested across contexts.

The study also compared government teachers with NGO programme delivery (green bars in Figure 1), with results showing both government teachers and NGO instructors are effective at improving student learning – government teachers improved learning by 0.314 standard deviations across both contexts, and NGO teacher aides improved learning by 0.263 standard deviations. The similarity in effect, with no statistically significant difference between them, demonstrates that government teachers can effectively implement these educational methods in emergency programmes.

Figure 1. Learning outcomes at scale across five countries

Notes: i) Data are presented as treatment effects relative to the control group, expressed in terms of standard deviations for comparable units. ii) The light grey and blue lines indicate 95 percent confidence intervals.

ii) Improvement in secondary outcomes like parents’ beliefs and non-cognitive skills: In addition to learning outcomes, we examined the impacts on beliefs about learning and demand for the intervention. Caregivers state that they think their child’s learning has progressed with large and statistically significant effects as a result of phone call tutorials. These results build on a growing literature exploring the importance of parents knowing their child’s learning level, enabling them to better support their education (Bergman 2021). Caregivers have displayed a high demand for the programme; in the phone call and SMS tutorial treatment, 100% of parents express interest in receiving the programme.

The study also found sizeable effects of the programme on non-cognitive skills, with 6.2 percentage point gains in ambition³, and found positive effects on measures of child wellbeing.

iii) Cost-effectiveness: The study’s estimates suggest the cost for one round of the phone call and SMS tutorials is about US$12 per child on average. In addition to being low cost, the approach has low procurement requirements since it uses existing household assets (mobile phones), a particularly attractive feature for governments.

Conclusion

Despite heterogeneous country contexts, our results show that the effectiveness of phone call tutorials can be scaled across diverse contexts. The study found consistently large and robust effect sizes on learning, with average effects of 0.30-0.35 standard deviations. Further, the intervention proved similarly effective when delivered either by NGO instructors or government teachers, indicating scalability within various systems. These effects were also highly cost-effective, delivering up to four years of high-quality instruction per US$100 spent.

Given the growing rate of mobile phone ownership globally, phone-based tutoring programmes like the one studied here have the potential to maintain schooling continuity and accelerate learning alongside emergencies. The low-cost, high-access, and ease of implementation of phone tutoring could build more resilience into education systems, enabling systems to better withstand frequent shocks, and, more generally, utilise cost-effective approaches to address a persistent global learning crisis.

Notes:

Additional information on each country context can be found in Angrist et al. (2023).
Standard deviation is a measure that is used to quantify the amount of variation or dispersion of a set of values from the mean value of that set.
Measured by adapting a questionnaire used by Carlana and La Ferrara (2021).