Synthetic Control Method: Opportunity for policy evaluation

Policy evaluation involves estimating the effect of an intervention, by comparing outcomes in units subjected to the intervention with otherwise similar units not subjected to intervention. However, this may be challenging if there is no appropriate comparison group. In this post, Karan Bhasin discusses how the ‘Synthetic Control Method’ can work in such cases, and its application for evaluating the impact of policies such as inflation targeting.

A key objective of policy evaluation is to estimate the effect of a policy intervention. Difference-in-difference (D-i-D) estimators have been a popular choice for policy evaluation when there is a clear treatment (subjected to intervention) and a control group (not subjected to intervention). The D-i-D estimator focuses on the difference in outcomes due to the adoption of a treatment (policy intervention) between two otherwise similar groups¹. When the outcome variable behaves similarly in the two groups prior to the adoption of the treatment (parallel trends), then any ex-post differences in outcomes can be attributed to the policy intervention in the treatment group. However, in some instances, an appropriate control group may not be available such as in the case of trade liberalisation, or the adoption of inflation targeting (or in general, adoption of any macroeconomic framework).

There have been several attempts to understand the effects of these policies, and these attempts have either focused on comparing outcomes in the pre-treatment and post-treatment periods; or have attempted to compare outcome variable between similar countries that did not undertake the treatment.²Differences in outcome variable is often interpreted as the effect of the policy intervention.

In this post, I discuss the ‘Synthetic Control Method’ (SCM) introduced by Abadie and Gardeazabal (2003), which allows construction of a synthetic counterfactual that provides a causal interpretation for the policy interventions. SCM does not require two key assumptions invoked by standard D-i-D estimators, which are parallel trends and no policy anticipation. SCM by construction ensures parallel trends and is flexible to accommodate instances where interventions were anticipated. The latter is important as anticipation of policy interventions can have preemptive effects on behaviour of individuals that may contaminate the estimates of treatment effects.

Let us consider the policy intervention of the adoption of inflation targeting (IT). Poland formally adopted IT in 1998 – the adoption of IT becomes the treatment and 1998 becomes treatment year. The period prior to 1998 is considered as pre-treatment period and the period after 1998 is the post-treatment period. Evaluation of IT in the Polish context could be through comparing inflation (outcome variable) in the pre-treatment and the post-treatment period. A decline in inflation in the post-treatment period would suggest that adoption of IT reduced inflation. However, a decline in inflation in the post-treatment period could also be driven by a strong declining trend in inflation. Furthermore, it is possible that global inflation could be muted during the post-treatment period. Therefore, to fairly assess the effects of IT, a counterfactual is needed. That is, what would Poland’s inflation be during the post-treatment period in the absence of IT? Synthetic control method allows the construction of such a counterfactual.

Synthetic Control Method: How it works

The central idea behind SCM is to construct a synthetic counterfactual for the treatment group using a weighted average of the control group. The control group does not receive the treatment and therefore, an average of the same can serve as an appropriate counterfactual. In the context of the example outlined above (IT in Poland), this would be taking a weighted average of countries that do not have IT. The difference between this weighted average and the actual post-treatment period would be the causal effect of adoption of IT. The intuition behind the SCM is that a combination of observations could serve as a better counterfactual than a single comparator.

To ensure that the synthetic counterpart serves as an appropriate comparator, the difference between treatment unit and the synthetic counterpart should be small for the pre-treatment period. It is possible to generate weights for the synthetic counterpart such that the synthetic counterpart displays similar dynamics to the outcome variable, and the distance between the outcome variable and a bunch of covariates for the pre-treatment period and the synthetic counterpart is minimised.

Statistical software such as Stata and R have programmes written that automatically obtains the appropriate weights. These weights are then used for constructing a synthetic counterpart that serves as a counterfactual for policy evaluation.

Using Synthetic Control Method to study the impact of inflation targeting

The ability to deploy SCM through existing statistical programmes makes it easy to implement the procedure to answer important policy questions where a counterfactual may not be readily available. This is particularly appealing while studying macroeconomic policy interventions such as the effects of reforms (Adhikari et al. 2018).

Early examples of the application of the method are of the increase in minimum age for smoking cigarettes in California, and its effect on cigarette sales (Abadie, Diamond and Hainmueller 2010); and that of German reunification and its effect on economic growth in West Germany (Abadie, Diamond and Hainmueller 2015).

In recent work, I along with co-authors (Bhalla et al. 2023), discuss the application of SCM to evaluate the treatment effect of IT on inflation and GDP (gross domestic product) growth rate. We argue that there has been a trend decline in inflation across the world. This makes it difficult to causally attribute this decline to IT. SCM allows the construction of a counterfactual for individual countries, allowing for a direct comparison of the actual outcomes in the post-treatment period with the counterfactual.

To construct an appropriate counterfactual, several structural features of the actual economy and its synthetic counterparts should match. Therefore, we derive the appropriate weights by minimising the distance between the vector of variables (inflation, per-capita GDP etc.) for the actual economy and its synthetic counterpart in the pre-treatment period. The co-movement of the outcome variable of interest (inflation) across the different units in the data is exactly what synthetic controls are designed to exploit. It is important to note that the weights are derived to match the dynamics in the pre-treatment period. Successful implementation would ensure lower errors between the actual and the synthetic counterpart in the pre-period and therefore, the assumption of parallel trends is not needed.³

In Figure 1, I reproduce charts from Bhalla et al. (2023) for Mexico and Poland, showing how the adoption of IT led to lower levels of inflation. The example for Poland demonstrates how closely inflation in synthetic counterpart follows the actual in the pre-treatment period and subsequently diverges post adoption of IT.

Figure 1. Adoption of IT led to lower levels of inflation in Mexico and Poland

Some considerations

The key advantage of SCM is that most policy interventions happen at an aggregate level such as the state, and aggregate data is more readily available. However, it is important to ensure that each of the control-group units (state or country) has not been exposed to the treatment. Doing so would contaminate the control group or donor pool from which the counterfactual is constructed. In addition, it is possible to get inconsistent weights as an outcome of the optimisation procedure. For example, it is theoretically possible for Meghalaya to get a higher weight than Karnataka while generating a counterfactual for Maharashtra. Comparison of the counterfactual constructed through these weights could pose as a potential problem in at least some instances. A solution to such a problem could be to restrict the control group to states that are similar in terms of population, etc.

Conclusion: Useful addition to policy evaluation toolkit

SCM provides an opportunity to revisit several important policy questions and debates from the past. A good example would be the debate surrounding 1991 reforms and the acceleration of India’s economic growth that pre-dated the reforms. Synthetic control allows for a closer re-examination of the causal effects of these reforms on India’s economic growth. Other useful applications of the method would include evaluating the effects of state-level interventions (such as free electricity, bus rides, cash or in-kind transfers) on development outcomes. The SCM is particularly appealing in the Indian context given the large number of states which could be used to construct the counterfactual. Furthermore, the method is appropriate when applied in a panel context (that is, over time) on aggregate data (such as state-level labour force participation rates or other development indicators). This is particularly useful when micro-data are not available, constraining the use of other estimators. Consequently, the SCM serves as an important and useful addition to the toolkit for policy evaluation for both applied macro- and micro-economists.

The views expressed in this post are solely those of the author, and do not necessarily reflect those of the I4I Editorial Board.

Notes:

A recent example of D-i-D in the Indian context is Sinha and Laha (2019), which compares consumption expenditure across rural and urban regions, pre and post a food price surge in 2008.
A difference in outcomes in the pre-treatment and post-treatment periods could be driven by an existing trend rather than the policy intervention itself; a similar country does not serve as a perfect control group, as it might have factors other than the policy intervention that are unique to it and impact the outcome variable.
Errors refer to the root-mean square errors, which quantifies the average magnitude of error in predictions.