Lessons Learned from Applying Computer Assisted Experimental Designs

1. Introduction

As a means of increasing efficiency, stratification on observed covariates is commonly used. An alternative, or complement, to stratification that has received attention lately is to utilize modern computational capabilities for finding assignments with balance in observed covariates (see e.g., [3,10,11,12,13,14,15,18]). This paper contributes to the methodological literature on computer assisted designs in providing guidance of how to make use of these new designs in practice. More importantly, the paper provides numerical evidence on efficiency gains on, more specifically, the `Mahalonobis based rerandomization design’, suggested in [18], based on real world data. The numerical exercise allows for a discussion of advantages and disadvantages with this specific design in applications.

In [18], assignments are repeatedly randomized until obtaining an assignment whose Mahalanobis distance in covariate means of treated and control units is smaller than a given threshold. This assignment is then the treatment vector in the Mahalonobis based rerandomization design. According to our understanding, ours is the first study in which any of these new type of computer assisted designs are applied. An important conclusion from the exercise is that regression-based post-stratification on relevant covariates might be a reasonably efficient alternative to the Mahalonobis based rerandomization design. The results should be of interest for especially analysis of experiments in organizations where it is costly to set up a computer assisted design.

The background of the experiment is that the Swedish Tax Agency was interested in analyzing the possibilities of estimating the effect of a low-cost intervention in a digital platform on the subsequent tax compliance of Swedish taxpayers who, according to the so called Common Reporting Standard (CRS), had foreign dividends during the income year of 2018.1 Although the direct cost of sending a digital message is low, the number of treated was limited to 500 individuals. The reason for this decision was uncertainty on behalf of the team responsible for sending the message about how the taxpayers would react to the message. In particular, there were concerns about potential service desk overload. Furthermore, there is no information on whether the recipient has read the digital message or not. In combination with the non-intrusive nature of the message, the intent-to-treat (ITT) effect size cannot be expected to be large. For these reasons it was important to consider a more efficient experimental design than a completely randomization design.

As the Swedish Tax Agency was also interested in analyzing heterogenous effects by gender and previous tax compliance we decided to stratify the experiment on gender and being compliant in 2018. Six covariates, deemed to be important in reducing the heterogeneity, were then used to further reduce the imbalance using Mahalanobis based rerandomization. The population considered for the experiment was 2,697 individuals. Instead of choosing all remaining as controls, only 500 were chosen as controls as this choice provided the opportunity for analyzing the efficiency gains from the experimental design, deemed important by the tax office for future implementations. The design was built on a random sample of 1,000 individuals from this population.2 The digital message was sent to 500 of these individuals in March 2020, about six weeks before the final day to file income taxes for the income year of 2019.

The results from the experiment show no statistically significant effects on declared capital income or final taxes paid. However, the point estimates for capital income are larger in magnitude among the previously non-compliant taxpayers and this is true for both women and men. The only case of a confidence interval that does not cover zero is for non-compliant women. Given that this is the group for which it was expected to have the largest effect on compliance from receiving the digital message (see [2]), this was seen as a useful result by the Swedish Tax Agency.

The paper proceeds with a description of the study sample, the intervention, and the outcome data. The experimental design is described in Section 3 and in the following Section 4, we discuss the analysis and the results from the experiment. Section 5 provide the results from the simulation exploring the gains from the chosen experimental design. The paper concludes with a discussion in Section 6.

2. The Intervention, Study Population and Outcome Data

The intervention consisted of sending the following message to the treated individuals and leaving the controls untreated:

Declaring foreign dividends/interest

Hi!

We get many questions on how to declare foreign income and have therefore developed a new online app in order to make this easier.

The Swedish Tax Agency has obtained information from a foreign tax authority that you have received dividends or interest from abroad during 2018.

If you have received dividends or interest from abroad also during 2019, you can use the online app when you file your taxes. The app will help you with the correct amount to file and how much foreign tax offset you have the right to claim.

You can find the app here:

https://app.skatteverket.se/klient-sifu-segmentering/

Sincerely,

The Swedish Tax Agency

The message was sent in Swedish (see Appendix A for the original) and the message in English above is our translation. This message was sent in digital form to the treated taxpayers via a so called digital mailbox. In Sweden, about half of the population above 16 years of age have a digital mailbox.3 This is a free service making it possible to receive mail in digital form from Swedish authorities and some large private firms. With regards to taxes, having a digital box implies that all communication from the Swedish Tax Agency that otherwise would have been sent as paper mail is sent digitally in a secure app. This includes pre-filled tax returns as well as various other messages. Taxpayers can file their income tax declaration securely in the app.

The online app mentioned in the message does not require login and is best described as a specialized calculator. The user fills in the type of foreign income (dividends or interest), amount, currency, date of receiving the amount, country where the income was received, and if applicable, the amount of foreign tax paid. Upon clicking Calculate, the app converts the amount to SEK, calculates the foreign tax offset, and indicates the specific tax declaration boxes where the amounts should be filed.

The message was sent to each treated taxpayer’s digital box on 25 March, 2020. This is one week after it became possible for taxpayers to send in their income declarations (17 March, 2020). Unfortunately we have no data on the share of taxpayers who sent in their declarations between the 17th and 25th of March. However, the absolute majority of taxpayers in Sweden file their tax returns during the week preceding the deadline (4 May, 2020). Thus, in our opinion any attenuation of the impact from early filing is non-existing or minor.

In addition to measuring the effect of receiving the message on the taxpayers’ tax compliance, it would have been of interest to assess whether receiving the message had an effect on using the online app mentioned in the message. However, since the app does not require the users to log in, this is not feasible.

Finally, as mentioned in the introduction, we have no means of knowing whether the taxpayers who were sent the digital message actually read it. Therefore, we can only draw inferences on the ITT effect.

2.1. The Study Population

The original population consisted of 10,344 individual taxpayers each of whom had a financial account with accrued foreign dividends summing to over 3,000 SEK (about 292 EUR) during tax year 2018, according to automatically exchanged information within the Common Reporting Standard (CRS). The study population was restricted to the population with a digital mailbox, aged 30 to 75 and having filed a tax declaration in 2018. In addition, some further selections were made with respect to missing data and outliers. For details on the restriction of the population see [2].

2.2. The Outcome Data

Data on the outcome variables comes from administrative data on taxpayers’ filed income tax returns. The latest day for sending in the income tax declaration for the income year 2019 was May 4, 2020. For various reasons, some taxpayers file their taxes later than the deadline. The most common reason for filing at a later date is that the individual has applied and been approved for respite with income tax returns. As of July 27, 2020 (the date when the estimation data set was collected from the registers), we had access to data on 99.3 percent of the study sample, i.e., the group of individuals who received the message and the control group.

There is no separate field in the income tax declaration for foreign dividends. Our main outcome variable is therefore capital income ($capinc$), where according to the tax code, the taxpayer should include the amount of foreign dividends. The variable $capinc$ does not include capital gains and can be zero but cannot be negative. In addition, we estimate the effect on total tax paid ($tax$). An increase in declared foreign dividends in $capinc$ should lead to an increase in $tax$ unless no offsetting adjustments are made in the income tax declaration. Since the treatment is expected to have an effect on $tax$ only through changes in foreign income which is a minor part of $tax$, we do not expect a large effect on $tax$. It is nevertheless important to include the final tax as an outcome variable since it is a measure of compliance directly related to tax revenues.

Also, although perhaps far fetched, receiving the message could potentially have a positive effect on overall compliance through an unintentional nudge. In other words, merely receiving a message from the Swedish Tax Agency could possibly nudge taxpayers into higher general compliance level which could be manifested as an increase in total tax paid not necessarily stemming from increased declared capital income.

3. Experimental Design

Even though the estimators from well-conducted experiments are unbiased in expectation, the estimates from any single experiment may still be far from the population average treatment effect due to an unlucky, albeit random, assignment. For this reason the experimental design is stratified on gender and historical tax compliance; two variables that we believe can be important determinants for the two outcomes under investigation. A second reason for stratification is that we are interested in examining group differences in behavior as a consequence of the intervention. Hypotheses on the signs of effect differences along previous compliance and gender are discussed in [2]. As we also have continuous covariates (historical data on the outcomes, earnings, age, etc.) we also would like to balance them within each stratum. To this end, we used the stratified rerandomization design suggested in [11].

As mentioned previously, the number of treated was limited to 500 individuals so given that we in total have 2,697 individuals in the population this means that we have in 2,197 non-treated individuals. However, as we also want to evaluate the efficiency gains of the design we do not want to use all these as controls. For simplicity we considered only 500 of these in the experimental design. The reason for choosing balanced designs is that they preferable to unbalanced designs in both Fisher [4] and Neyman-Pearson [6] inference with a fixed sample size. The standard error would in general be smaller if the number of controls would be larger than 500. However, as the sampling variance of the treated are constant the reduction in standard error will in general be moderate in the unbalanced case. For example, under the assumption that the variance of the potential outcomes if being treated or a control is the same the standard errors would in expectation be reduced by only around 20 percent if all 2,197 potential controls were used (i.e. a quadrupling of the sample size of the controls).4 With heterogeneous treatment effects the variance is likely to be higher in the treatment state and then the efficiency gains are in expectation even smaller.

The idea behind rerandomization is the same as with stratification or blocking, that is, to remove from consideration assignments with imbalance in observed covariates between treated and control units and then randomize within the set of assignments with balance on these covariates. Call the set of all assignments within a stratum $\mathcal{A}$ and the set of acceptable assignments ${\mathcal{A}}_a$. Thus, $Card\left({\mathcal{A}}_a\right)<Card\left(\mathcal{A}\right)$, where $Card\left(\mathcal{A}\right)$ and $Card\left({\mathcal{A}}_a\right)$ is the cardinality of $\mathcal{A}$ and ${\mathcal{A}}_a$, respectively. Morgan and Rubin (2012) suggested calculating the Mahalanobis distance between the means of K covariates of the potentially treated and controls and then to accept a specific random allocation only if this measure is less than a, where a is small. The Mahalanobis distance is chi-square distributed with K degrees of freedom, ${\chi }_K^2$. The criterion a, determining the set ${\mathcal{A}}_a$, can thus be decided implicitly by accepting a random allocation if the calculated Mahalanobis distance between the two means is smaller than a pre-specified probability derived from the ${\chi }_K^2$-distribution. If we let $Pr({\chi }_K^2\le a)=p_a$, the specific random allocation is one allocation of the $100\times p_a$ % allocations with the smallest difference in means between treated and controls. For details on the procedure, see Appendix B.

As shown in Appendix B, the percent reduction in variance in contrast to complete randomization can be shown to be equal to

$$100\times R^2(1-{\nu }_a),$$

(1)

where $R^2$ is the coefficient of determination of a regression of the outcome on $\mathbf{X}$ and

$${\nu }_a={\displaystyle \frac{Pr({\chi }_{K+2}^2\le a)}{Pr({\chi }_K^2\le a)}};0<{\nu }_a<1.$$

(2)

${\nu }_a$ is non-increasing with K and decreasing in $p_a$. The implication is that the choice of covariates is important. One should not add unnecessary covariates (i.e., covariates that are not partially correlated with the outcome) as this reduces the efficiency of the estimator. Furthermore, a strict criterion for a given K will increase the efficiency in comparison to a less strict criterion.

We define compliance by using information about whether the amount of foreign dividends obtained from the CRS-data for the income year of 2018 was less than or equal to total capital income during the same year, i.e., $\mathbf{1}[compliant=1]\equiv \mathbf{1}[fdiv\le capinc]$, where $\mathbf{1}[·]$ takes the value one if the expression within brackets is true and zero otherwise.5 As we observe the sex of individuals in our data this allows us to define the four strata.

The following six covariates, measured for the pre-experiment income year 2018, were used to calculate the Mahalanobis distance:

• $age$: the taxpayer’s age
• $fdiv$: foreign dividends
• $capinc$: capital income
• $tax$: total tax paid
• $earn$: earnings including labor income, sick pay, pension, etc.
• $finc\equiv \mathbf{1}\left[hasforeignincome\right]$: categorical variable based on a check box in the tax declaration which equals one if the box is checked and zero otherwise.

Using the 2,697 individuals we proceeded as follows:

• Divide the sample in a compliant (1,759) and a non-compliant group (938) where the number of observations is given within parentheses.
• Draw two simple random samples, each of size 500, from the two groups. These 1,000 individuals constitute the sampling frame of the trial.
• Create four strata; compliant women, compliant men, non-compliant women, and non-compliant men
• Within each stratum, randomly select an allocation with a Mahalanobis distance between treated and controls means of the six covariates to be less than 0.17. As $P\left({\chi }^2\left(6\right)<0.17\right)=0.0001$ this means that the specific random allocation is one allocation of the 0.01 % allocations with the smallest differences in means between the treated and the controls.

Table 1 shows descriptive statistics by stratum after performing the rerandomization as well as the resulting number of observations in each stratum. From this table we can see that we have $167\times 2$ non-compliant men, $83\times 2$ non-compliant women, $200\times 2$ compliant men, and $50\times 2$ compliant women. The mean difference of the covariates between the treated are, as expected, very small within each of the four strata.

4. Analysis and Results

A drawback with the rerandomization strategy is that the difference-in-means estimator is no longer asymptotically normally distributed (see [17]). However, [16] showed that standard asymptotic inference can be conducted using ordinary least squares (OLS). To be specific, let ${\mathbf{x}}_i$ be the $K\times 1$ vector of covariates used in the Mahalanobis distance for individual i, $W_i$ be the treatment indicator, and $Y_i$ the outcome. The treatment effect is the estimated coefficient on $W_i$ in the regression of $Y_i$ on $W_i$, ${\mathbf{x}}_i$ and $W_i({\mathbf{x}}_i-\overline{\mathbf{x}})$, where $\overline{\mathbf{x}}$ is the vector of sample means of the covariates. To construct asymptotically valid confidence intervals one should use the Eicker-Huber-White (EHW) robust standard error estimator (see [5]; [8]; [22]).

A balanced design simplifies the analysis and the tests for effect differences across strata. The reason is that the four effects estimated in the saturated linear regression model are the same as the four between-group differences in mean estimates, which are unbiased (see [11]). This holds also in a rerandomization design.

The following regression model is used in the analysis

$$\begin{array}{cc}\hfill Y_i=& {\alpha }_0+\tau W_i+{\alpha }_w\mathbf{1}[woman=0]+{\alpha }_c\mathbf{1}[compliant=1]\hfill \\ \hfill & +{\alpha }_{wc}\mathbf{1}[woman=0]\times \mathbf{1}[compliant=1]+{\mathbf{\beta }}_0^{\prime }{\tilde{\mathbf{x}}}_i+{{\mathbf{\beta }}_{\mathbf{1}}}^{\prime }{W}_i{\tilde{\mathbf{x}}}_i,\hfill \end{array}$$

(3)

where ${\tilde{\mathbf{x}}}_i\equiv {\mathbf{x}}_i-\overline{\mathbf{x}}$.

We have two outcomes ($capinc$ and $tax$ measured in 1000s SEK) and the test for overall effect for each outcome is $H_0:\tau =0$ against the alternative $H_1:\tau \ne 0$. We let the overall risk level for judging whether we have an effect or not to be 5% which means that each single test will be conducted at the 2.5% risk level.

4.1. Results

The raw mean differences of the two outcomes (capital income ($capinc$) and final tax ($tax$), respectively, for 2019, measured in 1000s SEK) for treated and controls and their differences overall and across the strata are shown in the first six columns in Table 2. Overall we see a higher declared capital income for those receiving the digital message, but also a lower declared taxed income. In addition, we see large differences in outcome levels and treatment-control differences across the strata. In the last six columns, we also show the share of taxpayers in the various strata who have declared any capital income, or who had to pay any tax. The differences between treated and controls with respect to these shares vary in sign and are relatively close to zero in all strata and for both variables.

The OLS-results from estimating Equation (3) on $capinc$ and $tax$ are presented in Table 3. The point estimate in column (1) shows an increase in the declared capital income of 5,225 SEK on average, or an increase with around 43% when evaluated at the mean declared capital income of 12,196 SEK. Column (2) shows a reduction on declared tax by 3,856 SEK on average, or a reduction by 1.23%, when evaluated at the mean declared taxed income of 313,290 SEK. Both parameters are however imprecisely estimated and none is statistically significant at the 10%-level.

4.2. Sub-group analysis

In the pre-analysis plan we stated that we would only test for effect heterogeneity across groups if the parameter estimate $\widehat{\tau }$ from (3) was found to be statistically significant at the 2.5 percent level. The reason for this restriction was to have control over the size of the test and at the same time to have power of detecting the ITT-effect.6

Since the point estimate $\widehat{\tau }$ was not statistically significant, we have not included the heterogeneity analysis in this paper. We did, however, perform exploratory analyses. The point estimates for $capinc$ are in line with the discussion in the pre-analysis plan of larger effects for the non-compliant.7 We also conjectured that the effect for women would be larger in magnitude than for men, and this is also supported by the point estimates for non-compliant taxpayers. However, the only case a confidence interval that does not cover zero is for non-compliant women.

5. Efficiency Gains from the Statistical Design

We perform the simulation using the subset of non-treated individuals from the original experiment. To recap, the experiment was based on 2,697 individuals (see Section 3), from which 1,000 individuals were sampled and 500 were treated. In the simulation, we exclude the 500 treated and remove those few who had not filed their declaration as of July 27, 2020, resulting in a sample of 2,190 taxpayers which we denote the simulation sample. As in the original experiment, we divide the sample in a compliant and non-compliant group. The first group which we denote (a) consists of 1,509 individuals and the second, (b) of 681 individuals. In each replication of the simulation ($r=1,2,\dots ,1000$), the following steps were performed:

• Draw two simple random samples, each of size 500, from (a) and (b) respectively, resulting in 1,000 individuals.
• Draw 500 individuals assumed to be treated according to three different experimental designs:
  • Rerandomization within stratum: Perform steps 3 and 4 described in Section 3.
  • Stratification, i.e. complete randomization within stratum: Create the four strata and randomly allocate 50% to be `treated’ within each stratum.
  • Complete randomization: Randomly allocate 500 individuals to be `treated’.
• Estimate $\tau$ for the three designs, with and without covariates, and store the estimates.
• For each of the designs and estimator calculate the standard deviation of the 1000 estimates.

The results from the simulation with and without the covariates are presented in panel I and II in Table 4, respectively. The regression estimator is used in all analysis with covariates based on the specification in Equation (3). Without covariates, the difference-in-means estimator is used in the completely randomized experiment while for the two other designs the regression estimator is used. Note that due to the balanced design the regression estimator with the stratum indicators and interaction with the treatment is equivalent to the stratified estimator, ${\widehat{\tau }}^{SE}={\sum }_{g=1}^4{f}_g\times ({\overline{y}}_{g1}-{\overline{y}}_{g0})$, where ${\overline{y}}_{g,W}$ are the mean outcomes of the treated ($W=1$) and controls ($W=1$) in stratum g and $f_g$ is the fraction of observations in stratum g. This is thus equivalent to Equation (3) without any covariates.

From panel I we can see that the efficiency gains from stratification over complete randomization on capital income and income tax are 2.94 and 5.60 percent, respectively (see row I.6). The corresponding efficiency gains from rerandomization within strata are substantial; 3.23 and 47.19 percent, respectively (see row I.4). Finally, the total efficiency gains from the stratified rerandomization on these two variables are 6.27 and 55.45 percent, respectively (see row I.5).

The percentage reduction in variance in a rerandomization design is $100\times R^2(1-{\nu }_a)$ (cf. Equation (1)). As ${\nu }_a=Pr({\chi }_8^2\le 0.17)/Pr({\chi }_6^2\le 0.17))=0.12$, the efficiency gains in our design should be approximately $100\times R^2(1-0.12)$ within each stratum. Using the $R^2$ from from Table 3 we find an expected gain of 2.73% and 48.66%, respectively. The observed efficiency gains displayed in panel I of Table 4 against complete randomization were 6.27% and 55.44% which is substantially larger. Most likely, this is a consequence of the stratification.

The results from the simulation with the covariate adjustment is presented in panel II. As expected, the efficiency gains are much more modest. The total efficiency gains for capital income and income tax from stratified rerandomization compared to stratification are 5.02 and 5.32 percent, respectively (see row II.5). The majority of the efficiency gain stems from the stratification. As can be seen in row II.4, for final tax, rerandomization within stratum even reduced the efficiency in contrast to only stratification.

For inference, standard errors need no be estimated, thus a more relevant analysis of the efficiency gains is to study the length of the estimated asymptotic confidence intervals given correct coverage rate. Here, we for each replicate and for all cells, estimate the standard errors and use them together with the point estimate to calculate the asymptotic confidence interval. The coverage rates are calculated as the fraction of replication where the estimated confidence interval covers the null of $\tau =0$ given a level of significance $\alpha$ of $1\%,5\%$ and $10\%$.

The coverage together with the average length of the confidence interval is displayed in Table 5. From the table we can see that the coverage rates are, as expected, close the the nominal level for all estimators. For both outcomes the stratified design has the shortest length, followed by the stratified rerandomization design. However, the average lengths of the confidence intervals are very similar, so in practice there is hardly any efficiency gains from the designs in contrast to ex-post covariate adjustments in the completely randomized design.

6. Discussion

Based on a field experiment conducted by the Swedish Tax Agency, this paper contributes to the research community by providing a better understanding of costs and benefits of using the Mahalonobis based rerandomization design. The Swedish Tax Agency was interested in analyzing the possibilities of precisely estimating the effect of a low-cost intervention using a digital platform on the subsequent tax compliance of Swedish taxpayers who had foreign dividends during the income year of 2018. Although the direct cost of sending a digital message is low, the number of treated was limited to 500 individuals by the Swedish Tax Agency. Furthermore, the effect size was not expected to be large. For these reason it was imperative to consider an efficient experimental design. To gain an understanding for future implementations the experiment was deliberately set up of to examine the efficiency gains of the experimental design.

The results show that the efficiency gains from the stratified rerandomization design compared to complete randomization and a stratified design can be substantive. As compared with the completely randomized design and the stratified experiment the efficiency gain for one of the outcomes, final tax paid, was 55.4 and 47.19 percent, respectively. The corresponding efficiency gains for reported capital income, where the covariates are less relevant for the outcome were only 6.3 and 3.23 percent, respectively. There are gains also when the treatment effect is estimated using a regression estimator adjusting for the same covariates that were used in the design. In this case the efficiency gains against a completely randomized experiment were more modest (5.3 percent for tax and 5.0 percent for capital income). Furthermore, compared to the results with a stratified experiment there were essentially no efficiency gains from the the stratified rerandomization design.

In the comparison of the asymptotic confidence intervals of the regression estimator we found for both outcomes that the average length of confidence intervals are the shortest under the stratified experiment. However, the differences in lengths are marginal across the three designs.

When covariates are used, the results thus suggests marginal, if any, efficiency gains in inference to a population from the stratified rerandomization design in contrast to a completely randomized experiment, or a stratified experiment, under the null of no effect. It may be the case that there are efficiency gains using the stratified rerandomization design with smaller sample sizes or under the alternative, as has previously been shown in Monte Carlo-simulations (see e.g. [9,10,11,23]). However, based on our experience with this experiment, in a large organization, there might be considerable administrative costs beyond the relatively small cost of writing the source code for the experiment, meaning that a standard randomized experiment would be a more practical option. For this and similar applications, our conclusion is that regression-based post-stratification along with including relevant covariates might be a reasonably efficient alternative.

It should be stressed that if one chooses the covariates in the regression models after the experiment is conducted, the resulting inference may be flawed as researchers are prone to searching for statistically significant results (see e.g [7]; [1]; [21]; [19]). To avoid difficult post-experiment decisions and for transparency, we recommend a careful choice process documented in a pre-analysis plan.