Level Weights for Modeling with Complex Survey Data

1. Introduction

Weighting is a correction technique applied to survey data to improve the accuracy of the survey estimates (Lavrakas 2008) especially for health surveys that are typically based on complex survey designs like stratified multistage clustered designs (Graubard and Korn 1996). Survey data is usually weighted for two reasons; to correct for unequal probabilities of selection that often have occurred during sampling and to compensate for survey nonresponse during data collection (Gelman 2007, Kalton and Flores-Cervantes 2003, Lavrakas 2008, Lepkowski, et al. 2013, Little and Vartivarian 2003, Pfeffermann-a 1993, Seaman and White 2013, Wagemaker 2020).

Though widely accepted in descriptive analyses, weights have been criticized by some modelers in analytical inference (Pfeffermann-a 1993). However, weighting sample observations produces consistent estimators of the model parameters (Bethlehem and Keller 1987, Skinner and Mason n.d.), averts model misspecification (Pfeffermann-b 1996) and reduces bias in the survey estimates (Chen, et al. 2015, Kim and Skinner 2013) while analyzing data from unequal probability designs (Nahorniak, et al. 2015). Despite efforts to maximize response rates, virtually all sample surveys are prone to nonresponses (Wun, et al. 2007).

One of the critical challenges faced by analysts using Demographic and Health Survey (DHS) program datasets for multilevel modeling and other survey datasets is that sampling weights related to the specific levels of the multistage design (level weights) are not incorporated in these datasets. Providing the exact level weights may create disclosure risks where dataset users may recognize specific clusters, households, or individuals within the clusters if they have access to sampling frames. Hence, instead of level-specific weights, the final survey weights are made available in the public-used datasets. Yet, multilevel modeling requires level-specific sampling weights instead of the final survey weights, the product of the level weights. The final survey weights are only adequate for most analytical purposes besides multilevel modeling (Elkasabi, Ren and Pullum 2020). Ignoring these level weights however, may lead to erroneous inferences with respect to the sample design (Hakim, et al. 2020, West, et al. 2015).

Notably, few guidelines for integrating weights into multilevel models are in place (Carle 2009). Hence, this paper aimed at demonstrating the estimation of level weights and comparing weighted models; one that applied the final survey weights and another that applied the estimated level weights, using the Uganda Malaria Indicator Survey (UMIS) data of 2018/19.

2. Methods

Source of Data and Study Population

The study applied secondary data from the latest UMIS of 2018/19. The data are based on a two-stage cluster and stratified sampling procedure, meeting the requirements of a complex survey design. A total of 7,632 children below 5 years of age that were tested for anaemia and malaria infection formed the study population. Blood samples from the children to be tested were collected by finger or heel prick. Anaemia and malaria testing was carried out by health technicians (NMCD, UBOS and ICF 2020).

Weighting

Sampling weights are required for analysis to ensure that the survey results are representative at the national level as well as the domain level due to the non-proportional allocation of the sample to different regions and to their urban and rural areas, and the possible differences in response rates (UBOS and ICF 2018). Since the 2018/19 UMIS sample is a two-stage stratified cluster sample, sampling weights were calculated separately, based on sampling probabilities for each sampling stage and for each cluster (NMCD, UBOS and ICF 2020).

In this study, we used a procedure for estimating level weights in Malaria Indicator Surveys (MIS) proposed by the Demographic Health Survey (DHS) program (Elkasabi, Ren and Pullum 2020). The procedure required information available in the UMIS datasets and the final report. The following steps were followed to approximate level-2 (household) and level-3 (enumeration area) as outlined in Table 1.

Where;

• HV005 is the final household survey weight variable, from the household recode (HR) dataset.
• $a_h^c$ is the number of finalized EAs in stratum or region $h$ for the strata. The number of interviewed EAs, $a_h^c$ was calculated from the household (HR) dataset.
• $M_h$ is the number of households in stratum $h$ for all strata.
• $M$ is the number of households in the whole of Uganda according to the Uganda Population and Housing Census of 2014.
• $M_{hi}$ is the number of households in EA $i$ per EA. These numbers were estimated using the average number of households in each EA in strata ${\overline{M}}_h$ according to the most recent Uganda Population and Housing Census data of 2014.
• $m^c$ is the number of complete households in the survey.
• $\widehat{M}$ is the approximated household number at the time of the survey in the whole country. This was approximated by the number of households in Uganda according to the Uganda Population and Housing Census of 2014.

The applied variation factor $f$ depends on the exponent specification $\alpha$, $0\le \alpha \le 1$ thus contributing different degrees to $w_{2\backslash 3}^{HH}$ and $w_3^{HH}$. The factor is fully assigned to $w_3^{HH}$ and also fully assigned to $w_{2\backslash 3}^{HH}$ when $\alpha =1$ and $\alpha =0$, respectively. In the case of this study, it is equally assigned to the two weights as $\alpha =0.5$. In case extreme values are applied, all the weight variation is attributed to either the level-2 or level-3 weight.

De-normalization of the final survey weight was done to restore the original scale of the survey weight using the available information in the UMIS dataset and the UMIS final report. The de-normalized final survey weights in the first step were estimated as:

$$d_{hi}^{HH}=HV{005}_{hi}\frac{\widehat{M}}{m^c}$$

where $M$ is the number of households in the entire country.

Multilevel mixed effects models

Weighted multilevel mixed effects logistic regression models were specified to explain the contextual area associations. The model is represented in the equation below:

$$ln\left(\frac{p_{ijk}}{1-p_{ijk}}\right)={\beta }_0+{\beta }_1{X}_{ijk}+{\eta }_k+{\xi }_{jk}$$

• $ln$ is the natural logarithm.
• $p_{ijk}$ is the probability of testing positive for malaria for the $i^{th}$ child in the $j^{th}$ household and $k^{th}$ EA.
• ${\beta }_0$ is the average log-odds of malaria infection.
• $X_{ijk}$ is a covariate at level-1 for the $i^{th}$ child in the $j^{th}$ household and $k^{th}$ EA.
• ${\beta }_1$ denotes the slope related with $X_{ijk}$ representing the association between the individual child covariates and the log-odds of malaria infection.
• ${\eta }_k$ is the $k^{th}$ EA random effect.
• ${\xi }_{jk}$ is the $j^{th}$ household random effect.

Note that the random effects are assumed to be independently and identically distributed to one another with zero mean and ${\sigma }_{\eta }$ and ${\sigma }_{\xi }$ variances, for EA and household respectively.

Model comparison

Design factor (deft) was used to compare model estimates since nonresponse results into loss in the precision of survey estimates, primarily due to reduced sample size (Brick 2013). A model was therefore considered better if it produced estimates with lower deft values in general. Lower deft values are associated with lower loss of precision of model estimates (Sturgis 2004). The deft was calculated as follows:

$$deft=\sqrt{deff}=\sqrt{1+rho\left(n-1\right)}$$

Where;

• $deff$ is the design effect.
• $rho$ is the intra-class correlation for the variable in question.
• $n$ is the size of the cluster.

3. Results

Comparison of Models Using Standard Errors of Model Estimates

Findings from this study indicated that the model weighted with the final survey weights underestimated standard errors compared to one that was weighted with the estimated level weights (Figure 1). Underestimated standard errors in turn resulted into narrow confidence intervals (Table 2 and Figure 3), presenting seemingly high accuracy of model estimates and likely overconfidence in the model estimates. This is worth paying attention to because overconfidence in results leads to rejecting the null hypothesis when it should not be the case (Type I error) which most likely misleads inference.

Comparison of Models Using Design Factor Values of Model Estimates

Design factor (deft) values for the model weighted with final survey weights were generally higher than those for the model weighted with level weights (Figure 2 and Table 3). This indicates that the use of final survey weights resulted into a considerble loss of precision for model estimates compared to the use of level weights. Almost half (5 of the 11) estimates for the model weighted with final survey weights produced deft values greater than 1.2, a value commonly taken to indicate sizeable variance inflation compared to only 1 estimate for the model weighted with level weights.

4. Discussions

Recently, the application of multilevel analysis on complex survey data has increased as well as the accessibility of public available survey datasets collected from large national samples. However, the challenge with modeling in these data is weighting, as researchers have continuously applied the final survey weights that are inappropriate for multilevel analysis (Elkasabi, Ren and Pullum 2020) instead of level weights at the sampled hierarchies (Rabe-Hesketh and Skrondal 2006). The use of the former weights has a number of implications on model estimates among which is misestimating standard errors and model parameters (Carle 2009). Ignoring specific level weights has consequences; at the first stage, it results into biased estimates on the intercept and variance of random effect, whereas at the second stage, it leads to slightly underestimated fixed effects and residual variance, in addition to the biased estimates on the intercept and variance of the random effect (Cai 2013).

A model that applied final survey weights produced more estimates with design factor values greater than 1.2 a, value commonly taken to indicate sizeable variance inflation (Sturgis 2004) hence increasing the variance of the estimated coefficients (Skinner and Mason n.d.), an indication that level weights are useful in reducing variance of model estimate (Dargatz and Hill 1996, Liao and Valliant 2012).

Computation of appropriate standard errors with complex survey data has important implications for policy research since standard errors are the foundation upon which statistical significance testing is based (Davern, et al. 2007) which informs inference. Besides, the DHS public available datasets like other complex survey data are used extensively for health policy research, with significant implications for national health policy formulation decisions. A model weighted by final survey weights underestimated standard errors for the model estimates. This has an effect on model estimates (Daniels, Dominici and Zeger 2004). Underestimation of standard errors leads to narrow confidence interval, affecting test of statistical significance and possibly misleading inference.

5. Conclusions

Use of level weights instead of the final survey weights produces correct values of standard errors hence correctly informing inference. Models that apply level weights also produce estimates with high precision. Researchers should always estimate and apply level weights while using survey data for multilevel analysis with final survey weights especially datasets under the DHS program, for better model estimates and inference.