Item-Oriented Personalized LDP for Discrete Distribution Estimation

1. Introduction

Discrete distribution estimation is widely used as a fundamental statistics tool and has achieved significant performance in various data analysis tasks, including frequent pattern mining [1], histogram publication [2], and heavy hitter identification [3]. With the deepening and expansion of application scenarios, these data analysis tasks inevitably involve more and more sensitive personal data. Due to the privacy concerns, individuals may not always be willing to truthfully provide their personal information. When dealing with such data, however, discrete distribution estimation is difficult to play its due role. For instance, a health organization plans to make statistics about two epidemic diseases: HIV and Hepatitis, so they issued a questionnaire survey containing three options: HIV, Hepatitis, and None, to inquire whether the citizens suffer from these two diseases. Undoubtedly, this question is highly sensitive, especially for people who actually have these two diseases. As a result, they have a high probability to give false information when filling the the questionnaire, and this will eventually lead to unpredictable biases in the estimation of the distribution of diseases. Therefore, under the requirement of privacy protection, how to conduct discrete distribution estimation is increasingly drawn the attention of researchers.

Differential Privacy (DP) [4,5] is an advanced and promising technique for privacy protection. Benefiting from its rigorous mathematical definition and lightweight computation demand, DP has rapidly become one of the trend in the field of privacy protection. Generally, we can categorize DP into Centralized DP (CDP) [5,6,7,8,9] and Local DP (LDP) [10,11,12]. Compared with the former, the latter does not require a trusted server, and hence it is much more appropriate for privacy protection in the tasks of the discrete distribution estimation. Based on Randomized Response (RR) [13] mechanism, LDP provides different degrees of privacy protection through the assignment of different privacy budget. Currently, typical LDP mechanisms, such as K-ary RR (KRR) [14] and RAPPOR [15], perturb all items in the domain with the same privacy budget, thus providing uniform protection strength. However, in practical scenarios, each item’s sensitivity is different rather than fixed, and the number of individuals involved is also inversely proportional to their sensitivity level. For example, in the questionnaire mentioned above, HIV undoubtedly has a much higher sensitivity than Hepatitis, but a relatively less population of individuals with it than that of the latter. Additionally, None is a non-sensitive option, which naturally accounts for the largest population. Therefore, if we provide privacy protection for all items at the same level without considering their distinct sensitivity, unnecessary perturbation will be imposed on those less sensitive (and even non-sensitive) items that account for a much more population, which severely degrades the data utility of the final result.

Recently, several works proposed to improve the utility by providing different levels of protection according to various sensitivities of items. Murakami et al. introduced the Utility-Optimized LDP (ULDP) [16], which partitions personal data into sensitive and non-sensitive data and ensures privacy protection for the sensitive data only. While ULDP have better utility than KRR and RAPPOR by distinguishing sensitive data from non-sensitive data, it still protects all the sensitive data at the same level without considering the different sensitivities among them. After that, Gu et al. proposed the Input-Discriminative LDP (ID-LDP) [17], which further improved utility by providing fine-grained privacy protection for items with different privacy budgets of inputs. However, under ID-LDP, the strength of perturbation is severely restricted by the minimum privacy budget. As the minimum privacy budget decreases, the corresponding perturbations imposed on different items will approach the maximum level, which greatly weakens the improvement of utility brought by differentiating handling for each item with a independent privacy budget, thereby limiting the applicability of this method.

Therefore, the current methods of discrete distribution estimation in local privacy setting leave much room to improve the utility. In this paper, we propose a novel notion of LDP named Item-Oriented Personalized LDP (IPLDP). Unlike previous works, IPLDP independently perturbs different items with different privacy budgets to achieve personalized privacy protection and utility improvement simultaneously. Through independent perturbation, the strength of perturbation imposed on those less sensitive items will never be influenced by the sensitivity of others. To satisfy IPLDP, we propose a new mechanism called Item-Oriented Personalized RR (IPRR), and it uses the direct encoding method as in KRR to guarantee the equivalent protection for inputs and outputs simultaneously.

Our main contributions are:

1.

We propose a novel LDP named IPLDP, which independently perturbs different items with different privacy budgets to achieve personalized privacy protection and utility improvement simultaneously.

2.

We propose IPRR mechanism to provide equivalent protection for inputs and outputs simultaneously using the direct encoding method.

3.

By calculating the $l1$ and $l2$ losses through the unbiased estimator of the gound-truth distribution under IPRR, we theoretically prove that our method has tighter upper bounds than that of existing direct encoding mechanisms.

4.

We evaluate our IPRR on a synthetic and a real-world dataset with the comparison with the existing methods. The results demonstrate that our method has better performance than existing methods in data utility.

The remainder of this paper is organized as follows. Section 2 lists the related works. Section 3 provides an overview of several preliminary concepts. Section 4 presents the definition of IPLDP. Section 5 discusses the design of our RR mechanism and its empirical estimator. Section 6 analyzes the utility of the proposed RR method. Section 7 shows the experimental results. Finally, in Section 8, we draw the conclusions.

2. Related Work

Since DP was firstly proposed by Dwork [4], it has attracted much attention from researchers, and numerous variants of DP have been studied including d-privacy [18], Pufferfish privacy [8], dependent DP [19], Bayesian DP [20], mutual information DP [7], R´enyi DP [21], Concentrated DP [6], and distribution privacy [22]. However, all of these methods require a trusted central server. To address this issue, Duchi et al. [11] proposed LDP, which quickly became popular in a variety of application scenarios, such as frequent pattern mining [12,23,24], histogram publication [25], heavy-hitter identification [3,26], and graph applications [27,28,29]. Based on RR [13] mechanism, LDP provides different degrees of privacy protection through the assignment of privacy budget. Currently, typical RR mechanisms, such as KRR [14] and RAPPOR [15], perturb all items in the domain with the same privacy budget, thus providing uniform protection strength.

In recent years, several fine-grained privacy methods have been developed for both centralized and local settings. For example, in the centralized setting, Personalized DP [30,31], Heterogeneous DP [32], and One-sided DP [33] have been studied. In the local setting, Murakami et al. proposed ULDP [16], which partitions the value domain into sensitive and non-sensitive sub-domains. While ULDP optimizes utility by reducing perturbation on non-sensitive values, it does not fully consider the distinct privacy requirements of sensitive values. Gu et al. introduced ID-LDP [17], which protects privacy according to the distinct privacy requirements of different inputs. However, the perturbation of each value is influenced by the minimum privacy budget. As the minimum privacy budget decreases, the perturbations of different items approach the maximum, which degrades the improvement of utility.

3. Preliminaries

In this section, we formally describe our problem. Then, we describe the definitions of LDP and ID-LDP. Finally, we introduce the distribution estimation and utility evaluation methods.

3.1. Problem Statement

A data collector or a server desires to estimate the distribution of several discrete items from n users. The set of all personal items held by these users and its distribution are denoted as $\mathcal{D}$ and $\mathbf{p}\in {\mathbb{S}}^{\left|\mathcal{D}\right|}$, respectively, where $\mathbb{S}$ stands for a probability simplex and $|·|$ is the cardinality of a set. For each $x\in \mathcal{D}$, we use ${\mathbf{p}}_x$ to denote its respective probability. We also have a set of random variables $X^n=\{X_1,...,X_n\}\in \mathcal{D}$ held by n users, which are drawn i.i.d. according to $\mathbf{p}$. Additionally, since the items may be sensitive or non-sensitive for users, we divide $\mathcal{D}$ into two disjoint partitions: ${\mathcal{D}}_S$, which contains sensitive items, and ${\mathcal{D}}_N$, which contains non-sensitive items.

Because of privacy issues, users perturb their items according to a privacy budget set $\mathcal{E}={\left\{{\epsilon }_x\right\}}_{x\in {\mathcal{D}}_S}$, where ${\epsilon }_x$ is the corresponding privacy budget of $x\in {\mathcal{D}}_S$. After perturbation, the data collector can only estimate $\mathbf{p}$ from users by observing $Y^n=\{Y_1,...,Y_n\}$ which is the perturbed version of $X^n$ through a mechanism $\mathbf{Q}$, and the mechanism $\mathbf{Q}$ maps an input item $x\in \mathcal{D}$ to an output $y\in \mathcal{D}$ with probability $\mathbf{Q}\left(y\right|x)$.

Our goals are: (1) to design $\mathbf{Q}$ that maps inputs $\forall x\in \mathcal{D}$ to outputs $\forall y\in \mathcal{D}$ according to the corresponding ${\epsilon }_y\in \mathcal{E}$, and improves data utility as much as possible; (2) to estimate the distribution vector $\mathbf{p}$ from $Y^n$.

We assume that the data collector or the server is untrusted, and users never report their data directly but randomly choose an item from $\mathcal{D}$ to send, where $\mathcal{D}$ is shared by both the server and users. $\mathcal{E}$ should be also public with $\mathcal{D}$, so that users can calculate $\mathbf{Q}$ for perturbation, and the server can calibrate the result according to $\mathbf{Q}$.

3.2. Local Differential Privacy

In LDP [11], each user perturbs its data randomly and then send the perturbed data to the server. The server can only access these perturbed results, which guarantees the privacy. In this section, we list two definitions of LDP notions, that is, the standard LDP and the ID-LDP [17].

Definition 1 

($\epsilon$-LDP). A randomized mechanism $\mathbf{Q}$ satisfies ϵ-LDP if, for any pair of inputs $x,x^{\prime }$, and any output y:

$$\begin{array}{c}\hfill e^{-\epsilon }⩽\frac{\mathbf{Q}\left(y\right|x)}{\mathbf{Q}\left(y\right|x^{\prime })}⩽e^{\epsilon },\end{array}$$

(1)

where $\epsilon \in {\mathbb{R}}^{+}$ is the privacy budget that controls the level of confidence an adversary can distinguish the output from any pair of inputs. Smaller ε means that an adversary feels less confidence for distinguishing y from x or $x^{\prime }$, which naturally provides a stronger privacy protection.

Definition 2 

($\mathcal{E}$-ID-LDP). For a given privacy budget set $\mathcal{E}={\left\{{\epsilon }_x\right\}}_{x\in \mathcal{D}}\in {\mathbb{R}}_{+}^{\left|\mathcal{D}\right|}$, a randomized mechanism $\mathbf{Q}$ satisfies $\mathcal{E}$-ID-LDP if, for any pair of inputs $x,x^{\prime }\in \mathcal{D}$, and any output $y\in Range\left(\mathbf{Q}\right)$:

$$\begin{array}{c}\hfill e^{-r({\epsilon }_x,{\epsilon }_{x^{\prime }})}⩽\frac{\mathbf{Q}\left(y\right|x)}{\mathbf{Q}\left(y\right|x^{\prime })}⩽e^{r({\epsilon }_x,{\epsilon }_{x^{\prime }})},\end{array}$$

(2)

where $r(·,·)$ is a system function of two privacy budgets.

Generally, we use $\mathcal{E}$-MinID-LDP in practical scenarios, where $r({\epsilon }_x,{\epsilon }_{x^{\prime }})=min($

${\epsilon }_x,{\epsilon }_{x^{\prime }})$.

3.3. Distribution Estimation Method

The empirical estimation [34] and the maximum likelihood estimation [34,35] are two types of useful methods for estimating discrete distribution in local privacy setting. We use the former method in our theoretical analysis and use both in our experiments. Here, we explain the details of the empirical estimation.

3.3.1. Empirical estimation method

The empirical estimation method calculates the emprical estimate $\widehat{\mathbf{p}}$ of $\mathbf{p}$ using the empirical estimate $\widehat{\mathbf{m}}$ of the distribution $\mathbf{m}$, where $\mathbf{m}$ is the distribution of the output of the mechanism $\mathbf{Q}$. Since both $\mathbf{p}$ and $\mathbf{m}$ are $\left|\mathcal{D}\right|$-dimensional vectors, $\mathbf{Q}$ can be viewed as a $\left|\mathcal{D}\right|\times \left|\mathcal{D}\right|$ conditional stochastic matrix. Then, the relationship between $\mathbf{p}$ and $\mathbf{m}$ can be given by $\mathbf{m}=\mathbf{p}\mathbf{Q}$. Once the data collector obtains the observed estimation $\widehat{\mathbf{m}}$ of $\mathbf{m}$ from $Y^n$, the estimation of $\mathbf{p}$ can be solved by $\widehat{\mathbf{m}}=\widehat{\mathbf{p}}\mathbf{Q}$. As n increases, $\widehat{\mathbf{m}}$ remains unbiased for $\mathbf{m}$, and hence $\widehat{\mathbf{p}}$ converges to $\mathbf{p}$ as well. However, when the sample count n is small, some elements in $\widehat{\mathbf{p}}$ can be negative. To address this problem, several normalization methods [35] can be utilized to truncate and normalize the result.

3.4. Utility Evaluation Method

In this paper, the $l_2$ and $l_1$ losses is utilized for our theoretical analysis of utility. Mathematically, they are defined as $l_2(\mathbf{p},\widehat{\mathbf{p}})={\sum }_{x\in \mathcal{D}}{({\widehat{\mathbf{p}}}_x-{\mathbf{p}}_x)}^2$, and $l_1(\mathbf{p},\widehat{\mathbf{p}})={\sum }_{x\in \mathcal{D}}\left|{\widehat{\mathbf{p}}}_x-{\mathbf{p}}_x\right|$. Both $l_2$ and $l_1$ losses evaluate the total distance between the estimate value and the ground-truth value. The shorter the distance, the better the data utility.

4. Item-Oriented Personalized LDP

In this section, we first introduce the definition of our proposed IPLDP. Then, we discuss the relationship between IPLDP and LDP. Finally, we compare IPLDP with MinID-LDP.

4.1. Privacy Definition

The standard LDP provides the same level of protection for all items using a uniform privacy budget, which can result in excessive perturbation for less sensitive items and lead to poor utility. To improve the utility, ID-LDP uses distinct privacy budgets for the perturbation of different inputs to provide fine-grained protection. Since all perturbations are influenced by the minimum privacy budget, the strength of all perturbations will be forced to approach the maximum value as the minimum privacy budget decreases. To avoid this problem, IPLDP uses different privacy budgets for outputs of the mechanism to provide independent protection for each item. However, using the output as the protection target may not provide equal protection for the input items. Therefore, in IPLDP, we force the input and output domains to be the same $\mathcal{D}$. Formally, IPLDP is defined as follows.

Definition 3 

($({\mathcal{D}}_S,\mathcal{E})$-IPLDP). For a privacy budget set $\mathcal{E}=\{{\epsilon }_1,\cdots ,{\epsilon }_{|{\mathcal{D}}_S|}\}\in {\mathbb{R}}_{+}^{|{\mathcal{D}}_S|}$, a randomized mechanism $\mathbf{Q}$ satisfies $({\mathcal{D}}_S,\mathcal{E})$-IPLDP if and only if it satisfies following conditions:

1.

for any $x,x^{\prime }\in \mathcal{D}$ and for any $x_i\in {\mathcal{D}}_S(i=1,\cdots ,|{\mathcal{D}}_S\left|\right)$,

$$\begin{array}{c}\hfill e^{-{\epsilon }_i}⩽\frac{\mathbf{Q}\left(x_i\right|x)}{\mathbf{Q}\left(x_i\right|x^{\prime })}⩽e^{{\epsilon }_i},\end{array}$$

(3)

2.

for any $x\in {\mathcal{D}}_N$ and for any $x^{\prime }\in \mathcal{D}$,

$$\begin{array}{c}\hfill \mathbf{Q}\left(x\right|x^{\prime })>0\mathrm{and}\mathbf{Q}\left(x\right|x^{\prime })=0\mathrm{for}\mathrm{any}x\ne x^{\prime }\end{array}$$

(4)

Since non-sensitive items need no protection, the corresponding privacy budget can be viewed as an infinity value. However, we cannot set the privacy budget to infinity in practice. Hence, inspired by ULDP, IPLDP handles ${\mathcal{D}}_S$ and ${\mathcal{D}}_N$ separately.

According to the definition, IPLDP guarantees that the adversary’s ability to distinguish any $y\in {\mathcal{D}}_S$ whether it is from any pair of inputs $x,x^{\prime }\in \mathcal{D}$ would not exceed the range determined by the respected ${\epsilon }_y$. That is to say, for $\forall x\in {\mathcal{D}}_S$, it should satisfy ${\epsilon }_x$-LDP. For $\forall x\in {\mathcal{D}}_N$, It can only be perturbed to any $x\in {\mathcal{D}}_S$ or itself.

4.2. Relationship with LDP

We hereby assume $\mathcal{D}={\mathcal{D}}_S$. Then, the obvious difference between LDP and IPLDP is the number of the privacy budgets. A special case is that, when all the privacy budgets are identical, i.e. ${\epsilon }_x=\epsilon$ for all $x\in \mathcal{D}$, then IPLDP becomes the general $\epsilon$-LDP. Without loss of generality, on the one hand, if a mechanism that satisfies $\epsilon$-LDP, it also satisfies $(\mathcal{D},\mathcal{E})$-IPLDP for all $\mathcal{E}$ with $min\left\{\mathcal{E}\right\}=\epsilon$. On the other hand, if a mechanism satisfies $(\mathcal{D},\mathcal{E})$-IPLDP, it also satisfies $max\left\{\mathcal{E}\right\}$-LDP. Therefore, IPLDP can be viewed as a relaxed version of LDP. Noticeably, the relaxation does not mean that IPLDP is weaker than LDP in terms of the privacy protection, but LDP is too strong for items with different privacy needs. IPLDP has the ability to guarantee the personalized privacy for each item.

4.3. Comparison with MinID-LDP

According to the definition of notion, the main difference between IPLDP and MinID-LDP lies in the corresponding target of the privacy budget. Our IPLDP controls the distinguishability according to the output, while MinID-LDP focuses on the any pair of inputs. Both notions can be considered as a relaxed version of LDP. However, from Lemma 1 in [17], $\mathcal{E}$-MinID-LDP relaxes LDP in $\epsilon =2min\left\{\mathcal{E}\right\}$ at most, which means that the degree of relaxation is much lower than IPLDP with the same $\mathcal{E}$. Therefore, as the minimum privacy budget of $\mathcal{E}$ decreases, the utility improvement under MinID-LDP is limited, and we will further experimentally verify this in Section 7.

5. Item-Oriented Personalized Mechanisms and Distribution Estimation

In this section, to provide personalized protection, we first propose our IPRR mechanism for the sensitive domain ${\mathcal{D}}_S=\mathcal{D}$. We then extend the mechanism to be compatible with the non-sensitive domain ${\mathcal{D}}_N$. Finally, we present the unbiased estimator of IPRR using the empirical estimation method.

5.1. Item-Oriented Personalized Randomized Response

According to our definition of IPLDP, it focuses on the indistinguishability of the mechanism’s output. Then, the input and output domains should keep the same to ensure the equivalent protection for both inputs and outputs. Therefore, the only way to design the mechanism $\mathbf{Q}$ is to use the same direct encoding method as in KRR. To use such method, we need to calculate ${\left|\mathcal{D}\right|}^2$ different probabilities for the $\left|\mathcal{D}\right|\times \left|\mathcal{D}\right|$ stochastic matrix of $\mathbf{Q}$. However, it is impossible to directly calculate these probabilities which make $\mathbf{Q}$ invertible and satisfy IPLDP constraints simultaneously. To calculate all the probabilities of $\mathbf{Q}$, a possible way is to find an optimal solution of minimizing the expectation of $l_2(\widehat{\mathbf{p}},\mathbf{p})$ subject to the constraints of IPLDP, i.e.

$$\begin{array}{ccc}\hfill \underset{\mathbf{Q}}{min}& \underset{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}{error}\left[l_2(\widehat{\mathbf{p}},\mathbf{p})\right]\hfill & \hfill \mathrm{s}.\mathrm{t}.ln\left|\mathbf{Q}\left(y\right|x)/\mathbf{Q}\left(y\right|x^{\prime })\right|⩽{\epsilon }_y,(\forall x,x^{\prime },y\in \mathcal{D}).\end{array}$$

(5)

Nevertheless, we still can not directly solve this optimization problem. Firstly, it is complicated to calculate a close-form of $\mathbf{Q}$, since the objective function is likely to be non-convex and all constraints are non-linear inequalities. Secondly, even if we solve this problem numerically, the complexity of each iteration will become very large as the cardinality of the items increases, since we have to calculate an inverse matrix of $\mathbf{Q}$ to calculate the objective function in (5).

To address this problem, we reconsider the relationship between the privacy budget and the data utility. The privacy budget determines the indistinguishability of each item, i.e., $y\in \mathcal{D}$, by controlling the bound of $\left|ln[\mathbf{Q}\left(y\right|x)/\mathbf{Q}\left(y\right|x^{\prime })]\right|$ for any $x,x^{\prime }\in \mathcal{D}$. Among all inputs, the contribution to the data utility comes from the honest answers (when $y=x$). Therefore, within the range controlled by the privacy budget, as long as the more honest answer can be distinguished from the dishonest ones, the more the utility can be improved. In other words, the ratio of $\mathbf{Q}\left(x\right|x)$ (denote as $q_x$) and $\mathbf{Q}\left(x\right|x^{\prime })$ (denote as ${\overline{q}}_x$) should be as large as possible within the bound dominated by ${\epsilon }_x$. Hence, we can reduce the computation complexity of probabilities from ${\left|\mathcal{D}\right|}^2$ to $2\left|\mathcal{D}\right|$ by making a tradeoff of forcing all ${\overline{q}}_x$ to be identical for all $x^{\prime }\ne x$, and

$$\begin{array}{c}\hfill q_x=e^{{\epsilon }_x}{\overline{q}}_x,x\in \mathcal{D}.\end{array}$$

(6)

Then, we can calculate each element ${\mathbf{p}}_x$ of $\mathbf{p}$ through each element ${\mathbf{m}}_x$ of $\mathbf{m}$ for all $x\in \mathcal{D}$ as follows:

$$\begin{array}{c}\hfill {\mathbf{m}}_x={\mathbf{p}}_x{q}_x+(1-{\mathbf{p}}_x){\overline{q}}_x={\mathbf{p}}_x(e^{{\epsilon }_x}-1){\overline{q}}_x+{\overline{q}}_x.\end{array}$$

(7)

Next, we use the estimate $\widehat{\mathbf{m}}$ and $\widehat{\mathbf{p}}$ with (7) to calculate our objective function in (5). Since $n{\widehat{\mathbf{m}}}_x$ follows the binomial distribution with parameters n and ${\mathbf{m}}_x$, its mean and variance are ${error}_{}\left(n\widehat{\mathbf{m}}\right)=n{\mathbf{m}}_x$ and $\mathrm{Var}\left(n\widehat{\mathbf{m}}\right)=n{\mathbf{m}}_x(1-{\mathbf{m}}_x)$. We now can calculate the objective function in (5) according to (7):

$$\begin{array}{cc}\hfill \underset{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}{error}\left[{\ell }_2(\widehat{\mathbf{p}},\mathbf{p})\right]& =\sum _{x\in \mathcal{D}}\underset{}{error}\left[{({\widehat{\mathbf{p}}}_x-{\mathbf{p}}_x)}^2\right]=\sum _{x\in \mathcal{D}}\frac{1}{{\left(e^{{\epsilon }_x}-1\right)}^2{\overline{q}}_x^2}\frac{{\mathbf{m}}_x-{\mathbf{m}}_x^2}{n}\hfill \\ \hfill & =\sum _{x\in \mathcal{D}}\frac{{\mathbf{p}}_x\left(e^{{\epsilon }_x}-1\right)+1}{n{\left(e^{{\epsilon }_x}-1\right)}^2{\overline{q}}_x}-\sum _{x\in \mathcal{D}}\frac{{\left[{\mathbf{p}}_x\left(e^{{\epsilon }_x}-1\right)+1\right]}^2}{n{\left(e^{{\epsilon }_x}-1\right)}^2}.\hfill \end{array}$$

(8)

The second term and n in (8) can be viewed as constants since they are irrelevant to ${\overline{q}}_x$. Therefore, by omitting these two constants, our final optimization problem can be given as

$$\begin{array}{ccc}\hfill \underset{{\{q_x,{\overline{q}}_x\}}_{x\in \mathcal{D}}}{min}& \sum _{x\in \mathcal{D}}\frac{{\mathbf{p}}_x\left(e^{{\epsilon }_x}-1\right)+1}{{\left(e^{{\epsilon }_x}-1\right)}^2{\overline{q}}_x}\hfill & \hfill \mathrm{s}.\mathrm{t}.q_x+\sum _{x^{\prime }\in \mathcal{D}\setminus \left\{x\right\}}{\overline{q}}_{x^{\prime }}=1,\forall x\in \mathcal{D}.\end{array}$$

(9)

Since the objective is a convex function of ${\overline{q}}_x$ for all $x\in \mathcal{D}$ and all the constraints are linear equations, we can efficiently calculate all the $q_x$ and ${\overline{q}}_x$ for all $x\in \mathcal{D}$ via the Sherman-Morrison formula [36] at the intersection point of the hyper planes formed by the constraints in (9). After solving the linear equaltion groups, we can finally define our Item-Oriented Personalized RR (IPRR) mechanism as follows.

($\mathcal{D},\mathcal{E}$)-IPRR).Definition 4 ( Let $\mathcal{D}=\{x_1,\cdots ,x_{\left|\mathcal{D}\right|}\},\mathcal{E}=\{{\epsilon }_1,\cdots ,{\epsilon }_{\left|\mathcal{D}\right|}\}\in {\mathbb{R}}_{+}^{\left|\mathcal{D}\right|}$, Then ($\mathcal{D},\mathcal{E}$)-IPRR is a mechanism that maps $x^{\prime }\in \mathcal{D}$ to $x\in \mathcal{D}$ with the probability ${\mathbf{Q}}_{\mathrm{IPRR}}\left(x\right|x^{\prime })$ defined by

$$\begin{array}{c}\hfill {\mathbf{Q}}_{\mathrm{IPRR}}\left(x\right|x^{\prime })=\left\{\begin{array}{cc}{q}_x\hfill & \mathrm{if}x=x^{\prime },\hfill \\ {\overline{q}}_x\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(10)

where ${\overline{q}}_x={\left[(e^{{\epsilon }_x}-1)(1+{\sum }_{x\in {\mathcal{D}}_S}{(e^{{\epsilon }_x}-1)}^{-1})\right]}^{-1}$ and $q_x=e^{{\epsilon }_x}{\overline{q}}_x$.

In addition, the special case is that, when all the elements in $\mathcal{E}$ are identical, IPRR becomes KRR.

Theorem 1.($\mathcal{D},\mathcal{E}$)-IPRR satisfies $(\mathcal{D},\mathcal{E})$-IPLDP.

5.2. IPRR with Non-sensitive Items

We hereby present a full version of IPRR that incorporates the non-sensitive domain ${\mathcal{D}}_N$. For all $x\in {\mathcal{D}}_N$, privacy protection is not needed, which is equivalent to ${\epsilon }_x\to \infty$. Thus, to maximize the ratio of $q_x$ and ${\overline{q}}_x$, we set ${\overline{q}}_x$ to zero. Then, inspired by URR, we define IPRR with the non-sensitive domain as follows.

(${\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E}$)-IPRR).Definition 5 ( Let ${\mathcal{D}}_S=\{x_1,\cdots ,x_{|{\mathcal{D}}_S|}\},\mathcal{E}=\{{\epsilon }_1,\cdots ,$

${\epsilon }_{|{\mathcal{D}}_S|}\}\in {\mathbb{R}}_{+}^{|{\mathcal{D}}_S|}$, then (${\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E}$)-IPRR is a mechanism that maps $x\in \mathcal{D}$ to $x_i\in \mathcal{D}$ with the probability ${\mathbf{Q}}_{\mathrm{IPRR}}\left(x\right|y)$ defined by:

$$\begin{array}{c}\hfill {\mathbf{Q}}_{\mathrm{IPRR}}\left(x\right|x^{\prime })=\left\{\begin{array}{cc}{q}_x\hfill & \mathrm{if}x\in {\mathcal{D}}_S\wedge x=x^{\prime },\hfill \\ {\overline{q}}_x\hfill & \mathrm{if}x\in {\mathcal{D}}_S\wedge x\ne x^{\prime },\hfill \\ \tilde{q}\hfill & \mathrm{if}x\in {\mathcal{D}}_N\wedge x=x^{\prime },\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(11)

where $\tilde{q}=1-{\sum }_{x\in {\mathcal{D}}_S}{\overline{q}}_x={(1+{\sum }_{x\in {\mathcal{D}}_S}\frac{1}{e^{{\epsilon }_x}-1})}^{-1}$.

In addition, the special case is that, when all the elements in $\mathcal{E}$ are identical (denoted as $\epsilon$), it is equivalent to the $({\mathcal{D}}_S,\epsilon )$-URR.

Theorem 2.$({\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E})$-IPRR satisfies $({\mathcal{D}}_S,\mathcal{E})$-IPLDP.

Figure 1 depicts an example of the $({\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E})$-IPRR, which illustrates the perturbation of our IPRR, and also shows detailed values of all involved probabilities under $\mathcal{E}=\{0.1,0.5,1.0\}$. As shown in Figure 1, as the privacy budget decreases, users with sensitive items are more honest, which is different from the perturbation style of mainstream RR mechanisms. In those mechanisms, the probability of an honest answer will decrease as the privacy budget decreases. However, data utility is hard to be improved if we follow the mainstream methods to achieve independent personalized protection. Inspired by Mangat’s RR [37], data utility can be further improved through a different style of RR while guaranteeing the privacy as long as we obey the definition of LDP. Mangat’s RR requires users in the sensitive group always answer honestly and uses dishonest answers from other users to contribute to the perturbation. Then, the data collector still can not distinguish one response whether it is an honest answer or not. Furthermore, in practical scenerios, the sensitivity of data shows an inverse relationship with the population size of respective individuals. As a result, indistinguishability can be guaranteed by a large proportion of dishonest responses from less sensitive or non-sensitive groups, even if individuals in the sensitive group are honest. Therefore, in our privacy scheme, we can guarantees the privacy of the clients with improvement of utility as long as both server and clients reach an agreement on this protocol.

5.3. Empirical Estimation under IPRR

In this subsection, we show the details of the emprical estimate of $\mathbf{p}$ under our $({\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E})$-IPRR mechanism. To calculate the estimate, we define a vector $\mathbf{r}$ and a function $\mathcal{S}$ for convenience, which are given by

$$\begin{array}{c}\hfill {\mathbf{r}}_x=\left\{\begin{array}{cc}\frac{1}{e^{{\epsilon }_x}-1}\hfill & \mathrm{if}x\in {\mathcal{D}}_S\hfill \\ 0\hfill & \mathrm{if}x\in {\mathcal{D}}_N\hfill \end{array}\right.,\mathrm{and}{\mathcal{S}}_{·}=\frac{1}{{\sum }_{x\in ·}{\mathbf{r}}_x+1},\end{array}$$

where ${\mathbf{r}}_x$ is the corresponding element of $\mathbf{r}$ to $x\in \mathcal{D}$, and “·” can be any domains, i.e., ${\mathcal{S}}_{{\mathcal{D}}_S}={[{\sum }_{x\in {\mathcal{D}}_S}{\mathbf{r}}_x+1]}^{-1}$. Then, for all $x\in {\mathcal{D}}_S$, we have ${\overline{q}}_x=\frac{1}{(e^{{\epsilon }_x}-1)}·\frac{1}{{\sum }_{i=1}^{\left|\mathcal{E}\right|}{(e^{{\epsilon }_i}-1)}^{-1}+1}={\mathbf{r}}_x{\mathcal{S}}_{{\mathcal{D}}_S}$, and, based on (7), we can calculate $\mathbf{m}$ and the estimate $\widehat{\mathbf{p}}$ with each element ${\mathbf{m}}_x$ and ${\widehat{\mathbf{p}}}_x$ for all $x\in \mathcal{D}$ as

$$\begin{array}{c}\hfill {\mathbf{m}}_x={\mathbf{p}}_x{\mathcal{S}}_{{\mathcal{D}}_S}+{\mathbf{r}}_x{\mathcal{S}}_{{\mathcal{D}}_S}\Rightarrow {\widehat{\mathbf{p}}}_x={\widehat{\mathbf{m}}}_x/{\mathcal{S}}_{{\mathcal{D}}_S}-{\mathbf{r}}_x.\end{array}$$

(12)

As the sample count n increases, $\widehat{\mathbf{m}}$ remains unbiased for $\mathbf{m}$, and hence $\widehat{\mathbf{p}}$ converges to $\mathbf{p}$ as well.

6. Utility Analysis

In this section, we first evaluate the data utility of IPRR based on the $l_2$ and $l_1$ losses of the empirical estimate $\widehat{\mathbf{p}}$. Then, for each loss, we calculate its tight upper bound independent to the unknown distribution $\mathbf{p}$. Finally, we discuss the upper bound in both high and low privacy regimes.

First, we evaluate the expectation of $l_2$ and $l_1$ losses under our IPRR mechanism.

Theorem 3.($l_2$ and $l_1$ losses of $({\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E})$-IPRR) According to the Definition 5 and the empirical estimator given in (12), for all $\mathcal{E}$, the expected $l_2$ and $l_1$ losses of the $({\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E})$-IPRR are given by

$$\begin{array}{c}\hfill \underset{}{error}\left[l_2(\mathbf{p},\widehat{\mathbf{p}})\right]=\underset{}{error}\left[\sum _{x\in \mathcal{D}}{({\widehat{\mathbf{p}}}_x-{\mathbf{p}}_x)}^2\right]=\frac{1}{n}\sum _{x\in \mathcal{D}}\left[({\mathbf{p}}_x+{\mathbf{r}}_x)\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathbf{p}}_x+{\mathbf{r}}_x)\right)\right],\end{array}$$

(13)

and for large n,

$$\begin{array}{cc}\hfill & \underset{}{error}\left[l_1(\mathbf{p},\widehat{\mathbf{p}})\right]=\underset{}{error}\left[\sum _{x\in \mathcal{D}}\left|{\widehat{\mathbf{p}}}_x-{\mathbf{p}}_x\right|\right]\approx \sqrt{\frac{2}{n\pi }}\sum _{x\in \mathcal{D}}\sqrt{({\mathbf{p}}_x+{\mathbf{r}}_x)({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathbf{p}}_x+{\mathbf{r}}_x))},\hfill \end{array}$$

(14)

where $a_n\approx b_n$ represents ${lim}_{n\to \infty }{a}_n/b_n=1$.

According to (13) and (14), we can see that the two losses share the similar structure. Hence, to conveniently discuss the property of the losses, we define a general loss L as follows.

Definition 6 

(general loss of $({\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E})$-IPRR). The general loss L of $({\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E})$-IPRR is can be defined as

$$\begin{array}{c}\hfill L(\mathcal{E};\mathbf{p},\widehat{\mathbf{p}},\mathcal{D})=C\sum _{x\in \mathcal{D}}g\circ f_x,\end{array}$$

(15)

where $f_x=({\mathbf{p}}_x+{\mathbf{r}}_x)\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathbf{p}}_x+{\mathbf{r}}_x)\right)$, g is any monotonically increasing concave function with $g\left(0\right)=0$, and C is a non-negative constant.

With this definition, we show that, for any distribution $\mathbf{p}$, privacy budget set $\mathcal{E}$, ${\mathcal{D}}_S$, and ${\mathcal{D}}_N$, both $l2$ and $l1$ losses of $({\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E})$-IPRR are lower than that of $({\mathcal{D}}_S,min\left\{\mathcal{E}\right\})$-URR.

Through the assignment of fine-grained privacy budgets to each items in IPRR, the emprical estimator in Section 5.3 is a general version for mechanisms which use the direct encoding method. Due to the generality of our estimator, we can use it to calculate the empirical estimate of URR or KRR as long as all the privacy budgets are identical in $\mathcal{E}$. Therefore, based on the general empirical estimator, (13) and (14) are also applicable to these two mechanisms, even other mechanisms that use direct encoding method. To show that the losses of IPRR are lower than that of URR, we first give a lemma below.

Lemma 1. 

Let $\widetilde{·}$ be a sorted version of any given set ·. For any two privacy budget sets ${\mathcal{E}}_1$ and ${\mathcal{E}}_2$ with same dimension k, $L({\mathcal{E}}_1;\mathbf{p},\widehat{\mathbf{p}},\mathcal{D})⩽L({\mathcal{E}}_2;\mathbf{p},\widehat{\mathbf{p}},\mathcal{D})$, if ${\mathcal{E}}_1⩽{\mathcal{E}}_2$, where $A⩽B$ stands for that, for all $a_i\in \tilde{A}$ and $b_i\in \tilde{B}(i=1,\cdots k)$, we have $a_i⩽b_i$.

Based on Lemma 1, for any distribution $\mathbf{p}$, privacy budget set $\mathcal{E}$, ${\mathcal{D}}_S$, and ${\mathcal{D}}_N$, since $\mathcal{E}⩾\{min\left\{\mathcal{E}\right\}\}\in {\mathbb{R}}_{+}^{\left|\mathcal{E}\right|}$, the general loss L of $({\mathcal{D}}_S,{\mathcal{D}}_N,\mathcal{E})$-IPRR are lower than that of $({\mathcal{D}}_S,min\left\{\mathcal{E}\right\})$-URR. Since $l_2$ and $l_1$ losses are specific versions of L when $g\left(x\right)=x$ and $g\left(x\right)=\sqrt{x},respectively$, both two losses of IPRR also lower than that of URR in the same setting.

Next, we evaluate the worst case of the loss L. Observe that L is closely related to the original distribution $\mathbf{p}$. However, since $\mathbf{p}$ is unknown for theoretical analysis, we need to calculate a tight upper bound of the loss that does not depend on the unknown $\mathbf{p}$. Then, to obtain the tight upper bound, we need to find an optimal $\mathbf{p}$ that maximizes L. To address this issue, we convert this problem to an optimization problem subject to $\mathbf{p}$ being a probability simplex as

$$\begin{array}{cccc}\hfill \underset{\mathbf{p}}{max}& \sum _{x\in \mathcal{D}}g\circ f_x\hfill & \hfill \mathrm{s}.\mathrm{t}.& \mathbf{p}\in {\mathbb{S}}^{\left|\mathcal{D}\right|}.\hfill \end{array}$$

(16)

Then, the optimal solution can be given as the following lemma.

Lemma 2. 

Let ${\mathcal{D}}^{*}$ be a subset of $\mathcal{D}$. For all $x\in \mathcal{D}$, if ${\mathbf{r}}_x$ satisfies

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\left(\right|{\mathcal{D}}^{*}|{\mathcal{S}}_{{\mathcal{D}}^{*}}{)}^{-1}-1<{\mathbf{r}}_x<\left(\right|{\mathcal{D}}^{*}{\left|{\mathcal{S}}_{{\mathcal{D}}^{*}}\right)}^{-1}\hfill & \mathrm{if}x\in {\mathcal{D}}^{*},\hfill \\ {\mathbf{r}}_x⩾\left(\right|{\mathcal{D}}^{*}{\left|{\mathcal{S}}_{{\mathcal{D}}^{*}}\right)}^{-1}\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(17)

${\mathbf{p}}^{*}$ is the optimal solution that maximizes the objective function in (16), which is given by

$$\begin{array}{c}\hfill {\mathbf{p}}_x^{*}=\left\{\begin{array}{cc}\left(\right|{\mathcal{D}}^{*}{\left|{\mathcal{S}}_{{\mathcal{D}}^{*}}\right)}^{-1}-{\mathbf{r}}_x\hfill & \mathrm{if}x\in {\mathcal{D}}^{*},\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(18)

According to Lemma 2, we can obtain the following general upper bounds of (13) and (14) as

Theorem 4 

(General upper bound of $l_2$ and $l_1$ losses of IPRR). Eq. (13) and (14) can be maximized by ${\mathbf{p}}^{*}$:

$$\begin{array}{cc}\hfill \underset{}{error}\left[l_2(\mathbf{p},\widehat{\mathbf{p}})\right]& ⩽\underset{}{error}\left[l_2({\mathbf{p}}^{*},\widehat{\mathbf{p}})\right]=\frac{1}{n}\left(\frac{1}{{\mathcal{S}}_{\mathcal{D}}^2}-\frac{1}{|{\mathcal{D}}^{*}|{\mathcal{S}}_{{\mathcal{D}}^{*}}^2}-\sum _{x\in \mathcal{D}\setminus {\mathcal{D}}^{*}}{\mathbf{r}}_x^2\right);\hfill \\ \hfill \underset{}{error}\left[l_1(\mathbf{p},\widehat{\mathbf{p}})\right]& \lesssim \underset{}{error}\left[l_1({\mathbf{p}}^{*},\widehat{\mathbf{p}})\right]\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & =\sqrt{\frac{2}{n\pi }}\left[\sum _{x\in \mathcal{D}\setminus {\mathcal{D}}^{*}}\sqrt{{\mathbf{r}}_x({\mathcal{S}}_{\mathcal{D}}^{-1}-{\mathbf{r}}_x)}+\sqrt{{\mathcal{S}}_{{\mathcal{D}}^{*}}^{-1}\left(\right|{\mathcal{D}}^{*}|{\mathcal{S}}_{\mathcal{D}}^{-1}-{\mathcal{S}}_{{\mathcal{D}}^{*}}^{-1})}\right],\hfill \end{array}$$

(20)

where $a_n\lesssim b_n$ represents ${lim}_{n\to \infty }{a}_n/b_n⩽1$.

Finally, we discuss the losses in the high and low privacy regimes based on the general upper bounds. Let ${\epsilon }_{min}=min\left\{\mathcal{E}\right\}$ and ${\epsilon }_{max}=max\left\{\mathcal{E}\right\}$.

Theorem 5 

($l_2$ and $l_1$ losses in high privacy regime). When ${\epsilon }_{max}$ is close to 0, for all $x\in \mathcal{D}$, we have $e^{{\epsilon }_x}-1\approx {\epsilon }_x$. Then, the worst case of $l_2$ and $l_1$ losses are:

$$\begin{array}{cc}\hfill \underset{}{error}\left[l_2(\mathbf{p},\widehat{\mathbf{p}})\right]& ⩽\underset{}{error}\left[l_2({\mathbf{p}}^{*},\widehat{\mathbf{p}})\right]\approx \frac{1}{n}\sum _{x\in {\mathcal{D}}_S}\sum _{x^{\prime }\in {\mathcal{D}}_S\setminus \left\{x\right\}}\frac{1}{{\epsilon }_x{\epsilon }_{x^{\prime }}};\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill \underset{}{error}\left[l_1(\mathbf{p},\widehat{\mathbf{p}})\right]& \lesssim \underset{}{error}\left[l_1({\mathbf{p}}^{*},\widehat{\mathbf{p}})\right]\approx \sqrt{\frac{2|{\mathcal{D}}_S|}{n\pi }{\sum }_{x\in {\mathcal{D}}_S}{\sum }_{x^{\prime }\in {\mathcal{D}}_S\setminus \left\{x\right\}}\frac{1}{{\epsilon }_x{\epsilon }_{x^{\prime }}}}.\hfill \end{array}$$

(22)

According to [16], in high privacy regime, the expectation of $l_2$ and $l_1$ losses of $({\mathcal{D}}_S,{\epsilon }_{min})$-URR are $\frac{|{\mathcal{D}}_S\left|\right(|{\mathcal{D}}_S|-1)}{n{\epsilon }_{min}^2}$ and $\sqrt{\frac{2}{n\pi }}·\frac{|{\mathcal{D}}_S|\sqrt{|{\mathcal{D}}_S|-1}}{{\epsilon }_{min}}$, accordingly. Thus, the losses of our method is much smaller than that of URR in current setting.

Theorem 6 

($l_2$ and $l_1$ losses in low privacy regime). When ${\epsilon }_{min}>ln\left(\right|{\mathcal{D}}_N|$

$+1)$, for all $x\in \mathcal{D}$, the worst case of $l_2$ and $l_1$ losses are:

$$\begin{array}{cc}\hfill \underset{}{error}\left[l_2(\mathbf{p},\widehat{\mathbf{p}})\right]& ⩽\underset{}{error}\left[l_2({\mathbf{p}}^{*},\widehat{\mathbf{p}})\right]={\left[\sum _{x\in {\mathcal{D}}_S}\frac{|{\mathcal{D}}_S|+e^{{\epsilon }_x}-1}{|{\mathcal{D}}_S|(e^{{\epsilon }_x}-1)}\right]}^2\left(1-\frac{1}{\left|\mathcal{D}\right|}\right);\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill \underset{}{error}\left[l_1(\mathbf{p},\widehat{\mathbf{p}})\right]& \lesssim \underset{}{error}\left[l_1({\mathbf{p}}^{*},\widehat{\mathbf{p}})\right]=\sqrt{\frac{2\left(\right|\mathcal{D}|-1)}{n\pi }}·\sum _{x\in {\mathcal{D}}_S}\frac{|{\mathcal{D}}_S|+e^{{\epsilon }_x}-1}{|{\mathcal{D}}_S|(e^{{\epsilon }_x}-1)}.\hfill \end{array}$$

(24)

According to [16], in low privacy regime, the expectation of $l_2$ and $l_1$ losses of $({\mathcal{D}}_S,{\epsilon }_{min})$-URR are $\frac{\left(\right|{\mathcal{D}}_S{|+e^{{\epsilon }_{min}}-1)}^2}{n{(e^{{\epsilon }_{min}}-1)}^2}\left(1-\frac{1}{\left|\mathcal{D}\right|}\right)$ and $\sqrt{\frac{2\left(\right|\mathcal{D}|-1)}{n\pi }}·\frac{|{\mathcal{D}}_S|+e^{{\epsilon }_{min}}-1}{e^{{\epsilon }_{min}}-1}$, accordingly. Thus, the losses of our method is much smaller than that of URR in current setting.

7. Evaluation

In this section, we evaluate the performance of our IPRR based on the emprical estimation method with the Norm-sub (NS) truncation method and maximum likelihood estimation (MLE) method, and compare it with the the KRR, URR, and Input-Discriminative Unary Encoding (IDUE) [17] satisfying ID-LDP.

7.1. Experimental Setup

7.1.1. Datasets

We conducted experiments over two datasets, and show their details in Table 1. The first dataset, Zipf, was generated by sampling from a Zipf distribution with an exponential parameter $\alpha =2$, followed by filtering the results using a specific threshold to control the size of the item domain and the number of users. The second dataset, Kosarak, is one of the largest real-world datasets, which contains millions of records related to the click-stream of news portals from users (e.g. see [23,38,39]). For Kosarak dataset, we randomly selected an item for every user to serve as the item they hold, and then applied the same filtering process used for the Zipf dataset.

7.1.2. Metrics

We use Mean Squar Error (MSE) and Relative Error (RE) as the metrics of the performance, which are defined as

$$\begin{array}{c}\hfill \mathrm{MSE}=\sum _{x\in \mathcal{D}}\frac{{(f_x-{\widehat{f}}_x)}^2}{n},\left(\right)\sum _{x\in \mathcal{D}}\frac{|f_x-{\widehat{f}}_x|}{f_x},\end{array}$$

(25)

where $f_x$ (resp. ${\widehat{f}}_x$) is the true (resp. estimated) frequency count of x. We take the sample mean of one hundred repeated experiments for analysis.

7.1.3. Settings

We conduct five experiments for both Zipf and Kosarak datasets, where the experiments #1∼#3 compare the utility of IPRR with that of KRR and URR under various privacy level groups with different sample ratios, the experiment #4 compares IPRR with URR under different $|{\mathcal{D}}_N|$, and the experiment #5 compares IPRR with IDUE under different sample ratios. We use $SR$ (sample ratio) to calculate $SR·\left|\mathcal{D}\right|$ as the sample count n. In each experiment, we evaluate the utility under various privacy levels. Since the sensitivity of data shows an inverse relationship with the population size of respective individuals, we first sort the dataset based on the count of each item, and items with smaller counts are assigned to higher privacy levels. Then, we choose items with larger size as ${\mathcal{D}}_N$ (others as ${\mathcal{D}}_S$) by using $NR$ (non-sensitive ratio), which controls the ratio of $|{\mathcal{D}}_N|$ over $\left|\mathcal{D}\right|$. For ${\mathcal{D}}_S$, we use ${\epsilon }_{min}$, ${\epsilon }_{max}$, $LC$ (level count) to divide ${\mathcal{D}}_S$ into different privacy levels, where $LC$ divides ${\mathcal{D}}_S$ and a range $[{\epsilon }_{min},{\epsilon }_{max}]$ evenly to assign the privacy level, accordingly. For example, assume we have $\mathcal{D}=\{A,B,C,D,E,F\}$ with sizes 6, 5, 4, 3, 2, and 1 for each item. Under ${\epsilon }_{min}=0.1$, ${\epsilon }_{max}=0.3$, $LC=3$, and $NR=0.5$, we can obtain ${\mathcal{D}}_N=\{A,B,C\}$, with privacy budgets of 0.3, 0.2, and 0.1 assigned to items D, E, and F, respectively. In practical scenarios, it is unnecessary to assign a unique privacy level to every item. Thus, we set $LC=4$ for all experiments in our settings. Additionally, in all experiments, KRR and URR satisfy ${\epsilon }_{min}$-LDP and $({\mathcal{D}}_S,{\mathcal{D}}_S,{\epsilon }_{min})$-ULDP, respectively. Table 2 shows the details of the parameter settings for all experiments.

7.2. Experimental Results

7.2.1. Utility under various privacy level groups

In Figure 2 we illustrate the results of the experiments #1∼#3. We conducted these experiments to compare the utility of our IPRR with other types of direct encoding mechanisms under various combinations of privacy budgets with fixed $NR$. Fristly, Figure 2(a) and (d) show the comparison of the utility in a high privacy regime among IPRR, URR and KRR on both datasets. As we can see in the high privacy regime, our method outperforms the others by approximate one order of magnitude. As the sample count n increases, the loss decreases as well. Noticeably, in the figure, the results of KRR under both the NS and MLE methods are almost indistinguishable, while our results of the latter method are improved significantly compared with that of the former method. The reason is that the former method may truncate the empirical estimate for lacking samples. Furthermore, the high privacy regime will also escalate the degree of truncation for the emprical estimate, which naturally introduces more errors than the latter method. Secondly, the results of Figure 2(b) and (e) present the comparison of the utility in a low privacy regime. It is clear that our method also has better performance than the others. Notably, in the current setting, the improvement of the MLE method over the NS method is less significant compared with that in the high privacy regime. We argue that a large privacy budget does not result in much truncation for the emprical estimate, so the results are close to each other. Finally, Figure 2(c) and (f) give the results of the hybrid high and low privacy regimes. The results are close to Figure 2(a) and (d), accordingly. Although the improvement is limited in this setting, it does not mean that all perturbations in our method are greatly influenced by the minimum privacy budget similar to IDUE.

Figure 3 shows the results of the experiment #4. In this experiment, we compare the utility of our IPRR with URR in the same privacy budget set to check the influence of different $|{\mathcal{D}}_N|$. We only compare with URR because only these two mechanisms support non-sensitive items. We restricted the maximum value of $NR$ to 80% for the Zipf dataset since $|{\mathcal{D}}_S|$ will be less than $LC=4$ if $NR$ exceeds 0.8. As one can see, as $|{\mathcal{D}}_N|$ increases, our method outperforms URR, and all metrics decrease.

After all, our IPRR shows better performance than URR and KRR on the two corresponding metrics in the two datasets, which verifies our theoretical analysis. Compared with KRR, URR reduces the perturbation for non-sensitive items by coarsely dividing the domain into sensitive and non-sensitive subsets, resulting in lower variance than that of KRR under the identical sample ratio. Thus, URR has better overall performance than KRR. Our method further divides the sensitive domain into finer-grained subsets with personalized perturbation for each item, which reduces the perturbation for less sensitive items. Therefore, with the same sample ratio, our method reduces much more total variance than URR, and thus our method performs better than other mechanisms.

7.2.2. Comparison with IDUE

In Figure 4, we show the results of the experiment #5. Since IDUE does not support non-sensitive items, we conducted this separate experiment to compare the utility of our method with IDUE in the same privacy setting. It is clear that IPRR owns better performance than IDUE over Zipf with both NS and MLE methods, while IDUE outperforms IPRR over Kosarak with the NS method. The reason is that the unary encoding method used by IDUE has more advantages when processing an item domain with a larger size, and this may reduce the truncation for the empirical estimate. However, under the MLE method without the influence of truncation, IPRR outperforms IDUE even over Kosarak with the larger $\left|\mathcal{D}\right|$. We think that our IPRR effectively reduces unnecessary perturbation for less sensitive items than IDUE since the strength of perturbation is highly affected by the minimum privacy budget. In the current setting of our experiment, the perturbation for items with the maximum privacy budget only needs to satisfy 10-LDP in our method, while IDUE can only relax their perturbation at most 0.2-LDP according to the Lemma 1 in [17].

8. Conclusion

In this paper, we first proposed a novel notion of LDP called IPLDP for discrete distribution estimation in local privacy setting. To improve utility, IPLDP perturbs items independently for personalized protection according to the outputs with different privacy budgets. Then, to satisfy IPLDP, we proposed a new mechanism called IPRR based on a common phenomenon that the sensitivity of data shows an inverse relationship with the population size of respective individuals. We prove that IPRR has tighter upper bound than that of existing direct encoding methods under both $l_2$ and $l_1$ losses of emprical estimate. Finally, we conducted related experiments on a synthetic and a real-world datasets. Both theoretical analysis and experimental results demonstrate that our scheme owns better performance than existing methods.

Appendix A.1. Proof of Theorem 1

Proof. 

Since ${\mathcal{D}}_N=\varnothing$, we only need to consider (3). Then, since $q_x/{\overline{q}}_x=e^{{\epsilon }_x}$, the inequality (4) holds.□ □

Appendix A.2. Proof of Theorem 2

Proof. 

For all $x\in {\mathcal{D}}_S$, since $q_x/{\overline{q}}_x=e^{{\epsilon }_x}$, the inequality (4) holds. Then, for all $x\in {\mathcal{D}}_N$, it follows from (3) that (4) also holds.□ □

Appendix B.1. Proof of Theorem 3

Proof. 

1. The $l_2$ loss of the estimate.

Since $n{\widehat{\mathbf{m}}}_x$ follows the binomial distribution with parameters n and ${\mathbf{m}}_x$, its mean and variance are ${error}_{}\left(n\widehat{\mathbf{m}}\right)=n{\mathbf{m}}_x$ and $\mathrm{Var}\left(n\widehat{\mathbf{m}}\right)=n{\mathbf{m}}_x(1-{\mathbf{m}}_x)$. Then,

$$\begin{array}{cc}\hfill {\mathbb{E}}_{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}\left[l_2(\widehat{\mathbf{p}},\mathbf{p})\right]& =\mathbb{E}\left[\sum _{x\in \mathcal{D}}{({\widehat{\mathbf{p}}}_x-{\mathbf{p}}_x)}^2\right]\hfill \\ \hfill & =\sum _{x\in \mathcal{D}}\mathbb{E}\left[{({\widehat{\mathbf{p}}}_x-{\mathbf{p}}_x)}^2\right]\hfill \\ \hfill & =\sum _{x\in \mathcal{D}}\mathbb{E}\left[{({\widehat{\mathbf{m}}}_x/{\mathcal{S}}_{{\mathcal{D}}_S}-{\mathbf{m}}_x/{\mathcal{S}}_{{\mathcal{D}}_S})}^2\right]\hfill \\ \hfill & =\frac{1}{{\mathcal{S}}_{{\mathcal{D}}_S}^2}\sum _{x\in \mathcal{D}}[\underset{}{error}\left({\widehat{\mathbf{m}}}^2\right)-{\mathbf{m}}^2]\hfill \\ \hfill & =\frac{1}{{\mathcal{S}}_{{\mathcal{D}}_S}^2}\sum _{x\in \mathcal{D}}\frac{{\mathbf{m}}_x-{\mathbf{m}}_x^2}{n}\hfill \\ \hfill & =\frac{1}{n{\mathcal{S}}_{{\mathcal{D}}_S}^2}\sum _{x\in \mathcal{D}}\left[{\mathbf{p}}_x{\mathcal{S}}_{{\mathcal{D}}_S}+{\mathbf{r}}_x{\mathcal{S}}_{{\mathcal{D}}_S}-{({\mathbf{p}}_x{\mathcal{S}}_{{\mathcal{D}}_S}+{\mathbf{r}}_x{\mathcal{S}}_{{\mathcal{D}}_S})}^2\right]\hfill \\ \hfill & =\frac{1}{n}\sum _{x\in \mathcal{D}}\left[({\mathbf{p}}_x+{\mathbf{r}}_x)\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathbf{p}}_x+{\mathbf{r}}_x)\right)\right]\hfill \end{array}$$

2. The $l_1$ loss of the estimate.

$$\begin{array}{cc}\hfill {\mathbb{E}}_{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}\left[l_1(\widehat{\mathbf{p}},\mathbf{p})\right]& =\mathbb{E}\left(\sum _{x\in \mathcal{D}}|{\widehat{\mathbf{p}}}_x-{\mathbf{p}}_x|\right)\hfill \\ \hfill & =\frac{1}{{\mathcal{S}}_{{\mathcal{D}}_S}}\sum _{x\in \mathcal{D}}\mathbb{E}\left[\right|\widehat{\mathbf{m}}-{\mathbf{m}}_x\left|\right]\hfill \\ \hfill & =\frac{1}{{\mathcal{S}}_{{\mathcal{D}}_S}}\sum _{x\in \mathcal{D}}\frac{\sqrt{\mathrm{Var}\left(n{\widehat{\mathbf{m}}}_x\right)}}{n}\underset{}{error}\left[\left|\frac{n{\widehat{\mathbf{m}}}_x-\mathbb{E}\left(n{\widehat{\mathbf{m}}}_x\right)}{\sqrt{\mathrm{Var}\left(n{\widehat{\mathbf{m}}}_x\right)}}\right|\right].\hfill \end{array}$$

It follows from the central limit theorem that $\frac{n{\widehat{\mathbf{m}}}_x-\mathbb{E}\left(n{\widehat{\mathbf{m}}}_x\right)}{\sqrt{\mathrm{Var}\left(n{\widehat{\mathbf{m}}}_x\right)}}$ converges to the normal distribution $\mathcal{N}(0,1)$. Hence,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\underset{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}{error}\left[\left|\frac{n{\widehat{\mathbf{m}}}_x-\mathbb{E}\left(n{\widehat{\mathbf{m}}}_x\right)}{\sqrt{\mathrm{Var}\left(n{\widehat{\mathbf{m}}}_x\right)}}\right|\right]=\sqrt{\frac{2}{n\pi }}\end{array}$$

Therefore,

$$\begin{array}{cc}\hfill {\mathbb{E}}_{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}\left[l_1(\widehat{\mathbf{p}},\mathbf{p})\right]& \approx \frac{1}{{\mathcal{S}}_{{\mathcal{D}}_S}}\sqrt{\frac{2}{n\pi }}\sum _{x\in \mathcal{D}}\sqrt{{\mathbf{m}}_x-{\mathbf{m}}_x^2}\hfill \\ \hfill & =\sqrt{\frac{2}{n\pi }}\sum _{x\in \mathcal{D}}\sqrt{({\mathbf{p}}_x+{\mathbf{r}}_x)\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathbf{p}}_x+{\mathbf{r}}_x)\right)}\hfill \end{array}$$

□

Appendix B.2. Proof of Lemma 1

Proof. 

$$\begin{array}{c}\hfill f_x=({\mathbf{p}}_x+{\mathbf{r}}_x)\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathbf{p}}_x+{\mathbf{r}}_x)\right)=({\mathbf{p}}_x+{\mathbf{r}}_x)\left(\sum _{x\in \mathcal{D}\setminus \left\{x\right\}}{\mathbf{r}}_x+1-{\mathbf{p}}_x\right).\end{array}$$

Then,

$$\left\{\begin{array}{cc}\frac{\partial f_x}{\partial {\mathbf{r}}_x}=\sum _{x\in \mathcal{D}\setminus \left\{x\right\}}{\mathbf{r}}_x+1-{\mathbf{p}}_x>0,\hfill & \\ \frac{\partial f_x}{\partial {\mathbf{r}}_{x^{\prime }}}={\mathbf{p}}_x+{\mathbf{r}}_x>0\hfill & \mathrm{if}{x}^{\prime }\in D\setminus \left\{x\right\}.\hfill \end{array}\right.$$

Apparently, $f_x$ is a monotonically increasing function of ${\mathbf{r}}_x$ for all $x\in \mathcal{D}$. Then,

$$\frac{\partial L}{\partial {\mathbf{r}}_x}=C{g}^{\prime }\circ f_x·\frac{\partial f_x}{\partial {\mathbf{r}}_x}+C\sum _{x^{\prime }\in \mathcal{D}\setminus \left\{x\right\}}{g}^{\prime }\circ f_{x^{\prime }}·\frac{\partial f_{x^{\prime }}}{\partial {\mathbf{r}}_x}>0$$

Therefore, L is a monotonically increasing function of ${\mathbf{r}}_x$ for all $x\in \mathcal{D}$. Moreover, due to the loss L is non-negative and ${\mathbf{r}}_x$ is a monotonically decreasing function of ${\epsilon }_x$ for $x\in \mathcal{D}$, the proposition holds. □

Appendix B.3. Proof of Lemma 2

Proof. 

To prove this lemma, we consider a more general optimization problem as

$$\begin{array}{c}\hfill F\left(w\right)=\underset{\theta }{max}\sum _{i=1}^{K}g\left[({\theta }_i+c_i)(C-({\theta }_i+c_i))\right]\mathrm{s}.\mathrm{t}.{\theta }^{\top }\mathbf{1}=w,\mathbf{0}⩽\theta ⩽\mathbf{1}w,\end{array}$$

where $\theta$ and $\mathbf{c}$ are vectors with k-dimension, $\mathbf{c}$ is a constant vector, $w\in (0,1)$, and C is a large enough positive constant (e.g. $C\gg {\mathbf{c}}^{\top }\mathbf{1}+1$).

First, we find a proper constant vectors $\mathbf{c}$ to obtain the optimal $\theta$ without zero elements. Since g is any monotonically increasing concave function with $g\left(0\right)=0$, according to Jensen inequality, we have

$$\begin{array}{c}\hfill \sum _{i=1}^{K}g\left[({\theta }_i+c_i)(C-({\theta }_i+c_i))\right]⩽g\left[K\sum _{i=1}^K\left[({\theta }_i+c_i)(C-({\theta }_i+c_i))\right]\right],\end{array}$$

where the equality holds iff ${\theta }_1+c_1=\cdots ={\theta }_K+c_K$. Hence, to satisfy the equality condition, we have

$$\begin{array}{c}\hfill \sum _{i=1}^K({\theta }_i+c_i)=w+\sum _{i=1}^K{c}_i\Rightarrow {\theta }_i=\frac{w+{\sum }_{j=1}^K{c}_j}{\left|k\right|}-c_i.\end{array}$$

Then, since $0<{\theta }_i<w$,

$$\begin{array}{c}\hfill 0<\frac{w+{\sum }_{j=1}^K{c}_j}{K}-c_i<w.\end{array}$$

Therefore, if all elements of $\mathbf{c}$ satisfy

$$\begin{array}{c}\hfill \frac{w+{\sum }_{j=1}^K{c}_j}{K}-w<c_i<\frac{w+{\sum }_{j=1}^K{c}_j}{K},\end{array}$$

we can ensure that the optimal $\theta$ has no zero elements, and the maximum value is

$$\begin{array}{cc}\hfill F\left(w\right)& =g\left[K\sum _{i=1}^K\left[\frac{w+{\sum }_{j=1}^K{c}_j}{K}\left(C-\frac{w+{\sum }_{j=1}^K{c}_j}{K}\right)\right]\right]\hfill \\ \hfill & =g\left[K^2·\frac{w+{\sum }_{i=1}^K{c}_i}{K}\left(C-\frac{w+{\sum }_{i=1}^K{c}_i}{K}\right)\right]\hfill \\ \hfill & =g\left[\left(w+\sum _{i=1}^K{c}_i\right)\left(KC-\left(w+\sum _{i=1}^K{c}_i\right)\right)\right]\hfill \end{array}$$

Next, we consider the general case, where the optimal $\theta$ contains zero elements. Let

$$\begin{array}{c}\hfill \tilde{F}\left(w\right)=\sum _{t=1}^T{\tilde{F}}_t\left(w\right)=\sum _{t=1}^{T}g\left[((1-w){\varphi }_t+c_t)(C-((1-w){\varphi }_t+c_t))\right],\end{array}$$

where $c_t$ is a constant which satisfies $c_t⩾\frac{w+{\sum }_{i=1}^K{c}_i}{K}$, and ${\varphi }_t$ is a constant which satisfies ${\sum }_{t=1}^T{\varphi }_t=1$ and $0<{\varphi }_t<1$.

Let $H\left(w\right)=TF\left(w\right)+\tilde{F}\left(w\right)$. Then,

$$\begin{array}{cc}\hfill H^{\prime }& =T{F}^{\prime }·\left[KC-\left(w+\sum _{i=1}^K{c}_i\right)-\left(w+\sum _{i=1}^K{c}_i\right)\right]\hfill \\ \hfill & +\sum _{t=1}^T{\varphi }_t{\tilde{F}}_t^{\prime }·[((1-w){\varphi }_t+c_t)-(C-((1-w){\varphi }_t+c_t))]\hfill \\ \hfill & =T\frac{KC-2\left(w+{\sum }_{i=1}^K{c}_i\right)}{1/F^{\prime }}+\sum _{t=1}^T\frac{2[(1-w){\varphi }_t+c_t]-C}{1/\left({\varphi }_t{\tilde{F}}_t^{\prime }\right)}\hfill \\ \hfill & =\sum _{t=1}^T\left[\frac{C-2\frac{w+{\sum }_{i=1}^K{c}_i}{K}}{1/\left(K{F}^{\prime }\right)}+\frac{2[(1-w){\varphi }_t+c_t]-C}{1/\left({\varphi }_t{\tilde{F}}_t^{\prime }\right)}\right]\hfill \\ \hfill & ⩾\sum _{t=1}^T\frac{C-2\frac{w+{\sum }_{i=1}^K{c}_i}{K}+2[(1-w){\varphi }_t+c_t]-C}{max\left[1/\left(K{F}^{\prime }\right),1/\left({\varphi }_t{\tilde{F}}_t^{\prime }\right)\right]}\hfill \\ \hfill & ⩾2\sum _{t=1}^T\frac{(1-w){\varphi }_t+c_t-\frac{w+{\sum }_{i=1}^K{c}_i}{K}}{max\left[1/\left(K{F}^{\prime }\right),1/\left({\varphi }_t{\tilde{F}}_t^{\prime }\right)\right]}\hfill \\ \hfill & ⩾0.\hfill \end{array}$$

Since $H^{\prime }⩾0$, $H\left(w\right)$ reaches its maximum when $w=1$.

Finally, because any general case of the optimization problem in this lemma can be convert to the function H, the lemma holds. □

Appendix B.4. Proof of Theorem 4

Proof. 

For save the pages, we won’t give the details of the proof, since we just get the result by substituting ${\mathbf{p}}^{*}$ of Lemma 2 into (13) and (14). □

Appendix B.5. Proof of Theorem 5

Proof. 

As ${\epsilon }_{max}$ is close to 0, ${\mathcal{D}}^{*}$ will become ${\mathcal{D}}_N$. Hence, according to Lemma 2, when ${\mathcal{D}}^{*}={\mathcal{D}}_N$, $min\left\{\mathbf{r}\right\}=\frac{1}{e^{{\epsilon }_{max}}-1}>\frac{1}{|{\mathcal{D}}_N|}\Rightarrow 0<{\epsilon }_{max}<ln\left(\right|{\mathcal{D}}_N|+1)$. In this case, ${\mathbf{p}}^{*}={\mathbf{p}}^{\mathrm{u}}$, where

$$\begin{array}{c}\hfill {\mathbf{p}}_x^{\mathrm{u}}=\left\{\begin{array}{cc}\frac{1}{|{\mathcal{D}}_N|}\hfill & \mathrm{if}x\in {\mathcal{D}}_N,\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

Therefore,

$$\begin{array}{cc}\hfill {\mathbb{E}}_{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}\left[l_2(\widehat{\mathbf{p}},\mathbf{p})\right]& ⩽{\mathbb{E}}_{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}\left[l_2(\widehat{\mathbf{p}},{\mathbf{p}}^{\mathrm{u}})\right]\hfill \\ \hfill & =\frac{1}{n}\left[\sum _{x\in {\mathcal{D}}_S}\left[{\mathbf{r}}_x\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-{\mathbf{r}}_x\right)\right]+\sum _{x\in {\mathcal{D}}_N}\left[\frac{1}{|{\mathcal{D}}_N|}\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-\frac{1}{|{\mathcal{D}}_N|}\right)\right]\right]\hfill \\ \hfill & =\frac{1}{n}\left[\sum _{x\in {\mathcal{D}}_S}\left[{\mathbf{r}}_x\left(\sum _{x^{\prime }\in \mathcal{D}}{\mathbf{r}}_{x^{\prime }}+1-{\mathbf{r}}_x\right)\right]+\sum _{x\in \mathcal{D}}{\mathbf{r}}_x+1-\frac{1}{|{\mathcal{D}}_N|}\right]\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \approx \frac{1}{n}\left[\sum _{x\in {\mathcal{D}}_S}\left[\frac{1}{{\epsilon }_x}\left(\sum _{x^{\prime }\in \mathcal{D}}\frac{1}{{\epsilon }_{x^{\prime }}}+1-\frac{1}{{\epsilon }_x}\right)\right]+\sum _{x\in \mathcal{D}}\frac{1}{{\epsilon }_x}+1-\frac{1}{|{\mathcal{D}}_N|}\right]\hfill \\ \hfill & =\frac{1}{n}\left[{\left(\sum _{x\in {\mathcal{D}}_S}\frac{1}{{\epsilon }_x}\right)}^2-\sum _{x\in {\mathcal{D}}_S}\frac{1}{{\epsilon }_x^2}+\sum _{x\in \mathcal{D}}\frac{2}{{\epsilon }_x}+1-\frac{1}{|{\mathcal{D}}_N|}\right]\hfill \\ \hfill & \approx \frac{1}{n}\left[{\left(\sum _{x\in {\mathcal{D}}_S}\frac{1}{{\epsilon }_x}\right)}^2-\sum _{x\in {\mathcal{D}}_S}\frac{1}{{\epsilon }_x^2}\right]\hfill \\ \hfill & =\frac{1}{n}\sum _{x\in {\mathcal{D}}_S}\sum _{x^{\prime }\in {\mathcal{D}}_S\setminus \left\{x\right\}}\frac{1}{{\epsilon }_x{\epsilon }_{x^{\prime }}},\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill {\mathbb{E}}_{Y^n\sim \mathbf{Q}}\left[l_1(\widehat{\mathbf{p}},\mathbf{p})\right]& ⩽{\mathbb{E}}_{Y^n\sim \mathbf{Q}}\left[l_1\left(\widehat{\mathbf{p}},{\mathbf{p}}^{\mathrm{u}}\right)\right]\hfill \\ \hfill & =\sqrt{\frac{2}{n\pi }}\left[\sum _{x\in {\mathcal{D}}_S}\sqrt{{\mathbf{r}}_x\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-{\mathbf{r}}_x\right)}+\sqrt{|{\mathcal{D}}_N|{\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-1}\right]\hfill \\ \hfill & ⩽\sqrt{\frac{2}{n\pi }}\left[\sqrt{|{\mathcal{D}}_S|{\sum }_{x\in {\mathcal{D}}_S}{\mathbf{r}}_x\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-{\mathbf{r}}_x\right)}+\sqrt{|{\mathcal{D}}_N|{\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-1}\right]\hfill \\ \hfill & \approx \sqrt{\frac{2}{n\pi }}\left[\sqrt{|{\mathcal{D}}_S|{\sum }_{x\in {\mathcal{D}}_S}\left[\frac{1}{{\epsilon }_x}\left({\sum }_{x^{\prime }\in \mathcal{D}}\frac{1}{{\epsilon }_{x^{\prime }}}+1-\frac{1}{{\epsilon }_x}\right)\right]}\right.\hfill \\ \hfill & \left.+\sqrt{|{\mathcal{D}}_N|{\sum }_{x\in \mathcal{D}}\frac{1}{{\epsilon }_x}+\left|{\mathcal{D}}_N\right|-1}\right]\hfill \\ \hfill & \approx \sqrt{\frac{2}{n\pi }}\sqrt{|{\mathcal{D}}_S|{\sum }_{x\in {\mathcal{D}}_S}\left[\frac{1}{{\epsilon }_x}\left({\sum }_{x^{\prime }\in \mathcal{D}}\frac{1}{{\epsilon }_{x^{\prime }}}+1-\frac{1}{{\epsilon }_x}\right)\right]}\hfill \\ \hfill & =\sqrt{\frac{2}{n\pi }}\sqrt{|{\mathcal{D}}_S|\left[{\left({\sum }_{x\in {\mathcal{D}}_S}\frac{1}{{\epsilon }_x}\right)}^2+{\sum }_{x\in {\mathcal{D}}_S}\frac{1}{{\epsilon }_x}-{\sum }_{x\in {\mathcal{D}}_S}\frac{1}{{\epsilon }_x^2}\right]}\hfill \\ \hfill & \approx \sqrt{\frac{2}{n\pi }}\sqrt{|{\mathcal{D}}_S|\left[{\left({\sum }_{x\in {\mathcal{D}}_S}\frac{1}{{\epsilon }_x}\right)}^2-{\sum }_{x\in {\mathcal{D}}_S}\frac{1}{{\epsilon }_x^2}\right]}\hfill \\ \hfill & =\sqrt{\frac{2}{n\pi }}\sqrt{|{\mathcal{D}}_S|{\sum }_{x\in {\mathcal{D}}_S}{\sum }_{x^{\prime }\in {\mathcal{D}}_S\setminus \left\{x\right\}}\frac{1}{{\epsilon }_x{\epsilon }_{x^{\prime }}}}\hfill \end{array}$$

□

Appendix B.6. Proof of Theorem 6

Proof. 

According to Lemma 2, when ${\epsilon }_{min}>ln\left(\right|{\mathcal{D}}_N|+1)$, $\mathbf{m}$ is a uniform distribution. In this case, ${\mathcal{D}}^{*}=\mathcal{D}$. Therefore,

$$\begin{array}{cc}\hfill {\mathbb{E}}_{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}\left[l_2(\widehat{\mathbf{p}},\mathbf{p})\right]& ⩽{\mathbb{E}}_{Y^n\sim \mathbf{m}\left(\mathbf{Q}\right)}\left[l_2(\widehat{\mathbf{p}},{\mathbf{p}}^{*})\right]\hfill \\ \hfill & =\frac{1}{n}\sum _{x\in \mathcal{D}}\left[({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\right)\right]\hfill \\ \hfill & =\frac{1}{n}\left|\mathcal{D}\right|\left[({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\right)\right]\hfill \\ \hfill & =\frac{1}{n}\left[{\mathcal{S}}_{\mathcal{D}}^{-1}\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\right)\right]\hfill \\ \hfill & =\frac{1}{n{\mathcal{S}}_{\mathcal{D}}}\left({\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\right)\hfill \\ \hfill & =\frac{1}{n{\mathcal{S}}_{{\mathcal{D}}_S}^2}\left(1-\frac{1}{\left|\mathcal{D}\right|}\right),\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill {\mathbb{E}}_{Y^n\sim \mathbf{Q}}\left[l_1(\widehat{\mathbf{p}},\mathbf{p})\right]& ⩽{\mathbb{E}}_{Y^n\sim \mathbf{Q}}\left[l_1\left(\widehat{\mathbf{p}},{\mathbf{p}}^{*}\right)\right]\hfill \\ \hfill & =\sqrt{\frac{2}{n\pi }}\sum _{x\in \mathcal{D}}\sqrt{({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\left[{\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\right]}\hfill \\ \hfill & =\sqrt{\frac{2}{n\pi }}\left|\mathcal{D}\right|\sqrt{({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\left[{\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-({\mathcal{S}}_{\mathcal{D}}{\left|\mathcal{D}\right|)}^{-1}\right]}\hfill \\ \hfill & =\sqrt{\frac{2}{n\pi }}\sqrt{{\mathcal{S}}_{\mathcal{D}}^{-1}\left(\left|\mathcal{D}\right|{\mathcal{S}}_{{\mathcal{D}}_S}^{-1}-{\mathcal{S}}_{\mathcal{D}}^{-1}\right)}\hfill \\ \hfill & =\sqrt{\frac{2}{n\pi }}\sqrt{{\mathcal{S}}_{\mathcal{D}}^{-2}\left(\right|\mathcal{D}|-1)}\hfill \\ \hfill & =\sqrt{\frac{2\left(\right|\mathcal{D}|-1)}{n\pi }}{\mathcal{S}}_{\mathcal{D}}^{-1}.\hfill \end{array}$$

□