Disaggregation Model: A Novel Methodology to Estimate Customers Profiles in a Low-Voltage Distribution Grid Equipped With Smart Meters

1. Introduction

The international negotiations on climate policies reveal that we are grappling with the impacts we have on the environment. One way to mitigate these impacts directly involves reducing the use of fossil fuels and increasing electrification. This leads to changes in the level the distribution electrical network is used. These changes extend from the generation and distribution of energy to the way customers use it. They are part of a phenomenon called energy transition. Its effects can be immediately seen when we look at the new directive of the European Union, "Fit for 55," which aims to reduce greenhouse gas emissions by 55% by 2030. To achieve this goal, one of the proposals is to increase the renewable energy target from 32% to at least 40% by 2030 [1].

We can also take a closer look at the energy transition by examining the French scenario in more detail. According to data from the DSO Enedis, responsible for energy distribution in 95% of metropolitan France, the number of production sites connected to the grid increased by 178% between the years 2012 and 2023. The same data show us that the total number of electric vehicle charging infrastructures connected to the grid grew by 2000% between 2015 and 2023 [2]. The French Ministry of Ecological Transition estimates the distribution of residential energy use in France by type. The heating system represents approximately 70% of the country's residential energy consumption. Among the various energy sources used for residential heating, electric sources have been growing proportionally year after year, as indicated by this data [3]. We can therefore observe that the fight against environmental impacts is reflected in a significant change in the habits of electricity production and consumption, centered in the distribution grid.

As the uses of the grid evolve, the distribution grid also evolves to keep pace with these changes. The electrical grids are becoming more digital, transforming our networks into smart grids. According to the International Energy Agency, “smart grids are electrical networks that use digital technologies, sensors, and software to better match the supply and demand for electricity in real-time while minimizing costs and maintaining the stability and reliability of the network.” The major enabler of this digitization lies in the replacement of electricity meters, transitioning from analog/digital meters to smart meters. Smart meters are electronic devices that record information such as electrical energy consumption and communicate this information to the consumer, electricity providers, and DSOs.

We can look at some examples of this evolution in the metering landscape around the world. The Italian DSO ENEL initiated the deployment of its first generation of smart meters in 2001 for your 32 million customers [4]. In 2016, the Italian regulatory body defined the minimum functions for the new generation of meters. This new generation of meters began to be distributed in 2017, with the aim of making energy consumption more sustainable and becoming a key element in the construction of smarter and more circular cities [5]. The U.S. DSO PG&E launched the SmartMeter™ program for the installation of smart meters in 2006, and the majority of its customers were equipped with them by 2012 [6]. The Canadian DSO BC Hydro is the primary electricity distributor in the British Columbia region, serving over 4 million customers in most areas of the province. The deployment of smart meters began in 2011 and was completed in 2012 [7]. The Korean Electric Power Corporation, KEPCO, is one of the largest energy companies in South Korea. Since 2012, the KEPCO has installed approximately 120,000 smart meters to promote the smart grid infrastructure. Until 2018, 6.8 million households in South Korea were equipped with smart meters. The goal was to reach 22.5 million households by the early 2020s [8]. TEPCO, the largest Japanese DSO, completed the installation of approximately 29 million smart meters by the end of 2020 [9]. The French DSO Enedis began rolling out its meters in 2015 and, from 2023 last figures, has over 37 million smart meters deployed. [10] It is possible to observe that the change in the metering landscape is a global phenomenon, occurring in parallel with changes in grids usage.

The exploitation of smart meter data makes it possible to explore new solutions that helps the grid to receive this new usage in the energy transition context. The objective of this paper is to propose a novel methodology to estimate the load curves of consumers on the low-voltage network using very few data from smart meters. This modelling will be based on data provided by the French DSO Enedis, but the model had been built to be compliant with data from most smart metering infrastructure. The possible constraints that may be faced when using smart meter data for load curve estimation are related to data privacy concerns and IT infrastructure limitations. The aim of this model is to explore the use of smart meter data for profile or load curve estimation and be a model that can be employed to assist in the integration of new usages. To achieve this, the model starts with the load curve of the secondary substation and disaggregates it into individual load curves of customers, as illustrated in Figure 1. The idea behind this model is to perform daily disaggregation, allowing us to access the customer-level load curve every day. The customer load curve is the input data for network calculation tools, enabling us to compute voltage and power transits in the network. With this daily customer-generated load curve data, it is possible to better estimate the hosting capacity of the network as well as the available flexibility of the demand.

Looking at the literature, T. Barbier [11] conducted a classification of different types of electrical consumption models. This classification is based on two factors. The first factor is a framework with three levels of consumption, including the individual level of the customer and their devices, the aggregated level by type of customer and type of device, as well as the aggregated level of the customer by spatial zone and their devices. The second factor is a framework of variables influencing consumption, including the device inventory, the technology of these devices, usage behavior, socioeconomic level, meteorology, variables related to the place of consumption, customer subscription, and the electricity market. Thus, he lands at 6 categories of models.

The first category includes models based on aggregated consumptions in a zone. This category includes models where the scale is the level of electrical consumption aggregated by spatial zone. These models primarily focus on the temporal component of consumption and are commonly referred to as “forecasting” models. In this category, [12,13,14,15] stand out in the analysis and evaluation of this type of forecasting model.

The second category consists of models based on consumptions by type of customer. This category includes models where the scale is the level of consumption per type of customer. This type of modelling produces different load curves for each type of customer (residential, industrial, etc.), and customers of the same type will have identical profiles, adjusted in energy [16,17,18]. This category can be subdivided based on the model's input data.

According to the author [11], the third category consists of models based on consumption by usage. An example of this type of model is one that decomposes thermosensitive uses from non-thermosensitive uses based on load curves from different zones [19]. It illustrates the example of thermosensitivity in France, allowing the Transmission System Operator and Distribution System Operators to understand the influence of meteorological hazards on network sizing.

The next category consists of models based on consumption by type of customer and usage. This category essentially includes models for the residential sector that are based on consumption by type of use of equipment in the sector. This category comprises models [20,21,22] that involve disaggregating electrical consumption profiles with information on the ownership rate of devices and possibly with socio-economic data. These are “bottom-up” and “top-down” models.

The fifth category consists of models based on consumption per customer. According to the author [1], models belonging to this category use data from smart meters to make short-term predictions of electrical consumption for individual customers [23,24,25].

The last category consists of models based on consumption per device. This category aims to estimate consumption at different levels through the aggregation of device consumption. An example of this is Non-Intrusive Load Monitoring models [26,27,28]. The idea is to have a sufficiently fine temporal resolution of measurements to capture the energy signatures of different devices.

Based on this analysis of the classification of different types of electricity consumption models conducted in [11], we can conclude that "bottom-up" and "top-down" models are the ones that come closest to the initial idea of disaggregation models. However, the scale at which we intend to build the disaggregation model is that of the secondary substation level to assess the load curve or profile of each LV customer attached to it. This is not clearly represented in this classification. The load curves of customers are the inputs for the DSO's tools to carry out network planning studies. Therefore, estimating them accurately becomes a necessary task in this context of evolving network usage. The use of usage decomposition methodologies such as "bottom-up" proves to be extremely dependent on measurements with fine temporal resolution. DSOs may not necessarily have this degree of precision in their measurements. Our disaggregation model avoids this situation, making it more accessible.

This literature review confirms that we are seeking to undertake innovative research that will contribute to expanding the horizons of the field. Keeping this in mind, let us explore the idea novel methodology into details.

2. Materials and Methods

The general principle of the model is to disaggregate the load curve of the substation into load curves for each downstream customer. To achieve this, we will carry out this process in two steps:

• Customer Segmentation: The N customers of a substation share similar characteristics, either because they belong to the same categories (residential, professional, etc.) or because they have similar consumption habits. In this perspective, the segmentation (or the number of clusters) of these customers is performed to identify K groups of similar customers among the N customers of the substation, where K < N.
• Secondary Substation Load Disaggregation: The load curve of the substation is then disaggregated into K curves, representing the K groups of similar customers at the substation. These K curves are then adjusted in energy to assign to each customer the curve of the group to which they belong.

The data scope chosen for the development of this model included 48 secondary substations, where load curve data was collected for the period from March 2022 to March 2023. Additionally, maximum power and energy data from Linky smart meters (a total of 5318 meters) were collected downstream of these substations for each day of the period. It is essential to specify that the Linky data of the customers used in this study are associated with a customer panel, who have given their consent to Enedis for the use of their data in the context of network studies. Including the one integrated into this article, in accordance with the General Data Protection Regulation (GDPR). Therefore, these data are not accessible beyond this perimeter. We have three types of information per secondary substation:

• The load curve of the substation.
• The maximum power value in watts measured by Linky for all customers connected in the substation.
• The time of day (hours and minutes) when the maximum power occurred.

With all this information in mind, we can delve into the details of each step of the model.

2.1. Customer Segmentation

The objective of this step is to identify K groups of similar customers at a substation with N customers. The idea behind this is to simplify the subsequent disaggregation step, as we will disaggregate the substation's load curve into fewer curves (K < N). Additionally, working at an individual customer level is extremely complex due to the significant variability in consumption. Therefore, working at a slightly higher level (K groups of similar customers) is considered the optimal strategy. Various temporal scales could be used to segment customers at a substation, such as daily, by days of the week, monthly, seasonal, yearly, among others.

A segmentation by season was chosen as it offers more advantages. It provides a sufficient volume of data for customer segmentation and allows to visualize the evolution of the behavior of all customers over time. Customers who exhibit similar behavior within a given season may not necessarily show similar behavior in other seasons. The data selected for building this dataset are maximum power data. The decision to use only this data is to simplify the construction process by leveraging one of the pieces of information collected by Linky meters. Moreover, the simpler the information used in model construction, the easier it will be to reproduce it in other research.

To preprocess the data, the dimensionality reduction technique called Principal Component Analysis (PCA) was employed [29]. PCA is a statistical technique used to reduce the dimensionality of data while preserving, as much as possible, the essential information contained in the data. Specifically, this method is beneficial for reducing the size of processed data, limiting data redundancy, and facilitating easier data visualization, leading to better understanding [29].

In the literature, various methods exist for performing segmentation. For the implementation of this disaggregation model step, we chose to work with partition methods, specifically k-means [30], due to their simplicity in understanding the proposed results. Kmeans clustering is one of the most well-known and widely used unsupervised learning algorithms. Generally, unsupervised algorithms make inferences from datasets using only input vectors without reference to known or labeled outcomes [31]. The goal of Kmeans clustering is to group similar data points and discover underlying clusters. To determine the number of groups K that will be assigned in our dataset, the elbow method [32] is employed. It aims to identify the number of clusters K that strikes a good balance between reducing the sum of squared errors (SSE) and the complexity of the model. It is an easy-to-understand and apply method, making it a practical tool for choosing K. However, the drawback is the subjectivity of the method, as identifying the elbow can be challenging. For this reason, we chose to use the Python library Kneed [33] to automatically detect elbows and eliminate the subjective aspect. The Figure 2 illustrates the step of customer segmentation.

2.2. Secondary Substation Load Disaggregation

The objective of this step is to formulate a method to disaggregate the load curve of a secondary substation into K curves that represent the K groups among the N customers identified in the segmentation step. To begin thinking about how to achieve this solution, let start by formulating the problem mathematically.

$${Load Curve}_{\begin{array}{c}secondary \\ substation\end{array}}=\sum _{k=1}^{K}Cluster Curve \left(k\right)$$

(1)

As shown in Equation 1, the disaggregation of the substation's load curve can be seen as the sum of K curves being equal to the substation's curve. Approaching the problem from this perspective, once the equations describing the K curves are defined, this problem can be solved by applying a curve fitting method. Curve fitting is a commonly used technique in data analysis to estimate the parameters of a mathematical function that best describes a set of data [34]. Its basic principle is to find the optimal parameters of a mathematical function that minimizes the square of the difference between the observed data and the values predicted by the function, as shown in Equation 2:

$$min( \sum _{i=1}^M{\left(y_i-f\left(x_i,\theta \right) \right)}^2 )$$

(2)

The goal is to find 𝜃_{𝑜𝑝𝑡𝑖𝑚𝑎𝑙} values that minimize quadratic error, called the cost function. In other words, curve fitting involves solving an optimization problem where the objective is to identify the values of the parameters 𝜃 that make the cost function as small as possible. To solve this curve fitting problem, the Python library lmfit [35] was chosen. As explained by the developers, lmfit provides a user-friendly interface for defining models, specifying parameters, fitting data, and retrieving results. It also allows the choice of a variety of optimization methods. In the case of this disaggregation problem, the chosen method is L-BFGS-B (Limited-memory Broyden-Fletcher-Goldfarb-Shanno with Bound constraints) [36]. With the definition of how the disaggregation problem will be approached, it is now necessary to define which mathematical function best describes our data. The data on which the curve fitting will be performed are the daily load curves of the secondary substations.

2.2.1. Function One – Pure Mathematical Model

The option chosen to address the question of which equations best describe the sum of K curves in Equation 2 is a sum of N Gaussians, as shown in Equation 3.

$$f\left(x,a,b,c\right)= \sum _{k=1}^N(a_k\ast {exp}^{\frac{{-(x-b_k)}^2}{{{2\ast c}_k}^2}})$$

(3)

Where: formally θ in the function f(x, θ) now corresponds to the parameters $a_k$, $b_k$, $c_k$. $a_k$ determines the height of the Gaussian, $b_k$ determines the position of the center of the Gaussian, and $c_k$ determines the width of the Gaussian.

The reason that leads to believe that this assumption is a good starting point is as follows: if we examine the shape of customer load profiles [37,38], the consumption peaks at different times of the day resemble Gaussian behavior. These profiles represent the behavior of a group of customers belonging to the same class. Furthermore, a Gaussian is a natural candidate because it is a classic function in statistics related to the central limit theorem. This theorem plays a fundamental role in statistical theory by showing that the mean of many independent and identically distributed random variables approximately follows a Gaussian distribution, regardless of the initial distribution of these variables. This demonstrates its ubiquity in modeling random phenomena, such as the electrical consumption of low-voltage customers.

2.2.2. Function Two – Adding Electrical Properties to the Purely Mathematical Model

The idea behind this function is to use the information available thanks to Linky meters to approximate the modeling of the real behavior of network customers. Starting with an analysis for a single customer, all the times at which their maximum power was recorded during the season can be examined. It can be observed that these times are not always identical. This is illustrated in Figure 3, where each blue point represents an occurrence of maximum power for 5 examples of customers during the winter season.

It can be observed that each customer exhibits different consumption patterns, and they vary considerably throughout the season. This leads us to believe that observing a group of aggregated customers could reveal patterns that are easier to identify. Figure 4 shows the occurrence of maximum power throughout the winter season for two different groups of customers. Group 1 consists of 70 consumers, and group 2 has 25. They are presented in Figure 4 through a histogram where each bar corresponds to a 10-minute interval that measures the occurrences of maximum power capture by Linky infrastructure.

Observing the occurrences by group makes the trends more apparent. For group 1, four concentrations of maximum power occurrences can be noted: around noon, 8 pm, 8 am, and midnight. For group 2, there is a significant concentration at 7 am and less pronounced concentrations around noon and 8 am. As the previous stage of disaggregation segments the customers from the secondary substation, we will have formed the groups of customers to analyze. Now, it is necessary to translate this information into adjusting the parameters of our function.

The concentrations of maximum power at a given period indicate whether it's a significant time for the cluster, deserving high resolution to finely adjust the peaks. Therefore, for each cluster, we can say that the most pronounced concentrations in the histograms will indicate the maximum number of Gaussians as well as the initial value of the "b" parameter for each of these Gaussians. To identify these concentrations, the “find_peaks” function from the Python SciPy library was used [39]. It allows for the identification and localization of local maxima in a one-dimensional dataset by simple comparison of neighboring values. The times of maxima will be denoted by the letter t, and the heights at which they appear will be denoted as $h_t$. This guides us towards another way of expressing our sum of Gaussians function f(x, a, b, c) and adjusting the parameter bounds. For each cluster k, we will have T numbers of Gaussians found from the histogram of maximum power occurrences, as indicated by Equation 4.

$$\begin{gathered} f\left(x,a,b,c\right)=\sum _{k=1}^K\sum _{t=1}^T\left(a_{t,k}\ast {exp}^{\frac{{-\left(x-b_{t,k}\right)}^2}{{{2\ast c}_{t,k}}^2}}\right) \\ 0 \le {\mathit{a}}_{\mathit{t},\mathit{k}}<\mathit{max}\left(Secondary Substation Curve\right) \\ { h}_t-0.5hours \le {\mathit{b}}_{\mathit{t},\mathit{k}} \le h_t+0.5hours \\ 1hour<{\mathit{c}}_{\mathit{t},\mathit{k}}<6hours \end{gathered}$$

(4)

Having the parameter $a_{t,k}$ presenting a possible value of zero allows the optimization to discard the Gaussian (t,k) if it is not significant for curve fitting, setting its height to zero. This accounts for the variable behavior indicating that customers in this group may not have their maximum power at that time every day. The parameter $b_{t,k}$ capable of varying by about half an hour around the time $h_t$, allows for moving the center of the Gaussian, which aligns with the occurrences of maximum power around each maximum. The parameter $c_{t,k}$ represents the width of the Gaussian, with the lower bound set to one hour. This prevents the presence of peaks that appear and disappear very quickly, as we assume that the consumption of the customer group has some inertia in its variations. The upper bound for the $c_{t,k}$ parameter is limited to 6 hours to avoid losing information about a possible important peak.

2.3. Global Vision of the Model

To facilitate the overall view of the model, a schematic gathering all its stages is presented in Figure 5.

2.4. Error Evaluation

The error produced by the model is calculated using symmetric mean average percentage error (SMAPE). SMAPE is a commonly used metric to assess the accuracy of forecasts or prediction models. It calculates the accuracy of predictions by comparing actual and predicted values in a symmetric way, as shown in the Equation 5:

$$SMAPE=\frac{1}{N}\times \sum _{n=1}^N\frac{\left|P_n-R_n\right|}{\raisebox{1ex}{$(\left|R_n\right|+\left|P_n\right|)$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$$

(5)

Where P is the predicted value and R is the real value. So, for each data point, it takes the absolute difference between the actual value and the predicted value, and then divides it by the sum of the actual and predicted values. SMAPE is robust to small values. Due to its symmetry, SMAPE ensures that the penalty for error is not sensitive to large differences between the prediction and the actual value.

With the help of SMAPE, we will evaluate the error between the reconstruction of the load curve of the secondary substation based on the model's result and the true load curve of the secondary substation. This way, we will be able to assess how well the disaggregation model can reconstruct the load curves of the secondary substation.

3. Results

The results of the disaggregation model are presented for both functions described in Section 2.2. The evaluation of the results for both models is conducted along two cases. On the first case, a random secondary substation is selected, and the model is applied to the load curve for the winter season. On the second case, the error is computed for all secondary substations on all days of the winter season.

3.1. Results for One Random Secondary Substation

A secondary substation was randomly chosen from the set of 48 in our dataset. This secondary substation has 198 connected consumers. The result of consumer segmentation, presented in 2.1, led us to the number K of similar customer groups equal to 6. The quantity of customers in each group can be observed in Table 1.

For each day of the winter season, our disaggregation model is put into practice using the two functions described in Section 2.2 and the 6 groups of customers. The results of the errors calculated are presented in Table 2.

In order to look at the load curves produced by the model with both functions, the days with minimal SMAPE error are select. These curves are illustrated by the Figure 6.

The individual load profiles of each cluster are presented by below. The individual load profiles produced by the model using function one and two are presented by Figure 7 and Figure 8 respectively.

3.2. Results for All Secondary Substation in the Dataset

For the 48 secondary substations, the total quantity of groups obtained by the segmentation step is presented in Table 3.

For each day of the winter season, our disaggregation model is put into practice using the two functions described in Section 2.2. The results of the errors calculated using SMAPE are presented in Table 4.

In order to look at the load curves produced by the model with both functions, the days for the secondary substations with minimal SMAPE error are selected. These curves are illustrated by the following Figure 7.

Figure 9. (a) Minimum SMAPE result produced by the model using function one; (b) Minimum SMAPE result produced by the model using function two.

Figure 9. (a) Minimum SMAPE result produced by the model using function one; (b) Minimum SMAPE result produced by the model using function two.

4. Discussion

The results show us that the disaggregation model using function 2 presents better outcomes, even though function 2 has fewer degrees of freedom than function 1. If we look closely at the results with both functions for one random secondary substation, we can observe that function 1, purely mathematical, manages to capture the large peaks and valleys of the load curve, as shown in Figure 6a. However, it fails to capture the secondary peaks, thus leaving the curve fitting result unsatisfactory. On the other hand, function 2 incorporates characteristics of customer consumption into the modeling, thanks to the data from Linky meters. The interest in considering the behavior of customers from the segmentation step is to ensure consistency with reality, thus bringing the model closer to what happens in the network. It can be observed in Figure 6b that the fitted curve has successfully captured the various consumption peaks represented by the secondary substation load curve. This result is reflected in the shapes of individual profiles, as shown in Figure 8. This highlights the consistency between the distribution of maximum power occurrences per group of customers and the consumption measurements of all customers observed by secondary substation load curve. This is reflected when comparing the errors of the functions. The error produced by function 2 is lower than that of function 1 as shown in Table 2.

The result obtained for a random secondary substation holds when we look at all the substations in our dataset. As presented in Table 4, function 2 produces results with lower errors than function 1. This assures us of the importance of the maximum power information measured by the Linky meter. Additionally, it underscores the significance of data measured by smart meters for network load modeling.

5. Conclusion and Perspectives

We have decided to reverse the natural process, which involves starting from the development of aggregated customer curves to obtain an estimate of the secondary substation load curve. By leveraging the richness of Linky meter data, we have created a way to disaggregate the substation curve from this measurement into customer profiles. Developing an estimation methodology based on data measured by smart meters allows us to better estimate customer’s consumption behavior and its changes. This is crucial in the context of integrating new uses of the grid, such as renewable energy sources and electric vehicles. Unlike classical "bottom-up" methodologies that are highly dependent on measurements with fine temporal resolution, our developed methodology is based on the occurrence time of the maximum daily power. Additionally, our approach involves a disaggregation at a higher grid level, specifically the secondary substation level. This modeling idea is not discussed in the scientific literature, making this model innovative and potentially opening doors to considering how smart meter data can assist in network sizing, integrating new uses, and adapting to changes in consumer behavior in general.

The next stage of studying and improving the model involves exploring the energy measured by Linky meters in the disaggregation process. The evolution of the model can be added to the schematic seen in 2.3. This is presented in Figure 10.

Imposing an energy constraint ensures that our result, when viewed from the aggregate of customers, can be distributed to the entire customer base of each group while respecting their consumption. This will further bring the results closer to the reality of consumption in the network. This will allow us to transition from curves aggregated by consumer segments to curves of individual consumers, properly accounting for the energy consumed by them and measured by the Linky meter.