A Novel Utility-Driven Residual Connection Inspired by Debreu's Theorem

1. Introduction

Since its introduction, residual connections have become an important part of modern deep learning architectures [1,2,3,4]. Its original purpose was to solve the learning difficulties that arise as the number of network layers increases, especially the gradient vanishing and degradation problems that are prone to occur in deep neural networks [5,6,7,8,9]. The residual structure effectively improves the information propagation capability by directly jumping the input of the previous layer to the next layer, making the optimization of deep networks easier [10,11,12]. Residual connections have been shown to significantly improve the stability and performance of many mainstream models such as ResNet, DenseNet and Transformer [13,14,15,16,17]. Therefore, residual connections have gradually evolved from a structural optimization method to a basic module for deep network design [18,19,20,21,22].

Although traditional residual connections have high practical value in engineering practice, their theoretical explanation is still relatively weak. Existing explanations mainly focus on numerical stability, gradient flow and feature reuse, and there is no theoretical model that can understand its role at a more abstract level [23,24,25,26]. In other words, although residual connections are efficient, there is still a lot of room for research on their effective mechanism and further optimization methods. Especially under the current research trend of emphasizing model interpretability and structural innovation, giving residual connections a clear theoretical meaning will help promote the network structure from "empirical construction" to "theoretical inductive construction" [27,28,29,30,31].

At the same time, in the field of economics, the utility representation theorem (i.e., Debreu's theorem) proposed by Gérard Debreu provides a solid theoretical basis for individual preference modeling [32,33,34,35,36,37,38]. The theorem states that as long as the individual's preference relationship satisfies the axiom conditions such as completeness, transitivity, and monotonicity, there exists a real-valued function (i.e., utility function) that can express preferences in numerical form and maintain order. This function is monotonically increasing without loss of generality, which means that an individual's preference for a certain state can be represented by a certain increasing utility value. This idea plays a core role in welfare economics, game theory, and microeconomics.

Interestingly, the "utility enhancement" mechanism constructed by Debreu's theorem naturally fits the needs of "information enhancement" and "feature importance enhancement" in deep learning. The output of each layer of a deep network can be regarded as the network's "intermediate representation" or "temporary judgment" of the input. By introducing a utility function that satisfies monotonicity, "preference evaluation" or "preference-driven control" can be simulated to perform nonlinear enhancement of features in a more abstract and explanatory way. The addition of this mechanism not only retains the technical advantages of traditional residual connections, but may also further enhance the expressive power of the model.

Therefore, the core of this paper is to integrate the mathematical concept of "preference-utility" mapping in Debreu's theorem into the residual connection structure of deep neural networks, and construct a new "utility residual connection mechanism" based on theoretical explanation. Structurally, this paper adopts a dual-path design, the main channel extracts traditional features, and the utility channel responds to features based on monotonic nonlinear preferences. Then the two are added and fused. This design is first applied to a standard multilayer perceptron (MLP), and used as an experimental basis to verify its effectiveness and versatility.

In summary, the contribution of this paper is not only to propose a new residual connection mechanism, but more importantly, inspired by the theory of economic preferences, to introduce an interdisciplinary theoretical perspective into the structural design of deep networks, expanding the research boundaries of artificial intelligence systems in structural representation and theoretical modeling.

2. Theoretical Motivation

Gérard Debreu's important contribution to economics, the famous Debreu theorem, puts forward a profound and widely applicable view: as long as the preference relationship of individuals satisfies several reasonable axioms, including completeness, transitivity and monotonicity, there must be a real-valued function that can represent this preference relationship. This function is called the "utility function". Its core meaning is that although human preferences are subjective and non-quantitative, under certain conditions, individual preferences can still be characterized and ranked through a monotonically increasing numerical mapping. The larger the utility function, the more preferred the state is, so the utility value can be regarded as a potential "preference intensity" signal.

This theoretical setting based on preference and aimed at enhancement provides us with a new perspective for thinking about the information processing process in neural networks. In traditional neural networks, the output of each layer is regarded as a series of intermediate representations, reflecting the model's "gradual understanding" or "feature extraction" of the input data. However, if we compare these intermediate representations to "preference states", we can further introduce a new mechanism: without changing the original structure, we use an auxiliary branch with monotonic properties to simulate the "preference enhancement" of the intermediate representation, that is, to respond positively to the value judgment of the information, thereby enhancing its influence in subsequent dissemination.

In this framework, the main channel still undertakes the traditional feature learning task, and continuously transforms the input data linearly and nonlinearly to capture the deep semantics in the hierarchical structure. The newly introduced auxiliary channel starts from the intermediate result of the main channel and applies a nonlinear response mechanism to simulate the dynamic correction of information by the preference mechanism. The core feature of this response mechanism is monotonicity - that is, for the intermediate representation given by the main channel, the auxiliary channel should provide a continuous positive mapping response. For example, by using activation methods such as Sigmoid and Softplus, the auxiliary channel can construct a "utility enhancement" performance path, so that the greater the input value, the stronger the enhancement, without reverse suppression or information interference.

Furthermore, when the outputs of the main channel and the utility channel are summed in the final stage, this process not only retains the original advantages of residual connection in numerical stability and gradient flow, but also introduces an interpretable semantic mechanism - that is, the current judgment of the network is not only a linear response to the underlying data, but also an adjustment and strengthening of its "preference intensity". This design is in line with the "utility mapping" in Debreu's theory: the stronger the output signal, the more inclined and trustful the model is, thereby increasing its contribution to the final decision.

In other words, the utility residual mechanism is no longer just a "shortcut connection" at the technical level, but an active enhancement path that contains "preference expression". It allows the neural network to not only "remember" certain key signals in the process of information transmission, but also "willing to emphasize" certain information that is considered more important. Compared with the unbiased weighting strategy of traditional residual connection, this mechanism has stronger controllability and expressiveness, and can better simulate the value judgment behavior of humans in the cognitive process.

In summary, transferring the idea of the monotonic mapping relationship between preference and utility in Debreu's theorem to the neural network structure can inject new theoretical significance into the design of residual connections. This mechanism with "utility monotonicity" as the core is not only intuitively explanatory, but also may improve the generalization ability and robustness of the model in complex tasks, thereby providing theoretical and practical support for the structural innovation of neural networks.

3. Model Structure

In order to evaluate the effectiveness of introducing Debreu's utility enhancement idea into the residual connection mechanism, this paper constructs and compares two neural network models, both of which are implemented based on the PyTorch framework, and conducts control experiments under the same data set and training strategy to ensure that the structural changes are the only variables to verify their impact on performance.

The first type of model is a traditional multi-layer perceptron (MLP) with a three-layer fully connected structure. The input features are first flattened, and then pass through two hidden layers in sequence. Each hidden layer is followed by a ReLU activation function to introduce nonlinear capabilities, and the final output layer is used for classification tasks. In this design, the model only relies on linear weights and activation functions to complete the layer-by-layer processing of information, without any jump connections or auxiliary structures. It is a typical deep feedforward network.

The second type of model is the utility residual MLP proposed in this study. Its overall architecture is consistent with the standard MLP, but an auxiliary path called "utility module" is introduced after each hidden layer. This module consists of a set of independent fully connected layers and nonlinear activation functions (such as Sigmoid). Its input is the current intermediate representation of the main channel, and its output is a set of transformed "utility response values". These response values are regarded as the importance weights or preference strengths of the output of this layer, and are added to the original output of the main channel to form an update mechanism similar to the residual structure. Through this operation, the network not only transmits the original representation, but also sends its "preference enhancement results" to the next layer, forming a signal update method with semantic directionality.

The core feature of this structure is that it not only retains the numerical stability advantage of traditional residual connections, but also introduces a learning path that conforms to the semantics of preference enhancement, so that the network can make responsive adjustments to the value of intermediate features. Especially in multi-layer nonlinear structures, standard residual connections often exist as a "blind retention" mechanism, that is, the output of the previous layer is directly passed as is. After the introduction of the utility module, the retention and enhancement of information becomes more selective, and the network can dynamically decide which signals are worth strengthening and which should be suppressed during parameter learning, thereby improving the overall expressiveness.

It is worth emphasizing that although this study is prototyped in the MLP architecture, the utility residual mechanism is not limited to fully connected networks. In fact, its design concept is highly versatile and scalable. The structures of the main branch and the utility branch can be flexibly replaced according to task requirements. For example, in a convolutional neural network, the main branch can be composed of a standard convolutional layer, while the utility module is implemented by a set of lightweight convolutions or depth-separable convolutions; in a graph neural network, the main branch can be a graph aggregation operation, while the utility module can learn the importance enhancement strategy of nodes or edges; in the Transformer architecture, this mechanism can also be used as a residual replacement path for multi-head attention, where the utility module can build dynamic weights based on attention scoring information.

In addition, since the utility module itself has the characteristics of nonlinear enhancement, it can also be combined with attention mechanisms, gating mechanisms, etc. to further develop into a more complex structure. For example, in sequence modeling tasks, the utility branch can weight the time dimension to highlight the information of key time points; in multimodal fusion, information of different modalities can also achieve preference mapping and fusion enhancement through independent utility paths. The universality of this structure gives it good migration potential and can play a role in various task scenarios such as images, texts, and voices.

In summary, the utility residual structure is not only a local improvement for residual connections, but also a general modeling mechanism that introduces "semantic enhancement". By adding a learnable utility mapping branch outside the trunk channel, the network can perform preference screening and dynamic reinforcement of intermediate features, thereby improving the model's expression depth, generalization ability, and flexibility in information selection. The proposal of this idea has opened up a new research direction for the design of connection mechanisms in deep learning, and also provided a theoretical basis and practical possibility for unified modeling across structures.

4. Experimental Setup and Results

We conducted comparative experiments on the standard MLP and utility residual MLP on the CIFAR-10 dataset, running each for 10 times to obtain a stable average. The experiments mainly observed classification accuracy, precision, recall, F1 score, and inference time.

The complete python code used for the experiment is as follows:

• import torch
  import torch.nn as nn
  import torch.nn.functional as F
  import torchvision
  import torchvision.transforms as transforms
  from torch.utils.data import Subset, DataLoader
  from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
  import time
  import numpy as np
  import random
  # Ensure reproducibility
  seed = 42
  torch.manual_seed(seed)
  np.random.seed(seed)
  random.seed(seed)
  # ---------------------
  # 1. Dataset (CIFAR-10)
  # ---------------------
  transform = transforms.Compose([
      transforms.ToTensor(),
      transforms.Normalize((0.5,), (0.5,))
  ])
  train_dataset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
  test_dataset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
• # Use only 5000 samples each
  train_subset = Subset(train_dataset, range(5000))
  test_subset = Subset(test_dataset, range(5000))
• train_loader = DataLoader(train_subset, batch_size=32, shuffle=True)
  test_loader = DataLoader(test_subset, batch_size=32, shuffle=False)
• # -------------------------
  # 2. Standard MLP Model
  # -------------------------
  class MLP(nn.Module):
    def __init__(self):
      super(MLP, self).__init__()
      self.flatten = nn.Flatten()
      self.fc1 = nn.Linear(32 * 32 * 3, 512)
      self.fc2 = nn.Linear(512, 256)
      self.fc3 = nn.Linear(256, 10)
•   def forward(self, x):
      x = self.flatten(x)
      x = F.relu(self.fc1(x))
      x = F.relu(self.fc2(x))
      x = self.fc3(x)
      return x
• # ---------------------------------------------------------
  # 3. Residual MLP with Debreu's Inspired Utility Connection
  # ---------------------------------------------------------
  class UtilityResidualMLP(nn.Module):
    def __init__(self):
      super(UtilityResidualMLP, self).__init__()
      self.flatten = nn.Flatten()
      self.fc1 = nn.Linear(32 * 32 * 3, 512)
      self.utility1 = nn.Sequential(
         nn.Linear(512, 512),
         nn.Sigmoid()
       )
      self.fc2 = nn.Linear(512, 256)
      self.utility2 = nn.Sequential(
         nn.Linear(256, 256),
         nn.Sigmoid()
       )
       self.fc3 = nn.Linear(256, 10)
•    def forward(self, x):
       x = self.flatten(x)
       out1 = F.relu(self.fc1(x))
       util1 = self.utility1(out1)
       res1 = out1 + util1
•      out2 = F.relu(self.fc2(res1))
       util2 = self.utility2(out2)
       res2 = out2 + util2
•      x = self.fc3(res2)
       return x
• # ---------------------
  # 4. Training Function
  # ---------------------
  def train(model, loader, criterion, optimizer):
     model.train()
     for images, labels in loader:
       optimizer.zero_grad()
       outputs = model(images)
       loss = criterion(outputs, labels)
       loss.backward()
       optimizer.step()
• # ---------------------
  # 5. Evaluation Function
  # ---------------------
  def evaluate(model, loader):
     model.eval()
     all_preds = []
     all_labels = []
     start_time = time.time()
     with torch.no_grad():
       for images, labels in loader:
         outputs = model(images)
         _, predicted = torch.max(outputs.data, 1)
         all_preds.extend(predicted.cpu().numpy())
         all_labels.extend(labels.cpu().numpy())
     end_time = time.time()
     return all_labels, all_preds, end_time - start_time
• # ---------------------
  # 6. Metrics Function
  # ---------------------
  def calculate_metrics(true, pred):
     return {
       "accuracy": accuracy_score(true, pred),
       "precision": precision_score(true, pred, average='macro', zero_division=0),
       "recall": recall_score(true, pred, average='macro', zero_division=0),
       "f1": f1_score(true, pred, average='macro', zero_division=0)
     }
• # ---------------------
  # 7. Run Experiment 10 Times per Model
  # ---------------------
  def run_experiments(model_class, model_name):
     all_results = {
       "accuracy": [],
       "precision": [],
       "recall": [],
       "f1": [],
       "time": []
     }
     print(f"\n==== Running 10 trials for {model_name} ====\n")
     for run in range(10):
       model = model_class()
       criterion = nn.CrossEntropyLoss()
       optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
•      for epoch in range(5):
         train(model, train_loader, criterion, optimizer)
•      true_labels, predicted_labels, elapsed = evaluate(model, test_loader)
       metrics = calculate_metrics(true_labels, predicted_labels)
•      all_results["accuracy"].append(metrics["accuracy"])
       all_results["precision"].append(metrics["precision"])
       all_results["recall"].append(metrics["recall"])
       all_results["f1"].append(metrics["f1"])
       all_results["time"].append(elapsed)
•      print(f"Run {run + 1}: Acc={metrics['accuracy']:.4f}, Prec={metrics['precision']:.4f}, "
         f"Recall={metrics['recall']:.4f}, F1={metrics['f1']:.4f}, Time={elapsed:.2f}s")
•    print(f"\n---- {model_name} Summary ----")
     for key in all_results:
       values = all_results[key]
       mean = np.mean(values)
       std = np.std(values)
       print(f"{key.capitalize()} - Mean: {mean:.4f}, Std: {std:.4f}")
     print("\n")
• # ---------------------
  # 8. Run Both Models
  # ---------------------
  run_experiments(MLP, "Standard MLP")
  run_experiments(UtilityResidualMLP, "Utility Residual MLP")
  The output of the experimental code is as follows:
  ---- Standard MLP Summary ----
  Accuracy - Mean: 0.4197, Std: 0.0094
  Precision - Mean: 0.4258, Std: 0.0074
  Recall - Mean: 0.4198, Std: 0.0096
  F1 - Mean: 0.4097, Std: 0.0122
  Time - Mean: 1.9911, Std: 1.6200
  ---- Utility Residual MLP Summary ----
  Accuracy - Mean: 0.4198, Std: 0.0068
  Precision - Mean: 0.4342, Std: 0.0041
  Recall - Mean: 0.4200, Std: 0.0068
  F1 - Mean: 0.4143, Std: 0.0089
  Time - Mean: 1.8464, Std: 0.7113

The experimental results show that the two models are basically the same in terms of accuracy, and the utility residual model is slightly better in precision and F1 score, and the inference time is slightly shorter, indicating that it has certain advantages without significantly increasing the computational burden.

More importantly, the core idea of this structure provides the network with a "preference enhancement-based explanation path", not just a technical optimization.

5. Versatility and Scalability

The "utility residual connection" structure proposed in this paper is not just a partial improvement of the traditional multi-layer perceptron (MLP), but a connection mechanism with high versatility and inter-structure adaptability. Its core idea is to introduce a utility module that simulates the "preference reinforcement" process in addition to the main information transmission path. This enables the neural network to not only rely on the original features in information processing, but also to induce and amplify the intermediate representations through learnable preference responses, thereby improving the model's responsiveness to useful information.

This mechanism can be widely applied to the entire network architecture. Specifically, the utility residual connection can be embedded in the original model in various forms and applied to images, text, graph structure data, and even multimodal tasks. For example, in a convolutional neural network (CNN), a small convolution path with an activation function (such as a sigmoid function or a Swish function) can be introduced after the output of each convolutional layer to form an auxiliary structure similar to a preference channel. This allows the importance of the current feature map to be increased pixel by pixel or channel by channel. Since CNN itself is more sensitive to local area features, the utility residual structure can more effectively distinguish important features from background noise and improve the recognition ability of the target area.

In a graph neural network (GNN), the representation of a node usually depends on the aggregation process of neighborhood information, which leads to obvious preference differences between nodes and graph structures. Therefore, introducing a graph-based utility mapping module after the graph aggregation process can amplify the contribution of important neighborhood nodes and suppress the interference of irrelevant information. The utility residual mechanism not only improves representation accuracy by guiding the network to focus on paths with high semantic value in the structure, but also provides more flexible adjustment capabilities for graph modeling.

The Transformer architecture has become a mainstream model in natural language processing and vision due to its extensive attention mechanism. Although traditional residual connections are widely used, these connections themselves do not have the ability to distinguish semantics. By embedding a set of auxiliary modules after each attention layer, the utility residual structure can capture which attention outputs are more preferred and adjust the attention distribution in a preference-driven manner. This extension not only improves the modeling effect, but also provides a new explanation path for understanding the selection bias of the attention mechanism.

In addition, in complex systems such as meta-learning and multi-task learning, different tasks and learning stages often have different representation requirements and preference tendencies. By introducing independent utility modules for each task, the network can make targeted preference adjustments for the feature representations required for specific tasks, and realize dynamic information selection and sharing between tasks. This mechanism provides strong support for building more general and adaptable models.

Another important value of the utility residual connection lies in its theoretical explanatory power and potential interpretability. In deep learning, the black box nature of the model decision process has been a major challenge for many years. The utility module has clear monotonicity and preference expression functions. The weight distribution learned during the learning process can be regarded as an explicit expression of the model's importance to information. By analyzing the changing trends of these weights, we can intuitively explain the model's focus and understand its selection preferences and risk sensitivity in various samples and tasks.

In summary, whether from the perspective of model structure design, task adaptability or model interpretability, the utility residual connection shows good scalability and theoretical basis. It is not only a local architectural change, but also a universal connection mechanism with cognitive motivation and applicability. It is suitable for integration into various deep neural network frameworks to promote the coordinated improvement of model performance and interpretability.

6. Conclusion

Based on the Debreu theorem in economics, this paper proposes a new residual connection mechanism-"utility residual connection". The core idea is to apply the concept of "preference monotonicity" to the structural design of neural networks, thereby improving the selectivity and reinforcement of the model in the process of information flow. Specifically, by introducing a monotonic utility function branch in the conventional residual path, the model adjusts the preference based on the original representation, realizing the dual processing method of "understanding + reinforcement".

The effectiveness of the structure was verified by preliminary experiments using a standard multi-layer perceptron framework. While maintaining basic learning efficiency, the utility residual model achieved a stable and small improvement in multiple performance indicators, especially in accuracy and F1 value. This shows that more effective information modeling can be achieved by introducing a preference reinforcement mechanism in a simple architecture.

More importantly, the mechanism proposed in this paper is not limited to a specific model architecture, but is extremely versatile and pluggable. A utility residual structure that can be flexibly adjusted according to scenarios, such as image recognition, graph modeling, text understanding, and cross-modal tasks, is a key factor in improving model performance and interpretability. In particular, the application value of this structure is particularly important in scenarios where it is necessary to select and amplify intermediate information representations.

From a theoretical perspective, this paper integrates the economic basis of connection mechanisms and utility functions in deep learning, expands the ideas of model structure design, and provides a new perspective for the interpretability of deep neural networks. It is expected that this cross-domain thinking model can further promote the study of the relationship between model structure and human cognitive mechanism, and promote deep learning in a direction that is easier for humans to understand.

Future research may make further progress from multiple perspectives. On the one hand, exploring more complex utility function forms, such as parameterizable preference modeling and nonlinear learnable reinforcement curves, can help express the importance of information more accurately. On the other hand, by combining this mechanism with existing attention mechanisms and gating mechanisms, a multi-level preference expression structure can be constructed to improve the modeling ability of complex semantic relationships. In addition, it can also be widely tested in practical tasks such as recommendation systems, sentiment computing, and medical imaging diagnosis to verify its effectiveness and advantages in real environments.

In other words, the utility residual connection provides a structural design scheme for deep neural networks that has both theoretical support and engineering practicality, and is expected to play a more important role in the modeling and interpretation of neural networks in the future.