Domain Adaptation with Sentiment Domain Adapter

1. Introduction

Sentiment analysis [1,2], also known as opinion mining, is a subfield of natural language processing (NLP) that involves identifying and categorizing opinions expressed in a piece of text to determine the writer’s attitude towards a particular topic, product, or service. This process typically classifies sentiments into categories such as positive, negative, or neutral, but can also extend to more nuanced emotions like joy, anger, or disappointment [3]. The utility of sentiment analysis is vast, ranging from businesses assessing customer reviews and feedback to gauge public opinion, to social media platforms monitoring user content to understand prevailing attitudes and trends. [4,5] The challenge lies in the subtleties of human language, including sarcasm, irony, and context-dependent meanings, which can skew straightforward computational interpretations.

Technologically, sentiment analysis involves various computational techniques, from simple rule-based algorithms that scan for positive or negative keywords to sophisticated machine learning models that leverage large datasets to understand context and linguistic nuances. With the advent of deep learning, models such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) have become particularly effective, as they can process textual data in sequence and capture temporal dependencies and contextual clues that are essential for accurate sentiment interpretation [3,11]. These models are often trained on vast corpora of labeled text data, where they learn to associate specific linguistic patterns with sentiment labels. As NLP continues to evolve, the integration of contextual embeddings and transformer models like BERT (Bidirectional Encoder Representations from Transformers) has further revolutionized sentiment analysis, offering even greater precision in capturing the complexities of human emotion expressed through text.

Domain adaptation is a critical technique in machine learning, aimed at addressing the problem of applying an algorithm trained in one setting (the source domain) to a different but related setting (the target domain). This is especially prevalent in fields like natural language processing, computer vision, and sentiment analysis, where data distribution can vary significantly across domains due to differences in language usage, image backgrounds, or contextual nuances [16,17]. The fundamental challenge in domain adaptation is to minimize the domain shift—whereby the model trained on the source domain underperforms in the target domain due to differences in feature distribution. Techniques like transfer learning, fine-tuning, and domain invariant feature extraction are commonly employed to mitigate this shift, improving the model’s ability to generalize across domains without the need for extensive labeling in each new domain.

Domain adaptation strategies benefit immensely from the abundance of labeled data in a source domain, enabling effective adaptation to similar, albeit unlabeled, data distributions in a target domain. In sentiment analysis, the articulation of emotions can vary significantly across domains [18]. For instance, delicious might convey positivity in the Food domain, while heartwarming might serve a similar purpose in the Movies domain. This variance often renders classifiers trained in one domain less effective in another.

In more advanced settings, domain adaptation strategies involve both theoretical and algorithmic innovations to create models that can automatically adjust to new environments. For instance, researchers have developed methods that align the statistical properties of data distributions between domains using techniques such as Maximum Mean Discrepancy (MMD) or adversarial training approaches [19,20]. These methods not only adjust the underlying distributions but also enhance the model’s interpretability by identifying which features are most relevant for both domains. Recent approaches have also explored the use of deep learning architectures, which can learn complex representations of data that are more adaptable to different domains. Such models often incorporate elements of feature disentanglement and attention mechanisms to focus on the most transferable aspects of the data, further refining the adaptation process for better accuracy and robustness in diverse applications.

Traditional domain adaptation approaches have emphasized the need to identify shared pivotal sentiment words and domain-specific non-pivotal words. Early methods like Structural Correspondence Learning (SCL) by Blitzer et al. [25,26] and Spectral Feature Alignment (SFA) by Pan et al. [27] have attempted to bridge these domains by aligning non-pivotal words with pivotal words. However, these methods treated the categories separately and lacked a unified approach.

With the advent of deep learning, new strategies have emerged to decrease the distributional shift in domain adaptation. Techniques like the Maximum Mean Discrepancy (MMD) and various adversarial training methods have become popular for their effectiveness in aligning domain characteristics [20,28,29,30,31,32,33,34,35]. Yet, these approaches often fall short in terms of interpretability, particularly in understanding the role of pivotal and non-pivotal sentiment words within the adaptation process.

To bridge this gap, we introduce the Sentiment Domain Adapter (SDA), a novel integration of CAN and CNN. This model views pivotal and non-pivotal words as a unified set of category attributes. SDA includes a Category Memory Module (CMM), a Dynamic Matching (DM) process, and a Category Attention (CA) layer within its architecture. The CMM stores a predefined set of category attributes, which are dynamically matched to each sample through the DM process. The CA layer then focuses on these attributes within the sample, enhancing the model’s attention to relevant features for domain adaptation [28,34]. SDA is applied to both the source domain, where CMM is specifically tailored, and the target domain, where CMM starts with a random initialization, offering insights into the transferability of domain knowledge.

Through extensive optimization of the model’s objectives, SDA not only focuses on relevant category features but also disregards non-pertinent ones, thus boosting performance in the target domain. The domain-aware CMM and the CA within SDA provide valuable interpretative insights into the process of domain adaptation. Our comprehensive evaluations across multiple real-world datasets reveal that SDA outperforms other models, with further analyses confirming that the domain-aware CMM effectively facilitates knowledge transfer from the source to the target domain, offering a robust interpretation of adaptable domain features.

2. Related Work

The exploration of domain adaptation techniques often revolves around the challenge of distinguishing between pivotal and non-pivotal elements within datasets. One seminal approach, the Structural Correspondence Learning (SCL) method, introduced by Blitzer et al. [25,26], employs a strategy to induce a shared low-dimensional feature space across domains, capitalizing on the co-occurrence of pivotal and non-pivotal elements. This method relies on multiple pivot prediction tasks to uncover the underlying connections between these elements, setting a foundational framework for further studies in this field.

Another notable technique, the Spectral Feature Alignment (SFA) method [27], seeks to cluster non-pivotal features from varied domains into unified groups using pivots as a cohesive link. This alignment facilitates a more seamless domain adaptation by bridging gaps between distinct data characteristics. Additionally, the Adversarial Memory Network (AMN) [42] introduces a dynamic mechanism to identify and leverage non-pivotal features, enhancing the model’s adaptability across domains with minimal pivot overlap.

Further advancements have been made with the Hierarchical Attention Transfer Network (HATN) [18], which innovatively captures both pivotal and non-pivotal features without segregating them into separate networks. This model utilizes a dual-network system, P-net and NP-net, each dedicated to handling specific types of features, yet it strives for a unified approach in feature processing, marking a significant step towards integrated domain adaptation models.

In the realm of deep learning, substantial progress has been made towards automating the extraction of robust feature representations for cross-domain applications, particularly in sentiment classification. Early work by Glorot et al. employed a Stacked Denoising Autoencoder (SDA) to derive meaningful, unsupervised feature representations, subsequently applying these features to train a specialized classifier. This approach paved the way for the integration of advanced techniques such as the Maximum Mean Discrepancy (MMD) measure, which has been widely adopted as a regularization strategy to minimize distribution mismatches across domains [28,29,30,50].

Long et al. enhanced this methodology by incorporating a multiple kernel variant of MMD (MK-MMD), originally proposed by Gretton et al. , into Convolutional Neural Networks (CNNs). This integration significantly improved dataset bias reduction and boosted the transfer capabilities within task-specific layers of CNNs. Furthermore, Dong and de Melo introduced a novel approach for inducing sentiment embeddings through supervision on out-of-domain data, integrating these embeddings into the model via a dedicated memory-based component to further refine the adaptation process.

The recent surge in popularity of adversarial training methods and Generative Adversarial Networks (GANs) has opened new avenues for domain adaptation. Key methodologies in this area include Domain Adaptation with Adversarial Training (DAAT) [34], Domain-Adversarial Neural Network (DANN) [31], Domain Separation Networks (DSN) [32], Selective Adversarial Networks (SAN) [33], among others [20,35,58]. These approaches, while effective, often grapple with issues of interpretability, as the direct learning of transferred category attribute words remains elusive. The lack of clear interpretability can pose significant challenges in user comprehension and trust, particularly when deploying these models in real-world applications, where understanding the basis of model decisions is crucial.

3. Methodology

In this section, we present the conceptual framework and the computational details of our proposed model, the Sentiment Domain Adapter (SDA).

3.1. Problem Formulation

Consider the problem of domain adaptation between a source domain ${\mathcal{D}}_s$ and a target domain ${\mathcal{D}}_t$, both concerned with a binary classification task. We have a set ${\mathit{X}}_s=\{({\mathit{x}}_1^s,y_1^s),\dots ,({\mathit{x}}_{N_s}^s,y_{N_s}^s)\}$ of $N_s$ labeled samples from ${\mathcal{D}}_s$, where each sample ${\mathit{x}}_i^s\in {\mathbb{R}}^L$ corresponds to a feature vector of length L and $y_i^s\in \{1,2,\dots ,C\}$ represents the associated category label. The target domain ${\mathcal{D}}_t$ provides a dataset ${\mathit{X}}_t=\{{\mathit{x}}_1^t,\dots ,{\mathit{x}}_{N_t}^t\}$ of $N_t$ unlabeled samples, each also of length L. The objective is to leverage ${\mathit{X}}_s^{train}$, the labeled data from ${\mathcal{D}}_s$, along with ${\mathit{X}}_t^{train}$, the unlabeled data from ${\mathcal{D}}_t$, to train a model that performs effectively on the test data ${\mathit{X}}_t^{test}$ from the target domain.

3.2. Category Attention Network (CAN)

The Category Attention Network (CAN) is designed to emphasize significant categorical features, such as specific sentiment-bearing words in sentiment analysis. Words like excellent and terrible frequently indicate positive and negative sentiments, respectively, and are crucial for classification accuracy. The CAN comprises several components: a Category Memory Module (CMM), a Dynamic Matching (DM) process, and a Category Attention (CA) layer, each contributing uniquely to the model’s performance.

3.2.1. Category Memory Module (CMM)

The CMM is a repository of category-specific attribute words, extracted from labeled data in the source domain. These words are distinctly frequent in their respective categories but rare in others. For instance, we extract attribute words for the positive category by identifying words with a significantly higher frequency in positive contexts than in negative ones, as per the formula:

$$a_1^{pos}=arg\underset{i}{max}\left(log{\mathit{T}}_i^{pos}-log{\mathit{T}}_i^{neg}\right),$$

(1)

where ${\mathit{T}}_i^{pos}$ and ${\mathit{T}}_i^{neg}$ are the occurrences of the i-th word in positive and negative samples, respectively. This process is mirrored to identify the most significant negative attribute words. The CMM thus comprises the top-M attribute words for each category.

3.2.2. Dynamic Matching (DM)

The DM process dynamically matches category attribute words from the CMM to each sample during training, using cosine similarity to identify the most relevant words for each instance. This is formulated as:

$$\begin{array}{cc}\hfill s_m^c& =\underset{l\in \{1,\cdots ,L\}}{max}cos({\mathit{w}}_l,{\mathit{a}}_m^c),\end{array}$$

(2)

$$\begin{array}{cc}\hfill q_1^c& =arg\underset{m\in \{1,\cdots ,M\}}{max}{s}_m^c,\end{array}$$

(3)

where ${\mathit{w}}_l$ is the embedding vector of the l-th word in a sample, and ${\mathit{a}}_m^c$ is the embedding of the m-th attribute word for category c. The top-K attribute words are selected for each category, providing a focused subset for further analysis.

3.2.3. Category Attention (CA)

The CA mechanism applies attention to the dynamically matched words, enhancing the model’s focus on relevant category-specific features. It calculates attention weights for each word in a sentence relative to the matched attribute words, significantly highlighting the most indicative features for classification. This is described by:

$$\begin{array}{c}\hfill s_{k,l}^c=\frac{exp\left({\mathit{q}}_k^{c\top }tanh({\mathit{W}}_a{\mathit{w}}_l+\mathit{b})\right)}{{\sum }_{l=1}^{L}exp\left({\mathit{q}}_k^{c\top }tanh({\mathit{W}}_a{\mathit{w}}_l+\mathit{b})\right)},\end{array}$$

(4)

where ${\mathit{W}}_a$ and $\mathit{b}$ are the weight and bias parameters of the attention mechanism, respectively. This focused attention helps in isolating the most discriminative features within the input data.

3.3. Integration of CAN and CNN for Domain Adaptation

The integration of the Category Attention Network (CAN) with a Convolutional Neural Network (CNN) forms the core of our domain adaptation approach. The CNN, following the architecture of TextCNN [59], extracts broad contextual features from the text, while the CAN focuses on category-specific attributes. The combined features from both networks are then used to predict the final category labels.

The optimization of our model involves a composite loss function that addresses classification accuracy, domain adaptation, and category-specific feature alignment. This includes a supervised classification loss ${ }_C\left(\theta \right)$ for labeled data in the source domain, a domain adaptation loss ${ }_I\left(\theta \right)$ using Maximum Mean Discrepancy (MMD) to minimize the difference between source and target feature distributions, and a distribution loss ${ }_D\left(\theta \right)$ that aligns the attention weights of category attribute words across domains. Each component is essential for ensuring that the model not only performs well on the source domain data but also adapts effectively to the target domain. It is worth noting that our approach is a general framework and the optimization objective acts on the CAN and the features extractor CNN. So the CNN can be replaced by any other efficient feature extractors (e.g., LSTM [60], Transformer [61]).

Table 1. Top five category-specific attribute words identified by CMM in three sentiment analysis domains: Consumer Reviews (CR), Amazon Fine Foods (AFF), and Movie Reviews (MR).

CR		AFF		MR
Positive	Negative	Positive	Negative	Positive	Negative
excellent	poor	tasty	awful	captivating	dull
satisfied	problematic	yummy	bad	compelling	pointless
best	disappointing	flavorful	disappointing	fascinating	lackluster
superb	worst	scrumptious	unappealing	masterpiece	bland
perfect	terrible	mouthwatering	horrible	inspiring	dreary

Table 1. Top five category-specific attribute words identified by CMM in three sentiment analysis domains: Consumer Reviews (CR), Amazon Fine Foods (AFF), and Movie Reviews (MR).

CR		AFF		MR
Positive	Negative	Positive	Negative	Positive	Negative
excellent	poor	tasty	awful	captivating	dull
satisfied	problematic	yummy	bad	compelling	pointless
best	disappointing	flavorful	disappointing	fascinating	lackluster
superb	worst	scrumptious	unappealing	masterpiece	bland
perfect	terrible	mouthwatering	horrible	inspiring	dreary

Table 2. Comparativeaccuracy performance of the proposed SDA model against various baseline models, utilizing 10-fold cross-validation across different domain adaptation scenarios.

	Model	MR→CR	AFF→CR	CR→AFF	MR→AFF	CR→MR	AFF→MR
Direct Transfer	fastText-random	0.6290	0.6720	0.6790	0.6900	0.5750	0.5850
	fastText-finetuned	0.6680	0.7470	0.7240	0.7480	0.6550	0.6890
	CNN-char	0.5600	0.6670	0.7140	0.6620	0.5610	0.5930
	CNN-random	0.6070	0.6990	0.7130	0.6750	0.5900	0.6010
	CNN-finetuned	0.6900	0.7580	0.7520	0.7630	0.6680	0.6920
Domain Adaptation	SDA	0.6080	0.6650	0.6750	0.6930	0.6250	0.6350
	mSDA	0.5960	0.6430	0.6810	0.7060	0.6210	0.6390
	SDA-fine-tuned	0.6230	0.6940	0.6900	0.7150	0.6310	0.6430
	DAAT	0.6990	0.7310	0.7220	0.7440	0.6240	0.6530
	SDA (shared CMM)	0.7150	0.7500	0.7660	0.7810	0.6550	0.6970
	SDA	0.7320	0.7650	0.7890	0.7930	0.6800	0.7100

Table 2. Comparativeaccuracy performance of the proposed SDA model against various baseline models, utilizing 10-fold cross-validation across different domain adaptation scenarios.

	Model	MR→CR	AFF→CR	CR→AFF	MR→AFF	CR→MR	AFF→MR
Direct Transfer	fastText-random	0.6290	0.6720	0.6790	0.6900	0.5750	0.5850
	fastText-finetuned	0.6680	0.7470	0.7240	0.7480	0.6550	0.6890
	CNN-char	0.5600	0.6670	0.7140	0.6620	0.5610	0.5930
	CNN-random	0.6070	0.6990	0.7130	0.6750	0.5900	0.6010
	CNN-finetuned	0.6900	0.7580	0.7520	0.7630	0.6680	0.6920
Domain Adaptation	SDA	0.6080	0.6650	0.6750	0.6930	0.6250	0.6350
	mSDA	0.5960	0.6430	0.6810	0.7060	0.6210	0.6390
	SDA-fine-tuned	0.6230	0.6940	0.6900	0.7150	0.6310	0.6430
	DAAT	0.6990	0.7310	0.7220	0.7440	0.6240	0.6530
	SDA (shared CMM)	0.7150	0.7500	0.7660	0.7810	0.6550	0.6970
	SDA	0.7320	0.7650	0.7890	0.7930	0.6800	0.7100

4. Experiments

We assess the efficacy of the SDA model against several benchmarks within the scope of three distinct sentiment analysis datasets. The experimental setup includes diverse datasets and multiple domain adaptation scenarios, reflecting real-world challenges in sentiment analysis.

4.1. Datasets

The datasets employed in our study are: 1) CR: Customer Review dataset from Amazon, covering various products. 2) AFF: Amazon Fine Foods Review dataset, a subset of which was randomly selected. 3) MR: Movie Review dataset from Cornell University, containing diverse film critiques.

These datasets lack a predefined train/test split; therefore, we implement a 10-fold cross-validation approach. This method ensures robustness and generalizability of our findings. The chosen datasets, rich in domain-specific nuances, are ideal for evaluating the adaptability of the SDA model.

4.2. Implementation Details

We construct six domain adaptation tasks from the aforementioned datasets, forming combinations such as MR→CR, AFF→CR, and so on. To ensure a fair comparison across all experiments, we standardize model parameters such as filter widths, feature maps, and embedding dimensions based on the established TextCNN configurations. Hyperparameters such as batch size and learning rate are also unified across all setups.

The CAN component utilizes a configuration of 50 attribute words per category, dynamically selecting 5 for matching in real-time processing. The hyperparameters $\alpha$ and $\beta$, determining the strength of the loss function components, are optimized through grid search on the validation set of the CR dataset.

4.3. Baseline Models

Our benchmarks include direct transfer models such as fastText and CNN, each with variations in word vector initialization (random vs. fine-tuned). We also consider domain adaptation models like SDA, mSDA, and adversarial training approaches like DAAT, comparing their performance without needing to explicitly handle pivots and non-pivots separately.

4.4. Performance Comparison

We present our findings in Table 2, where the SDA model generally outperforms all baselines across different adaptation scenarios, highlighting its robustness and effectiveness in leveraging domain-specific knowledge. The results illustrate that pre-trained embeddings and fine-tuning significantly contribute to performance improvements in domain adaptation tasks. Further, the shared CMM approach underlines the importance of domain-aware adaptation, confirming that direct application of source domain knowledge to target domains without adjustments tends to reduce performance.

Additionally, to dissect the impact of individual components of the SDA model, we conduct an ablation study, presented in Table 3. This study confirms that each component of the loss function contributes meaningfully to the overall effectiveness of the model, with the full configuration achieving the best results.

4.5. Interpretability Analysis

As mentioned before, the Category Memory Module (CMM) in our Sentiment Domain Adapter (SDA) model utilizes labeled data to distill category-defining attribute words, which poses a limitation when dealing with unlabeled target domain data. Notably, as seen in Table 1, distinct domains manifest unique sets of category attribute words for identical categories (e.g., the words "delicious" and "fast" for positive sentiments in AFF differ from "heartwarming" and "vividly" in MR). Therefore, the transfer of a CMM configured for one domain directly to another, as attempted in the CAN-CNN-shared model, results in suboptimal performance, as evidenced by the comparative outcomes in Table 2.

In the SDA model, the target domain’s CMM is initialized randomly and undergoes adaptive refinement during training, aligning with domain-specific semantic contexts. Table 4 showcases the evolution of vocabulary words closely aligned with category attribute words in the target domain’s CMM from their initial random state to their post-training, contextually enriched state. Initially, the resemblance to random words is apparent, but post-training, a clear thematic alignment emerges, reflecting effective domain adaptation. This dynamic adaptation not only illustrates the transfer of contextual knowledge between domains but also underscores the model’s ability to internally recalibrate its interpretative focus, aligning it with emergent domain-specific semantics.

4.6. Case Study on Target Domain Sentiment Analysis

We further elucidate the interpretability of the SDA model through a detailed case study, focusing on the visualization of category attention weights in the target domain of Movie Reviews (MR). The weights are averaged from the top-K matched category attribute words, denoted as ${\mathit{s}}^c=\frac{1}{K}{\sum }_k{\mathit{s}}_k^c$, and depicted in Table 5. Here, the intensity of the color overlay on each word corresponds to its computed category attention weight, offering a visual representation of the model’s focus.

The analysis reveals nuanced insights into the model’s operational dynamics. For instance, in the positive example (1), the words "funny" and "heartbreaking" are highlighted as significant, aligning well with the sample’s positive sentiment. Conversely, in the negative example (1), the word "boring" receives substantial emphasis, accurately reflecting its negative sentiment impact. This visual analysis not only confirms the model’s effectiveness in identifying sentiment-critical words but also showcases its ability to dynamically adjust focus within different contextual frames, ensuring robust domain adaptation.

5. Conclusion and Future Work

In our study, we introduced the Sentiment Domain Adapter (SDA), an innovative model designed to enhance the efficiency of domain adaptation tasks while also providing a mechanism for interpretability. This was achieved through the integration of a Category Attention Network (CAN) with a conventional Convolutional Neural Network (CNN). Our approach simplifies the complex dynamics of learning in domain adaptation by treating both pivots and non-pivots as unified category attributes. This obviates the need for separate network designs for different types of words, streamlining the learning process.

The core innovation of our model lies in its ability to dynamically learn and adjust the category attribute words within the target domain. This functionality not only aids in the adaptive learning process but also enhances the model’s capacity to interpretatively determine the most salient features to transfer from the source to the target domain. The empirical validation of our model on three distinct sentiment analysis datasets—spanning across different domains—showcases significant improvements in performance when compared against a range of existing baseline models.

The findings underscore the SDA model’s effectiveness in handling domain discrepancies and its adeptness at learning domain-specific nuances without extensive manual feature engineering. The integration of category attention mechanisms has particularly proven beneficial in refining the feature representations to be more domain-adaptive.

Looking ahead, several avenues remain open for further enhancing the SDA model. Future work could explore the incorporation of more granular attention mechanisms that could fine-tune the interpretation capabilities of the model. Additionally, extending the model’s architecture to support multi-lingual datasets could vastly increase its applicability in global sentiment analysis tasks. Moreover, experimenting with different forms of neural network architectures, such as Transformers, may provide deeper insights and potentially yield improvements in both adaptability and accuracy.

Another promising direction would be to enhance the model’s ability to handle larger and more diverse datasets, possibly incorporating unsupervised or semi-supervised learning elements to reduce the dependency on labeled data. Finally, a deeper exploration into the interpretability aspect could involve developing visualization tools that allow users to see and understand how the model makes its predictions, thereby increasing trust and transparency in automated decision-making processes.

In conclusion, the SDA model represents a significant step forward in the domain adaptation field, particularly within the realm of sentiment analysis. Its ability to seamlessly adapt and interpret across domains holds great promise for real-world applications, where understanding and reacting to user sentiments across various platforms and demographics is crucial.