LexScore: A Semantic Approach to Scoring Domain Specific Sentiment Lexicons

The sentiment of a word varies based on its context of usage: the words used around it and the part-of-speech it is used as. This paper proposes a technique to suggest the sentiment of a word by combining its part-of-speech and the semantic similarities of its cooccurrences with both context-specific and pre-trained embeddings to achieve powerful and fast results. A study was conducted across domains and subdomains to measure variance of sentiment by switching domains and switching context within the same domain. Re-scoring a commonly used polarity lexicon showed that 10% of words changed scores while switching domains and 8% changed scores within domains while switching context. Part of Speech analysis on 65,353 commonly used sentiment lexicons showed that 81% of sentiment bearing (non-neutral) lexicons were of the tags NN (Common Noun), JJ (Adjective) or


Introduction
Assigning sentiment scores to domain and context-specific words for lexicon-based sentiment analysis requires immense domain knowledge. The word crazy would have a negative connotation in a scientific setup whereas it would be considered positive in the informal news domain (Figure 1). Understanding the overarching theme of the sentiment of crazy with respect to the domain is relatively easier; although the magnitude of the sentiment score to assign still remains a task to calculate owing to the many contexts of usage of the word within the same domain and most times, expensive and timeconsuming (Mohammad and Turney, 2010; Fast et al., 2016) since it may require domain experts to work together with annotators; which also allows room for bias based on domain-specific usage or demographic variation (Deng et al., 2014;Hovy, 2015;Yang and Eisenstien, 2015). Large scale web-derived lexicon generation techniques (Velikovich et al., 2010) do not cater well to variation in context of usage and may introduce bias which may further impact the study of opinion and research.
To address these requirements, the paper introduces : an algorithm that learns

LexScore: A Semantic Approach to Scoring Domain Specific Sentiment Lexicons
Shantanu Kumar inFeedo 4864/24, Jain Bhawan Daryaganj, Delhi 110002 shantanu@infeedo.com from domain-specific corpora and enhances preexisting sentiment lexicons. combines a widely used method of using co-occurrences with word embeddings and part of speech tags to generate sentiment scores for input words. The algorithm is designed to maintain baseline accuracies while building heavily on speed to cater to both researchers and industry implementations where infrastructure spends are limited.
The key contributions of this work are: 1. A novel sentiment scoring algorithm combining word embeddings from domain-specific corpora and part-ofspeech tags. 2. The re-scored Jockers (2015b) and Rinker's augmented  polarity lexicons (Rinker, 2015a) to cater to 51 contexts across 7 domains.
This work aims to collate multiple ways of scoring sentiment-bearing lexicons into one algorithm, maintaining speed and scalability. The re-scored lexicons along with code for the preprocessing and the algorithm will be made available in an R package.

Related Work
This work builds upon a wealth of previous opinion papers and research on inducing and assigning scores to sentiment lexicons. Often, the two categories being followed for this task are corpus-based (Hatzivassiloglou and McKeown, 1997;Wiebe, 2000;Turney and Littman, 2003;Kanayama and Nasukawa, 2006) and relying on co-occurrence (Igo and Riloff, 2009;Riloff and Shepherd, 1997;Turney and Littman, 2003). Recent work involves construction of lexical graphs and label propagation (Hamilton et al., 2016).
This work also takes into account the implicit consensus that dictionary-based approaches generate higher-quality lexicons due to their generation requiring domain experts and being cleaner (Hamilton et al., 2016)the aim is to combine both categories but would fall in the category of dictionary-based approaches.
There is work that shows different POS categories contribute to sentiment in varying degrees and uses Maximum Entropy Modelling to achieve results (Nicholls and Song, 2009). In the interest of speed and scalability of the algorithm, the paper introduces a vanilla filtering approach for words.
Existing methods of generating lexicons primarily cater to sentence-based sentiment analysis (i.e., they provide weighted sentiments only per sentence, e.g., the R-package sentimentr, Rinker, 2015b). This paper aims to draw out sentiment scores for each word and not just the ones that would normally be sentiment-bearing. For example, "Woah crap that car is amazing!" has a positive connotation but the sentiment bearing words here (crap, amazing) cancel each other out in effect, resulting in a neutral sentiment. The aim is to reassign a positive score to "crap" to capture the true sentiment of this word in the sentence.
This work is inspired by (Hamilton et al., 2016;Nicholls and Song, 2009;Qiu et al., 2009) but aims to achieve similar results using a novel approach that gives importance to speed and interpretability.

Polarity Assignment
The proposed algorithm first initializes pre-trained embeddings or builds domain-specific embeddings to have high quality semantic representations for words using a vector space model and then constructs a term-co-occurrence matrix from a tokenized corpus using a skip-gram model that calculates weights given by the formula where the is the distance of each word from the current word (text2vec.org -Selivanov and Wang, 2016;Pennington et al., 2014) with varying window sizes based on the size of the corpus and the accuracy/speed tradeoff. A preexisting sentiment lexicon is then used as input to re-create for the specific domain. The algorithm is created to address three key requirements: 1. Replicable: The algorithm takes inputs as preexisting lexicons that are re-scored according to the domain.

Resource-light:
Captures accurate polarity scores with small corpora.

Accuracy Speed Tradeoff:
Allows for manually controlling the tradeoff to address various use-cases. This work re-scores the Jockers (2015b) and Rinker's augmented  polarity lexicons for 51 contexts using pre-trained GloVe embeddings (Pennington et al., 2014).

Downweighing to fit TCM
The first step in this approach is to check if the input word has been used in the corpus. The termco-occurrence matrix created would be scanned by row and column to find its co-occurrence statistics. This approach introduces a downweighing value when the exact word isn't found but a word semantically similar exists in the matrix. The formula used (cosine similarity) is given by where A and B are vector representations of the input word and the word most similar to it semantically.

POS filtering over co-occurrence vector
Once the co-occurrence vectors are extracted in the previous step, the initial domain-specific corpus is checked with the pre-existing lexicon set to find the POS-tags of maximum occurring sentiment bearing words. The co-occurrence vector is then tagged using the R package tagger (Rinker, 2015c) that uses openNLP and the Penn Treebank (Marcus et al., 1994). The words whose tags match the previous ones with maximum occurrence are filtered. Once we have these words with their cooccurrence weights, we now find most semantically similar words from the embeddings and check for their polarity scores in the existing lexicon. We combine these pre-existing polarity scores with their co-occurrence weights and a multiplier, which we set at 1.5x of the previous multiplication for the words in the new set matching the previous co-occurrence vector. The final set of the new words that are sentimentbearing are now semantically compared to the input word. The formula for calculating the final weights for each word is given by (3) where is the cosine similarity between the input word and the current word. The last set of words is then averaged (downweighed zero averaging) (Rinker, 2015b) and multiplied with the initial downweighing value D, if it wasn't in the TCM; this is given by the formula which calculates the final sentiment score for the input word in the algorithm.

Conclusion
LexScore allows industry and academic researchers to re-score commonly used lexicons to fit their particular domain and context. The paper combines the use of domain-specific cooccurrences with word-embeddings to transform commonly used lexicons to domain-specific ones for opinion mining and research purposes. The method hopes to help further thoughts in the space of context-dependency of sentiment taking into account the speed and interpretability of algorithms to further be able to scale usage beyond researchers. While the lexicons re-scored by LexScore are not the best, they do provide a more structured way of understanding the sentiment of certain words in different contexts. In the future, this could be coupled with lexicon-inducing algorithms to further improve both scoring and inducing these domain-specific lexicons.