A Fuzzy Synthetic Evaluation Approach to Assess Usefulness of Reviews by considering Bias Inherited in Sentiments and Articulacy

1. Introduction

In today’s digital era, understanding the underlying feelings and potential biases within these reviews has become critical, when online reviews and user-generated material have significant impact over customer decisions. Sentiment analysis, also known as opinion mining, is the computational analysis of people's views, feelings, emotions, and attitudes about entities such as products, services, issues, events, ideas, and their attributes [1]. As opinions and sentiments are widely expressed on many platforms ranging from e-commerce websites to social media, the necessity for accurate sentiment analysis technologies that not only recognize emotions but also detect potential biases has become critical, thus improving reviews’ usefulness.

Analysing the pertinent literature, it can be inferred that sentiment analysis is a rich area for research across various fields, such as e-business, e-learning, marketing, social networking sites, customer feedback, political discourse, etc.

Firstly, in the area of e-business, it has been used to assess the sentiments of customers and receive their preference on either products or services [2,3,4,5,6]. Indeed, online commerce platforms can have availability of a large pool of data in the form of customer feedback or reviews that can serve as a valuable source of information towards building their promotion strategy. As such, companies apply sentiment analysis techniques on this data to gain important insights on customer satisfaction and take corresponding decisions to improve customer experience. A recent review on sentiment analysis in e-business attests that sentiment analysis has been widely used in e-commerce using mainly machine learning algorithms.

In the context of e-learning, sentiment analysis finds application in analysing students' feedback on learning objects, forum discussions, and course evaluations [7,8,9,10,11]. This process yields valuable insights into learners' emotions and sentiments, enabling the creation of personalized learning experiences. By understanding students' feelings, educators can deliver tailored learning materials, exercises, and collaborative opportunities. Moreover, sentiment analysis serves as a valuable tool for identifying areas of improvement in the e-learning environment, helping educators enhance the overall learning experience. A recent review work of 2023 [12] highlighted that sentiment analysis has shown to be effective for educators as it helped them improve their teaching methodology and tailor the course content to students. Also, concerning learners, sentiment analysis has helped them advance their knowledge and has provided them with access to qualitative learning.

Furthermore, sentiment analysis has also been widely used in social networking sites [13,14,15,16,17], since understanding the opinion of people holds significant implications. Researchers have developed domain-specific sentiment analysis techniques to tackle the unique challenges posed by noisy social media data. A review study of 2023 [18] attests that sentiment analysis can have great implications in this field while the techniques used are mainly neural networks and Support Vector Machines (SVMs).

Also, sentiment analysis finds application in political discourse analysis [19,20,21,22,23], where analysing public sentiment towards political candidates, policies, and events can inform campaign strategies and policy decisions. By analysing sentiments expressed in political tweets, news articles, and online forums, valuable insights into public perceptions can be gained.

As shown in the aforementioned studies, two prevalent approaches have been used for sentiment analysis. The first approach is the lexicon-based techniques. The lexicon-based methods assign positive, negative, or neutral sentiment scores to individual words and then aggregate the scores to determine the overall sentiment of a text. While simple and easy to implement, lexicon-based methods suffer from limited context awareness and may struggle with sarcasm, idioms, or language nuances.

Machine learning approaches in sentiment analysis that can perform better in situations where lexicon-based techniques may present obstacles. Researchers have resorted to machine learning techniques such as supervised learning and deep learning to categorize text sentiments automatically. Supervised learning entails training models on labelled datasets where each text is assigned a sentiment label such as positive, negative, or neutral. These models learn to recognize patterns in the data that can be applied to new situations with similar sentiments. Among the popular supervised learning algorithms used in sentiment analysis are Support Vector Machines, Naive Bayes (NB), and Random Forests. These algorithms are adept at classifying sentiments based on the patterns they learn from the labelled training data.

Deep learning approaches, including Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs), have shown important effectiveness in sentiment analysis applications. RNNs perform well for sequential data and can capture long-term dependencies in text. CNNs are well-suited in learning local patterns and features from text. Moreover, the advent of pre-trained language models, such as BERT and GPT, has significantly advanced sentiment analysis, as these models can have great performance on specific sentiment analysis tasks.

In conclusion, a recent review by [24]highlights the extensive application of sentiment analysis, with social networks being the predominant field of use. The techniques predominantly employed in sentiment analysis involve traditional machine learning approaches, particularly Support Vector Machines and Naive Bayes.

Moreover, fuzzy logic, with its ability to model imprecise and uncertain information, provides a robust framework for identifying the sentiment of reviews and evaluating the potential bias that might be present in the expressions.

Although bias in sentiment analysis has attracted the attention of many researchers [25,26,27,28,29,30,31], sentiment analysis systems confront several challenges [32]. The effectiveness of sentiment analysis methods depends on the bias embedded in documents, such as gender or nationality bias as well as on how well the method addresses the so called domain adaptation problem [33].

Research studies suggest there is an urgent need to develop sentiment analysis techniques that can identify and quantify bias [29,32,34]. Bias in reviews can be attributed among other, to personal characteristics such as gender or cultural differences [27,29,35,36]. However, a few studies focus on understanding the role of cultural differences in user content generation [35,36,37]. This study proposes a FSE approach to calculate reviews’ usefulness by incorporating bias which is embedded in reviews of different nationalities users. This study considers reviews’ document and title sentiment, and articulacy as determinants of usefulness. Although fuzzy logic has been used in sentiment analysis, there is little-to-no research that utilises fuzzy logic to model and analyse usefulness and bias in sentiments.

The remainder of this paper is organized as follows: 2. Literature review on reviews usefulness and reviewers’ bias; 3. Methodology; 4. Methods; 5. Results; 6. Limitations of the study and Future Research; and 7. Conclusions.

2. Reviews’ Usefulness and Reviewers’ Bias in Sentiment Analysis

A plethora of reviews have flooded the online platforms such as Tripadvisor, Booking, etc., since reviews have been recognised as an important source of information for customers [38,39]. As a result, when users need to focus on the most useful opinions, they encounter vast numbers of reviews that imply high search costs and information overload [40,41]. It is argued [42,43] that the adoption usefulness voting systems can benefit both consumers and businesses. Customers may find assistance to tackle the sheer numbers of reviews by focusing on the most appropriate reviews and businesses are expected to develop revenue streams. Thus, a major research question arises with respect to how consumers identify the useful reviews and how they perceive high-quality ones [38,39]. In the relevant literature the usefulness of reviews has been examined by two perspectives namely: the review and the reviewers’ related factors [38]. Review related factors include length of review, sentiment extremity, novelty, depth, rating, and information inconsistency [38,39,40,41,44,45,46,47,48,49,50]. Reviewer related factors include expertise, experience, identity, rank and reputation [46,51]. It is also argued that since users read lots of reviews, the usefulness of a review does not solely rely on the reviews’ characteristics, but also on the characteristics of the reviews that have been read previously by the user as well as on the products context factors such as product satisfaction, product popularity, intangibility, etc. [38]. However, research studies reach contradicting results [38]. Some studies argue that consumers prefer reviews with depth, i.e. more words in a review [42,52]. Other studies argue that there is no significant relationship between review depth and usefulness [46,53]. It is argued that if a review’s length exceeds a certain threshold, then its readability diminishes as it would require consumer to spend more time to read it [53]. However, is such a threshold the same for all reviewers? In the same vein, studies indicate that reviews’ sentiment extremity, has a positive effect on usefulness [41,50], others report a negative effect [44,50], while other results show no effect at all [43,52,53]. The question that arises is if the same extremes are perceived similarly by all reviewers. Of ‘course, not all users express themselves or perceive reviews extremes the same way. Therefore, despite the undeniable value of reviews and sentiment analysis applications, biases in users’ reviews which are often overlooked may subsequently result to misleading and often contradicting judgements. Indeed, several research studies focused on investigating bias in sentiment analysis [30,31]. The association of certain words and expressions with males or females as an indicator of gender bias, is examined in [29]. Results indicate that women tend to use more direct language than males, to express either positive or negative sentiments. Research findings suggest that there are gender differences regarding the extent to which sentiments are expressed in user reviews [25,26,27,28]. Gender bias is also found in political writing [34]. Sentiment analysis results indicate that sentiment is less positive when a political article refers to a female figure than a male. Behavioural differences were also discussed in [54], where tweets were analysed and a method for gender identification was proposed.

Differences on sentiments can also be attributed to different nationalities. Users’ cultural background implies that they may have different priorities or they may seek information from different sources. Customers from Greece and Portugal prefer to rely on word of mouth rather than commercial marketing sources which are the choice of the customers from the UK and USA [36]. Users from the USA tend to appreciate reviews’ evaluations from their compatriots more than those from other countries [35]. Asian hotel visitors exhibit different complaint behaviour, since they seem to rather refrain themselves for expressing their complaints publicly compared to the non-Asian visitors [37]. A research study [35] indicates that guests from western countries usually express their feelings in a more positive and informative way than others. Visitors’ nationality is also identified as a discriminator factor since statistically significant differences were found in sentiments expressed by people from America, Europe, Middle East, and Asia [55].

3. Methodology

This study proposes methodology that utilises fuzzy logic in order to assess reviews usefulness by incorporating cultural differences and biases embedded in user reviews, thus increasing reviews’ readability by recommending the most appropriate reviews to users according to their preferences. This study approaches cultural differences in terms of reviews’ sentiments and articulacy since relevant research has identified both as determinants of reviews’ usefulness [47,49]. Thus, it aims to:

Analyse reviews and to investigate how lenient users of different nationalities are in their reviews when expressing their sentiments.

To assess any differences in the way users, express themselves, i.e. to contrast sentiments polarity and strength in their reviews’ title and full bodies.

• Examine how informative the users from different nationalities are, i.e., examine cultural differences in terms of reviews’ articulacy.
• Propose a fuzzy synthetic evaluation approach that allows users to identify the most useful reviews to read. Although this study considers the sentiment and articulacy determinants of reviews’ usefulness, the proposed approach allows users to specify their personalised perspective of usefulness by incorporating their individual biases. Figure 1 illustrates the steps of the proposed methodology.

Figure 1. The steps of the proposed methodology.

Figure 1. The steps of the proposed methodology.

This study collected hotel reviews from Tripadvisor over the years 2020-2022 using open source web crawler Python Scrapy. Data included the review’s text, its title, and the nationality of the reviewer. In total, 95.678 reviews were selected for 5 nationalities namely: British, US, Australian, Greek and Dutch. The sample consists of 42.678 reviews from British citizens (44,6%), 40.311 from the USA (42,1%), 9.293 from Australians (9,7%), 1.734 from Dutch (1,8%) and 1.662 from Greek citizens (1,6%). The sample is clearly biased, since the majority of the reviews is selected from reviewers whose mother tongue is meant to be English. KNIME visual programming software platform was used for data preparation and analysis. KNIME is an open-source platform that provides tools to manipulate and prepare data and machine learning algorithms for data analysis. A node is the fundamental unit which performs tasks as, delete stop words, create a bag of words, calculate TF-IDF scores, train a neural network, etc. Nodes are combined through drag-and-drop to develop a workflow in KNIME. KNIME has been used in several studies such as in [32,56]. The collected dataset was initially cleaned, anonymized and pre-processed in order to be imported in KNIME as a CSV file. Subsequently the documents were checked for misspelling errors, they were converted in lower cases, stop words were removed and reviews’ texts were tokenized.

This study uses MS Azure Machine learning to calculate the sentiment strength and polarity for both the review’s document and its title. The sentiment strength returns values in the interval [0,1]. Values close to 0 indicate negative sentiment while values close to 1 indicate positive sentiment. The polarity is quantified with a positive, neutral, or negative value. The MS Azure sentiment analysis service is used in [57] in order to calculate the sentiment of the mood that prevails in forum discussions regarding listed companies and include it in stock market data analysis. Jiang et al. (2022) [58], assessed the effectiveness of MS Azure sentiment analysis service in conjunction with other sentiment analysis tools with metaphoric testing. In another study [59], the MS Azure sentiment machine learning was used to calculate the sentiment and satisfaction expressed by patients regarding online doctor services.

4. Methods

This study represents sentiment and articulacy as triangular fuzzy number (TFN). TFNs are represented by triple (a, b, c). The membership function $f_A\left(x\right)$ of TFN $\tilde{A}$ can be calculated according to the following equation [60]

$$f_A\left(x\right)=\left\{\begin{array}{ccc}\frac{x-c}{a-c}& ,& c\le x\le a\\ & & \\ \frac{b-x}{b-a}& ,& a\le x\le b\\ & & \\ 0& ,& otherwise\end{array}\right.$$

(1)

where a, b, c are real numbers.

4.1. Fuzzy Relations

Fuzzy relations are important for they can describe the strength of interactions between variables [61,62]. Fuzzy relations, which are fuzzy sets, are fuzzy subsets of $X\times Y$, that is mapping from $X\to Y$. Let $X,Y\subseteq R$ be universal sets. Then

$$\tilde{R}=\{\left(\left(x,y\right),{\mu }_R\left(x,y\right)\right)|\left(x,y\right)\in X\times Y\}$$

(2)

is called a fuzzy relation on $X\times Y$ [62]. A fuzzy relation on a single universe $X$ is also a relation from $X$ to $X$. It is a fuzzy tolerance relation if the two following properties define it:

Reflexivity:${\mu }_R\left(x_i,x_i\right)=1$ and

Symmetry:${\mu }_R\left(x_i,x_j\right)={\mu }_R\left(x_j,x_i\right)$

The resulting fuzzy relation is the tolerance matrix, which indicates the similarity degrees between related concepts.

The “usefulness” of reviews is calculated by applying fuzzy relation composition. The composition is implemented by the Cartesian product of two fuzzy sets. Assume fuzzy set $\tilde{A}$ on universe $X$ and $\tilde{B}$ on universe ${\rm Y}$, then the Cartesian product will result in relation $R$, which is contained in the Cartesian product space so that $A\times B=R\subset X\times Y$. The membership function of fuzzy relation $R$ is calculated according to equation (3).

$${\mu }_R\left(x,y\right)={\mu }_{A\times {\rm B}}\left(x,y\right)=\min ({\mu }_A\left(x\right),{\mu }_B\left(y\right))$$

(3)

4.2. Fuzzy Synthetic Evaluation

The Fuzzy Synthetic Evaluation (FSE) has been widely used to assess muti-criteria problems [63,64,65,66]. The FSE conceptualizes a decision making problem at three levels: the indicators, the criteria, and the alternatives [63,64,65]. It associates the three levels drawing on fuzzy relations. The steps of FSE follow:

i)

Assume that $\mathrm{C}=\left\{C_i,\mathrm{i}=1,\mathrm{...},\mathrm{n}\right\}$ is the set of criteria, and $C_i$ indicates criterion (i). This study assumes the criterion “usefulness” thus, $n=1$.

ii)

Assume $\mathrm{I}=\left\{I_j,\mathrm{j}=1,\mathrm{...},\mathrm{m}\right\}$ is the set of indicators, where $I_j$ indicates indicator (j). It consists of the “title-sentiment (t_s)”, “review-sentiment (r_s)”, “title-articulacy (t_a)” and the “review-articulacy (r_a)” indicators thus, $m=4$.

iii)

$\mathrm{AG}=\left\{{\mathrm{AG}}_p,\mathrm{p}=1,\mathrm{...},\mathrm{s}\right\}$ is the set of assessment grades for criteria, indicators, and alternatives, with ${\mathrm{AG}}_p$ indicating assessment grades.

More specifically,

the set of assessment grades ${\mathrm{AG}}_p^{Ci}$ used for the criterion “usefulness” is, ${\mathrm{AG}}^{Ci}=\left\{{\mathrm{AG}}_p^{Ci},\mathrm{p}=1,\mathrm{...}{,\mathrm{k}}_{\mathrm{Ci}}\right\}=\left\{\mathrm{Low},\mathrm{Medium},\mathrm{High}\right\}$.

Respectively for the indicators,

${\mathrm{AG}}^{\mathrm{Ij}}=\left\{{\mathrm{AG}}_p^{\mathrm{Ij}},\mathrm{p}=1,\mathrm{...}{,\mathrm{k}}_{\mathrm{Ij}}\right\}$, for $I_1,I_2,I_3,I_4$, which in our case are $I_{ts},I_{rs},I_{ta},I_{ra}$

$${\mathrm{AG}}^{I_1}=\left\{{\mathrm{AG}}_p^{I_1},\mathrm{p}=1,\mathrm{...}{,\mathrm{k}}_{I_1}\right\}=\left\{{\mathrm{AG}}_p^{t_s},\mathrm{p}=1,\mathrm{...},3\right\}=\left\{\mathrm{Negative},\mathrm{Neutral},\mathrm{Positive}\right\}$$

$${\mathrm{AG}}^{I_2}=\left\{{\mathrm{AG}}_p^{I_2},\mathrm{p}=1,\mathrm{...}{,\mathrm{k}}_{I_2}\right\}=\left\{{\mathrm{AG}}_p^{r_s},\mathrm{p}=1,\mathrm{...},3\right\}=\left\{\mathrm{Negative},\mathrm{Neutral},\mathrm{Positive}\right\}$$

$${\mathrm{AG}}^{I_3}=\left\{{\mathrm{AG}}_p^{I_3},\mathrm{p}=1,\mathrm{...}{,\mathrm{k}}_{I_3}\right\}=\left\{{\mathrm{AG}}_p^{t_a},\mathrm{p}=1,\mathrm{...},3\right\}=\left\{\mathrm{Low},\mathrm{Medium},\mathrm{High}\right\}$$

$${\mathrm{AG}}^{I_4}=\left\{{\mathrm{AG}}_p^{I_4},\mathrm{p}=1,\mathrm{...}{,\mathrm{k}}_{I_4}\right\}=\left\{{\mathrm{AG}}_p^{r_a},\mathrm{p}=1,\mathrm{...},3\right\}=\left\{\mathrm{Low},\mathrm{Medium},\mathrm{High}\right\}$$

All the above consist of three grades, therefore $k_{C_1}=k_{I_1}=k_{I_2}=k_{I_3}=k_{I_4}=3$.

iv)

$\mathrm{A}=\left\{r_v,\mathrm{v}=1,\mathrm{...},\mathrm{z}\right\}$ is the set of the alternatives, where (z) is the number of reviews that are potentially considered by the users when seeking advice for a destination.

v)

Establish the membership function matrix $\mathrm{R}$ for each nationality $\mathrm{Nat}=\left\{{\mathrm{Nat}}_g,\mathrm{g}=1,\mathrm{...},\mathrm{d}\right\}$,

(in our case $d=5,\left\{\mathrm{British},\mathrm{US},\mathrm{Australian},\mathrm{Greek},\mathrm{Dutch}\right\}$):

(4)

where $r_{z,I_j}$ indicates the membership degrees to which $I_j$ satisfies assessment grade ${\mathrm{AG}}_{k_{\mathrm{Ij}}}^{\mathrm{Ij}}$, to the total of reviews (z).

For example, in order to calculate the membership degree to which indicator “title sentiment - $t_s$”, indicator, $I_1$, satisfies assessment grade, ${\mathrm{AG}}_1^{I_1}$, “Negative” (element (1,1) of the matrix above), we calculate the percentage of the total reviews that are related as “Negative”. Respectively, we calculate the percentage of the total reviews that are related as “Neutral” and “Positive”.

Let assume that the calculation returns that 17% of the British reviews are rated as “Negative”, 23% as “Neutral” and 60% as “Positive”; then the membership function of the “title-sentiment - $t_s$”, is given by (5):

$$\frac{0,17}{\mathrm{Negative}},\frac{0,23}{\mathrm{Neutral}},\frac{0,60}{\mathrm{Positive}}$$

(5)

Each of the three assessment grades is assigned to a rating factor $S_p=1,2,3$, e.g. Negative=1, Neutral=2, and Positive=3, and Low=1, Medium=2, and High=3, as used in other studies [63,64,66,67].

$$\frac{0,17}{\mathrm{Negative}},\frac{0,23}{\mathrm{Neutral}},\frac{0,60}{\mathrm{Positive}}\to \frac{0,17}{1}+\frac{0,23}{2}+\frac{0,60}{3}$$

vi)

Calculate the weights ${\mathrm{WI}}_j$ for each indicator $I_j$. This study adopts the ordered weight averaging aggregation (OWA), which is often used in fuzzy logic [63,67]. The weights are calculated using the equations (6) and (7):

$${}_{}{}^{{\mathrm{Nat}}_g}\mathrm{I}{\mathrm{mp}}_{I_j}={{\displaystyle \sum }}_p\left(S_p\times r_{z,I_j}\right)$$

(6)

$${}_{}{}^{{\mathrm{Nat}}_g}\mathrm{W}{\mathrm{I}}_{I_j}=\frac{\left({}_{}{}^{{\mathrm{Nat}}_g}\mathrm{I}{\mathrm{mp}}_{I_j}\right)}{{{\displaystyle \sum }}_{j=1}^m\left({}_{}{}^{{\mathrm{Nat}}_g}\mathrm{I}{\mathrm{mp}}_{I_j}\right)}$$

(7)

where

${}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{I}{\mathrm{mp}}_{{\mathrm{I}}_{\mathrm{j}}}$ is the aggregated importance vector for indicator $I_j$,

$S_p$ is the rating factor given to assessment grade ${\mathrm{AG}}_{k_{\mathrm{Ij}}}^{\mathrm{Ij}}$, and

m is the number of indicators under one criterion.

Therefore, the vector of weights for the (m) indicators is given by:

$${}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}\mathrm{I}=\left[\begin{array}{c}{}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}{\mathrm{I}}_{{\mathrm{I}}_1}\\ {}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}{\mathrm{I}}_{{\mathrm{I}}_2}\\ \cdots \\ {}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}{\mathrm{I}}_{{\mathrm{I}}_{\mathrm{m}}}\end{array}\right]=\left[\begin{array}{c}{}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}{\mathrm{I}}_{{\mathrm{t}}_{\mathrm{s}}}\\ {}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}{\mathrm{I}}_{{\mathrm{r}}_{\mathrm{s}}}\\ {}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}{\mathrm{I}}_{{\mathrm{t}}_{\mathrm{a}}}\\ {}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}{\mathrm{I}}_{{\mathrm{r}}_{\mathrm{a}}}\end{array}\right]$$

(8)

vii)

Calculate the weights ${\mathrm{WC}}_i$, for each criterion $C_i$. The weights are calculated using the equation (9).

$${}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{WC}_{\mathrm{i}}=\frac{{({\displaystyle \sum _{j=1}^m{}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}{\mathrm{I}}_{{\mathrm{I}}_{\mathrm{j}}}})}_i}{{({\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^m{}^{{\mathrm{Nat}}_{\mathrm{g}}}\mathrm{W}{\mathrm{I}}_{{\mathrm{I}}_{\mathrm{j}}}}})}_i},$$

(9)

for all indicators under criterion $C_i$.

This study assumes one criterion, i.e. the “usefulness” of reviews.

viii)

Establish the membership function matrix APF of the alternatives’ performance for each nationality

(10)

$$\begin{array}{r}{\mathrm{A}}_{{\mathrm{Nat}}_1}{=\text{{}\mathrm{r}}_{{\mathrm{v}}_{{\mathrm{Nat}}_1}}{,\mathrm{v}}_{{\mathrm{Nat}}_1}=1,\mathrm{...}{,\mathrm{z}}_{{\mathrm{Nat}}_1}{\text{}},\mathrm{where}\mathrm{z}}_{{\mathrm{Nat}}_1}\mathrm{is}\mathrm{the}\mathrm{total}\mathrm{number}\mathrm{of}\mathrm{British}\mathrm{Reviews}\\ {\mathrm{A}}_{{\mathrm{Nat}}_2}{=\text{{}\mathrm{r}}_{{\mathrm{v}}_{{\mathrm{Nat}}_2}}{,\mathrm{v}}_{{\mathrm{Nat}}_2}=1,\mathrm{...}{,\mathrm{z}}_{{\mathrm{Nat}}_2}\text{}},\\ \dots \\ {\mathrm{A}}_{{\mathrm{Nat}}_5}{=\text{{}\mathrm{r}}_{{\mathrm{v}}_{{\mathrm{Nat}}_5}}{,\mathrm{v}}_{{\mathrm{Nat}}_5}=1,\mathrm{...}{,\mathrm{z}}_{{\mathrm{Nat}}_5}\text{}},\end{array}$$

$$\mathrm{A}={\displaystyle \sum }_{h=1}^d{A}_{{\mathrm{Nat}}_h}$$

$$\mathrm{z}={\displaystyle \sum }_{h=1}^d{z}_{{\mathrm{Nat}}_h}$$

$a_{z,p}$ indicates the membership degrees to which alternative $z$ satisfies assessment grade ${\mathrm{AG}}_{k_{\mathrm{Ij}}}^{\mathrm{Ij}}$. This study considers the set of reviews that a user may consider as the set of alternatives.

ix)

Aggregate performance evaluation for alternative $z$ using fuzzy composition [61,63,65]as shown in equation (11):

$${\omega }_z=W\otimes AP{F}_z$$

(11)

The three scalars of ${\omega }_p^z$, represent the membership degrees of each assessment grade $p$, for the alternative$z$, thus

$${\omega }_z\to \frac{{\omega }_1^z}{\mathrm{Negative}}+\frac{{\omega }_2^z}{\mathrm{Neutral}}+\frac{{\omega }_3^z}{\mathrm{Positive}}$$

A crisp value for ${\omega }_z$ can be obtained after defuzzification. This study adopts the equations (15) and (16) used in [63] in order to calculate the score for alternative$z$:

$${\omega }_z=5{\omega }_{\mathrm{Negative}}^z+50{\omega }_{\mathrm{Neutral}}^z+100{\omega }_{\mathrm{Positive}}^z$$

(12)

$$\mathrm{A}\mathrm{final}\mathrm{usefulness}\mathrm{score}\mathrm{is}\mathrm{given}\mathrm{by}{\Delta }_z=100-{\omega }_z$$

(13)

5. Results

5.1. Reviews’ Sentiments membership functions

Results indicate that the British exhibit more consistent behaviour with their sentiments expressed in reviews’ titles and bodies in all three sentiments categories than the other nationalities in the sample.

Table 1. Title and Review sentiment percentages for each nationality in the sample.

	Sentiment
	Title	Review	Title	Review	Title	Review
	Negative		Neutral		Positive
British	4,08	4,68	6,83	6,65	89,08	88,67
USA	4,73	32,21	5,89	7,10	89,37	60,67
Australian	2,60	28,74	6,46	6,40	90,94	64,86
Greek	3,55	6,26	6,68	24,13	89,77	69,61
Dutch	3,86	30,68	6,69	4,56	89,45	64,76

Table 1. Title and Review sentiment percentages for each nationality in the sample.

	Sentiment
	Title	Review	Title	Review	Title	Review
	Negative		Neutral		Positive
British	4,08	4,68	6,83	6,65	89,08	88,67
USA	4,73	32,21	5,89	7,10	89,37	60,67
Australian	2,60	28,74	6,46	6,40	90,94	64,86
Greek	3,55	6,26	6,68	24,13	89,77	69,61
Dutch	3,86	30,68	6,69	4,56	89,45	64,76

The sentiments British expressed in either their review’s titles or the reviews’ documents are almost identical. All nationalities in the sample are unanimous in expressing more positive than negative sentiments to a large extent, which is a good sign for the quality of the services reviewers received during their visits.

Differences range from a 20,16% more positive evaluations in titles than in full texts in the Greek sample, to a 28,7% in the USA sample. Similarly, more negative evaluations are found in full documents than in titles. Sentiment frequencies vary from a minimum of 2,71% more negatives in the full documents in the Greek sample, to the maximum of 27,48% in the USA sample. This implies that when the Greeks express negative sentiments in their titles, they do so more concisely and more accurately than the other three nationalities. For the Australians, and Dutch samples show larger percentage differences which are 26,14% and 26,82% respectively. With respect to neutral sentiments, percentage differences between titles and full reviews are rather small ranging from almost 0 to 2,13%, with the exception of the Greek sample (17,45%).

To accommodate the differences in users’ behaviour, this study proposes to represent sentiments from different nationalities as triangular fuzzy sets $\tilde{A}\left(l,m_i,u_i\right)$. According to [64,66,67], the membership function of each sentiment fuzzy set is formed as follows: In the sample of the total 42.678 British reviews, 1.742 expressed negative sentiment in their title, which returns a 4% negative reviews. Similarly, neutral titles account for a 6%, and positive ones for an 89%. Thus, for the British title sentiments the membership function is: $\frac{0,04}{Negative}+\frac{0,06}{Neutral}+\frac{0,89}{Positive}\to \frac{0,04}{1}+\frac{0,06}{2}+\frac{0,89}{3}$. The rating given to each assessment grade (i.e. Negative=1, Neutral=2, Positive=3), is adopted by [63,64,66,67]. The membership functions for both titles and reviews sentiments for each nationality, are shown in Table 2.

Figure 2 clearly depicts the differences between the British and the USA reviews’ sentiment fuzzy sets, which imply the behavioural differences between the two nationalities.

The sentiments’ fuzzy sets with their membership functions show the level of “positiveness” in reviews for each nationality. The diagrams show that British and Greek reviewers are inclined towards more positive comments than the other nationalities in the sample. The USA, Australians and Dutch users’ sentiments are closer for both titles and reviews.

5.2. Reviews’ Articulacy membership functions

Table 3 shows the results regarding the articulacy of reviews by calculating the mean and standard deviation of both the titles and the full review for each nationality.

The normalisation of the number of words in titles and reviews is performed by applying equation (14),

$$w_{i,j}^{norm}=\frac{w_{i,j}}{\max ({{\displaystyle \sum }}_{i,j}{w}_{i,j})}$$

(12)

where,

$w_{i,j}^{norm}$ indicates the normalised values of the number of words in the title for nationality $\left(i\right)$ and review $\left(j\right)$, and

$w_{i,j}$, represents the original number of words in the title for nationality and review.

Next, the fuzzification of titles and reviews word numbers is performed by using equation (1), and the TFN membership functions shown in Table 4.

Following the fuzzification, the articulacy for both titles and reviews is characterised in terms of the fuzzy sets {Low, Medium, High}, similarly as the sentiment is calculated as {Negative, Neutral, Positive}. The membership functions of articulacy are calculated according to [64,66,67], in the same way this study calculates as sentiment membership functions. Of the total 42.678 British reviews in the sample, 31% of the titles were characterized as of low, 41% as of medium and 26% as of high longevity, respectively. Thus, for the British title articulacy the membership function is:

$$\frac{0,32}{Negative}+\frac{0,42}{Neutral}+\frac{0,26}{Positive}\to \frac{0,32}{1}+\frac{0,42}{2}+\frac{0,26}{3}$$

The rating given to each assessment grade (i.e. Negative=1, Neutral=2, Positive=3), is adopted by [63,64,66,67]. The resulting membership functions for title and reviews’ articulacy for each nationality are shown in Table 5.

The articulacy fuzzy sets provide an indication of how informative the reviewers from each nationality are. Figure 3 shows diagrammatically the differences in review titles’ longevity between the British and the USA users.

Figure 3 diagrams show that the British are closer to the USA and the Dutch users in terms of the titles’ articulacy, while the Greeks are closer to the Australians. With respect to the reviews the Greeks and the USA exhibit similar behaviour, while the British are more similar to the Dutch and the Australians. An analysis of similarities and differences among nationalities that would include other indicators such as gender, age, preferences, may be used in order to develop a more comprehensive users’ profile.

5.3. Assessing Usefulness of Reviews by incroporating users biases

Drawing on the membership functions shown in Table 2 and Table 5, by using equation (4) the membership function matrix $\mathrm{R}$ for the British users, is formed as follows:

$${}_{}{}^{\mathrm{British}}R=\left(r_{z,I_j}\right)=\begin{array}{c}grades\\ t_s\\ r_s\\ t_a\\ r_a\end{array}\left[\begin{array}{ccc}\mathrm{Negative}/\mathrm{Low}& \mathrm{Neutral}/\mathrm{Medium}& \mathrm{Positive}/\mathrm{High}\\ 0,04& 0,06& 0,90\\ 0,05& 0,07& 0,88\\ 0,32& 0,42& 0,26\\ 0,30& 0,30& 0,40\end{array}\right]$$

Similarly, membership function matrices are calculated for all nationalities in the sample.

The importance matrix and the weights for each indicator are calculated using equations (6) and (7):

$${}_{}{}^{\mathrm{British}}\mathrm{I}{\mathrm{mp}}_{I_1}=\begin{array}{c}grades\\ t_s\end{array}\left[\begin{array}{ccc}Negative& Neutral& Positive\\ 0,04& 0,12& 2,70\end{array}\right]$$

$${}_{}{}^{\mathrm{British}}\mathrm{I}{\mathrm{mp}}_{I_2}=\begin{array}{c}grades\\ r_s\end{array}\left[\begin{array}{ccc}Negative& Neutral& Positive\\ 0,05& 0,14& 2,64\end{array}\right]$$

$${}_{}{}^{\mathrm{British}}\mathrm{I}{\mathrm{mp}}_{I_3}=\begin{array}{c}grades\\ t_a\end{array}\left[\begin{array}{ccc}Low& Medium& High\\ 0,32& 0,84& 0,78\end{array}\right]$$

$${}_{}{}^{\mathrm{British}}\mathrm{I}{\mathrm{mp}}_{I_4}=\begin{array}{c}grades\\ r_s\end{array}\left[\begin{array}{ccc}Low& Medium& High\\ 0,30& 0,60& 1,20\end{array}\right]$$

Thus,

$${}_{}{}^{\mathrm{British}}\mathrm{I}\mathrm{mp}==\begin{array}{c}\\ grades\\ t_s\\ r_s\\ t_a\\ r_a\end{array}\left[\begin{array}{ccc}Negative& Neutral& Positive\\ Low& Medium& High\\ 0,04& 0,12& 2,70\\ 0,05& 0,14& 2,64\\ 0,32& 0,84& 0,78\\ 0,30& 0,60& 1,20\end{array}\right]=\left[\begin{array}{c}2,86\\ 2,83\\ 1,94\\ 2,10\end{array}\right]$$

Finally, the weights for the British users are:

$${}^{\mathrm{British}}\mathrm{W}\mathrm{I}=\left[\begin{array}{c}{}^{\mathrm{British}}\mathrm{W}{\mathrm{I}}_{{\mathrm{I}}_1}\\ {}^{\mathrm{British}}\mathrm{W}{\mathrm{I}}_{{\mathrm{I}}_2}\\ {}^{\mathrm{British}}\mathrm{W}{\mathrm{I}}_{{\mathrm{I}}_3}\\ {}^{\mathrm{British}}\mathrm{W}{\mathrm{I}}_{{\mathrm{I}}_4}\end{array}\right]=\left[\begin{array}{c}{}^{\mathrm{British}}\mathrm{W}{\mathrm{I}}_{{\mathrm{t}}_{\mathrm{s}}}\\ {}^{\mathrm{British}}\mathrm{W}{\mathrm{I}}_{{\mathrm{r}}_{\mathrm{s}}}\\ {}^{\mathrm{British}}\mathrm{W}{\mathrm{I}}_{{\mathrm{t}}_{\mathrm{a}}}\\ {}^{\mathrm{British}}\mathrm{W}{\mathrm{I}}_{{\mathrm{r}}_{\mathrm{a}}}\end{array}\right]=\begin{array}{c}{t}_s\\ r_s\\ t_a\\ r_a\end{array}\left[\begin{array}{c}0,29\\ 0,29\\ 0,20\\ 0,22\end{array}\right]$$

Similarly, the weights for each nationality are:

$${}_{}{}^{\mathrm{USA}}\mathrm{W}\mathrm{I}=\begin{array}{c}{t}_s\\ r_s\\ t_a\\ r_a\end{array}\left[\begin{array}{c}0,30\\ 0,24\\ 0,22\\ 0,24\end{array}\right],{}_{}{}^{\mathrm{Greek}}\mathrm{W}\mathrm{I}=\begin{array}{c}{t}_s\\ r_s\\ t_a\\ r_a\end{array}\left[\begin{array}{c}0,31\\ 0,28\\ 0,21\\ 0,20\end{array}\right],$$

$${}_{}{}^{\mathrm{Australian}}\mathrm{W}\mathrm{I}=\begin{array}{c}{t}_s\\ r_s\\ t_a\\ r_a\end{array}\left[\begin{array}{c}0,31\\ 0,25\\ 0,21\\ 0,23\end{array}\right],{}_{}{}^{\mathrm{Dutch}}\mathrm{W}\mathrm{I}=\begin{array}{c}{t}_s\\ r_s\\ t_a\\ r_a\end{array}\left[\begin{array}{c}0,31\\ 0,26\\ 0,22\\ 0,21\end{array}\right]$$

The results indicate the differences as well as the similarities among the nationalities, which have been diagrammatically depicted in Figure 2 and Figure 3. Larger are the differences found in titles and reviews sentiments as compared to the differences in articulacy.

In order to calculate usefulness of reviews, assume Review-1 as an alternative in the FSE model. Review-1 can be a certain review or a collection of reviews over a period of time, e.g. 5 years. The aggregation of reviews over a period of time is suggested as especially useful [38]. By using MS Azure sentiment analysis, Review-1, sentiment polarity is quantified with a positive, neutral, or negative value. The numerical values of Review-1 articulacy for both title and review are then fuzzified using equation (1) and the TFN membership functions shown in Table 4.

Then the performance membership function matrix for Review-1, using equation (10) follows:

$${\mathrm{APF}}^{\mathrm{Review}-1}=(a_{z,p})=\begin{array}{c}\\ grades\\ t_s\\ r_s\\ t_a\\ r_a\end{array}\left[\begin{array}{ccc}Negative& Neutral& Positive\\ Low& Medium& High\\ 0,00& 0,00& 0,25\\ 0,25& 0,50& 0,75\\ 0,25& 0,50& 0,75\\ 0,50& 0,75& 1,00\end{array}\right]$$

The aggregated performance evaluation, regarding the usefulness of alternative”$\mathrm{Re}view-1$” is calculated using the fuzzy composition in equation (11):

$${\mathsf{\omega }}_{\mathrm{Review}-1}^{\mathrm{British}}=\begin{array}{c}\mathrm{High}\\ \mathrm{Medium}\\ \mathrm{Low}\end{array}\left[\begin{array}{c}0,23\\ 0,41\\ 0,66\end{array}\right]$$

Similarly, for the rest of the nationalities in the sample:

$${\mathsf{\omega }}_{\mathrm{Review}-1}^{\mathrm{USA}}=\begin{array}{c}\mathrm{High}\\ \mathrm{Medium}\\ \mathrm{Low}\end{array}\left[\begin{array}{c}0,23\\ 0,41\\ 0,66\end{array}\right],$$

$${\mathsf{\omega }}_{\mathrm{Review}-1}^{\mathrm{Greek}}=\begin{array}{c}\mathrm{High}\\ \mathrm{Medium}\\ \mathrm{Low}\end{array}\left[\begin{array}{c}0,22\\ 0,39\\ 0,64\end{array}\right],$$

$${\mathsf{\omega }}_{\mathrm{Review}-1}^{\mathrm{Australian}}=\begin{array}{c}\mathrm{High}\\ \mathrm{Medium}\\ \mathrm{Low}\end{array}\left[\begin{array}{c}0,23\\ 0,40\\ 0,65\end{array}\right]$$

$${\mathsf{\omega }}_{\mathrm{Review}-1}^{\mathrm{Dutch}}=\begin{array}{c}\mathrm{High}\\ \mathrm{Medium}\\ \mathrm{Low}\end{array}\left[\begin{array}{c}0,22\\ 0,39\\ 0,64\end{array}\right].$$

Finally, by using equations (12) and (13), the “Usefulness Score” =12,80. This score indicate how useful is for the British users to read the$\mathrm{Re}view-1$. Similarly, the usefulness score is calculated for each nationality as follows:

“Usefulness Score (British)” =12,80,

“Usefulness Score (USA)” =12,89,

“Usefulness Score (Greek)” =14,65,

“Usefulness Score (Australian)” =13,44,

“Usefulness Score (Dutch)” =14,73.

The “Usefulness Score” shows that $\mathrm{Re}view-1$, is not equally useful for all nationalities. It is more useful for the Greeks, the Australians, and the Dutch to read, but not so, for the British and the USA users. The difference in usefulness lies in the fact that not all users exhibit the same behaviour when expressing themselves. Some may express themselves by using more words in their reviews or others may be more lenient than others or exhibit a behavior in between. As already discussed in the relevant literature [35,37,55]not necessarily all nationalities will select the same set of reviews. The “Usefulness Score” can be used to rank reviews or a collection of reviews and assist users to focus on the most useful rev

6. Discussion

Behavioural differences exist among different nationalities in the way they express their sentiments. Some nationalities express their sentiments in the titles while others become more precise in their full reviews. Therefore, titles do not always convey in full what users’ sentiments imply that they may create a false perception of other reviewers’ experiences. An explanation may be that reviewers find more space in reviews’ documents to express their sentiments in more details. In particular, the British, followed by the Greeks exhibit a more consistent behaviour in expressing themselves. Users of both nationalities express themselves concisely in their reviews’ titles. By reading just the titles of reviews written by a British or a Greek reviewer, one can understand the reviewer’s negative feelings rather clearly. With respect to neutral sentiments the Greeks exhibit quite a varying behaviour, as opposed to the rest nationalities. Thus, neutral judgments are not so informative when they are expressed by Greek users. Regarding positive sentiments, reviews tend to be more lenient in their titles and more critical in their full documents. With the exception of the British rather balanced evaluations, positive scores are more frequent in titles than in reviews full documents for all other nationalities in the sample. Therefore, when positive sentiments are expressed, it is suggested that one should read the whole review document to have a more detailed understanding of the users’ experiences. By just reading the review title will not suffice to comprehend the thoughts of the user. The results of this research are in line with other studies [35,36,55], and suggests that nationality may imply differences in the way users write review and express their sentiments.

This study indicates that there are differences among nationalities in the sample with respect to the number of words in the title, and the reviews documents. Although differences may not be as profound as in sentiments, results show that the British, the USA, the Greeks and the Australian reviewers exhibit similar behaviour regarding their reviews length. However, the Dutch reviewers are more frugal in their full reviews, since their average number of words is approximately half the average number of the rest of the nationalities.

This study suggests that fuzzy logic can be used to not only represent the differences in behaviour among nationalities as fuzzy sets but also to calculate the “usefulness” of a review by utilising the FSE. Fuzzy sets provide the means for quantifying the differences among users that imply the biases of different nationalities. The FSE of reviews’ usefulness can be used to suggest users to read the most suitable reviews for them through fuzzy relations composition. The most useful reviews are those which reflect their behaviour better. Therefore, the proposed approach can be used within the context of a voting system or online review platform [38] that aims to assist users identify useful reviews that reflect their personalised preferences, behaviour, and perception. The current work, considers usefulness as the only criterion. The FSE and the fuzzy relations’ compositions can be used to combine several other characteristics of the reviews, e.g., publication date, features mentioned in reviews, etc.

7. Limitations of the study and Future Research

With respect to the limitations of this study, this research uses only one sentiment analysis method and focuses only on reviews’ sentiment polarity, strength and articulacy for certain nationalities. The plethora of data that can be collected from platforms such as Tripadvisor, Booking, etc., and the subsequent analysis of features such as gender, date of visit, date of review publication, etc., would result in more detailed insights of users’ ways of expressing their feelings, thus providing a more comprehensive view of biases. Furthermore, testing the efficiency of the proposed approach to accurately assess users’ sentiment when combined with sentiment analysis algorithms, would probably lead to developing domain specific sentiment analysis methods.

Future research may also focus on extending the modelling approach to include a broader perspective of users’ behaviour. Research efforts may focus on developing models that capture the different ways that users express themselves. Users do not all give the same sentiment strength even if they use the same or similar words in their reviews. Thus, the expressions “good service”, or “enjoyable experience” may not convey exactly the same meaning for all users. Since fuzzy logic allows for dealing with subjectivity, future research efforts may attempt to define different fuzzy sets for same concepts in order to reflect how different users perceive the same or similar expressions.

8. Conclusions

This study suggests that since bias affects users’ perception of reviews sentiment and length, fuzzy logic provides the necessary theoretical and methodological foundation to measure reviews’ usefulness. User reviews’ sentiment is subjective. Bias attributed to gender, nationality, etc., inherited in sentiment analysis has been examined in many studies. Fuzzy logic provides the means to deal with impartial information which is often found in reviews and represent the subjectivity that is embedded in how people with different cultural backgrounds, from different age groups, with varying preferences express their experiences. This study analysed users' reviews and calculated the membership functions of fuzzy sets that exhibit the bias as well as the similarities inherited among nationalities and expressed by the way they communicate their sentiments and write their reviews. Results identify aspects, such as the bias attributed to different nationalities as expressed in terms of sentiment expressed in titles or the reviews as well as titles and reviews articulacy. Such aspects, if not overrepresented or underrated, may increase the accuracy of sentiment analysis techniques thus improving reviews readability and usefulness when judging a service or a product.