Logic Operations For Assessment of Experts’ Weight In Fuzzy Rule-Based Systems

1. Introduzione

Multi-Criteria Group Decision-Making (MCGDM) extends Multi-Criteria Decision-Making (MCDM) to settings in which multiple decision-makers evaluate alternatives and a collective outcome is required [1]. In standard MCDM, criteria are identified and weighted according to their relative importance [2,3]. In MCGDM, an additional layer is needed, namely the weighting of the decision-makers themselves [1,4]. This issue is crucial because experts may differ in competence, objectives, and cognitive bias, and assigning equal importance to all evaluators may lead to distorted or unstable decisions [4,5,6].

The literature usually distinguishes between subjective weighting schemes, based on self-assessment or external judgment, and objective weighting schemes, based on mathematical indicators such as consistency, consensus, distance, or performance measures [3,7]. Despite their usefulness, both families present practical limitations, especially when weighting is perceived as a personal evaluation of the participating experts rather than as a modelling choice [1,5]. For this reason, recent research has explored hybrid and automated strategies in which artificial experts support or partially replace human experts in fuzzy MCGDM pipelines [8,9,10].

Artificial experts, such as optimisation algorithms, recommender systems, simulation-driven agents, and predictive models, are increasingly relevant in decision processes where scalability, repeatability, and traceability are required. In these settings, they should not be regarded merely as computational tools, but as decision contributors whose outputs may affect the collective ranking of alternatives. This motivates the need for principled weighting mechanisms capable of integrating human and artificial experts within the same aggregation pipeline, while preserving interpretability and operational flexibility. The broader interest in soft-computing approaches and in their application to new domains is coherent with this perspective [11,12,13,14].

In this work, an extension of the work in [10], we follow this direction and study a fuzzy framework in which expert weights are produced through a Fuzzy Rule-Based System (FRBS). The key idea is to encode weighting policies as interpretable IF–THEN rules. This allows us to obtain weights in the interval $[0,1]$ that are transparent, adaptable, and suitable for mixed groups including both human and artificial decision-makers.

A broader panoramic view places this proposal within the long development of fuzzy-based decision systems, from early rule-based approximate reasoning models and linguistic control architectures to more recent fuzzy MCDM and MCGDM procedures [15].

In this paper, we propose a new version of the state-of-the-art in fuzzy MCGDM [15]. This new method has been tested on an Large Language Model (LLM) scoring task. The FRBS has been able to weigh performances, technological requirements and the open-source of LLM tested. In conclusion, has been proven that the proposed method is flexible enough to address real-world tasks. Since real-world has not have a unique direction of optimisation, in this contribution it is proven that optimisation algorithms should not either.

The remainder of this paper is organised as follows. Section 2 reviews the literature most relevant to expert weighting, fuzzy rule-based systems, and group decision-making. Section 3 introduces the mathematical background on fuzzy sets, FRBSs, and the MCGDM framework required for the subsequent developments. Section 4 presents the proposed methodology and describes the corresponding operational pipeline. Section 5 illustrates the approach through an application involving the ranking of four open-source LLM. Finally, Section 6 summarises the main findings of the paper and outlines possible directions for future research.

2. Related Works

From the decision-analytic and data-driven viewpoint, user weighting is commonly addressed through TOPSIS [16], VIKOR [17,18,19], scoring systems, and train–test evaluation protocols. TOPSIS ranks users by closeness to an ideal profile [16], while VIKOR provides a compromise ranking between collective utility and individual regret [17,18,19]. Scoring systems based on reinforcement learning and curriculum learning capture dynamic behaviour, long-term utility, and performance under increasing task difficulty; this line is consistent with performance-based expert weighting [20,21]. Train–test protocols improve robustness by separating calibration from out-of-sample evaluation, reducing overfitting and enabling fairer comparison across human experts and AI agents; related aggregation strategies include adaptive weighting and evaluator-reliability estimation [1,4,22].

A second relevant line of work concerns the role of Fuzzy Rule-Based Systems themselves. Classical contributions by Mamdani and by Takagi–Sugeno established FRBSs as interpretable mechanisms for modelling control and decision policies through linguistic IF–THEN rules [23,24,25]. Later studies clarified the semantics and design of fuzzy rules, and surveyed the evolution of FRBSs from control-oriented architectures to broader knowledge-based and decision-support settings [26,27,28]. More recent reviews highlight that FRBSs remain attractive when transparency, modularity, and explainability are important, especially in application domains where purely black-box optimisation is difficult to justify [29,30,31]. This makes FRBSs a natural candidate for expert-weighting problems, because they allow weighting criteria to be stated explicitly and revised without abandoning formal aggregation procedures.

Within MCGDM, the literature usually distinguishes subjective, objective, and hybrid approaches to weighting decision makers. Subjective approaches rely on direct judgments, pairwise comparisons, or elicited confidence assessments, and are close in spirit to classical expert-judgment and preference-elicitation traditions [2,20,21]. Objective approaches infer weights from observable properties of the decision process, for example consistency, consensus, dispersion, deviation, or reliability indicators [22,32,33]. Hybrid approaches combine both sources of information, often blending structural indicators with performance or consensus information in order to avoid the rigidity of purely subjective schemes and the impersonality of purely objective ones [4,7,34]. The survey literature confirms that no single family dominates across applications, and that weighting design remains strongly dependent on the decision context, the available data, and the desired level of interpretability [1,3,7].

This point becomes even more delicate when some evaluators are artificial experts rather than human experts. In decision-support systems, artificial experts may take the form of optimisation routines, learned predictors, recommender components, simulation agents, or explainable AI modules that generate scores, rankings, or recommendations [8,9,35]. Their inclusion raises specific methodological questions: unlike human experts, artificial experts can be highly repeatable and scalable, yet their outputs may depend on training data, design assumptions, or hidden model constraints; therefore, assigning them a weight is not merely a social choice, but also a modelling decision about trust, accountability, and comparability across heterogeneous decision sources [36,37]. For this reason, explainability is not an accessory feature but a central requirement when human and artificial expertise are aggregated in the same MCGDM pipeline [8,30].

Against this background, the main gap addressed by the present study is not the absence of weighting methods per se, but the limited application-oriented validation of FRBS-based weighting schemes for artificial experts and the still scarce use of explicit sensitivity analyses for this specific setting. The methodological building blocks are available in the broader MCGDM and FRBS literature, and a first focused proposal for weighting artificial experts through fuzzy rules has already been presented in [10]; however, the literature still offers little evidence on how such a scheme behaves when transferred to realistic decision tasks, compared with more conventional weighting strategies, and perturbed under different modelling assumptions. Since sensitivity analysis is widely recognised as a key instrument in MCDM and MCGDM for testing robustness, credibility, and dependence on parameter choices, this remains a non-trivial omission [38,39].

In this sense, the present article differs both from the previous methodological contribution and from the broader families of weighting methods already available in the literature. Relative to [10], the contribution here is steered less by the formal introduction of the FRBS mechanism and more by its validation in an application-oriented scenario inspired by the assessment of LLMs, with attention to how performance, technological requirements, and openness can be jointly encoded in the weighting process. Relative to standard subjective, objective, or hybrid schemes, the proposed strategy emphasises an interpretable rule base, making the rationale for assigning different weights to human and artificial experts more explicit and auditable. This orientation is consistent with broader attempts to move fuzzy methods toward new application domains while preserving semantic clarity and practical usability [11,12,13,14].

3. Preliminaries

In this section, the main preliminary notions concerning Fuzzy Set Theory (FST) and Fuzzy Rule-based Systems (FRBS) will be reviewed briefly. In the following section, the essential elements of the Multi-Criteria Decision-Making (MCDM) and Multi-Criteria Group Decision-Making (MCGDM) frameworks will be introduced.

3.1. Fuzzy Set Theory

In this section, we recall essential terminology from Fuzzy Set Theory (FST), which underpins FRBSs. For more details on fuzzy sets, we refer the reader to [40,41,42,43,44,45].

A fuzzy set [46] is specified by its membership function, namely a mapping from a Universe of Discourse (UoD) U to a complete lattice – typically the unit interval:

$${\mu }_A:U\mapsto [0,1],$$

(1)

where A denotes the (symbolic) label of the set and ${\mu }_A$ its (semantic) interpretation. In order to keep notation compact, one shall simply write fuzzy set A in place of the fuzzy set labelled A with membership function ${\mu }_A$, and one writes ${\mu }_A\left(x\right)$ for the membership degree of $x\in U$. The family of all fuzzy sets over U is denoted by $\mathcal{F}\left(U\right)$.

Fuzzy set connectives generalise intersection, union, and complement via t-norms, t-conorms (s-norms), and negations.

Definition 1.

(cf. [40]) A function $\otimes :[0,1]\times [0,1]\to [0,1]$ is a t-norm if, for all $x,y,z\in [0,1]$,

$$\begin{array}{}\hfill \text{(Monotonicity)}& y\le z\Rightarrow x\otimes y\le x\otimes z,\hfill \hfill \text{(Associativity)}& x\otimes (y\otimes z)=(x\otimes y)\otimes z,\hfill \hfill \text{(Commutativity)}& x\otimes y=y\otimes x,\hfill \hfill \text{(Boundary Condition)}& x\otimes 1=x.\hfill \end{array}$$

Definition 2.

(cf. [40]) A function $\oplus :[0,1]\times [0,1]\to [0,1]$ is a t-conorm if it satisfies the axioms of def:t-norm with the boundary condition replaced by

$$\begin{array}{}\hfill \text{(Boundary Condition)}& x\oplus 0=x,\hfill \end{array}$$

for all $x\in [0,1]$.

Three logics are commonly used: Lukasiewicz, Gödel, and Product Logic [41] (see Table 1). Zadeh’s operators, defined by $a{\otimes }_{Z}b=min(a,b)$ and $a{\oplus }_{Z}b=max(a,b)$, coincide with Gödel logic for conjunction and disjunction, while differing in the treatment of negation and implication. In order to distinguish the logic used, we write the symbols in Table 2.

A fuzzy relation R on $U_X\times U_Y$ is a fuzzy subset of the Cartesian product:

$${\mu }_R:U_X\times U_Y\mapsto [0,1].$$

(2)

Beyond the usual fuzzy set operations, relations admit a composition operator, fundamental to fuzzy inference. Given $R_1$ on $U_X\times U_Y$ and $R_2$ on $U_Y\times U_Z$, their (sup–min) composition is

$${\mu }_{R_2\circ R_1}(x,z)=\underset{y\in U_Y}{sup}min\{{\mu }_{R_1}(x,y),{\mu }_{R_2}(y,z)\}.$$

(3)

Analogously, the composition of a fuzzy set A on $U_X$ with a relation R on $U_X\times U_Y$ yields

$${\mu }_{R\circ A}\left(y\right)=\underset{x\in U_X}{sup}min\{{\mu }_A\left(x\right),{\mu }_R(x,y)\},$$

(4)

which is known as the compositional rule of inference (CRI) [42].

Intuitively, Equation (4) matches a fact (expressed as a fuzzy set $A_X$ on $U_X$) with a relation R to infer a new fact (expressed as a fuzzy set $A_Y$ on $U_Y$). Viewing the relation R as a fuzzy rule $A_X\to A_Y$, the fuzzy composition of relation underpins the Generalized Modus Ponens (GMP), which is the cornerstone of approximate reasoning:

$$\frac{A_X^{\prime },\hspace{1em}{A}_X\to A_Y}{A_Y^{\prime }}$$

(5)

which, for any premise $A_X^{\prime }\in U_X$, infers a conclusion $A_Y^{\prime }\in U_Y$.

3.1.1. Fuzzy Numbers

A fuzzy number is a specific type of fuzzy subset within the set of real numbers $\mathbb{R}$, characterised by additional defining properties (see, e.g., [41,47]). For these notions, we refer to the definitions and results used in the 1983 paper by P.J.M. van Laarhoven and W. Pedrycz [48]. In this paper, we consider triangular and trapezoidal fuzzy numbers defined as follows.

Definition 3.

Atriangular fuzzy number (TFN) $\tilde{t}=(a,b,c)$, with $a,b,c\in \mathbb{R}$ and $a\le b\le c$, is a fuzzy number whose membership function has a trapezoidal shape, i.e., for every $x\in \mathbb{R}$,

$$\tilde{t}\left(x\right)=(a,b,c)\left(x\right)=\mu \left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{x-a}{b-a}},\hfill & x\in [a,b],\hfill \\ 1\hfill & x=b,\hfill \\ {\displaystyle \frac{x-b}{c-b}},\hfill & x\in [b,c],\hfill \\ 0\hfill & otherwise.\hfill \end{array}\right.$$

Definition 4.

(cf. [49, Definition 2.3]) A trapezoidal fuzzy number (TpFN) $\widetilde{t_p}=(a,b,c,d)$, with $a,b,c,d\in \mathbb{R}$ and $a\le b\le c\le d$, is a fuzzy number whose membership function has a trapezoidal shape, i.e., for every $x\in \mathbb{R}$,

$$\widetilde{t_p}\left(x\right)=(a,b,c,d)\left(x\right)=\mu \left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{x-a}{b-a}},\hfill & x\in [a,b],\hfill \\ 1\hfill & x\in [b,c],\hfill \\ {\displaystyle \frac{x-d}{c-d}},\hfill & x\in [c,d],\hfill \\ 0\hfill & otherwise.\hfill \end{array}\right.$$

3.2. Fuzzy Rule-Based Systems

Fuzzy sets have been used in many applications, but their most notable success is undoubtedly in control systems. Many control systems employ the well-known Fuzzy Rule-Based Systems (FRBSs), which exploit a rule base formed by a simple set of IF–THEN rules in the form

$$\mathrm{IF}\mathtt{antecedents}\mathrm{THEN}\mathtt{consequent},$$

where the antecedents are usually a composition of logical operations (AND, OR, and NOT) of fuzzy sets, while the consequent is usually a fuzzy set. This particular setting of antecedents and consequent is typical of Mamdani FRBSs [23]. For more details on FRBSs, we refer the reader to [27,28,29,30,31].

In particular, FRBSs are explainable systems composed of (see Figure 1 for a graphical representation):

• Fuzzifier: The fuzzifier receives the real-world input to the fuzzy system. In the fuzzy-systems literature, this is commonly termed a crisp input because it represents an exact value of the parameter under consideration. The role of the fuzzifier is to map this precise quantity onto linguistic categories, such as large, medium, or high, by assigning an appropriate degree of membership. This degree typically lies within the real interval $[0,1]$.
• Knowledge Base: The Knowledge Base constitutes the core of the fuzzy system and comprises both the data base and the rule base. The data base specifies the membership functions associated with the fuzzy sets employed in the rules, whereas the rule base contains a collection of fuzzy IF–THEN statements.
• Inference System: The Inference System, also referred to as the decision-making unit, carries out the reasoning process over the rule set. Its function is to determine how the rules are evaluated and combined in order to generate the fuzzy output.
• Defuzzifier: The output produced by the inference stage is inherently fuzzy. Since practical applications generally require a precise output, the defuzzifier converts this fuzzy result into a crisp value that can be interpreted in real-world terms. In this respect, it performs the inverse operation of the input stage.

In essence, FRBSs produce mathematical functions in multiple variables to model almost any complex control system by employing this simple rule base. For instance, if each rule in the rule base contains two antecedents and one consequent, it defines a three-dimensional surface over the fuzzy-set domains associated with the antecedent and consequent variables.

3.3. Multi-Criteria Group Decision-Making

In this section, we briefly recall the basic setting of Multi-Criteria Decision-Making (MCDM), i.e., a branch of decision theory concerned with the comparison of several feasible alternatives in order to identify the optimal one, namely the option that best satisfies given goals, objectives, values, and priorities.

In formal terms, one considers a collection of nalternatives$\mathcal{A}=\{A_1,A_2,\dots ,A_n\}$ and a family of mcriteria$\mathcal{C}=\{C_1,C_2,\dots ,C_m\}$. The latter, sometimes referred to as attributes or goals, specify the dimensions under which the available alternatives are evaluated.

For each pair $(A_i,C_j)$, a value – denoted as score – $a_{ij}$ is assigned in order to express the degree to which the alternative $A_i$ satisfies the criterion $C_j$.

The purpose of this framework is to determine the best alternative, denoted as $A^{*}$. In order to achieve this objective, each criterion $C_j$ is also associated with a coefficient $w_j\in [0,1]$, representing its importance in the decision problem. These coefficients are commonly normalised in such a way that ${\sum }_{j=1}^m{w}_j=1$. Their determination may depend either on the judgement of a single decision-maker or on an aggregated assessment provided by several experts. Methods for deriving and eliciting such weights have been extensively studied in the literature; see, for instance, [50].

In addition to identifying the optimal alternative, MCDM typically yields an overall value $x_i$, usually called rank, for each alternative $A_i$, with $i=1,\dots ,n$. This value summarises the global goodness of the alternative and makes it possible to rank all feasible options according to their overall suitability within the decision problem.

The information concerning alternatives and criteria is often arranged in a decision matrix, where rows are indexed by the alternatives and columns by the criteria. Such a matrix compactly collects the entries (scores) $a_{ij}$ and provides the starting point for the application of aggregation procedures combining criterion evaluations with their corresponding weights.

Among the classical techniques employed to determine this ranking value, one may consider the simplest one, namely the Weighted Sum Method (WSM). In this case, for every $i=1,\dots ,n$, the score assigned to $A_i$ is given by

$$x_i={\displaystyle \sum _{j=1}^m}{a}_{ij}{w}_j.$$

For further details on other methods used to determine ranking values, we refer the reader to [51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67].

The alternatives are then ranked from the largest to the smallest value of $x_i$, and the preferred choice is the one corresponding to the maximum score, that is,

$$A^{*}=\arg \underset{A_i}{max}{x}_i.$$

When multiple experts are involved, the setting is referred to as Multi-Criteria Group Decision-Making (MCGDM), in which the optimal alternatives identified by the individual experts must be combined into a single overall optimal alternative. Let p denote the number of experts, and let $E_k$, for $k=1,\dots ,p$, be the l-th expert. Each expert $E_k$ determines an optimal alternative $A_k^{*}$ by means of an MCDM method. The method adopted by one expert need not coincide with those adopted by the others, so that different experts may employ different MCDM procedures.

Accordingly, the aggregator (see, e.g., [68]) is a function that associates the p optimal alternatives $A_1^{*},\dots ,A_p^{*}$ with a new alternative, denoted by $A^{*}$ for the reader’s convenience, representing the overall optimal choice. In order to determine this aggregated optimal alternative, a weight ${\pi }_k$ is assigned to each expert $E_k$, for $k=1,\dots ,p$. These weights are usually normalised so that ${\sum }_{k=1}^p{\pi }_k=1$, and act as scalar coefficients expressing the relative importance attributed to the optimal alternative $A_k^{*}$ selected by the expert $E_k$. Thus, ${\pi }_k$ may be interpreted as a measure of the expertise of $E_k$, so that its optimal alternative has a greater influence on the determination of the aggregated optimum $A^{*}$.

In general, no universal constraints are imposed on the form of the aggregator, as its definition depends on the specific decision problem under consideration. For example, if the problem concerns the location of a nuclear power plant and the alternatives correspond to physical sites, then the aggregator cannot reasonably be defined as a mean – with weights $\{{\pi }_1,\dots ,{\pi }_p\}$ – of the alternatives $A_k^{*}$, since the resulting alternative $A^{*}$ might fail to belong to the set $\{A_1^{*},\dots ,A_p^{*}\}$. In such a case, a more appropriate aggregator could be one that minimises the distance between the aggregated optimum $A^{*}$ and the set of alternatives $\{A_1^{*},\dots ,A_p^{*}\}$.

Finally, it is possible to extend the MCDM and MCGDM using fuzzy logic. For instance, fuzzy sets can be employed for the scores $a_{ij}$ of the decision matrix and/or the weights $w_j$ of the criteria. For MCGDM, the weights $p_k$ of the experts can also be handled with this methodology. In this case, the frameworks are referred to as Fuzzy MCDM and Fuzzy MCGDM, respectively.

4. Method

In this section, we present our method for determining, in an MCGDM framework with p experts, the weights ${\pi }_k$ associated with the experts $E_k$, for $k=1,\dots ,p$.

First, we provide a brief overview of the method proposed in [10].

4.1. Overview

Firstly, a concise overview of the methodology employed in [10] will be provided. The referenced study proposed a preliminary framework for the weighting of artificial experts in the context of MCGDM or Fuzzy MCGDM, utilising a Larsen-like FRBS architecture. The fundamental proposition underlying this study was that, in scenarios where artificial users are utilised in lieu of or in conjunction with human experts, their contributions should not be aggregated indiscriminately. Instead, it is recommended that each expert (artificial and human) be assigned a weight that reflects properties relevant to its anticipated behaviour within the decision process. This perspective was motivated by the observation that no ranking method is uniformly preferable across all decision settings and that the performance of a given method may vary substantially according to the structure of the problem, the criteria considered, and the stability of the rankings it produces [50].

Within that framework, the weighting problem was reformulated as a fuzzy inference task. Artificial experts were described through a collection of linguistic variables representing salient characteristics, e.g., their precision. These properties were modelled by means of triangular membership functions and combined through an interpretable rule base of IF–THEN rules in order to derive a crisp (or fuzzy) weight ${\pi }_k$ for each expert. The resulting output could then be defuzzified and normalised, thus making it directly usable within standard MCGDM procedures, while still preserving the possibility of a fully fuzzy treatment whenever required by the application, e.g., in [49,69].

Consequently, the preceding contribution established a structured and extensible basis for the evaluation of artificial experts. The primary value of the study was not limited to the specific example examined. It demonstrated that the allocation of expert weights can be facilitated through a transparent, rule-based mechanism that is capable of incorporating heterogeneous criteria and qualitative assessments. In particular, the selection of linguistic variables and their fuzzy sets exhibited consistency with the artificial experts that were taken into account. Concurrently, the framework was deliberately proffered as an inaugural methodological step, thereby leaving the question of how such a weighting scheme might be specialised, refined, or operationalised in more targeted decision contexts unresolved.

The present study builds on this foundation, progressing from the general problem of weighting artificial experts to the development of a more focused methodological contribution. In this sense, the earlier framework should be understood as the conceptual and inferential substrate upon which the current proposal is constructed. It provides the rationale for differentiating among artificial decision-makers, the formal language through which such differentiation can be expressed, and the methodological bridge between qualitative expert descriptors and quantitatively deployable weights in subsequent group decision-making models.

4.2. Methodology

In this section, we describe the extension of the procedure adopted in [10] to produce either a crisp or a fuzzy weight for a given decision domain.

Crucially, the linguistic variables and rule base are designed independently for each application domain, reflecting the relevant properties of the experts to be weighted.

For each domain under consideration, the method follows a structured pipeline composed of four main phases.

First, we define the domain-specific linguistic variables. Let $(X_1,\dots ,X_l)$ denote the l antecedent variables selected to characterise the experts in the considered domain. Each variable is defined on a universe of discourse that is appropriate for the application context and is partitioned into fuzzy sets, so as to capture the linguistic assessments relevant to that domain. In addition, we introduce the consequent variable, denoted by Weight, whose universe is fixed to the interval $[0,10]$.

Second, we formulate the domain-specific fuzzy IF–THEN rules. These rules encode the relationship between the antecedent variables and the resulting expert weight. The antecedent part of each rule may combine linguistic conditions through the logical connectives AND and OR. In particular, conjunction is modelled by means of the some t-norm ⊗, whereas disjunction is represented through some t-conorm ⊕, as those given in Table 1. The consequent part of each rule assigns the output to an appropriate Weight class.

Third, once the rule base has been defined, we perform inference and defuzzification. For each expert $E_k$, described by the crisp profile $(x_1^{\left(k\right)},\dots ,x_l^{\left(k\right)})$, where $x_i^k\in U\left(X_i\right)$, i.e., $x_i^k$ belongs to the Universe of Discourse of $X_i$, the input values are first fuzzified with respect to the previously defined antecedent fuzzy partitions. The firing strength of each rule is then computed according to the membership degrees of the inputs and the adopted fuzzy connectives (AND/OR). Next, the implication is applied to the consequent fuzzy sets, obtaining the corresponding firing strengths. The resulting outputs of all activated rules are subsequently aggregated. Finally, for each expert, a crisp weight $w_k$ is obtained through defuzzification of the Weight using one of the classic defuzzification methods, such as the Centre of Gravity (CoG) or the Mean of Maximum (MOM).

Fourth, the obtained weights are normalised in order to derive relative expert weights $({\pi }_1,\dots ,{\pi }_k)$. Specifically, for each expert $E_k$, the normalised weight is computed as

$${\pi }_k={\displaystyle \frac{w_k}{{\sum }_{i=1}^p{w}_p}}.$$

(6)

This final step ensures that the resulting weights are expressed on a comparable scale and can be directly interpreted as relative contributions within the set of experts.

4.2.1. Antecedents Memberships

In this section, we present the procedure adopted to model the antecedent variables by means of triangular and trapezoidal fuzzy numbers.

More specifically, the antecedent variables are derived from empirical results associated with a given metric M, such as precision@10. In this case, all results are available, and it is therefore possible to compute, for each metric under consideration, the pair $(\mu ,\sigma )$, where $\mu$ denotes the mean and $\sigma$ the standard deviation. Assuming that the metric values are approximately Gaussian, a natural mapping from the metric M to a corresponding fuzzy number.

In a three-partition fuzzy-set scheme, that is, whenever a linguistic variable is described by the standard family of three fuzzy sets Low, Medium, and High, let M be a metric and let $\mathbf{x}=(x_1,\dots ,x_n)$ denote the vector of observed values of M over n rounds. Let $\overline{x}$ be the mean of $\mathbf{x}$, and let $\sigma$ be its standard deviation. Moreover, let $\mathcal{N}(\overline{x},{\sigma }^2)$ denote the Gaussian distribution with mean $\overline{x}$ and standard deviation $\sigma$. Then, a suitable approximation of Low, Medium, and High by means of triangular fuzzy numbers is

$$\begin{array}{cc}\hfill \mathtt{Low}& =\tilde{t}\left(\overline{x}-3\sigma ,\overline{x}-3\sigma ,\overline{x}\right),\hfill \\ \hfill \mathtt{Medium}& =\tilde{t}\left(\overline{x}-3\sigma ,\overline{x},\overline{x}+3\sigma \right),\hfill \\ \hfill \mathtt{High}& =\tilde{t}\left(\overline{x},\overline{x}+3\sigma ,\overline{x}+3\sigma \right).\hfill \end{array}$$

A similar approximation can be defined using trapezoidal fuzzy numbers:

$$\begin{array}{cc}\hfill \mathtt{Low}& =\widetilde{t_p}\left(\overline{x}-3\sigma ,\overline{x}-3\sigma ,\overline{x}-2\sigma ,\overline{x}-{\displaystyle \frac{\sigma }{2}}\right),\hfill \\ \hfill \mathtt{Medium}& =\widetilde{t_p}\left(\overline{x}-2\sigma ,\overline{x}-{\displaystyle \frac{\sigma }{2}},\overline{x}+{\displaystyle \frac{\sigma }{2}},\overline{x}+2\sigma \right),\hfill \\ \hfill \mathtt{High}& =\widetilde{t_p}\left(\overline{x}+{\displaystyle \frac{\sigma }{2}},\overline{x}+2\sigma ,\overline{x}+3\sigma ,\overline{x}+3\sigma \right).\hfill \end{array}$$

It is worth noting that the above partitions define a Ruspini [70] partition such that

$$\sum _{x\in \mathbf{x}}{\mu }_{\mathtt{Low}}\left(x\right)+{\mu }_{\mathtt{Medium}}\left(x\right)+{\mu }_{\mathtt{High}}\left(x\right)=1.$$

In particular, the universe of discourse of both fuzzy representations is defined over the interval from $\overline{x}-3\sigma$ to $\overline{x}+3\sigma$, which corresponds to approximately $99.7\%$ of the area under $\mathcal{N}(\overline{x},{\sigma }^2)$. For trapezoidal fuzzy numbers, the plateau is taken as the interval from $\overline{x}-{\displaystyle \frac{\sigma }{2}}$ to $\overline{x}+{\displaystyle \frac{\sigma }{2}}$, which corresponds to approximately the $38.3\%$ of the area under $\mathcal{N}(\overline{x},{\sigma }^2)$. This choice yields a simple and interpretable fuzzy partition of the observed range of the metric into three overlapping linguistic regions. In Figure 2, we provide a graphical representation of both approximations.

5. Applications

In this section, we present our experiments using a FRBS for weighting four LLMs:

• apertus:8b [71],
• gemma4:e4b [72],
• mistral-small3.2:24b [73],
• nemotron-cascade-2:30b [74].

5.1. Dataset Introduction

The dataset has been created starting from the SDF dataset. Each LLM has been asked to answer 100 questions, and for every LLM, this process has been repeated 10 times. From these repetitions, the quantities $\overline{x}$ and $\sigma$ have been calculated for each language, where $\overline{x}$ denotes the average number of correct answers and $\sigma$ denotes the corresponding standard deviation. In this task, allowing a tolerance of $\pm 10\%$ is reasonable because the answers are numerical and small deviations from the target value should not be treated as fully incorrect. For this reason, the dataset used in the following examples is built from the measurements obtained under this tolerance level. These values are later transformed into trapezoidal fuzzy numbers (as defined in Section 4.2.1).

In Table 3 the multilingual results as $\overline{x}\pm \sigma$ is shown, where $\overline{x}$ is the average number of correct answers among the 10 runs, and $\sigma$ is the standard deviation of these runs. In Table 4, we provide the resource-side criteria, namely the VRAM requirement and the qualitative open-source score adopted in the methodology.

The dataset already suggests the qualitative trend later captured by the fuzzy method (see Figure 3 and Figure 4): nemotron-cascade-2:30b is strongest on multilingual quality, mistral-small3.2:24b remains competitive on the language dimensions, while apertus:8b and gemma4:e4b are more favourable on the resource side.

The VRAM score was assigned on the basis of the VRAM allocation size, expressed in kMiB, on a GPU equipped with 32 GB of VRAM. It was verified that RAM offloading was negligible and therefore did not materially affect the assessment.

By contrast, the Open-Sourceness score was determined from the following considerations. The model apertus:8b is both open-weight and open-source, in the sense that the entire training pipeline is openly available. For this reason, it was assigned a score of 10. The LLMs nemotron-cascade-2:30b and gemma4:e4b provide open weights and are available for commercial use; accordingly, each was assigned a score of $7.5$. Moreover, these models have consistently remained available for commercial use over time.

The case of mistral-small3.2:24b is more nuanced. Although the model was released under an open-weight licence, specifically Apache 2.0, its commercial use was initially subject to restrictions. While its weights have only recently become available for commercial use, this evolving licensing regime may constitute a significant source of uncertainty for anyone intending to build a business upon this LLM. For this reason, it was assigned a score of 5.

5.2. Fuzzy Inference System

The fuzzy inference scheme takes five linguistic variables into consideration for the input and one for the output. In particular, the input variables are English, Italian, Portuguese, VRAM, and Open-Sourceness, while the output variable is Weight. The inputs VRAM and Open-Sourceness are resource-side criteria.

For each input variable, we defined three trapezoidal fuzzy numbers Low, Medium, and High. In particular, the linguistic input variables English, Italian, and Portuguese are represented with the trapezoidal parameters given Table 5 (see Figure 5 for a graphical representation). On the other hand, the resource-side criteria VRAM and Open-Sourceness are represented using the trapezoidal parameters given in Table 6 (see Figure 6 for a graphical representation).

In particular, VRAM is modelled to reflect the idea of what is considered acceptable in relation to the price, based on the GPU’s capacity. On the other hand, Open-Sourceness is described through its qualitative score range, which is sufficient for a smooth separation between low, intermediate, and high openness.

LLM openness is best understood as a multidimensional property rather than a binary distinction between open source and closed models. Relevant dimensions include the availability of model weights, which permits local execution but does not necessarily imply reproducibility; the release of source code, training pipelines, and, in stronger cases, training data under an OSI-compatible license; disclosure or publication of pretraining and fine-tuning corpora, enabling auditability and replication; documentation of the model architecture and design choices through papers or technical reports; and the legal permissions granted by the model license, including rights to commercial use, modification, and redistribution. Additional axes include open access through public APIs, which provides usability without revealing weights or internal implementation; open evaluation, in which benchmark results, safety assessments, and testing protocols are published; and open fine-tuning or customisation, which allows users to adapt the model through methods such as LoRA or full parameter fine-tuning. Under the gradient of openness perspective, these dimensions are independent: a system may be open along some axes while remaining closed along others. For example, some models release weights but restrict licensing and withhold training data, whereas more fully open systems release weights, code, data documentation, and evaluation artefacts; proprietary API-only systems, by contrast, generally provide access without substantive openness of weights, data, or training methodology. In assigning this score has been taken in account also the stability over time and the clearness of license assigned to each model, as concluding factor of trustfulness of the model.

Finally, the output variable Weight is partitioned into five fuzzy numbers Very Low, Low, Medium, High, and Very High (see Table 7 and Figure 7). This latter partition is denser than the input ones, which is useful at the aggregation stage because it provides a more expressive scale for translating rule consequents into final scores.

For the inference part, we consider the probabilistic t-norm ${\otimes }_P$ and t-conorm ${\oplus }_P$ for the interpretation of the logical operation AND, and OR, respectively.

5.3. First Example

As a first working example, for our inference system, we consider the following rule base:

$R_1$	IF Italian is High AND Portuguese is High THEN Weight is Very High
$R_2$	IF Italian is Medium AND Portuguese is High THEN Weight is High
$R_3$	IF Italian is Low AND Portuguese is Low AND English is High THEN Weight is Medium
$R_4$	IF Italian is Low AND Portuguese is High THEN Weight is Very Low
$R_5$	IF Italian is High AND Portuguese is Low THEN Weight is Very Low

This first example uses only linguistic inputs to assign a final weight. The idea is simple: strong competence in Italian and Portuguese is valued because both languages reflect a solid background in Latin. English is also useful, but, in this case, it plays a secondary role and can only partially compensate for weaker performance in the other two languages.

Each rule reflects a specific preference. When both Italian and Portuguese are High, the candidate receives a Very High weight because strong performance in both Latin languages is especially desirable. If Portuguese remains high while Italian is only Medium, the result is still High, showing that Portuguese carries slightly greater importance. When both Latin-language scores are Low, a high English score can still recover part of the evaluation, leading to a Medium weight. By contrast, a strong imbalance between Italian and Portuguese leads to a Very Low weight, because the model penalises inconsistent competence across the two key languages.

In Table 8, we show the final ranking for this first example, where the normalised score is the $L_1$ normalisation of the raw score. In this particular case study, nemotron-cascade-2:30b stands out clearly from the other candidates, with the highest raw score and a much higher normalised value. mistral-small3.2:24b follows at a moderate distance, while gemma4:e4b and apertus:8b receive substantially lower scores. This distribution shows that, under the language-focused rules of the first scenario, the fuzzy system is able to identify a strong preference for models that better match the desired Latin language profile.

5.4. Second Example

In this second working example, we consider the following rule base:

$R_1$	IF Italian is Medium AND Portuguese is Medium AND English is Medium AND VRAM is Low THEN Weight is High
$R_2$	IF Italian is Medium AND Portuguese is Medium AND English is Medium AND Open-Sourceness is High THEN Weight is Very High
$R_3$	IF Italian is High AND Portuguese is Medium AND English is Medium THEN Weight is High
$R_4$	IF Italian is Low AND Portuguese is Medium AND English is Medium AND VRAM is High AND Open-Sourceness is Low THEN Weight is Very Low
$R_5$	IF English is High AND Open-Sourceness is Medium AND VRAM is Medium THEN Weight is Low

This second example extends the evaluation by including resource-related criteria in addition to language skills. Alongside Italian, Portuguese, and English, the model now also considers available VRAM and the degree of commitment to open-source software. This allows the final weight to reflect not only linguistic suitability, but also practical and ethical constraints.

Each rule highlights a different balance between these factors. When all three languages are at a Medium level, and VRAM is Low, the weight is still High because limited hardware resources make efficiency especially valuable. If the language profile remains balanced and Open-Sourceness is High, the weight becomes Very High, reflecting a strong preference for principled technical choices. A strong score in Italian, combined with Medium values in Portuguese and English, also produces a High weight because Italian is given special importance in this example. On the other hand, if Italian is Low, VRAM is High, and weak open-source alignment leads to a Very Low weight, since the candidate performs poorly on several key criteria. Finally, strong English alone is not enough to compensate for only Medium technical and ethical values, so that case receives a Low weight.

Table 9 summarises the final ranking produced in the second scenario, where the normalised score is again the $L_1$ normalisation of the row score. The results suggest that gemma4:e4b is the most suitable model under these preferences, with a clear lead in both raw and normalised score. The remaining models form a closer group, with apertus:8b, mistral-small3.2:24b, and nemotron-cascade-2:30b receiving lower but still comparable evaluations. Overall, the table shows how the fuzzy rule base can separate candidates according to the combined influence of language quality, resource efficiency, and open-source alignment.

6. Conclusions and Future Works

This work has examined the use of a Fuzzy Rule-Based System for assigning weights to experts within a Multi-Criteria Group Decision-Making framework, with particular attention to settings in which artificial experts contribute alongside, or in place of, human decision-makers. Building on the preliminary methodological proposal introduced in [10], the present contribution has focused on an application-oriented validation of the approach, showing how an interpretable fuzzy weighting mechanism can be adapted to a concrete decision domain and how its behaviour changes under different modelling choices.

The results obtained from the LLM case study confirm that the proposed framework is sufficiently flexible to encode heterogeneous weighting policies through transparent IF–THEN rules. In particular, the experiments have shown that the final weights are not determined by performance alone, but may be shaped in a controlled and auditable manner by combining multilingual quality, computational requirements, and openness-related characteristics. This is precisely one of the main advantages of the FRBS perspective: rather than imposing a single optimisation principle, it allows the weighting scheme to reflect the priorities of the decision context. In this sense, the method appears well-suited to real-world scenarios, where decision quality is rarely reducible to one isolated dimension.

The empirical examples also highlight an important substantive point. Small changes in the rule base, or in the logical structure of the antecedents, may produce substantial changes in the final ranking. This should not be viewed as a weakness of the method, but rather as evidence of its sensitivity to explicit modelling assumptions. Hence, the framework makes the value judgements underlying the weighting process visible, instead of leaving them implicit in a black-box optimisation step. From the perspective of MCGDM, this is a significant methodological advantage, especially when artificial experts are involved, and accountability is required.

At the same time, the study has shown that some modelling components are particularly influential. Among them, the design of the rule base is the most decisive, since it directly encodes the weighting policy adopted in the application. The choice of membership functions and the partition of the linguistic variables also affect the outcome, especially in borderline cases where an expert lies near the overlap of adjacent fuzzy sets. Likewise, the selection of logical operators and the adopted defuzzification procedure may alter the relative magnitude of the final scores. For this reason, the proposed framework should not be interpreted as a fully automatic mechanism, but as a structured and interpretable tool whose reliability depends on a careful alignment between fuzzy modelling choices and domain-specific objectives.

From a practical standpoint, the proposed approach offers a viable way of integrating artificial experts into decision-support pipelines without treating them as interchangeable black boxes. By assigning weights through linguistic variables and interpretable rules, the framework supports a more nuanced form of aggregation in which the role of each expert can be justified explicitly. This is particularly relevant in contemporary decision environments, where algorithmic agents, recommender systems, predictive models, and large language models increasingly act as decision contributors rather than as mere auxiliary tools.

The present study nevertheless has some limitations. First, the empirical validation has been conducted on a specific case study involving four LLMs and a selected set of criteria; therefore, the conclusions should not be overgeneralised without further testing on broader classes of decision problems. Secondly, the construction of the fuzzy partitions and the rule base remains application-dependent and requires domain knowledge. Thirdly, although the examples considered here clearly illustrate the interpretability and flexibility of the method, a broader benchmarking analysis against alternative expert-weighting strategies would be necessary in order to assess its comparative performance more systematically.

Several lines of future research follow naturally from these observations. A first direction concerns the extension of the experimental analysis to additional application domains, including more traditional MCGDM settings and larger groups of heterogeneous experts. A second direction is the development of a more systematic sensitivity analysis covering membership-function design, logical connectives, inference schemes, and defuzzification operators. A third possible development is the introduction of semi-automatic or data-assisted procedures for constructing the rule base, so as to reduce the burden of manual design while preserving interpretability. Finally, it would be worthwhile to investigate hybrid settings in which FRBS-based weighting is combined with empirical validation, calibration procedures, or benchmark-based evaluation protocols for artificial experts.

In conclusion, the main contribution of this paper lies not in proposing yet another abstract weighting formula, but in showing that expert weighting can be formulated as an interpretable fuzzy inference problem and meaningfully instantiated in an application-oriented setting. The study, therefore, strengthens the case for FRBS-based weighting as a transparent and operationally credible approach for MCGDM, especially in scenarios where human and artificial expertise must be combined under multiple, and potentially competing, criteria.