Uniting Psychometric Modelling and Poisson Distributions: A Metrological Study of Elementary Counting

1. Introduction

Probability mass functions (PMFs) (Section 2.1) provide a means of defining — for discrete and qualitative data — the basic metrics, viz. location (Equation (5)) and dispersion (Equation (6)), of metrology when assuring the quality of measurement in terms of traceability and uncertainty. Quality-assurance of those metrics, following satisfaction of constraints concerning statistical sufficiency and additivity, in turn supports, respectively, interoperability and defining limits on the quality of products and services of all kinds [1,2,3].

Deploying PMFs for discrete and qualitative data thereby complements the dominating role for these location and dispersion metrics traditionally played by PDFs, the corresponding probability density functions, in quantitative and continuous Metrology in Physics.

Location and dispersion metrics for discrete and qualitative data are increasingly required since the turn of the millennium in topical and challenging quality-assurance applications, amongst others, in the human sciences [4], sustainability [5] and in Artificial Intelligence [6].

The metrology of PMFs remains so far relatively undeveloped, and the present work presents a reanalysis of a study of elementary counting (of simple clouds of dots by the Munduruku indigenous people of Brazil, [7,8]) as a means of comparing and contrasting the metrology of three kinds of probability mass functions (PMFs) for the four scenarios shown in Table 1: (i) quantitative response to a discrete range of counts (Section 3.3), [9], (ii) the classic Poisson distribution of miscounts (Section 4.3.3), and (iii) psychometric (Rasch) distributions of counting task difficulty and person counting ability, (Section 5) for one of the conceptually simplest cases of measurement. The analyses illustrate the typical challenges normally faced when determining quality-assured metrics with the different kinds of PMF.

Figure 1. Analytical PMFs of distributions of perceived counting errors for three counts: (1 dot, 6 dots and 8 dots) in the study [7] of elementary counting tasks (’items’, j) of sets of clouds of increasing number of dots and task difficulty. Lines for each count PMF assume a Gaussian distribution of counting error, $p_x=N((\mu -{\mu }_0),{\sigma }^2$), $\mu =$ perceived count, Equation (5); $\sigma =1$ dot, perceived counting dispersion, Equation (6); ${\mu }_0=$true count.

Figure 1. Analytical PMFs of distributions of perceived counting errors for three counts: (1 dot, 6 dots and 8 dots) in the study [7] of elementary counting tasks (’items’, j) of sets of clouds of increasing number of dots and task difficulty. Lines for each count PMF assume a Gaussian distribution of counting error, $p_x=N((\mu -{\mu }_0),{\sigma }^2$), $\mu =$ perceived count, Equation (5); $\sigma =1$ dot, perceived counting dispersion, Equation (6); ${\mu }_0=$true count.

The Discussion (Section 6) provides the motives for choosing an elementary construct. Classification, such as counting dots, despite sometimes having relatively large uncertainties and being considered ’trivial’, is one of conceptually simplest metrological cases with a well-defined measurand, as will be illustrated. In a summary of this and other limitations in the present study, perspectives will also be given on future work including consideration of other measurement systems where a Human as a Measurement Instrument is replaced by for instance an AI agent and multidimensional measurement theories. This includes proper account of ’anti-racist’, ’culturally responsive’ and ’culturally specific’ measurement applications being developed, for example, by Mallinson [10] and Sul [11,12,13].

While the metrology of the 20th Century was dominated by measurement uncertainty studies in the bottom-right entry in Table 1, based on ’analytical’ measurement of physical properties in trade and industry [14], the first quarter century of the millennium has seen an emergence [3] of metrology in all the other entries, discrete and (for want of a better term) ’clinical’ measurements, Section 3.2 . This reflects a growing trend towards more accountability in terms of meaning, value and effectiveness [15], and not mere numbers and efficiency.

2. PMFs and Measurement

Examples of PMFs abound in the human sciences, education, sustainability and the health sciences (in surveys, sensory panels and educational examinations) and by extension increasingly in the area of Artificial Intelligence as well as other applications outside of traditional physical metrology [3]. Models used when quality-assuring PMFs, particularly those based on qualitative observations, such as item response theory and log-odds Equation (17) in the educational, medical and social sciences, are not exclusive to human agents but apply in many cases equally well (Section 4.3.1) to more technical and physical measurement systems where they have so far not been regarded as belonging to mainstream Metrology.

2.1. Probability Mass Functions

The discrete counterparts of PDFs—probability mass functions (PMF)—are distributions of probability density (ordinate) over one or more discrete ranges of a variable (abscissa).

Let X be a discrete random variable with a range of C categories:

$$R_X=x_1,x_2,...,x_C$$

(1)

An event $A=X=x_c$ is defined as the set of outcomes, s, in the sample space,S, for which the corresponding value of X is equal to $x_c$ for a response in category c. In particular, the event:

$$A=s\in S|X\left(s\right)=x_c$$

(2)

The probabilities of events $X=x_c$ are formally shown by the probability mass function (PMF) of X:

$$P_X\left(x\right)=\left\{\begin{array}{cc}P(X=x_c)& ifx\in R_X\\ 0& ifx\notin R_X\end{array}\right\}$$

(3)

for the range $R_X=x_1,x_2,...$(finite or countably infinite).

The probabilities for the different categories, c, $P(X=x_c)=N_c/N_S$, that is, the occupancy, $N_c$, of a bin for category c as a fraction (or corresponding probability) of the sample size, $N_S$, must be non-negative and for PMFs are normalised to sum up to 1, respectively:

$$P_X\left(x\right)\ge 0$$

and

$$\sum _x{P}_X\left(x\right)=1$$

(4)

In the examples referred to in this paper, mention will be made of:

• Probabilities
  The probabilities can be either derived from actual frequencies (occupancy per bin) or estimated in terms of prior knowledge or even judgment when scores are set. Figure 2 shows several Poisson PMFs (Section 4.2) as an example. When setting response scores as probabilities P, choices have to be made about how to define the scale-end scores of 0 and 100. Inferentially stable measurement also requires that the score varies monotonically across the scale in a manner allowing summarization which exhausts the data of all available information in a minimally sufficient statistic, [16,17,18]. (These requirements, [19], are subsequently checked (Section 5.5)).
• Ranges

  -

  intrinsically discrete ranges,$R_X$ (Equation (1)) (such as when counting dots (Section 3.3)).

  -

  discrete ranges chosen for convenience (such as in sensory panel responses) where the observed variable is on a continuous scale, but the large uncertainties make it practical to round off the score, say to the nearest integer.

  -

  ranges can be as short as two, such as for the binomial distribution from binary Bernoulli trials (Section 4.1) as well as long, polytomous ranges spanning several categories (Section 4.3.2).

• Scales
  In different application areas, the scales associated with both X (abscissa axis) and P (ordinate axis) of a PMF can be:

  -

  either fully quantitative (Section 3.3),

  -

  or more qualitative (Section 5)

  ranging, respectively, from ratio and interval scales to the ordinal and nominal scales.

There is no obvious one-to-one correlation between discrete and continuous ranges versus qualitative and quantitative scales [20]. Table 1 shows a matrix of possible correlations.

2.2. Accuracy of Classification

The accuracy of classification, in terms of both precision and trueness ISO 5725 [21], involves, respectively, differences between observed and ’correct’ in terms of the location, Equation (5) and the dispersion of PMFs, Equation (6) [2] (Section 4.1.1): where (provided distances between different categories are known, Section 4.1.2)

$${\widehat{p}}_x=\sum _{x}x·p_x;$$

(5)

where $p_x=P(X=x_c)$.

$$\sigma \left({\widehat{p}}_x\right)={\displaystyle \frac{1}{C-1}}\sqrt{\sum _x\left({(x·p_x-{\widehat{p}}_x)}^2\right)}.$$

(6)

Here, trueness is a measure of closeness of the mean to the ’true’ value while precision is the dispersion of repeated responses (without reference to a ’true’ value).

The concept of Accuracy can be considered for both analytic and clinical performance metrics (Section 3.2):

• analytical accuracy.
  In an ’analytical’ scenario 1, the ’accuracy’ estimation of the ’measurand’, the quantitative number of each set of discrete objects (with the number of dots increasing from 1 to 10 in the present counting case), can be expressed in terms of:

  -

  (i) trueness—estimated as the difference between the perceived and true dot count, $x-\mu$.

  -

  (ii) precision—estimated in terms of dispersion, such as $\sigma$ dots, (Figure 1).

• clinical accuracy
  In a ’clinical’ scenario 2, distances between different categories of classification on the abscissa of a PMF (such as when measuring the difference before or after an intervention, or when estimating dispersion measures) may not be fully known or even meaningful (such as on nominal scales (Section 4.1)). Appropriate methods for dealing with such scales include log-odds ratio transformations, including GLMM and the Rasch psychometric theory with which ’measurands’ such as counter ability and task difficulty can be identified, as dealt with later in the paper in our clinical counting example (Section 5).

Alongside concepts about PMFs found in the wider literature, such as so-called ’measure theoretic formulations’, a proper metrological approach describes how PMFs should be associated with various stages in the measurement process (Section 3.1).

3. Analytical and Clinical Performance Metrics

3.1. PMF and Measurement System Analysis (MSA)

PMFs are a common tool of statistics, but here there are also metrological aspects. Uncertainty in counting, for instance, arises both from the identification of what is being counted and from the experimental counting procedure [22].

A proper metrological approach describes how PMFs should be associated at various stages in the measurement process, corresponding to the propagation stage of establishing a measurement result. This means consideration of how measurement information is ’transmitted’ (as in any communication system) through a measurement system [object – instrument – observer – environment and method, ([24], Figure 14) and Figure 3, ([2] p.198)] with a particular focus on an object as ’probed’ with an instrument of some kind. This MSA analysis will guide the formulation and propagation stages for PMFs in establishing a measurement result from initial observations through to restitution of the measurand(s) [23]. For the present case of measurement systems involving discrete PMFs, inputs, such as $X_1$, when restituting the measurands, can be characterised in terms of informational entropy ([2] Figure 5.4), assigned on the basis of available knowledge (Section 5.3).

3.2. Analytical and Clinical Performance Criteria

Different attributes associated with various elements (MSA object – instrument – observer – environment and method) of the measurement system (Section 3.1) can be analysed in terms of PMFs. For each MSA element attribute, two principal kinds of performance criteria — ’analytical’ and ’clinical’ — can be identified, which Hofmann [25] places as essential intermediaries between, on the one hand, epistemic relevance and, on the other, meaningfulness (ultimately for patient health). This can be topically exemplified in the context of regulation and conformity assessment of in vitro devices, specified in EU regulations [26,27], although equally applicable to our elementary case of counting dots (Figure 4):

1.

’Analytical’ performance criteria for determining, e.g., how much (quality characteristic: concentration) of a particular analyte (MSA object) is present in a sampled object (by "variable"), such as, first, analytical method accuracy (trueness and precision, [21]) (Section 2.2) and, second, sensitivity, such as instrument limit of detection (MSA measurement instrument), as exemplified in the analytical interpretation of the elementary counting case of the present study, Section 3.3. According to [26], "…Analytical performance focuses on the gathering of evidence that the measurement instrument in question reliably, accurately and consistently measures and or detects an analyte". This is closely related to terminology in acceptance sampling standards [28], §3.1 where Inspection by variables is inspection by measuring the magnitude(s) of a characteristic(s) of an item.

2.

’Clinical’ performance , according to [26], "aims to demonstrate that the measurement instrument can achieve clinically relevant outputs through predictable and reliable use by the intended users". This is closely related to terminology in the definition §3.1.3 of [29], where Inspection by attribute is "inspection whereby either the item is classified simply as conforming or nonconforming with respect to a specified requirement or set of specified requirements, or the number of nonconformities in the item is counted". Commonly used clinical performance metrics of the measurement systems include: selectivity, Equation (22) and specificity, Equation (23), which are plotted against each other on receiver operating characteristic curves, [30], when sampling by ’attribute’. A psychometric treatment of these clinical performance metrics [31,32] yields quantitative estimates for quality characteristics such as task difficulty (MSA object) and agent ability (MSA measurement instrument) as attributes of the different elements of the measurement system illustrated in Figure 3, corresponding to the top-right entry in Table 1.

3.3. PMFs for Counting and Related Tasks. ’Analytical’ and ’Sampling by Variable’

The elementary counting of dots by less numerate agents (e.g., children or indigenous people [7,8], Figure 4) presents a case of the metrology of discrete counting as a typical ’analytical’ or ’sampling by variable’ procedure, point Section 1 and bottom-left entry in Table 1.

This study of elementary counting has several advantages, including the opportunity of analysing the measurement response where the analytical measurand (quantity intended to be measured) — the number of dots ($1...10$) — is:

• (i) in fact known exactly
• (ii) conceptually simple

In fact, this elementary case can exemplify each of the elements of Table 1, as will be shown throughout this paper.

As the number of dots to be counted exceeds about 5 (the number of digits on one hand traditionally used by the agents for counting), each estimated count becomes successively more under-estimated [7], Figure 4. Similar observations of limitations in counting ability can be found in children and educated adults when they are given only a limited time to perform each counting task. An initial ’analytical’ scenario 1 is the estimation of the measurand: the quantitative number of a set of discrete objects (with the number of dots increasing from 1 to 10) in the data shown in Figure 4. A count of 5 dots, where perceived counts deviate substantially from the known, true number of dots, could be considered a ’limit of detection’ as a analytical performance metric.

These successive count under-estimations can be described in terms of PMFs distributed over an increasing number of counting categories and shifted towards successively underestimated values, shown in Figure 1. Analytical PMFs of distributions of perceived counting errors can be plotted, for example, for three counts: (1 dot, 6 dots and 8 dots) in the study [7] of elementary counting tasks (’items’, j) of sets of clouds of increasing number of dots and task difficulty. The lines drawn for each count PMF assume a Gaussian distribution of counting error, $p_x=N((\mu -{\mu }_0),{\sigma }^2$), $\mu =$ perceived count, Equation (5); $\sigma =1$ dot, perceived counting dispersion, Equation (6); ${\mu }_0=$true count.

4. Case Studies of PMFs: Quantitative Statistical Process Control

PMFs have been invoked for many decades for a number of well-known sampling distributions in the context of trials for statistical process control (SPC, [33]) as one major area of application. In general, the parameters of a production process are unknown and can change with time. Procedures to estimate the parameters of probability distributions and solve other inference or decision-oriented problems related to them need to be developed. In this section, we will motivate how SPC PMFs can be deployed when tackling the metrology of clinical performance 2, with a particular focus on the elementary counting example, Figure 4.

4.1. Binomial Distribution and Dichotomous Bernoulli Trials in SPC

A basic kind of PMF is the dichotomousBernoulli variant shown in Figure 5. For an elementary counting case, a counter may initially estimate the number of dots as one or other of two adjacent integers, say ’9’ or ’10’ when counting a cloud of ten dots, corresponding to the top-left entry in Table 1.

For a dichotomous [35] ’trial’ , where a ’success’ is scored 1 and a ’failure’, 0, let the probability of categorising a ’successful’ response be p. Values on the ordinate (vertical) axis of the binomial PMF shown in Figure 5 are probability density values:

$$P_X\left(x\right)=\left\{\begin{array}{cc}P(X=x_c)={\displaystyle \frac{coun{t}_c}{N_{sample}}}={\displaystyle \frac{{\sum }_{i=1}^{N_{sample}}[z_i=c]}{N_{sample}}}& ifx\in R_X\\ 0& ifx\notin R_X\end{array}\right\}$$

(7)

On a quantitative scale on the abscissa, the PMF plotted in Figure 5 (based on Monte Carlo (MC) simulations of sampling trials [34]) has the following means and standard deviations: $\widehat{p}=0.5;\sigma \left(\widehat{p}\right)=0.28$, while equations (5) and (6) yield:

$$\widehat{p}=\sum _{c=0}^{c=1}c·p_c=0.5$$

(8)

$$\sigma \left(\widehat{p}\right)={\displaystyle \frac{\sqrt{{\sum }_{c=0}^{c=1}{(c·p_c-\widehat{p})}^2}}{C-1}}=0.3.$$

(9)

The binomial distribution is a limiting form of the Poisson distribution (Section 4.2) for an infinite number of Bernoulli trials and an infinitely small probability of non-conforming product [33].

4.1.1. Dichotomous Decision-Making with Uncertainty

A typical decision of conformity is whether the attribute of an entity (e.g., a product or sample) can be classified as above (‘positive’) or below (‘negative’) a specification limit, SL, for that quality characteristic. However, measurement uncertainty leads generally to the risk of incorrect decisions and classifications in such conformity assessments, as described in [14] JCGM 106:2012.

An observed response in a performance test is typically scored binarily (binomial PMF, Figure 5) in one of two categories on the abscissa: as $c=1$ for correct and $c=0$ for incorrect and presented on the PMF ordinate as $P_{success,i,j}$ for each instrument, i, to a specific object, j, when examined in terms of Measurement System Analysis (Section 3.1). This simplest binary classification can be extended to polytomous decision-making over a range of classification categories Section 4.2. In the elementary counting case, this would correspond to ’guessing’ the count to lie in a range of integer numbers of dots for a given cloud as an analytic measurand, as is evident in the results shown in Figure 1. A corresponding clinical performance study (2) would identify two measurands: the (i) ability of the counter and the (ii) difficulty of each counting task derived from the probability of correct classification, corresponding, respectively to the top-right and top-left entries in Table 1. In any case, the decision quandary reflects uncertainty.

4.1.2. Distances on Categorical Scales: Counted Fractions

Distances on categorical scales, such as on a binomial PMF, Figure 5, (but also PMFs for other distributions, such as the Poisson and psychometric) can be challenging to evaluate correctly in the case of hidden, unrecognised non-linearity or simply a non-numerical scale (3). This ’scale ordinality’ obliges relatively more emphasis to be placed on the PMF ordinate values compared with more quantitative cases with PDFs such as the analytical (1). Typical clinical performance (2) cases include assessing the effects of an intervention or the relative performance of two counters (Figure 1) in terms of distances between PMFs.

As defined in Equation (4), the probabilities for the different categories of any PMF, c, $P(X=x_c)$, must be non-negative and sum up to 1: $P_X\left(x\right)\ge 0$ and ${\sum }_x{P}_X\left(x\right)=1$.

A general feature of categorical data in which fractions on a finite scale are counted on any bounded scale is referred to as the ’counted fraction’ effect, [36]. Put simply, any score on a fixed scale on bounded by 0 and 100% becomes increasingly non-linear at either extreme of the scale. A difference of, for example, 1% percentage points in score $P_{success}$ of successful classification is a different amount at mid-scale (50%) compared with scores at 5% or 95%. This effect will be explained and compensated for as follows:

An early statement by Pearson [37]: “Beware of attempts to interpret correlations between ratios whose numerators and denominators contain common parts”.

The counted fraction effect has been explained in terms of the occurrence of the same term, $Z_j$, in the numerator and denominator of the score $Z_j=Z_j/\left({\sum }_{k=1}^K{Z}_k\right)$.

Tukey [36] wrote further: “As a general consequence we should expect that scales which have a finite range are likely to give us trouble unless all our observations tend to be safely away from any ends which are present. Hence the fact that percentages go only from one end (at 0%) to another (at 100%) suggests that, whenever even moderately extreme percentages are likely to occur, we are likely to have to ’stretch the tails’, while, if really extreme percentages occur, we may have to stretch hard enough so that there are no ends (at any finite values).”

Filzmoser [38] stated for basically the same concept in the context of mineral compositional data studies.: “If not all variables or components have been analysed, this constant sum property, (i.e., ${\sum }_x{P}_X\left(x\right)=1$ (Section 2.1)) makes the relation between variables not ’real’ but ’forced’…For example,if the concentrations of chemical elements are measured in soil samples, and if an element like SiO₂ has a big proportion of, say, 70%, then automatically the sum of the remaining element concentrations is at most 30%. Increasing values of SiO₂ automatically lead to decreasing values of the other elements, and even if not all elements of the soil have been measured, the correlations will be mainly driven by the constant sum constraint.”

Non-linearity in the raw response scale, such as arises from the counted fraction effect [39] can be readily revealed by plotting, as in Figure 6 and Figure 7, the marginal probabilities for the different items j (cloud of dots to be counted, for example), given by:${\displaystyle \frac{{\Sigma }_{i=1}^{N_{TP}}{y}_{i,j}}{N_{TP}}}$ against the more quantitative scales from Rasch Measurement Theory (RMT) [40] for the task difficulties, ${\delta }_j$, (logits), as will be described below.

Scale non-linearity caused by the counted-fraction effect is ubiquitous to all PMFs because of the limited percentage scale, Equation (4), but if not recognised (as in Classical Test Theory, CTT [41]) can lead to serious underestimation of true scores at either end of the response scale, as shown in Figure 6 and Figure 7, compared with the substantially linear region close to mid-scale, Equation (10) (Section 4.1.3).

4.1.3. Compensating for Counted-Fraction Scale Non-Linearity

There is a substantial and growing body of literature ([2], p. 88 - 9) for a diversity of applications, such as psychometry [40] and compositional data analysis [38,42] where the counted fraction effect is recognised. The ’counted fraction’ effect can be compensated for using a log-odds (GLMM) approach, Equation (11):

Ordinate values compensated for the counted fractions effect are given as a straight line $y=m·z+c$ as plotted in Figure 7 and Figure 6 where two statistics are known:

• the intercept $c=0.5$ at $z=0$
• the slope

  $$m={\displaystyle \frac{dp}{dz}}=p\left(z\right)·(1-p\left(z\right))=0.25$$

  (10)

  at $z=0,p=0.5$

These two parameters, which apply to all bounded-scale fractions, define the anchor (at $P_{success}=$ 50 %) and the span (the slope) of the linear, quantitative scale resulting from a log-odds transformation Equation 11, thus giving the corrections to raw performance data for the counted-fraction effect for any PMF.

Ordinality in responses is generally compensated for the effects of counted fractions by taking log-odds ratios (OR) [43] i.e.,

$$log\left(OR\right)=log\left({\displaystyle \frac{x_j}{1-x_j}}\right)=z$$

(11)

The RHS of Equation (11) is referred to as a ’link function’ in the literature in the Generalised Linear Measurement Models (GLMM) approach [44] and provides the means for common-person and common-item equating in psychometrics, following Wright’s expansions of Rasch’s models ([45] pp. 109-117). For current application, see [46].

4.2. Poisson Distributions in SPC

The Poisson is another widely distribution used to model count data and random occurrences over time or space, such as arrivals in a queue, radioactive decay events or rare defects in SPC production, [47]. We will also examine (Section 4.3.3) how the Poisson distribution can be useful when defining clinical performance metrics for the elementary counting case, Figure 4.

Let $\lambda$ denote the expected number of events per interval, and assume $\lambda$ is known. Then, the distribution of X is Poisson with parameter $\lambda$:

$$X\sim \mathrm{Poisson}\left(\lambda \right).$$

The PMF for X is

$$p_X\left(x\right)=\left\{\begin{array}{cc}\hfill {\displaystyle \frac{{\lambda }^x{e}^{-\lambda }}{x!},}& x=0,1,2,\dots ,\hfill \\ \hfill 0,& \mathrm{otherwise}.\hfill \end{array}\right.$$

(12)

X is the quality characteristic, x, being classified (e.g., number of defects) in n observations.

X has expectation and variance ([33], p. 55).

$$E\left(X\right)=\lambda ,V\left(X\right)=\lambda .$$

(13)

The Poisson and binomial distributions are closely related: The Poisson distribution can be regarded as a limiting form of the binomial distribution (Section 4.1). Allowing the binomial parameters $n\to \infty$ and $p\to 0$ in such a way that $n·p=\lambda$ results in a Poisson distribution [33]. According to Meredith [48] the Poisson distribution: "also occurs as a limit distribution for a number of discrete distributions other than the binomial. Two of the more important instances of this are the negative binomial or Pascal distribution and the hypergeometric distribution."

As for the binomial (dichotomous Bernoulli trials) case (Section 4.1), values on the ordinate (vertical) axis of the Poisson PMF are probability density values for each x:

$$p_x=coun{t}_x/N_{sample}=\sum _{i=1}^{N_{sample}}[z_i=x]/N_{sample}$$

(14)

On a quantitative scale on the abscissa, each Poisson PMF — plotted in Figure 2 for the elementary counting case — has the mean and standard deviation given by:

$${\widehat{p}}_x=\sum _{x=0}^{x=X}x·p_x;$$

(15)

$$\sigma \left({\widehat{p}}_x\right)=1/(X-1)·\sqrt{\sum _{x=0}^{x=X}\left({(x·p_x-{\widehat{p}}_x)}^2\right)}.$$

(16)

4.3. Rasch Psychometric Model. Principle of Specific Objectivity

The output of the uncertainty model (Section 3.1) will ultimately provide (through ’restitution’), from an MSA perspective, Figure 3, our best estimates of the instrument attribute (typically an ’ability’ of a rater to make a ’correct’ classification), alongside the object attribute which together constitute a ’conjugate’ pair of measurands. This procedure for clinical properties involving ordinal (and nominal) PMFs, has the same role as the corresponding procedure for data on fully quantitative scales, when calibrating the sensitivity of, for instance, a weighing machine in order to metrologically determine the mass of an unknown weight. Providing separate estimates of measurement system attributes from the analysis of PMFs on the less quantitative scales will need to deal with possible scale non-linearity, Figure 7. Again, a psychometric model will be useful when defining clinical performance metrics for the elementary counting case, Figure 4 and corresponding to the top-right entry in Table 1.

Amongst a family of log-odds transformation models (through restitution from the ordinal or nominal response scores, $Y=P_{success}$, of the measurement system), a particularly metrological approach is the psychometric [40] modern measurement (dichotomous) theory [49,50], as a special case of the log-odds, Equation (11):

$$y_{i,j}=P_{success,i,j}={\displaystyle \frac{e^{({\theta }_i-{\delta }_j)}}{1+e^{({\theta }_i-{\delta }_j)}}}$$

(17)

or in the form of the log-odds:

$$ln({\displaystyle \frac{P_{success,i,j}}{(1-P_{success,i,j})}}={\theta }_i-{\delta }_j$$

(18)

not only compensates for counted fraction non-linearity (Section 4.1.2) but also allows separate estimates, by logistic regression of Equation 18 to the response data (often using a maximum likelihood criterium), of the quantitative, continuous variables ${\delta }_j$: task difficulty of each of the set of objects,j, and ${\theta }_i$: agent ability of each of the set (’cohort’) of instruments, i as the principal clinical measurands, for instance in our elementary case. (Note that this logistic regression is made on the whole matrix of responses of i instruments and j objects similar to, for instance, a least-squares subdivision, as opposed to bilateral, pairwise comparisons more commonly found in metrology (section 4.1.1 in [2].)

The principle of specific objectivity is one major approach to providing separate estimates of these attributes as measurands [40]. Following the Principle of Specific Objectivity, MSA responses are not measures of the instrument’s (classifier’s or decision maker’s) ability, nor the classification task (object) difficulty, but depend on both. Rasch [40] attempted to motivate his principle by drawing analogies to relations between several variables and the universal equations of Physics (such as Newton’s 2nd law).

The MSA approach of engineering metrology is felt to be a closer point of departure [50] (Section 3.1). In accordance with Rasch’s principle of specific objectivity, task difficulty and agent ability need to be treated separately:

• ’agent’ ability, $\theta$
• ’task’ difficulty, $\delta$

In making separate estimates of the measurands, ability and task difficulty, following the Principle of Specific Objectivity, when reliability and validity are to be assessed, the normal rules of statistics apply, as well as more metrological requirements which go beyond requiring mere numbers to making judicious and representative sampling across the full range of measurement scales (Section 5.5).

Alongside his better-known log-odds expression ((17), the Poisson PMF distribution Figure 2 of SPC was chosen by Rasch [51] when originally proposing his psychometric approach (Section 4.3.3) to determining the ability of individual cohort members to perform psychometric tests such as reading. Misreadings in such tests correspond with the typical application of the Poisson distribution (Section 4.2) in quality control as a model of the number of defects or non-conformities that occur in a ’unit of product’ [33]. In Rasch’s psychometric models [40,51], the Poisson rate factor $\lambda =k/h$, where ’task’ difficulty, $\delta =log\left(k\right)$ and ‘agent’ ability, $\theta =log\left(h\right)$ [52,53] is recognized as "a necessary condition for specific objectivity of comparisons between tests and between students", with "the result that for the Poisson distribution the multiplicative structure (IX:2) of its parameter A; is both necessary and sufficient for obtaining specific objectivity can be extended to the whole class of additive exponential models...." ([40], pp. 92-92). How the Poisson distribution can be useful when defining clinical performance metrics for the elementary counting case, Figure 4, is further examined in (Section 4.3.3).

4.3.1. Agnostic Rasch Models

Most applications of Rasch’s measurement models [40,51] have to date been made in education, health care, and survey research, involving persons who act as instruments [50], such as in the elementary counting example (Section 5). But that model is not mathematically limited to psychometrics or human respondents, but should also be applicable to more technical agents, as Rasch [54] recognized in his retirement lecture, which he closed by saying that the potential for use of the models he developed "stretches to all sciences where the subjects are comparisons that must be objective".

Diverse examples, such as:

• of a material hardness indenter [55] to make an impression (JCGM VIM [56] EXAMPLE 1 Rockwell C hardness)
• of a hospital to provide a service (A & E [57], surgery [58])
• in psychophysics, that is, where various physical fields and forces impinge on the five senses of an agent [50,59].

are amenable to an analysis in terms of the measurands (i) an ’ability’ of an agent and the ’difficulty’ of a task, or alternatively, (ii) in usability studies [60], the ’leniency’ of an agent and the ’quality’ of a product or service, by exploiting the agnostic aspect of the Rasch measurement model (Section 4.3).

4.3.2. Polytomous Measurement Models

Rasch’s dichotomous measurement model version, Equation 17, has subsequently been complemented by the corresponding polytomous measurement model variants as well as multilevel [61,62], multidimensional [63], multifaceted [64,65], mixture [66], and other models. As an example, JCGM VIM [56] EXAMPLE 4 Subjective level of abdominal pain on a scale from zero to five belongs here.

Registering the response, $P_{success}$, of the measurement system when instrument resolution is limited or levels of uncertainty are high in the measurement process, is often made more practically in terms of a Probability Mass Function (PMF) with discrete categories rather than as a continuous response scale (so-called ’visual (analogue) score’) by rounding off to the nearest integer, even when the object quantity varies on a continuous scale.

Polytomous classification is the term used when more than two categories are assigned. In the elementary counting case, Figure 4, this would correspond to ’guessing’ the count to lie in a range of integer numbers of dots for a given cloud, as a clinical performance study 2.

Approaches to polytomous response analysis include the so-called ’partial credit model’ (PCM, [67,68]) in which the binary (dichotomous) log-odds ratios (OR) models are extended to polytomous cases, as summarised as follows:

Probability, $q_{i,j,k}$ of response $x_{i,j}$ of instrument (agent) i;object (item) j, over k categories for PCM:

$$q_{i,j,k}={\displaystyle \frac{exp{\sum }_{c=0}^k({\theta }_i-{\delta }_{j,c})}{exp{\sum }_{k=0}^{C_j}·\left({\sum }_{c=0}^k({\theta }_i-{\delta }_{j,c})\right)}}$$

(19)

An alternative to the PCM is the Rasch Rating Model [69]:

$$Pr(Y_{i,j}=q)=q_{i,j,k}={\displaystyle \frac{exp{\sum }_{c=0}^k({\theta }_i-{\delta }_{j,c}-{\tau }_c)}{exp{\sum }_{k=0}^{C_j}·\left({\sum }_{c=0}^k({\theta }_i-{\delta }_{j,c}-{\tau }_c)\right)}}$$

(20)

where ${\tau }_c$ is the threshold of category c.

Even the present case of elementary counting where, for each task of counting a number of dots, an individual cohort member is making a polytomous decision: "how many dots do I see?" among the set of potential dot numbers (Section 4.3.3).

4.3.3. Poisson PMFs for Counting

At about the same time as Rasch [40] formulated his psychometric model, Equation (17), he also considered [51] an alternative formulation referring to a probabilistic Poisson distribution (Section 4.2) well known from SPC as a model of the number of defects or nonconformities that occur in a unit of product when classifying it:

$$p\left(x\right)={\displaystyle \frac{e^{-\lambda }{\lambda }^x}{x!}};x=0,1,\dots ,n$$

(21)

where x is the ’quality characteristic’ being classified (number of defects) in n observations.

The rate parameter $\lambda$ is equal directly both to the mean and variance of the Poisson distribution ([33] p. 55 and (Section 4.2)). In [51] a Rasch Poisson PMF was given as a model for the number of ’misreadings’ by a child (i) for a text (j) of a particular level of difficulty. As quoted in [53], and references therein: "Although it is one of the earliest models that Rasch has developed, the Rasch Poisson Counts Model has received less attention than other binary or polytomous Rasch models".

A much earlier observation by Meredith [48] noted: "a thorough understanding and application of such a particularly simple [Poisson] model is often the key to further development of suitable stochastic models in scientific work".

The fundamental value obtained via the Poisson counts model was only belatedly realized by Rasch ([70] p. 66) when, after a 1959 conversation with a colleague, he suddenly realized "that the possibility of separating two sets of parameters must be a fundamental property of a very important class of models," and that his study of children’s misreadings had led him to devise a new "class of probability models [having] the property in common with the Multiplicative Poisson Model, that one set of parameters can be eliminated by means of conditional probabilities while attention is concentrated on the other set, and vice versa." Andrich [71] then showed that the Poisson distribution is necessary and sufficient for constructing measures from discrete observations.

In the present work, the example of the elementary counting of dots, akin to the Rasch [51] counting study, not only provides an ’analytical, sampling by variable’ PMF (Section 3.3) but will also exemplify how Poisson PMFs can be benchmarked against full psychometric studies as ’clinical, sampling by attribute’ PMF studies (Section 5).

In the case study of dot counting by the Munduruku (indigenous people of Brazil), [7,8], Poisson PMFs can be derived with Equation (21) since task difficulty can be calculated from first principles, Equation (26), from the number of dots being counted, thanks to the conceptual simplicity of the task [72], pp.66ff. Table 2 shows the PMF values from the Poisson rate, $\lambda$, calculated by entering the simulated counting difficulty $\delta$ into Equation (21), where $k_j=e^{{\delta }_j}$ and taking a cohort-average ability, ${\theta }_{mean}$, to yield $h_i=e^{{\theta }_i}$.

Figure 2 shows the resulting plots of PMFs for three cases (true counts of dots: (1), (6) and (8)). These PMFs show the distribution of probabilities for each object count (no. of dots), with a shifting maximum — e.g. at $x=5$ mis-classifications for 8 dots — corresponding to the most likely value of x, the quality characteristic being classified (number of mis-classifications) in n observations.

• In contrast to Rasch’s [40] psychometric model, it is important to note that Poisson [73] placed important restrictions on the applicability of his model: particularly that the misclassification events were to be "very rare", as should apply to the PMFs shown in Figure 2. Although there appears in the literature to be no exact threshold where the Poisson approximation breaks down, a ’rule of thumb’ quotes: $\lambda =n·p$ is moderate (typically $n·p\le 5-10$ for good accuracy) ([33], Figure 2-24). Our psychometric simulations (Section 5) of task difficulty and counter ability do not suffer from the same restrictions and allow the choice of response level, $P_{success}$, over a complete range Figure 8: from the most mis-classifications (for a low-ability counter attempting a difficult counting task) to the least number of mis-classifications where the Poisson approximation should be valid in the latter case (i.e., for a high-ability counter performing an easy counting task). The Poisson rule-of-thumb range can be seen to be comparable with the mis-classification rates shown in Figure 2. However, to make a full comparison between the classic ’rule of thumb’ limit to the Poisson distribution and the present psychometric simulations requires a proper account of the applicability of the ergodic principle: that is, to what extent the classic group-statistic Poisson approach [33,73] corresponds to the individual statistics of Rasch [51] psychometric modelling. For instance, the Poisson rate $\lambda =n·p$ has in some way to be interpreted in psychometric cases where $n=1$, i.e., just one counting agent (MSA: Instrument) while, at the same time, the overall number of degrees of freedom needs to be sufficiently large to achieve adequate reliability and validity (Section 5.5).
• The Poisson PMF number of mis-classifications shown in Figure 2 has no information about which numbers are perceived correctly or not, but should in principle correspond — for each item (j) — to the occupancies of the analytical PMFs shown in Figure 1. Similarly, the Poisson PMF number of mis-classifications has no obvious relation to the corresponding ’clinical’ metrics, that is, the counting task difficulty and the counter ability. (In principle, one can estimate the expected count by inverting Equation (21) since — in the present case of an elementary construct — the relation between task difficulty and the number of dots is known, Equation (26)).

5. ’Clinical’ Psychometric Study: Counting Dots

The elementary counting of dots by less numerate agents (e.g., children or indigenous people [7,8]) also presents a case typical of the metrology of counting as a ’clinical’ or ’sampling by attribute’ procedure (2) corresponding to the top row of Table 1, as follows.

5.1. Dichotomous and Polytomous Mis-Classification Probabilities. CTT and Rasch Measurement Theory

Regularly used clinical performance metrics (2) of measurement systems include: ’selectivity’, Equation (22) and ’specificity’, Equation (23), which are plotted against each other on receiver operating characteristic curves, [30], when sampling by ’attribute’, and are often analysed with CTT despite its known limitations (Section 4.1.2).

Dichotomous mis-classification probabilities [74,75] due to binary decision quandary arising from uncertainties are:

• $FAP=P[Y=1|X=0];$ False Acceptance (or positive, FPR) Probability

• $FRP=P[Y=0|X=1];$ False Rejection (or negative, FNR) Probability

These two probabilities lie on the off-diagonal of a 2 x 2 Decision matrix for a binary test, where the true positive and true negative classification probabilities lie on the diagonal. Each pair of probabilities — positive (TP, FP) and negative (FN, TN) — can be plotted as binary PMFs like the binomial plots shown in Figure 5.

Performance metrics can be defined for the case of a binary decision A or B about an entity when assessed for conformity with respect to a specification limit, SL, for the quantity, z, of interest, based on a signal (SN with noise) in the presence of noise, N,30]:

True positive rate (TPR) aka Selectivity:

$$P(A\mid SN)={\displaystyle \frac{{\sum }_{z>SL}{N}_{z,\mathrm{detected}}}{{\sum }_z{N}_{z,\mathrm{detected}}}}$$

(22)

True negative rate (TNR) aka Specificity:

$$P(B\mid N)={\displaystyle \frac{{\sum }_{z<SL}{N}_{z,\mathrm{not}\mathrm{detected}}}{{\sum }_z{N}_{z,\mathrm{detected}}}}$$

(23)

The elementary counting case Section 5 corresponds to a polytomous extension (Section 4.3.2) of each of these classic clinical performance metrics.

Recall that "clinical performance aims to demonstrate that the measurement instrument can achieve clinically relevant outputs through predictable and reliable use by the intended users", 2,26]. Considering the known (but not always recognised) limitations in CTT (Section 4.1.2), a more reliable approach to clinical performance assessment [32,76,77,78] is therefore to deploy Rasch Measurement Theory (Section 4.3) which:

• (i) compensates for counted-fraction scale non-linearity (Section 4.1.3)
• (ii) exchanges the analytical measurands (such as the number of dots, Section 3.3) for the clinical measurands: counting task difficulty and counter ability.

In the ’clinical’ scenario 2, the level of difficulty attributed to each object counted and the ability attributed to each counter (the person or, in MSA terms Figure 3, the ’instrument’), both contribute in some way to the probability of successful counting. For a proper metrological analysis of the clinical analysis of counting, PMFs (occupancy: probability of responses in each category (number of dots) for a range of tasks of increasing levels of difficulty can be analysed psychometrically where the quantities (measurands) of interest are primarily the ability of each counter and the level of difficulty of a certain number of dots (rather than the actual number of dots – which of course can be determined by any capable counter).

5.2. Counted-Fraction Scale Non-Linearity When Counting Dots. ICC

Figure 9 exemplifies for the elementary counting example (Section 5) so-called ICC (Item Characteristic Curves, [79]) for a particular item, calculated for a selected counting task (difficulty ${\delta }_j$, $j=4$, using PCM modelling of polytomous data distributed across a cohort of test agents (persons counting) calculated as follows:

For an item with categories $c=0,1,2,...,k$ and Andrich thresholds ${\tau }_c$ Equation 20, with ${\tau }_0$ chosen to be 0, then: At location x on the latent variable (relative to the item difficulty), the probability of observing category c is $P_c\left(x\right)={\displaystyle \frac{e^{c·x-{\sum }_{c=0}^k{\tau }_c}}{sump\left(x\right)}}$ where $sump\left(x\right)={\sum }_{c=0}^k{e}^{c*x-sum\left({\tau }_c\right)}$ for $c=0$ to k.

The expected score y at location x is given by: $y={\sum }_{c=0}^k(c·P_c\left(x\right))$

Further publications referring to the simulation of psychometric ICC models can be found in, for example: [80,81]. In the present work, the simulated values of as described in (Section 4) simply use the usual dichotomous Rasch expression Equation 17 (which for benign data sets should give at least as good estimates as rounding to the nearest category, c, as described by [81]).

5.3. Communication of Measurement Information Throughout the Measurement Process. Amount of Entropy

To meet the challenge posed by PMF axes being often on less quantitative or even nominal scales where even the most basic arithmetic operations cannot be assumed to always work (Section 4.1.3) a viable alternative to calculating distances (even on ordinal and nominal scales) of PMF plots is to take differences in informational (Shannon) entropy.

The amount of measurement information on the categorical scales of signals in each classification category, c, at any one point and state in the measurement process (from object, through instrument and operator to restitution, Figure 3) can be expressed as a [82] ’surprisal’ $-ln\left(q_c\right)$, while the relative contribution to the total entropy is weighted with the relative occupancy, q_c ([2], chapter 5).

The propagation of measurement information depicted in Figure 3, where information can be lost, distorted or gained, is described more readily with sums and differences of informational entropy than with the corresponding combination of probabilities and PMFs. An example of additions of entropy is found in the so-called Mutual information: $I(Z;Y)=H\left(Z\right)-H\left(Z\right|Y)$ which has been used by Benish [83] where an individual’s disease state is denoted Z while the diagnostic test result is Y. The well-known Kullback-Leibler (K-L) metric $d_{KL}(P,Q)$ for a response Y, PMF $Q\left(y\right)$, to an object attribute Z, PMF $P\left(z\right|A)$, is:

$$d_{KL}(P,Q)=\int -z·d{P}_{success}=-[P_{success}·log\left(P_{success}\right)+(1-P_{success})·log(q-P_{success})]=H(P,Q)-H\left(P\right).$$

(24)

The K-L parameter. Equation (24), is admittedly a valid metric only for infinitesimally small distances and is only one of several candidates for PMF distance metrics widely used, for example, in automatic image classification [84]. It can also be shown (section 5.5.3 in [2]) that log-odds transformed data correspond to entropy, Equation 11.

The start of the measurement process has a deficit in overall entropy (that is, the amount of ’useful’ measurement information initially available) which for a discrete PMF is $\Delta H\left(P\right)=-{\sum }_k{p}_k·ln\left(p_k\right)$, where $p_k$ is the occupancy of category k from the actual quantity attributed to the measurement object. For conceptually simple tasks—-such as the counting of dots—this calculated entropy will provide a metrological reference for calibration of classification properties, as illustrated in the current case study (Section 5.4.1). From a measurement system perspective, the response of the instrument (often a score, $P_{success}$, made by a ’rater’) will provide an input, $X_1$, when establishing the measurement result (Section 3.1).

5.4. Simulated PMFs for the Elementary Counting Case

PMFs are shown in Figure 8 for simulated data for a set of clouds of dots (of increasing (a) (red histograms) task (’object’) counting difficulty, based on entropy theory Equation (26) (Section 5.4.1) and (b) (blue histograms) counter ability (Section 5.4.2), as the conjugate pair of psychometric measurands [7,8]:

5.4.1. Counting Task Difficulty Simulated. Object (Task) Entropy

For the present guide on PMF simulations, ’seed’ values will be considered in a study of elementary counting (of dots by Munduruku indigenous people in Brazil, [7]), a set of clouds of dots was presented visually for each counter as a sequence of counting tasks (’objects’). Task difficulty is in general proportional to entropy – a more ordered (less entropy) task is easier to perform [72,85].

For the current purpose of demonstrating best practice when simulating uncertainties associated with PMFs, of particular interest will be cases where attributes – for instance, the level of difficulty of tasks – are sufficiently simple conceptually so that simulations can be “seeded” with realistic start values. Beyond numerical simulation, it will be particularly valuable if ab initio estimates of categorical quantities such as task difficulty are available. Recent work has demonstrated how such knowledge-informed seed values can indeed be calculated even for categorical properties in a manner analogous to establishing metrological reference values in the more quantitative metrology in Chemistry and the Material sciences, as demonstrated in the neurodegeneration studies of legacy memory tests [86] where recall task difficulty could be expressed with so-called Construct Specification Equation (CSE, [87,88,89,90])

$$\delta ={\Sigma }_k{\beta }_k·Z_k$$

(25)

in terms of entropy-based explanatory variables $Z_k$. The PMF plotted (red histograms) in Figure 8 against the measurand, counting task difficulty, ${\delta }_j$, are those predicted to increase discretely, reflecting the increasing number, G, of integer dots in successive clouds according to the basic Hartley entropy [91]:

$${\delta }_j=ln\left(G_j\right)$$

(26)

assuming a random distribution of identical dots in each cloud counted. (The addition of some symmetry to a cloud of dots would make the counting task easier, in terms of a lowering in the overall task entropy, as has been studied for elementary memory recall tests [72], p. 68.)

5.4.2. Counting Classifier Ability Simulated

• Excel:

  $$NORM.INV\left(RAND\right();0;2)$$

  (27)
  Returns a vector of random numbers having the Normal distribution of the measurand counter abilities across the cohort shown with the blue histograms in Figure 8.

5.5. Reliability and Validity

In making separate estimates of the conjugate attributes, ${\delta }_j$ and ${\theta }_i$ — which are not merely the measurands, but also for ’clinical’ purposes the quality characteristics, when conformity assessment is made — the normal rules of statistics of course apply – including making sufficient numbers of observations of items and instruments – to ensure minimum levels of reliability and power as these are relevant in fit-for-purpose designs distinguishing between the varying demands of screening, diagnostic, accountability, and research applications. Additional, more metrological requirements go beyond requiring mere numbers to making judicious and representative sampling across the full range of measurement scales. For metrological purposes — concerning both precision and trueness (for metrological invariance, comparability and interoperability) — the underlying scale needs to be unidimensional, linear, quantitative, well targeted, free of differential item functioning, etc., and there is a whole battery of conventional tests of model validity [85].

The standard uncertainty in the ’height’ (i.e., occupancy, $P_{success}$) of a PMF column is:

$$u\left(P_{success}\right)=\sqrt{{\displaystyle \frac{P_{success}·(1-P_{success})}{n}}},$$

(28)

derived from the underlying binomial distribution Equation 7. One should however always respect the inherent non-linearity of the scale, such as due to the counted-fractions effect (Section 4.1.2). A more proper estimation of this uncertainty is to first calculate the uncertainties in the link function, as given by the GLMM log-odds approach, Equation 11 .

Uncertainties (so-called Standard Errors, SE, i.e., standard uncertainties) in Rasch analyses with the logistic regression, Equation 18 are those quoted by the WINSTEPS® program (section 21.120), where according to that program’s manual (p. 805):

$$SE({\theta }_i,\tilde{\delta })=\sqrt{{\displaystyle \frac{1}{{\Sigma }_{j=1}^{N_{items}}{P}_{success,i,j}·(1-P_{success,i,j})}}}$$

(29)

and

$$SE(\tilde{\theta },{\delta }_j)=\sqrt{{\displaystyle \frac{1}{{\Sigma }_{i=1}^{N_{TP}}{P}_{success,i,j}(1-P_{success,i,j})}}}$$

(30)

Thereafter, one can apply the inverse of the Rasch transformation in the case the enduser wants to relate to the original PMFs. Uncertainty intervals in $P_{success}$) calculated in that way will become increasingly asymmetric, the closer the score is to either end of the scale.)

Reliability means assessing the influence of measurement uncertainty as an estimate of limited measurement quality. In psychometrics [45], a reliability coefficient, $R_{\beta }$, for a Rasch variable $\beta ={\beta }^{\prime }+{ϵ}_{\beta }$, (for either Rasch attribute: $\beta =\theta$ or $\delta$) including an error term, ${ϵ}_{\beta }$, is defined as:

$$R_{\beta }={\displaystyle \frac{var\left({\beta }^{\prime }\right)}{var\left(\beta \right)}}={\displaystyle \frac{var\left(\beta \right)-var\left({ϵ}_{\beta }\right)}{var\left(\beta \right)}}$$

(31)

where the ’true’ variance is $var\left({\beta }^{\prime }\right)$ and the observed variance is $var\left(\beta \right)$.

The consequences of the decisions to be made determine what the actual limits of reliability are, for instance, by setting a maximum permissible uncertainty. Traditional psychometric limits for so-called ’high-stake’ decisions are typically $R_{\beta }>0.8$ (Equation (31)) which corresponds to at most half of the observed dispersion being assigned to measurement uncertainty.

5.6. Validation of Simulated LOGISTIC regressions

The results of logistic regressions of partial credit model (Equation 19) to raw scores (y-axes of Figure 9; calculated with the dichotomous Rasch formula for ${\delta }_j$, Equation 17) across a simulated cohort of varying (x-axes) counting ability, ${\theta }_i$ Figure 8(b) for a pair of selected counting tasks (difficulties ${\delta }_j$ shown in Figure 8(a)) of elementary counting tasks. Using the WINSTEPS ® program for this logistic regression, the raw scores were coded into categories by integer rounding of the % scores, as is particularly evident in the steps shown in Figure 9.

The validity of this procedure is investigated in Figure 10 and Figure 12 by comparing the restituted estimates (y-axes) of $\delta$ and $\theta$ with the original simulated values (x-axes) plotted, respectively.

A second validation was made by comparing (Figure 11) simulated (y-axis) and original experimental data (x-axis) counting task difficulties from psychometric analysis (Rasch Equation (17)).

Figure 10. Comparison of simulated (y-axis) and calculated (x-axis) counting task difficulties from psychometric analysis (Rasch 17) of a cohort (a) (blue) Task difficulty and (b) (red) 10 x residuals of differences between simulated and calculated difficulties. Uncertainty coverage factor, $k=2$

Figure 10. Comparison of simulated (y-axis) and calculated (x-axis) counting task difficulties from psychometric analysis (Rasch 17) of a cohort (a) (blue) Task difficulty and (b) (red) 10 x residuals of differences between simulated and calculated difficulties. Uncertainty coverage factor, $k=2$

Figure 11. Comparison of simulated (y-axis) and original experimental data (x-axis) counting task difficulties from psychometric analysis (Rasch Equation 17). Uncertainty coverage factor, $k=2$

Figure 11. Comparison of simulated (y-axis) and original experimental data (x-axis) counting task difficulties from psychometric analysis (Rasch Equation 17). Uncertainty coverage factor, $k=2$

Figure 12. Comparison of simulated (y-axis) and calculated (x-axis) counting agent abilities from psychometric analysis (Rasch 17) of a cohort (a) (blue) Agent ability and (b) (red) 10 x residuals of differences between simulated and calculated abilities. Uncertainty coverage factor, $k=2$

Figure 12. Comparison of simulated (y-axis) and calculated (x-axis) counting agent abilities from psychometric analysis (Rasch 17) of a cohort (a) (blue) Agent ability and (b) (red) 10 x residuals of differences between simulated and calculated abilities. Uncertainty coverage factor, $k=2$

6. Discussion and Conclusions

The choice of construct — quantities attributed to an elementary counting process — in the current case study is made deliberately. The conceptual simplicity ensures the unidimensionality required in metrological psychometrics [40,50,81]. Additionally, it offers the opportunity to define — from first principles — metrological references (’gold standards’) [72] for counting task difficulty in terms of Construct Specification Equations [87,88,89,90,92] with explanatory variables based on informational entropy [93,94], as demonstrated in the simulation of counting task difficulty (Section 5.4.1). Such metrological references for calibration of categorical properties enable in turn interoperability and defining limits on quality of products and services of all kinds. (Other explanatory approaches beyond CSE include [95,96].)

This conceptual simplicity facilitates contrasting the metrology of three kinds of probability mass functions (PMFs) (Table 1): (i) quantitative response to a discrete range of counts, Figure 1, (ii) the classic Poisson distribution of miscounts, Figure 6, and (iii) psychometric (Rasch) distributions of counting task difficulty and person counting ability, Figure 8. New insights are provided when benchmarking the Rasch Poisson Counts Model (Section 4.3.3) which has received less attention, against full psychometric Rasch modelling (Section 5). Limits (traditionally, rules-of-thumb on small probabilities and large number of observations) on the validity of the Poisson distribution can be tested with access to psychometric simulations across the full range of task difficulty and counter ability.

A human being acting as a counter (MSA: A human as a Measurement Instrument [50]) is a prototype for more advanced applications, including counting, decision-making and similar tasks performed by Artificial Intelligence agents [6,72,97,98]. Logistic regression (Section 4.3) provides an essential complement when handling discrete and sometimes qualitative observations, for example, to a typical use of Machine Learning in metrology where traditionally regular regression tasks are made on a continuous quantity [99]. The proposed methods will be useful when the traditional role of Physics in metrology is to be complemented by a typical ’turn-of-the-millennium’ focus on the new metrology of ’clinical’ and other applications where meaning and value, and not just numbers and analytics, are sought.

Future work is needed to extend the present studies to include less elementary and more complex constructs. There is much current research addressing multidimensional measurement theory and related topics in this field [32,100,101,102,103].