Masterminding the Giant : Shannonian Entropic Threshold Perplexity AI

2. Crossroads between Perplexity AI, and Shannonian Entropy

When assessing language models, one popular statistic to utilise is ambiguity [9]. Consider the built-in metric of perplexity in scikit-learn’s implementation of the topic-modelling algorithm Latent Dirichlet Allocation.

In numerous Natural Language challenges, a common configuration [9] is to have a language L and aim to construct a model M for it. A particular genre or corpus, such as “English Wikipedia,” “Nigerian Twitter,” or “Shakespeare,” or (at least conceptually) a general term, such as “French,” could constitute the “language.”

Specifically [9], by a language L, we mean a process for generating text. Given a history h consisting of a series of previous words in a sentence, the language L is the probability that the next word is w:

$$h a series of words(w_1,w_2,\dots ,w_{n-1})$$

$$L\left(w|h\right)=Prob\left(next word is w\right|previous words are h)$$

(1)

$$A language model is a collection of probaility distributions$$

If L is “English,” for instance, I’m ready to bet that:

➢

L(dog | The swift brown fox leaps over the slow brown) ≈ 1.

➢

L(ipsum | Lorem) ≈ 1.

➢

L(wings | Buffalo buffalo buffalo Buffalo buffalo) ≈ 0

$$\begin{gathered} L\left(s\right)=L\left(w_1|w_0\right)\times L\left(w_2|w_0{w}_1\right)\dots \times \left(w_n|w_0{w}_1\dots w_{n-1}\right) \\ {=\prod }_{k=1}^{n}L\left(w_k|w_0{w}_1\dots w_{k-1}\right) \end{gathered}$$

(2)

$$The language L gives the probability of a sentence s$$

Nonetheless, it is typical practice to omit the product’s initial phrase as well, or even to utilise an even longer beginning context [9].Obtaining an impeccable imitation of $L$ in, instance, spoken American English is surprisingly simple. Simply signal any person who is fluent in English as they pass by. Typically, our goal is to teach the model to a computer, which is why machine learning was created. Thus, $M$ will represent any language model that we have been able to construct computationally. This configuration, comprising of a language $L$ and a model $M$, is highly versatile and is utilised in numerous Natural Language jobs such as speech-to-text, autocorrection, autocomplete, machine translation, and so forth.

Given the words someone has entered so far, autocomplete is the most obvious example; pick the completion with the highest probability and try to estimate what they could type next.

Eventually [9], if given a model $M$, the perplexity on a sentence $s$, reads:

$$Perplexity\left(M\right)= {\left({\prod }_{k=1}^n\left(\frac{1}{M\left(w_k|w_0{w}_1\dots w_{k-1}\right)}\right)\right)}^{\frac{1}{n}}$$

(3)

$$Perplexity of a language model M$$

This is the opposite of the geometric mean of the terms in the product’s denominator, as the second line makes clear. This can be understood as a per-word metric since the likelihood of each word is only computed once, contingent upon the history. This indicates that sentence length has no bearing on the level of bewilderment, other things being equal. We want our odds to be high, which denotes less confusion. The perplexity would be 1 and the model would accurately predict the text if all the probability were 1. Conversely, there will be more confusion with worse language models.

The language itself sets a lower bound on bewilderment. However, this highlights a typical aspect of NLP metrics: a simple metric, such as perplexity, may not always be the best indicator of a model’s actual performance. For development (validation), confusion is beneficial, but not always for evaluation. Human evaluation is still considered the gold standard in evaluation.

3. Results and Experimental Validation

Theorem 1. The Shannonian perplexity AI,

$$\omega \left(p\left(n\right)\right)=2^{H\left(p\right(n)}, H\left(p\left(n\right)\right)=-\sum _{n=1}^{N}p\left(n\right)lnp\left(n\right)$$

(6)

(i) Increases forever in $\lambda$ if and only if $\exists \lambda >0$ satisfying that:

$$p\left(n\right)= e^{-(\lambda +1)},\omega \left(p\left(n\right)\right)= 2^{(\lambda +1)},\lambda = -lnp\left(n\right)-1, p\left(n\right)\in \left[\mathrm{0,1}\right], H\left(0\right)=0$$

(7)

(ii) Decreases forever in $\rho$ if and only if $\exists \rho >0$ satisfying that:

$$p\left(n\right)= e^{(\rho -1)}, \omega \left(p\left(n\right)\right)= 2^{(1-\rho )}, \rho = lnp\left(n\right)+1,p\left(n\right)\in \left[\mathrm{0,1}\right], H\left(0\right)=0$$

(8)

Proof

Let $\omega \left(x\right),$defines perplexity,

$$\omega \left(p\left(n\right)\right)=2^{H\left(p\right(n\left)\right)}, H\left(p\left(n\right)\right) \mathrm{d}\mathrm{e}\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{S}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{a}\mathrm{n}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{y}\mathrm{o}\mathrm{f}\left(\mathrm{c}.\mathrm{f}.,\left(4\right)\right)$$

(9)

$$H\left(p\left(n\right)\right)=-\sum _{n=1}^{N}p\left(n\right)lnp\left(n\right)$$

(10)

$$\frac{d\omega }{dp\left(n\right)}=2^{H\left(p\right(n\left)\right)}\left(-\sum _{n=1}^N\left(1+lnp\left(n\right)\right)\right)$$

(11)

Communicating(11), and IDT, one gets

$$\frac{d\omega }{dp\left(n\right)}>0 ⟺ 2^{H\left(p\left(n\right)\right)}\left(-\sum _{n=1}^N\left(1+lnp\left(n\right)\right)\right)>0$$

(12)

Since $2^{H\left(p\left(n\right)\right)}>0,$

$$\frac{d\omega }{dp\left(n\right)}>0 ⟺ 1+lnp\left(n\right)<0$$

(13)

$1+lnp\left(n\right)< 0$ translates to the existence of a positive real number, say $\lambda$, satisfying:

$$1+lnp\left(n\right)= -\lambda$$

(14)

Or

$$p\left(n\right)=e^{-(\lambda +1)}$$

(15)

The normalization property implies:

$$\sum _{n=1}^{N}p\left(n\right)=1$$

(16)

Engaging (7), and (16), it is implied that:

$$N{e}^{-(\lambda +1)}=1\to N= e^{(\lambda +1)}$$

(17)

(17) translates to

$$\lambda =\mathit{ln}(-1(c.f.,(7\left)\right)$$

(18)

Wrapping up (9), (15),(17), and (18) yields:

$$\omega \left(\lambda \right)= 2^{H\left(p\right(n\left)\right)}= 2^{-\left(\sum _{n=1}^N{e}^{-\left(\lambda +1\right)}\left(-\left(\lambda +1\right)\right)\right)}=2^{\left(\lambda +1\right)N{e}^{-(\lambda +1)}}= 2^{\left(\lambda +1\right)} \left(\mathrm{c.f.,}\right(7\left)\right)$$

Moreover,

$$\frac{d\omega }{dp\left(n\right)}<0 ⟺ 2^{H\left(p\left(n\right)\right)}\left(-\sum _{n=1}^N\left(1+lnp\left(n\right)\right)\right)>0$$

(19)

Equation (19) holds only if $\exists$ a positive real number $\rho$, such that:

$$1+lnp\left(n\right)= \rho$$

(20)

Engaging the same logic as in (i), the proof of (ii) follows.

Experimental setup:

The increasability zones

Observing Figure 2, it can be seen perplexity AI increases by the progressive increase of the fine- tuner, $\lambda$.

On another separate note, reading the represented data of Table 1, the only permitted case, is for $\rho =1$, otherwise, this violates the axiomatic probability logic.

Theorem 2. The upper and lower bounds of Shannonian perplexity AI, ${UB}_{Perplexity AI} {,UB}_{Perplexity AI},$respectively, are given by the following inequality:

$${UB}_{Perplexity AI}={\left(1+\frac{H(p\left(n\right)}{n}\right)}^{\left(n+H(p\left(n\right)\right)}>\omega \left(p\left(n\right)\right)>H(p\left(n\right)={LB}_{Perplexity AI}$$

(21)

$$H\left(p\left(n\right)\right)=-\sum _{n=1}^{N}p\left(n\right)lnp\left(n\right)$$

Proof 

Following Taylor’s expansion of the exponential function, $e^x$, for $x>0$

$$e^x=\sum _{k=0}^{\infty }\left(\frac{x^k}{k!}\right)>x$$

(22)

Moreover, engaging Lehmer’s inequality(c.f., [18]),

$$e^x\le {\left(1+\frac{x}{n}\right)}^{\left(\frac{x+n}{2}\right)} \mathrm{for} x,n>0$$

(23)

Since,

$${\left(1+\frac{x}{n}\right)}^{\left(\frac{x+n}{2}\right)}<{\left(1+\frac{x}{n}\right)}^{\left(x+n\right)} \mathrm{holds} \mathrm{for} \mathrm{all} x,n>0$$

(24)

Setting $x=H\left(p\left(n\right)\right),$combined with communicating (22)-(24), the proof follows.

4. Conclusion, Open Problems, and Next Research Phase

The current influential work showcases for the first time ever, the fine-tuning of AI, is revealed by using a novel revolutionary mathematical approach. The analytic findings are validated numerically. Notably, the upper and lower bounds of Perplexity AI are calculated.

The following are some emerging open problems:

• Assuming other media, such as “Twitter” or “Spoken English,” and other languages, such as “Russian,” the true entropy of written English text remains undetermined.
• Can we replace $H\left(x\right)$, by other entropies, such as Rényi, Tsallis, or $\mathbb{Z}$- entropies to fine tune the perplexity? The question is still open.
• Is it feasible to calculate strict upper and lower bounds for perplexity AI?

The next phase of research includes attempting to solve the proposed complex-still-open to solve problems, as well as the exploration of more unprecedented innovative mathematical techniques to advance AI.