Entropy Estimation: From Statistical Physics to Classification

Driven by the common entropy formalism in statistical physics and information theory here we show that methods developed for the analysis of molecular conformational ensembles may be successfully used for classification. Entropy is estimated using the k-th nearest neighbour method combined with the Maximum Information Spanning Tree method to account for the mutual informations between variables. This formalism is used to classify new samples using their estimated cross-entropy with the training set. The method is tested using diverse datasets and the results obtained are comparable to those obtained by machine learning methods, or better when the number of samples is small. The methods described here have the advantage, compared to other classification methods, of directly linking variables and pairs of variables to the assigned class, a valuable feature when aiming at identifying causative relationships between features and classes (e.g. the relationship between genes and diseases).