Accelerating Extreme Search of Multidimensional Functions Based on Natural Gradient Descent with Dirichlet Distributions

In this work, we explore the extreme searching of multidimensional functions by natural gradient descent based on Dirichlet and generalized Dirichlet distributions. The natural gradient is based on describing multidimensional surface with probability distributions, which allows us to reduce changing the accuracy of gradient and step-size. In this article, we propose an algorithm of natural gradient descent based on Dirichlet and generalized Dirichlet distributions. We demonstrate that the natural gradient descent with step-size adaptation with Dirichlet and generalized Dirichlet distributions has higher accuracy and does not take a large number of iterations for minimizing test functions than gradient descent and Adam.