Preprint Article Version 1 Preserved in Portico This version is not peer-reviewed

Learning Numerosity Representations with Transformers

Version 1 : Received: 17 May 2021 / Approved: 18 May 2021 / Online: 18 May 2021 (09:50:01 CEST)

How to cite: Boccato, T.; Testolin, A.; Zorzi, M. Learning Numerosity Representations with Transformers. Preprints 2021, 2021050401 (doi: 10.20944/preprints202105.0401.v1). Boccato, T.; Testolin, A.; Zorzi, M. Learning Numerosity Representations with Transformers. Preprints 2021, 2021050401 (doi: 10.20944/preprints202105.0401.v1).

Abstract

One of the most rapidly advancing areas of deep learning research aims at creating models that learn to disentangle the latent factors of variation from a data distribution. However, modeling joint probability mass functions is usually prohibitive, which motivates the use of conditional models assuming that some information is given as input. In the domain of numerical cognition, deep learning architectures have successfully demonstrated that approximate numerosity representations can emerge in multi-layer networks that build latent representations of a set of images with a varying number of items. However, existing models have focused on tasks requiring to conditionally estimate numerosity information from a given image. Here we focus on a set of much more challenging tasks, which require to conditionally generate synthetic images containing a given number of items. We show that attention-based architectures operating at the pixel level can learn to produce well-formed images approximately containing a specific number of items, even when the target numerosity was not present in the training distribution.

Subject Areas

Deep neural networks; Disentangled representations; Attention mechanisms; Generative models; Density estimation; Out-of-distribution generalization; Numerical cognition; Visual perception; Cognitive modeling

Comments (0)

We encourage comments and feedback from a broad range of readers. See criteria for comments and our diversity statement.

Leave a public comment
Send a private comment to the author(s)
Views 0
Downloads 0
Comments 0
Metrics 0


×
Alerts
Notify me about updates to this article or when a peer-reviewed version is published.
We use cookies on our website to ensure you get the best experience.
Read more about our cookies here.