Submitted:
23 October 2025
Posted:
27 October 2025
You are already at the latest version
Abstract
Keywords:
Related Work
Methodology
Data Pipeline
Prompt
Data Filtration
Text to Speech for Audio Generation
| Emotional Tone | Pitch | Description |
| Happy | 1.2 | Increase pitch for happy |
| Sad | 0.8 | Lower pitch for sad |
| Angry | 1.4 | Increase pitch and stress for angry |
| Neutral | 1 | Normal pitch for Neutral |
| Emotional Tone | Rate | Description |
| Happy | 1.5 | Increase rate for happy |
| Sad | 0.7 | Slower rate for sad |
| Angry | 1.2 | Faster rate for angry |
| Neutral | 1 | Normal rate for neutral |
Datasets
Experiments & Results
| Model | CIDEr | BLEU4 |
| Audio-Flamingo | 0.6 | 0.1 |
| Audio-Flamingo | 1.2 | 0.3 |
| LTU | 0.48 | 0.17 |
| LTU | 0.67 | 0.25 |
| Qwen-Audio | 0.53 | 0.03 |
Conclusion
References
- S. Ruder, R. Johnson, and P. R. Kumar, “Dialogue datasets for multilingual conversational AI: A survey,” in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 1–10.
- L. M. Rojas-Barahona and P. A. Perez, “Universal dialogue state tracking with a pretrained language model,” in Proc. Conf. Natural Language Learning (CoNLL), 2020, pp. 65–75.
- Y. Ji, D. H. Kim, and D. Lee, “Cross-lingual transfer learning for dialogue generation,” in Proc. Neural Information Processing Systems (NeurIPS), 2019, pp. 1124–1135.
- J. Tiedemann and S. Thottingal, “OPUS-MT: A massive multilingual machine translation system,” in Proc. Workshop on Neural Generation and Translation (WNGT), 2020, pp. 19–23.
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186.
- C. Raffel, N. Shazeer, A. Roberts, et al., “T5: Exploring the limits of transfer learning with a unified text-to-text transformer,” in Proc. Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 4171–4186.
- L. Li, Y. Song, Z. Wang, et al., “Building multilingual dialogue systems with pretrained language models,” in Proc. Int. Conf. Learning Representations (ICLR), 2020, pp. 125–136.
- Y. Zhang, S. Chen, and X. Zeng, “Towards multilingual text generation for dialogue systems,” in Proc. Int. Conf. Computational Linguistics (COLING), 2020, pp. 2205–2214.
- R. Singh, D. N. D. M., and T. K. Chakraborty, “Exploring multilingual conversational agents: Challenges and approaches,” in Proc. Workshop on Multilingual Natural Language Processing (MLNLP), 2020, pp. 34–42.
- X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang, “Wavcaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,” arXiv preprint arXiv:2303.17395, 2023.
- A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. Int. Conf. Machine Learning (ICML), 2023, pp. 28492–28518.
- C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generating captions for audios in the wild,” in Proc. Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019, pp. 119–132.
- Y. Gong, S. Khurana, L. Karlinsky, and J. Glass, “Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers,” arXiv preprint arXiv:2307.03183, 2023.
- S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,” arXiv preprint arXiv:2305.11834, 2023.
- Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919, 2023.
- Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, “Listen, think, and understand,” arXiv preprint arXiv:2305.10790, 2023.
- S. Liu, A. S. Hussain, C. Sun, and Y. Shan, “Music understanding llama: Advancing text-to-music generation with question answering and captioning,” arXiv preprint arXiv:2308.11276, 2023.
- A.-M. Oncescu, A. Koepke, J. F. Henriques, Z. Akata, and S. Albanie, “Audio retrieval with natural language queries,” arXiv preprint arXiv:2105.02192, 2021.
- R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y. Wu, Z. Hong, J. Huang, J. Liu, et al., “Audiogpt: Understanding and generating speech, music, sound, and talking head,” arXiv preprint arXiv:2304.12995, 2023.
- L. Salewski, S. Fauth, A. Koepke, and Z. Akata, “Zero-shot audio captioning with audio-language model guidance and audio context keywords,” arXiv preprint arXiv:2311.08396, 2023.
- A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, et al., “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325, 2023.
- E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “Fsd50k: an open dataset of human-labeled sound events,” IEEE/ACM Trans. Audio, Speech, and Language Processing, vol. 30, pp. 829–852, 2021.
- H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.
- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
- A. Adigwe, N. Tits, K. E. Haddad, S. Ostadabbas, and T. Dutoit, “The emotional voices database: Towards controlling the emotion dimension in voice generation systems,” arXiv preprint arXiv:1806.09514, 2018.
- B. Peng, C. Li, P. He, M. Galley, and J. Gao, “Instruction tuning with GPT-4,” arXiv preprint arXiv:2304.03277, 2023.
- P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, et al., “Audiopalm: A large language model that can speak and listen,” arXiv preprint arXiv:2306.12925, 2023.
- Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro, “Audio Flamingo: A novel audio language model with few-shot learning and dialogue abilities,” arXiv preprint arXiv:2402.01831, 2024.
- B. Elizalde, S. Deshmukh, and H. Wang, “Natural language supervision for general-purpose audio representations,” arXiv preprint arXiv:2309.05767, 2023. [Online]. Available: https://arxiv.org/abs/2309.05767.
- J. Gardner, S. Durand, D. Stoller, and R. M. Bittner, “Llark: A multimodal foundation model for music,” arXiv preprint arXiv:2310.07160, 2023.
- S. Lipping, P. Sudarsanam, K. Drossos, and T. Virtanen, “Clotho-AQA: A crowdsourced dataset for audio question answering,” in Proc. 30th Eur. Signal Process. Conf. (EUSIPCO), 2022, pp. 1140–1144.
- Z. Yang, W. Ping, Z. Liu, V. Korthikanti, W. Nie, D.-A. Huang, L. Fan, Z. Yu, S. Lan, B. Li, et al., “Re-ViLM: Retrieval-augmented visual language model for zero and few-shot image captioning,” in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), 2023.
- H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” arXiv preprint arXiv:2304.08485, 2023.
- J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al., “Flamingo: A visual language model for few-shot learning,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 23716–23736, 2022.
- H. Wang, H. Wu, Z. He, L. Huang, and K. W. Church, “Progress in machine translation,” Engineering, vol. 18, pp. 143–153, 2022.
- W. S. El-Kassas, C. R. Salama, A. A. Rafea, and H. K. Mohamed, “Automatic text summarization: A comprehensive survey,” Expert Systems with Applications, vol. 165, p. 113679, 2021.
- R. Luo, Z. Zhao, M. Yang, J. Dong, M. Qiu, P. Lu, T. Wang, and Z. Wei, “Valley: Video assistant with large language model enhanced ability,” arXiv preprint arXiv:2306.07207, 2023.
- C. Lyu, M. Wu, L. Wang, X. Huang, B. Liu, Z. Du, S. Shi, and Z. Tu, “Macaw-LLM: Multi-modal language modeling with image, audio, video, and text integration,” arXiv preprint arXiv:2306.09093, 2023.
- M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-ChatGPT: Towards detailed video understanding via large vision and language models,” arXiv preprint arXiv:2306.05424, 2023.
- H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “CREMA-D: Crowd-sourced emotional multimodal actors dataset,” IEEE Trans. Affective Computing, vol. 5, no. 4, pp. 377–390, 2014.
- C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,” Lang. Resources and Evaluation, vol. 42, pp. 335–359, 2008.
- P. Barros, N. Churamani, E. Lakomkin, H. Siqueira, A. Sutherland, and S. Wermter, “The OMG-Emotion behavior dataset,” in Proc. Int. Joint Conf. Neural Networks (IJCNN), 2018, pp. 1–7.
- Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner, “MUSDB18-HQ: An uncompressed version of MUSDB18,” Aug. 2019. [Online]. Available: . [CrossRef]
- S. Hershey, D. P. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, “The benefit of temporally strong labels in audio event classification,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 366–370.
- Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5.
- R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “CIDEr: Consensus-based image description evaluation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2015, pp. 4566–4575.
- K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” in Proc. 40th Annu. Meeting of the Association for Computational Linguistics (ACL), 2002, pp. 311–318.
- C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Proc. Text Summarization Branches Out, 2004, pp. 74–81.
- V. Pendyala, R. Raja, A. Vats, R. Para, D. Krishnamoorthy, U. Kumar, S. R. Narra, S. Bharadwaj, D. Nagasubramanian, P. Roy, D. Roy, D. Pant, and S. Lohani, “The Cognitive Nexus, Vol. 1, Issue 2: Advances in AI Methodology, Infrastructure, and Governance,” IEEE Computational Intelligence Society, Santa Clara Valley Chapter, Oct. 2025. Magazine issue editorial. Available at: https://www.researchgate.net/publication/396179773_The_Cognitive_Nexus_Vol_1_Issue_2_Advances_in_AI_Methodology_Infrastructure_and_Governance.
- V. Pendyala, R. Raja, A. Vats, N. Krishnan, L. Yerra, A. Kar, N. Kalu-Mba, M. Venkatram, and S. R. Bolla, “The Cognitive Nexus, Vol. 1, Issue 1: Computational Intelligence for Collaboration, Vision–Language Reasoning, and Resilient Infrastructures,” IEEE Computational Intelligence Society, Santa Clara Valley Chapter, July 2025. Magazine issue editorial. Available at: https://www.researchgate.net/publication/396179779_The_Cognitive_Nexus_Vol_1_Issue_1_Computational_Intelligence_for_Collaboration_Vision-Language_Reasoning_and_Resilient_Infrastructures.
- V. Pendyala, R. Raja, A. Vats, R. Para, D. Krishnamoorthy, U. Kumar, S. R. Narra, S. Bharadwaj, D. Nagasubramanian, P. Roy, D. Roy, D. Pant, and S. Lohani, “The Cognitive Nexus, Vol. 1, Issue 2: Advances in AI Methodology, Infrastructure, and Governance,” Preprints, Oct. 2025. DOI: 10.20944/preprints202510.0091.v1. Available at: . [CrossRef]


Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).