Version 1
: Received: 10 January 2023 / Approved: 12 January 2023 / Online: 12 January 2023 (08:58:03 CET)
Version 2
: Received: 29 March 2023 / Approved: 30 March 2023 / Online: 30 March 2023 (03:51:37 CEST)
T. Liu, Q. Xiong and S. Zhang, "When to Use Large Language Model: Upper Bound Analysis of BM25 Algorithms in Reading Comprehension Task," 2023 5th International Conference on Natural Language Processing (ICNLP), Guangzhou, China, 2023, pp. 1-4, doi: 10.1109/ICNLP58431.2023.00049.
T. Liu, Q. Xiong and S. Zhang, "When to Use Large Language Model: Upper Bound Analysis of BM25 Algorithms in Reading Comprehension Task," 2023 5th International Conference on Natural Language Processing (ICNLP), Guangzhou, China, 2023, pp. 1-4, doi: 10.1109/ICNLP58431.2023.00049.
T. Liu, Q. Xiong and S. Zhang, "When to Use Large Language Model: Upper Bound Analysis of BM25 Algorithms in Reading Comprehension Task," 2023 5th International Conference on Natural Language Processing (ICNLP), Guangzhou, China, 2023, pp. 1-4, doi: 10.1109/ICNLP58431.2023.00049.
T. Liu, Q. Xiong and S. Zhang, "When to Use Large Language Model: Upper Bound Analysis of BM25 Algorithms in Reading Comprehension Task," 2023 5th International Conference on Natural Language Processing (ICNLP), Guangzhou, China, 2023, pp. 1-4, doi: 10.1109/ICNLP58431.2023.00049.
Abstract
Large language model (LLM) is a representation of a major advancement in AI, and has been used in multiple natural language processing tasks. Nevertheless, in different business scenarios, LLM requires fine-tuning by engineers to achieve satisfactory performance, and the cost of achieving target performance and fine-tuning may not match. Based on the Baidu STI dataset, we study the upper bound of the performance that classical information retrieval methods can achieve under a specific business, and compare it with the cost and performance of the participating team based on LLM. This paper gives an insight into the potential of classical computational linguistics algorithms, and which can help decision-makers make reasonable choices for LLM and low-cost methods in business R&D.
Keywords
Large Language Model; natural language processing; reading comprehension; computational lin-guistics; information retrieval; BM25
Subject
Computer Science and Mathematics, Data Structures, Algorithms and Complexity
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Commenter: Tingzhen Liu
Commenter's Conflict of Interests: Author
2. Adjusted the narrative order of Chapter 4