Version 1
: Received: 15 February 2024 / Approved: 15 February 2024 / Online: 15 February 2024 (15:41:17 CET)
Version 2
: Received: 16 February 2024 / Approved: 16 February 2024 / Online: 16 February 2024 (11:04:53 CET)
Version 3
: Received: 16 February 2024 / Approved: 16 February 2024 / Online: 17 February 2024 (09:06:25 CET)
Version 4
: Received: 19 February 2024 / Approved: 20 February 2024 / Online: 20 February 2024 (09:09:43 CET)
Version 5
: Received: 22 February 2024 / Approved: 23 February 2024 / Online: 24 February 2024 (08:49:48 CET)
Version 6
: Received: 14 March 2024 / Approved: 17 March 2024 / Online: 18 March 2024 (07:25:57 CET)
How to cite:
García, W. Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference. Preprints2024, 2024020858. https://doi.org/10.20944/preprints202402.0858.v6
García, W. Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference. Preprints 2024, 2024020858. https://doi.org/10.20944/preprints202402.0858.v6
García, W. Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference. Preprints2024, 2024020858. https://doi.org/10.20944/preprints202402.0858.v6
APA Style
García, W. (2024). Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference. Preprints. https://doi.org/10.20944/preprints202402.0858.v6
Chicago/Turabian Style
García, W. 2024 "Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference" Preprints. https://doi.org/10.20944/preprints202402.0858.v6
Abstract
The human-readable simplicity with which the CSV format was devised, together with the absence of a standard that strictly defines this format, has allowed the proliferation of several variants in the dialects with which these files are written. The latter has meant that the exchange of information between data management systems, or between countries and regions, requires human intervention during the data mining and cleansing process. This has led to the development of various computational tools that aim to accurately determine the dialects of CSV files, in order to avoid data loss at data loading stage in a given system. However, the dialect detection is a complex problem and current systems have limitations or make assumptions that need to be improved and/or extended. This paper proposes a method for determining CSV file dialects through table uniformity, a statistical approach based on table consistency and records dispersion measurement along with the detection of data type over each field. The new method has a 100\% accuracy on a dataset with 148 CSV files composed of samples coming from a data load testing framework and some others added as verification of the parsing routines. In tests on truly messy data, the proposed solution outperforms the state-of-the-art tool by achieving an improvement of about 10\% in the accuracy with which dialects are detected. Furthermore, the proposed method is accurate enough to determine dialects by reading only ten records, requiring more data to disambiguate those cases where the first records do not contain the necessary information to conclude with a dialect determination.
Keywords
comma separated values; CSV dialect detection; data mining
Subject
Computer Science and Mathematics, Data Structures, Algorithms and Complexity
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
