Preprint Technical Note Version 1 Preserved in Portico This version is not peer-reviewed

AWS-S3-Integrity-Check: An Open-Source Bash Tool to Verify the Integrity of a Dataset Stored on Amazon S3

Sonia García-Ruiz
ORCID logo ,
Regina H Reynolds
,
Melissa Grant-Peters
,
Emil K Gustavsson
,
Aine Fairbrother-Browne
,
Zhongbo Chen
,
Jonathan W Brenton
,
Mina Ryten
*
Version 1 : Received: 7 August 2023 / Approved: 7 August 2023 / Online: 8 August 2023 (14:12:01 CEST)

How to cite: García-Ruiz, S.; H Reynolds, R.; Grant-Peters, M.; K Gustavsson, E.; Fairbrother-Browne, A.; Chen, Z.; W Brenton, J.; Ryten, M. AWS-S3-Integrity-Check: An Open-Source Bash Tool to Verify the Integrity of a Dataset Stored on Amazon S3. Preprints 2023, 2023080603. https://doi.org/10.20944/preprints202308.0603.v1 García-Ruiz, S.; H Reynolds, R.; Grant-Peters, M.; K Gustavsson, E.; Fairbrother-Browne, A.; Chen, Z.; W Brenton, J.; Ryten, M. AWS-S3-Integrity-Check: An Open-Source Bash Tool to Verify the Integrity of a Dataset Stored on Amazon S3. Preprints 2023, 2023080603. https://doi.org/10.20944/preprints202308.0603.v1

Abstract

Amazon Simple Storage Service (Amazon S3) has become a widely used and reliable platform for storing large biomedical datasets. However, unintended changes to the original data can occur during the data writing and transmission, ultimately altering the original contents of the object transferred and producing unexpected results when later accessed. Despite the interest in verifying end-to-end data integrity, there are no existing open-source and easy-to-use tools to accomplish this mission. To bridge this gap, here we present aws-s3-integrity-check, a user-friendly, lightweight and reliable bash tool to verify the integrity of a dataset stored within an Amazon S3 bucket. By using this tool, we completed the integrity verification of 1,045 records ranging between 5 Bytes and 10 Gigabytes (GB) in size and occupying a total of ~935 GigaBytes (GB) of Amazon S3 cloud storage space in ~114 minutes. The aws-s3-integrity-check tool also provides file-by-file on-screen and log-file-based information about the status of each individual integrity check. To the best of our knowledge, the aws-s3-integrity-check bash tool is the only open-source tool that allows verifying the integrity of a dataset uploaded to the Amazon S3 Storage system in a quick, reliable and efficient manner. The aws-s3-integrity-check tool is freely available for download and use at https://github.com/SoniaRuiz/aws-s3-integrity-check and https://hub.docker.com/r/soniaruiz/aws-s3-integrity-check.

Keywords

AWS; Amazon S3; MD5 Checksum; File Integrity; Bash Script; Bash Tool; ETag; Amazon S3 ETag; multipart upload; EGA; GigaDB

Subject

Computer Science and Mathematics, Software

Copyright: This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download PDF

Comments (0)

We encourage comments and feedback from a broad range of readers. See criteria for comments and our Diversity statement.

Leave a public comment
Send a private comment to the author(s)
* All users must log in before leaving a comment
Views 0
Downloads 0
Comments 0
Metrics 0
Get PDF Cite Share 0
Bookmark BibSonomy Mendeley Reddit Delicious
×
Alerts
Notify me about updates to this article or when a peer-reviewed version is published.
We use cookies on our website to ensure you get the best experience.
Read more about our cookies here.