How to process 100gb tsv and XML files?

#1

Post by anindha » Wed, Aug 28, 2024, 4:12 PM UTC

I am trying to parse a music data file that is close to 100gb. What app or programming language is best for handling a file like this?

Thanks!

Re: How to process 100gb tsv and XML files?

#2

Post by anindha » Wed, Aug 28, 2024, 4:30 PM UTC

I found this https://klogg.filimonov.dev

Re: How to process 100gb tsv and XML files?

#3

Post by datadrivenangel » Wed, Aug 28, 2024, 8:42 PM UTC

What kind of single music data file is 100gb?

Also how is it structured? If it's actually a tab separated value file, consider using something like polars or DuckDB?

Re: How to process 100gb tsv and XML files?

#4

Post by FlyingAvatar » Wed, Aug 28, 2024, 10:05 PM UTC

It really depends on what you need to do with the data, but in most cases Python could do this pretty easily with csv.reader (with a \t delimiter for TSV) or xml.etree.ElementTree.iterparse (for XML) in streaming fashion such that you're not loading the whole file at once.

Re: How to process 100gb tsv and XML files?

#5

Post by abdusco » Thu, Aug 29, 2024, 12:04 PM UTC

For TSV, you might wanna consider importing it into a Sqlite database, then querying it however you please.

https://stackoverflow.com/a/35454070/5298150

You can also use datasette & sqlite utils for it

https://sqlite-utils.datasette.io/en/stable/cli.html#inserti...

Re: How to process 100gb tsv and XML files?

#6

Post by mobilio » Thu, Aug 29, 2024, 7:50 PM UTC

Great solution is DuckDB: https://duckdb.org/docs/data/csv/overview.html

Re: How to process 100gb tsv and XML files?

#7

Post by pradeepchhetri » Fri, Aug 30, 2024, 3:14 AM UTC

You can leverage ClickHouse to process your music data. ClickHouse supports both TSV[0] and XML[1] data formats.

[0] https://clickhouse.com/docs/en/interfaces/formats#tabseparat...

[1] https://clickhouse.com/docs/en/interfaces/formats#xml