Improving Parquet Dedupe on Hugging Face Hub
huggingface.co
Improving Parquet Dedupe on Hugging Face Hub
1–10 of 18 posts
Re: Improving Parquet Dedupe on Hugging Face Hub
#2Wouldn't be it easier to extend delta-rs to support deduplication?
Re: Improving Parquet Dedupe on Hugging Face Hub
#3Wouldn't be it easier to extend delta-rs to support deduplication?
Can you elaborate? As I understand Delta Lake provides transactions on top of existing data and effectively stores "diffs" because it knows what the transaction did. But when you have regular snapshots, its much harder to figure out the effective diff and that is where deduplication comes in. (Quite like how git actually stores snapshots of every file version, but very aggressively compressed).
Re: Improving Parquet Dedupe on Hugging Face Hub
#4I'm surprised that Parquet didn't maintain the Arrow practice of using mmap-able relative offsets for everything. Although these could be called relative to the beginning of the file.
Re: Improving Parquet Dedupe on Hugging Face Hub
#5I'm surprised that Parquet didn't maintain the Arrow practice of using mmap-able relative offsets for everything. Although these could be called relative to the beginning of the file.
I believe Parquet predates Arrow. That's probably why.
Re: Improving Parquet Dedupe on Hugging Face Hub
#6How does this compare to rsync/rdiff?
Re: Improving Parquet Dedupe on Hugging Face Hub
#7Love this post and the visuals! Great work
Re: Improving Parquet Dedupe on Hugging Face Hub
#8I just don't understand how these guys could literally give terrabytes of free storage and free data transfer to everyone. I was doing some calculation of cost from my storage and transfers and if they used something like S3 it would costed them 1000s of dollar. And I don't pay them anything.
Re: Improving Parquet Dedupe on Hugging Face Hub
#9How does this compare to rsync/rdiff?
Great question! Rsync also uses a rolling hash/content defined chunking approach to deduplicate and reduce communication. So it will behave very similarly.
Re: Improving Parquet Dedupe on Hugging Face Hub
#10I just don't understand how these guys could literally give terrabytes of free storage and free data transfer to everyone. I was doing some calculation of cost from my storage and transfers and if they used something like S3 it would costed them 1000s of dollar. And I don't pay them anything.
> As Hugging Face hosts nearly 11PB of datasets with Parquet files alone accounting for over 2.2PB of that storage
11PB on S3 would cost ~$250k per month / $3m per year.
HuggingFace has raised almost $400M.
Not saying it's nothing, but probably not a big deal to them (e.g. ~10 of their 400+ staff cost more).