Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache iceberg the Hadoop of the modern-data-stack?

blog.det.life

Post reply on HN
Print view View original

21–30 of 67 posts

Re: Apache iceberg the Hadoop of the modern-data-stack?

#21

Post by foobiekr »

twoodfin wrote: post #16
datadrivenangel wrote: post #12

Earlier quoted context omitted.

The only reason Microsoft is using Delta is to emphasize to CTOs and investors that fabric is as good as databricks, even when that is obviously false to anyone who has smelled the evaporative scent of vaporware before.

Very different business, of course, but Databricks v. Fabric reminds me a lot of Slack v. Teams. Regardless of the relative merits now, I think everyone agrees that a few years ago Slack was clearly superior. Microsoft could have certainly bought Slack instead of pumping probably billions into development, marketing, discounts to destroy them. I think Microsoft could and would consider buying Databricks—$80–100B is a…

Anti-trust is the reason a lot of the kinds of deals you’re talking about don’t happen.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#24

Post by bloomingkales »

Gasp0de wrote: post #8
paulsutter wrote: post #6

Earlier quoted context omitted.

If you want to keep them in S3, consolidate into sorted parquet files. You get random access to row groups, and only the columns you need are read so it’s very efficient. DuckDB can both build and access these files efficiently. You could compact files hourly/nightly/weekly whatever Of course you could also use Aurora for a clean scalable Postgres that can survive zone failures for a simpler solution

The problem is that the initial writing is already so expensive, I guess we'd have to write multiple sensors into the same file instead of having one file per sensor per interval. I'll look into parquet access options, if we could write 10k sensors into one file but still read a single sensor from that file that could work.

Something like Redis instead? [sensorid-timerange] = value. Your key is [sensorid-timerange] to get the values for that sensor and that time range.

No more files. You might be able to avoid per usage pricing just by hosting this on a regular vps.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#25

Post by simlevesque »

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb files, etc...

I feel like this could be a game changer for the ecosystem. It's more cpu and network heavy for writes but the reads are always fast. And the writes are still faster than pyiceberg.

I want to hear opinions or how this could never work.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#26

Post by necrobrit »

spothedog1 wrote: post #11
Gasp0de wrote: post #8

Earlier quoted context omitted.

The problem is that the initial writing is already so expensive, I guess we'd have to write multiple sensors into the same file instead of having one file per sensor per interval. I'll look into parquet access options, if we could write 10k sensors into one file but still read a single sensor from that file that could work.

New S3 Table Buckets [1] do automatic compaction [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tab...

Table buckets are currently quite hard to use for a lot of use cases as they _only_ support primitive types. No nested types.

Hopefully this will come at some point. Product looks very cool otherwise.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#27

Post by Gasp0de »

this_user wrote: post #17
Gasp0de wrote: post #4

Does anyone have a good alternative for storing large amounts of very small files that need to be individually queriable? We are dealing with a large amount of sensor readings that we need to be able to query on a per sensor basis and a timespan, and we are dealing with the problem mentioned in the article, that storing millions of small files in S3 is expensive.

Do you absolutely have to write the data to files directly? If not, then using a time series database might be the better option. Most of them are pretty much designed for workloads with large numbers of append operations. You could always export to individual files later on if you need it. Another option if you have enough local storage would be to use something like JuiceFS that creates a virtual file system where…

We tried some readymade options but they were way more expensive than our custom built S3 solution (by a factor of x10 approximately). I think we tried timescale and AWS Timestream. I haven't heard of SeaweedFS.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#28

Post by Gasp0de »

bloomingkales wrote: post #24
Gasp0de wrote: post #8

Earlier quoted context omitted.

The problem is that the initial writing is already so expensive, I guess we'd have to write multiple sensors into the same file instead of having one file per sensor per interval. I'll look into parquet access options, if we could write 10k sensors into one file but still read a single sensor from that file that could work.

Something like Redis instead? [sensorid-timerange] = value. Your key is [sensorid-timerange] to get the values for that sensor and that time range. No more files. You might be able to avoid per usage pricing just by hosting this on a regular vps.

We use Redis for buffering for a certain timeperiod, and then we write data for one sensor for that period to S3. However we fill up large Redis clusters pretty fast, so we can only buffer for a shortish period.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#29

Post by thom »

simlevesque wrote: post #25

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb f…

Interesting. My personal feeling is that we're slowly headed to a world where we can have our cake and eat it: fast bulk ingestion, fast OLAP, fast OLTP, low latency, all together in the same datastore. I'm hoping we just get to collapse whole complex data platforms into a single consistent store with great developer experience, and never look back.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#30

Post by mritchie712 »

This is a bit overblown.

Is Iceberg "easy" to set up? No.

Can you get set up in a week? Yes.

If you really need a datalake, spending a week setting it up is not so bad. We have a guide[0] here that will get you started in under an hour.

For smaller (e.g. under 10tb) data where you don't need real-time, DuckDB is becoming a really solid option. Here's on setup[1] we've played around with using Arrow Flight.

If you don't want to mess with any of this, we[2] spin it all up for you.

0 - https://www.definite.app/blog/cloud-iceberg-duckdb-aws

1 - https://www.definite.app/blog/duck-takes-flight

2 - https://www.definite.app/

Post reply on HN