51–60 of 67 posts
This article is just shameless advertising for Estuary Flow, a company that the author is working for. "Operational Maturity", as if Iceberg, Delta or Hudi are not mature. These are battle-tested frameworks that have been in production for years. The "small files problem" is not really a problem because every framework supports some way of compacting smaller files. Just run a nightly job that compacts the small files…
This is a bit overblown. Is Iceberg "easy" to set up? No. Can you get set up in a week? Yes. If you really need a datalake, spending a week setting it up is not so bad. We have a guide[0] here that will get you started in under an hour. For smaller (e.g. under 10tb) data where you don't need real-time, DuckDB is becoming a really solid option. Here's on setup[1] we've played around with using Arrow Flight. If you don…
If you're already in AWS, why wouldn't you use AWS Glue Catalog + AWS SDK for pandas + Athena? You can setup a data lake, save data and start doing queries in like 10 minutes with this setup.
This pain is too real, and too close to home. I've seen this outcome turn the entire business off of consuming their data via hadoop because it turns into a wasteland of delayed deliveries, broken datasets, op's teams who cannot scale, and architects overselling too robust designs.
I've tried to scale down hadoop to the business user with visual etl tools like Alteryx, but there again compatibility between Alteryx and hadoop suck via ODBC connectors. I came from an AWS based stack into a poorly leapfrogged data stack and it's hard not to pull my hair out between the business struggling to use it and infra + op's not keeping up. Now these teams want to push to iceburg or big query while ignoring the mountains of tech debt they have created.
Don't get me wrong Hadoop isn't a bad idea, its just complex and a time suck, and unless you have time to dedicate to properly deploy these solutions which most business do not, your implementation will suffer, your business will suffer.
"While the parallels to Hadoop are striking, we also have the opportunity to avoid its pitfalls." no one in IT learns from their failures unless they are writing the checks, most will flip before they feel the pain.
This is a bit overblown. Is Iceberg "easy" to set up? No. Can you get set up in a week? Yes. If you really need a datalake, spending a week setting it up is not so bad. We have a guide[0] here that will get you started in under an hour. For smaller (e.g. under 10tb) data where you don't need real-time, DuckDB is becoming a really solid option. Here's on setup[1] we've played around with using Arrow Flight. If you don…
heads up the logo on your site needs to be 2x'd in pixel density it comes across as blurry on hidpi displays. or convert it to an svg/vector.
Earlier quoted context omitted.
Do you absolutely have to write the data to files directly? If not, then using a time series database might be the better option. Most of them are pretty much designed for workloads with large numbers of append operations. You could always export to individual files later on if you need it. Another option if you have enough local storage would be to use something like JuiceFS that creates a virtual file system where…
We tried some readymade options but they were way more expensive than our custom built S3 solution (by a factor of x10 approximately). I think we tried timescale and AWS Timestream. I haven't heard of SeaweedFS.
https://github.com/seaweedfs/seaweedfs/wiki/Cloud-Drive-Bene...
https://github.com/seaweedfs/seaweedfs/wiki/Cloud-Tier
https://github.com/seaweedfs/seaweedfs/wiki/Benchmarks
https://github.com/seaweedfs/seaweedfs/wiki/Words-from-Seawe...
https://github.com/seaweedfs/seaweedfs/wiki/Amazon-S3-API
...your true issue is it seems like you're using the filesystem as the "only" storage layer in play, but you also need time and entity querying(!?!).
>> we need to be able to query on a per sensor basis and a timespan
...look at the "Cloud-Tier" wiki page. If you're truly in an "everything's hot all the time" situation, you really should be using a database. If you're pulling "usually recent stuff, occasionally old stuff" then fronting with something like SeaweedFS seems like it might "just" transparently reduce your overall costs.
Really, I'd nudge towards "write .txt ; compact ... ; SELECT ... && cat .txt".
Basically, keep your inbound writes cached to (eg) seaweed as unit files. "Compact them" every hour by appending rows to some appropriate database (I mean: migrate to using litefs, turso, postgres, something like that). When you read, you may need to supplement "tip" data from your incoming files, but the majority should be hitting a "real" remote database, there's plenty to choose from!
A nifty note, sqlite can connect to multiple DB's at once: https://www.sqlite.org/lang_attach.html ... https://stackoverflow.com/posts/10020/revisions
...something like `select * from raw union (select * from one_hour) union (select * from today) union (select * from historical) ...`
Earlier quoted context omitted.
If you want to keep them in S3, consolidate into sorted parquet files. You get random access to row groups, and only the columns you need are read so it’s very efficient. DuckDB can both build and access these files efficiently. You could compact files hourly/nightly/weekly whatever Of course you could also use Aurora for a clean scalable Postgres that can survive zone failures for a simpler solution
The problem is that the initial writing is already so expensive, I guess we'd have to write multiple sensors into the same file instead of having one file per sensor per interval. I'll look into parquet access options, if we could write 10k sensors into one file but still read a single sensor from that file that could work.
Do you ever go back and reaggregate older data into bigger, sorted files? That is, maybe you originally partitioned by hour, but stale data is so infrequently accessed, you could roll up into partitions per week/month/whatever. Depending on the specifics, you might save some space from less file overhead and better compression statistics.
Does anyone have a good alternative for storing large amounts of very small files that need to be individually queriable? We are dealing with a large amount of sensor readings that we need to be able to query on a per sensor basis and a timespan, and we are dealing with the problem mentioned in the article, that storing millions of small files in S3 is expensive.
You can also do this with a landing table or even branches+WAP.
This is a huge challenge with Iceberg. I have found that there is substantial bang for your buck in tuning how parquet files are written, particularly in terms of row group size and column-level bloom filters. In addition to that, I make heavy use of the encoding options (dictionary/RLE) while denormalizing data into as few files as possible. This has allowed me to rely on DuckDB for querying terabytes of data at low…
Earlier quoted context omitted.
If you're already in AWS, why wouldn't you use AWS Glue Catalog + AWS SDK for pandas + Athena? You can setup a data lake, save data and start doing queries in like 10 minutes with this setup.
Athena is really expensive though and you will often run into a hard limit on the size of your query.
And some times, if your query is CPU extensive but the queried data size is not huge you can get a ridiculous value for money, like many CPU-days in 10 minutes for just $5 if your query covers 1TB after partitioning.
Query size limits are also configurable.
Obviously it depends on what data you are working on, but not having to set up and pay for a computational cluster is a huge cost saving.