Live data from Hacker News

AWS Acquires DuckLabs

ducklabs.com

191–200 of 293 posts

Re: AWS Acquires DuckLabs

#191
post #157
post #147

I'm an idiot, and a lot of Hacker News goes over my head, but I still read it. Would anyone mind answering this question? Are databases not a solved problem? Why are there lots of different databases? Why is one faster than the other? What's different between them?

It's all about tradeoffs. Same reason there are so many kinds of wheeled transport and so many kinds of bridges. There's no single set of requirements and desired properties that people have for databases. What queries does it accept? How does it persist data? How does it manage replication and partitioning across multiple servers? Are questions with many answers and the right one varies by application.

> What queries does it accept? How does it persist data? How does it manage replication and partitioning across multiple servers? Are questions with many answers and the right one varies by application.

nothing prevents to build single database which would cover all such answers. Its engineering, funding and distribution problems: no-one built it yet.

Re: AWS Acquires DuckLabs

#192

Earlier quoted context omitted.

>...sustainable engineering, correctness, and keeping DuckDB open and MIT-licensed for everyone Hah. Never have I ever heard a sentence that described Amazon less.

what do you mean? amazon maintains a ton of open source projects, including a pure FOSS version of Elasticsearch called Opensearch

Top-level troll: that's not a "version" of Elastic Search, it's a fork of ES specifically created due to ES's licence change that made it more difficult for AWS to extract money from it at Elastic's expense.

Not that I like either actor in this particular play, but this isn't exactly an example of community oriented goodwill on Amazon's part.

Re: AWS Acquires DuckLabs

#195

What's the relationship between MotherDuck ( https://motherduck.com ) and DuckLabs?

TL;DR AWS will either acquire MD next/soon, or (more likely) keep them around for the optics that they are totally not going to starve the community of anything other than DuckDB as a hosted AWS service.

From the MD "about page":

The idea for MotherDuck came after Jordan Tigani, MotherDuck co-founder and chief duck-herder, saw DuckDB in action and thought, “Wow, this is amazing! someone should really build a serverless version.”

...

Hannes and Mark, who founded DuckLabs to focus on the core technology and build the world’s best analytics database, were looking for partners who would build a commercial cloud offering.

Re: AWS Acquires DuckLabs

#196

I wasn't expecting this, and I am pleasantly surprised. I think AWS has been a bad player in the data space and very much behind all the other warehouse providers for many years now. This could lead into interesting integrations of DuckDB on top of all the data that lives in S3 and provide a leaner, faster and a more modern alternative to Athena.

Feel like AWS got a lot of the concepts right with S3 Tables* ("serverless" Iceberg catalog on top of S3) ... would love, love to see a version of S3 Tables that lets you choose between Ducklake or Iceberg as the catalog.

* Cloudflare also seems to have gotten the same idea right with R2 Data Catalog

Re: AWS Acquires DuckLabs

#197
post #70

Earlier quoted context omitted.

Yup, they are very serious about DynamoDB and it is a very good product. Wish they released more information about it, though.

Man I admire AWS marketing team for convincing people DynamoDB is a very good product. 100 items max per Transaction BatchGet 100 items, 16 MB max low write limits on same key Item size 400 KB max etc.

Thanks, I don't really get any chance to seriously use or manage OLTP databases (mostly working with OLAP ones myself) so it is good to know about this. The 400KB limit is indeed very limiting -- I had to truncate some data because of that.

I wish I could get a role to work on OLTP databases. PostgreSQL seems to be a fascinating topic so that's on my plate.

Re: AWS Acquires DuckLabs

#198
post #147

I'm an idiot, and a lot of Hacker News goes over my head, but I still read it. Would anyone mind answering this question? Are databases not a solved problem? Why are there lots of different databases? Why is one faster than the other? What's different between them?

It's a fair question!

My attempt at an answer: no, databases are not solved.

Specifically, different databases are better or worse for different use-cases.

For example, Postgres is a great "all around database" - you can use it for a lot of different things. As a "relational" database, it's really good if you have a table full of users, a table full of order, and you want to see all orders made by a user with ID=123. You need to answer questions like that a lot (eg every time someone on a website loads a page) and you need the answer fast (hundreds of miliseconds at most)

However, say your use-case is more like... you've got 100 billion rows of billing data ("joe was charged $123.45 on 2026-03-07 for a shirt, blue, size 11, brand foobar") in one table. You don't care much about joe, but you want to be able to find out how much was billed, total, in 2026-03 for blue shirts (or all year for brand foobar, or all time, for size 11). Postgres would struggle with data of that volume - you'd need a really big expensive database. A "columnar" database like duckdb (or clickhouse) might be able to answer those questions better.

Anyway, different databases are better/worse for:

- Large piles of data that you need to query in seconds

- Huge (petabytes) of data that you need to query in minutes, but can query in parallel

- Many related piles (like a standard relational database)

- Cases where you're mostly getting or retrieving single items (key-value stores)

- Huge piles of data that represent a long stream of events in time (time-series datbases)

- Piles of data that look and act more like files (object stores)

- When you need strict transactions

- When your need is very write-heavy

- When your need is very read-heavy

- and probably many others - I'm not even a huge data guy :)

So it all depends on your use-case. There are still cases that are not served well by any existing database - eg "filtering billions of rows, in milliseconds, by an arbitrary portion of several dozen very-high-cardinality columns" (to use an example that came up recently for me IRL) :)

Re: AWS Acquires DuckLabs

#199
post #7

Maybe they learned AWS were planning on releasing a managed clone without their involvement?

That seems unlikely because AWS needs to build proprietary features within their services to lock in paying (enterprise) users. The problem now is that labs will inevitably focus on this work, which I view as opposed to local, in-process when your paycheque comes from selling hosted services.

Re: AWS Acquires DuckLabs

#200
post #70

Earlier quoted context omitted.

Yup, they are very serious about DynamoDB and it is a very good product. Wish they released more information about it, though.

Man I admire AWS marketing team for convincing people DynamoDB is a very good product. 100 items max per Transaction BatchGet 100 items, 16 MB max low write limits on same key Item size 400 KB max etc.

Some of the largest, highest TPS, highest throughput systems in the world use DDB.

The constraints are what let this happen. Unconstraining it might make a better generalist product but part of what you're opting into with DDB is the dumb "put an item in get an item out semantics" and the other side is knowing that it will still work if that volume increases dramatically.

Post reply on HN