Viewing profile — Sirupsen

Yep! Other than random reads (~p99=200ms on larger ranges), it's essential to get good download performance of a single file. A single (range) request can "only" drive ~500 MB/s, s…

Due to the batching, this will only consume a few million class B per month. They are $5/million

(cofounder of tpuf here) We don't have a relational database, otherwise that would work great for a queue! You can imagine us continuing to iterate here to Step 5, Step 6, ... Step…

Our query planner has that built in! We've spent a lot of time making high recall with any selectivity in the fitler work.

The short of it is that building a database on top of object storage has generally required a complicated, distributed system for consensus/metadata. CAS makes it possible to build…

To avoid any dependencies other than object storage, we've been making use of this in our database (turbopuffer.com) for consensus and concurrency control since day one. Been waiti…

It works great. We’ve had SPANN in production since October of 2023 at https://turbopuffer.com/

tpuf’s ANN index uses a variant of SPFresh, yup. These are the only two production implementations I am aware of. I don’t think it is in production at MSFT yet

Ya, the world needed S3 to become fully consistent. This didn't happen until end of 2020!

Yeah, thinking about this more I now understand Clickhouse to be more of an operational warehouse similar to Materialize, Pinot, Druid, etc. if I understand correctly? So bunching …

Most production storage systems/databases built on top of S3 spend a significant amount of effort building an SSD/memory caching tier to make them performant enough for production …

Emil if you email me at info@turbopuffer.com I can let you into the alpha :)

While I never owned a VanMoof, I read they painted pictures of flat screen TVs on the shipping boxes after months of issues with bikes being damaged by the time they got to custome…

Are you aware of a good write-up on how Clickhouse/other columnar databases do the intersection?

MySQL can do skip-scans https://dev.mysql.com/doc/refman/8.0/en/range-optimization.h...

Ideally, I would add three graphs to the post: (1) Table size on the x-axis, and time on the y-axis for index merge vs composite index (2) Number of columns on the x-axis, and time…

You're absolutely right! I forgot to move this around when I updated the article's structure. This is only relevant when doing the index merge. The article has been updated

Excellent, thank you! I'll add that to the article.

I've added this to the article, thanks! Composite index (int64, int64): ~70 MiB in Postgres, ~350 MiB in MySQL Single index (int64): ~70 MiB in Postgres, ~240 MiB in MySQL If you a…