Supertoast tables • Hacker News Board

Re: Supertoast tables

#11

Post by atombender » Sat, Mar 07, 2026, 12:21 AM UTC

huntaub wrote: post #8

atombender wrote: post #7

I really wish there was a seamless system for this. Once you try to do this kind of thing, you run into all sorts of rabbit holes and cans of worms. For example, coalescing blobs into "superblobs" to avoid a proliferation of small objects means you invent a whole system for tracking "subfiles" within a bigger file. And you'll need a compacting job to ensure old, deleted data is expunged, which may be more important t…

Well, I think this is what our company, Archil, is working on. We basically built an SSD clustering layer that proxies/caches/and assembles requests into object storage so that you can run a POSIX file system directly on top. There's also some really great projects like SlateDB in this space, which could be more like what you're looking for (~RocksDB like API that runs on S3).

Your product looks very interesting, I will take a look!

Re: Supertoast tables

#12

Post by carderne » Sat, Mar 07, 2026, 7:21 AM UTC

abelanger wrote: post #5

carderne wrote: post #4

How does this work with self-hosting? Is the assumption that self-hosters won’t run into this problem? For most use-cases I’d probably prefer to just delete the payloads some time after the job completes (persisting that data is business logic problem). And keep the benefits of “just use Postgres”, which you guys seem to have outgrown.

Candidly we're still trying to figure that out: all of the plumbing is there in the open source, but the actual implementation of writes to S3 are only on the cloud version. This is partially because we're loath to introduce additional dependencies, and partially because this job requires a decent amount of CPU and memory and would have to run separate from the Hatchet engine, which adds complexity to self-hosted set…

I guess you don't get the luxury of being opinionated enough to say: forget your old data.

Anyway great write-up, even though I'm sure it's painful having to run this system on top of your once-elegant Postgres solution.

Re: Supertoast tables

#13

Post by johnthescott » Wed, Mar 11, 2026, 7:02 AM UTC

no need to store blob in database. too expensive.

why is blob storage even a "problem"? a crypto digest of the blob as a key covers vast majority of cases.

for blobs that do not change store only the crypto hash (160bits) of the large blob in the database?

    ripemd(sha256(sha256(blob)))

for the blobs themselves use any object store you please.

yawn.