Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Supabase Storage now supports the S3 protocol

supabase.com

Post reply on HN
Print view View original

141–150 of 194 posts

Re: Supabase Storage now supports the S3 protocol

#141

Post by ovaistariq »

kiwicopple wrote: post #31
filleokus wrote: post #29

Do you think Supabase Storage (now or in the future) could be an attractive standalone S3 provider as an alternative to e.g MinIO?

It's more of a "accessibility layer" on top of S3 or any other s3-compatible backend (which means that it also works with MinIO out-of-the-box [0]) I don't think we'll ever build the underlying storage layer. I'm a big fan of what the Tigris[1] team have built if you're looking for other good s3 alternatives [0] https://github.com/supabase/storage/blob/master/docker-compo... [1] Tigris: https://tigrisdata.com

I appreciate the callout to Tigris. Now it’s super easy to use Tigris with Supabase for all those files.

Re: Supabase Storage now supports the S3 protocol

#142

Post by ovaistariq »

crubier wrote: post #121
bdcravens wrote: post #111

Earlier quoted context omitted.

S3 is one of the original AWS services (SQS predates it), and has been around for 18 years. The idea of a propriety API becoming the industry defacto standard isn't uncommon. The same thing happened with Microsoft's XMLHttpRequest.

Also, the S3 API is simple and makes sense, no need to reinvent something different just for the pleasure of it

There are valid reasons for extending and redoing some parts of the API. I will give you one example. Suppose you want to extend list objects to support ordering by last modified or you want to support filtering of objects by user metadata. Right now doing this is quite clunky via headers.

Re: Supabase Storage now supports the S3 protocol

#143

Post by ihateolives »

kiwicopple wrote: post #136
replwoacause wrote: post #135

Earlier quoted context omitted.

I was pretty unhappy with the self hosted offering. It’s neutered compared to the cloud, which was disappointing.

> neutered Can you share more? There is nothing missing, it’s all the same code that we run in production (besides some login changes to keep it decoupled from the platform)

From top of my head: reports and multiple projects. Also I kept getting weird "fetch failed" errors when creating rules or adding users from gui working behind reverse proxy so I gave up on the end.

Re: Supabase Storage now supports the S3 protocol

#144

Post by replwoacause »

kiwicopple wrote: post #136
replwoacause wrote: post #135

Earlier quoted context omitted.

I was pretty unhappy with the self hosted offering. It’s neutered compared to the cloud, which was disappointing.

> neutered Can you share more? There is nothing missing, it’s all the same code that we run in production (besides some login changes to keep it decoupled from the platform)

Sure. I wouldn't quite agree with you that "there is nothing missing" though.

https://github.com/orgs/supabase/discussions/4444 https://github.com/orgs/supabase/discussions/4421

Re: Supabase Storage now supports the S3 protocol

#146

Post by vbezhenar »

Rapzid wrote: post #57

I like to Lob my BLOBs into PG's storage. You need that 1-2TB of RDS storage for the IOPS anyway; might as well fill it up. Large object crew, who's with me?!

I don't. S3-compatible storages usually are significantly cheaper, allow to offload HTTP requests. Also huge databases make backups and recoveries slow.

The only upside of storing blobs in the database is transactional semantics. Buf if you're fine with some theoretical trash in S3, that's trivially implemented with proper ordering.

Re: Supabase Storage now supports the S3 protocol

#147

Post by inian »

pier25 wrote: post #145

At $0.1/GB of egress it’s not super attractive compared to B2 or R2 for anything but trivial projects. I wish they would offer a plan with just the pg database. Any news on pricing of Fly PG?

We are hosted on aws and are just passing the cost over to our users. We make no margin on egress fees. Deploying storage on other clouds including Fly.io is planned.

We are actively working on our Fly integration. At the start, the pricing is going to be exactly the same as our hosted platform on aws - https://supabase.com/docs/guides/platform/fly-postgres#prici...

Re: Supabase Storage now supports the S3 protocol

#148

Post by inian »

tln wrote: post #133
kiwicopple wrote: post #58

Earlier quoted context omitted.

that's not how this works. files are stored in s3, metadata in postgres

Will the files get deleted with ON CASCADE DELETE somehow? That would be awesome. Then for GDPR, when you delete a user, the associated storage can be deleted. One could cobble this together with triggers, some kind of external process, and probably repetititious code so there is one table of metadata per "owning" id, although it would be nice to be packaged.

We have discussed this internally before since we have seen some users delete the metadata in the storage schema and expect the underlying object to be deleted too and if we should convert our entire storage server to just be a Postgres extension.

The source of truth also matters here - if it's the database or the underlying s3 bucket. I think having the underlying storage bucket to be the source of truth would be more useful. In that scenario we would sync the metadata in the database to match what's actually being stored and if we notice metadata of a object missing, we add that in as opposed to deleting the object in storage. This would make it easier for you to bring in your own s3 bucket with existing data and attach it to Supabase storage.

Re: Supabase Storage now supports the S3 protocol

#149

Post by inian »

SOLAR_FIELDS wrote: post #88
fenos wrote: post #48

Earlier quoted context omitted.

Our S3 Driver is compatible with any S3 Compatible object Storage so you don’t have to write one :)

Gentle reminder here that S3 compatability is a sliding window and without further couching of the term it’s more of a marketing term than anything for vendors. What do I mean by this statement? I mean that you can go to cloud vendor Foo and they can tell you they offer s3 compatible api’s or clients but then you find out they only support the most basic of operations, like 30% of the API. Vendor Bar might support 50…

I agree that s3 compatibility is a bit of a moving target and we would not implement any of the AWS specific actions.

We are transparent with what's the level of compatibility - https://supabase.com/docs/guides/storage/s3/compatibility

The most often used APIs are covered but if something is missing, let me know!

Post reply on HN