Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 isn't getting cheaper

matt-rickard.com

Post reply on HN
Print view View original

51–60 of 215 posts

Re: S3 isn't getting cheaper

#51

Post by hedora »

jenny91 wrote: post #26
hedora wrote: post #22

Earlier quoted context omitted.

They use wide erasure code stripes, so redundancy is ~ 1x within a data center. Let's assume 3 data centers. It's well known how to build a storage node whose cost is mostly disks, let's say 50% of the hardware is not disks. So, redundancy and server costs explain up to a 6x mark up. Power and networking really shouldn't account for the other 54x.

I agree with you but if you explain the 6x mark up, you only have 10x to explain out of 60x.

It depends on if you're comparing to raw disk, or the replicated storage, but yeah, spending 9x more on network and power than on machines is still pretty extreme.

Re: S3 isn't getting cheaper

#52

Post by ipaddr »

jonatron wrote: post #21
vorpalhex wrote: post #16

Earlier quoted context omitted.

DigitalOcean, Linode, Hetzner, Vultr...

DO's spaces didn't even work for the most basic test of putting a few images on a page when I tried it. I couldn't say it's comparable to S3.

Something tells me you are doing something wrong.

Re: S3 isn't getting cheaper

#53

Post by tnolet »

wmfiv wrote: post #42
tnolet wrote: post #24

For my company, neither egress nor storage cost are the big issue. It’s the API call (PUT) cost. We deal with payloads that are just a little too big for a database (we run Postgres and Clickhouse) but just too frequent (~100 per second) and small (think largish json blobs) to be effective on S3. We are write heavy. Reads are probably 1% but need to be instant for a good UI and API experience.

Have you looked at Kinesis Firehose? It was pretty much build for this use case although you will still need to see if you can define a partitioning scheme probably in combination with an S3 Select query to meet your query requirements. https://aws.amazon.com/kinesis/data-firehose/?nc=sn&loc=0 https://aws.amazon.com/blogs/aws/s3-glacier-select/

We are using Kinesis. It’s fine. Great actually. We still need to store user logs and generated data persistently. Cold storage is also not an option. This is data that needs to be accessible the moment the event that generates the data happened. Don’t want to push my product too much but I run a synthetic monitoring comopany. Check my bio and you’ll get a gist of the type of workloads.

Re: S3 isn't getting cheaper

#54

Post by iknownothow »

tnolet wrote: post #36
YawningAngel wrote: post #30

Earlier quoted context omitted.

This is a very naive question so I might be very wrong, but isn't postgres pretty flexible about objects now?

Yes, but it does blow up TOAST and has a lot of impact on the deletion behavior on busy tables. We removed all larger json blobs from PG. typical settings or config stuff in json in PG are fine. We use that all the time. But larger json blobs of several kilobytes are still an issue for semi timeseries data.

Could you elaborate on the TOAST issues you're having? We're pretty liberal with our use of large JSONB objects and might hit a billion objects in a year or so.

Re: S3 isn't getting cheaper

#56

Post by judge2020 »

impulser_ wrote: post #50
fxtentacle wrote: post #11

The main issue we have with S3 is the extortionary egress bandwidth fee. Storage pricing seems OK, but what's the point if I can't send those files to users?

Have you looked into other S3 compatible services? We switched to DO Spaces because of the lower bandwidth and storage fees. The savings was actually quite a bit and no noticeable differences for our use case. I know there are others services out there that are also s3 compatible and cheaper.

For my use case, DO Spaces was quite bad in terms of artifacts being served from stale caches after being replaced and the 'purge' API being called on the bucket; this was over 2 years ago so maybe it's better now.

Re: S3 isn't getting cheaper

#57

Post by aabhay »

Dunedan wrote: post #6

> Yet, AWS S3 pricing hasn't decreased as fast as the underlying storage costs. > […] > Another blog post analyzes the same theory for compute and finds a similar story using pricing data from AWS EC2. AWS used to do frequent price reductions years ago. At a certain point they seem to have stopped doing that and are now only doing them rarely. That's really a shame as there are still a lot of AWS offerings which are…

One of the benefits of inflation!

Re: S3 isn't getting cheaper

#58

Post by coder543 »

js4ever wrote: post #23
bicijay wrote: post #3

Lately im doing Backblaze B2 + Bunny CDN and im very satisfied with the prices

Blackblaze reliability and performance are below aws, same for bunny cdn. Although I understand it can be interesting for some use cases where perf/reliability is not critical.

Can you cite your sources?

AWS "reliability" has been the direct cause of a number of sleepless nights for me over the years. Comparing to a few years ago when I worked on a large-scale product hosted on bare metal servers that worked beautifully, I don't think AWS is all it is hyped up to be.

Anecdotal, I know, but even with no experience using Backblaze or Bunny, the bar they would have to meet is a lot lower than you're implying.

Re: S3 isn't getting cheaper

#59

Post by vorpalhex »

jonatron wrote: post #21
vorpalhex wrote: post #16

Earlier quoted context omitted.

DigitalOcean, Linode, Hetzner, Vultr...

DO's spaces didn't even work for the most basic test of putting a few images on a page when I tried it. I couldn't say it's comparable to S3.

I use DO spaces heavily - for images no less. It works great. What issues did you have?

Re: S3 isn't getting cheaper

#60

Post by vorpalhex »

upupandup wrote: post #46
vorpalhex wrote: post #16

Earlier quoted context omitted.

DigitalOcean, Linode, Hetzner, Vultr...

Which raises my question: Is there some open source github project that lets you create an S3 API compatible with underlying heterogeneous VPS hosts clustered? I'm guessing even with this storage would still be expensive, so how does Backblaze pull this off?

Yes, several. Minio is the big one but I believe Ceph can do this too. Others exist.

Disk is cheap. Real cheap.

Post reply on HN