Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to: Compete with Amazon S3 without Buying Hardware

blog.adamsmith.cc

Post reply on HN
Print view View original

31–40 of 40 posts

Re: How to: Compete with Amazon S3 without Buying Hardware

#31

Post by mleonhard »

A big cost of running a redundant data storage service is data transfer.

To store two replicas of each piece of data, you must receive the data at one replica, transmit it to the other replica, and receive it at that replica. The data goes in at one server, then back out, and then in at the other server. To store 1 GB of data, you must pay for 3 GB of data transfer. Data transfer is expensive.

Amazon works around this problem by building data centers in clusters, interconnected with low-cost connections. When you upload to S3, your data goes over the Internet only once.

Re: How to: Compete with Amazon S3 without Buying Hardware

#32

Post by kozubik »

driverdan wrote: post #17

I've been watching the storage industry for years as a hobby-passion. Adam pretty much hit all the major points. The storage space is still open for disruption but is hard with high risk. It's not like building a website. You need serious funding for hardware. You need people to manage the hardware. You need complex software to manage the data and ensure security. One serious breach early on and you're done. Competin…

We've[1] been doing this for 11 years now, just as you describe. We built the bare metal ourselves, we own it, and the buck stops here.

Most importantly, unlike the OP who speaks of "the big hosting guys don't have a track record of building complex systems software" and your own post speaking of "complex software", we run an architecture that is as simple as possible.

Our failures are always boring ones, just like our business is.

You are correct that a chain of vendors, ending in a behemoth[2] that nobody will ever interact with, and will never take responsibility, is a bad model.

So too is a model whose risk you cannot assess. You have no idea how to model the risk of data in an S3 container. You can absolutely model the risk of data in a UFS filesystem running on FreeBSD[3].

[1] rsync.net

[2] Amazon

[3] ZFS deployment occurs in May, 2012

Re: How to: Compete with Amazon S3 without Buying Hardware

#33

Post by stevewilhelm »

> Amazon S3 has high margins today. ...

> ... despite the fact that hard drive costs fall 50% per year.

Citations for both statements please.

Even if both are true, it may be the case that hard drives are not the primary cost of running a large cloud storage service.

Re: How to: Compete with Amazon S3 without Buying Hardware

#34

Post by toomuchtodo »

mbell wrote: post #29
driverdan wrote: post #17

I've been watching the storage industry for years as a hobby-passion. Adam pretty much hit all the major points. The storage space is still open for disruption but is hard with high risk. It's not like building a website. You need serious funding for hardware. You need people to manage the hardware. You need complex software to manage the data and ensure security. One serious breach early on and you're done. Competin…

It really isn't that expensive see http://www.backblaze.com/ and specifically http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v... Obviously they are targeted at backups but you wouldn't need to change a lot to improve performance (mostly it would be in software + some caching boxes I think).

We have ~8PB of spinning storage that we built with the backblaze storage pods, and use Openstack's Swift Object Storage for the software layer. Works like a champ.

Re: How to: Compete with Amazon S3 without Buying Hardware

#35

Post by Corrado »

toomuchtodo wrote: post #34
mbell wrote: post #29

Earlier quoted context omitted.

It really isn't that expensive see http://www.backblaze.com/ and specifically http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v... Obviously they are targeted at backups but you wouldn't need to change a lot to improve performance (mostly it would be in software + some caching boxes I think).

We have ~8PB of spinning storage that we built with the backblaze storage pods, and use Openstack's Swift Object Storage for the software layer. Works like a champ.

This sounds very interesting and is something that I'm thinking about doing as well. One "limitation" that I see with the BackBlaze pods is the possibility that they don't perform well in heavy everyday use. They were designed to be mostly write-only devices, but my use case would be very read heavy and I'm not sure how they would hold up.

Do you have any information on what/how you went about building your storage system? If not, would you be so kind as to create some text (blog, how-to's, etc.) that detailed your setup and how it performs under your workload?

Re: How to: Compete with Amazon S3 without Buying Hardware

#37

Post by toomuchtodo »

Corrado wrote: post #35
toomuchtodo wrote: post #34

Earlier quoted context omitted.

We have ~8PB of spinning storage that we built with the backblaze storage pods, and use Openstack's Swift Object Storage for the software layer. Works like a champ.

This sounds very interesting and is something that I'm thinking about doing as well. One "limitation" that I see with the BackBlaze pods is the possibility that they don't perform well in heavy everyday use. They were designed to be mostly write-only devices, but my use case would be very read heavy and I'm not sure how they would hold up. Do you have any information on what/how you went about building your storage s…

Before my current startup, I worked at Fermi National Accelerator Lab on the CMS detector data taking team for the LHC. I spent a year there getting to admin the spinning storage (~5PB) on Nexsan Satabeasts (very nice, but very expensive for ~48-96TB of disk per enclosure) and ~17PB of storage on Storagetek tape silos (also, of less consequence, ~5500 nodes that reconstructed collider data from raw data we streamed over 40Gb/s optical links from CERN).

After my experience there (both technical and political), I left to do big storage. We settled on the Backblaze enclosures due solely to cost (cheap is cheap); read performance is sub-optimal, but we try to compensate with heavy caching in memory and intelligent read assumptions ("what might someone request next based on past read requests") at the app level (sitting on top of Nova).

I could do a blog post, but I have to check with my partner to make sure they're cool with me spilling that much info =)

Hope this has helped a bit. If you haven't guessed yet, I love object storage.

Re: How to: Compete with Amazon S3 without Buying Hardware

#39

Post by rbanffy »

oijaf888 wrote: post #14
chx wrote: post #9

So... let me get this straight. You came up with an idea which requires building a strong brand which takes a lot of money and that Amazon can squelch any second there's a hint of possibility of a success. And this made HackerNews frontpage. What?

Or building an open source tool that individual users could use to manage raw storage in an S3 fashion. Its an interesting idea.

Or one many little startups everywhere could use to compete locally with Amazon in the SMB space.

Re: How to: Compete with Amazon S3 without Buying Hardware

#40

Post by driverdan »

toomuchtodo wrote: post #37
Corrado wrote: post #35

Earlier quoted context omitted.

This sounds very interesting and is something that I'm thinking about doing as well. One "limitation" that I see with the BackBlaze pods is the possibility that they don't perform well in heavy everyday use. They were designed to be mostly write-only devices, but my use case would be very read heavy and I'm not sure how they would hold up. Do you have any information on what/how you went about building your storage s…

Before my current startup, I worked at Fermi National Accelerator Lab on the CMS detector data taking team for the LHC. I spent a year there getting to admin the spinning storage (~5PB) on Nexsan Satabeasts (very nice, but very expensive for ~48-96TB of disk per enclosure) and ~17PB of storage on Storagetek tape silos (also, of less consequence, ~5500 nodes that reconstructed collider data from raw data we streamed o…

A blog post would be really interesting, even if you couldn't disclose everything.

What are your startup and blog URLs? You don't have any profile info.

Post reply on HN