Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to: Compete with Amazon S3 without Buying Hardware

blog.adamsmith.cc

Post reply on HN
Print view View original

11–20 of 40 posts

Re: How to: Compete with Amazon S3 without Buying Hardware

#11

Post by adamsmith »

thereallurch wrote: post #8
adamsmith wrote: post #6

Earlier quoted context omitted.

> This article is basically advocating RAID 5 across many storage providers. That is correct. And to be more precise, I'm advocating RAID 5 across storage providers as a service , so people who just want to store data don't have to manage anything.

Wouldn't the next issue become bandwidth? Sure, you have cheap storage, but you still need to manipulate the data somehow. I know S3 only charges for outbound data. What about the other companies? Is bandwidth free and plenty?

This is a really interesting question.

At scale, this could be managed, I think, through a combination of (a) shipping hard drives around, (b) caching, and (c) peering between storage providers. For example, shipping hard drives around would be expedient if you wanted to switch out a raw storage provider. The optimal strategy also depends on the access patterns and latency requirements.

It seems solvable, but not trivial.

Re: How to: Compete with Amazon S3 without Buying Hardware

#12

Post by revolutions »

pandemicsyn wrote: post #5
thereallurch wrote: post #3

"In 2006 a 320 GB hard drive cost $120. Today (Thailand floods aside) that much money will snag you a 3 TB drive." Floods or not, the current price isn't $120 dollars. It's 50% higher than that. http://camelcamelcamel.com/Western-Digital-Caviar-Green-Desk... Shows one of the cheapest 3tb non enterprise drives. It looks like 3tb was $120 for ~2 weeks. Looking at enterprise drives, 3tb is closer to $300. This article i…

People always conveniently forget as well that when you store data at S3 or Rackspace they're doing 3 replica's. If you store 3TB's, its not just the cost of a single 3TB drive..its 3x. To do it at home you'd need to buy 3x3TB Drives @300$ each.

Suppose the prices of _x_ amount of storage was halved recently. If they stored their data in three replicas, shouldn't their price become (1/2)^3, or 1/8th the original price? That would only serve to prove his point further, or I'm missing something.

Re: How to: Compete with Amazon S3 without Buying Hardware

#13

Post by thereallurch »

revolutions wrote: post #12
pandemicsyn wrote: post #5

Earlier quoted context omitted.

People always conveniently forget as well that when you store data at S3 or Rackspace they're doing 3 replica's. If you store 3TB's, its not just the cost of a single 3TB drive..its 3x. To do it at home you'd need to buy 3x3TB Drives @300$ each.

Suppose the prices of _x_ amount of storage was halved recently. If they stored their data in three replicas, shouldn't their price become (1/2)^3, or 1/8th the original price? That would only serve to prove his point further, or I'm missing something.

provider1=[d1,1][d1,2][d1,3]

provider2=[d2,1][d2,2][d2,3]

provider3=[d3,1][d3,2][d3,3]

Each drive cost $x. You have 9$x = cost to store 1 drive of data, across 3 providers, who each store 3 copies. If drive prices halve, it's still 9$x.

Re: How to: Compete with Amazon S3 without Buying Hardware

#14

Post by oijaf888 »

chx wrote: post #9

So... let me get this straight. You came up with an idea which requires building a strong brand which takes a lot of money and that Amazon can squelch any second there's a hint of possibility of a success. And this made HackerNews frontpage. What?

Or building an open source tool that individual users could use to manage raw storage in an S3 fashion. Its an interesting idea.

Re: How to: Compete with Amazon S3 without Buying Hardware

#15

Post by soult »

Tahoe-LAFS[1] is a storage system that works similarly. It splits data into n fragments, of which any k fragments are enough to restore the data. This leads to a replication factor of only DATA_SIZE/k*n, while you can still lose n-k pieces without data loss. Additionally all data is encrypted, signed and optionally deduplicated.

One of the authors of Tahoe-LAFS started a company that ported the whole system over to cloud storage providers.[2] It's still in alpha, but it's definitly worth a look if you want secure, encrypted storage without relying on a single cloud provider.

1: https://tahoe-lafs.org/trac/tahoe-lafs

2: https://leastauthority.com/

Re: How to: Compete with Amazon S3 without Buying Hardware

#16

Post by adamsmith »

soult wrote: post #15

Tahoe-LAFS[1] is a storage system that works similarly. It splits data into n fragments, of which any k fragments are enough to restore the data. This leads to a replication factor of only DATA_SIZE/k*n, while you can still lose n-k pieces without data loss. Additionally all data is encrypted, signed and optionally deduplicated. One of the authors of Tahoe-LAFS started a company that ported the whole system over to c…

RE Least Authority, awesome!, thanks for the link!

Re: How to: Compete with Amazon S3 without Buying Hardware

#17

Post by driverdan »

I've been watching the storage industry for years as a hobby-passion. Adam pretty much hit all the major points. The storage space is still open for disruption but is hard with high risk.

It's not like building a website. You need serious funding for hardware. You need people to manage the hardware. You need complex software to manage the data and ensure security. One serious breach early on and you're done.

Competing with Amazon is especially hard since S3 is well established and entrenched. If you use EC2 you're going to use S3.

Pricing would be a primary factor in competing. 3x redundancy is unnecessary. I'm not sure why services still do that. Reed-Solomon or similar redundancy algorithms can provide better protection and use less space. They have CPU overhead but CPUs aren't going to be the bottleneck for a storage service, bandwidth and hard drives will be.

Edit: This would be if you built from the hardware up. I don't think offering a service like S3 on top of other storage services would work as a business. You'd have to deal with too many vendors, too much variation in APIs / software / hardware, lack of control, latency issues, and much tighter margins. IMO you'd be better off starting with bare metal. You could do something like this for personal, smaller scale storage but growing it to scale would be a nightmare.

Re: How to: Compete with Amazon S3 without Buying Hardware

#18

Post by wmf »

thereallurch wrote: post #13
revolutions wrote: post #12

Earlier quoted context omitted.

Suppose the prices of _x_ amount of storage was halved recently. If they stored their data in three replicas, shouldn't their price become (1/2)^3, or 1/8th the original price? That would only serve to prove his point further, or I'm missing something.

provider1=[d1,1][d1,2][d1,3] provider2=[d2,1][d2,2][d2,3] provider3=[d3,1][d3,2][d3,3] Each drive cost $x. You have 9$x = cost to store 1 drive of data, across 3 providers, who each store 3 copies. If drive prices halve, it's still 9$x.

I don't think anyone is proposing 3 providers, who each store 3 copies. Either use one provider who stores 3 copies (Amazon) or three providers who each store one copy.

Re: How to: Compete with Amazon S3 without Buying Hardware

#20

Post by notmyname »

adamsmith wrote: post #6
thereallurch wrote: post #3

"In 2006 a 320 GB hard drive cost $120. Today (Thailand floods aside) that much money will snag you a 3 TB drive." Floods or not, the current price isn't $120 dollars. It's 50% higher than that. http://camelcamelcamel.com/Western-Digital-Caviar-Green-Desk... Shows one of the cheapest 3tb non enterprise drives. It looks like 3tb was $120 for ~2 weeks. Looking at enterprise drives, 3tb is closer to $300. This article i…

> This article is basically advocating RAID 5 across many storage providers. That is correct. And to be more precise, I'm advocating RAID 5 across storage providers as a service , so people who just want to store data don't have to manage anything.

If this were built on top of (or in conjunction with) Openstack swift (http://swift.openstack.org), it could be done with a common storage backend. There are already several public cloud storage providers using swift (Rackspace, HP, Softlayer, Internap, KT, and others), and a growing number of private deployments as well (Wikipedia probably being the most recognizable name).

If you'd like to talk more about this, send me an email (address in profile).

Post reply on HN