Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Let's talk about a hypothetical public-facing service”

reddit.com

Post reply on HN
Print view View original

31–40 of 72 posts

Re: “Let's talk about a hypothetical public-facing service”

#31

Post by Sanddancer »

sneak wrote: post #29
Sanddancer wrote: post #27

Earlier quoted context omitted.

Yes. One can build a system that holds a petabyte of data, fits in less than one rack, has multiple redundancy, for about a hundred grand these days. Multiplied by three locations, that'll get you to $300000. Add in another $35000 for three years of hosting at three different datacenters, and you get to $335000. Sounds like a lot, right? Amazon's even more expensive. Storing one petabyte on Amazon, at the slowest, ch…

It costs about a hundred and fifty grand annually (fully loaded, base salary 80-90k) to have the lowest end person who could conceivably keep one of these things running well for three years. n+1 means you need two of them in case one gets the flu. Tell me again about how Amazon's a ripoff, please.

You're still going to need n+1 sysadmins to maintain the rest of your infrastructure. Maintaining a storage system like this really isn't much different for the day-to-day operations. So, yes, it's one more system, but you're already going to need those admins anyways.

Re: “Let's talk about a hypothetical public-facing service”

#32

Post by codezero »

ChuckMcM wrote: post #30
nadams wrote: post #26

Earlier quoted context omitted.

> you can stop buying disks from Amazon every few months This is something that has been puzzling me. Many years ago I purchased 4x 2TB 5900 RPM drives for a 4 bay ReadyNAS (cost about ~$300 for drives plus ReadyNAS). They have been spinning nonstop for ~4 years [1] and haven't had to replace a single one. Not even an increase in errors to signal that the drive is going. Yet - I've worked on a SAN that cost hundreds…

AFR buddy, AFR. AFR is the annual failure rate, its typically 2 - 5% of the population per year. 4 drives you don't have a large enough set to see this in action, just every day you're in danger of losing a drive by a small statistical amount. In the Blekko cluster we have just under 10,000 drives. We have a two 20 drive 'boxes' (40 drives) from Western Digital, as drives fail we pull replacements from the 'new/refur…

Just out of curiosity, has anyone done any research into determining whether a drive which fails after X days/years has some properties in the first Y days that could be a signal for future failure?

Re: “Let's talk about a hypothetical public-facing service”

#33

Post by morganvachon »

zatkin wrote: post #2

I was hoping they would stay offline. It's really disappointing to see a good service turn into a money wringing desperation.

The way I discovered SF was down was when I was trying to install some Slackbuilds and kept getting MD5 mismatches on the source downloads. I noticed I was getting the same MD5 for several different packages, so I tried to manually grab a package and that's when I hit SF's error page (which sbopkg was happily downloading and trying to pass off as somesourcepkg.tar.gz).

I found myself in the odd position of hoping SF would come back up so I could finish what I was doing, when I'd normally welcome the news that they had shut down.

Re: “Let's talk about a hypothetical public-facing service”

#34

Post by myth_buster »

glass- wrote: post #25
intrasight wrote: post #3

Got a good chuckle. Such is life in IT. Gotta admit I missed the "Extremely Massive Corporation" hint.

It reminds me of Douglas Crockford's talk where he mentioned that a company asked him for an exemption to the "do no evil" clause in the JSON License, and he said he didn't want to name the company as that would embarrass them so he would instead give their initials: IBM.

The comment in question

https://www.youtube.com/watch?v=-hCimLnIsDA

Re: “Let's talk about a hypothetical public-facing service”

#35

Post by objectivefs »

ThePhysicist wrote: post #19
smacktoward wrote: post #5

> You have a moment where you envision the future of virtualized storage and think about how great it will be when storage is nearly free and outsourceable and you can stop buying disks from Amazon every few months The future is now! Instead of you sending money to Amazon and them sending you disks, they keep the disks and you send them the money anyway. Progress! :-D

Sadly, depending on what you want to accomplish with your storage the future is still not here. For example, if you want to share a virtual disk with multiple servers through NFS you will still need to manually create a NFS high-availability cluster using e.g. DRBD on most cloud providers (e.g. EC2 or Azure). To be fair, MS Azure offers fully managed SMB shares as a service, but the transfer speed is quite low (60 MB…

For sharing a filesystem between servers without managing your own high-availability cluster, you could try our ObjectiveFS [1].

[1] https://objectivefs.com

Re: “Let's talk about a hypothetical public-facing service”

#36

Post by sneak »

Sanddancer wrote: post #31
sneak wrote: post #29

Earlier quoted context omitted.

It costs about a hundred and fifty grand annually (fully loaded, base salary 80-90k) to have the lowest end person who could conceivably keep one of these things running well for three years. n+1 means you need two of them in case one gets the flu. Tell me again about how Amazon's a ripoff, please.

You're still going to need n+1 sysadmins to maintain the rest of your infrastructure. Maintaining a storage system like this really isn't much different for the day-to-day operations. So, yes, it's one more system, but you're already going to need those admins anyways.

You're also going to need connectivity, of course. We were doing an apples to apples comparison in a vacuum, I thought.

Re: “Let's talk about a hypothetical public-facing service”

#37

Post by Smerity »

codezero wrote: post #32
ChuckMcM wrote: post #30

Earlier quoted context omitted.

AFR buddy, AFR. AFR is the annual failure rate, its typically 2 - 5% of the population per year. 4 drives you don't have a large enough set to see this in action, just every day you're in danger of losing a drive by a small statistical amount. In the Blekko cluster we have just under 10,000 drives. We have a two 20 drive 'boxes' (40 drives) from Western Digital, as drives fail we pull replacements from the 'new/refur…

Just out of curiosity, has anyone done any research into determining whether a drive which fails after X days/years has some properties in the first Y days that could be a signal for future failure?

I'm certain there are more recent statistics but Google's "Failure Trends in a Large Disk Drive Population"[1] (2007) is a good start:

"In addition to presenting failure statistics, we analyze the correlation between failures and several parameters generally believed to impact longevity."

There is also a more recent open source dataset from Backblaze[2] that includes:

"Every day, the software that runs the Backblaze data center takes a snapshot of the state of every drive in the data center, including the drive’s serial number, model number, and all of its SMART data"

which forms the basis of an article correlating SMART data with drive failures at Backblaze[3].

[1]: http://static.googleusercontent.com/media/research.google.co...

[2]: https://www.backblaze.com/blog/hard-drive-data-feb2015/

[3]: https://www.backblaze.com/blog/hard-drive-smart-stats/

Re: “Let's talk about a hypothetical public-facing service”

#38

Post by Sanddancer »

sneak wrote: post #36
Sanddancer wrote: post #31

Earlier quoted context omitted.

You're still going to need n+1 sysadmins to maintain the rest of your infrastructure. Maintaining a storage system like this really isn't much different for the day-to-day operations. So, yes, it's one more system, but you're already going to need those admins anyways.

You're also going to need connectivity, of course. We were doing an apples to apples comparison in a vacuum, I thought.

That's covered in the cost estimates. A petabyte takes up about a half rack these days, and renting a half rack from a colo is about $350/mo.

Re: “Let's talk about a hypothetical public-facing service”

#39

Post by [deleted] »

objectivefs wrote: post #35
ThePhysicist wrote: post #19

Earlier quoted context omitted.

Sadly, depending on what you want to accomplish with your storage the future is still not here. For example, if you want to share a virtual disk with multiple servers through NFS you will still need to manually create a NFS high-availability cluster using e.g. DRBD on most cloud providers (e.g. EC2 or Azure). To be fair, MS Azure offers fully managed SMB shares as a service, but the transfer speed is quite low (60 MB…

For sharing a filesystem between servers without managing your own high-availability cluster, you could try our ObjectiveFS [1]. [1] https://objectivefs.com

[deleted]

Re: “Let's talk about a hypothetical public-facing service”

#40

Post by nkozyra »

idunno246 wrote: post #23
ThePhysicist wrote: post #19

Earlier quoted context omitted.

Sadly, depending on what you want to accomplish with your storage the future is still not here. For example, if you want to share a virtual disk with multiple servers through NFS you will still need to manually create a NFS high-availability cluster using e.g. DRBD on most cloud providers (e.g. EC2 or Azure). To be fair, MS Azure offers fully managed SMB shares as a service, but the transfer speed is quite low (60 MB…

Ec2 has efs at least, no idea if it's any good

I'm sure it's fine, at least as well as a manually-configured NFS, but it's only US-West (Oregon) now.
Post reply on HN