Live data from Hacker News

Sun engineer responds to the Backblaze "Petabytes on a budget" design

c0t0d0s0.org

91–100 of 109 posts

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#91

Earlier quoted context omitted.

10x the rack space, 10x the cooling, 10x the power ...

People often forget that the rack space, the actual physical space, is very costly. Depending on the datacenter, the power might run you just as much too. Last company I worked at was essentially forced out for using too much power per U.

Would you really buy a hackish low-spec solution like this and then place it in a premium rack?

Given the idea, buy 10 and place them at geographically diverse points, you may as well put them in the back room at branch offices or something. Who cares if one goes offline for a while. Hell, put them in countries with good consumer internet connectivity (japan, korea, sweden etc) and just use that.

If you're going ghetto, go all the way!

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#92

I agree that what the article misses is that Backblaze is spending the money on software rather than hardware, since the price difference is so huge (and the software solution is a fixed cost). What I'm more interested in is whether the lower MTBF of the cheap drives and home-brewed chassis ends up with a higher cost per year due to higher failure rates. If a desktop drive costs $100 and fails three times a year, but…

Well, remember that if you were seriously on a budget you'd just return all the failed drives for warranty replacement (Seagate is 5yrs). The hardware costs would never add up to more than the enterprise, it's that the labour costs of dealing with it would be more, and worse - more likely to lose data. Enough more likely to justify the additional cost? Hm, I doubt it, not for this use case.

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#93
post #29

Earlier quoted context omitted.

Haha! Unexpectedly needing a new cage at your DC, or an extra 50kW power, or being told you can't put anything more in your half-full racks because you've exceeded floor loading gets expensive real quick.

Upvoted. I would upvote a million times over if I could. So many developers seem to not realize just how expensive physical plant (and the people to administer it) really is -- "hardware is cheap" fails to account for the facility costs involved in acquiring necessary space, power, cooling to run it -- often ahead of time. Managing the IT team and a growing data center installation for a medium size business was one…

A few years ago I did some consulting for a datacentre provider that you've almost certainly heard of who were giving serious consideration to buying a utility and operating their own power station, to get their power and cooling at cost. They'd already contemplated and rejected moving their entire operation to northern Alaska/Canada/wherever and just paying to plumb in the bandwidth (customers wanted to be able to get at their kit if they needed to).

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#94

Ok I must be missing something. Everyone is knocking this backblaze thing because it doesnt do ZFS or because it is not a "super duper high end san replacement" or because the components are not rated for enterprise level work. It seems to me that a company whose product is personal and small business backups does not need any of those things. They have software to do mirroring. Why would they need any fast network f…

The answer to your question is in the price comparisons that Backblaze provides. If you compare the price of something you imply that it fills the same or similar needs. Since there is no explicit caveat, and they don't explain why their solution is so much cheaper, competitors will come out and provide that explanation for them.

I'm pretty sure that the Backblaze solution is actually cheaper and that they are cutting some of the right corners for their particular application scenario. But it remains an open question how much cheaper it ends up being.

I think their price comparison is a bit misleading since it doesn't include replacement costs for the desktop grade hard disks they use and it doesn't include any of the costs incurred by the software/labor/maintainance required to make this a reliable backup solution.

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#95

Earlier quoted context omitted.

I think their use case is sufficiently a-typical to get away with this. They're primarily write-only, only when a customer retrieves their data does stuff get read. If they spin down the drives when the volume is not being accessed the mttf for a single drive goes up a lot, possibly beyond the point where it matters.

That made me think of MAID (and COPAN), which is decent approach. http://www.copansystems.com/

Floor loading (and rack size) is a problem with their 'very big box of disks'.

That, and the fact that the company seems to be struggling to get sales.

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#96
post #62

Ok I must be missing something. Everyone is knocking this backblaze thing because it doesnt do ZFS or because it is not a "super duper high end san replacement" or because the components are not rated for enterprise level work. It seems to me that a company whose product is personal and small business backups does not need any of those things. They have software to do mirroring. Why would they need any fast network f…

Exactly - this is a replacement for a tape library, not a database server.

Indeed. It would be nice to see some comparisons to the architectures (and, as everyone seems to care so much, prices) in the various 'VTLs' sold by the likes of Falconstor, HP, EMC, and StorageTek/Sun!

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#97
post #67

Ok I must be missing something. Everyone is knocking this backblaze thing because it doesnt do ZFS or because it is not a "super duper high end san replacement" or because the components are not rated for enterprise level work. It seems to me that a company whose product is personal and small business backups does not need any of those things. They have software to do mirroring. Why would they need any fast network f…

"how did so many people get the idea that this article about a specific solution to a specific problem was actually some sort of general purpose solution attacking all the big name people?" Backblaze did it, They did it when they decided to put this graph in the article: http://blog.backblaze.com/wp-content/uploads/2009/08/cost-of... It's an apples to oranges comparison.

It's an apples to oranges comparison.

Maybe for the low-level tech, but certainly not what it's used for. There are lots of online backup services out there. Some of them publicly make it known they use Amazon S3 for storage. I'm sure there are some other companies, who own their hardware, like Backblaze, but didn't optimise their cost structure using custom builds.

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#98
post #86
post #82

Earlier quoted context omitted.

"That said, our experience with the Sun X4500s hasn't been great from an I/O point of view." Of course it's not. It's an IBM PC down inside. I bet it can still boot MS-DOS and run GW-BASIC. ;-) PCs are not a nice architecture for servers: there are starvation points all over the system, from registers (AMD64 solves some of it) to memory to I/O. Sun could have based the Thumper on a more server-ish (SPARC?) design, wi…

PCs are not a nice architecture for servers: there are starvation points all over the system, from registers (AMD64 solves some of it) to memory to I/O. Could you clarify the ways in which you think a PC architecture will hobble a data server? It looks like the X4500 runs dual Opterons, which relatively speaking have no shortage of registers. And it supports 16 GB of memory, which doesn't seem too shabby. And while I…

- It's not the amount of memory, it's how much memory you can transfer without disrupting program execution due to bus contentions. It makes a huge difference whether your 16 GB of memory are in two, four or eight sockets (different sockets can (or at least could) be accessed simultaneously) or if it's attached to a single processor or pooled system-wide.

- AMD64 is a little better than vanilla 32-bit x86, but it still has few registers compared to POWER or SPARC architectures. This increases the risk of memory access, which is bad. I suppose there is a point when it's pointless to add registers and x86s do some convoluted stuff with shadow registers, so the picture is not really clear. Optimizing compilers should alleviate this too, but, like car builders say, there is no substitute for the cubic-inch.

- Still about processors, the least a multi-threaded CPU can do for you is to keep an execution context in-chip and prevent a context swap from memory. That saves a lot of memory bus time that cannot be used by other parts of the system. AMD64s (and their Intel counterparts, AFAIK) max out at 2 threads/core. POWER and SPARC max out at 4 and 8 tpc respectively (again, a number off the top of my head).

- On PCs (defined here as "a computer that can run Windows"), there is little distributed intelligence. I never saw a PC where CPU, disk controller and network interfaces could do the chat I described. Contrast it with the typical vintage mainframe design, where there are as many things going in parallel as designers can think of. As as example, there is an IBM disk-drive in the Computer Museum where you can see two sets of heads/arms on opposing sides of the disk, effectively being able to read/write different cylinders simultaneously. While I don't believe such machines are in current use, this serves to illustrate how far a server designer is willing to go in order to beat a throughput record.

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#99

Ok I must be missing something. Everyone is knocking this backblaze thing because it doesnt do ZFS or because it is not a "super duper high end san replacement" or because the components are not rated for enterprise level work. It seems to me that a company whose product is personal and small business backups does not need any of those things. They have software to do mirroring. Why would they need any fast network f…

The answer to your question is in the price comparisons that Backblaze provides. If you compare the price of something you imply that it fills the same or similar needs. Since there is no explicit caveat, and they don't explain why their solution is so much cheaper, competitors will come out and provide that explanation for them. I'm pretty sure that the Backblaze solution is actually cheaper and that they are cuttin…

desktop grade hard disks

I believe the data dispel this marketing myth.

http://usenix.org/events/fast07/tech/schroeder/schroeder_htm...

http://usenix.org/events/fast07/tech/full_papers/pinheiro/pi...

Re: Sun engineer responds to the Backblaze "Petabytes on a budget" design

#100
post #75

Earlier quoted context omitted.

Well no, it's not an apples to oranges comparison because no hardware vendors offer a solution which is comparable to the one Backblaze have devised. That graph is simply comparing their solution to the closest commercial equivalents. It just so happens that these are all way off because all hardware manufacturers want to design their hardware to provide the highest throughput. (That said, our experience with the Sun…

It is an apples to oranges comparison because Blackblaze is only including the cost of components in that graph for their solution, but they're including all of the research, development, assembly, and support costs in the other vendors' solutions. What are the labor costs for testing all their hardware (with 10 sata controllers no less)? What are the labor costs for assembling all those systems? What is the labor co…

Good point, but I suspect even given the amount of money they've spent on development, and given the production runs on these things are tiny, they'll probably still beat Sun on total price. Hell, even if they only made one of the things they could spend $800k on developing/manufacturing it and still beat Sun.

As far as I'm concerned, you'd have to write the custom software even if you used Sun's boxes. I wouldn't trust data to a single machine of anyone's design. Though that's probably just me.

The S3 comparison is totally invalid though, I agree.

Post reply on HN