Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Online backup to S3 for the Mac

haystacksoftware.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Online backup to S3 for the Mac

#31

Post by sreitshamer »

PStamatiou wrote: post #30

I have been using Arq for the last 3 weeks. It's very basic software at the moment, but it does the job. It lets me see file revisions and restore files as necessary. Unfortunately these files can't be seen directly in S3 due to the packing nature of how they're stored.. so if your mac crashes and you want a file, you'll need to find another mac and install Arq to retrieve your files.

Right. The files are named using the SHA1 hash of their contents (to enable de-duplication) and small files are packed together (to dramatically improve performance). I thought that was the best solution given the limits of the S3 API. JungleDisk does something similar I think.

What sort of alternative for restoring would you be interested in? Maybe an open-source command-line utility?

Re: Online backup to S3 for the Mac

#32

Post by zacharypinter »

sreitshamer wrote: post #31
PStamatiou wrote: post #30

I have been using Arq for the last 3 weeks. It's very basic software at the moment, but it does the job. It lets me see file revisions and restore files as necessary. Unfortunately these files can't be seen directly in S3 due to the packing nature of how they're stored.. so if your mac crashes and you want a file, you'll need to find another mac and install Arq to retrieve your files.

Right. The files are named using the SHA1 hash of their contents (to enable de-duplication) and small files are packed together (to dramatically improve performance). I thought that was the best solution given the limits of the S3 API. JungleDisk does something similar I think. What sort of alternative for restoring would you be interested in? Maybe an open-source command-line utility?

A command-line utility is always a big plus for those who like scripting. Maybe a FUSE script/app/plugin?

Re: Online backup to S3 for the Mac

#33

Post by rarrrrrr »

pclark wrote: post #10

I can't judge how much it'll cost to back my iMac up. I have about 750Gb of data.

I'm a bit biased, but I don't feel S3 is the best possible fit for a frugal consumer's backup data. S3 is a high performance, general purpose solution capable of very low latency.

Having that performance makes it more expensive than an archival specific solution, which tolerates latency just fine as long as high throughput is maintained.

There's a reason why Mozy, Carbonite, SpiderOak, etc. all run our own data centers. I can sell a 100GB account for usable across unlimited computers for $50 per year and be profitable.

That said, the market certainly has plenty of room for more backup choices, and any option to help people avoid data loss is a win. An exclusively Mac app seems like a good idea.

Re: Online backup to S3 for the Mac

#34

Post by sreitshamer »

rarrrrrr wrote: post #33
pclark wrote: post #10

I can't judge how much it'll cost to back my iMac up. I have about 750Gb of data.

I'm a bit biased, but I don't feel S3 is the best possible fit for a frugal consumer's backup data. S3 is a high performance, general purpose solution capable of very low latency. Having that performance makes it more expensive than an archival specific solution, which tolerates latency just fine as long as high throughput is maintained. There's a reason why Mozy, Carbonite, SpiderOak, etc. all run our own data cente…

I agree. For me though, backup comes down to trust. I trust Amazon to provide the S3 service reliably. I find it harder to trust Mozy, Carbonite and the rest, partly because they don't even bother to restore my Mac files' metadata correctly.

Re: Online backup to S3 for the Mac

#35

Post by mikepurvis »

pclark wrote: post #10

I can't judge how much it'll cost to back my iMac up. I have about 750Gb of data.

I don't know how other people do it, but certainly for me, I have things that are at different levels of importance to me. And a lot of the most important are not really all that big, nor are they changing particularly frequently. I wouldn't pay a premium of $100 a month just for the convenience of having everything backed up in this super-redundant, universally-accessible way.

I would happily use a product like Arq (or Time Machine) for my documents and current photos, to make sure the backup is always current, and then whenever the backup hits 50gb, move it over to an off-site drive (at work, a relative's house, even in a safety deposit box).

Re: Online backup to S3 for the Mac

#36

Post by mikepurvis »

Willie_Dynamite wrote: post #21
sreitshamer wrote: post #16

Earlier quoted context omitted.

I was thinking of adding Pogoplug support. That way storage would be much cheaper, although backing up to a Pogoplug at a friend's house may be much less reliable than backing up to S3. Would anyone be interested in this?

I'd be far more interested in a generic ssh target thingy, than support for specific peripherals.

I second the motion. I have a ton of storage on Dreamhost that I could totally be using as backup. (And have tried a bunch of times, but it's a pain getting cron and rsync to do the right thing, and then remembering to set it up again when my computer gets changed around...)

Re: Online backup to S3 for the Mac

#37

Post by sreitshamer »

mikepurvis wrote: post #36
Willie_Dynamite wrote: post #21

Earlier quoted context omitted.

I'd be far more interested in a generic ssh target thingy, than support for specific peripherals.

I second the motion. I have a ton of storage on Dreamhost that I could totally be using as backup. (And have tried a bunch of times, but it's a pain getting cron and rsync to do the right thing, and then remembering to set it up again when my computer gets changed around...)

So, SFTP support then?

Re: Online backup to S3 for the Mac

#38

Post by cperciva »

sreitshamer wrote: post #28
cperciva wrote: post #26

As the author of Tarsnap, I'm a bit biased, so I'm going to refrain from comparing the two; but I have one question for the author: Aren't S3 per-request fees a problem? One of the reasons I have all Tarsnap data go through the Tarsnap server is to allow me to "bundle" blocks together and amortize per-PUT costs.

The per-request fees ($.01 per 1000 PUTs and $.01 per 10,000 GETs) aren't a problem for 2 reasons: 1) Arq packs small files (under 64k) together into "packs", a bit like git does. So instead of thousands of PUTs of tiny files (which would also be very slow) it does fewer larger PUTs. 2) Arq caches indexes of those pack files locally, and caches the list of larger S3 objects as well. So it doesn't have to do a GET to…

Arq packs small files (under 64k) together into "packs"...

Ok, so if you have lots of 64 kB files they're all PUT separately, using 15625 PUTs/GB (i.e., $0.15/GB)? I guess that's not too bad.

The file format is open

Nice! Good to see someone who believes in openness. (How about the source code, so that people can confirm that you're doing what the spec says?)

One thing caught my eye glancing through that document, though: "The name of each blob is its SHA1 hash." That's the SHA1 hash of the encrypted data, right? (If not, it definitely should be!)

Re: Online backup to S3 for the Mac

#40

Post by sreitshamer »

cperciva wrote: post #38
sreitshamer wrote: post #28

Earlier quoted context omitted.

The per-request fees ($.01 per 1000 PUTs and $.01 per 10,000 GETs) aren't a problem for 2 reasons: 1) Arq packs small files (under 64k) together into "packs", a bit like git does. So instead of thousands of PUTs of tiny files (which would also be very slow) it does fewer larger PUTs. 2) Arq caches indexes of those pack files locally, and caches the list of larger S3 objects as well. So it doesn't have to do a GET to…

Arq packs small files (under 64k) together into "packs"... Ok, so if you have lots of 64 kB files they're all PUT separately, using 15625 PUTs/GB (i.e., $0.15/GB)? I guess that's not too bad. The file format is open Nice! Good to see someone who believes in openness. (How about the source code, so that people can confirm that you're doing what the spec says?) One thing caught my eye glancing through that document, th…

Ok, so if you have lots of 64 kB files they're all PUT separately, using 15625 PUTs/GB (i.e., $0.15/GB)? I guess that's not too bad.

I guess I could ask you to choose the threshold instead of picking 64k, but generally I've tried to choose reasonable defaults so that the user doesn't have to make those kinds of decisions.

Good to see someone who believes in openness. (How about the source code, so that people can confirm that you're doing what the spec says?)

Well, you could actually write your own code to read the files, since the format is open.

That's the SHA1 hash of the encrypted data, right?

It's the SHA1 of the contents of the object, and the object is encrypted data, so yes.

Post reply on HN