The main problem I have with Backblaze and AWS is that there is no budget limit on transfers.
I would like to tell the provider; limit downloads to $500, then reply 509 to further requests.
Backblaze Storage Pod 5.0
181–190 of 222 posts
Re: Backblaze Storage Pod 5.0
#182I really love these, and I desperately wanted to be a Backblaze customer, but I was put off by the abysmal desktop software :( Storage execution seems top-notch. End-user software deserves some love.
With Backblaze B2 you'll be able to do it yourself -> www.backblaze.com/b2 ;-)
Re: Backblaze Storage Pod 5.0
#183Why are they not offering 8GB drives. It's late 2015.
The most important thing for us is Cost per GB. Right now the 4TB drives we're using are still much cheaper than the 8TBs, so we go with the 4s, but the Pods work with 8TBs if you want to drop the dough on them!
Re: Backblaze Storage Pod 5.0
#184Earlier quoted context omitted.
Interesting and thanks for the enlightening comments. I proposed similar things to a few people esp the partnership with hardware development, working around faulty (cheap) stuff, custom firmware, and kernel I/O modifications. I expected each of these in a high-end solution. My architecture didn't have an IP stack at all: guards or front-end acted as I.P. gateway while internally using much simpler protocol that stil…
Wow, very cool! So, candidate for offloading that's really commonly considered is "scrubbing". Essentially, protecting against passive bit flips by checking your data against your own checksums. If you have some N TB of data under management, it's really expensive to be striping these large serial reads all the way to an application's userland. So finding ways to keep these as low-priority "background" reads (that al…
"So, candidate for offloading that's really commonly considered is "scrubbing". Essentially, protecting against passive bit flips by checking your data against your own checksums."
"Or even offload it to hardware--some custom host adapter, possibly, using custom ATA/SCSI commands to control it and query it. (and the `sg` driver)."
Now you're thinking. Chips like Octeon already do hardware acceleration of checksums and possibly other I/O boosters. So did mainframe I/O. There's many for FPGA's and ASIC's. Cray's ChipKill technology, implemented in Oracle SPARC CPU's, does something similar for RAM in combination with ECC RAM. So, an ASIC with several SATA connectors and logic for this should be able to do it. Might handle, partially or totally, the protocol for processing the data, too.
Wanted to see if I was really thinking ahead by Googling for academic papers. If it's a good idea, there's usually a few people that have done it for years already. Turned up some hits, including on your Reed-Solomon.
FPGA-accelerated decision-tolerant coding for reliable distributed storage https://dl.acm.org/citation.cfm?id=1763276
FPGA-accelerated, flash store for analytics http://people.csail.mit.edu/wjun/papers/fpga2014-wjun.pptx
DB acceleration http://www.cse.buffalo.edu/~vipin/papers/2010/todd1.pdf
Filesystem in GPU. Shows GPU's or DSP's can be used for this. Probably Adapteva's chips, too, esp since they might negotiate for volume deals. https://people.csail.mit.edu/idish/ftp/silberstein13asplos-g...
Tilera's were used in a 100Gbps NIDS that I recall. I expect your protocol and algorithms to be simpler. ;) http://www.tilera.com/products/
So, goes to show either custom, FPGA, or better-suited chips can handle that part at great performance per watt. Cost varies with implementation strategy. Nonetheless, simple functions with streaming data model greatly benefit from this strategy.
Re: Backblaze Storage Pod 5.0
#185Earlier quoted context omitted.
Host Managed (or Host Aware) SMR disks are fine. You need to do your own zone management using ZAC/ZBC, and throw out your filesystem. The disks all have a conventional section near LBA 0 for metadata management (e.g. indices). Usual tradeoff--more work in software, but far less $.
Ah good, now we're talking specifics. Does REPORT ZONES not point to this conventional section always? I cannot wait for a FORMAT ALL ZONES or something command to control the number/amount of conventional sections at the sacrifice of capacity, though it looks like it's not coming. The power guarantees in the zbc spec (as of r03) seem to not be held by some drives. Writing less than a full zone seems to be a bad idea…
> Ah good, now we're talking specifics.
Well, specifics within a hypothetical. Hypotheticals are always game to to discuss. :-)
> Does REPORT ZONES not point to this conventional section always?
Yep, REPORT ZONES includes the conventional zone.
> I cannot wait for a FORMAT ALL ZONES
I agree that would be nice, but there are no indications that anyone but the vendor will control conventional vs. sequential layout, sequential sizes, or maximum # of explicitly open zones, anytime soon (absent asking for special favors for a large batch order).
Re: Backblaze Storage Pod 5.0
#186Earlier quoted context omitted.
Details? I don't work for BB, but have worked in the backup industry for some time in my past, and always curious.
Not the OP, have used BB for 5 years (and still using, the service is totally fine): Initial sync is a nighmare. The upload is on the slow side. Biggest gripe so far is that I just cannot tell the client that some files are more important than the others. I would love my pdf invoices and changed KDBX files to be synced before uploading the uncompressed blu ray of Guardians of the galaxy. On the other hand there are p…
FWIW, I'm in the UK and initial backup was great and fast (on a 1Gbps upload connection, I was getting maybe 10-15MBps upload consistently, which isn't bad considering it's across the Atlantic!). The biggest thing for this has been the multithreading that BB have introduced recently - setting 10 upload threads makes the backup super-snappy.
WRT priority backup - I ended up just excluding the "big folders" manually (reducing backup size by 90%) then including them when the initial backup of the important files was complete. Also, the client backs up small files first, then moves onto bigger files, which is a nice touch - generally I'd tend to care more about losing a large number of files than a few huge files.
Not affiliated with the company, just love their product!...
Re: Backblaze Storage Pod 5.0
#187Earlier quoted context omitted.
the bad thing about your good mission is that big guys starting to buy cheap hdds which means that prices will go up :(
Prices for the HDDs themselves? Doubtful, but maybe. The nice thing about consumer-grade HDDs is that consumers are not very price elastic, meaning if consumer HDD prices go up too much, consumers will just stop buying them (we saw this happen after the Thailand Drive Crisis). Hopefully what it'll mean is that drive manufacturers will instead start making lower-cost "professional" drives for this exact purpose. Cheap…
I still haven't been able to work out if prices ever went back to pre-Thailand prices. It pretty much coincided with the beginning of the Aussie dollar becoming worthless so that's skewed things a bit.
Re: Backblaze Storage Pod 5.0
#188Great to see a smaller, very open company remaining competitive against the giants in the space like Amazon. I'm particularly excited to see how their B2 service will stack up to Glacier for half the cost.
Yev from Backblaze here -> We're pretty excited about it. Beta users are providing some good feedback too! We think it'll definitely be the right choice for a lot of folks!
I keep checking my email for the invite link ;)
I think it'd be pretty interesting to expose it as a Windows filesystem with Dokan [0] and I'm sure Jeff from ExpanDrive [1] is looking at it too!
Re: Backblaze Storage Pod 5.0
#189Earlier quoted context omitted.
For a use like this EEC RAM is worth the money.
Alright, even with ECC and much faster DDR4 ram: http://www.crucial.com/usa/en/ct16g4rfd4213 and motherboard: http://www.newegg.com/Product/Product.aspx?Item=N82E16813157...
You might have missed the fact that the motherboard they use has built-in dual 10G Ethernet: http://www.supermicro.com/products/motherboard/Xeon/C600/X9S...
Once you account for the cost of a 10G NIC and the fact that having it integrated frees up another expansion slot, I'd say it's a pretty good deal.
Also, I'm not sure if there are any non-Xeon processors that support ECC.
Re: Backblaze Storage Pod 5.0
#190Earlier quoted context omitted.
Alright, even with ECC and much faster DDR4 ram: http://www.crucial.com/usa/en/ct16g4rfd4213 and motherboard: http://www.newegg.com/Product/Product.aspx?Item=N82E16813157...
You might have missed the fact that the motherboard they use has built-in dual 10G Ethernet: http://www.supermicro.com/products/motherboard/Xeon/C600/X9S... Once you account for the cost of a 10G NIC and the fact that having it integrated frees up another expansion slot, I'd say it's a pretty good deal. Also, I'm not sure if there are any non-Xeon processors that support ECC.
An extreme example of a non-Xeon that supports ECC: Atom C2750 :)