Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Highly Available Block Storage

digitalocean.com

Post reply on HN
Print view View original

131–140 of 145 posts

Re: Highly Available Block Storage

#131

Post by ymse »

wiremine wrote: post #125
olavgg wrote: post #79

Earlier quoted context omitted.

Thank you for telling us these details. I can't help but I note that you benchmark the drive with a block size of 1024. Minimum block size of a modern SSD is 4096, anything lower would just cause unnecessary load. Also if you want to benchmark a drive/block storage I highly recommend using fio.

Thx for the tip, I'll give that a try!

    fio --direct=1 --rw=write --ioengine=libaio --runtime=300 --bs=4k --numjobs=4 --iodepth=32 --size=2G --name=4096-direct --filename=/dev/vdb --group_reporting
This is my "dd" test for any storage system. Filename can be a regular file; you can also pass --directory instead. I would love to see numbers for both this test as well as with --bs=4M.

Also experiment with --rw=rw|randrw|read :)

(and make sure your device is large enough to hold numjobs*size)

Re: Highly Available Block Storage

#132

Post by Jedd »

andybak wrote: post #52
Jedd wrote: post #37

Earlier quoted context omitted.

> Anyone see a flaw in this? Perhaps not a flaw, but some issues with your setup are implied. If you're rebuilding from scratch because you're not sure that you can update things, then you're probably in need of a configuration management tool (I'm a big fan of saltstack[1], mostly because I don't like Ruby or DSL's, but there's lots of options out there[2]) If you're worried you're going to lose transitory data, it…

I probably wasn't clear: 1. I am already using Ansible - but I want to run my "build from scratch" playbook every time I push a change instead of my "git pull etc." playbook. 2. I'm not scared of losing data because of backups or similar - it's because I want to start from a clean droplet every time I push a change. I do know in theory that playbooks are idempotent and running the 'build from scratch' playbook on a r…

  >  1. Someone has inadvertently modified the environment at some point.
  >  A quick-fix or accidental change or an alteration in my playbook that
  >  isn't clearly reflected in the running instance
This will cause you comparable problems whether you're building from scratch or upgrading in place (iff you're using a tool like ansible). As noted by another responder, you've got a human problem there, not necessarily a technical one.

  >  2. Not all ansible modules are perfectly idempotent. It's a
  > fairly leaky abstraction.
I've not used ansible, but have used a few other tools, so I understand how this is (easily) possible. As in other cases, the trick, of course, is to fix this. ; )

  >  3. If you never build from scratch apart from the first time then
  >  you're not really testing your automated deployment. The next time
  > you really need to build from scratch you might get a shock.
Totally agreed.

Your original post suggested the problem you had was with transitory data - evidently your concern is more to do with the ability to consistently and reliably rebuild a known and trusted environment.

Persistent block storage won't help you with that.

Re: Highly Available Block Storage

#133

Post by jclulow »

3pt14159 wrote: post #43

I have been asking for non-SSD on DO for a long time now. My heart jumped when I saw the HN title, only to be dashed on the rocks. What are us data nerds supposed to do? We want to take 10 terabytes, run a batch process on it, keep the 20TB, then continue with about 5GB of working data until the next month's terabyte comes in, then we want to batch through the 21TB. Right now the price slider doesn't even go up to 21…

If you have a lot of data that you want to ingest and store, without keeping it in an active instance, you might want to take a look at Joyent Manta[1]. We allow you to run compute jobs directly on the storage servers themselves -- basically anything you can do in a UNIX environment with a script you can run against an object in the store, from "grep" up to "python" and beyond. If you were to select "single copy" storage, the public service pricing[2] might be more interesting for you; if you grow too big or want to move onto your own hardware, the stack is open source[3] and/or we can sell you support!

Disclaimer: I work at Joyent.

[1]: https://www.joyent.com/manta

[2]: https://www.joyent.com/pricing/manta

[3]: https://github.com/joyent/manta

Re: Highly Available Block Storage

#135

Post by boulos »

boulos wrote: post #62
mwcampbell wrote: post #55

I think this might be a mistake. Ever since Joyent's commentary on one of the big Amazon EBS failures in 2011 [1] [2] [3], I've been suspicious of all network-attached block storage. Then again, I haven't heard of any big EBS failures recently; I wonder what changed. [1]: https://www.joyent.com/blog/on-cascading-failures-and-amazon... [2]: https://www.joyent.com/blog/magical-block-store-when-abstrac... [3]: https://w…

Network block storage isn't inherently broken, the initial EBS implementation was frankly just unreliable. We've not had anything like those dark days with Persistent Disk. It's still true that having your storage across the network opens you to networking failures taking out your storage, but the gain in durability and maintenance pays for it (in our case, live migration would just be crazy with local spinning disks…

Too late to edit, so I'll put that up here: we do migrate your local SSD but it's best effort and it takes a while (something like 30 minutes maybe?). But for comparison, our SSDs have crazy amounts of read/write bandwidth (1 gigabyte per second) compared to a single spinning disk. And while your VM is still going, it's not that disruptive to be re-syncing an SSD, randomly seeking around a hard drive while someone is trying to use it for say streaming reads will take their reasonable performance and tank it.

tl;dr: Yes, we do this for our scratch disk SSDs, but you wouldn't want to rely on this for a guaranteed durable storage (as it'll take a really really long time to move 100 GiB of HDD).

Re: Highly Available Block Storage

#136

Post by vidarh »

brazzledazzle wrote: post #89
vidarh wrote: post #48

Earlier quoted context omitted.

I tend to avoid those config management tools other than for basic bootstrapping exactly because while you can use those too to recreate from scratch, when you don't do that, you leave the door open for undocumented, unknown state, since most of them basically take a system in an unknown-but-hopefully-mostly-consistent state and try to bring them to a known state. But they'll only be in a known state in that case if…

I actually agree with your overall point but generally speaking if someone is making changes outside of your standardized toolchain you have a human problem. Emergencies aside, you use those tools for a reason. Straddling the fence is almost the worst of both worlds.

Agreed. But in my experience, the easier you make going outside the process the easier it becomes to invent excuses for why it is ok. A lot of my job involves making the right thing to do the path of least resistance, because when it isn't, because humans overall tend to be the cause of a whole lot more of the problems than the servers.

Re: Highly Available Block Storage

#137

Post by walkertraylor »

marcstreeter wrote: post #26
drtse4 wrote: post #17

A bit pricey for the long term but great if you just need to add some disk space to your vm and don't need the other improvements more expensive vms give you. I use DO mostly to compile stuff on Linux when i don't have access to a physical server, and storage size is always a problem.

The purpose of the block storage in this instance isn't about giving your vm/droplet more space. It's separation. That way any data that's on that device can be attached to another vm/droplet. It probably would be more cost effective just to upgrade the vm/droplet if space were a concern. It's at least how we've marketed the same feature for the past year or two via Codero's portal. Not to say I don't like how DO has…

Exactly! It's about having an easy upgrade path, generally reducing the amount of work for common operations, and for more flexibility engineering your cloud architecture. It's nice that it happens to be SSD, but highest performance or lowest cost per GB isn't necessarily the only cost saving factor.

This a main reason I still use Amazon AWS: I can create an instance and if it doesn't perform, upgrade it until it does. Then when I'm finished, kill the instance and save the volume. Next time I need it, just create the instance for the job, perhaps at spot pricing, then kill it again.

Re: Highly Available Block Storage

#138

Post by onnoonno »

notacoward wrote: post #118
e1ven wrote: post #94

Earlier quoted context omitted.

I usually use Bonnie++ when testing disk performance. http://www.coker.com.au/bonnie++/

The only thing worse than dd is bonnie++. Please, folks, use fio, or at least iozone, with multiple threads and/or a queue depth greater than one.

Are you sure? Bonnie++ has byte-wise and block-wise tests. Yes, the byte-wise tests are CPU bound (as expected), but I have not seen that for the block-wise tests on any machine so far?

Re: Highly Available Block Storage

#139

Post by slgeorge »

brazzledazzle wrote: post #85
fweespeech wrote: post #70

Earlier quoted context omitted.

Expecting a hosting provider to wade into fair use waters hand-in-hand with you is generally unwise. The vast majority will not unless you are a large customer with your own legal staff on retainer to provide the appropriate legalese/notices/etc.

I agree, but why not just take action based on the only things you have to? Namely court orders and DMCA notifications. It seems like a more complex balancing act to assume that any notification you receive is in good faith.

The primary thing commentators on HN don't acknowledge is that dealing with legal issues, whether in court or phone calls/letters, costs money and time. That takes a company away from their core objective of delivering service to their customers (in the round) and making a profit. Most companies would rather spend their money on product/customer development than hire more lawyers. But, there's a lot of low-level 'rights holders' hassle for many digital companies. A steady and constant stream of Copyright and (separately) DMCA take-down notices. When I was in a service provider we didn't think it was our job to resolve those issues (it's for the end-user site to defend whether it is or is not copyright infringing) because we were just providing a service: as a SP you're just caught in the cross-fire.

It comes down to them making a risk assessment on whether they're more likely to incur cost from taking down versus ignoring. In some cases the balancing act is due to the nature of the outcomes: on the one side if you do nothing it's unlikely (low probability) but you could land-up in court and incur millions, versus on the other side refunding a customer their annual subscription and taking a reputation hit - it's the asteroid crash problem. It's notable that they asked their customer for a response and didn't get (what they consider) a satisfactory one. Who knows the specifics in this situation, but they won't have done it just on a whim!

Re: Highly Available Block Storage

#140

Post by andybak »

Jedd wrote: post #132
andybak wrote: post #52

Earlier quoted context omitted.

I probably wasn't clear: 1. I am already using Ansible - but I want to run my "build from scratch" playbook every time I push a change instead of my "git pull etc." playbook. 2. I'm not scared of losing data because of backups or similar - it's because I want to start from a clean droplet every time I push a change. I do know in theory that playbooks are idempotent and running the 'build from scratch' playbook on a r…

> 1. Someone has inadvertently modified the environment at some point. > A quick-fix or accidental change or an alteration in my playbook that > isn't clearly reflected in the running instance This will cause you comparable problems whether you're building from scratch or upgrading in place (iff you're using a tool like ansible). As noted by another responder, you've got a human problem there, not necessarily a techn…

What I was hoping it would do is enable me to use a single droplet but get a fast, minimal down-time 'from scratch' deployment that retains user data.

The reason I want a single droplet is mainly cost. I use one droplet per client rather than a multitenant setup. I could use S3 and a separate hosted db but for various reasons I'd prefer not to.

Post reply on HN