Live data from Hacker News

A million-dollar engineering problem

segment.com

151–160 of 265 posts

Re: A million-dollar engineering problem

#151
post #142

For us the ECS autoscaling feature is way too naive. We need to do our own controller on some more responsive metrics. The customization support is pretty bad.

Would you mind saying more about what you had to build, and what you would liked to see? One thing I'd like to see in the future is scaling actions that supported multipliers, as well as metric math support (scale service to request_count/acceptable_requests_per_service).

One thing we experienced at Segment was the fact that we needed to quickly handle a surge in volume but couldn't overload partners. Essentially we wanted something that scaled up quickly at first but was pretty conservative after that.

We settled on constant increases/decreases using queue depth thresholds but ideally ECS would support feeding multiple metrics and doing some basic math to figure out how much we're currently process, how long it would take to flush the queue, and how many more containers we need to drain in a timely fashion.

Re: A million-dollar engineering problem

#153
> In some cases, we’re currently packing 100-200 containers per instance.

I have a very basic "Docker 101" question here.. Can someone explain how you might get to >100 containers on a given instance? That seems to imply that there are multiple containers of the same microservice running on one machine. Why would you pack duplicate containers like that instead of using one larger container?

Re: A million-dollar engineering problem

#155
post #53

Earlier quoted context omitted.

I've always found it interesting that Netflix has not at least tried to go off AWS; if only because Amazon has a competing service with Amazon video. I'm sure its not an easy problem to solve, but Dropbox has gone off AWS as well; it seems like they would be much better served with their own "Video Cloud" with specialized hardware for streaming/processing videos.

Netflix realizes that the value presented by AWS for outweighs the costs. If you feel differently, you should probably ask yourself what information you are missing, rather than just dismissing Netflix and their decision. It's clear that Netflix knows more about operating applications with tens of millions of simultaneous global users than most companies...

>If you feel differently, you should probably ask yourself what information you are missing

That's something that should be applied almost universally. :)

Re: A million-dollar engineering problem

#157
However they miss the easiest fix: Calling their account rep at AWS and cutting a deal.

AWS loves startups that could end up being huge customers so they're willing to slash bills upfront to help you get to growth stage; not only will they assign you an account rep but they'll have a rep whose job it is to build a good relationship with your VC. Speak to your account rep. Have your VC speak to their Amazon rep. Push them to cut your prices.

I've known startups which have cut their monthly bill by six figures going this route. AWS doesn't want your startup to fail and on the margin these resources cost them close to zero so they're willing to deal, make use of it.

Re: A million-dollar engineering problem

#158
post #150
post #139

Earlier quoted context omitted.

Why is it always colo vs cloud in these arguments? What about renting everything monthly from ovh, rackspace, softlayer etc? Many of them can provision dedicated servers within minutes, too.

Also some combination of cloud and private servers. If I was starting up a Netflix competitor for instance, the site could probably run fine from owned hardware, but the content itself would be better off being cloud based.

Ironically netflix does the exact opposite. Also, you definitely don't want to have the content on the cloud - the data transfer pricing is where they make significant profit. Both AWS and GCE are ~$100/TB outgoing.

Re: A million-dollar engineering problem

#159

Earlier quoted context omitted.

The company I work for is backed by a bunch of data in a MySQL database, scripts to aggregate and process more data into that database, and simple internal webapp to both manage inventory, track tasks and run lots of reporting on the aggregated data. We made do for quite a while on a single colocated database with 32 GB of RAM and a few fast SSDs (for the time, which was about 6 years ago), and a few multi-TB archiva…

May I make a suggestion: Try Amazon Aurora. You'll get 500,000 read operations per second, and 100,000 writes per second. It supports up to 64TB of total capacity, and 244GB RAM on your primary master. You can also have up to 15 read replicas with a similar footprint. You should seriously try it. It's amazing.

Maybe on the next upgrade iteration. As it is now, the server is performing well, and it can scale to 1.5 TB of RAM, so that aspect has quite a bit of room.

Re: A million-dollar engineering problem

#160

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

You are absolutely correct, and I have seen this sort of thing time and again... "cloud" doesn't mean that the server is magically faster - it usually means "a great deal slower" because it is a shared resource.
Post reply on HN