Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A million-dollar engineering problem

segment.com

Post reply on HN
Print view View original

171–180 of 265 posts

Re: A million-dollar engineering problem

#171

Post by austinhutch »

aresant wrote: post #17
teh wrote: post #6

I've been joking with friends that my next job will be AWS efficiency guru. I've somewhat optimized our own use, but I think I could use similar, simple rules to get 20% out of a 500k / month budget. Give me what I save you in 2 months and I'll have a good business :)

Go do it! I used that exact same model in Conversion Rate Optimization - get your conversion rate up, give me 30% of what we improve. And built that into a 20+ person digital agency billing millions of dollars a year before being bought out. Exactly how I did that and you can to: (1) Wrote topical, detail rich posts similar to the parent here about problems I was solving in CRO for a handful of customers, never discl…

When you say "trust symbol" do you mean the "verisign" logos and similar? If so - do you have real data to prove these make a difference in conversion rate? I have clients that come to me parroting the same tips and I imagine they are all reading the same nearly identical blog posts out there making this claim.

Re: A million-dollar engineering problem

#172

Post by mattmanser »

rbranson wrote: post #123
gravypod wrote: post #120

Earlier quoted context omitted.

> on top of that there's a 3-5ms delay between the database and website which causes all sorts of performance problems Yea that's something I don't get too. With all of that IPC going over amazon's network for all of their services how much time are programs wasting sending and waiting for messages from other amazon services? I never really thought about that. If someone could measure this it would be interesting.

I can't comment on Azure, but most of the complaints about networking issues in AWS went away when they redesigned their networks around an SDN model with VPC. We see consistent <1ms RTTs within region, and <300µs within availability zone (datacenter).

I've got a reply from Google Compute before after complaining about how bad Azure was saying they'd has similar problems but solved the latency between servers and databases. It basically just sounds like the Azure guys have severely mucked up their infrastructure.

We have a test page that runs 100 "SELECT 1" SQL queries ON THE SAME CONNECTION between an Azure website and an Azure database and it takes a whopping 250ms to complete at best. That's literally all it does. And their infrastructure is so bad that it'll vary from 250ms to 600ms in the space of 20 seconds in peak periods. That's damning, cloud infrastructures are there to part you from your cash, nothing more.

It's ridiculously bad, in my opinion no professional should EVER recommend Azure. My client is enamoured with telling people the system runs on Azure, with its "secure" network, nothing I've told him makes a difference, but he'd have a much better service if he moved it off Azure.

Because we develop with the entity framework it's quite easy to muck up and forget an include or two and then suddenly the queries spawn a hundred or two hundred basic SELECT queries to populate a `Order.OrderItem` or something as trivial. On a standard dedicated server setup or even your worst 5 year-old crappy dev laptop that's a ms or 2 extra, but on Azure it's performance death.

There are advantages to using Azure, I admit, the easy deploy from github for example, but that's more because setting up deploying from a repo to IIS is such a bad workflow at the moment and the IIS management too. They don't want to make it easy. With other clients I've setup moderately complicated deployment scripts, and once the initial work is done, it's much better than Azure, you run a bat, boom, deploy much faster than Azure manages. 10-20 seconds without even pre-compiling the pages, Azure will take 5 or 10 minutes to finally get round to doing it and woe betide you doing it at peak times as it will use up your memory allocation and simply hang for 20 minutes while you frantically try and restart it while the azure management portal has a massive spaz (yes, we unfortunately deployed a serious bug, fixed it, tried to re-deploy while under heavy load, cue website down for a ridiculous amount of time while their management portal threw a ton of weird and inscrutable errors before we finally managed to restart it).

Re: A million-dollar engineering problem

#173

Post by hluska »

sillysaurus3 wrote: post #134
sdflkd wrote: post #38

Earlier quoted context omitted.

How does one begin to learn about these things? Minimizing cost of running services sounds super interesting but as a student I've never had to deal with it and am basically starting with 0 knowledge.

I worked through the labs at https://pdos.csail.mit.edu/6.824/ for fun. It's more along the lines of "How can we write a distributed fault-tolerant database?" but you might like it anyway. Lab 4 is a beast with more lives than . The moment you think you've finally written your distributed system correctly, the unit tests will prove your service fails during XYZ network partition topologies. It's very worthwhile to be…

Thanks for the link to 6.824! It looks amazing and I can't to start working through the labs.

Re: A million-dollar engineering problem

#174

Post by rbranson »

ig1 wrote: post #157

However they miss the easiest fix: Calling their account rep at AWS and cutting a deal. AWS loves startups that could end up being huge customers so they're willing to slash bills upfront to help you get to growth stage; not only will they assign you an account rep but they'll have a rep whose job it is to build a good relationship with your VC. Speak to your account rep. Have your VC speak to their Amazon rep. Push…

That was definitely part of the effort; it just doesn't make for terribly compelling technical content. We have a healthy, active relationship with our AWS team. They worked with us to put in place minimum commits for several products, and examined both EDP as well as bulk RI purchase discounts.

Re: A million-dollar engineering problem

#175

Post by dangoldin »

Really interesting read. We've been in a similar mindset and it really comes down to understanding what's driving your costs and being creative around ways to minimize it. As others on here mentioned bandwidth and data transfer is a huge cost so if there's a way to make your caching more aggressive it's a big and easy win. Beyond that the low hanging fruit is in using reserved instances - there are a few options there but are significantly cheaper than paying the on demand price.

A shameless plug is I built a small visualization for the AWS detail log to help understand the cost drivers. All it is loading the data and visualizing it across a few dimensions but might come in handy for others - https://github.com/dangoldin/aws-billing-details-analysis

Re: A million-dollar engineering problem

#176

Post by curun1r »

illumin8 wrote: post #146
pm90 wrote: post #53

Earlier quoted context omitted.

I've always found it interesting that Netflix has not at least tried to go off AWS; if only because Amazon has a competing service with Amazon video. I'm sure its not an easy problem to solve, but Dropbox has gone off AWS as well; it seems like they would be much better served with their own "Video Cloud" with specialized hardware for streaming/processing videos.

Netflix realizes that the value presented by AWS for outweighs the costs. If you feel differently, you should probably ask yourself what information you are missing, rather than just dismissing Netflix and their decision. It's clear that Netflix knows more about operating applications with tens of millions of simultaneous global users than most companies...

I don't think Netflix and Amazon really compete yet. Very few people see the two services as either/or. They're more allies against the cable companies at the moment.

Re: A million-dollar engineering problem

#177

Post by logfromblammo »

EnFinlay wrote: post #132
kafkaesq wrote: post #116

Earlier quoted context omitted.

A blanket "never use AWS" isn't going to address it constructively. But that's not what the commenter was saying, of course. More like (paraphrasing) "before you go full-hog on AWS, do some simple math first." Which an amazingly high percentage of people neglect to do, these days.

> At this point simply not using AWS is a competitive advantage.

I read this as suggesting that a simple arbitrary rule to not use AWS is likely to save both money and also the time that would otherwise be spent tuning your use of AWS to cost less money.

Your time is valuable too. It could be spent developing something novel instead of messing around with server configuration.

Re: A million-dollar engineering problem

#178

Post by patrickg_zill »

brianwawok wrote: post #164
patrickg_zill wrote: post #161

Earlier quoted context omitted.

The instant you wrote "128vCPU" you undermined your own argument. What is a vCPU?

It's early defined in the docs. The machine also has a physical core count that is yours.

There is nothing stopping Amazon from having a machine that is double that config, and renting you half of it, however.

As soon as 32GB DIMMs drop enough in price, I would expect them to do exactly that. You can configure a Dell R930 with e7 Haswell CPUs online, if you want to double-check it. Not sure if the R930 can go to 8 CPUs (which the e7 Haswells CPUs support).

Re: A million-dollar engineering problem

#179

Post by rbranson »

mattmanser wrote: post #172
rbranson wrote: post #123

Earlier quoted context omitted.

I can't comment on Azure, but most of the complaints about networking issues in AWS went away when they redesigned their networks around an SDN model with VPC. We see consistent <1ms RTTs within region, and <300µs within availability zone (datacenter).

I've got a reply from Google Compute before after complaining about how bad Azure was saying they'd has similar problems but solved the latency between servers and databases. It basically just sounds like the Azure guys have severely mucked up their infrastructure. We have a test page that runs 100 "SELECT 1" SQL queries ON THE SAME CONNECTION between an Azure website and an Azure database and it takes a whopping 250…

Yikes. I've only really had true production experience with AWS. I guess all clouds aren't made the same.

Re: A million-dollar engineering problem

#180

Post by closeparen »

ryeguy wrote: post #139
bearjaws wrote: post #106

I work in a start up, we own all our own hardware, and it is HELL. We are forced to pay extremely large sums of money to upgrade our infrastructure as any purchase requires a redundant piece as well. For example we have used 90% of our SANs storage, our IO is suffering and now were looking at purchasing two $10k SANs to upgrade. In the meantime, we have probably spent over 10k worth of development time to compress, c…

Why is it always colo vs cloud in these arguments? What about renting everything monthly from ovh, rackspace, softlayer etc? Many of them can provision dedicated servers within minutes, too.

AWS provides a SAN in the form of EBS.

Colocation provides a SAN in the form of calling up a datacenter storage company and having some consultants install you a SAN.

What are you going to do to get similar functionality on OVH, Rackspace, or Softlayer?

Post reply on HN