Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A million-dollar engineering problem

segment.com

Post reply on HN
Print view View original

211–220 of 265 posts

Re: A million-dollar engineering problem

#211

Post by doppel »

bearjaws wrote: post #106

I work in a start up, we own all our own hardware, and it is HELL. We are forced to pay extremely large sums of money to upgrade our infrastructure as any purchase requires a redundant piece as well. For example we have used 90% of our SANs storage, our IO is suffering and now were looking at purchasing two $10k SANs to upgrade. In the meantime, we have probably spent over 10k worth of development time to compress, c…

I have the opposite experience. We have a quite storage-heavy service (video delivery, multiple versions of each), and we recently had to upgrade our SAN to a) a newer system and b) more capacity.

As part of this process, we looked into AWS and similar products, and with a naive 1:1 move it would be 3 times more expensive than getting a hardware deal with ramp-up (make a deal for X, buy 20% * X upfront, then buy the rest as needed). Implement some better scaling up and down might have decreased the cost to only a 2x multiplier.

Having your own hardware is not a walk in the part, and you need at least one experienced sysadmin to help with server setup, but I feel that for a steady company the cost benefit alone is worth it. We still use a CDN on top of our setup to alleviate pressure to our servers, but otherwise everything runs on our own hardware.

Re: A million-dollar engineering problem

#212

Post by Curious42 »

edanm wrote: post #208
hueving wrote: post #202

Earlier quoted context omitted.

Gross. AWS is becoming the Cisco of cloud computing. Get people hooked when they are low revenue and twist the screws in later when they get big.

In what way is this gross? They're giving people what is essentially a loan, which they are uniquely able to give, to help them start a business, in the hopes that this will be an even bigger business. This is almost the definition of a win-win. Customers are free to leave at any time or work with other providers, but they choose AWS because they give them more value.

Wouldn't it irritate AWS if startups left once they got big, despite receiving special care services from AWS? How do they deal with that?

Re: A million-dollar engineering problem

#213

Post by marknadal »

Great article. I wish/hope/expect everybody to take this type of work/attention as seriously as they have. I am as much of a penny pincher as you can get, so our team built a database adapter on AWS that can handle 100M+ records for $10/day, demo here: https://www.youtube.com/watch?v=x_WqBuEA7s8 .

I'm surprised not more people are price/performance sensitive. My goal is to get a system up to 10K table inserts/second on a free Heroku instance, we're already doing on low end hardware 1750 table inserts/second across a federated system. But price tuning is one of the most important things when engineering for high load, as well.

Glad to see articles like this. :)

Re: A million-dollar engineering problem

#214

Post by metafunctor »

QuinnyPig wrote: post #62
qeternity wrote: post #59

Earlier quoted context omitted.

I am curious your thoughts on the following. It seems like there is a trough of benefit to AWS. In the mid-size world, especially where computing needs are relatively stable/predictable, you are much better leasing dedicated servers from the usual suspects. Below that size, the linear scaling allows small scrappy startups to really scale/burst with their needs. And at large scale, but below Google/Facebook et al scal…

I agree with a lot of what you're saying-- but that trough is a lot wider than most people think. At the single instance very small scale, it simply doesn't matter all that much. "Our business failed because our infrastructure bill was $400 instead of $150 every month" isn't that common of a story. Base load can be effectively migrated to AWS, depending upon how it's built / structured. It's hard to lose money on inf…

Resizing, bursting, and duplicating on the fly aren't exclusive to AWS, though.

A lot of teams rent space from places like OVH or Hetzner because the raw power is so much cheaper, and they already have the talent and tools to scale and manage a computing fleet (people like to think it's harder than it actually is).

Re: A million-dollar engineering problem

#215

Post by joelbondurant »

zenlikethat wrote: post #135
zzzcpan wrote: post #86

Earlier quoted context omitted.

"With Amazon you have so many options when it comes to storage, compute power, load balancing..." It used to be different, but today their most important offerings have open source equivalents. They are essentially off the shelf, because for your own projects there is no need to care about abuse or noisy neighbors. Most importantly, though, it's going to be cheaper and you still will be able to use cloud where it shi…

> It used to be different, but today their most important offerings have open source equivalents. Seriously? What about S3, RDS, VPC, IAM? The engineering $$$ (in time) to set up and maintain open source equivalents of these would not be cheap.

S3 is shit compared to HDFS.

Re: A million-dollar engineering problem

#216

Post by ig1 »

rbranson wrote: post #174
ig1 wrote: post #157

However they miss the easiest fix: Calling their account rep at AWS and cutting a deal. AWS loves startups that could end up being huge customers so they're willing to slash bills upfront to help you get to growth stage; not only will they assign you an account rep but they'll have a rep whose job it is to build a good relationship with your VC. Speak to your account rep. Have your VC speak to their Amazon rep. Push…

That was definitely part of the effort; it just doesn't make for terribly compelling technical content. We have a healthy, active relationship with our AWS team. They worked with us to put in place minimum commits for several products, and examined both EDP as well as bulk RI purchase discounts.

I'd say especially in your case you could be pushing for additional discounts beyond what's available to everyone else.

You're the ideal case for this, not only will you be a large customer in your own right if you succeed but you have a multiplier effect in that your own clients become heavier users of AWS because of your product encouraging users to move large amounts of data from Salesforce, etc. into AWS.

I don't what level the rep you're dealing with is, but if you're not doing so already I'd strongly encourage you to build higher level relationships with AWS.

Re: A million-dollar engineering problem

#217

Post by tiplus »

gkop wrote: post #141
user5994461 wrote: post #109

Earlier quoted context omitted.

-looks at this datacenter- Well, actually it doesn't. RAM and Xeons are expensive.

So how many does it buy?

I would estimate the cost for three years like this:

- 1U single socket Supermicro Xeon E5v4-2630 10core+1GTX1070 GPU = 1500EUR

- operation costs (power+cooling) over 3 years = 1500EUR

- cost of 2 hardware admins 160k/year

- renting local rack space 15EUR/U

This equals about 145 servers for 3 years for us for 1 Million. The setup is highly optimized for _one_ set of data crunching tasks, not DB. We do not need much ram, or network but require a specific CPU/GPU ratio.

Re: A million-dollar engineering problem

#218

Post by stymaar »

I don't know if LBO funds are doing this already but I think it's most likely going to become a trend in the future :

- buy a SaaS company which hosts all its infra on the cloud

- move everything to bare metal

- double the company's margin

- profit

Re: A million-dollar engineering problem

#219

Post by julienba »

DISCLAIMER: It can be viewed as an advertisement for my project

Few months back I have start a company for that exact purpose : helping you decreasing you AWS bill. https://wizardly.eu/

The focus is around reservation, unused resources and tracking cost evolution on a day to day basics.

I don't get that much traction ( I'm not a marketer and it's my first SaaS project, still a lot to learn in that area). I'm looking for more users feedback, if you willing to invested few minutes in my project that would really helpful. It's also completely free now.

As the article say, their is no silver bullet. Specially if you need infrastructure changed. But it's possible to have an interesting cut by using all the AWS pricing options.

Making teams aware and responsable of their costs is also making a big differences. Often, people creating and managing infrastructure have no view of what it is costing for their company.

Re: A million-dollar engineering problem

#220

Post by SudhindraBhat »

ars wrote: post #198
whalesalad wrote: post #140

Earlier quoted context omitted.

The problem is not AWS. The problem is poorly architected systems and software. We see a similar issue with the "we switched from PHP to Haskell on .NET in Azure and improved performance 3000x" -- the issue wasn't the fault of the language, it was the way it was used. You put bald tires that are over-inflated on a Porsche and you're gonna have a shitty experience. Sure, bare-metal is gonna have performance benefits o…

> You put bald tires that are over-inflated on a Porsche and you're gonna have a shitty experience. This is off topic, but bald tires have better traction than grooved ones in dry conditions. (But the slightest bit of rain and they will not hold.) Also tire inflation does not have a strong correlation with traction (there is some, but it's not large), so it's not so obvious that over-inflated means a bad experience.

Bald tires also mean that the tire compound has degraded. So it is going to have worse traction. Slick tires on the other hand are new without any grooves.
Post reply on HN