Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A million-dollar engineering problem

segment.com

Post reply on HN
Print view View original

91–100 of 265 posts

Re: A million-dollar engineering problem

#91

Post by omarchowdhury »

aresant wrote: post #17
teh wrote: post #6

I've been joking with friends that my next job will be AWS efficiency guru. I've somewhat optimized our own use, but I think I could use similar, simple rules to get 20% out of a 500k / month budget. Give me what I save you in 2 months and I'll have a good business :)

Go do it! I used that exact same model in Conversion Rate Optimization - get your conversion rate up, give me 30% of what we improve. And built that into a 20+ person digital agency billing millions of dollars a year before being bought out. Exactly how I did that and you can to: (1) Wrote topical, detail rich posts similar to the parent here about problems I was solving in CRO for a handful of customers, never discl…

That's awesome, do you have more information about your acquisition?

Re: A million-dollar engineering problem

#92

Post by mattmanser »

djchen wrote: post #37
gravypod wrote: post #34

I've always been set aback by how much AWS servers cost. Maybe I'm just too cheap but you can go to very reputable hosting companies and get things at fractions of the price. For example, if I need 16GB of memory and 4 cores here are my options: * AWS (t2.xlarge) $137.62 * OVH (EG-16) $79 * So You Start/OVH (SYS-IP-1S) $42 * Kimsufi/OVH (KS-3B) $29 * Hetzner (EX41) €46.41 [Lowest cost model is 32GB of ram] * Joe's Da…

Most of those providers you mentioned aren't as reliable and scalable as AWS, Google Cloud, Azure, etc. That isn't an apples to apples comparison. I would not want to host my business on So You Start, Kimsufi, Hetzner, and especially not Joe's Data Center. I have personally used Joes DC and they have had numerous outages in the past. Hetzner is known for terminating you for any sort of "DOS" like traffic, including h…

Reliability wise, do you actually use these services, as in my experience that's a complete myth.

One of my clients on a dedicated server has never gone down. The site is blazingly fast, barely touches 5% CPU and pages have sub 50ms response times. Deploys take 10 seconds or so, I could make it faster but it's not really worth the cost/benefit.

My client on Azure, with a significantly lower visitor count, pays 3-10 times as much, the site is sluggish, takes a long a time to spin up after deploys, hangs ocassionally, we've caught the whole site being offline a couple of times then it mysteriously starts working again with nothing in the logs, had a deploy to one site take down other sites and on top of that there's a 3-5ms delay between the database and website which causes all sorts of performance problems when a page makes too many DB requests.

They've had to "scale up" to premium database in the past to handle loads I know a much cheaper dedicated server would have handled without even thinking about it.

On top of that, Azure's management portal is super slow, regularly fails to execute commands and is incredibly frustrating to navigate and use.

The claimed machine you get on cloud aren't anything like as performant as supposedly similar machines on dedicated.

I'll admit I've generally found AWS to be significantly better than Azure, but still very expensive.

And AWS went down a couple of weeks ago too.

Re: A million-dollar engineering problem

#93

Post by that_colo_guy »

Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

anon account for obvious reasons.

Came here to share my story, which is very close to what you describe.

tl;dr: Moved from AWS to colo in desperate attempt to make balance sheet more attractive. It worked.

Before I took over the engineering bits at this plateau/failing startup, they were all in on AWS. Monthly costs were running at around £3k. Which is not much, but since the revenue was tiny, the costs of hosting was "considerable". After a monumental effort on my part to try to make things more efficient and streamlined, I eventually run out of motivation (but that's another story) so I expressed my wish to leave to the founder and he finally agreed to my years old suggestion of going lean-and-mean and in one last attempt to make the company more attractive to one last investment round (E by then). Make it or break it, as the say goes.

On that same week I came across a HPE "Buy One Get One Free" offer (similar to this one https://www.serversdirect.co.uk/pdf/BOGOF-Gen9-Servers.pdf). Then found guy who was re-selling colo space at a under-used datacentre in London. Since I (sysadmin/DevOps/InfoSec/Backend engineer Jack-of-all-trades master of none) was the person who would end up managing the new metal, I picked a location within an hours drive from home.

And so it happened.

2x 1U HPE proliant servers with decent CPU and 32G of RAM. And 6x Samsung 850 SSDs off Amazon. Total bill for the hardware (BOGOF refund included): £2600 ish.

Post migration, monthly hosting costs: £50 for the colo; £50-ish AWS S3+CloudFront (10M assets would increase the SSD storage costs too much); £40 in taxi fares when I visit the DC once a month for kernel updates.

And just like that, we broke even the very next month after the last "big" AWS bill. Our product wasn't all that exciting, but 2 months later we found a buyer who was happy to snatch this "non loss making" operation.

3 months after the acquisition, they had already migrated the whole thing back to AWS and the HPE proliant servers were gathering dust in their office.

Re: A million-dollar engineering problem

#94

Post by user5994461 »

rbranson wrote: post #28
nik736 wrote: post #5

Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.

It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows…

Try all the same on Google Cloud, you'll save half the money compared to AWS, probably more.

Disclaimer: I am NOT affiliated to Google in any way.

Re: A million-dollar engineering problem

#95

Post by user5994461 »

rbranson wrote: post #85
lafay wrote: post #72

Earlier quoted context omitted.

This article makes a good case for having at least one early team member who knows how to build out physical infrastructure at scale, even if you initially decide on AWS / GCE / Azure: http://firstround.com/review/the-three-infrastructure-mistak...

This might sound a bit extreme, but contrary to most FRR pieces this is garbage advice. Startups should focus first on getting their product-market fit before trying to future-proof themselves from a future that doesn't yet exist.

Register your startup to Amazon or Google, you'll get 100k of free credits.

Keep a look on your bill because you don't want to run anything. But take advantage of having no hassle for your first years.

Re: A million-dollar engineering problem

#96

Post by user5994461 »

asptimothy wrote: post #11
nik736 wrote: post #5

Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.

I was curious about that myself. What's the best way to model the costs associated with maintenance overhead that comes with bare-metal, vs the savings from managing bare-metal servers using a service (like packet), co-locating your own hardware or running your own datacenter. My gut feeling is that you have to get to an extraordinary size to realize any meaningful savings, but that's primarily based on Dropbox's mig…

It's rather simple, you count the days you are wasting on handling the low level hardware.

The entry cost is £500 per day or $1000 for an engineer.

5 hours on the phone to find the hardware and agree on the order with DELL + an afternoon with customer support because they shipped the servers without hard drives + your project is delayed by an entire week because you don't have the resources => 1 day + 1 day + half a week.

These things would have been 5 minutes on a cloud.

Re: A million-dollar engineering problem

#97

Post by aresant »

omarchowdhury wrote: post #91
aresant wrote: post #17

Earlier quoted context omitted.

Go do it! I used that exact same model in Conversion Rate Optimization - get your conversion rate up, give me 30% of what we improve. And built that into a 20+ person digital agency billing millions of dollars a year before being bought out. Exactly how I did that and you can to: (1) Wrote topical, detail rich posts similar to the parent here about problems I was solving in CRO for a handful of customers, never discl…

That's awesome, do you have more information about your acquisition?

This was several years ago before we had the "growth hackers" lexicon.

One of our customers bought the entire company to get a hold of the core team + essentially continued the "30%" deal as a long term incentive as convertible equity.

Was a good run, and reinforces my post of if you can bring measurable / substantial value to large enterprise companies amazing things can happen.

Large enterprise have fun accounting terms like "capitalizing an acquisition", eg they don't buy you out of cash flow. And can even carry debt on the purchase / incentive programs etc that not only make you more valuable to them, but create incentives for them to buy smaller companies.

Happy to answer any more specific questions.

Re: A million-dollar engineering problem

#98

Post by kornish »

user5994461 wrote: post #84
kornish wrote: post #26

The Dynamo incident highlights an important lesson when using consistently hashed distributed data stores: make sure the actual distribution of hash keys mirrors the expected distribution. (though to their credit, someone writing an automated test using a hard-coded key was beyond their control). Incidents like this are generally why rate limits exist, which they don't currently have [0], but perhaps they'll consider…

> the per-user ID write bottleneck The basics is to have less machines but more powerful. It helps to handle the targeted bursts. The advanced is to have a layer of queuing before the ingestion, where you can do magic with distribution rules, rate limiting and dropping peak traffic.

Yep, you're absolutely right. In multitenant SaaS apps with extremely uneven distribution of traffic, it's pretty common for large customers to get their own dedicated DB servers.

> The advanced is to have a layer of queuing before the ingestion, where you can do magic with distribution rules, rate limiting and dropping peak traffic.

And batch loading – don't forget batching!

Re: A million-dollar engineering problem

#99

Post by Im_a_throw_away »

aresant wrote: post #17
teh wrote: post #6

I've been joking with friends that my next job will be AWS efficiency guru. I've somewhat optimized our own use, but I think I could use similar, simple rules to get 20% out of a 500k / month budget. Give me what I save you in 2 months and I'll have a good business :)

Go do it! I used that exact same model in Conversion Rate Optimization - get your conversion rate up, give me 30% of what we improve. And built that into a 20+ person digital agency billing millions of dollars a year before being bought out. Exactly how I did that and you can to: (1) Wrote topical, detail rich posts similar to the parent here about problems I was solving in CRO for a handful of customers, never discl…

This sounds super interesting! Where can we learn more about your story? Do you have an email or Twitter to contact you?

Re: A million-dollar engineering problem

#100

Post by QuinnyPig »

rhizome wrote: post #78
QuinnyPig wrote: post #62

Earlier quoted context omitted.

I agree with a lot of what you're saying-- but that trough is a lot wider than most people think. At the single instance very small scale, it simply doesn't matter all that much. "Our business failed because our infrastructure bill was $400 instead of $150 every month" isn't that common of a story. Base load can be effectively migrated to AWS, depending upon how it's built / structured. It's hard to lose money on inf…

Are bursting and resizing an issue in the trough, though? GP's premise is that these companies have platform and load stability. There can be marginal gains in having a dynamic architecture, sure, but when it also costs "months of engineering," I don't see where the savings would be compelling, or even guaranteed.

There are companies "in the trough" who are contently spending tens of millions a year on AWS. It's hard to give generic advice for all situations-- as with so much else, context matters heavily.

Those marginal gains could potentially pay for a dozen engineers...

Post reply on HN