Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A million-dollar engineering problem

segment.com

Post reply on HN
Print view View original

181–190 of 265 posts

Re: A million-dollar engineering problem

#181

Post by brazzledazzle »

QuinnyPig wrote: post #127
alexkon wrote: post #105

Earlier quoted context omitted.

How do you approach a horrifying AWS bill? Can you recommend any reading? I’m going to need to optimize a tiny fleet on GCE pretty soon.

It gets into the specifics of "what makes it horrifying" pretty quickly. It usually boils down to "nobody's responsible for cleaning things up," a misunderstanding of the billing model, and by the time anyone cares to fix it the billing report is multiple gigabytes. Step 1 is always "figure out what you've got, and why it is this way."

So if the best way to avoid the mess is accountability for keeping things clean, how do you do that from the start? Would something as simple as tagging resources to specific teams/departments be a good place to start? I'm helping with setting up a completely new account/environment and I know once the buzzword fever passes and the annual costs are reviewed there will be frantic requests to cut down the bill. Getting ahead of that inevitability with a plan and some preparation will save us a headache.

Re: A million-dollar engineering problem

#182

Post by Dylan16807 »

bearjaws wrote: post #106

I work in a start up, we own all our own hardware, and it is HELL. We are forced to pay extremely large sums of money to upgrade our infrastructure as any purchase requires a redundant piece as well. For example we have used 90% of our SANs storage, our IO is suffering and now were looking at purchasing two $10k SANs to upgrade. In the meantime, we have probably spent over 10k worth of development time to compress, c…

And how much would AWS cost for your total storage, per month? Because if that's $5k then it sounds like you still picked the cheaper route...

Re: A million-dollar engineering problem

#183

Post by vacri »

QuinnyPig wrote: post #64
a13n wrote: post #42

Earlier quoted context omitted.

Also as a SaaS founder running on AWS, I would totally do this once our AWS bill is in the 4-5 figures.

It's almost like clockwork. Companies start wondering around $10K a month; they start doing something about it at $50K a month. I can almost set my watch by it. This turns into a fun parlor trick when I can estimate a client's bill based upon the story they tell me!

We got pinged by our CEO to reduce our AWS bill which was $8k at the time. After a bit of work, we got it down by a little over a grand. One lunch, he said "guys, what are you doing about that bill?"... "What, we got it down by over a grand!"... "Yeah, but the exchange rate has gone the other way..."

Gotta love the Australian dollar. The Australian economy is solid - about to set a world record for longest continuing period without recession, including the GFC years - but the AUD swings around like a mad animal.

Re: A million-dollar engineering problem

#184

Post by grkvlt »

CobrastanJorji wrote: post #74

> After a three months of focused work, we managed to cut our AWS bill by over one million dollars annually. That's great, but now I'm wondering how many engineers were paid for three months of full time work to save $1 million/year on AWS, and how much it costs the company to pay those engineers for 3 months. It's also interesting to wonder about how much growth your startup could have managed if it had had three ad…

If you assume USD 250k per engineer, they would need to have had more than 16 engineers working on this for the 3 months to spend more than a million on salary. So, I'm guessing they probably came out ahead on that...?

Re: A million-dollar engineering problem

#185

Post by vacri »

mattmanser wrote: post #92
djchen wrote: post #37

Earlier quoted context omitted.

Most of those providers you mentioned aren't as reliable and scalable as AWS, Google Cloud, Azure, etc. That isn't an apples to apples comparison. I would not want to host my business on So You Start, Kimsufi, Hetzner, and especially not Joe's Data Center. I have personally used Joes DC and they have had numerous outages in the past. Hetzner is known for terminating you for any sort of "DOS" like traffic, including h…

Reliability wise, do you actually use these services, as in my experience that's a complete myth. One of my clients on a dedicated server has never gone down. The site is blazingly fast, barely touches 5% CPU and pages have sub 50ms response times. Deploys take 10 seconds or so, I could make it faster but it's not really worth the cost/benefit. My client on Azure, with a significantly lower visitor count, pays 3-10 t…

> One of my clients on a dedicated server has never gone down. The site is blazingly fast, barely touches 5% CPU and pages have sub 50ms response times. Deploys take 10 seconds or so, I could make it faster but it's not really worth the cost/benefit.

One of your clients... where? Without even giving a location, this doesn't even qualify as 'anecdata'.

Re: A million-dollar engineering problem

#187

Post by cobookman »

TheHydroImpulse wrote: post #151
dastbe wrote: post #142

Earlier quoted context omitted.

Would you mind saying more about what you had to build, and what you would liked to see? One thing I'd like to see in the future is scaling actions that supported multipliers, as well as metric math support (scale service to request_count/acceptable_requests_per_service).

One thing we experienced at Segment was the fact that we needed to quickly handle a surge in volume but couldn't overload partners. Essentially we wanted something that scaled up quickly at first but was pretty conservative after that. We settled on constant increases/decreases using queue depth thresholds but ideally ECS would support feeding multiple metrics and doing some basic math to figure out how much we're cu…

Have you looked into kubernetes?

Re: A million-dollar engineering problem

#188

Post by TheHydroImpulse »

cobookman wrote: post #187
TheHydroImpulse wrote: post #151

Earlier quoted context omitted.

One thing we experienced at Segment was the fact that we needed to quickly handle a surge in volume but couldn't overload partners. Essentially we wanted something that scaled up quickly at first but was pretty conservative after that. We settled on constant increases/decreases using queue depth thresholds but ideally ECS would support feeding multiple metrics and doing some basic math to figure out how much we're cu…

Have you looked into kubernetes?

A bunch of people at Segment have looked into it. We manage our infrastructure with Terraform and operationally IMO Kubernetes introduces too much complexity compared with ECS and Terraform.

ECS is pretty dead simple (just run an agent on the host) and while it doesn't offer nearly the same feature set, it's really good operationally.

Some people here are tinkering with it for some non-core services.

Re: A million-dollar engineering problem

#189

Post by QuinnyPig »

brazzledazzle wrote: post #181
QuinnyPig wrote: post #127

Earlier quoted context omitted.

It gets into the specifics of "what makes it horrifying" pretty quickly. It usually boils down to "nobody's responsible for cleaning things up," a misunderstanding of the billing model, and by the time anyone cares to fix it the billing report is multiple gigabytes. Step 1 is always "figure out what you've got, and why it is this way."

So if the best way to avoid the mess is accountability for keeping things clean, how do you do that from the start? Would something as simple as tagging resources to specific teams/departments be a good place to start? I'm helping with setting up a completely new account/environment and I know once the buzzword fever passes and the annual costs are reviewed there will be frantic requests to cut down the bill. Getting…

Propagate tags to secondary resources, like EBS volumes. Separate out production from your dev environment-- preferably with a second AWS account with consolidated billing. Buy in to something like Terraform or CloudFormation for building the non-instance parts of your stack.

As you grow, start adding project tags to things. Automatically tag resources with the IAM user who creates them. Have an "untagged resources report" that goes out on a schedule.

Build with an eye towards "when someone freaks out in a year about the bill, what questions will they have, and how can I best position Future Me to answer them?"

Re: A million-dollar engineering problem

#190

Post by daurnimator »

brianwawok wrote: post #167
empath75 wrote: post #148

Earlier quoted context omitted.

Of course, because how do you scale up when you're building everything yourself? Suddenly you have to hire a whole bunch of hardware guys, storage guys, db guys etc, etc.

Why do you need a DB guy if you host it on your server or AWSs? You have a DB either way. You can have a guy if you want or not... Physical location means nothing.

DB guy to tune the instance the database runs on + figure out your streaming backups etc. It's not a full time job, but it's a couple of days a week; and completely taken care of if you use RDS
Post reply on HN