Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A million-dollar engineering problem

segment.com

Post reply on HN
Print view View original

51–60 of 265 posts

Re: A million-dollar engineering problem

#51

Post by QuinnyPig »

Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

> At this point simply not using AWS is a competitive advantage.

Respectfully, I'm going to disagree. I consult full time on AWS cost optimization / reduction / understanding.

If you blindly run things on AWS without an understanding of the costing model, that'll work for a time. As you scale, you start to realize "oh my god it runs on money."

There are myriad ways around that, but a blanket "never use AWS" isn't going to address it constructively.

Re: A million-dollar engineering problem

#52

Post by QuinnyPig »

teh wrote: post #6

I've been joking with friends that my next job will be AWS efficiency guru. I've somewhat optimized our own use, but I think I could use similar, simple rules to get 20% out of a 500k / month budget. Give me what I save you in 2 months and I'll have a good business :)

I actually do this as a full time thing; I started a consultancy to fix horrifying AWS bills.

Something I've learned is that flat fee pricing makes the most sense-- while tempting, the other models are off-putting.

Hourly is a great way to starve to death, and "percentage of savings" grows difficult to quantify. "Okay, you just recommended the following reserved instance purchases. Is this really the best for us, or does it boost the number you're taking a percentage of?"

It's very easy to end up misaligned with your clients as you go down that path...

Re: A million-dollar engineering problem

#53

Post by pm90 »

GFischer wrote: post #41
user5994461 wrote: post #22

Earlier quoted context omitted.

You shouldn't be on AWS in the first place if everything you do can fit on a single server. Use the right tool for the job.

I disagree. Depends on the service's business model obviously, but there are services where the convenience will more than offset the extra costs. Once the company grows big enough, yes, then it might make sense to go off AWS (or not - see: Netflix).

I've always found it interesting that Netflix has not at least tried to go off AWS; if only because Amazon has a competing service with Amazon video. I'm sure its not an easy problem to solve, but Dropbox has gone off AWS as well; it seems like they would be much better served with their own "Video Cloud" with specialized hardware for streaming/processing videos.

Re: A million-dollar engineering problem

#55

Post by rbranson »

kornish wrote: post #47
rbranson wrote: post #33

Earlier quoted context omitted.

Thanks for the warm feedback! We currently have an internal project underway to detect hot keys in our pipeline and collapse back-to-back writes before they're written out to DynamoDB. It's difficult to apply throttling on these conditions synchronously within the ingestion API (i.e. return a 429 based on too many writes to one key) because of the flexibility of the product: that workload is perfectly acceptable for…

Ah, gotcha. Yeah, it makes sense to avoid synchronously turning away data as that does defeat the point of the product. And the cost for rejecting false negatives is high because the moment when a client is receiving lots of data is when it's most important for them to store it. If you don't mind answering: for your warehouse offering, do you pull data from some services (e.g. Zendesk, SFDC), have them push it to you…

For downstream partners I mean data flows from user -> Segment -> partner. Event data is ingested through our API & SDKs, and this is fed to partner's APIs. For Cloud Sources, generally data is pulled from partners using their public APIs at an interval and pushed to customer warehouses in batches. In a few special cases partners push data to our APIs.

Re: A million-dollar engineering problem

#56

Post by secabeen »

GFischer wrote: post #41
user5994461 wrote: post #22

Earlier quoted context omitted.

You shouldn't be on AWS in the first place if everything you do can fit on a single server. Use the right tool for the job.

I disagree. Depends on the service's business model obviously, but there are services where the convenience will more than offset the extra costs. Once the company grows big enough, yes, then it might make sense to go off AWS (or not - see: Netflix).

Remember that Netflix runs their core product (content distribution) internally on their own CDN. All of the supporting technologies (billing, content discovery, etc.) is on AWS, but the core product is not.

Re: A million-dollar engineering problem

#57

Post by QuinnyPig »

rbranson wrote: post #28
nik736 wrote: post #5

Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.

It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows…

> It could be if our workload was relatively stable

+1. If your entire environment is steady-state, it may be worth considering a migration to bare metal.

That said, you can expect:

* Worse uptime

* A fair bit of retraining / hiring different skillsets

* A loss of engineering focus as they work on the migration instead of feature expansion

It can make sense economically, but that discussion goes well past "the monthly AWS bill."

Re: A million-dollar engineering problem

#58

Post by pokler »

pm90 wrote: post #53
GFischer wrote: post #41

Earlier quoted context omitted.

I disagree. Depends on the service's business model obviously, but there are services where the convenience will more than offset the extra costs. Once the company grows big enough, yes, then it might make sense to go off AWS (or not - see: Netflix).

I've always found it interesting that Netflix has not at least tried to go off AWS; if only because Amazon has a competing service with Amazon video. I'm sure its not an easy problem to solve, but Dropbox has gone off AWS as well; it seems like they would be much better served with their own "Video Cloud" with specialized hardware for streaming/processing videos.

Netflix doesn't use AWS for streaming video.

https://openconnect.netflix.com/en/

Re: A million-dollar engineering problem

#59

Post by qeternity »

QuinnyPig wrote: post #51
Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

> At this point simply not using AWS is a competitive advantage. Respectfully, I'm going to disagree. I consult full time on AWS cost optimization / reduction / understanding. If you blindly run things on AWS without an understanding of the costing model, that'll work for a time. As you scale, you start to realize "oh my god it runs on money." There are myriad ways around that, but a blanket "never use AWS" isn't goi…

I am curious your thoughts on the following. It seems like there is a trough of benefit to AWS. In the mid-size world, especially where computing needs are relatively stable/predictable, you are much better leasing dedicated servers from the usual suspects. Below that size, the linear scaling allows small scrappy startups to really scale/burst with their needs. And at large scale, but below Google/Facebook et al scale, it's cheaper than building your own datacenter yourself. At least this has been my casual observations. And fwiw, the startup I have just launched fits into this trough perfectly. So hopefully I am not just biased. I love the cloud for the spot market. But not for our base load. Spot is the nat gas of the server world, and OVH is the coal.

Re: A million-dollar engineering problem

#60

Post by user5994461 »

secabeen wrote: post #56
GFischer wrote: post #41

Earlier quoted context omitted.

I disagree. Depends on the service's business model obviously, but there are services where the convenience will more than offset the extra costs. Once the company grows big enough, yes, then it might make sense to go off AWS (or not - see: Netflix).

Remember that Netflix runs their core product (content distribution) internally on their own CDN. All of the supporting technologies (billing, content discovery, etc.) is on AWS, but the core product is not.

The core products are the shows, not the tech.
Post reply on HN