Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A million-dollar engineering problem

segment.com

Post reply on HN
Print view View original

41–50 of 265 posts

Re: A million-dollar engineering problem

#41

Post by GFischer »

user5994461 wrote: post #22
Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

You shouldn't be on AWS in the first place if everything you do can fit on a single server. Use the right tool for the job.

I disagree. Depends on the service's business model obviously, but there are services where the convenience will more than offset the extra costs.

Once the company grows big enough, yes, then it might make sense to go off AWS (or not - see: Netflix).

Re: A million-dollar engineering problem

#42

Post by a13n »

a13n wrote: post #40
teh wrote: post #6

I've been joking with friends that my next job will be AWS efficiency guru. I've somewhat optimized our own use, but I think I could use similar, simple rules to get 20% out of a 500k / month budget. Give me what I save you in 2 months and I'll have a good business :)

Or charge 20% of what you save them over the next year. This way you're charging more overall (especially if their costs are growing). Also your revenue will be more recurring rather than a one time thing. And by the time the 12 months is up, maybe they'll need your service again. :P

Also as a SaaS founder running on AWS, I would totally do this once our AWS bill is in the 4-5 figures.

Re: A million-dollar engineering problem

#43

Post by yegle »

Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

So, how about backup, availability and a potential additional employee to make sure the server runs fine?

If they can indeed fit their footprint in one single server, then it should be fisible to just host the server at someone's home with enough upload speed, no?

Re: A million-dollar engineering problem

#44

Post by rbranson »

Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

Totally. If the growth rate is reasonable (<50% y/y) and haven't really broken beyond what can be done on a single machine, using AWS or GCE or Azure is a ludicrous decision, short of a "serverless" style architecture. It's pretty easy to get sucked into the "best practices" and overengineer something that costs a lot of money.

Re: A million-dollar engineering problem

#45

Post by voidlogic »

Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

I used to do data-center and virtualization consulting- we always designed our enterprise client's systems to handle the baseload in-house and burst to public cloud. Not rock science. We even automatically live-migrated load from and shutdown onsite hypervisors at night when the system had surplus capacity...

I think the real issue is that people assume all workloads are public cloud workloads. The bigger and less dynamic your workload the less that is true.

Re: A million-dollar engineering problem

#46

Post by gravypod »

djchen wrote: post #37
gravypod wrote: post #34

I've always been set aback by how much AWS servers cost. Maybe I'm just too cheap but you can go to very reputable hosting companies and get things at fractions of the price. For example, if I need 16GB of memory and 4 cores here are my options: * AWS (t2.xlarge) $137.62 * OVH (EG-16) $79 * So You Start/OVH (SYS-IP-1S) $42 * Kimsufi/OVH (KS-3B) $29 * Hetzner (EX41) €46.41 [Lowest cost model is 32GB of ram] * Joe's Da…

Most of those providers you mentioned aren't as reliable and scalable as AWS, Google Cloud, Azure, etc. That isn't an apples to apples comparison. I would not want to host my business on So You Start, Kimsufi, Hetzner, and especially not Joe's Data Center. I have personally used Joes DC and they have had numerous outages in the past. Hetzner is known for terminating you for any sort of "DOS" like traffic, including h…

I think that:

1) Most problems can, but shouldn't necessarily be, solved with adding more servers.

2) SYS & Kimsufi are just OVH resellers. Same datacenters just the last generation of hardware under a different brand.

3) Hetzner is great for if you want to serve up content from the EU. Great ping times all across there. (Your friends over the pond will thank you)

4) OVH is a huge, and very reliable, hosting company. I don't think there's anything wrong with them. They also do supply such an auto-scaling API.

5) You don't necesarily need to run mission-critical systems on a traditional hosting solution to see big savings. Move your high-compute/high-bandwith programs onto a dedicated server and save big. This is really great for batch analytics type systems where after a day of operation, you dump a backup file and want to pull some data out of it for the morning. It's great for dev systems and thousands of other non-mission-critical systems.

Re: A million-dollar engineering problem

#47

Post by kornish »

rbranson wrote: post #33
kornish wrote: post #26

The Dynamo incident highlights an important lesson when using consistently hashed distributed data stores: make sure the actual distribution of hash keys mirrors the expected distribution. (though to their credit, someone writing an automated test using a hard-coded key was beyond their control). Incidents like this are generally why rate limits exist, which they don't currently have [0], but perhaps they'll consider…

Thanks for the warm feedback! We currently have an internal project underway to detect hot keys in our pipeline and collapse back-to-back writes before they're written out to DynamoDB. It's difficult to apply throttling on these conditions synchronously within the ingestion API (i.e. return a 429 based on too many writes to one key) because of the flexibility of the product: that workload is perfectly acceptable for…

Ah, gotcha. Yeah, it makes sense to avoid synchronously turning away data as that does defeat the point of the product. And the cost for rejecting false negatives is high because the moment when a client is receiving lots of data is when it's most important for them to store it.

If you don't mind answering: for your warehouse offering, do you pull data from some services (e.g. Zendesk, SFDC), have them push it to you (which is what I interpreted your "downstream partners" comment to mean – though perhaps those are "upstream partners"), or a mix of both?

Re: A million-dollar engineering problem

#48

Post by TheHydroImpulse »

WithHighProb wrote: post #36

For us the ECS autoscaling feature is way too naive. We need to do our own controller on some more responsive metrics. The customization support is pretty bad.

CPU + memory autoscaling has worked pretty well for most services at Segment. We did have to build a custom queue-based autoscaler that feeds into ECS for our integration stack but the limitations of the ECS autoscaling rules makes it a bit clunky.

Re: A million-dollar engineering problem

#49

Post by amelius »

Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

I guess you could go even lower by not renting, but buying a server instead.

Re: A million-dollar engineering problem

#50

Post by [deleted] »

user5994461 wrote: post #21
throwawaysed wrote: post #20

Earlier quoted context omitted.

A lot of people are in denial that there isn't using some magic efficiency to cloud services that other datacenters don't have. Primary cost savings on cloud VM's is from overprovisioning. The more abstracted away the service is from the hardware, the more they can overprovision without customers noticing. The 50%+ profit margins have to be coming from somewhere. AWS is not made of magic, it's made from largely the s…

> The 50%+ profit margins Charge more than what it costs you. That's how to make money.

[deleted]
Post reply on HN