Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A million-dollar engineering problem

segment.com

Post reply on HN
Print view View original

81–90 of 265 posts

Re: A million-dollar engineering problem

#81

Post by ropiku »

burntwater wrote: post #68

Had an interesting experience when I went to the pricing page for Segment - Had a little sales guy pop up in the lower right corner asking "Would (my enterprise employer name) like to know if you qualify for a volume discount?" I only recently started working (and web browsing) from an enterprise network, so maybe this has been done before, but it's my first time seeing it and I thought it was really interesting. Edi…

I don't get it right now but it's possible that is Intercom (chat) with data from Clearbit (https://clearbit.com/reveal).

Re: A million-dollar engineering problem

#82

Post by user5994461 »

klodolph wrote: post #77
voidlogic wrote: post #45

Earlier quoted context omitted.

I used to do data-center and virtualization consulting- we always designed our enterprise client's systems to handle the baseload in-house and burst to public cloud. Not rock science. We even automatically live-migrated load from and shutdown onsite hypervisors at night when the system had surplus capacity... I think the real issue is that people assume all workloads are public cloud workloads. The bigger and less dy…

This is a great approach. A company I worked at was running mostly on dedicated hardware, and due to the pain of moving jobs around and getting new hardware, we focused hard on optimizing our code to run within our existing capacity whenever we ran into problems. After a couple years of customer growth, I'm sure we would have been running on at least 10x as many machines if we had never done that. We then set up clou…

> I'm sure we would have been running on at least 10x as many machines

Seeing the opposite trend.

Because your own machines are so expensive and annoying to manage, you only get 500GB memory 40 cores servers. You don't optimize and you run stuff randomly on whatever seem available.

Whereas in the cloud, you put VMs per role, with appropriate sizing. And when someone asks for 5 machines with 16 cores, you can be like, WTF are you running with all that power???

Re: A million-dollar engineering problem

#83

Post by davidlumley »

ropiku wrote: post #81
burntwater wrote: post #68

Had an interesting experience when I went to the pricing page for Segment - Had a little sales guy pop up in the lower right corner asking "Would (my enterprise employer name) like to know if you qualify for a volume discount?" I only recently started working (and web browsing) from an enterprise network, so maybe this has been done before, but it's my first time seeing it and I thought it was really interesting. Edi…

I don't get it right now but it's possible that is Intercom (chat) with data from Clearbit ( https://clearbit.com/reveal ).

Close! It's Drift (drift.com), not Intercom.

Re: A million-dollar engineering problem

#84

Post by user5994461 »

kornish wrote: post #26

The Dynamo incident highlights an important lesson when using consistently hashed distributed data stores: make sure the actual distribution of hash keys mirrors the expected distribution. (though to their credit, someone writing an automated test using a hard-coded key was beyond their control). Incidents like this are generally why rate limits exist, which they don't currently have [0], but perhaps they'll consider…

> the per-user ID write bottleneck

The basics is to have less machines but more powerful. It helps to handle the targeted bursts.

The advanced is to have a layer of queuing before the ingestion, where you can do magic with distribution rules, rate limiting and dropping peak traffic.

Re: A million-dollar engineering problem

#85

Post by rbranson »

lafay wrote: post #72
rbranson wrote: post #28

Earlier quoted context omitted.

It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows…

This article makes a good case for having at least one early team member who knows how to build out physical infrastructure at scale, even if you initially decide on AWS / GCE / Azure: http://firstround.com/review/the-three-infrastructure-mistak...

This might sound a bit extreme, but contrary to most FRR pieces this is garbage advice. Startups should focus first on getting their product-market fit before trying to future-proof themselves from a future that doesn't yet exist.

Re: A million-dollar engineering problem

#86

Post by zzzcpan »

bluedino wrote: post #79
gravypod wrote: post #34

I've always been set aback by how much AWS servers cost. Maybe I'm just too cheap but you can go to very reputable hosting companies and get things at fractions of the price. For example, if I need 16GB of memory and 4 cores here are my options: * AWS (t2.xlarge) $137.62 * OVH (EG-16) $79 * So You Start/OVH (SYS-IP-1S) $42 * Kimsufi/OVH (KS-3B) $29 * Hetzner (EX41) €46.41 [Lowest cost model is 32GB of ram] * Joe's Da…

AWS is so much more than servers. If you just want a server to run some stuff on, you're right, it's faster and cheaper to buy a dedicated server from a reseller. With Amazon you have so many options when it comes to storage, compute power, load balancing... They have a huge portfolio of products and you can get everything in one location from one vendor.

"With Amazon you have so many options when it comes to storage, compute power, load balancing..."

It used to be different, but today their most important offerings have open source equivalents. They are essentially off the shelf, because for your own projects there is no need to care about abuse or noisy neighbors. Most importantly, though, it's going to be cheaper and you still will be able to use cloud where it shines.

Re: A million-dollar engineering problem

#87

Post by walrus01 »

Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

A million dollars buys a shitload of fast 1U dual socket xeon servers each with 512GB of RAM.

Re: A million-dollar engineering problem

#89

Post by omarchowdhury »

user5994461 wrote: post #60
secabeen wrote: post #56

Earlier quoted context omitted.

Remember that Netflix runs their core product (content distribution) internally on their own CDN. All of the supporting technologies (billing, content discovery, etc.) is on AWS, but the core product is not.

The core products are the shows, not the tech.

Without the tech, no one sees the core products.

Re: A million-dollar engineering problem

#90

Post by [deleted] »

aresant wrote: post #17
teh wrote: post #6

I've been joking with friends that my next job will be AWS efficiency guru. I've somewhat optimized our own use, but I think I could use similar, simple rules to get 20% out of a 500k / month budget. Give me what I save you in 2 months and I'll have a good business :)

Go do it! I used that exact same model in Conversion Rate Optimization - get your conversion rate up, give me 30% of what we improve. And built that into a 20+ person digital agency billing millions of dollars a year before being bought out. Exactly how I did that and you can to: (1) Wrote topical, detail rich posts similar to the parent here about problems I was solving in CRO for a handful of customers, never discl…

[deleted]
Post reply on HN