Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Dropbox from Nginx to Envoy

dropbox.tech

Post reply on HN
Print view View original

41–50 of 243 posts

Re: Migrating Dropbox from Nginx to Envoy

#41

Post by SaveTheRbtz »

e40 wrote: post #40

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anythin…

The subject of monetizing opensource software is a tricky one. Some companies pursue the Open-Core principle, others monetize through the consulting services or cloud infrastructures.

As for investing into opensource, Dropbox is trying to do that when possible, for example we (along with Automattic) did sponsor HTTP/2 development in Nginx.

Re: Migrating Dropbox from Nginx to Envoy

#42

Post by Angostura »

e40 wrote: post #40

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anythin…

In the case of buying from a small company this can make sense. If they fold it is good to know that the software will still be around.

Re: Migrating Dropbox from Nginx to Envoy

#43

Post by sanxiyn »

rmoriz wrote: post #22
techntoke wrote: post #15

Who does Dropbox compete with these days? They have pretty much the highest prices for the least amount of value. The only reason I see them mentioned here frequently is their connection with Y Combinator.

I've admired Drew and the early Dropbox team for getting things done and shipped even when compiled python GUIs was edgy as the initial Rails version of Twitter was. But they shipped and validated the market. Now adding all those fancy and cool tech mentioned in the blog post will increase the complexity by a lot but it's not clear what are the real benefits. Does a decreased number of machines running really justify…

> Does a decreased number of machines running really justify the migration and addition of complexity?

Of course it does. I am not sure why you think it doesn't.

Re: Migrating Dropbox from Nginx to Envoy

#44

Post by user5994461 »

SaveTheRbtz wrote: post #38
user5994461 wrote: post #31

Earlier quoted context omitted.

Well, a single server doesn't really need to do more than 10Gbps or 100k connections. Going above is a "simple" matter of managing horizontal scaling. What I wonder about is how do you distribute the traffic on the higher level? I imagine there are separate clusters of envoys to serve different configurations/applications/locations? How many datacenters does dropbox have? I was running a comparable setup in a large c…

We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is: * First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients. * User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!) * Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS,…

Great. Exactly what I was looking for =)

Re: Migrating Dropbox from Nginx to Envoy

#46

Post by dmicher »

ci5er wrote: post #39
dmicher wrote: post #36

Earlier quoted context omitted.

When we are comparing C, Lua (Nginx) and C++ (Envoy). Yes C++ is better :).

Honest question: In what way? Platform wars are over-ish. We have the same compile targets. What they call "Undefined Behavior) is relegated to ... well ... platforms we are not supporting. C is fast - simple - easy(ish) to learn, and easy to "fuzz" in testing. I can't speak to LUA - but C++ looks like a mine-field (to me). Why do you declare that C++ is "better"? (Seriously interested - I don't even know enough thes…

I'm not declaring that C++ is better for everything. In our case it is better because it makes this part of the infrastructure more sustainable: there are more engineers who can code well in C++ vs C in our company and industry overall. Also it is easier to code in C++ as it is general purpose programming language with a lot of libraries available, open source projects, community around it etc.

Re: Migrating Dropbox from Nginx to Envoy

#47

Post by user5994461 »

stock_toaster wrote: post #17
SaveTheRbtz wrote: post #4

Earlier quoted context omitted.

I think the best slice of who's migrating to Envoy can be observed via EnvoyCon talks[1][2]: * Lyft (of course) * Spotify * Stripe * Square * eBay * Yelp * Pinterest Plus the support from major cloud providers: Google, Microsoft, and Amazon. [1] https://envoyconna18.sched.com/ [2] https://envoycon2019.sched.com/

So, seems like nginx is fine until your company reaches the "we are worth billions now" scale?

nginx is never fine for load balancing, they put basic features like metrics behind the paid edition. It's not sane to operate in production.

https://thehftguy.com/2016/10/03/haproxy-vs-nginx-why-you-sh...

Re: Migrating Dropbox from Nginx to Envoy

#48

Post by alexeldeib »

SaveTheRbtz wrote: post #38
user5994461 wrote: post #31

Earlier quoted context omitted.

Well, a single server doesn't really need to do more than 10Gbps or 100k connections. Going above is a "simple" matter of managing horizontal scaling. What I wonder about is how do you distribute the traffic on the higher level? I imagine there are separate clusters of envoys to serve different configurations/applications/locations? How many datacenters does dropbox have? I was running a comparable setup in a large c…

We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is: * First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients. * User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!) * Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS,…

Cool to see someone using Katran in production. Really interesting stack you have there.

Re: Migrating Dropbox from Nginx to Envoy

#49

Post by user5994461 »

ci5er wrote: post #39
dmicher wrote: post #36

Earlier quoted context omitted.

When we are comparing C, Lua (Nginx) and C++ (Envoy). Yes C++ is better :).

Honest question: In what way? Platform wars are over-ish. We have the same compile targets. What they call "Undefined Behavior) is relegated to ... well ... platforms we are not supporting. C is fast - simple - easy(ish) to learn, and easy to "fuzz" in testing. I can't speak to LUA - but C++ looks like a mine-field (to me). Why do you declare that C++ is "better"? (Seriously interested - I don't even know enough thes…

I think what they're trying to say is that developing plugins for nginx in Lua is not great. It's not a rant about languages.

This has to do with nginx not having the required features (features blocked behind the paid edition or GRPC non existent), forcing to develop plugins in Lua to compensate (the only supported language) and Lua is too slow for this sort of stuff (processing Gbps of traffic in real time is no joke).

Re: Migrating Dropbox from Nginx to Envoy

#50

Post by user5994461 »

SaveTheRbtz wrote: post #4
Havoc wrote: post #3

Sensing a bit of a trend here. Didn't another major player recently make the same switch?

I think the best slice of who's migrating to Envoy can be observed via EnvoyCon talks[1][2]: * Lyft (of course) * Spotify * Stripe * Square * eBay * Yelp * Pinterest Plus the support from major cloud providers: Google, Microsoft, and Amazon. [1] https://envoyconna18.sched.com/ [2] https://envoycon2019.sched.com/

They must all be GRPC users. Developers are pushing GRPC and protobuf pretty hard in companies. The next step down the road is to move to envoy as the load balancer. Otherwise these protocols don't work well over traditional HTTP infrastructure.
Post reply on HN