Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Dropbox from Nginx to Envoy

dropbox.tech

Post reply on HN
Print view View original

51–60 of 243 posts

Re: Migrating Dropbox from Nginx to Envoy

#51

Post by alena_bsu »

rmoriz wrote: post #22
techntoke wrote: post #15

Who does Dropbox compete with these days? They have pretty much the highest prices for the least amount of value. The only reason I see them mentioned here frequently is their connection with Y Combinator.

I've admired Drew and the early Dropbox team for getting things done and shipped even when compiled python GUIs was edgy as the initial Rails version of Twitter was. But they shipped and validated the market. Now adding all those fancy and cool tech mentioned in the blog post will increase the complexity by a lot but it's not clear what are the real benefits. Does a decreased number of machines running really justify…

With that move it is actually decreasing the complexity and improving manageability and sustainability.

Re: Migrating Dropbox from Nginx to Envoy

#52

Post by ci5er »

dmicher wrote: post #46
ci5er wrote: post #39

Earlier quoted context omitted.

Honest question: In what way? Platform wars are over-ish. We have the same compile targets. What they call "Undefined Behavior) is relegated to ... well ... platforms we are not supporting. C is fast - simple - easy(ish) to learn, and easy to "fuzz" in testing. I can't speak to LUA - but C++ looks like a mine-field (to me). Why do you declare that C++ is "better"? (Seriously interested - I don't even know enough thes…

I'm not declaring that C++ is better for everything. In our case it is better because it makes this part of the infrastructure more sustainable: there are more engineers who can code well in C++ vs C in our company and industry overall. Also it is easier to code in C++ as it is general purpose programming language with a lot of libraries available, open source projects, community around it etc.

Maybe we program in different verticals. I have not found it to be so. (I have still upvoted your comment).

Re: Migrating Dropbox from Nginx to Envoy

#53

Post by ci5er »

user5994461 wrote: post #49
ci5er wrote: post #39

Earlier quoted context omitted.

Honest question: In what way? Platform wars are over-ish. We have the same compile targets. What they call "Undefined Behavior) is relegated to ... well ... platforms we are not supporting. C is fast - simple - easy(ish) to learn, and easy to "fuzz" in testing. I can't speak to LUA - but C++ looks like a mine-field (to me). Why do you declare that C++ is "better"? (Seriously interested - I don't even know enough thes…

I think what they're trying to say is that developing plugins for nginx in Lua is not great. It's not a rant about languages. This has to do with nginx not having the required features (features blocked behind the paid edition or GRPC non existent), forcing to develop plugins in Lua to compensate (the only supported language) and Lua is too slow for this sort of stuff (processing Gbps of traffic in real time is no jo…

That makes sense. Thanks. (I still prefer C - but I am admittedly getting old and C++ sucked in the mid-90s or so)

EDIT: BTW -- I am not going to argue with LUA throughput. I'm still not sure what the thinking was there (maybe time-to-prototype?) - but C plugins run faster than Apache's do. By, like, a lot. (And I like Apache! ...Having used it since 1996)

Re: Migrating Dropbox from Nginx to Envoy

#54

Post by rebelnz »

e40 wrote: post #40

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anythin…

Exactly my thoughts when I read the article - a hugely successful company not contributing to an open source project which enabled them to succeed in the first place ...

Re: Migrating Dropbox from Nginx to Envoy

#55

Post by euroelessar »

weitzj wrote: post #32

I did not quite get how they configure envoy? Did they write their own control plane? Use ambassador/Istio/Gloo?

We have built our own control plane in golang tightly integrated with an existing infrastructure (service discovery, secrets/certificates management, configs delivery, feature gating, and so on).

Re: Migrating Dropbox from Nginx to Envoy

#56

Post by euroelessar »

ci5er wrote: post #39
dmicher wrote: post #36

Earlier quoted context omitted.

When we are comparing C, Lua (Nginx) and C++ (Envoy). Yes C++ is better :).

Honest question: In what way? Platform wars are over-ish. We have the same compile targets. What they call "Undefined Behavior) is relegated to ... well ... platforms we are not supporting. C is fast - simple - easy(ish) to learn, and easy to "fuzz" in testing. I can't speak to LUA - but C++ looks like a mine-field (to me). Why do you declare that C++ is "better"? (Seriously interested - I don't even know enough thes…

There are many reasons which lead to a cleaner codebase, some of them are: RAII, smart pointers, constant types, reusable containers, standard algorithms library, cleaner way to define interfaces, etc.

Overall our experience is that C++ code is smaller, simpler to write/read, and has a smaller chance of mistakes than equivalent logic written in C.

Of course many of this points are relevant only for relatively modern C++ (c++11/c++14 or newer), before that the cost/benefit ratio was much less clear.

Re: Migrating Dropbox from Nginx to Envoy

#57

Post by mehrdada »

e40 wrote: post #40

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anythin…

Why does it blow your mind? Various obvious and sane reasons for this, including cost. I bet many of those companies you have in mind “buy” Windows and macOS, and if they are sufficiently big, most certainly buy Oracle or SAP for their corporate operations, finances, and accounting. It’s usually only the production side of sufficiently large internet-scale companies that is biased towards building. Most of the time you can easily explain it with “it’s cheaper than the contract with supplier”. Often, it is strategic ownership over your fate and not being locked into the vendor that comes into play as well. The vendor positioning in the market and its leverage may also change over time and pose a risk down the road (getting acquired, changing focus, abandoning the product, going out of business, etc.) Many times it is just that their problem is so unique to their scale that the generic solution does not technically work for them or the pricing model is not designed to be a fit.

In the particular case of nginx, I can tell you their reputation is not great in adapting to the users’ needs.

Re: Migrating Dropbox from Nginx to Envoy

#58

Post by dilyevsky »

SaveTheRbtz wrote: post #38
user5994461 wrote: post #31

Earlier quoted context omitted.

Well, a single server doesn't really need to do more than 10Gbps or 100k connections. Going above is a "simple" matter of managing horizontal scaling. What I wonder about is how do you distribute the traffic on the higher level? I imagine there are separate clusters of envoys to serve different configurations/applications/locations? How many datacenters does dropbox have? I was running a comparable setup in a large c…

We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is: * First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients. * User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!) * Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS,…

Katran - nice! Any issues with it at all? Do you use it with xdp capable hardware or just normal driver offload?

Re: Migrating Dropbox from Nginx to Envoy

#59

Post by SaveTheRbtz »

alexeldeib wrote: post #48
SaveTheRbtz wrote: post #38

Earlier quoted context omitted.

We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is: * First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients. * User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!) * Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS,…

Cool to see someone using Katran in production. Really interesting stack you have there.

Actually, all the props for that go to Katran's author himeself. When we hired Nikita V. Shirokov (tehnerd), the first thing he did was replacing IPVS with XDP/eBPF-based Katran, which improved our Edge servers throughput by 10x, from ~2MPps to ~20Mpps.

He also contributed a lot to Envoy migration migrating our desktop client to it and adding perf-related thing like TLS session tickets' lifetime to SDS.

Re: Migrating Dropbox from Nginx to Envoy

#60

Post by dmicher »

ci5er wrote: post #52
dmicher wrote: post #46

Earlier quoted context omitted.

I'm not declaring that C++ is better for everything. In our case it is better because it makes this part of the infrastructure more sustainable: there are more engineers who can code well in C++ vs C in our company and industry overall. Also it is easier to code in C++ as it is general purpose programming language with a lot of libraries available, open source projects, community around it etc.

Maybe we program in different verticals. I have not found it to be so. (I have still upvoted your comment).

I value your opinion.
Post reply on HN