Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Dropbox from Nginx to Envoy

dropbox.tech

Post reply on HN
Print view View original

31–40 of 243 posts

Re: Migrating Dropbox from Nginx to Envoy

#31

Post by user5994461 »

dmicher wrote: post #2

I know some people might find it a little controversial, but I’m super excited about our load balancing future and that we probably have the biggest Envoy deployment in the world now. When we moved most of Dropbox traffic to Envoy, we had to seamlessly migrate a system that already handles tens of millions of open connections, millions of requests per second, and terabits of bandwidth. This effectively made us into o…

Well, a single server doesn't really need to do more than 10Gbps or 100k connections. Going above is a "simple" matter of managing horizontal scaling.

What I wonder about is how do you distribute the traffic on the higher level? I imagine there are separate clusters of envoys to serve different configurations/applications/locations? How many datacenters does dropbox have?

I was running a comparable setup in a large company, all based on HAProxy, there was a significant amount of complexity in routing requests to applications that might ultimately be in any of 30 datacenters.

Re: Migrating Dropbox from Nginx to Envoy

#33

Post by ci5er »

dmicher wrote: post #5
Havoc wrote: post #3

Sensing a bit of a trend here. Didn't another major player recently make the same switch?

It may actually become a trend. For well known reasons: - Community - Nginx served us well for almost a decade. But it didn’t adapt to current development best-practices - Operationally Nginx was quite expensive to maintain - C++ - Observability and monitoring etc...

Is C++ generally considered to be "better"?

I've always looked at it (esp. with STL) as kind of a "Swiss-Army-Chainsaw" and you were going to shoot your eye out. Maybe that view is old and things are better - but I learned a while back that sending a young gun into a C++ application's code-base would lead to a world of pain)

Maybe that learning is no longer accurate? What do you think?

Re: Migrating Dropbox from Nginx to Envoy

#34

Post by SaveTheRbtz »

user5994461 wrote: post #29

A shame they picked nginx in the first place, it has all the stats and critical features behind the paid edition. HAProxy is always a better choice for load balancing. Besides that, it looks like the move was significantly driven by GRPC and profobuf. No surprise here, GRPC really doesn't work well over HTTP. Once a company start using the google stack, they have to move to more of the google stack to make it usable.

Our technology stack is very gRPC friendly, so developer experience is actually better with it, than without (though this is very subjective.)

As for the middleboxes, using gRPC-WEB[1] allowed us to switch Desktop Client App to gRPC even behind firewalls/IDSes that do not speak HTTP/2 yet.

As for the HAProxy, Dropbox used to use (circa 2013) it specifically for loadbalancing, but we eventually replaced it with our Golang proxy. That said, recent HAProxy improvements (v2.0+) make it quite an awesome dataplane and an excellent loadbalancer!

[1] https://github.com/grpc/grpc-web

Re: Migrating Dropbox from Nginx to Envoy

#35

Post by ci5er »

thevagrant wrote: post #23
techntoke wrote: post #15

Who does Dropbox compete with these days? They have pretty much the highest prices for the least amount of value. The only reason I see them mentioned here frequently is their connection with Y Combinator.

I noticed on social media a lot of negativity toward Dropbox and sometimes even on HN. The negative sentiment appears to come from tech circles who feel One Drive offers a better price point or iCloud works great for them, so Dropbox shouldn't exist. Personally, I prefer Dropbox. I found problems with One Drive. Google Drive client was always hit and miss and I could not rely on it. iCloud is not cross platform (afai…

Insync solves a lot of Google Drive issues (not all! - the fundamental organization (and ability to search - at Google!) is horrible), Box.com is not bad for auditibility and observability, but One Drive keeps trying to suck you in (You think you are out! But you are not!),

For the extra buck-or-two per user per month - I just like the fact that it "just works" for most people and little tech support. (Although I do miss the RSS feed on events that they removed that helped me keep track of all of the "stuff" "the people" were doing with "all the files". I'm sure there was a reason - but that was actually the only feature that made me think that they and Box.com might be comparable in that area)

Re: Migrating Dropbox from Nginx to Envoy

#36

Post by dmicher »

ci5er wrote: post #33
dmicher wrote: post #5

Earlier quoted context omitted.

It may actually become a trend. For well known reasons: - Community - Nginx served us well for almost a decade. But it didn’t adapt to current development best-practices - Operationally Nginx was quite expensive to maintain - C++ - Observability and monitoring etc...

Is C++ generally considered to be "better"? I've always looked at it (esp. with STL) as kind of a "Swiss-Army-Chainsaw" and you were going to shoot your eye out. Maybe that view is old and things are better - but I learned a while back that sending a young gun into a C++ application's code-base would lead to a world of pain) Maybe that learning is no longer accurate? What do you think?

When we are comparing C, Lua (Nginx) and C++ (Envoy). Yes C++ is better :).

Re: Migrating Dropbox from Nginx to Envoy

#37

Post by ci5er »

techntoke wrote: post #20
greymalik wrote: post #16

Earlier quoted context omitted.

Google and Apple, to start.

They don't compete though. Google offers 17+GB free with email, office suite, unlimited photos, drive and Voice with a free number for unlimited calling and texting. Dropbox has much stricter bandwidth limits as well. For $9.99 you get all that plus 2TB storage with Google One. Dropbox has a minimum for of 3 users for their business plan, but with 1 user on G Suite for $12/mo I get unlimited storage and all the Goodi…

I'm learning to believe that there are various sub-sects of the HN crowd. The people who would rather pay $3/month to host a slow-ass VPS, and those where $10/month delta, as long as their work-flow is optimized is a "win".

In fact, I think that drives a lot of these "what is better" debates in threads here. Some people go "Google Drive is better, because I get 2TB/month for a flat-fee that bundles the other services" (I do that plan too - I just subscribe to everything - to match the client's work flow. Where it really sucks is that I commercially subscribe to 3 (4?) commercial video conferencing systems).

I am not going to choose to save $5 when it stands in the way of me making $100. I find that thinking impoverishing, and time-wasting, and frankly stupid.

Even if I have to pay a designer $1000 to remake my slides after the content is settled for a client trying to pay me $15K, so that they can raise $2.5M, I'll gladly pay it! That doesn't seem to be the mindset here? (Or maybe I'm just coming across the people that shill for Vultr over Digital Ocean (or God-forbid - AWS!) - instead of focused on velocity of earnings. Maybe I just come across the wrong posts)

But, that's me - and I suspect I am not the majority here.

Re: Migrating Dropbox from Nginx to Envoy

#38

Post by SaveTheRbtz »

user5994461 wrote: post #31
dmicher wrote: post #2

I know some people might find it a little controversial, but I’m super excited about our load balancing future and that we probably have the biggest Envoy deployment in the world now. When we moved most of Dropbox traffic to Envoy, we had to seamlessly migrate a system that already handles tens of millions of open connections, millions of requests per second, and terabits of bandwidth. This effectively made us into o…

Well, a single server doesn't really need to do more than 10Gbps or 100k connections. Going above is a "simple" matter of managing horizontal scaling. What I wonder about is how do you distribute the traffic on the higher level? I imagine there are separate clusters of envoys to serve different configurations/applications/locations? How many datacenters does dropbox have? I was running a comparable setup in a large c…

We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is:

* First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients.

* User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!)

* Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS, now Katran[4]) that encapsulate traffic and DSR it to L7 balancers (previously nginx, now mostly Envoy.)

Overall, we have ~25 PoPs and 4 datacenters.

[1] https://dropbox.tech/infrastructure/dropbox-traffic-infrastr... [2] https://dropbox.tech/infrastructure/intelligent-dns-based-lo...

[3] https://www.dropbox.com/peering [4] https://github.com/facebookincubator/katran

Re: Migrating Dropbox from Nginx to Envoy

#39

Post by ci5er »

dmicher wrote: post #36
ci5er wrote: post #33

Earlier quoted context omitted.

Is C++ generally considered to be "better"? I've always looked at it (esp. with STL) as kind of a "Swiss-Army-Chainsaw" and you were going to shoot your eye out. Maybe that view is old and things are better - but I learned a while back that sending a young gun into a C++ application's code-base would lead to a world of pain) Maybe that learning is no longer accurate? What do you think?

When we are comparing C, Lua (Nginx) and C++ (Envoy). Yes C++ is better :).

Honest question: In what way?

Platform wars are over-ish. We have the same compile targets. What they call "Undefined Behavior) is relegated to ... well ... platforms we are not supporting.

C is fast - simple - easy(ish) to learn, and easy to "fuzz" in testing.

I can't speak to LUA - but C++ looks like a mine-field (to me).

Why do you declare that C++ is "better"? (Seriously interested - I don't even know enough these days to have a debate. I just gave up on the C++ hell-hole years (decades?) ago, and maybe should have kept up)

Re: Migrating Dropbox from Nginx to Envoy

#40

Post by e40 »

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features.

It always kills me when very successful companies don't buy software from other companies.

I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anything but OSS. What they make? Non-OSS software.

Just blows my mind.

Post reply on HN