Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Dropbox from Nginx to Envoy

dropbox.tech

Post reply on HN
Print view View original

141–150 of 243 posts

Re: Migrating Dropbox from Nginx to Envoy

#141

Post by rleigh »

chii wrote: post #124
pjmlp wrote: post #100

Earlier quoted context omitted.

All fine and dandy, except that those patches and collaboration don't pay bills.

> patches and collaboration don't pay bills. implying that just because your open source project is being used, that it is entitled to fund the bills of the project maintainers. I think patches and contributions are a form of bill paying.

Patches and contributions take a non-negligible amount of time and resources to review, test and integrate, as well as adding to the ongoing maintenance burden. They might be welcome, but they are absolutely not cost-free and I wouldn't consider them a "form of bill paying", the benefit (if any) is far too indirect and it doesn't directly help the bottom-line in any way.

Re: Migrating Dropbox from Nginx to Envoy

#142

Post by pjmlp »

Nitramp wrote: post #137
Slartie wrote: post #130

Earlier quoted context omitted.

I had to chuckle as well when I read that article and the part about gRPC. Seems like the pendulum is swinging into the other direction again - back to where we've already been ten or twenty years ago. New name of course, but same concepts. One really starts to feel old at such occasions.

It certainly does look like that. I do think though that we've learned a number of central lessons in the process: - treat messaging as a first class concept, not something to hide & abstract away. - do not attempt to implement polymorphism in a messaging protocol. Do not bind your messaging protocol to a programming language's type system (they serve different purposes). - bake fundamental monitoring & maintainabili…

Sure we did understand it, that is why they used an IDL. independent of programming language's type system.

Apparently DCE IDL now comes in proto files.

What newcomers did not bother to understand is why we were using those formats in first place.

Rest assured, maybe in 20 years we will be introducing this cool RPC protocol, based on YAML on something. Thankfully by then I should be retired.

Re: Migrating Dropbox from Nginx to Envoy

#143

Post by pjmlp »

chii wrote: post #124
pjmlp wrote: post #100

Earlier quoted context omitted.

All fine and dandy, except that those patches and collaboration don't pay bills.

> patches and collaboration don't pay bills. implying that just because your open source project is being used, that it is entitled to fund the bills of the project maintainers. I think patches and contributions are a form of bill paying.

When supermarkets and landlords start accepting them, then yes.

Re: Migrating Dropbox from Nginx to Envoy

#144

Post by DarkWiiPlayer »

SaveTheRbtz wrote: post #119
MuffinFlavored wrote: post #95

Can somebody speak to why dynamic upstreams included in a file paired with `sudo service nginx reload` for prod deploys stopped scaling?

It is easy enough for simple cases (and we used it for quite a while, until we moved to using Lua for that.) For more complex scenarios you will have new `server` blocks, certificates, tls tickets, log files / syslog endpoints, so the automation will end up interacting not with just a single dynamic upstream file but with rather large amount of system interfaces. Control-plane ends up being distributed between config…

nginx is not well suited for constantly reconfiguring your infrastructure on very hot servers. This is a problem when you expose such infrastructure configurations to users (think cloudflare), but otherwise you can just mitigate this problem by having a sane deployment strategy.

Re: Migrating Dropbox from Nginx to Envoy

#145

Post by DarkWiiPlayer »

atonse wrote: post #118
mholt wrote: post #89

Earlier quoted context omitted.

It is a great choice for a general purpose proxy. (That's kind of the point.)

But they mentioned that they wanted to use C++ instead of go to get even that extra performance out. I use Caddy a lot and it's perfectly fine for my scale, but at dropbox's scale, maybe go wouldn't be enough for the ingress part?

> C++ instead of go to get even that extra performance out

Might as well use C then (with some hand-written asm sprinkled in where the compiler gets confused and doesn't see an obvious optimization) for that. And I'm not even being sarcastic here (I wish I was though)

Re: Migrating Dropbox from Nginx to Envoy

#146

Post by mehrdada »

pjmlp wrote: post #143
chii wrote: post #124

Earlier quoted context omitted.

> patches and collaboration don't pay bills. implying that just because your open source project is being used, that it is entitled to fund the bills of the project maintainers. I think patches and contributions are a form of bill paying.

When supermarkets and landlords start accepting them, then yes.

Could it be that OSS is simply not a sustainable business model for the long haul and it was simply successful in a period of history when vast money was made quickly by landgrab expansion of technology to consolidate/provide many basic services and the code itself wasn’t the competitive differentiator? I don’t know but that’s a possibility too. I question why one would be concerned in keeping OSS alive, as a business, assuming it cannot survive on its own feet. There’s no inherent reason OSS should somehow forcefully live. It’s already changing its character via AGPL and Mongo license-style things in the face of AWS cloud simply deploying and milking cash.

(The above is assuming the concern that it is funding that’s a problem today; I don’t quite see it that way [for instance, I strongly suspect Nginx to have made more money than DBX so far, so who are we to say who’s been more successful; market cap ain’t everything], but that’s a hypothetical to think about.)

Moreover, supporting a project does not equate supporting its existing maintainers. It could mean taking some partial ownership including the review side and having some developers on your own payroll. Seems like that’s how the big project are done most of the time. The Open Core model we are focusing on is a niche and arguably more akin to fremium products than free software as a thing with communal ownership.

Re: Migrating Dropbox from Nginx to Envoy

#147

Post by aliswe »

chii wrote: post #126
dragonwriter wrote: post #82

Earlier quoted context omitted.

> If you have an enterprise, paid version of your OSS product it has to be impossible for an enterprise to use it for free. Why? Most enterprises, especially ones that aren't tech firms, are going to shell out for enterprise support even if there are no additional features. Crippling the community version doesn't necessarily help enterprise sales, it can reduce overall mindshare reducing enterprise traction or, worse…

> shell out for enterprise support even if there are no additional features. i don't feel this to be true. Also, an enterprise that's large would want some features that are irrelevant to a small shop. For example, single-sign-on integration with various providers.

I do though ... Imagine a manager being dependent on a system he ~~bought~~ installed for free, which he didn't buy support for, and is now malfunctioning. It's his fault. And this is what I've seen in practice as well.

Just think about the commercial success of SUSE Linux?

Re: Migrating Dropbox from Nginx to Envoy

#148

Post by rmoriz »

sanxiyn wrote: post #43
rmoriz wrote: post #22

Earlier quoted context omitted.

I've admired Drew and the early Dropbox team for getting things done and shipped even when compiled python GUIs was edgy as the initial Rails version of Twitter was. But they shipped and validated the market. Now adding all those fancy and cool tech mentioned in the blog post will increase the complexity by a lot but it's not clear what are the real benefits. Does a decreased number of machines running really justify…

> Does a decreased number of machines running really justify the migration and addition of complexity? Of course it does. I am not sure why you think it doesn't.

servers are a commodity and cheap compared to runtime complexity

Re: Migrating Dropbox from Nginx to Envoy

#149

Post by Nitramp »

pjmlp wrote: post #142
Nitramp wrote: post #137

Earlier quoted context omitted.

It certainly does look like that. I do think though that we've learned a number of central lessons in the process: - treat messaging as a first class concept, not something to hide & abstract away. - do not attempt to implement polymorphism in a messaging protocol. Do not bind your messaging protocol to a programming language's type system (they serve different purposes). - bake fundamental monitoring & maintainabili…

Sure we did understand it, that is why they used an IDL. independent of programming language's type system. Apparently DCE IDL now comes in proto files. What newcomers did not bother to understand is why we were using those formats in first place. Rest assured, maybe in 20 years we will be introducing this cool RPC protocol, based on YAML on something. Thankfully by then I should be retired.

I think IDLs are an important point among many others (I attempted to list some above). I think we might be missing substantial improvements if we'd just say "oh so we're back to a static IDL description, all old is new again".

And even within IDLs, we've made major progress. Compare the mess of SOAP's data type system, various attempts at inheritance and polymorphism in SOAP and CORBA, pointers in CORBA etc.

Re: Migrating Dropbox from Nginx to Envoy

#150

Post by spacewander »

One of my friends brought me up this post in the morning. The post is awesome and inspirational (caused a discussion in our chant group), though I can't agree with some trivial points.

> Nginx performance without stats collections is on part with Envoy, but our Lua stats collection slowed Nginx on the high-RPS test by a factor of 3. This was expected given our reliance on lua_shared_dict, which is synchronized across workers with a mutex.

The `a factor of 3` is quite large to me. Maybe you put all your stats in lua_shared_dict? You don't need to synchronize the stats every time. Since the collection regularly happens in per-minute frequency, you can put the stats as Lua table, and synchronize them once per 5/10 seconds.

It look like that the compared Nginx is configured with a system which has been survived for years and not up-to-date. The company I worked with used a single virtual server to hold all traffic and routed them with Lua code dynamically. And the upstream is chosen by Lua code too. There is no need to reload Nginx when a new route/upstream is added. We even implemented 'Access Log Service' like feature so that each user can have her favorite access log (by modifying the Nginx core, of course).

However, I don't think this post is incorrect. What Envoy surpasses Nginx is that it has a more thriving developer community. There are more features added into Envoy than Nginx in the recent years. Not only that, opening discussion of Nginx development is rare.

Nginx is an old, slow giant.

Post reply on HN