Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Dropbox from Nginx to Envoy

dropbox.tech

Post reply on HN
Print view View original

231–240 of 243 posts

Re: Migrating Dropbox from Nginx to Envoy

#231

Post by ncmncm »

Matthias247 wrote: post #94
ram_rar wrote: post #68

I feel so old now. There was a time, when I used to discuss with senior engineers @ Yahoo! to use NginX over Apache. Nginx was the hot thing, popularizing C10k [1]. Now in my current team, I have junior devs in my team pushing for Envoy over HAProxy/Nginx setup. Is this trend happening primarily because devs are pushing for GRPC over REST? What benefits does Envoy offer over Nginx, if you're still a REST based servic…

The sibling comments point towards the difference in configuration if you take the "out of the box" product. But there is also a vast difference in how code is organized, in case you ever have to touch it. From my point of view Nginx feels "old". It's a C codebase without a great amount of abstractions and interfaces, and instead having a bunch of #ifdefs here and there. Unit-tests and comments are not to be found. B…

I looked around at the code in Envoy.

"As modern as it gets"? Very, very far from it. Everywhere I looked it was all-over public virtual functions. It looked, more than anything, like Java, which is essentially, more or less, C++92 with some bells on.

The code might be OK, but, as with typical Java code, everywhere I looked was boilerplate, hardly any of it doing any actual work. I would hate for somebody to look at Envoy and think that was what good, modern C++ code looks like.

Virtual functions are a good answer to certain problems that come up, once in a while--in C, for such a problem, you would use function pointers. Inheritance is a pretty good answer to certain problems that come up a little more often.

But neither is a good answer to any organizational need, and a big project that reaches for virtual functions and inheritance as first resort makes me shiver.

Re: Migrating Dropbox from Nginx to Envoy

#232

Post by rolls-reus »

spacewander wrote: post #150

One of my friends brought me up this post in the morning. The post is awesome and inspirational (caused a discussion in our chant group), though I can't agree with some trivial points. > Nginx performance without stats collections is on part with Envoy, but our Lua stats collection slowed Nginx on the high-RPS test by a factor of 3. This was expected given our reliance on lua_shared_dict, which is synchronized across…

> The `a factor of 3` is quite large to me. Maybe you put all your stats in lua_shared_dict? You don't need to synchronize the stats every time. Since the collection regularly happens in per-minute frequency, you can put the stats as Lua table, and synchronize them once per 5/10 seconds.

Any pointers on how to achieve this for someone just starting out with lua and openresty? I have the exact same thing (lua_shared_dict) for stats collection, would love to learn a better way.

Re: Migrating Dropbox from Nginx to Envoy

#233

Post by MrBuddyCasino »

SaveTheRbtz wrote: post #219
MrBuddyCasino wrote: post #157

Earlier quoted context omitted.

> uses Bazel for builds Is this unanimously good? I've heard both praise and horror, never used it myself.

One of the senior engineers once said to me that "Bazel is like a sewer: you get back what you put in." Bazel requires a lot of upfront effort but the power of (a programmatically accessible/modifiable) dependency graph and a common build/test system across all the languages is very hard to underestimate.

> very hard to underestimate

Are you sure?

Re: Migrating Dropbox from Nginx to Envoy

#234

Post by nickdothutton »

e40 wrote: post #40

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anythin…

Business is motivated to avoid anything which is a tax. Said another way, they are motivated to avoid or escape from anything that grows in line with earnings. If their infrastructure grows, their bill from nginx will grow, modulo the skills and efficiency of their infrastructure teams and the speed of whatever servers they are buying.

Re: Migrating Dropbox from Nginx to Envoy

#235

Post by adolph »

footlose_3815 wrote: post #102
e40 wrote: post #40

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anythin…

Reminds me of private companies who profit from public resources. Like selling tap water in bottles.

Is tap water not sold for commercial use at market rates? The public resource steward is leaving money on the table if they aren't.

Re: Migrating Dropbox from Nginx to Envoy

#236

Post by SaveTheRbtz »

spacewander wrote: post #150

One of my friends brought me up this post in the morning. The post is awesome and inspirational (caused a discussion in our chant group), though I can't agree with some trivial points. > Nginx performance without stats collections is on part with Envoy, but our Lua stats collection slowed Nginx on the high-RPS test by a factor of 3. This was expected given our reliance on lua_shared_dict, which is synchronized across…

We've made a note about how inefficient our solution was and what was the plan to fix it. Sadly, to get proper stats in nginx we needed two things:

* C interface for stats, so we can would have access to from C code.

* Instrument all `ngx_log_error` calls so we would have access not only to per-request stats but also various internal error conditions (w/o parsing logs.)

That said, we could indeed just improve our current stat collection in the short term (e.g. like you suggested with a per-worker collection and periodic lua_shared_dict sync.) But that would not solve the longterm problem of lacking internal stats. We could even go further and pour all the resources that were used for Envoy migration into nginx customizations but that would be a road with no clear destination because we would unlikely to succeed in upstreaming any of that work.

Re: Migrating Dropbox from Nginx to Envoy

#237

Post by JoshTriplett »

pm90 wrote: post #209
boris wrote: post #167

Earlier quoted context omitted.

What if it were a single source-available version (that also allowed you to, say, get third-party support/customizations)?

If the licensing model asks for companies to pay for the "Enterprise" features which are still OSS, it would resolve this problem for me at least (not sure about the OP). In general, its been my experience that the closed source enterprise only crap that most companies push for is exactly that: crap. I suspect its because those features are treated as a business expense, and thus built to keep costs low. Almost every…

> GCP's support for urgent issues and product feature requests has been somewhat reliable and predictable.

Can you give an example of a product feature request that succeeded via that support channel?

Re: Migrating Dropbox from Nginx to Envoy

#238

Post by pm90 »

JoshTriplett wrote: post #237
pm90 wrote: post #209

Earlier quoted context omitted.

If the licensing model asks for companies to pay for the "Enterprise" features which are still OSS, it would resolve this problem for me at least (not sure about the OP). In general, its been my experience that the closed source enterprise only crap that most companies push for is exactly that: crap. I suspect its because those features are treated as a business expense, and thus built to keep costs low. Almost every…

> GCP's support for urgent issues and product feature requests has been somewhat reliable and predictable. Can you give an example of a product feature request that succeeded via that support channel?

- we had requested some way of being able to remove IAM permissions that went stale. GCP introduced a “recommendations” feature that indicates which IAM roles have been unused for a long time and can safely be removed.

- we requested a way to prohibit the provision of ILBs on shared VPC subnets without explicit grants; GCP introduced a role for that

- we had issues with the number of compute instances being too high on our VPC and reaching limits because of the number of vpc peerings. In the past, every GKE cluster created a new peering. We have a bunch of GKE clusters, and as we added more the Max number of instances we could provision was reduced significantly. GCP introduced and fast tracked a feature that enabled all GKE clusters to use a single peering rather than create a peer per GKE cluster.

There’s a bunch more. But on this front I have been a happy customer.

Re: Migrating Dropbox from Nginx to Envoy

#239

Post by JoshTriplett »

pm90 wrote: post #238
JoshTriplett wrote: post #237

Earlier quoted context omitted.

> GCP's support for urgent issues and product feature requests has been somewhat reliable and predictable. Can you give an example of a product feature request that succeeded via that support channel?

- we had requested some way of being able to remove IAM permissions that went stale. GCP introduced a “recommendations” feature that indicates which IAM roles have been unused for a long time and can safely be removed. - we requested a way to prohibit the provision of ILBs on shared VPC subnets without explicit grants; GCP introduced a role for that - we had issues with the number of compute instances being too high…

Thanks for the details!

Re: Migrating Dropbox from Nginx to Envoy

#240

Post by afiori »

JoshTriplett wrote: post #217
boris wrote: post #167

Earlier quoted context omitted.

What if it were a single source-available version (that also allowed you to, say, get third-party support/customizations)?

If you're saying "source-available" as opposed to Open Source: complete non-starter. I'm looking for Open Source, not a faux knock-off of it; don't try to give me a subset of Open Source license terms that you think would placate me while not actually being open.

the free software people say the same thing about open source
Post reply on HN