Migrating Dropbox from Nginx to Envoy • Hacker News Board

Re: Migrating Dropbox from Nginx to Envoy

#111

Post by e12e » Fri, Jul 31, 2020, 5:18 AM UTC

eric4smith wrote: post #104

It's interesting almost no web server provides an easy way to deal with multi-tenant multi-domain architectures in a good way that includes automatic SSL. Caddy is the closest, but still not near enough. There is this small segment of the market that we operate in that requires thousands of TLS connected domains to be hosted behind a dynamic backend. It's services like Tumblr, Wordpress.com, or any other hosting serv…

Traefik makes it fairly easy (inasmuch as it makes anything easy). But it's just a proxy, not a web server.

Re: Migrating Dropbox from Nginx to Envoy

#112

Post by PopeDotNinja » Fri, Jul 31, 2020, 5:19 AM UTC

gitgud wrote: post #86

SaveTheRbtz wrote: post #41

Earlier quoted context omitted.

The subject of monetizing opensource software is a tricky one. Some companies pursue the Open-Core principle, others monetize through the consulting services or cloud infrastructures. As for investing into opensource, Dropbox is trying to do that when possible, for example we (along with Automattic) did sponsor HTTP/2 development in Nginx.

Personally I think that monetisation of open-source goes against the consumer of the OSS in practically all cases. - Open-Core::: Features are not added to core, as they want people to upgrade. - Consulting::: Ease of use is ignored, as if it's too easy people won't need consultants. - Sponsoring Goals::: Software is almost held at ransom, until goals are reached. The best way to help open-source software is to donat…

Do you have an example of sponsorship goals actively gating software development? I haven’t seen this one. “I’m not patching this zero day until I get to $1,000,000!”

Re: Migrating Dropbox from Nginx to Envoy

#113

Post by SaveTheRbtz » Fri, Jul 31, 2020, 5:22 AM UTC

pmlnr wrote: post #92

If anyone was wondering, this is solely for proxying, not for oldschool web server functionalities, eg. static file serving.

We've actually started experimenting with converting our static file serving to "proxying to S3 + caching." This is simpler from deployment and development perspectives (for companies that do not have a distributed filesystem, like Google with its GFS):

* for deployment we do not need to maintain a pool of stateful boxes with files on them and keep these files in sync.

* for development, engineers now have a programatic interface for managing your static assets.

Re: Migrating Dropbox from Nginx to Envoy

#114

Post by pjmlp » Fri, Jul 31, 2020, 5:22 AM UTC

ram_rar wrote: post #68

I feel so old now. There was a time, when I used to discuss with senior engineers @ Yahoo! to use NginX over Apache. Nginx was the hot thing, popularizing C10k [1]. Now in my current team, I have junior devs in my team pushing for Envoy over HAProxy/Nginx setup. Is this trend happening primarily because devs are pushing for GRPC over REST? What benefits does Envoy offer over Nginx, if you're still a REST based servic…

Yep, gRPC is the new toy for distributed computing, after everyone realised that DCOM, CORBA, RMI, Remoting actually made sense instead of parsing XML and JSON text formats all the time.

Re: Migrating Dropbox from Nginx to Envoy

#115

Post by jhgg » Fri, Jul 31, 2020, 5:31 AM UTC

SaveTheRbtz wrote: post #4

Havoc wrote: post #3

Sensing a bit of a trend here. Didn't another major player recently make the same switch?

I think the best slice of who's migrating to Envoy can be observed via EnvoyCon talks[1][2]: * Lyft (of course) * Spotify * Stripe * Square * eBay * Yelp * Pinterest Plus the support from major cloud providers: Google, Microsoft, and Amazon. [1] https://envoyconna18.sched.com/ [2] https://envoycon2019.sched.com/

discord is also on that list - although we have not spoken much about it yet!

Re: Migrating Dropbox from Nginx to Envoy

#116

Post by neximo64 » Fri, Jul 31, 2020, 5:31 AM UTC

e40 wrote: post #40

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anythin…

Makes sense actually. Your motives are conflicted so you can't see it.

Also if I can ask, is your product also closed source (in any nature at all), but made with open source components?

Re: Migrating Dropbox from Nginx to Envoy

#117

Post by ganfortran » Fri, Jul 31, 2020, 5:32 AM UTC

e40 wrote: post #40

Also note that we’ll cover the open source version of the Nginx, not its commercial version with additional features. It always kills me when very successful companies don't buy software from other companies. I remember being at a lunch with a prospective client that really loved our technology. About 1/2 way through, he said he really would love to purchase our software, but the CEO doesn't allow them to use anythin…

This ain't mind-blowing by any means IMO.

If the said company has unknown track record, then doing business with them is risky.

What if the company goes out of business in near future? Or get acquired (actually I think A lot of infra companies's end goal is to get acquired)? What if they raise the price out of sudden? How extensible/customizable their solution is?

The trust is the key here. If I am in the position to buy software from somewhere and cost isn't the primary concern, the money would goes to a known/stable figure in the industry.

Re: Migrating Dropbox from Nginx to Envoy

#118

Post by atonse » Fri, Jul 31, 2020, 5:34 AM UTC

mholt wrote: post #89

SaveTheRbtz wrote: post #76

Earlier quoted context omitted.

To tell you the truth, we didn't consider it. From what I can get from the architecture docs[1], it can be a decent platform for apps, but might not be the best choice for a general purpose ingress/egress proxy (at least for now.) [1] https://caddyserver.com/docs/architecture

It is a great choice for a general purpose proxy. (That's kind of the point.)

But they mentioned that they wanted to use C++ instead of go to get even that extra performance out.

I use Caddy a lot and it's perfectly fine for my scale, but at dropbox's scale, maybe go wouldn't be enough for the ingress part?

Re: Migrating Dropbox from Nginx to Envoy

#119

Post by SaveTheRbtz » Fri, Jul 31, 2020, 5:34 AM UTC

MuffinFlavored wrote: post #95

Can somebody speak to why dynamic upstreams included in a file paired with `sudo service nginx reload` for prod deploys stopped scaling?

It is easy enough for simple cases (and we used it for quite a while, until we moved to using Lua for that.) For more complex scenarios you will have new `server` blocks, certificates, tls tickets, log files / syslog endpoints, so the automation will end up interacting not with just a single dynamic upstream file but with rather large amount of system interfaces. Control-plane ends up being distributed between config generation, filesystem state, service configuration (e.g. syslog.)

On a more practical note, each nginx `reload` will double the number of workers, almost doubling memory consumption and significantly increasing CPU usage (need to re-establish all TCP connections, re-do TLS handshake, etc.) So there is only that many reloads that you can do in an hour.

Re: Migrating Dropbox from Nginx to Envoy

#120

Post by pjmlp » Fri, Jul 31, 2020, 5:34 AM UTC

ci5er wrote: post #37

techntoke wrote: post #20

Earlier quoted context omitted.

They don't compete though. Google offers 17+GB free with email, office suite, unlimited photos, drive and Voice with a free number for unlimited calling and texting. Dropbox has much stricter bandwidth limits as well. For $9.99 you get all that plus 2TB storage with Google One. Dropbox has a minimum for of 3 users for their business plan, but with 1 user on G Suite for $12/mo I get unlimited storage and all the Goodi…

I'm learning to believe that there are various sub-sects of the HN crowd. The people who would rather pay $3/month to host a slow-ass VPS, and those where $10/month delta, as long as their work-flow is optimized is a "win". In fact, I think that drives a lot of these "what is better" debates in threads here. Some people go "Google Drive is better, because I get 2TB/month for a flat-fee that bundles the other services…

I am with you, also feel that HN has kind of turned into Slashdot of the yearly days.

On a website that started to discuss business ideas and how to turn them into profitable endeavours, it is a bit surreal that every time someone brings commercial products out, one gets endless posts about free beer alternatives and how such software is doing a disservice to the community.