Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proxygen, Facebook's C++ HTTP Framework

code.facebook.com

Post reply on HN
Print view View original

91–98 of 98 posts

Re: Proxygen, Facebook's C++ HTTP Framework

#91

Post by nbm »

johnbellone wrote: post #88
nbm wrote: post #58

Earlier quoted context omitted.

By expensive, not just capital costs, but costs around operating them - they weren't as reliably configurable, health-checkable, and instrumentable as we'd want, and Proxygen (and a later L4 load balancer) were. Also the previous load balancers had constraints we weren't willing to accept - they required special connectivity to our networks, we could not use particular combinations of options, and we had to rely on v…

Yeah, we are hitting the same wall right now and we've been going down the same route using HAProxy/Chef. Our plan is to put an API in front of it and treat it similar to an ELB. Are you still using hardware load balancers for SSL termination?

The machines running Proxygen do the TLS termination.

Re: Proxygen, Facebook's C++ HTTP Framework

#92

Post by dcsommer »

dimman wrote: post #85

Facebook oughta hire some better scripters, the deps.sh is of terribly low quality. I didn't get more than a couple of lines until i stumble upon this (which tells me the author has no clue :): 'start_dir=`pwd`; trap "cd $start_dir" EXIT;'... No need to say that the script can be dangerous, in case directory change fails for instance, there's no checks but sudo make uninstall is run anyway in another dir than the int…

Bash isn't my expertise and I put this together pretty quickly. Please send pull requests! Forgive my ignorance, but what's the danger of the cd'ing in the EXIT trap? Also, I did set -e, so there's no problem of running "sudo make uninstall" from the wrong directory, afaict.

Re: Proxygen, Facebook's C++ HTTP Framework

#93

Post by nbm »

e12e wrote: post #76
dcsommer wrote: post #56

Earlier quoted context omitted.

The blog post goes into more detail, but Proxygen started with an effort to write a L7 reverse proxy that could deeply integrate into FB internal services. We pulled out a lot of the non-FB specific stuff into this open source release. Before that, we used hardware load balancers for this role, which was expensive.

I'll have a look at the blog for more -- but this was something that couldn't be solved with haproxy and/or trafficserver?

haproxy only got TLS support in 2012, whereas work on proxygen began quite before that. In addition, haproxy does not support SPDY natively (although it can forward it to something that does).

Apache Traffic Server only got SPDY support in a release about 45 days ago according to their release notes.

There are many other relatively basic features besides those, and similar considerations exist for the other projects that existed back then (and even now).

Re: Proxygen, Facebook's C++ HTTP Framework

#94

Post by dimman »

dcsommer wrote: post #92
dimman wrote: post #85

Facebook oughta hire some better scripters, the deps.sh is of terribly low quality. I didn't get more than a couple of lines until i stumble upon this (which tells me the author has no clue :): 'start_dir=`pwd`; trap "cd $start_dir" EXIT;'... No need to say that the script can be dangerous, in case directory change fails for instance, there's no checks but sudo make uninstall is run anyway in another dir than the int…

Bash isn't my expertise and I put this together pretty quickly. Please send pull requests! Forgive my ignorance, but what's the danger of the cd'ing in the EXIT trap? Also, I did set -e, so there's no problem of running "sudo make uninstall" from the wrong directory, afaict.

Last msg sounded a bit harsh, sry about that. Anyway, some things to consider:

1. You don't need bash, but rather use /bin/sh to be more compatible with other shells (I don't have bash, neither does a lot of other systems after latest Shellshock incident). There's really no need to limit it to bash (bash is one of many shells but very commonly mistaken for "shell script").

2. The script is executed in a subshell, so the directory your script is in when exiting is irrelevant, it doesn't affect the caller at all. Try by creating a new script that just does 'cd a_dir_that_exists' and run it from a terminal. :)

3. set -e makes the program stop in case of _unhandled_ errors yes, so you're right, the example I gave is indeed wrong and it would stop on the failed cd attempt.

Instead of using '|| true' (to deliberatly ignore errors), the std way to do it is '|| :' (which doesn't fork the true binary). However I would really recommend taking care and handling possible errors.

Re: Proxygen, Facebook's C++ HTTP Framework

#95

Post by matt_d »

dcsommer wrote: post #2

Hey there, I work on Proxygen at Facebook. I'm happy to answer any questions you have about the project.

Hi! Looks interesting!

I have a question -- in particular, the blog post mentions that the framework "includes both server and client code", any my question is about the second part :-)

I'm wondering, how does it compare to the other C++ HTTP client solutions: Is it closer to higher-level libraries like cpp-netlib, Casablanca, or POCO -- or more on a lower-level / comparable to Asio (or Boost.Asio)?

In your view, what are the main relative advantages/disadvantages (namely in the scenario mentioned in the blog post, i.e., integration into existing applications)?

Re: Proxygen, Facebook's C++ HTTP Framework

#96

Post by taha_jahangir »

nbm wrote: post #93
e12e wrote: post #76

Earlier quoted context omitted.

I'll have a look at the blog for more -- but this was something that couldn't be solved with haproxy and/or trafficserver?

haproxy only got TLS support in 2012, whereas work on proxygen began quite before that. In addition, haproxy does not support SPDY natively (although it can forward it to something that does). Apache Traffic Server only got SPDY support in a release about 45 days ago according to their release notes. There are many other relatively basic features besides those, and similar considerations exist for the other projects…

He ask the reason for `not contributing` to that projects, not `using` them.

Re: Proxygen, Facebook's C++ HTTP Framework

#97

Post by datashovel »

datashovel wrote: post #83
dcsommer wrote: post #13

Earlier quoted context omitted.

Yup, we still use Proxygen (the library) in our reverse proxy. Maybe some day we'll be able to open source the reverse proxy too, but it's pretty deeply integrated with internal FB code right now so that's tricky. We already use the Proxygen HTTP code for the webserver part of HHVM internally. We hope to release that webserver part too (in the HHVM project).

I would love to see Proxygen integrated as a HackLang extension. Especially the HTTP parser. As far as I know, the PHP world is sorely lacking a robust HTTP parser.

In response to my comment here, I did some more experimentation with pecl_http. Turns out version 2.1.4 looks relatively stable. Just be sure not to make the same mistake I made of referring to the documentation on the php.net website. Instead one should look here for version 2.x documentation:

http://devel-m6w6.rhcloud.com/mdref/http/

Re: Proxygen, Facebook's C++ HTTP Framework

#98

Post by cbsmith »

dcsommer wrote: post #42
amelius wrote: post #39

Skimming through the article, it seems to me that this server spawns a thread per connection, is that correct?

We use a very different model actually. Since spawning OS threads is expensive, we opted for the popular nonblocking-IO approach. Each worker thread (usually 1 per core on the CPU) is given connections in a round robin fashion from the listening socket. The worker thread runs an event loop processing events on the accepted socket.

Note: as google discovered a few years ago, round-robining the connections is probably less than ideal.
Post reply on HN