Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's TCP vs. RPC All over Again

systemsapproach.substack.com

Post reply on HN
Print view View original

31–40 of 116 posts

Re: It's TCP vs. RPC All over Again

#31

Post by chasil »

cryptonector wrote: post #12

> the Internet seems to be missing a standard protocol for the request/response paradigm, with repeated attempts to force-fit TCP leading to inevitable mismatches HTTP is that. QUIC is that over UDP . We've discussed Ousterhout's paper about replacing TCP in the data center. Last I looked the answer was "not with what is described in the paper". > I have never understood why the Internet has worked so persistently to…

>d) having one protocol with bindings to all applicable ULPs will help us get over the kernel support issues.

Isn't the purpose of QUIC to hide all of this from the kernel, at the cost of statically binding the library to the app (as Go might)?

>Cross-site linking is so incredibly useful that it can't be done without. And now you need a fancy user-agent (browsers), so the web folks built one (browsers).

Just set the MIME to text/plain?

Re: It's TCP vs. RPC All over Again

#32

Post by cjsplat »

So the question is why hasn't an optimized RPC implementation emerged for the data center that avoids the headaches that come with layering over TCP.

Turns out integrating with the control plane is hard at data center scale.

When everything is working, an optimized protocol stack is great and everyone is happy.

When congestion happens (fabric, kernel, memory bus, cache footprint, etc), and your optimized app causes a legacy production app to behave strangely, all hell breaks loose.

For example, a distributed file system SRE sees some strange tail latency effects and doesn't know that a cluster is shared with a non-TCP app and of course the optimized stack doesn't respond to congestion the way TCP would.

Worst case, the SREs notice this just after an update is pushed in their app, and they spend a random amount of time trying to figure out if the update is what changed the behavior.

Best case, they know there are non-TCP apps in the area, and they ping the SRE for that stack and say "my app is behaving strange, please hit your big red off button so I can see if that fixes the problem."

Someplace in between those options, the SRE is using TCP aware network debugging tools to try to figure out why this day is different from yesterday, or this cluster is different from the one where everything is working fine.

Regardless, you get an unhappy SRE. But of course you never get just one unhappy SRE.

So you need to generate a lot of value got justify their pain.

Re: It's TCP vs. RPC All over Again

#33

Post by cjsplat »

cryptonector wrote: post #13
irq-1 wrote: post #6

Really interesting article. Wish there was a discussion of performance for various RPCs from the past.

There's so many... Apollo Domain's. ONC RPC. DCE RPC (and MSFT RPC, which derives from DCE RPC). SOAP. LustreRPC. And who knows how many others. The biggest problem with RPC in general is that historically it was synchronous because that's what was easy to implement in the 80s. Fix that and an RPC sucks only as much as the encoding system it uses. But RPC == remote procedure call, and that causes people to instantly…

Well of course once you say "procedure call", people think synchronous.

For the sender, asynchronous RPC is just a convenient marshalling and return interface for the sender. It is pretty clunky for the A-RPC caller to create an illusion in the code of the RPC being "just like" a local procedure call.

For the called procedure an Async RPC can look just like a local sync procedure call, except for that whole address space thing.

I don't think that sync RPC was popular because it was easy to implement.

I think it was popular because it was easy to code to. Multi-threaded coding is easy to get wrong, and was poorly understood at the time.

Re: It's TCP vs. RPC All over Again

#35

Post by cjsplat »

Animats wrote: post #30

There have been lots of RPC protocols. Here are some still in use. Transport level: * Sun RPC [1]. QNX still uses this. It can run over UDP or over raw Ethernet. It just transfers an array of bytes and gets an array of bytes back - marshalling is a higher level problem. It handles messages bigger than one packet, and retransmission. It's simple and performance is good, but there is no security. * Stream Control Trans…

Yeah, but the question at hand is why layer it over TCP.

SunRPC over UDP is close, but then you push reliability and congestion management into every application.

Re: It's TCP vs. RPC All over Again

#36

Post by cjsplat »

AceJohnny2 wrote: post #9
iambvk wrote: post #4

Good to learn a bit of history about TCP. It always felt weird developing RPC layers on top of TCP cause requests and responses end up tied to the underlying socket -- which doesn't need to be the case ever.

How else would the kernel know to which application it should route a response?

Destinations can be described in the packet headers. That is how the current systems work.

Re: It's TCP vs. RPC All over Again

#37

Post by roncesvalles »

You could just use UDP instead of TCP and implement some subset of TCP features that you need in-band, like sequence numbers. This is not a new concept. The Facebook Memcache paper [1] in 2013 described using UDP for intra-datacenter requests.

[1] https://research.facebook.com/publications/scaling-memcache-...

Re: It's TCP vs. RPC All over Again

#39

Post by cryptonector »

chasil wrote: post #31
cryptonector wrote: post #12

> the Internet seems to be missing a standard protocol for the request/response paradigm, with repeated attempts to force-fit TCP leading to inevitable mismatches HTTP is that. QUIC is that over UDP . We've discussed Ousterhout's paper about replacing TCP in the data center. Last I looked the answer was "not with what is described in the paper". > I have never understood why the Internet has worked so persistently to…

>d) having one protocol with bindings to all applicable ULPs will help us get over the kernel support issues. Isn't the purpose of QUIC to hide all of this from the kernel, at the cost of statically binding the library to the app (as Go might)? >Cross-site linking is so incredibly useful that it can't be done without. And now you need a fancy user-agent (browsers), so the web folks built one (browsers). Just set the…

> Isn't the purpose of QUIC to hide all of this from the kernel, [...]

Yes, but the moment you want to use a new ULP (which is what TFA seems to be angling for) you'll basically need kernel support, except for kernel-mode / baremetal apps that aren't really relevant to this discussion.

Re: It's TCP vs. RPC All over Again

#40

Post by ur-whale »

User23 wrote: post #23
bob1029 wrote: post #17

There are some things that current RPC frameworks simply cannot touch. I've been up to my elbows in TCP lately. Working on a low-latency, select-based socket server for streaming gaming applications. Each instance runs on 1 thread so there is absolutely no context switching delay when servicing player requests. The TCP streams are responsible for moving player inputs and game state as quickly as possible between syst…

A well written C mux is absurdly performant. I was handling thousands of requests per second with efficient hand rolled binary protocols in the early oughts when I could make UDP work, which I could since it was an intra datacenter app and a super lightweight exponential backoff retry scheme on the client side was entirely adequate. “Modern” network programming is hilariously inefficient. I think games are the only m…

This.

The waste of doing RPC over TCP is simply astounding.

Post reply on HN