Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The hidden complexity of scaling WebSockets

composehq.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: The hidden complexity of scaling WebSockets

#61

Post by [deleted] »

klabb3 wrote: post #47
pk-protect-ai wrote: post #34

Earlier quoted context omitted.

I agree here. I have had an experience of scaling WebSockets server to 20M connections on a single server (with this one https://github.com/ITpC/LAppS.git ). However there are several issues with scaling WebSockets, on the backends as well: mutex locking, non-parallel XOR of input stream, utf8 validation. I do not know the state of the above repository code, it seems that it was never updated for at least 5 years. Th…

> non-parallel XOR of input stream I remember this one in particular making me upset, simply because of another extra buffer pass for security reasons that I believe are only to prevent proxies doing shit they never should have done in the first place?

[deleted]

Re: The hidden complexity of scaling WebSockets

#64

Post by akshayKMR »

jwr wrote: post #53

My SaaS has been using WebSockets for the last 9 years. I plan to stop using them and move to very simple HTTP-based polling. I found that scalability isn't a problem (it rarely is these days). The real problem is crappy network equipment all over the world that will sometimes break websockets in strange and mysterious ways. I guess not all network equipment vendors test with long-lived HTTP websocket connections wit…

What are the typical payload sizes in your WebSocket messages? Could you share the median and p99 values?

I've also discovered similar networking issues in my own application while traveling. For example, in Vietnam right now, I was facing recurring issues like long connection establishment times and loss of responsiveness mid-operation. I thought I was losing my mind - I even configured Caddy to not use HTTP3/QUIC (some networks don't like UDP).

I moved some chunkier messages in my app to HTTP requests, and it has become much more stable (though still iffy at times).

Re: The hidden complexity of scaling WebSockets

#65

Post by mirekrusin »

crabmusket wrote: post #51
mirekrusin wrote: post #31

Earlier quoted context omitted.

It's a bit misnomer because it defines rpcs _and_ notifications. What people seem to be often missing for some reason is that those two map naturally to existing semantics of the programming language they're already using. What it means in practice is that you are exposing and consuming functions (ie. on classes) – just like you do in ordinary libraries. In js/ts context it usually means async functions on classes an…

That do you see as the difference between an RPC and a notification? The terminology is not ideal, I grant, but a JSON-RPC "notification" (a request with no id) is just a request where the client cannot, and does not, expect any response, not even a confirmation that the request was received and understood by the server. It's like UDP versus TCP. > emitting individual objects for array results This is interesting! Ho…

Yes that’s all there is in the difference remote procedure call expects response, remote notification doesn’t.

Our implementation emits notifications for entries and rpc returns done payload (which is largely irrelevant just the fact of completion is relevant).

As I said it would be nice if they’d support generator functions at the protocol level.

Re: The hidden complexity of scaling WebSockets

#66

Post by hinkley »

ignoramous wrote: post #40
hinkley wrote: post #16

Earlier quoted context omitted.

Functional core, imperative shell makes testing and this fast iteration a lot easier. It’s best if your business logic knows very little about transport mechanisms. I think part of the problem is that early systems wanted to eagerly process requests while they are still coming in. But in a system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache line…

> system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache lines At what point should one scale up & switch to chips with embedded DRAMs ("L4 cache")?

I haven’t been tracking price competitiveness on those. What cloud providers offer them?

But you don’t get credit for having three tasks halfway finished instead of one task done and two in flight. Any failover will have to start over with no forward progress having been made.

ETA: while the chip generation used for EC2 m7i instances can have L4 cache, I can’t find a straight answer about whether they do or not.

What I can say is that for most of the services I benchmarked at my last gig, M7i came out to be as expensive per request as the m6’s on our workload (AMD’s was more expensive). So if it has L4 it ain’t helping. Especially at those price points.

Re: The hidden complexity of scaling WebSockets

#67

Post by catlifeonmars »

jwr wrote: post #53

My SaaS has been using WebSockets for the last 9 years. I plan to stop using them and move to very simple HTTP-based polling. I found that scalability isn't a problem (it rarely is these days). The real problem is crappy network equipment all over the world that will sometimes break websockets in strange and mysterious ways. I guess not all network equipment vendors test with long-lived HTTP websocket connections wit…

This is surprising to me as I would expect network equipment to just see a TCP connection given both HTTP and Websockets are an application layer protocol and that long lived TCP connections are quite ubiquitous (databases, streaming services, SSH, etc).

Re: The hidden complexity of scaling WebSockets

#68

Post by bruce343434 »

ignoramous wrote: post #40
hinkley wrote: post #16

Earlier quoted context omitted.

Functional core, imperative shell makes testing and this fast iteration a lot easier. It’s best if your business logic knows very little about transport mechanisms. I think part of the problem is that early systems wanted to eagerly process requests while they are still coming in. But in a system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache line…

> system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache lines At what point should one scale up & switch to chips with embedded DRAMs ("L4 cache")?

When you've profiled the code running in production and identified memory bottlenecks that can not be solved by algorithmic/datastructural optimizations.

Re: The hidden complexity of scaling WebSockets

#69

Post by atul-jalan »

Sytten wrote: post #63

The comment about Render/Railway gracefully tranferring connections seems weird? I am pretty sure it just kills the service after the new one is alive which will kill the connections. Not some fancy zero downtime reconnect.

It does, but it will generally give a grace period for outstanding requests to resolve prior to killing the service.

It's a good idea for short-lived HTTP requests, but will cause problems for a persistent connection.

Re: The hidden complexity of scaling WebSockets

#70

Post by jwr »

akshayKMR wrote: post #64
jwr wrote: post #53

My SaaS has been using WebSockets for the last 9 years. I plan to stop using them and move to very simple HTTP-based polling. I found that scalability isn't a problem (it rarely is these days). The real problem is crappy network equipment all over the world that will sometimes break websockets in strange and mysterious ways. I guess not all network equipment vendors test with long-lived HTTP websocket connections wit…

What are the typical payload sizes in your WebSocket messages? Could you share the median and p99 values? I've also discovered similar networking issues in my own application while traveling. For example, in Vietnam right now, I was facing recurring issues like long connection establishment times and loss of responsiveness mid-operation. I thought I was losing my mind - I even configured Caddy to not use HTTP3/QUIC (…

I transmit a lot over websockets. Large messages and large amounts of data. I don't think it makes sense to move bigger messages to HTTP requests while keeping the websockets — I heard that advice, but if I am to do that, I'd rather go all the way and stop using websockets altogether.
Post reply on HN