Live data from Hacker News

Everybody gets WebSockets

blog.cloudflare.com

61–70 of 78 posts

Re: Everybody gets WebSockets

#61
post #42

Neat. We've been using CF websockets for some time now on the Enterprise plan. If you're intending to use CF websockets, be prepared for random (and potentially massive) connection drops, and be sure you're architected to handle these disconnects gracefully. Cloudflare rolling restarts have caused hundreds of thousands of websocket connections to be disconnected in a matter of minutes for us. If you plan to operate a…

Ugh, how do you shard websocket connections when you have more connection than 1 server can handle ?

Re: Everybody gets WebSockets

#62
post #60
post #28

Earlier quoted context omitted.

Precisely. Because of CloudFlare's position regarding Tor users, along with the erroneous idea that an identity is an IP address, these changes do nothing for the Tor user and developer community at large. Tor is used for more than just routing around censorship. I use it to create a seamless network of all my computers all via hidden services. So every machine has a "hidden service domain name" of [hash].onion . Kno…

Tor for remote access? There are much faster ways to create seamless networks of machines.

Indeed.

I have my IoT system across multiple networks, some of which I do not control the router. One possible solution was to have my VPS machine provide tunnels in a star topology. It's pretty sucky for the star center, because they end up getting all the traffic. And it's also pretty hard.

Another solution was to dyndns, port forwarding, router redirection, poking through NATs and all that... for each location I'm in. That's bad. Real bad.

Perhaps I could do some point-to-point trickery... but that doesn't work when both machines are NATted.

Tor Hidden Services provides a way to automatically breach the network seamlessly, and provide a routable address to that machine, no matter where that machine is. I take it to a cafe in Washington DC? Within 10 seconds, it's back on Tor. South America? 10 seconds.

The topology, once done, looks like a humongous ethernet hub, with no promiscuity mode. And each node is the 16 char hash.

Then, I can code against .onion addresses. They just work, and I know if I establish a connection, I can send data.

I'm already sending MQTT telemetry data from one network to my broker in another house 30 miles away. And it's sending pictures and metadata both. And it just works.

EDIT response:

>What kind of throughput & latency do you usually see?

Latency is a bigger one, obviously. It depends on the construction of the bridge. If I'm not using any overlays (OBFS3, OBFS4, scramblesuit, etc) then initial lag times can go in excess of 30 seconds. Once that initial connection is established, then lag times go down to about 200-300ms range.

Using overlays, because the network blocks various vanilla types of Tor, can take a lot longer. That's because those overlays are beat on by China and Iran. Sometimes they will dead-route packets (5% of the time). Initial transit I've seen up to 1 minute, with avg ping times around 500 ms.

Throughput is a different beast. The only network I haven't saturated was my 1GBps desktop at work. I can stream movies directly with the speeds I routinely get. Just that initial bridge construction will make you think something went wrong.

(responded here because "I'm responding too fast")

Re: Everybody gets WebSockets

#63
post #62
post #60

Earlier quoted context omitted.

Tor for remote access? There are much faster ways to create seamless networks of machines.

Indeed. I have my IoT system across multiple networks, some of which I do not control the router. One possible solution was to have my VPS machine provide tunnels in a star topology. It's pretty sucky for the star center, because they end up getting all the traffic. And it's also pretty hard. Another solution was to dyndns, port forwarding, router redirection, poking through NATs and all that... for each location I'm…

What kind of throughput & latency do you usually see?

I imagine this is a lot better than hitting an exit node.

Re: Everybody gets WebSockets

#64
post #42

Neat. We've been using CF websockets for some time now on the Enterprise plan. If you're intending to use CF websockets, be prepared for random (and potentially massive) connection drops, and be sure you're architected to handle these disconnects gracefully. Cloudflare rolling restarts have caused hundreds of thousands of websocket connections to be disconnected in a matter of minutes for us. If you plan to operate a…

Ugh, how do you shard websocket connections when you have more connection than 1 server can handle ?

It's not ideal for sure but you can use DNS to divvy it up by adding several IPs to your domain name.

Re: Everybody gets WebSockets

#65

Earlier quoted context omitted.

Ugh, how do you shard websocket connections when you have more connection than 1 server can handle ?

It's not ideal for sure but you can use DNS to divvy it up by adding several IPs to your domain name.

You can also do it out of band and have an API call that returns the server you're supposed to be connected to.

Re: Everybody gets WebSockets

#66
post #62
post #60

Earlier quoted context omitted.

Tor for remote access? There are much faster ways to create seamless networks of machines.

Indeed. I have my IoT system across multiple networks, some of which I do not control the router. One possible solution was to have my VPS machine provide tunnels in a star topology. It's pretty sucky for the star center, because they end up getting all the traffic. And it's also pretty hard. Another solution was to dyndns, port forwarding, router redirection, poking through NATs and all that... for each location I'm…

Not to diminish your TOR advocacy, but check out tinc.

Re: Everybody gets WebSockets

#67
post #42

Neat. We've been using CF websockets for some time now on the Enterprise plan. If you're intending to use CF websockets, be prepared for random (and potentially massive) connection drops, and be sure you're architected to handle these disconnects gracefully. Cloudflare rolling restarts have caused hundreds of thousands of websocket connections to be disconnected in a matter of minutes for us. If you plan to operate a…

Ugh, how do you shard websocket connections when you have more connection than 1 server can handle ?

If you use Elixir/Phoenix, you can have many servers running same code, connected with each other as Erlang nodes. When client wants to broadcast something in whole network, it is broadcasted locally but is also sent do all servers and servers broadcasts it to their local clients. It's very easy to write.

Re: Everybody gets WebSockets

#68

Cloudflare is underrated, I am personally looking forward to their future.

I like Cloudflare a lot, but I am worried about their prevalance.

Similar to other companies I give up a good control of my life to, Cloudflare essentially becoming a new internet backbone is ever so slightly terrifying.

Cloudflare itself is a "single point of failure." (Though massively unlikely with their incredible infra).

Cloudflare could shut off all of it's servers, or it's nameservers could go haywire, and there would be a massive internet outage. Much like when AWS goes down.

Re: Everybody gets WebSockets

#69
It's nice but it was never a real technological barrier as much as it was a psychological barrier. You could always have added your WebSocket service under a different subdomain and then turned off CF for that specific subdomain only (CF always had this feature).

You can have different subdomains map to the same server IP address so I can't think of a reason why this would be a problem. That's what we did for https://baasil.io/

It's still nice though from the point of view that you no longer have to think about it. A lot of users wrongly assumed that you couldn't have Cloudflare AND WebSockets but you always could. At least now there won't be any confusion since it will just work by default.

Also, now that CF supports WebSocket as part of their offering, maybe they will also offer features like WebSocket rate limiting and such (for protecting against DoS)... And that would be useful.

Re: Everybody gets WebSockets

#70
post #42

Neat. We've been using CF websockets for some time now on the Enterprise plan. If you're intending to use CF websockets, be prepared for random (and potentially massive) connection drops, and be sure you're architected to handle these disconnects gracefully. Cloudflare rolling restarts have caused hundreds of thousands of websocket connections to be disconnected in a matter of minutes for us. If you plan to operate a…

As you know, when terminating a WebSocket connection due to releases CloudFlare now signals this action to both client and origin server by sending the 1001 status code (aka "going away", see section 7.4.1 of RFC 6455), so both sides are aware that the WebSocket termination is only a transient event, and that they can expect to immediately re-establish a connection again on retry.

We're working on additional refinements to the release process to minimize disruptions.

(I work at CloudFlare.)

Post reply on HN