We abused Slack's TURN servers to gain access to internal services • Hacker News Board

Re: We abused Slack's TURN servers to gain access to internal services

#21

Post by Sean-Der » Tue, Apr 07, 2020, 5:57 PM UTC

gfodor wrote: post #12

wrkronmiller wrote: post #5

As a complete novice in this area, I don't understand the advantage of using a proxy-like service such as TURN. What is the advantage over simply routing the media streams through application servers (i.e. user A connects to server which links to user B) which can then perform application-specific authentication, enforce restrictions on payloads, etc... Performance?

If you have a centralized server, you have a SFU. SFUs typically expose a range of UDP (and/or TCP) ports for communication. Peer connections are allocated on a port basis. So if a user is connected to your SFU, they take up a port, and need to be able to egress over a large UDP/TCP port range to connect, since the port is assigned randomly. However, many firewalls block port ranges, or even UDP entirely. What you re…

There is no reason an SFU couldn't run everything over one port though! Then you can just use the 3-tuple to route stuff to the proper connection.

Someone is doing this right now for Pion, really excited to see it. I am especially excited to see what it means for deploys, right now asking people to expose port ranges adds so much overhead vs 1 UDP and 1 TCP for media.

Re: We abused Slack's TURN servers to gain access to internal services

#22

Post by mypalmike » Tue, Apr 07, 2020, 6:00 PM UTC

wrkronmiller wrote: post #10

aclavelle wrote: post #9

Earlier quoted context omitted.

My knowledge is about 2 years old on this but I can try to explain: TURN/STUN are to facilitate users communicating behind NAT and firewalls. TURN routes all traffic through a central server and pushes it to clients which it has an established connection with, thus getting around NAT/Firewall. STUN is a bit more lightweight in that it really just helps users to negotiate a normal P2P connection and then they send mes…

Thanks! That's in-line with what I thought was going on. It sounds like TURN is very close to being an open proxy. Rather than falling back from p2p to STUN to TURN, why not replace TURN with something more application/protocol-specific? Perhaps a webrtc-only proxy that performs authentication and can perform authorization along the lines of: user A is (only) allowed to connect to user B using protocol WebRTC.

A custom intermediary needs to perform some expensive operations, such as decryption and re-encryption of the DTLS and SRTP going through it. It's much simpler and cheaper to just forward packets.

Re: We abused Slack's TURN servers to gain access to internal services

#23

Post by StillBored » Tue, Apr 07, 2020, 6:01 PM UTC

gfodor wrote: post #19

jackewiehose wrote: post #17

Earlier quoted context omitted.

It's not really a bug in old coturn, just a feature in the protocol. According to the article newer versions just disable routing to 127.0.0.1 by default but there are still other network addresses you might have to consider (see article for a recommended list of "denied-peer-ips").

Yup, sorry if my phrasing implied it's a bug. It's just better defaults, and there's evidence to support the idea slack was just running the software with the old defaults.

But its not necessarily the defaults. The article isn't clear if the private services are on 127/8 or the TURN server has access to other things in the DMZ/VPC/whatever.

I'm guessing its actually the latter as everyone is so fond of the 1:1 VM/container->service model. Meaning its likely a config problem with the denied-peer-ips the parent here links.

Re: We abused Slack's TURN servers to gain access to internal services

#24

Post by 0az » Tue, Apr 07, 2020, 6:13 PM UTC

Sean-Der wrote: post #21

gfodor wrote: post #12

Earlier quoted context omitted.

If you have a centralized server, you have a SFU. SFUs typically expose a range of UDP (and/or TCP) ports for communication. Peer connections are allocated on a port basis. So if a user is connected to your SFU, they take up a port, and need to be able to egress over a large UDP/TCP port range to connect, since the port is assigned randomly. However, many firewalls block port ranges, or even UDP entirely. What you re…

There is no reason an SFU couldn't run everything over one port though! Then you can just use the 3-tuple to route stuff to the proper connection. Someone is doing this right now for Pion, really excited to see it. I am especially excited to see what it means for deploys, right now asking people to expose port ranges adds so much overhead vs 1 UDP and 1 TCP for media.

Can you elaborate on what this means?

Re: We abused Slack's TURN servers to gain access to internal services

#25

Post by Sean-Der » Tue, Apr 07, 2020, 6:20 PM UTC

0az wrote: post #24

Sean-Der wrote: post #21

Earlier quoted context omitted.

There is no reason an SFU couldn't run everything over one port though! Then you can just use the 3-tuple to route stuff to the proper connection. Someone is doing this right now for Pion, really excited to see it. I am especially excited to see what it means for deploys, right now asking people to expose port ranges adds so much overhead vs 1 UDP and 1 TCP for media.

Can you elaborate on what this means?

Right now most SFUs start up an ICE Agent [0] and listen to a random port. ICE is used to establish the connection between two peers. Basically both sides exchange a list of peers, and try to find the best path.

With an SFU you end up having thousands of remote peers each with their own port on your server. However you could easily listen on a single port and then handle the inbound packet depending on what the remote 3-tuple is (clients ip/port/protocol). Effectively you would just be running all your ICE Agents on one port, but doing one additional step of processing.

I need to fill out [1] more to fully explain the idea, but I think it could make a huge difference when making it easier to deploy WebRTC SFUs.

[0] https://github.com/pion/ice

[1] https://github.com/pion/webrtc/wiki/SinglePortMode

Re: We abused Slack's TURN servers to gain access to internal services

#26

Post by jlokier » Tue, Apr 07, 2020, 6:34 PM UTC

wrkronmiller wrote: post #10

aclavelle wrote: post #9

Earlier quoted context omitted.

My knowledge is about 2 years old on this but I can try to explain: TURN/STUN are to facilitate users communicating behind NAT and firewalls. TURN routes all traffic through a central server and pushes it to clients which it has an established connection with, thus getting around NAT/Firewall. STUN is a bit more lightweight in that it really just helps users to negotiate a normal P2P connection and then they send mes…

Thanks! That's in-line with what I thought was going on. It sounds like TURN is very close to being an open proxy. Rather than falling back from p2p to STUN to TURN, why not replace TURN with something more application/protocol-specific? Perhaps a webrtc-only proxy that performs authentication and can perform authorization along the lines of: user A is (only) allowed to connect to user B using protocol WebRTC.

A TURN server has to do much less computation, and it also doesn't need to decrypt the payload. It's more or less a fancy packet forwarder.

In addition, only a fraction of users will need TURN; the rest can use direct peer connections with the aid of NAT traversal; the two kinds of connection are more or less the same to higher layers. Conversely, if the application depended on an application server to process data, chances are you wouldn't implement a second version of the same protocol that works without the server.

So a single TURN server can handle a lot more traffic than an application server, is potentially more secure, and is more easily shared between different applications, and even different owners.

If you want it geo-distributed for latency, the ability to share the same TURN servers between different applications and owners gives you cost-latency advantages too.

Re: We abused Slack's TURN servers to gain access to internal services

#27

Post by jerf » Tue, Apr 07, 2020, 6:38 PM UTC

jackewiehose wrote: post #17

gfodor wrote: post #11

Huh, I happen to be knee-deep in this stuff right now. This article noted that Slack seemed to be running an old TURN server (pre-coturn): https://webrtchacks.com/slack-webrtc-slacking/ Given that the latest coturn has this vulnerability mitigated by default, perhaps all this boils down to is "Slack runs outdated software, we exploited it."?

It's not really a bug in old coturn, just a feature in the protocol. According to the article newer versions just disable routing to 127.0.0.1 by default but there are still other network addresses you might have to consider (see article for a recommended list of "denied-peer-ips").

You may already know this, but it's worth getting the word out. Do not just deny routing to 127.0.0.1. 127.0.0.1 is merely the conventional "localhost" address; however, ALL 127.x.x.x is "localhost". You can check this now on your local command line with "ping 127.1.2.3".

(This just seems to be one of those bugs that every proxy goes through at some point, just like pretty much any attempt to write a web server that serves files off disk will have at least one directory traversal bug.)

Re: We abused Slack's TURN servers to gain access to internal services

#28

Post by jrockway » Tue, Apr 07, 2020, 6:45 PM UTC

Things like this are why mTLS internally are so important. If a hole is found in your firewall, services still don't trust each other until they have a valid TLS certificate.

Re: We abused Slack's TURN servers to gain access to internal services

#29

Post by rvz » Tue, Apr 07, 2020, 6:52 PM UTC

jsjddbbwj wrote: post #4

kylek wrote: post #2

tldr- November 2017: added TURN abuse to our stunner toolset December 2017: discovered and reported TURN vulnerability in private customer of Enable Security February 2018: briefly tested Slack and discovered the vulnerability April 2018: submitted our report to Slack, helped them reproduce and address the issue through various rounds of testing May 2018: Slack pushed patch to live servers which was retested by Enabl…

Stop using code blocks if you're not posting code

  Sorry big brother.

Re: We abused Slack's TURN servers to gain access to internal services

#30

Post by gfodor » Tue, Apr 07, 2020, 6:52 PM UTC

Sean-Der wrote: post #25

0az wrote: post #24

Earlier quoted context omitted.

Can you elaborate on what this means?

Right now most SFUs start up an ICE Agent [0] and listen to a random port. ICE is used to establish the connection between two peers. Basically both sides exchange a list of peers, and try to find the best path. With an SFU you end up having thousands of remote peers each with their own port on your server. However you could easily listen on a single port and then handle the inbound packet depending on what the remot…

Yup that's a great point. I'd love to see this approach explored further. Is there any risk of tuple collisions in some bizarro NAT situation? I'd guess not, since the remote tuple needs to route to a single destination, but there's some weird stuff out there... eg one could imagine a router abusing the IP protocol to somehow route packets to different destinations despite them having the same return IP/port combo. i'm no networking wizard, but in general i assume if its possible, someone is doing it :)