11–20 of 169 posts
I wish that more companies could be like this and skip the corporate BS, it shows that they really have something outstanding to offer.
Earlier quoted context omitted.
This is where the article lost me a little bit. I (think I) technically got the part of running a TCP/IP stack in an unprivileged user process, so you don't have to elevate privilege for adding a network interface and using the host OS TCP/IP stack. And maybe that's already very cool. But: - What other benefits does it give you? - This isn't a new problem and presumably has prior best practices for mitigation. What i…
The problem isn't new but the previous best practices involved giving the tool super user. Usually through an installation process. See most other VPNs. UserMode TCP/IP stacks aren't very common in practice. This is why what fly.io did is interesting.
I keep thinking of the nightmare of keeping up with the world of Internet middleboxens, broken net layer implementations and icmp hacks that the Linux kernel supports and makes 'just work'. The jump to usermode tcp seems interesting if you're not worried about that (and I've been watching the formal-proven ip/tcp stack space like a hawk for years), but I've been burned so many times with non standard stacks and 'oh you need to connect to that non-updated lynxos system and huh' or 'hah could you enable ecn or this obscure tcp option because... Legacy?'... And sometimes I need tc/netem and netlink and I don't know...
Fly.io's blog posts are incredible, they really seem to really enjoy what they do and want to share what they've made with everyone else. I love them for that. I wish that more companies could be like this and skip the corporate BS, it shows that they really have something outstanding to offer.
(And happens to be a top HN contributor)
I'd have to dig into the details more, but something like this might allow you to implement a simple tunneling system based on WireGuard that runs in the kernel if you have the privileges, otherwise falls back to usermode and is no worse than QUIC in terms of performance. That would be awesome.
Because 'sho_hn brought this up, here's a stab at a pro/con list of building TCP/IP directly into our API the way `flyctl` does: Pro: + Can just run "native" SSH directly over it (or, in our case, use x/crypto/ssh, without modification). + Lets `flyctl` offers a `flyctl proxy` command to users, so they can plug their own programs into whatever application they need to use, without asking us to change some proxy we ru…
Tangent: that's debatable IMO. In my company's current AWS infrastructure, there's no shell access to either the production containers or the host machines. I did write a script to create an ephemeral container that lets me (and future staff) run a shell inside the production network. And the thing I usually do in that shell is run psql; I suppose that's not ideal for auditability. But still, I can't poke around in the live containers or the host machines; in theory they could be distroless, with no shell at all. I'm trying to take immutable infrastructure to the max here; this seems like a good thing for security. It does mean that for debugging production problems, I can only go by what I find in logs and the database. But that has been acceptable so far.
Edit: Given tptacek's security background, I was surprised that he considered production shell access essential for a new app platform.
Because 'sho_hn brought this up, here's a stab at a pro/con list of building TCP/IP directly into our API the way `flyctl` does: Pro: + Can just run "native" SSH directly over it (or, in our case, use x/crypto/ssh, without modification). + Lets `flyctl` offers a `flyctl proxy` command to users, so they can plug their own programs into whatever application they need to use, without asking us to change some proxy we ru…
Question: ultimately all the packets are actually being sent via a Golang net.UDPConn right? ie you're simulating raw network packets by wrapping them in UDP packets, then running TCP in Golang over those wrapped packets?
Because 'sho_hn brought this up, here's a stab at a pro/con list of building TCP/IP directly into our API the way `flyctl` does: Pro: + Can just run "native" SSH directly over it (or, in our case, use x/crypto/ssh, without modification). + Lets `flyctl` offers a `flyctl proxy` command to users, so they can plug their own programs into whatever application they need to use, without asking us to change some proxy we ru…
The experience of...
1) build a thing because it's immediately useful for a specific use-case, 2) someone reuses it for another use-case because it's already there and saves some work, 3) times passes 4) oops really important stuff now relies on this thing in ways that weren't originally intended
... seems like a common pattern (see: JWT succeeding as the format for interoperable tokens by dint of just being around).
In this case, it seems like the pros are basically user-centered pros (`flyctl proxy`, existing tool interop, etc.) and the cons are basically fly-centered cons (state synchronization, maintaining the agent and making it work right).
The cons that do affect users (slowness, maybe they can't speak 51820/udp) seem _annoying_ but not deal-breaking for a lot of use cases. If the slowness persists over a long time it will be interesting to see how users opt to route around it (architect applications / processes to not rely on this channel).
Earlier quoted context omitted.
Question: ultimately all the packets are actually being sent via a Golang net.UDPConn right? ie you're simulating raw network packets by wrapping them in UDP packets, then running TCP in Golang over those wrapped packets?
That is effectively what we're doing, yeah.
Fly.io's blog posts are incredible, they really seem to really enjoy what they do and want to share what they've made with everyone else. I love them for that. I wish that more companies could be like this and skip the corporate BS, it shows that they really have something outstanding to offer.
I think their super power here is employing a renowned security expert who is an incredibly good communicator! (And happens to be a top HN contributor)