Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

cloud.google.com

Post reply on HN
Print view View original

141–150 of 158 posts

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#141

Post by jabl »

jeffbee wrote: post #122
jabl wrote: post #119

Earlier quoted context omitted.

There seems to be quite a lot of overlap between Falcon and what the Ultra Ethernet Consortium is ostensibly working on. As well as Amazon's Scalable Reliable Datagram (SRD) thing. All of them, in a way, are about addressing deficiencies in RoCEv2 for large scale latency sensitive networking that you see in HPC and DL training. But none of these are things you can buy today. Well, there's InfiniBand, but if you're we…

I think it's a mistake to assume that an organization this large and sophisticated simply failed to try RoCE. They probably gave it a go but it didn't work out for some technical or economic reason.

I'm sure all the big players in this space have tried RoCE, and found it wanting. Hence all these new approaches.

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#142

Post by em-bee »

tsegratis wrote: post #118
thereisnospork wrote: post #86

Earlier quoted context omitted.

>You can't route by MAC-address because it's effectively random. You'd have to store the port number for every device separately. This works fine at LAN scale, but not for the whole Internet. Not that I see any advantages to the approach but it's almost workable(?), if a little silly, at internet scale: If every device had a 64byte ID, guesstimating 10billion people * 100 devices/head gets us a 'measly' 64TB of stora…

Nice maths. Would each router then hold 64TB and doing a lookup per request in that volume of data would be slow Question: how does dns lookup differ from MAC lookup. Why is domain name lookup feasible, but not MAC?

the router needs to remember where each address goes. with MAC addresses being random, there is no shortcut. DNS is distributed and you look it up one subdomain level at a time, and that can be cached. same for IP, the router only needs to store the subnet for each destination, not all ip addresses.

a central lookup database for mac addresses (which could be distributed by having separate servers for a segment of the address space) doesn't make much sense because the distance of a server to the location of the device is to great and would make updates expensive.

so the router has to remember each address used. but at least it would not have to store all addresses in existence. actually, i think the storage needs are similar to those for NAT. well, except backbone routers which have to store a lot more.

the actual problem is the initial discovery of a MAC address. where does the routing information for a MAC address come from?

you need some peer finding protocols like DHT, and those are slower.

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#143

Post by elcritch »

mintplant wrote: post #17

Oh, I was hoping this would be something built more directly over Ethernet, rather than on top of UDP/IP (if I'm understanding the layer diagram correctly). I've been working with Ethernet devices a lot lately, using the network as a communication bus, essentially. I find that there's a lot of complexity that we simply don't need: ARP, DHCP, DNS... So many points of failure. We know all the devices on our LAN and the…

Move to ipv6 and drop ARP and DHCP, which eliminates a good chunk of the older cruft. IPv6 builds everything on top of multicast support which is required for IPv6 switches/routers. It's so much cleaner. You could even avoid DNS if you really want.

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#144

Post by pclmulqdq »

MichaelMoser123 wrote: post #103
mintplant wrote: post #17

Oh, I was hoping this would be something built more directly over Ethernet, rather than on top of UDP/IP (if I'm understanding the layer diagram correctly). I've been working with Ethernet devices a lot lately, using the network as a communication bus, essentially. I find that there's a lot of complexity that we simply don't need: ARP, DHCP, DNS... So many points of failure. We know all the devices on our LAN and the…

There are systems like this: Fibre Channel has it's own data link layer, actually they do reliability at the data link layer! I think InfiniBand is similar to this respect. Actually it's interesting that google didn't choose any of these, for their high bandwidth storage needs. They have the money to do their own thing, but why should they?

Infiniband and most other specialized protocols have data-link-layer reliable messaging. The downside of this is that a congested switch cannot drop packets, so you end up backpressuring your network to death unless the people writing software really know what they're doing. Google was not able to make this work at scale.

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#145

Post by speed_spread »

vore wrote: post #68
speed_spread wrote: post #57

Earlier quoted context omitted.

And QUIC/HTTP3 was invented by Google to deliver more ads, faster. Yay.

This is like saying planes were invented to deliver you to more businesses, faster.

At least I enjoy taking the plane and I decide when to take it. Planes serve a purpose. Adtech is just parasitic. An Internet without ads wouldn't need QUIC.

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#146

Post by bluGill »

Grazester wrote: post #135
bluGill wrote: post #134

Google has lost trust with so many of the things they have released in the past becoming unsupported and obsolete.

If this is an open standard then anyone who uses it and wishes to support it can do so no, or am I mistaken?

Maybe, but often in practice if the amount of supported needed gets high. It needs some very involved people to keep support going, otherwise it will bit rot and fail to function after a time.

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#147

Post by Dylan16807 »

tuetuopay wrote: post #140
Dylan16807 wrote: post #37

Earlier quoted context omitted.

1. In a single data center scale computing system, though? 2. Use some bytes outside of the header? 3. I get the impression this needs hardware to really use well anyway.

1. Especially in the datacenter. When you add VMs to the mix you get LOADS of devices to address. Add on top that a single device has multiple connections (management, internet, storage, etc), you'd run out of capacity almost instantly. 2. The point is to find some bytes that are constant for a given logical stream of related packets. Taking bytes outside of the header means taking bytes from the payload, that is by…

> Especially in the datacenter.

Okay, but my point there was saying "Google has millions of servers" isn't relevant, we're not looking at the entire company.

Even with a few addresses per VM, how many racks do you need to put into the same shared-compute mass? One data center is the upper limit, but it doesn't have to be the entire data center.

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#148

Post by wmf »

Zandikar wrote: post #139
kortilla wrote: post #78

Earlier quoted context omitted.

2. Don’t do this. It causes reordering between flows and software doesn’t like out of order packets.

I get the impression that if this comment chain continues we're just gonna reinvent UDP

Exactly. QUIC and Falcon use UDP because they basically have to. You can't avoid it.

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#149

Post by DoingIsLearning »

deely3 wrote: post #128
DoingIsLearning wrote: post #85

Earlier quoted context omitted.

Same thing goes for their single monolith repository philosophy.

Could you elaborate on this please?

There is a fairly amount of technical writing describing Google's use of a single repository across a lot of enginering teams.

This works for them at their scale with quite a lot of internal tooling specifically to make it work.

However many people defend it as _the_ solution whereas it is a way with trade offs like most solutions.

Re: Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem

#150

Post by MichaelZuo »

NavinF wrote: post #113
mintplant wrote: post #17

Oh, I was hoping this would be something built more directly over Ethernet, rather than on top of UDP/IP (if I'm understanding the layer diagram correctly). I've been working with Ethernet devices a lot lately, using the network as a communication bus, essentially. I find that there's a lot of complexity that we simply don't need: ARP, DHCP, DNS... So many points of failure. We know all the devices on our LAN and the…

If you're flexible enough to forgo UDP/IP, why not use infiniband instead of ethernet? That gets rid of all the complexity you mentioned but still gives you ordered streams

I agree, it's hard to see what real world use case would benefit from dumping UDP/IP while holding on to Ethernet, and not moving over to an Infiniband or similar solution. That also can't be satisfied by one of the existing specialized solutions that another user mentioned, such as EtherCAT.
Post reply on HN