Earlier quoted context omitted.
Yes.
Will we ever see the day where computers over the Internet talk to each other over something other than UDP/TCP over IPv4/IPv6?
Given Google and Apple's efforts to de-ossify the Internet, maybe. But no time soon.
We need a replacement for TCP in the datacenter [pdf]
311–320 of 329 posts
Re: We need a replacement for TCP in the datacenter [pdf]
#312FWIU, barring FTL; superluminal communication breakthroughs and controls, Deep Space Networking needs a new TCP, as well:
From https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_t... :
void tcp_retransmit_timer(struct sock *sk) {
/* Increase the timeout each time we retransmit. Note that
* we do not increase the rtt estimate. rto is initialized
* from rtt, but increases here. Jacobson (SIGCOMM 88) suggests
* that doubling rto each time is the least we can get away with.
* In KA9Q, Karn uses this for the first few times, and then
* goes to quadratic. netBSD doubles, but only goes up to *64,
* and clamps at 1 to 64 sec afterwards. Note that 120 sec is
* defined in the protocol as the maximum possible RTT. I guess
* we'll have to use something other than TCP to talk to the
* University of Mars.
*
* PAWS allows us longer timeouts and large windows, so once
* implemented ftp to mars will work nicely. We will have to fix
* the 120 second clamps though!
*/
Re: We need a replacement for TCP in the datacenter [pdf]
#313Earlier quoted context omitted.
I think it would be an overkill to use a networking protocol to connect exactly two devices. Plus if you have something specific to video stream transfer you could maybe do some optimization specific to that use case, although I can't think of any at the moment.
Excepting HDMI the parents examples are all networks with more than one peer. Thunderbolt and USB3 can both have arbitrary trees of nodes.
The parent comment was specifically wondering about HDMI, the other examples are given saying that those have some reason to exist, while he didn't see any reason for having HDMI instead of recycling some other protocol.
Re: We need a replacement for TCP in the datacenter [pdf]
#314Earlier quoted context omitted.
Fibre channel is dead, long live fibre channel. I agree a lot of things are on TCP, but I don't think its a massive problem, unless you are running close to the limit of your core network. And one solution to that is to upgrade your core network.... Failing that, implementing some load balancing/partitioning systems to make sure data-processing affinity is best matched. This the better solution, because it yields oth…
What is supposed to be a good scheduler? Genuine question as I (as a software dev) have no clue how modern DCs are build in detail.
Its very much down to your workload and how you want it to work.
Short answer: k8s/fargate/ECS/batch will do what most people want. Personally I'd steer clear of k8s until you 100% need that overhead. Managed services are ok.
Long answer:
K8s has a whole bunch of scheduling algorithms but its a jack of all trades, and only really deals with very shallow dependencies (there are plugins but I've not used them). For Like most things there are tradeoffs. Do you want to prioritise resiliency/uptime over efficiency? do you want to have batch processing? do you want it to manage complex dependencies (ie service x needs 15 other services to run first, which then need 40 other services) are you running on unreliable infra (ie spot instances to save money) do you need to partition services based on security? are you running over many datacentres and need the concept of affinity?
More detail:
The scheduler/dispatcher is almost always deigned to run the specific types of workload that you as a company run. The caveat being that this only applies if you are running multiple datacentres/regions with thousands of machines. Google have a need to run both realtime and batch processing. But as they have millions of servers, making sure that all machines are running at 100% utilisation (or as close to it as practical) is worth hundreds of millions. Its the same with facebook.
Netflix I guess has a slightly different setup as they are network IO orientated, so they are all about data affinity, and cache modelling. For them making sure that the edge serves as much as possible is a real cost saving, as bulk bandwidth transfer is expensive. The rest is all machine learning and transcoding I suspect (but that's a guess)
Re: We need a replacement for TCP in the datacenter [pdf]
#315Earlier quoted context omitted.
You're missing the point that maybe, just maybe, I'm part of a team that looks after >5 million servers. You might also divine that while TCP can be a problem, a bigger problem is data affinity. Shuttling data from a next door rack costs less than one that's in the next door hall, and significantly less than the datacentre over. With each internal hop, the risk of congestion increases. You might also divine that chan…
> quick maths: a faster top of rack switch is possibly the same cost as 5 days engineering wage for a mid level google employee. How many new switches do you think you could buy with the engineering effort required to port everything to the new protocol, and have it stable and observable? So your 5M machines / 40 in the best case of all 1U boxes is 125K TOR-switch-SWE-week-equivalents / 52 weeks in a year which comes…
> explicitly spending on SWE-hours instead of Cisco.
I strongly doubt that TOR switches are cisco
Re: We need a replacement for TCP in the datacenter [pdf]
#316Earlier quoted context omitted.
What is supposed to be a good scheduler? Genuine question as I (as a software dev) have no clue how modern DCs are build in detail.
Its very much down to your workload and how you want it to work. Short answer: k8s/fargate/ECS/batch will do what most people want. Personally I'd steer clear of k8s until you 100% need that overhead. Managed services are ok. Long answer: K8s has a whole bunch of scheduling algorithms but its a jack of all trades, and only really deals with very shallow dependencies (there are plugins but I've not used them). For Lik…
Thanks for the reply!
Not really an answer to the scheduler question, but at least it mirrors some of my experience.
That K8s is something to avoid, and that it does not scale, is a known (at least to me).
But that doesn't answer what people would put on the metal when building DCs…
I was not asking out of the perspective of an end-user. I was asking about (large) DC scale infra. (As dev I know the end-user stuff).
As I see it: You can build your own stuff from scratch (which is not realistic in most cases I guess), or you can use OpenStack or Mesos. There are no more alternatives at the moment I think, and it's unlikely that someone comes up with something new. OTOHS that's OK. A lot of people will never need to build their own DC(s). For smaller setups (say one or two racks) there are more options of course. (You could run for example Proxmox and maybe something on top).
Re: We need a replacement for TCP in the datacenter [pdf]
#317Earlier quoted context omitted.
That’s interesting. And I’m sure they have some very knowledgable people working for them who may(/will) know things I don’t. That being said, it wouldn’t surprise me if they were pushing 17G of UDP on 100G transports. Probably with some pretty high-end/expensive network hardware with huge buffers. I.e you can do it if you’ve got the money, but I bet TCP would still have better raw throughput.
Yep, 100G switches are common nowadays since the cost has come down so much, and you can easily carve a port to 4x10G, 4x25G, and 40G. In financial trading you tend to avoid switches with huge buffer as that comes to a huge cost in latency. For example, 2 megabytes of buffer is 1.68ms of latency on a 10G switch which is an eon in trading. Most opt for cut-through switches with shallow buffers measured in 100s of nano…
That is a really good point that I hadn’t considered. Presumably this comes at the risk of dropped packets if the upstream link becomes saturated? Does one just size the links accordingly to avoid that?
Re: We need a replacement for TCP in the datacenter [pdf]
#318Earlier quoted context omitted.
Google is using remote memory accesses rather than TCP for at least some classes of traffic (e.g. a caching system). They've been publishing details about how it all works too. Also, they have a transport (Pony express) developed specifically for RPCs, rather than byte streams or datagrams. Links: https://research.google/pubs/pub51341/ , https://research.google/pubs/pub50590/ , https://research.google/pubs/pub48630/…
Can someone ELI5 how remote memory access works?
I could be wrong but I believe they have a unified address space. There’s dedicated hardware that then owns a given memory range. On an access it will fetch it from the remote location matching that address on demand and store it in real memory in space allocated to it. Presumably it evicts stuff if there’s insufficient memory. Once the memory is brought over either a virtual address range is remapped to point to main memory or the ASIC just has a TLB itself.
This is pure speculation based on seeing the word ASIC in one of the summaries but it seems like it could be reasonable.
Re: We need a replacement for TCP in the datacenter [pdf]
#319Earlier quoted context omitted.
Why don’t you actually build gasp a prototype before asking for money
yeah, thank you for the kind comments about not needing money (aka: my time as no value) and asking me to build the prototype with irony. As I said, the project was started a few years back, and since I did not have the time to work on it, maybe it means my life does not give me the time and money to build this on the side. But I'll always find it funny how half of the people go "you need to have solid theory proofs…
Doubtful anyone is going to throw money at you for some random idea, that you won’t even invest some of your free time on. You clearly don’t value it, so no one else will.