Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

github.com

Post reply on HN
Print view View original

41–50 of 51 posts

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#41

Post by fulafel »

nly wrote: post #30
fulafel wrote: post #19

Earlier quoted context omitted.

Yes, TCP is pretty hungry for buffers. The bandwidth*delay product can eat gigs of memory on a server. You have to be ready to retransmit anything that's in flight / haven't received the ack for yet.

The bandwidth delay product for a 10Gbps stream for a 300ms RTT theoretically only requires ~384MB One option is just to simply keep buffers small and fixed and disconnect blocked clients on write() after some timeout

We're up to hundreds of gbps per server, have been for some years now. Eg 400 gbps uses a lot even with much smaller avg rtt. That's not going ng to be one stream of course, but a zillion smaller streams still add up to the same reqs.

This is far from little embedded device territory of course. But still, latest wifi is closer to 10 than 1 gbps already.

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#42

Post by Veserv »

fulafel wrote: post #41
nly wrote: post #30

Earlier quoted context omitted.

The bandwidth delay product for a 10Gbps stream for a 300ms RTT theoretically only requires ~384MB One option is just to simply keep buffers small and fixed and disconnect blocked clients on write() after some timeout

We're up to hundreds of gbps per server, have been for some years now. Eg 400 gbps uses a lot even with much smaller avg rtt. That's not going ng to be one stream of course, but a zillion smaller streams still add up to the same reqs. This is far from little embedded device territory of course. But still, latest wifi is closer to 10 than 1 gbps already.

I do not understand the point you are trying to make. The person you replied to showed how to evaluate it with simple math.

400 Gb/s is 50 GB/s. RTT of 300 ms would only require 15 GB of buffers. That would not even run a regular old laptop out of memory let alone a server driving 400 Gb/s of traffic. That would be single-digit percents to possibly even sub-percent amounts of memory on such a server.

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#43

Post by notepad0x90 »

preisschild wrote: post #39
notepad0x90 wrote: post #37

Earlier quoted context omitted.

that's great, but when you have a networking issue, you have to deal with two stacks for troubleshooting. it would be much less effort to use just ipv4. You're not paying for IPv4 addresses I'm sure, so did ipv6 solve anything for you? This is why i meant by zealots keeping it alive. you use ipv6 for the principle of it, but tech is suppose to solve problems, not facilitate ideologies.

> it would be much less effort to use just ipv4. Or just use IPv6-only. Thats what I do. Legacy ipv4 only services can be reached via DNS64/NAT64

But that's slow, and it's one more thing you have to setup and that could fail. What is the benefit to me if I used ipv6 and those nat services? what if I run into a service that blocks those nat IPs because they generate lots of noise/spam since they allow anyone to proxy through their IP? Not only does it not benefit me, if this was commercial activity I was engaging in, it could lead to serious loss of money.

At the risk of more downvotes, I again ask, why? am I supposed to endure all this trouble so that IPv4 is cheaper for some corporation? even then, we've hit the plateau as far as end user adaption goes. and I'll continue to argue that using IPv6 is a serious security risk, if you just flip it on and forget about it. you have to actually learn how it works, and secure it properly. These are precious minutes of people's lives we're talking about, for the sake of some techno ideology. The billions and billions spent on ipv4 and no one in 2026 is claiming ipv4 shortage will cause outages anytime within the next decade or two.

My suggestion is to come up with a solution that doesn't require any changes to the IP stack or layer3 by end users. CGNAT is one approach, but there are spare fields in the IPv4 Header that could be used to indicate some other address extension to ipv4 (not an entire freaking replacement of the stack), or just a minor addition/octet that will solve the problem for the next century or so by adding an "area code" like value (ASN?).

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#44

Post by notepad0x90 »

sedatk wrote: post #14
notepad0x90 wrote: post #10

Earlier quoted context omitted.

It's just not worth it. the only thing keeping it alive is people being overly zealous over it. if the cost to implement is measured as '1', the cost to administer it is like '50'.

> the only thing keeping it alive is people being overly zealous over it Hard disagree. It turned out to be great for mobile connectivity and IoT (Matter + Thread). > the cost to administer it is like '50'. I'm not sure if that's true. It feels like less work to me because you don't need to worry about NAT or DHCP as much as you need with IPv4.

To start with it requires support v4 as a separate network, at least for internal networks, since many devices don't support ipv6 (I have several AP's, IoT devices,etc.. bought in recent years that are like that). Then the v4->v6 nat/gateway/proxy approaches don't work well for cases where reliability and performance are critical. You mentioned NAT, but lack of NAT means you have to configure firewall rules, many get a public ip by their ISP directly to the first device that connects to the ISP modem,exposing their device directly to the internet. Others need to do expose a lan service on devices (port forwarding) which is more painful with v6. DHCP works very simply, v6 addressing can be made simply too (especially with the v4 patterned addressing - forgot its name) but you have multiple types of v6 addresses, the only way to easily access resources with v6 is to use host names. with v4 you can just type an IP easily and access a resource. Same with troubleshooting, it's more painful because it is more complex, and it requires more learning by users, and if you have dual stack, that doesn't add to the management/admin burden, it multiplies it. It's easier to tcpdump and troubleshoot arp, dhcp and routing with v4 than it is ND,RA,anycast,linklocal,etc.. with v6.

For mobile connectivity, ipv4 works smoothly as well in my experience, but I don't know about your use case to form an opinion. I don't doubt IPv6 makes some things much easier to solve than ipv4. I am also not dismissing IPv6 as a pointless protocol, it does indeed solve lots of problems, but the problem it solves is largely for network administrators, even then you won't find a private network in a cloud provider with v6, for good reason too.

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#45

Post by fulafel »

Veserv wrote: post #42
fulafel wrote: post #41

Earlier quoted context omitted.

We're up to hundreds of gbps per server, have been for some years now. Eg 400 gbps uses a lot even with much smaller avg rtt. That's not going ng to be one stream of course, but a zillion smaller streams still add up to the same reqs. This is far from little embedded device territory of course. But still, latest wifi is closer to 10 than 1 gbps already.

I do not understand the point you are trying to make. The person you replied to showed how to evaluate it with simple math. 400 Gb/s is 50 GB/s. RTT of 300 ms would only require 15 GB of buffers. That would not even run a regular old laptop out of memory let alone a server driving 400 Gb/s of traffic. That would be single-digit percents to possibly even sub-percent amounts of memory on such a server.

I introduced the concept of bandwidth * delay product to the conversation...

The question was about why use dynamic allocation. In this branch of the thread we ere discussing the question "Are there TCP/IP stacks out there in common use that are allocating memory all the time?"

We'd not be happy to see the server or laptop statically reserving this worst-case amount of memory for TCP buffers, when it's not infact slinging around max nr of tcp connections, each with worst-case bandwidth*delay product. Nor would be happy if the laptop or server only supported little tcp windows that limit performance by capping the amount of data in-flight to a low number.

We are happier if the TCP stack dynamically allocates the memory as needed, just like we're happier with dynamic allocation on most other OS functions.

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#46

Post by 1718627440 »

bobmcnamara wrote: post #31
cpach wrote: post #27

Earlier quoted context omitted.

“GPL is toxic for embedded commercial software” Why is that?

Many bare metal or RTOS systems consist of a handful of statically linked programs (one or two bootloaders and the main application), many companies would rather find a non-GPL library rather than open up the rest of the system's code. Sometimes a system contains proprietary code that may not be open sourced as well.

In the embedded world you don't really sell software you sell devices with firmware. Unless the library OS is AGPL, it doesn't matter too much.

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#47

Post by CyberDildonics »

fulafel wrote: post #40
CyberDildonics wrote: post #34

Earlier quoted context omitted.

Needing memory doesn't have to mean allocating memory over and over. Memory allocation is expensive. If someone is doing that reusing memory is going to be by far the best optimization.

Well, allocating and freeing according to need is reusing. Modern TCP perf is not bottlenecked by that. There's pools of recycled buffers that grow and shrink according to load etc.

Well, allocating and freeing according to need is reusing

That's a twisted definition. It seems like you're playing around with terms, but allocating memory from a heap allocator is obviously what people are talking about with "dynamic memory allocation". Reusing memory that has already been grabbed from an allocator is not reallocating memory. If you have a buffer and it works you don't need to do anything to reuse it.

Modern TCP perf is not bottlenecked by that. There's pools of recycled buffers that grow and shrink according to load etc.

If anything is allocating memory from the heap in a hot loop it will be a bottleneck.

Reusing buffers is not allocating memory dynamically.

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#48

Post by tjoff »

1718627440 wrote: post #46
bobmcnamara wrote: post #31

Earlier quoted context omitted.

Many bare metal or RTOS systems consist of a handful of statically linked programs (one or two bootloaders and the main application), many companies would rather find a non-GPL library rather than open up the rest of the system's code. Sometimes a system contains proprietary code that may not be open sourced as well.

In the embedded world you don't really sell software you sell devices with firmware. Unless the library OS is AGPL, it doesn't matter too much.

Yes it matters a lot?

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#49

Post by fulafel »

CyberDildonics wrote: post #47
fulafel wrote: post #40

Earlier quoted context omitted.

Well, allocating and freeing according to need is reusing. Modern TCP perf is not bottlenecked by that. There's pools of recycled buffers that grow and shrink according to load etc.

Well, allocating and freeing according to need is reusing That's a twisted definition. It seems like you're playing around with terms, but allocating memory from a heap allocator is obviously what people are talking about with "dynamic memory allocation". Reusing memory that has already been grabbed from an allocator is not reallocating memory. If you have a buffer and it works you don't need to do anything to reuse…

Sorry but there's shades of gray between heap allocation and TCP specific free lists in TCP impls. It's not a black and white free list vs malloc API situation.

For example in Linux there are middle level abstraction layers in play as follows:

For the payload there's a per socket runway of memory to be used for example (sk_page_frag). Then, if there's a miss on that pool, instead of calling the malloc (or kmalloc in the case of Linux) API, it invokes the page allocator to get a bunch of VM pages in one go, which is again more efficient than using the generic heap API. The page allocation API will recycle recently freed large clusters of memory, and the page alloc is again backed by a CPU-local per cpu pageset etc. It's turtles all the way down.

For the metadata (sk_head) there's separate skbuff_head_cache that facilitates recycling for the generic socket metadata, which is again not a TCP specific thing but lower level than generic heap allocator, somewhere between a TCP free list and malloc in the tower of abstractions.

Re: WolfIP: Lightweight TCP/IP stack with no dynamic memory allocations

#50

Post by CyberDildonics »

fulafel wrote: post #49
CyberDildonics wrote: post #47

Earlier quoted context omitted.

Well, allocating and freeing according to need is reusing That's a twisted definition. It seems like you're playing around with terms, but allocating memory from a heap allocator is obviously what people are talking about with "dynamic memory allocation". Reusing memory that has already been grabbed from an allocator is not reallocating memory. If you have a buffer and it works you don't need to do anything to reuse…

Sorry but there's shades of gray between heap allocation and TCP specific free lists in TCP impls. It's not a black and white free list vs malloc API situation. For example in Linux there are middle level abstraction layers in play as follows: For the payload there's a per socket runway of memory to be used for example (sk_page_frag). Then, if there's a miss on that pool, instead of calling the malloc (or kmalloc in…

It's not a black and white free list vs malloc API situation.

It is in the sense that if finding a buffer that's the right length was just as slow as malloc (and free) then you would just use malloc.

Not only that but malloc is shared with the entire program and can do a lot of locking. On top of that there is the memory locality of using the same heap as the rest of the program.

If you just make your own heap there is a big difference between using the system allocator over and over and reusing local memory buffers for a specific thread and purpose.

What you're describing here the same thing, avoiding the global heap allocator.

Post reply on HN