Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The curious case of a memory leak in a Zig program

iamkroot.github.io

Post reply on HN
Print view View original

61–70 of 80 posts

Re: The curious case of a memory leak in a Zig program

#61

Post by dcminter »

2h wrote: post #59
throwawaymaths wrote: post #55

Earlier quoted context omitted.

there's a lot of sunlight between the class of things that are idiomatic and the class of things that are anti-patterns; and there are likely things that are idiomatic but still anti-patterns (yes, you can do this, and if you did this it would look like this, and it causes no regression in this particular case, but don't get in the habit of doing it this way because it can cause a hard-to-spot regression in the gener…

> anti-patterns I agree with all that, but patternic is not a word.

To you. English has a variety of suffixes that allow for exactly this kind of constructive generation of new words. Thus 'patternic' is a word precisely because it has just been coined. Or do you only ever approve of "dictionary" words without pondering where they might come from?

Re: The curious case of a memory leak in a Zig program

#62

Post by krut-patel »

rntz wrote: post #52

> If you are hell-bent on using FixedBufferAllocator only and you want to avoid copies, there is a way. Using two buffers (and separate allocators backed by them), it is possible to keep swapping between them after every iteration. I found this bit lovely: the author has independently reinvented the core idea of semispace copying garbage collectors (see eg https://wingolog.org/archives/2022/12/10/a-simple-semi-space.…

And I am not the only one :) https://old.reddit.com/r/Zig/comments/11vbiv1/the_curious_ca...

Re: The curious case of a memory leak in a Zig program

#63

Post by jpcfl »

renox wrote: post #7
jpcfl wrote: post #5

Earlier quoted context omitted.

What an awful API design choice. It’s a stack allocator that leaks your memory if you don’t free in reverse order. Why would anybody ever want that behavior, let alone as the default?

It's not the 'default' it's the behaviour of this allocator. This makes this allocator fast, but it should clearly be named/described I agree.

Thanks. Yeah, that's my point. The naming doesn't make it obvious.

Personally, I would have called this a StackAllocator, that way the alloc/free order requirement is obvious. I would have made the default behavior to 'panic()' if you don't satisfy the precondition of freeing the most recently allocated buffer.

If somebody really wanted to make free a no-op, I'd offer a feature flag to turn that on.

Re: The curious case of a memory leak in a Zig program

#64

Post by smegsicle »

dcminter wrote: post #36
2h wrote: post #34

Earlier quoted context omitted.

> anti-patternic patternic is not a word.

Why? All words are coined at some point.

it's an idiotic construction— the prefix may be originally greek, but has taken a specific connotation in english, while 'pattern' is from french, and the '-y' suffix is more common in english anyway, so the correct form for the idea would of course be anti-pattern–y

Re: The curious case of a memory leak in a Zig program

#65

Post by krut-patel »

eatonphil wrote: post #32
krut-patel wrote: post #17

Earlier quoted context omitted.

Thanks for the pointers! > use a GeneralPurposeAllocator and setting "enable_memory_limit" and "requested_memory_limit" Interesting! I hadn't looked at GeneralPurposeAllocator too closely, but yes these seem like the right way to do things instead of abusing FixedBufferAllocator as I did. > If the purpose is to "only use the stack"... Not really, I just had to decide on some arbitrary upper bound on the mem usage, an…

TigerBeetle writes to disk for long-term storage. Data over time is the part you can't fit into memory (eventually). :)

> TigerBeetle writes to disk for long-term storage

But how does it determine when it should write to disk? Does every write to a potentially OOM operation get preceeded by a check? Take the case of a HashAggregate. The DB clearly cannot know at compile time how many unique keys will be present in the hashtable; it needs to resize at runtime. So does that mean all the hashtables are still using some form of Bump/Arena allocators backed by the pre-allocated memory?

Maybe I should just read the source code :)

Re: The curious case of a memory leak in a Zig program

#66

Post by eatonphil »

krut-patel wrote: post #65
eatonphil wrote: post #32

Earlier quoted context omitted.

TigerBeetle writes to disk for long-term storage. Data over time is the part you can't fit into memory (eventually). :)

> TigerBeetle writes to disk for long-term storage But how does it determine when it should write to disk? Does every write to a potentially OOM operation get preceeded by a check? Take the case of a HashAggregate. The DB clearly cannot know at compile time how many unique keys will be present in the hashtable; it needs to resize at runtime. So does that mean all the hashtables are still using some form of Bump/Arena…

> But how does it determine when it should write to disk?

You write fixed sized number of key-value pairs to the file at a time. This is how LSM trees work, you chunk your data up into N sorted keys per chunk. I don't myself understand all the specifics but this is the gist.

> Does every write to a potentially OOM operation get preceeded by a check?

If you allocate memory upfront and don't allocate any more memory, you can't OOM after the initial allocation. That's what TigerBeetle does.

Zig has some nice standard library containers for adding items while asserting that there's capacity. If we miscalculate, it is caught during tests because assertions fail.

Re: The curious case of a memory leak in a Zig program

#67

Post by jesse__ »

Dwedit wrote: post #60

Perhaps that allocator could print a warning message if you're not deleting the last element (when built in debug mode). That would make it a lot more clear how that kind of allocator should be used.

I use this pattern a lot and my allocators print a huge warning when they detect this kind of leak. +1 for this suggestion. It's a hard bug to track down in nontrivial code.

Re: The curious case of a memory leak in a Zig program

#69

Post by charcircuit »

kps wrote: post #56
charcircuit wrote: post #27

Earlier quoted context omitted.

>Is there a better way? Just let the kernel handle it. The virtual memory and mapped memory abstraction the kernel has makes your program's implementation simpler.

Zig is a low-level language. You might not have an MMU. You might not have a kernel. You might be the kernel.

Maybe if we went back a few decades. But in 2023 having access to a MMU and a kernel is normal.

Re: The curious case of a memory leak in a Zig program

#70

Post by [deleted] »

dcminter wrote: post #61
2h wrote: post #59

Earlier quoted context omitted.

> anti-patterns I agree with all that, but patternic is not a word.

To you. English has a variety of suffixes that allow for exactly this kind of constructive generation of new words. Thus 'patternic' is a word precisely because it has just been coined. Or do you only ever approve of "dictionary" words without pondering where they might come from?

[deleted]
Post reply on HN