Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything old is new again: memory optimization

nibblestew.blogspot.com

Post reply on HN
Print view View original

61–70 of 168 posts

Re: Everything old is new again: memory optimization

#61

Post by muskstinks »

Capricorn2481 wrote: post #55
muskstinks wrote: post #21

I'm always confused as hell how little insight we have in memory consumption. I look at memory profiles of rnomal apps and often think "what is burning that memory". Modern compression works so well, whats happening? Open your taskmaster and look through apps and you might ask yourself this. For example (lets ignore chrome, ms teams and all the other bloat) sublime consumes 200mb. I have 4 text files open. What is it…

> sublime consumes 200mb. I have 4 text files open. What is it doing? Huh? Sublime Text? I have like 100 files open and it uses 12mb. Sublime is extremely lean. Do you have plugins installed?

I do not have plugins installed and i have only a handful of files open on macos.

Memroy statistics says 200mb and a peak of 750mb in the past (for whatever reason)

Re: Everything old is new again: memory optimization

#62

Post by tombert »

I've been rewriting a lot of my stuff in Rust to save memory.

Rust is high-level enough to still be fun for me (tokio gives me most of the concurrency goodies I like), but the memory usage is often like 1/10th or less compared to what I would write in Clojure.

Even though I love me some lisp, pretty much all my Clojure utilities are in Rust land now.

Re: Everything old is new again: memory optimization

#63

Post by muskstinks »

pjc50 wrote: post #49
muskstinks wrote: post #21

I'm always confused as hell how little insight we have in memory consumption. I look at memory profiles of rnomal apps and often think "what is burning that memory". Modern compression works so well, whats happening? Open your taskmaster and look through apps and you might ask yourself this. For example (lets ignore chrome, ms teams and all the other bloat) sublime consumes 200mb. I have 4 text files open. What is it…

https://learn.microsoft.com/en-us/sysinternals/downloads/vmm... for an empty sublime text window gives me: - 100MB 'image' (ie executable code; the executable itself plus all the OS libraries loaded.) - 40MB heap - 50MB "mapped file", mostly fonts opened with mmap() or the windows equivalent - 45MB stack (each thread gets 2MB) - 40MB "shareable" (no idea) - 5MB "unusable" (appears to be address space that's not usabl…

Tx for the breakdown. I will play around with it later on my windows machine.

But isn't it crazy how we throw out so much memory just because of random buffers? It feels wrong to me

Re: Everything old is new again: memory optimization

#64

Post by Filligree »

dgb23 wrote: post #53
VorpalWay wrote: post #17

Earlier quoted context omitted.

> A fairer comparison would be to stream the file in C++ as well and maintain internal state for the count. Wouldn't memory mapping the data in Python be the more fair comparison? If the language doesn't support that, then this seems to absolutely be a fair comparison. > For most people that would be the first/naive approach as well when they programmed something like this I think. I disagree, my mind immediately goe…

I'd make the bet that "most people" (who can program) would not think of mmap, but either about streaming or would even just load the whole thing into memory. Ask a bunch of coding agents and they will give you these two versions, which means it's likely that the LLMs have seen these way more often than the mmap version. Both Opus and GPT even pushed back when I asked for mmap, both said it would "add complexity".

It does add complexity, and the optimal solution is probably not to use it. Consider what happens if a 4kB page has only a single unique word in it—you’d still need to load it to memory to read the string, it just isn’t accounted against your process (maybe).

I would have expected something like this:

- Scan the file serially.

- For each word, find and increment a hash table entry.

- Sort and print.

In theory, technically, this does require slightly more memory—but it’s a tiny amount more; just a copy of each unique word, and if this is natural language then there aren’t very many. Meanwhile, OOP’s approach massively pressures the page cache once you get to the “print” step, which is going to be the bulk of the runtime.

It’s not even a full copy of each unique word, actually, because you’re trading it off against the size of the string pointers. That’s… sixteen bytes minimum. A lot of words are smaller than that.

Re: Everything old is new again: memory optimization

#65

Post by wat10000 »

pjc50 wrote: post #49
muskstinks wrote: post #21

I'm always confused as hell how little insight we have in memory consumption. I look at memory profiles of rnomal apps and often think "what is burning that memory". Modern compression works so well, whats happening? Open your taskmaster and look through apps and you might ask yourself this. For example (lets ignore chrome, ms teams and all the other bloat) sublime consumes 200mb. I have 4 text files open. What is it…

https://learn.microsoft.com/en-us/sysinternals/downloads/vmm... for an empty sublime text window gives me: - 100MB 'image' (ie executable code; the executable itself plus all the OS libraries loaded.) - 40MB heap - 50MB "mapped file", mostly fonts opened with mmap() or the windows equivalent - 45MB stack (each thread gets 2MB) - 40MB "shareable" (no idea) - 5MB "unusable" (appears to be address space that's not usabl…

Turning these numbers into "memory consumption" gets complicated to the point of being intractable.

The portions that are allocated but not yet used might just be page table entries with no backing memory, making them free. Except for the memory tracking the page table entries. Almost free....

A lot of "image" will be mmapped and clean. Anything you don't actually use from that will be similarly freeish. Anything that's constantly needed will use memory. Except if it's mapped into multiple processes, then it's needed but responsibility is spread out. How do you count an app's memory usage when there's a big chunk of code that needs to sit in RAM as long as any of a dozen processes are running? How do you count code that might be used sometime in the next few minutes or might not be depending on what the user does?

Re: Everything old is new again: memory optimization

#66

Post by Capricorn2481 »

muskstinks wrote: post #61
Capricorn2481 wrote: post #55

Earlier quoted context omitted.

> sublime consumes 200mb. I have 4 text files open. What is it doing? Huh? Sublime Text? I have like 100 files open and it uses 12mb. Sublime is extremely lean. Do you have plugins installed?

I do not have plugins installed and i have only a handful of files open on macos. Memroy statistics says 200mb and a peak of 750mb in the past (for whatever reason)

Is that in Task Manager, or is that not a reliable place to look for these statistics?

Edit: From what I can tell, Sublime is allocated 100mb of virtual memory even if it's only using about 10mb in practice.

Re: Everything old is new again: memory optimization

#67

Post by adrian_b »

pjc50 wrote: post #52
gwbas1c wrote: post #43

A lot of frameworks that use variants of "mark and sweep" garbage collection instead of automatic reference counting are built with the assumption that RAM is cheap and CPU cycles aren't, so they are highly optimized CPU-wise, but otherwise are RAM inefficient. I wonder if frameworks like dotnet or JVM will introduce reference counting as a way to lower the RAM footprint?

Reference counting in multithreaded systems is much more expensive than it sounds because of the synchronization overhead. I don't see it coming back. I don't think it saves massive amounts of memory, either, especially given my observation with vmmap upthread that in many cases the code itself is a dominant part of the (virtual) memory usage.

Incrementing or decrementing a shared counter is done with an atomic instruction, not with a locked critical section.

This has negligible overhead in most cases. For instance, if the shared counter is already in some cache memory the overhead is smaller than a normal non-atomic access to the main memory. The intrinsic overhead of an atomic instruction is typically about the same as that of a simple memory access to data that is stored in the L3 cache memory, e.g. of the order of 10 nanoseconds at most.

Moreover, many memory allocators use separate per-core memory heaps, so they avoid any accesses to shared memory that need atomic instructions or locking, except in the rare occasions when they interact with the operating system.

Re: Everything old is new again: memory optimization

#68

Post by inetknght »

muskstinks wrote: post #21

I'm always confused as hell how little insight we have in memory consumption. I look at memory profiles of rnomal apps and often think "what is burning that memory". Modern compression works so well, whats happening? Open your taskmaster and look through apps and you might ask yourself this. For example (lets ignore chrome, ms teams and all the other bloat) sublime consumes 200mb. I have 4 text files open. What is it…

> I look at memory profiles of rnomal apps and often think "what is burning that memory".

As a corrolary to this: I look at CPU utilization graphs. Programs are completely idle. "What is burning all that CPU?!"

I remember using a computer with RAM measured in two-digit amounts of MiB. CPU measured in low hundreds of MHz. It felt just as fast -- sometimes faster -- as modern computers. Where is all of that extra RAM being used?! Where is all of that extra performance going?! There's no need for it!

Re: Everything old is new again: memory optimization

#69

Post by HPsquared »

inetknght wrote: post #68
muskstinks wrote: post #21

I'm always confused as hell how little insight we have in memory consumption. I look at memory profiles of rnomal apps and often think "what is burning that memory". Modern compression works so well, whats happening? Open your taskmaster and look through apps and you might ask yourself this. For example (lets ignore chrome, ms teams and all the other bloat) sublime consumes 200mb. I have 4 text files open. What is it…

> I look at memory profiles of rnomal apps and often think "what is burning that memory". As a corrolary to this: I look at CPU utilization graphs. Programs are completely idle. "What is burning all that CPU?!" I remember using a computer with RAM measured in two-digit amounts of MiB. CPU measured in low hundreds of MHz. It felt just as fast -- sometimes faster -- as modern computers. Where is all of that extra RAM b…

Work expands to fill the available time. This applies to CPU time just as it does to project management.

Re: Everything old is new again: memory optimization

#70

Post by ElectronCharge »

MrBuddyCasino wrote: post #26
biorach wrote: post #4

"copyright infringement factories"

I don't know how anyone can call the most amazing invention in computer science of the last 20 years "copyright infringement factories". We went from the ST:NG ship computer being futuristic tech to "we kinda have this now". Its like calling cars "air pollution factories", as if that was their only purpose and use. A fundamentally anti-civilisational mindset.

LLMs are amazing technology. It's crazy to interact with something that knows a lot about effectively everything that's ever been written, as well as mimicking human cognition to a large degree.

What LLMs are NOT is intelligent in the same way as a human, which is to say they are not "AGI". They may be loosely AGI-equivalent for certain tasks, software development being the poster child. LLMs have no equivalent of "judgement", and they lie ("hallucinate") with impunity if they don't know the answer. Even with coding, they'll often do the wrong thing, such as writing tests that don't test anything.

It seems likely that LLMs will be one component of a truly conscious AI (AGI+), in the same way our subconscious facility to form sentences is part of our intelligence. We'll see how quickly the other pieces arrive, if ever.

Post reply on HN