Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything old is new again: memory optimization

nibblestew.blogspot.com

Post reply on HN
Print view View original

91–100 of 168 posts

Re: Everything old is new again: memory optimization

#91

Post by zahlman »

contravariant wrote: post #38
akx wrote: post #29

Earlier quoted context omitted.

This doesn't do the same thing though, since it's not Unicode aware. >>> 'x\u2009 a'.split() ['x', 'a'] # incorrect; in bytes mode, `\S` doesn't know about unicode whitespace >>> list(re.finditer(br'\S+', 'x\u2009 a'.encode())) [ , ] # correct, in unicode mode >>> list(re.finditer(r'\S+', 'x\u2009 a')) [ , ]

There's bound to be a way to turn a stream of bytes into a stream of unicode code points (at least I think that's what python is doing for strings). Though I'm explicitly not volunteering to write the code for it.

Sure, but making one string from the file contents is surely much better than having a separate string per word in the original data.

... Ah, but I suppose the existing code hasn't avoided that anyway. (It's also creating regex match objects, but those get disposed each time through the loop.) I don't know that there's really a way around that. Given the file is barely a KB, I rather doubt that the illustrated techniques are going to move the needle.

In fact, it looks as though the entire data structure (whether a dict, Counter etc.) should a relatively small part of the total reported memory usage. The rest seems to be internal Python stuff.

Re: Everything old is new again: memory optimization

#92

Post by pjc50 »

muskstinks wrote: post #63
pjc50 wrote: post #49

Earlier quoted context omitted.

https://learn.microsoft.com/en-us/sysinternals/downloads/vmm... for an empty sublime text window gives me: - 100MB 'image' (ie executable code; the executable itself plus all the OS libraries loaded.) - 40MB heap - 50MB "mapped file", mostly fonts opened with mmap() or the windows equivalent - 45MB stack (each thread gets 2MB) - 40MB "shareable" (no idea) - 5MB "unusable" (appears to be address space that's not usabl…

Tx for the breakdown. I will play around with it later on my windows machine. But isn't it crazy how we throw out so much memory just because of random buffers? It feels wrong to me

As pointed out below, quite a lot of that isn't in RAM - see "working set".

There's a common noob complaint about "Linux using all my RAM!" where people are confused about the headline free/buffers numbers. If there's a reasonable chance data could be used again soon it's better to leave it in RAM; if the RAM is needed for something else, the current contents will get paged out. Having a chunk of RAM be genuinely unallocated to anything is doing nothing for you.

Re: Everything old is new again: memory optimization

#93

Post by zahlman »

est wrote: post #86
akx wrote: post #29

Earlier quoted context omitted.

This doesn't do the same thing though, since it's not Unicode aware. >>> 'x\u2009 a'.split() ['x', 'a'] # incorrect; in bytes mode, `\S` doesn't know about unicode whitespace >>> list(re.finditer(br'\S+', 'x\u2009 a'.encode())) [ , ] # correct, in unicode mode >>> list(re.finditer(r'\S+', 'x\u2009 a')) [ , ]

OP's .split_ascii() doesn't handle U+2009 as well. edit: OP's fully native C++ version using Pystd

Hmm? Which code are you looking at?

Re: Everything old is new again: memory optimization

#94

Post by VorpalWay »

inetknght wrote: post #68
muskstinks wrote: post #21

I'm always confused as hell how little insight we have in memory consumption. I look at memory profiles of rnomal apps and often think "what is burning that memory". Modern compression works so well, whats happening? Open your taskmaster and look through apps and you might ask yourself this. For example (lets ignore chrome, ms teams and all the other bloat) sublime consumes 200mb. I have 4 text files open. What is it…

> I look at memory profiles of rnomal apps and often think "what is burning that memory". As a corrolary to this: I look at CPU utilization graphs. Programs are completely idle. "What is burning all that CPU?!" I remember using a computer with RAM measured in two-digit amounts of MiB. CPU measured in low hundreds of MHz. It felt just as fast -- sometimes faster -- as modern computers. Where is all of that extra RAM b…

I too wonder that. And it is true on an OS level as well. The only worthwhile change in desktop environments since the early 2000s has been search as you type launchers. Other than that I would happily use something equivalent to Windows XP or (more likely) Linux with KDE 3. It seems everything else since then has mostly been bloat and stylistic design changes. The latter being a waste of time in my opinion.

Of course, some software other than desktop environments have seen important innovation, such as LSPs in IDEs which allows avoiding every IDE implementing support for every language. And SSDs were truly revolutionary in hardware, in making computers feel faster. Modern GPUs can push a lot more advanced graphics as well in games. And so on. My point above was just about your basic desktop environment. Unless you use a tiling window manager (which I tried but never liked) nothing much has happened for a very long time. So just leave it alone please.

Re: Everything old is new again: memory optimization

#95

Post by zahlman »

dgb23 wrote: post #15

Not a C++ programmer and I think the solution is neat. But it's not necessarily an apples to apples comparison. It's not unfair to python because of the runtime overhead. It's unfair because it's a different algorithm with fundamentally different memory characteristics. A fairer comparison would be to stream the file in C++ as well and maintain internal state for the count. For most people that would be the first/nai…

> It's unfair because it's a different algorithm with fundamentally different memory characteristics. A fairer comparison would be to stream the file in C++ as well and maintain internal state for the count.

The C++ code is still building a tally by incrementing keys of a hash map one at a time, and then dumping (reversed) key/value pairs out into a list and sorting. The file is small and the Python code is GCing the `line` each time through the outer loop. At any rate it seems like a big chunk of the Python memory usage is just constant (sort of; stuff also gets lazily loaded) overhead of the Python runtime, so.

Re: Everything old is new again: memory optimization

#96

Post by menaerus »

VorpalWay wrote: post #76
menaerus wrote: post #47

Earlier quoted context omitted.

Retrofitting new patterns or ideas is underutilized only when it is not worth the change. string_view example is trivial and anyone who cared enough about the extra allocations that could have happened already (no copy-elision taking place) rolled their own version of string_view or simply used char+len pattern. Those folks do not wait for the new standard to come along when they can already have the solution now. st…

Existing APIs for file IO in STL don't return string views into the file buffer of the library (when using buffered IO). That is something you could do, as an example. Optional being opinionated I don't think I agree with. It is better to have an optional of something that can't be null (such as a reference) than have everything be implicitly nullable (such as raw pointers). This means you have to care about the null…

I work on a codebase which is heavily influenced by the same sentiment you share wrt optional and I can tell you it's a nightmare. Has the number of bugs somehow magically decreased? No, it did not, as a matter of fact the complexity that it introduces, which is to be honest coupled along with the monadic programming patterns which are normally enforced within such environments, just made it more probable to introduce buggy code at no obvious advantage but at the great cost - ergonomics, reasoning about the code, and performance. So, yeah, I will keep the position that it is heavily opinionated and not solving any real problem until I see otherwise - the evidence in really complex C++ production code. I have worked with many traditional C and C++ codebases so that is my baseline here. I prefer working with latter.

Re: Everything old is new again: memory optimization

#97

Post by gwbas1c »

pjc50 wrote: post #52
gwbas1c wrote: post #43

A lot of frameworks that use variants of "mark and sweep" garbage collection instead of automatic reference counting are built with the assumption that RAM is cheap and CPU cycles aren't, so they are highly optimized CPU-wise, but otherwise are RAM inefficient. I wonder if frameworks like dotnet or JVM will introduce reference counting as a way to lower the RAM footprint?

Reference counting in multithreaded systems is much more expensive than it sounds because of the synchronization overhead. I don't see it coming back. I don't think it saves massive amounts of memory, either, especially given my observation with vmmap upthread that in many cases the code itself is a dominant part of the (virtual) memory usage.

That's why Rust has Rc for single-threaded structs, and Arc for thread-safe structs.

Re: Everything old is new again: memory optimization

#98

Post by fluoridation »

pjc50 wrote: post #92
muskstinks wrote: post #63

Earlier quoted context omitted.

Tx for the breakdown. I will play around with it later on my windows machine. But isn't it crazy how we throw out so much memory just because of random buffers? It feels wrong to me

As pointed out below, quite a lot of that isn't in RAM - see "working set". There's a common noob complaint about "Linux using all my RAM!" where people are confused about the headline free/buffers numbers. If there's a reasonable chance data could be used again soon it's better to leave it in RAM; if the RAM is needed for something else, the current contents will get paged out. Having a chunk of RAM be genuinely una…

Nitpick: What you're describing is the disk cache. If a process requests more memory than is free, the OS will not page out pages used for the cache, it will simply either release them (if they're on the read cache) or flush them (if they're on the write cache).

Re: Everything old is new again: memory optimization

#99

Post by muskstinks »

Capricorn2481 wrote: post #66
muskstinks wrote: post #61

Earlier quoted context omitted.

I do not have plugins installed and i have only a handful of files open on macos. Memroy statistics says 200mb and a peak of 750mb in the past (for whatever reason)

Is that in Task Manager, or is that not a reliable place to look for these statistics? Edit: From what I can tell, Sublime is allocated 100mb of virtual memory even if it's only using about 10mb in practice.

Real Memory: 138 MB Virtual Memory: 390 GB <<<<<< wtf? :) Shared Memory: 143 MB Private Memory: 34 MB
Post reply on HN