Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything old is new again: memory optimization

nibblestew.blogspot.com

Post reply on HN
Print view View original

141–150 of 168 posts

Re: Everything old is new again: memory optimization

#141

Post by zahlman »

contravariant wrote: post #39
tzot wrote: post #6

Well, we can use memoryview for the dict generation avoiding creation of string objects until the time for the output: import re, operator def count_words(filename): with open(filename, 'rb') as fp: data= memoryview(fp.read()) word_counts= {} for match in re.finditer(br'\S+', data): word= data[match.start(): match.end()] try: word_counts[word]+= 1 except KeyError: word_counts[word]= 1 word_counts= sorted(word_counts.…

For reasons I never quite understood python has a collections.Counter for the purpose of counting things. It's a bit cleaner.

> It's a bit cleaner.

That's pretty much the reason why. Raymond Hettinger explains the philosophy well while discussing the `random` standard library module: https://www.youtube.com/watch?v=Uwuv05aZ6ug

I feel like much of this has been forgotten of late, though. From what I've seen, i's really quite hard to get anything added to the standard library unless you're a core dev who's sufficiently well liked among other core devs, in which case you can pretty much just do it. Everyone else will (understandably) be put through a PhD thesis defense, then asked to try the idea out as a PyPI package first (and somehow also popularize the package), and then if it somehow catches on that way, get declined anyway because it's easy for everyone to just get it from PyPI (see e.g. Requests).

I personally was directed to PyPI once when I was proposing new methods for the builtin `str`. Where the entire point was not to have to import or instantiate anything.

Re: Everything old is new again: memory optimization

#142

Post by hrmtst93837 »

pjc50 wrote: post #7
griffindor wrote: post #3

Nice! > Peak memory consumption is 1.3 MB. At this point you might want to stop reading and make a guess on how much memory a native code version of the same functionality would use. I wish I knew the input size when attempting to estimate, but I suppose part of the challenge is also estimating the runtime's startup memory usage too. > Compute the result into a hash table whose keys are string views, not strings If t…

>> Peak memory consumption is 1.3 MB. At this point you might want to stop reading and make a guess on how much memory a native code version of the same functionality would use. At this point I'd make two observations: - how big is the text file? I bet it's a megabyte, isn't it? Because the "naive" way to do it is to read the whole thing into memory. - all these numbers are way too small to make meaningful distinctio…

[flagged]

Re: Everything old is new again: memory optimization

#143

Post by rcxdude »

groundzeros2015 wrote: post #125
rcxdude wrote: post #50

Earlier quoted context omitted.

Which isn't very good for substrings due to the null-termination requirement.

Struct Substring { char start, end }; My point is ownership being transferred implicitly in a struct assignment is a complexity introduced by C++. In C the concern of allocating memory and using it is separate. String_view is attempt to add more separation. But C programmers were already there.

Well yeah, and you could always do the same thing in C++, but having a standard structure for it makes interoperability a lot easier.

Re: Everything old is new again: memory optimization

#144

Post by BiteCode_dev »

muskstinks wrote: post #21

I'm always confused as hell how little insight we have in memory consumption. I look at memory profiles of rnomal apps and often think "what is burning that memory". Modern compression works so well, whats happening? Open your taskmaster and look through apps and you might ask yourself this. For example (lets ignore chrome, ms teams and all the other bloat) sublime consumes 200mb. I have 4 text files open. What is it…

rnomal ?

Re: Everything old is new again: memory optimization

#146

Post by HappMacDonald »

pjc50 wrote: post #92
muskstinks wrote: post #63

Earlier quoted context omitted.

Tx for the breakdown. I will play around with it later on my windows machine. But isn't it crazy how we throw out so much memory just because of random buffers? It feels wrong to me

As pointed out below, quite a lot of that isn't in RAM - see "working set". There's a common noob complaint about "Linux using all my RAM!" where people are confused about the headline free/buffers numbers. If there's a reasonable chance data could be used again soon it's better to leave it in RAM; if the RAM is needed for something else, the current contents will get paged out. Having a chunk of RAM be genuinely una…

Of course it's doing something for you. Room to defrag other areas of RAM, room to load something new without moving something else out of the way first.

Your perspective sounds like the concept that space in a room does nothing for you until/unless you cram it full of hoarded items.

Re: Everything old is new again: memory optimization

#147

Post by guenthert »

rcxdude wrote: post #50
groundzeros2015 wrote: post #35

Earlier quoted context omitted.

In C you have char*

Which isn't very good for substrings due to the null-termination requirement.

strtok() happily punches those holes in. Now you could argue, the resulting strings, while null-terminated, aren't true substrings (as the origin string is now corrupted), but in the context of parsing (particularly here using white-space as delimiter), that wouldn't be much an issue.

Re: Everything old is new again: memory optimization

#148

Post by rcxdude »

1718627440 wrote: post #145
rcxdude wrote: post #50

Earlier quoted context omitted.

Which isn't very good for substrings due to the null-termination requirement.

An object of type char * isn't necessarily null-terminated.

No, but that makes it no longer a string as far as most C functions are concerned.

Re: Everything old is new again: memory optimization

#149

Post by alok-g »

baud9600 wrote: post #113

Strange days we live in. Python and C++? What about a line of bash: tr -s '[:space:]' '\n' I’d like to know the memory profile of this. The bottleneck is obviously sort which buffers everything in memory. So if we replace this with awk using a hash map to keep count of unique words, then it’s a much smaller data set in memory: tr -s '[:space:]' '\n' I’m guessing this will beat Python and C++?

Isn't using bash effectively saying, I have a bunch of functions already written in say C which I'll use but would not count those towards the lines of code? You could do the same in C and C++ itself too.

In other words, I am not sure if the comparison you are making is a fundamental one.

Re: Everything old is new again: memory optimization

#150

Post by cataphract »

gmueckl wrote: post #136
cataphract wrote: post #134

Earlier quoted context omitted.

Assuming the symbol is defined in the library, when the static linker runs (ld -- we're not talking ld.so), it will decide whether the global variable is preemptable or not, that is, if it can be resolved to a symbol outside the dso. Generally, by default it is, though this depends on many things -- visibility attributes, linker scripts, -Bsymbolic, etc. If it is, ld will have the final code reach into the GOT. If no…

I've never observed a (non-LTO) linker exchange instructions. I want to see an example before I can believe this.

I'm not sure if you're just trolling, but I'll give the same example I gave before (you can get even wilder simplifications -- called relaxations -- with TLS, since there are 4 levels of generality there). I'm not sure what you meant by "changing isntructions", but in the first case the linker did the fixup indicated by the relocation and in the second reduced the generality of the reference (one less level of indirection by changing mov to lea) because it knew the symbol could not be preempted (more exactly, the R_X86_64_REX_GOTPCRELX relocation allows the linker to do the relaxation if it can determine that it's safe to)

  root@1f0775a74fd7:/tmp# cat a.c
  int glob;
  int main() {
   return glob;
  }
  root@1f0775a74fd7:/tmp# gcc -c a.c -fPIC -o a.o
  root@1f0775a74fd7:/tmp# objdump --disassemble=main a.o
  
  a.o:     file format elf64-x86-64
  
  
  Disassembly of section .text:
  
  0000000000000000 :
     0: f3 0f 1e fa           endbr64
     4: 55                    push   %rbp
     5: 48 89 e5              mov    %rsp,%rbp
     8: 48 8b 05 00 00 00 00  mov    0x0(%rip),%rax        # f 
     f: 8b 00                 mov    (%rax),%eax
    11: 5d                    pop    %rbp
    12: c3                    ret
  root@1f0775a74fd7:/tmp# readelf -rW a.o | grep glob
  000000000000000b  000000030000002a R_X86_64_REX_GOTPCRELX 0000000000000000 glob - 4
  root@1f0775a74fd7:/tmp# gcc -shared -o a.so a.o
  root@1f0775a74fd7:/tmp# objdump --disassemble=main a.so
  (...)
  00000000000010f9 :
      10f9: f3 0f 1e fa           endbr64
      10fd: 55                    push   %rbp
      10fe: 48 89 e5              mov    %rsp,%rbp
      1101: 48 8b 05 b8 2e 00 00  mov    0x2eb8(%rip),%rax        # 3fc0 
      1108: 8b 00                 mov    (%rax),%eax
      110a: 5d                    pop    %rbp
      110b: c3                    ret
  (...)
  root@1f0775a74fd7:/tmp# readelf -r a.so | grep glob
  000000003fc0  000600000006 R_X86_64_GLOB_DAT 000000000000400c glob + 0
  root@1f0775a74fd7:/tmp# gcc -shared -Wl,-Bsymbolic -o a.symb.so a.o
  root@1f0775a74fd7:/tmp# readelf -r a.symb.so | grep glob
  root@1f0775a74fd7:/tmp# objdump --disassemble=main a.symb.so
  (...)
  Disassembly of section .text:
  
  00000000000010f9 :
      10f9: f3 0f 1e fa           endbr64
      10fd: 55                    push   %rbp
      10fe: 48 89 e5              mov    %rsp,%rbp
      1101: 48 8d 05 04 2f 00 00  lea    0x2f04(%rip),%rax        # 400c 
      1108: 8b 00                 mov    (%rax),%eax
      110a: 5d                    pop    %rbp
      110b: c3                    ret
  (...)
Post reply on HN