Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Memory Management, Part 1: C

educatedguesswork.org

Post reply on HN
Print view View original

71–80 of 95 posts

Re: Understanding Memory Management, Part 1: C

#71

Post by gizmo686 »

pjmlp wrote: post #68
gizmo686 wrote: post #55

Earlier quoted context omitted.

In practice C let's you control memory layout just fine. You might need to use __attribute__((packed)), which is technically non standard. I've written hardware device drivers in pure C where you need need to peek and poke at specific bits on the memory bus. I defined a struct that matched the exact memory layout that the hardware specifies. Then cast an integer to a pointer to that struct type. At which point I coul…

Now split the struct across registers in C. You are aware that some of those casting tricks are UB, right?

Casting integers to pointers in C is implementation defined, not UB. In practice compilers define these casts as the natural thing for the architecture you are compiling to. Since mainstream CPUs don't do anything fancy with pointer tagging, that means the implementation defined behave does exactly what you expect it to do (unless you forget that you have paging enabled and cannot simply point to a hardware memory address).

If you want to control register layout, then C is not going to help you, but that is not typically what is meant by "memory layout".

And if you want to control cache usage ... Some architectures do expose some black magic which you would need to go to assembly to access. But for the most part controlling cache involves understanding how the cache works, then controlling the memory layout and accesses to work well with the cache.

Re: Understanding Memory Management, Part 1: C

#72

Post by gpderetta »

MarkSweep wrote: post #36
jvanderbot wrote: post #5

Earlier quoted context omitted.

> which is just getting things done end-to-end as fast as possible, not careful at every step that we have no memory errors. One horrible but fun thing a former professor of mine pointed out: If your program isn't going to live long, then you never have to deallocate memory. Once it exits, the OS will happily clean it up for you. This works in C or perhaps lazy GC languages, but for stateful objects where destructors…

There is this old chestnut about “null garbage collectors”: https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98... > This sparked an interesting memory for me. I was once working with a customer who was producing on-board software for a missile. In my analysis of the code, I pointed out that they had a number of problems with storage leaks. Imagine my surprise when the customers chief software engineer said…

Untill the software is reused for a newer model with longer range and they forget to increase the ram size.

But of course that would never happen, wouldn't it?

Re: Understanding Memory Management, Part 1: C

#73

Post by PhilipRoman »

pjmlp wrote: post #69
PhilipRoman wrote: post #50

Earlier quoted context omitted.

Most kernel resources are fairly well behaved, as they will automatically decrement their refcount when a process exits. Even mutexes have a "robust" flag for this exact reason. Programs which rely on destructors or any other form or orderly exit are always brittle and should be rewritten to use atomic operations.

Which kernel, on which specific OS? This is a very non portable assumption, even we constrain it to only across UNIX/POSIX flavours.

As far as assumptions go, it's actually one of the most portable ones and for a good reason, considering it is a basic part of building a reliable system. Quoting POSIX:

Consequences of Process Termination

Process termination caused by any reason shall have the following consequences:

[..] All of the file descriptors, directory streams, conversion descriptors, and message catalog descriptors open in the calling process shall be closed.

[..] Each attached shared-memory segment is detached and the value of shm_nattch (see shmget()) in the data structure associated with its shared memory ID shall be decremented by 1.

For each semaphore for which the calling process has set a semadj value (see semop()), that value shall be added to the semval of the specified semaphore.

[..] If the process is a controlling process, the controlling terminal associated with the session shall be disassociated from the session, allowing it to be acquired by a new controlling process.

[..] All open named semaphores in the calling process shall be closed as if by appropriate calls to sem_close().

Any memory locks established by the process via calls to mlockall() or mlock() shall be removed. If locked pages in the address space of the calling process are also mapped into the address spaces of other processes and are locked by those processes, the locks established by the other processes shall be unaffected by the call by this process to _Exit() or _exit().

Memory mappings that were created in the process shall be unmapped before the process is destroyed.

Any blocks of typed memory that were mapped in the calling process shall be unmapped, as if munmap() was implicitly called to unmap them.

All open message queue descriptors in the calling process shall be closed as if by appropriate calls to mq_close().

Re: Understanding Memory Management, Part 1: C

#74

Post by atiedebee »

astrobe_ wrote: post #34
bluetomcat wrote: post #6

This isn't proper usage of realloc: lines = realloc(lines, (num_lines + 1) * sizeof(char *)); In case it cannot service the reallocation and returns NULL, it will overwrite "lines" with NULL, but the memory that "lines" referred to is still there and needs to be either freed or used. The proper way to call it would be: tmp = realloc(lines, (num_lines + 1) * sizeof(char *)); if (tmp == NULL) { free(lines); lines = NUL…

The program abort()s if the reallocation fails. But indeed, for an educational example, it's not good to be too smart. I believe the test if(!num_lines) is unnecessary, because reallocating a NULL pointer is equivalent to malloc(). This is also a bit "smart", but I think it is also more correct because you don't use the value of one variable (num_lines is 0) to infer the value of another (lines is NULL). To go furthe…

> I believe the test if(!num_lines) is unnecessary, because reallocating a NULL pointer is equivalent to malloc().

I thought that this behaviour was deprecated in C23, but according to cop reference it is still there[0].

An I thinking of realloc with 0 size or was this actually a thing that was discussed?

[0] https://en.cppreference.com/w/c/memory/realloc

Re: Understanding Memory Management, Part 1: C

#75

Post by erlkonig »

Using abort() every time malloc and kin fail isn't really satisfying anything except the idea that the program should crash before showing incorrect results.

While the document itself is pretty good otherwise, this philosophical failing is a problem. It should give examples of COPING with memory exhaustion, instead of just imploding every time. It should also mention using "ulimit -Sd 6000" or something to lower the limit to force the problems to happen (that one happens to work well with vi).

Memory management is mature when programs that should stay running - notably user programs, system daemons, things where simply restarting will lose precious user data or other important internal data - HANDLE exhaustion, clean up any partially allocated objects, then either inform the user or keep writing data out to files (or something) and freeing memory until allocation starts working again. E.g. Vi informs the user without crashing, like it should.

This general philosophy is one that I've seen degrade enormously over recent years, and a trend we should actively fight against. And this trend has been greatly exacerbated by memory overcommit.

Re: Understanding Memory Management, Part 1: C

#76

Post by pjmlp »

atiedebee wrote: post #74
astrobe_ wrote: post #34

Earlier quoted context omitted.

The program abort()s if the reallocation fails. But indeed, for an educational example, it's not good to be too smart. I believe the test if(!num_lines) is unnecessary, because reallocating a NULL pointer is equivalent to malloc(). This is also a bit "smart", but I think it is also more correct because you don't use the value of one variable (num_lines is 0) to infer the value of another (lines is NULL). To go furthe…

> I believe the test if(!num_lines) is unnecessary, because reallocating a NULL pointer is equivalent to malloc(). I thought that this behaviour was deprecated in C23, but according to cop reference it is still there[0]. An I thinking of realloc with 0 size or was this actually a thing that was discussed? [0] https://en.cppreference.com/w/c/memory/realloc

Section 7.24.3.7 The realloc function

https://open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf

> If ptr is a null pointer, the realloc function behaves like the malloc function for the specified size. Otherwise, if ptr does not match a pointer earlier returned by a memory management function, or if the space has been deallocated by a call to the free or realloc function, or if the size is zero, the behavior is undefined. If memory for the new object is not allocated, the old object is not deallocated and its value is unchanged.

Re: Understanding Memory Management, Part 1: C

#77

Post by returningfory2 »

dataflow wrote: post #58
returningfory2 wrote: post #26

Earlier quoted context omitted.

I feel like this comment is misleading because it gives the impression that the code in the article is wrong or unsafe, whereas I think it's actually fine? In the article, in the case when `tmp == NULL` (in your notation) the author aborts the program. This means there's no memory leak or unsafety. I agree that one can do better of course.

You're confusing the code with the program it compiles to. The program is fine, okay. But the code is only "fine" or "safe" if you view it as the final snapshot of whatever it's going to be. If you understand that the code also influences how it's going to evolve in the future (and which code doesn't?) then no, it's not fine or safe. It's brittle and making future changes more dangerous. Really, there's no excuse wha…

This is an article introducing people to memory management, targeted at beginners. The code snippets are there to illustrate the ideas. The author made the correct pedagogical decision to prioritize readability over optimal handling of an OOM edge case that would be confusing to introduce to beginner readers at this early stage.

Talking about "making future changes" seems to be missing the point of what the author is doing. They're not committing code to the Linux kernel. They're writing a beginner's article about memory management.

Re: Understanding Memory Management, Part 1: C

#78

Post by returningfory2 »

erlkonig wrote: post #75

Using abort() every time malloc and kin fail isn't really satisfying anything except the idea that the program should crash before showing incorrect results. While the document itself is pretty good otherwise, this philosophical failing is a problem. It should give examples of COPING with memory exhaustion, instead of just imploding every time. It should also mention using "ulimit -Sd 6000" or something to lower the…

It's a beginners article about memory management. I think it's weird that so many comments here are judging the code snippets as if they're commits to production systems. When writing articles like these there are pedagogical decisions to be made, such as simplifying the examples to make them easier to understand.

Re: Understanding Memory Management, Part 1: C

#79

Post by dataflow »

returningfory2 wrote: post #77
dataflow wrote: post #58

Earlier quoted context omitted.

You're confusing the code with the program it compiles to. The program is fine, okay. But the code is only "fine" or "safe" if you view it as the final snapshot of whatever it's going to be. If you understand that the code also influences how it's going to evolve in the future (and which code doesn't?) then no, it's not fine or safe. It's brittle and making future changes more dangerous. Really, there's no excuse wha…

This is an article introducing people to memory management, targeted at beginners. The code snippets are there to illustrate the ideas. The author made the correct pedagogical decision to prioritize readability over optimal handling of an OOM edge case that would be confusing to introduce to beginner readers at this early stage. Talking about "making future changes" seems to be missing the point of what the author is…

> This is an article introducing people to memory management, targeted at beginners

I realize, and that's what makes it even worse. First impressions have a heck of a stronger effect than 10th impressions. Beginners need to learn the right way in the beginning, not the wrong way.

Whenever did "safety first" stop being a thing? This is like like skipping any mention of goggles when teaching chemistry or woodworking for "pedagogical reasons". You're supposed to first you teach your students the the best way to do things, then you can teach them how to play fast and loose if it's warranted. Not the other way around!

Re: Understanding Memory Management, Part 1: C

#80

Post by returningfory2 »

dataflow wrote: post #79
returningfory2 wrote: post #77

Earlier quoted context omitted.

This is an article introducing people to memory management, targeted at beginners. The code snippets are there to illustrate the ideas. The author made the correct pedagogical decision to prioritize readability over optimal handling of an OOM edge case that would be confusing to introduce to beginner readers at this early stage. Talking about "making future changes" seems to be missing the point of what the author is…

> This is an article introducing people to memory management, targeted at beginners I realize, and that's what makes it even worse . First impressions have a heck of a stronger effect than 10th impressions. Beginners need to learn the right way in the beginning, not the wrong way. Whenever did "safety first" stop being a thing? This is like like skipping any mention of goggles when teaching chemistry or woodworking f…

The code in the article is not wrong. It is not unsafe. The author explicitly handles the OOM case correctly. It is true that there are more optimal ways to do it if you do have an OOM handling strategy.

And no, you're not supposed to teach your students the best way to do things at the start. That's not how teaching works. You start with the simpler (but still correct) way, and then work towards the best way. This is why introductions to Rust are full of clone calls. The best Rust code minimizes the number of clones. But when you're introducing people to something, you don't necessarily do the optimal thing first because that disrupts the learning process.

Post reply on HN