Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

161–170 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#161

Post by jagrsw »

Koromix wrote: post #159
jagrsw wrote: post #151

Earlier quoted context omitted.

> The proper semantics for starting a new process is.. :) Ah.. I guess when "I was younger" (TM) I was also so dogmatic on most things computer-related. Anyways, when we need to do something a bit more complex than equivalent of system(), then it quickly becomes evident, that in many cases we need to prepare ourselves for the future execve(). Here's the list of syscalls/c-funcs, which are called in two random project…

I think that in a proper API, we could have: int process_handle = spawn("/bin/true", SP_PAUSED); // create process but don't execute, and return a handle // most API would take a process handle, thus you could do stuff (such as prctl) on the new process unpause(process_handle); There's a bit of a move in this direction in the Linux API with the PIDFD stuff.

Sounds reasonable. I guess most process controlling syscalls (ptrace, prctl, seccomp, bpf, remotely operating on pid's FDs, setsid, signal masks) will have to be replaced to be able to operate on remote process or on a fd, but yeah, this can work, potentially.

Re: In C, how do you know if the dynamic allocation succeeded?

#162

Post by foxfluff »

jagrsw wrote: post #151
bitwize wrote: post #99

Earlier quoted context omitted.

That's just the thing. fork(2), like much of POSIX, was a mistake. The proper semantics for starting a new process is something like posix_spawn or Win32 CreateProcess, i.e., you specify an executable image to start.

> The proper semantics for starting a new process is.. :) Ah.. I guess when "I was younger" (TM) I was also so dogmatic on most things computer-related. Anyways, when we need to do something a bit more complex than equivalent of system(), then it quickly becomes evident, that in many cases we need to prepare ourselves for the future execve(). Here's the list of syscalls/c-funcs, which are called in two random project…

If it's all syscalls (i.e. no libraries messing with hidden state) and you don't mess with parent's memory, you might be able to do this all with clone(CLONE_VM|CLONE_VFORK) instead of fork(). This is what musl uses to implement posix_spawn().

Re: In C, how do you know if the dynamic allocation succeeded?

#163

Post by ayende »

dahart wrote: post #124
Const-me wrote: post #82

That’s one of the many reasons I think Windows NT kernel is generally better than Linux. Windows doesn’t do that. When you don’t have enough memory and not enough page file space either, these allocation functions usually do fail returning nullptr.

But this feature is also why malloc on Windows can take a long time to return - seconds or more if you malloc too much. I recently bumped into this problem on the GPU. Since GPU memory in the Windows Display Driver Model requires it to be backed by the host machine’s virtual memory, a malloc on the GPU when running low on space can end up searching & even defragging virtual memory to satisfy the request. Crazy! Just…

What would you rather have, in this case?

Assuming that the work needs to be done? You can either do that on allocation (which you can predict, batch, etc) or you can do that when you need the next page to be used. Which is far harder to predict and work with.

Re: In C, how do you know if the dynamic allocation succeeded?

#164

Post by jagrsw »

foxfluff wrote: post #162
jagrsw wrote: post #151

Earlier quoted context omitted.

> The proper semantics for starting a new process is.. :) Ah.. I guess when "I was younger" (TM) I was also so dogmatic on most things computer-related. Anyways, when we need to do something a bit more complex than equivalent of system(), then it quickly becomes evident, that in many cases we need to prepare ourselves for the future execve(). Here's the list of syscalls/c-funcs, which are called in two random project…

If it's all syscalls (i.e. no libraries messing with hidden state) and you don't mess with parent's memory, you might be able to do this all with clone(CLONE_VM|CLONE_VFORK) instead of fork(). This is what musl uses to implement posix_spawn().

I think setenv() will use malloc() (it might use mutexes), same for error printing if it uses printf() or perror().

Also, this is all mess if fork() is called from a multi-threaded context. Esp. via clone(), which up to certain glibc() version cached getpid() result, and returned parent'd PID in the child (for performance reasons of course ;).

Re: In C, how do you know if the dynamic allocation succeeded?

#165

Post by foxfluff »

dahart wrote: post #124
Const-me wrote: post #82

That’s one of the many reasons I think Windows NT kernel is generally better than Linux. Windows doesn’t do that. When you don’t have enough memory and not enough page file space either, these allocation functions usually do fail returning nullptr.

But this feature is also why malloc on Windows can take a long time to return - seconds or more if you malloc too much. I recently bumped into this problem on the GPU. Since GPU memory in the Windows Display Driver Model requires it to be backed by the host machine’s virtual memory, a malloc on the GPU when running low on space can end up searching & even defragging virtual memory to satisfy the request. Crazy! Just…

That sounds like poor implementation rather than something inherent.

Re: In C, how do you know if the dynamic allocation succeeded?

#166

Post by josefx »

dekhn wrote: post #67
joosters wrote: post #65

Earlier quoted context omitted.

I’m sorry, but I’m still not following… why must a cache be (in theory) flushed more often without overcommit?

if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation. Since most applications don't touch pages they allocate, that means the OS could have avoided dropping the caches. Since it did drop the caches, other parts of the system will then have to do more work to reconstruct the caches (looking up an dentry explicitly, or loading a page from disk ins…

> if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation.

Couldn't the system just reserve the pages for future use by the application but still use them for caching until the application actually tries to use them?

Re: In C, how do you know if the dynamic allocation succeeded?

#169

Post by spacechild1 »

marcan_42 wrote: post #110
saagarjha wrote: post #75

Earlier quoted context omitted.

I mean, posix_spawn exists. It's a messy function, but its job is messy for exactly the reasons you describe. (FWIW, there are very few things you can legally perform between fork and exec.)

Are there any things that are illegal between fork and exec? It is perfectly legal to exec whenever you want, and it it perfectly legal to fork whenever you want. I'm not aware of any requirements around the fork/exec sequence.

From the fork() man page:

“After a fork() in a multithreaded program, the child can safely call only async-signal-safe functions (see signal-safety(7)) until such time as it calls execve(2)“

For example, you can't do any dynamic memory allocation or call a function that may allocate.

Re: In C, how do you know if the dynamic allocation succeeded?

#170

Post by vbezhenar »

lokedhs wrote: post #157
pcwalton wrote: post #69

Earlier quoted context omitted.

> wouldn't you just account the COW pages against the parent until they are copied? That's what Linux does. But how do you return ENOMEM when the copy does happen and now the system is out of memory? Memory writes don't return error codes. The best you could do is send a signal, which is exactly what the OOM killer does.

That's why you ensure you have enough swap to cover the worst case scenario. You as swap not because you need to actually use it, but on order to be able to guarantee there is enough memory available if the worst case scenario happens. In normal circumstances, swap should never really be utilised.

> In normal circumstances, swap should never really be utilised.

I don't agree. There're plenty of dormant virtual memory pages which will never be used. Keeping them in RAM is wasting precious resources.

Post reply on HN