Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

141–150 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#141
post #13

One of my interview questions starts with "Can a program allocate more memory than is physically available on the server?" Everybody gets this wrong (which is funny for a binary question) but it starts an interesting discussion through which I hope to learn how much they know about OS and virtual memory.

Should the question be like this? "Can a program ~~allocate~~ use more memory than is physically available on the server (assuming no swap)?" Because, malloc allocates from virtual memory, not from the physical memory. Only when we access the the memory explicitly by read/write for the first time, then the page fault happens and page allocation begins. If page can't be allocated, then program gets terminated with a SIGNAL. Here, malloc was successful since allocation from VM was successful. But that is no guarantee that we have all memory we got. We can still run out of memory even though we allocated everything at start of program... It is so un predictable ... So, I assume, your question is asked with above intention. Please correct me if I am wrong. Your question and whole thread made me to think! -p

Re: In C, how do you know if the dynamic allocation succeeded?

#142
Life is hard for C coders.

Kubernetes made writing poor code a breeze. At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. So we can concentrate on churning features fast instead of writing good code.

Re: In C, how do you know if the dynamic allocation succeeded?

#144

Life is hard for C coders. Kubernetes made writing poor code a breeze. At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. So we can concentrate on churning features fast instead of writing good code.

K8s might save you from crashes affecting availability but what about data corruption, logicals errors sound APIs etc.? I'm guessing if code quality is neglected all of these suffer.

Re: In C, how do you know if the dynamic allocation succeeded?

#145

Life is hard for C coders. Kubernetes made writing poor code a breeze. At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. So we can concentrate on churning features fast instead of writing good code.

> At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods.

This probably makes a lot of people have strong negative emotions, but at the same time i feel that you're not wrong and it's the only way to deal with the modern web dev, where clients/business push for features instead of quality, versus something like kernel/system software development, where there is more pushback against this for historial and cultural reasons.

At work, we have this one monolith system that's in the center of everything else within a particular project - it's not really scalable and it has multiple scheduled processes within it, as well as serves a lot of external API requests, oh and also has an administrative UI. So far, my attempts to warn people against having a single point of failure like this have fallen on deaf ears and we still have outages where the JVM misbehaves or scheduled processes gobble up all of the server's memory and GC slows everything down on a regular basis.

Contrast this to me finally getting to implement something more like microservices in another project - the services are containerized and run on servers that have been configured with Ansible, are horizontally scalable and have proper load balancing. Furthermore, the scheduled process functionality and others can sit behind feature flags and be enabled within a particular instance, all while not having multiple separate projects and keeping things simple with a single, modular codebase. I actually dubbed this approach "moduliths", horizontally scalable and modular monoliths, since there is no way that this org can handle "proper" microservices, about which i wrote more here: https://blog.kronis.dev/articles/modulith-because-we-need-to...

That said, even the older monolith projects can benefit from modern approaches like Ansible for configuration management (which also prevents situations where environment configuration diverges over time and no one has any idea why) as well as being put into containers - the horrible monolith application now also lives within a container (not yet in prod, sadly) and has built in health checks. Were it ever to break and fail to recover in a set time, then it will automatically restart, making an outage that lasts an hour and possibly makes someone get paged in the middle of the night instead be a minute long interruption before everything restarts.

Personally, i think that with the direction that the industry is headed all services and even servers should be restarted every now and then anyways, since with JVM/CLR you sometimes get weird things happening after a service being up for months or years. Knowing why that happens would be nice, of course, but no one actually has the time to address those.

Re: In C, how do you know if the dynamic allocation succeeded?

#146
post #144

Life is hard for C coders. Kubernetes made writing poor code a breeze. At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. So we can concentrate on churning features fast instead of writing good code.

K8s might save you from crashes affecting availability but what about data corruption, logicals errors sound APIs etc.? I'm guessing if code quality is neglected all of these suffer.

Just enough to get a new round of funding, a bonus, then to parachute out and let someone else deal with the bills when they come due

Re: In C, how do you know if the dynamic allocation succeeded?

#147

Life is hard for C coders. Kubernetes made writing poor code a breeze. At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. So we can concentrate on churning features fast instead of writing good code.

> At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. This probably makes a lot of people have strong negative emotions, but at the same time i feel that you're not wrong and it's the only way to deal with the modern web dev, where clients/business push for features instead of quality, versus something like kernel/system software development, whe…

> Knowing why that happens would be nice, of course, but no one actually has the time to address those.

More than likely bad user code. Perhaps even race conditions. But in case of the JVM, with flight recorder and other forms of logging you could find out the problem with quite a good chance.

Which version of the JVM do you use?

Re: In C, how do you know if the dynamic allocation succeeded?

#148

Life is hard for C coders. Kubernetes made writing poor code a breeze. At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. So we can concentrate on churning features fast instead of writing good code.

the basic pattern is having a load balancer sitting in front of N services, and then having a service manager keep an eye on each service and restart them if they crash. so kubernetes can get the job done, but you can do essentially the same thing with a load balancer & some VMs & using the service manager that comes with your operating system (even windows has one)

Re: In C, how do you know if the dynamic allocation succeeded?

#149
post #144

Life is hard for C coders. Kubernetes made writing poor code a breeze. At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. So we can concentrate on churning features fast instead of writing good code.

K8s might save you from crashes affecting availability but what about data corruption, logicals errors sound APIs etc.? I'm guessing if code quality is neglected all of these suffer.

Code quality is not neglected to the point we have logical errors. Architecture is reasonable and we have an extensive suite of end to end tests and integration tests.

Re: In C, how do you know if the dynamic allocation succeeded?

#150
post #37

Earlier quoted context omitted.

Yeah. And the issue is that the actual problem happens sometime later when the application actually tries to use that memory. So you replaced an error that is relatively simple to handle with something that is impossible to handle reliably. So the operating system very much doesn't like to admit it doesn't have physical memory to back the area you are trying to use. Now it does not have a simple way to signal this to…

Interestingly, Linux's OOMKiller actually gives you (the sysadmin or system designer) more control over what happens when the system is low on memory than disabling overcommit. In a system without overcommit, every process is taking memory out of the shared pool, until some random process is the unlucky one that can't allocate more. In a happy case, that unlucky process also has some data that it can let go of. But t…

There is userspace integration available as well for OOM killers, afaik Facebook using/having developed them. They can take into account much more fine-grained selection criteria before killing a process.
Post reply on HN