Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Factorio – Statistics improvements, Linux adventures

factorio.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Factorio – Statistics improvements, Linux adventures

#41

Post by nh2 »

gizmo686 wrote: post #22
bno1 wrote: post #16

Earlier quoted context omitted.

I wonder how they deal with multi-threading. Forking a multi-threaded process is very error-prone (e.g. allocator locks can remain orphaned).

The good thing about this usecase is that they do not need to solve the general problem of forking a multithreades program. The child process is doing a very specific job that amounts to serializing the contents of memory and writing it to a file. Once that is done is can simply exit, and all the orphans are cleaned up. While this is admittadly more complicated, it is the same general idea behind fork+exec. Sure, you…

It still seems super error prone:

It is very difficult to write general purpose C++ code that does not call malloc somehere.

Something fails and you want to display a message `"Error: " + reason`? Bam, malloc right here, serialisation process may hang forever in a malloc lock.

You quit the game, the parent process exits, and the serialisation process gets reparented to init, invisibly using up your RAM until you reboot.

fork()+exec() works in C because C has no invisible memory allocation, and even there you'd usually try to not call any function in between the two to be very sure.

Using fork() without being 100% sure there can be no malloc usually means inviting years of rare, hard-to-reproduce random weird bug reports.

Beyond that, as the post mentions, fork() needs "requires a significant amount of RAM to work" if many pages are touched due to copy-on-write, and copy-on-write also slows down the main game.

It seems much safer to use a thread for saving the game state.

Re: Factorio – Statistics improvements, Linux adventures

#42

Post by glandium »

diath wrote: post #33
earthling8118 wrote: post #13

Earlier quoted context omitted.

It's time to make WaylandX, for running Wayland applications on X. Well, I happily use Wayland so I won't use it, but maybe you would.

That's already a thing, called XWayland, and it's the only way to make Wayland actually usable on Linux desktop.

XWayland is the exact opposite of what GP is asking for.

Re: Factorio – Statistics improvements, Linux adventures

#43

Post by [deleted] »

diath wrote: post #33
earthling8118 wrote: post #13

Earlier quoted context omitted.

It's time to make WaylandX, for running Wayland applications on X. Well, I happily use Wayland so I won't use it, but maybe you would.

That's already a thing, called XWayland, and it's the only way to make Wayland actually usable on Linux desktop.

[deleted]

Re: Factorio – Statistics improvements, Linux adventures

#45

Post by gizmo686 »

nh2 wrote: post #41
gizmo686 wrote: post #22

Earlier quoted context omitted.

The good thing about this usecase is that they do not need to solve the general problem of forking a multithreades program. The child process is doing a very specific job that amounts to serializing the contents of memory and writing it to a file. Once that is done is can simply exit, and all the orphans are cleaned up. While this is admittadly more complicated, it is the same general idea behind fork+exec. Sure, you…

It still seems super error prone: It is very difficult to write general purpose C++ code that does not call malloc somehere. Something fails and you want to display a message `"Error: " + reason`? Bam, malloc right here, serialisation process may hang forever in a malloc lock. You quit the game, the parent process exits, and the serialisation process gets reparented to init, invisibly using up your RAM until you rebo…

In principle, using a thread for saving the game state is a much more difficult problem, since the game state itself will mutate. You need to be extremely careful to assure that the state you serialize is consistent and is at least a plausible game state (even if there is a bit of wiggle room to allow for game states that technically never existed). This complexity invades every aspect of game logic; and bugs here tend to be subtle corruption of your save data that will take a while to notice; thereby obscuring the relationship to the corruption, and the save/reload cycle.

In contrast, forking with its COW semantics is conceptually easy. You just fork. The main process can continue running, and the child process gets a frozen snapshot. There is a bunch of overhead from the copy part of copy-on-write. However, most of that overhead will likely be spent in the first frame; which is still a significant improvement over the pause time associated with stop the world saving. In practice, coding for the child process is tricky. However, it is self contained and responsible only for a relatively simple problem. No complex problems to solve, just a relatively small amount of code that needs to be written carefully.

The RAM usage is a real trade-off inherent in the approach.

> You quit the game, the parent process exits, and the serialisation process gets reparented to init, invisibly using up your RAM until you reboot.

Or until the short-lived child process finishes its work and exits on its own.

Re: Factorio – Statistics improvements, Linux adventures

#46

Post by raiguard »

atq2119 wrote: post #37
bno1 wrote: post #16

Earlier quoted context omitted.

I wonder how they deal with multi-threading. Forking a multi-threaded process is very error-prone (e.g. allocator locks can remain orphaned).

They mention that there's a rare freeze bug. That does sound like it's most likely one of those orphaned lock problems or similar.

The post was already getting long so I didn't mention this, but I already found an fixed a potential deadlock due to each process not closing the redundant ends of the pipe, which was specifically touted as being a potential cause for freezing. However, someone reported the bug again after that change went out, so there's still SOMETHING else causing it to hang that I haven't tracked down yet.

Re: Factorio – Statistics improvements, Linux adventures

#47

Post by nh2 »

gizmo686 wrote: post #45
nh2 wrote: post #41

Earlier quoted context omitted.

It still seems super error prone: It is very difficult to write general purpose C++ code that does not call malloc somehere. Something fails and you want to display a message `"Error: " + reason`? Bam, malloc right here, serialisation process may hang forever in a malloc lock. You quit the game, the parent process exits, and the serialisation process gets reparented to init, invisibly using up your RAM until you rebo…

In principle, using a thread for saving the game state is a much more difficult problem, since the game state itself will mutate. You need to be extremely careful to assure that the state you serialize is consistent and is at least a plausible game state (even if there is a bit of wiggle room to allow for game states that technically never existed). This complexity invades every aspect of game logic; and bugs here te…

How do you know that the game state is in a consistent state in the child?

A stopped thread may be in the middle of mutating an array or struct.

Re: Factorio – Statistics improvements, Linux adventures

#48

Post by nh2 »

gizmo686 wrote: post #45
nh2 wrote: post #41

Earlier quoted context omitted.

It still seems super error prone: It is very difficult to write general purpose C++ code that does not call malloc somehere. Something fails and you want to display a message `"Error: " + reason`? Bam, malloc right here, serialisation process may hang forever in a malloc lock. You quit the game, the parent process exits, and the serialisation process gets reparented to init, invisibly using up your RAM until you rebo…

In principle, using a thread for saving the game state is a much more difficult problem, since the game state itself will mutate. You need to be extremely careful to assure that the state you serialize is consistent and is at least a plausible game state (even if there is a bit of wiggle room to allow for game states that technically never existed). This complexity invades every aspect of game logic; and bugs here te…

> Or until the short-lived child process finishes its work and exits on its own.

I was talking about the malloc-deadlocked child. That will not finish.

Re: Factorio – Statistics improvements, Linux adventures

#49

Post by failbuffer »

mksybr wrote: post #36
naikrovek wrote: post #28

Earlier quoted context omitted.

Ok, so make it optional to use custom window chrome/decorations, rather than make it mandatory. GNOME devs are very arrogant. “Do it our way or go away.” They make it very hard to hold onto any hope that there will ever be a reasonable, cohesive, Linux desktop experience.

KDE is reasonable and cohesive

I really like the hearty amount of configurability they make available thru the control panels. They're not afraid of giving you a lot of options, yet it's organized well, easy to use, and the defaults are sensible.

Re: Factorio – Statistics improvements, Linux adventures

#50

Post by glandium »

earthling8118 wrote: post #13
lupusreal wrote: post #4

If they drop X11 support that will finally be the end of my cracktorio addiction.

It's time to make WaylandX, for running Wayland applications on X. Well, I happily use Wayland so I won't use it, but maybe you would.

Looks like your wish has been granted. https://news.ycombinator.com/item?id=40175731
Post reply on HN