Deterministic multithreading is hard (2024)
factorio.com
Deterministic multithreading is hard (2024)
1–10 of 14 posts
Re: Deterministic multithreading is hard (2024)
#2if you haven't read the factorio devblog before, please do!
after HN, it's one of my favourite places on the internet because i constantly learn new, random, insane things almost every time. imho it teaches you how to think + shows you what great engineering taste looks like. sorry if i'm overly effusive but each post is so deeply technical and well-written that i can't believe it's free.
you don't need to know anything about factorio or gamedev btw (i don't), just pick a random number between 1 to 438 and start reading :)
Re: Deterministic multithreading is hard (2024)
#3My favorite paper on it doing a clever workaround:
Re: Deterministic multithreading is hard (2024)
#4My favorite paper on it doing a clever workaround: https://github.com/emeryberger/dthreads
Paper link (2011): https://people.cs.umass.edu/~emery/pubs/dthreads-sosp11.pdf
> DTHREADS works by exploding multithreaded applications into multiple processes, with private, copy-on-write mappings to shared memory... Experimental results show that DTHREADS substantially outperforms a state-of-the-art deterministic runtime system, and for a majority of the benchmarks evaluated here, matches and occasionally exceeds the performance of pthreads.
Re: Deterministic multithreading is hard (2024)
#5if you haven't read the factorio devblog before, please do! after HN, it's one of my favourite places on the internet because i constantly learn new, random, insane things almost every time. imho it teaches you how to think + shows you what great engineering taste looks like. sorry if i'm overly effusive but each post is so deeply technical and well-written that i can't believe it's free. you don't need to know anyth…
I used to say the same about the Eve Online dev blog, but at some point I stopped reading because they were creating an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang.
Concurrency is hard. Blizzard added progressively more and more concurrency over time to rescue orphaned resources assigned to a single shard that was undersubscribed while another shard in the same AZ was seeing flash mobs. But the way they documented it was more of a tea leaves situation. Only enough data to guess what they had done if you were familiar with the space.
Re: Deterministic multithreading is hard (2024)
#6My favorite paper on it doing a clever workaround: https://github.com/emeryberger/dthreads
What’s interesting to me about Prof Berger’s work is that while many of his ideas are seemingly powerful and should be very impactful, it’s underutilized in practice. I think that’s because he’s failed to figure out how to really make it easy to use and productized. For example, Coz should be the definitive tool and mechanism people use to do performance analysis. However because it’s so hard to use and fails in various ways and is barely maintained if at all it ends up seeing limited use.
It’s a shame. The real world of development would be significantly richer if these ideas had better funding and dedicated long term development.
Re: Deterministic multithreading is hard (2024)
#7if you haven't read the factorio devblog before, please do! after HN, it's one of my favourite places on the internet because i constantly learn new, random, insane things almost every time. imho it teaches you how to think + shows you what great engineering taste looks like. sorry if i'm overly effusive but each post is so deeply technical and well-written that i can't believe it's free. you don't need to know anyth…
Yep FFF is such a treat to read. They do an amazing job of explaining complex problems and their solution in a way where anyone can understand.
Re: Deterministic multithreading is hard (2024)
#8One of the most interesting things to me when reading this was that it was treated as a bug even though it was that hard to reproduce. Most dev shops would not have the bandwidth and management but in to spend the time to dig into something like that unless it was high severity, and also it sounds like it was also getting caused from a modded version of the software
Re: Deterministic multithreading is hard (2024)
#9Almost impossible
Re: Deterministic multithreading is hard (2024)
#10My favorite paper on it doing a clever workaround: https://github.com/emeryberger/dthreads
Note that this wouldn't have been useful for Factorio, because Factorio deals with the harder problem of needing deterministic results with varying amount of parallelism, whereas from what I understand Dthread only give the same results if you run the program with the same number of "threads".