I usually write my own data handling functions rather than trying to play PyTorch’s game here. I find their abstractions here confusing or not useful.
A common mistake when NumPy’s RNG with PyTorch
41–50 of 54 posts
Re: A common mistake when NumPy’s RNG with PyTorch
#42Aside from the infuriating clickbait title (which I shall not dignify with an upvote), this is part of why I preprocess augmented images. I don't like too much magic in my custom derived (PyTorch) Dataset objects.
Re: A common mistake when NumPy’s RNG with PyTorch
#43Earlier quoted context omitted.
I believe the point is that the error will be more obvious if the state is passed around explicitly.
Possibly but this is the kind of boilerplate which people tend to ignore, especially when a program is non-trivial. It’s really easy to notice if you’re doing something like `seed_rng(); fork();` but once there’s distance and more than one thing being passed around I’d be surprised if you didn’t find the same pattern, perhaps a bit less common. Fundamentally, there two problems: fork() is a performance trick to try t…
Additionally, I think people make a hidden assumption that they don't even realize they're making: that when you ask for random numbers from numpy, they're more or less "true" random numbers, not seeded ones. Like, I think the intention of the programmers is just "give me a bunch of random numbers, I don't really care how as long as they're random", and assumes that that is what that numpy function does. But it doesn't: it provides you a pseudo-random sequence – not true randomness – so of course the sequence is identical after the fork.
Like, they think they're reading from /dev/random, but they're not: they're just running rand() (metaphorically speaking).
Re: A common mistake when NumPy’s RNG with PyTorch
#44Earlier quoted context omitted.
Possibly but this is the kind of boilerplate which people tend to ignore, especially when a program is non-trivial. It’s really easy to notice if you’re doing something like `seed_rng(); fork();` but once there’s distance and more than one thing being passed around I’d be surprised if you didn’t find the same pattern, perhaps a bit less common. Fundamentally, there two problems: fork() is a performance trick to try t…
Additionally, I think people make a hidden assumption that they don't even realize they're making: that when you ask for random numbers from numpy, they're more or less "true" random numbers, not seeded ones. Like, I think the intention of the programmers is just "give me a bunch of random numbers, I don't really care how as long as they're random", and assumes that that is what that numpy function does. But it doesn…
Definitely - back when I supported a computational neuroscience group that came up multiple times (not numpy but similar contexts), along with the various quirks around floating point math. Even experienced people do things like that because they’re focused on the actual problem and this is a leaky implementation detail.
Re: A common mistake when NumPy’s RNG with PyTorch
#45Is there something specific about numpy here, or would it be any RNG?
I'm looking at some code that uses random.random() to randomly apply augmentations, I suspect that will have the same issue right?
Re: A common mistake when NumPy’s RNG with PyTorch
#46This seems like another reason to never use fork() without exec(). Fork is really a mine field when used this way (and a pretty big maintenance burden on the kernel, by my understanding, to provide the illusion of sharing read-only state with the parent process).
Re: A common mistake when NumPy’s RNG with PyTorch
#47Yeah, I'd run into this 2 years ago and ended up also reporting an issue on the Centernet repo [1] The solution I have in that issue adapts from the very helpful discussions in the original Pytorch issue [2] `worker_init_fn=lambda id: np.random.seed(torch.initial_seed() // 2*32 + id)` I will admit that this is *very* easy to mess up as evidenced by the fact that examples in the official tutorials for Pytorch and othe…
Should be 2**32 above. HN formatting swallowed an asterisk.
Re: A common mistake when NumPy’s RNG with PyTorch
#48Note official TensorFlow tutorials make the exact same mistake. I've reported it but it hasn't been fixed. [1]
Re: A common mistake when NumPy’s RNG with PyTorch
#49Yeah, I'd run into this 2 years ago and ended up also reporting an issue on the Centernet repo [1] The solution I have in that issue adapts from the very helpful discussions in the original Pytorch issue [2] `worker_init_fn=lambda id: np.random.seed(torch.initial_seed() // 2*32 + id)` I will admit that this is *very* easy to mess up as evidenced by the fact that examples in the official tutorials for Pytorch and othe…
This field contains the worker-specific seed:
torch.utils.data.get_worker_info().seed
worker_init_fn=lambda id: np.random.seed(torch.utils.data.get_worker_info().seed)Re: A common mistake when NumPy’s RNG with PyTorch
#50Earlier quoted context omitted.
NumPy does auto-seed the RNG if you don't pass a seed yourself, using platform-specific code to pull some entropy from the OS. So that common case is handled reasonably well, unlike with C. In fact if you want exactly reproducible results (e.g. in testcases), you have to seed with a known seed, to avoid that default behavior. The issue here is a little more subtle: if you fork 10 copies of your Python process, all 10…
It’s even slightly more subtle than that. Python multiprocessing doesn’t use fork on Windows. It starts a new process and so shouldn’t be affected by this. So to trigger this you need to have num_processes != 0 on your DataLoader and be running on a non-Windows platform.
Starting with Python 3.8, multiprocessing will also use new processes by default on MacOS (due to some system libraries not being fork-safe).
IMHO cross-platform Python projects should call `multiprocessing.set_start_method('spawn')` to get the same behavior everywhere.