A common mistake when NumPy’s RNG with PyTorch • Hacker News Board

Re: A common mistake when NumPy’s RNG with PyTorch

#51

Post by Salgat » Mon, Apr 12, 2021, 2:40 AM UTC

timzaman wrote: post #24

Iirc the bug Karpathy mentioned in his tweet was actually due to the seed being the same across multigpu data parallel workers! You need to account for this too. So the author hasnt solved it. I know this bc I fixed the bug. And probably caused it. Hehe. Also you dont just want to set ur numpy seed but also the native python one and the torch one.

Yeah this isn't really a bug or even an issue with the library. If you instantiate an RNG with a seed and then fork your process, well duh of course the RNG will be repeated across the forks.

Re: A common mistake when NumPy’s RNG with PyTorch

#52

Post by Salgat » Mon, Apr 12, 2021, 2:41 AM UTC

Too wrote: post #28

.NET has a similar pitfall, but not due to forking but rather that the Random() default seed is based on the system clock. So starting several threads constructing new Random objects with the hope that they are unique might in fact give you same RNG sequences.

In .NET Core they actually fixed this thank goodness https://stackoverflow.com/questions/57905143/did-microsoft-c...

Re: A common mistake when NumPy’s RNG with PyTorch

#53

Post by rurban » Mon, Apr 12, 2021, 7:22 AM UTC

phonebucket wrote: post #15

jandrese wrote: post #5

Forgetting to seed your RNG is a really classic bug. IMHO RNGs should auto seed unless explicitly set not to, but since the opposite behaviour was baked into C so many years ago it's kind of the default. The worst part is how easy a bug this is to miss unless you're explicitly printing out the first set of random numbers for some strange reason.

I’m of the opposite opinion and would get away from all auto RNG seeding: 1) this will help reproducibility a great deal, which is a pain so often. 2) forcing users to actually understand the seeding of RNGs from the point that they are novice programmers could help allay bugs of the sort seen in this post, which I believe stems from having too much faith that RNGs will simply work out of the box as substitutions for…

I have again a different opinion. Allow both: srand() - explicit seed initialization, as well as autoseed.

But you really need to change selected known bad seeds, which destroy the PRNG statistical properties. Most PRNG's have a couple of known bad seeds, but nobody does anything against it. Same for hash functions.

Re: A common mistake when NumPy’s RNG with PyTorch

#54

Post by rurban » Mon, Apr 12, 2021, 7:26 AM UTC

andrew_v4 wrote: post #45

Is there something specific about numpy here, or would it be any RNG? I'm looking at some code that uses random.random() to randomly apply augmentations, I suspect that will have the same issue right?

It is a well-known trap that forked or threaded RNG's keep using the same seed. It can be called a feature, eg when you need to process different ranges, using the same sequence of random numbers for each range, but mostly it's a bug. You need to init your rng seed for each thread or fork to get different random sequences.

For splitting up sequential ranges, a good rng typically has an advance function to advance the seed for each range. So you can get reproducibility.