Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux /dev/urandom and concurrency

drsnyder.us

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Linux /dev/urandom and concurrency

#21

Post by mikeash »

jerf wrote: post #17
helper wrote: post #12

Earlier quoted context omitted.

I guess that means all go applications that use crypto/rand are considered misdesigned then[1]. [1]: http://golang.org/src/pkg/crypto/rand/rand_unix.go#L30

If you're using crypto /rand to yank a whole bunch of random numbers out for the purpose of deciding which DNS record to use when multiple DNS records were returned, yes, the Go application is misdesigned. Such applications should be using math /rand. Seeding your math/rand from crypto/rand isn't a bad idea, but you don't need to be hammering on /dev/urandom in such code.

You don't need to be, but why not? It should be plenty fast and work well. If it's turning out to be too slow due to too much locking, that should be fixed.

Re: Linux /dev/urandom and concurrency

#22

Post by hosay123 »

aidenn0 wrote: post #18

Seed a secure userspace PRNG from urandom, perhaps?

Adding to aidenn0's comment, if you trust /dev/urandom to produce 4kb of random data, it follows that you trust it to produce 128 bits.

128 bits (32 bytes) is sufficient to initialize a PRNG into any one of 115792089237316195423570985008687907853269984665640564039457584007913129639936 states (that's 1 with 77 digits). Consequently, hitting the kernel constantly for so much data is utterly inefficient in the first instance, and totally unnecessary in the second.

Blog author could improve his design's efficiency >128x just by seeding a PRNG with a single 32 byte read at the start of the subprocess

Re: Linux /dev/urandom and concurrency

#23

Post by sebcat »

As a user of libcares (which is awesome for bulk DNS lookups btw) I'll add that I've only ever needed one ares_channel per process. Having one ares_channel for every CURL-handle seems a bit excessive. This is probably the main problem here, not the kernel spinlock.

Edit: Come to think about it, why isn't the CURL-handle reused? Sounds like a new CURL-handle is inited for every request, which I don't recall being necessary.

Re: Linux /dev/urandom and concurrency

#24

Post by mcpherrinm »

hosay123 wrote: post #22
aidenn0 wrote: post #18

Seed a secure userspace PRNG from urandom, perhaps?

Adding to aidenn0's comment, if you trust /dev/urandom to produce 4kb of random data, it follows that you trust it to produce 128 bits. 128 bits (32 bytes) is sufficient to initialize a PRNG into any one of 115792089237316195423570985008687907853269984665640564039457584007913129639936 states (that's 1 with 77 digits). Consequently, hitting the kernel constantly for so much data is utterly inefficient in the first ins…

Userland PRNGs are one of the easiest ways to introduce security vulnerabilities into your programs. I would recommend being very, VERY careful before trying to do this, like the traditional "Don't roll your own crypto" advice.

Re: Linux /dev/urandom and concurrency

#25

Post by mike-cardwell »

Interestingly enough, I have actually been working on writing a DNS client library in C++ with Boost ASIO this very afternoon. I was going to get my source of random data using the following C++11 standard library code. I would really appreciate any comments from people here if there is anything wrong with what I'm doing:

  #include 
  std::uniform_int_distribution dist;

  // Seed a Mersenne twister PRNG with random data:
  std::mt19937 eng;
  std::random_device rd;
  eng.seed(dist(rd));

  // Now to generate random numbers, simply:
  uint32_t random_number = dist(eng);

Re: Linux /dev/urandom and concurrency

#26

Post by helper »

jerf wrote: post #17
helper wrote: post #12

Earlier quoted context omitted.

I guess that means all go applications that use crypto/rand are considered misdesigned then[1]. [1]: http://golang.org/src/pkg/crypto/rand/rand_unix.go#L30

If you're using crypto /rand to yank a whole bunch of random numbers out for the purpose of deciding which DNS record to use when multiple DNS records were returned, yes, the Go application is misdesigned. Such applications should be using math /rand. Seeding your math/rand from crypto/rand isn't a bad idea, but you don't need to be hammering on /dev/urandom in such code.

There are lots of applications that require a cryptographically secure random number generator, which math/rand doesn't claim to be.

Re: Linux /dev/urandom and concurrency

#27

Post by throwaway2048 »

jerf wrote: post #17
helper wrote: post #12

Earlier quoted context omitted.

I guess that means all go applications that use crypto/rand are considered misdesigned then[1]. [1]: http://golang.org/src/pkg/crypto/rand/rand_unix.go#L30

If you're using crypto /rand to yank a whole bunch of random numbers out for the purpose of deciding which DNS record to use when multiple DNS records were returned, yes, the Go application is misdesigned. Such applications should be using math /rand. Seeding your math/rand from crypto/rand isn't a bad idea, but you don't need to be hammering on /dev/urandom in such code.

This "some random data is more important then other random data" musical chair dance going on with /dev/random vs /dev/urandom vs userland [CS]PRNGs (often gathering from extremely poor sources, or using broken algos) has been nothing short of an unmitigated security and useability disaster.

We have the abillity to make the /dev/urandom CSPRNG secure enough and fast enough for (almost) any randomness purpose. We need to cut all the rest of this insane crap.

People choose the wrong RNGs and get burned, or wont use the right ones because of speed or imaginary entropy exhaustion issues. This matters.

Re: Linux /dev/urandom and concurrency

#28

Post by aidenn0 »

mike-cardwell wrote: post #25

Interestingly enough, I have actually been working on writing a DNS client library in C++ with Boost ASIO this very afternoon. I was going to get my source of random data using the following C++11 standard library code. I would really appreciate any comments from people here if there is anything wrong with what I'm doing: #include std::uniform_int_distribution dist; // Seed a Mersenne twister PRNG with random data: s…

I don't know what DNS uses the randomness for, but if a malicious attacker can gain from guessing the randomness, don't use MT, as the state can be extracted from MT by observing a relatively small number of outputs.

Re: Linux /dev/urandom and concurrency

#29

Post by raverbashing »

mikeash wrote: post #21
jerf wrote: post #17

Earlier quoted context omitted.

If you're using crypto /rand to yank a whole bunch of random numbers out for the purpose of deciding which DNS record to use when multiple DNS records were returned, yes, the Go application is misdesigned. Such applications should be using math /rand. Seeding your math/rand from crypto/rand isn't a bad idea, but you don't need to be hammering on /dev/urandom in such code.

You don't need to be, but why not? It should be plenty fast and work well. If it's turning out to be too slow due to too much locking, that should be fixed.

1 - Rand is faster

2 - You don't need the crypto qualities of it and you're emptying the entropy pool for nothing

3 - You're doing much more work, especially if you're reading one byte at a time from /dev/urandom (doing a syscall, etc), while rand is just a calculation

Re: Linux /dev/urandom and concurrency

#30

Post by ef47d35620c1 »

zatkin wrote: post #13
gizmo686 wrote: post #10

Earlier quoted context omitted.

From the mail: >This patch solves a problem where simultaneous reads to /dev/urandom can cause two processes on different processors to get the same value. We're not using a spinlock around the random generation loop because this will be a huge hit to preempt latency. So instead we just use a mutex around random_read and urandom_read. Yeah, it's not as efficient in the case of contention, if an application is calling…

So, would a possible solution be to check how many people are using the random generator at once? If only one process is currently accessing /dev/urandom, then avoid the spinlock and problem solved. Or, I am completely wrong.

Would a possible solution be to check how many people are using the random generator...?

One preson may have multiple processes reading from /dev/urandom.

Post reply on HN