Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux /dev/urandom and concurrency

drsnyder.us

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Linux /dev/urandom and concurrency

#51

Post by tptacek »

throwaway2048 wrote: post #27
jerf wrote: post #17

Earlier quoted context omitted.

If you're using crypto /rand to yank a whole bunch of random numbers out for the purpose of deciding which DNS record to use when multiple DNS records were returned, yes, the Go application is misdesigned. Such applications should be using math /rand. Seeding your math/rand from crypto/rand isn't a bad idea, but you don't need to be hammering on /dev/urandom in such code.

This "some random data is more important then other random data" musical chair dance going on with /dev/random vs /dev/urandom vs userland [CS]PRNGs (often gathering from extremely poor sources, or using broken algos) has been nothing short of an unmitigated security and useability disaster. We have the abillity to make the /dev/urandom CSPRNG secure enough and fast enough for (almost) any randomness purpose. We need…

Yes. Exactly this.

Re: Linux /dev/urandom and concurrency

#52

Post by tptacek »

ekimekim wrote: post #47
mike-cardwell wrote: post #33

Earlier quoted context omitted.

Ah. You appear to be right. I'm glad I asked now. [edit] I'm going to skip using the Mersenne twister engine and just use std::random_device for all random data, instead of as a seed. It seems on Linux at least that random_device is basically /dev/urandom. I assume the source will be sane on other OS's too.

Please don't do this. This exact approach is what caused the problems described in the article. If you need secure random numbers, do what you had above (though in light of other comments, perhaps consider a different algo besides MT).

This is a recommendation that says that a developer should actually consider using the Mersenne Twister for secure random numbers to avoid a potential performance problem.

Re: Linux /dev/urandom and concurrency

#53

Post by tptacek »

raverbashing wrote: post #29
mikeash wrote: post #21

Earlier quoted context omitted.

You don't need to be, but why not? It should be plenty fast and work well. If it's turning out to be too slow due to too much locking, that should be fixed.

1 - Rand is faster 2 - You don't need the crypto qualities of it and you're emptying the entropy pool for nothing 3 - You're doing much more work, especially if you're reading one byte at a time from /dev/urandom (doing a syscall, etc), while rand is just a calculation

There is in practice no such thing as "entropy depletion". The retail side of a CSPRNG is very similar to a stream cipher. The idea behind "entropy depletion" is structurally the same as the idea of a stream cipher "depleting its key". You can run AES-CTR as a stream cipher for several exbibytes before the output starts becoming distinguishable (which is not the same thing as "reveals the key").

Re: Linux /dev/urandom and concurrency

#54

Post by tptacek »

ekimekim wrote: post #46
azinman2 wrote: post #42

Earlier quoted context omitted.

And while that's interesting from the perspective of fetching web content via curl (and kudos to the author tracing it down to that), that doesn't mean the fundamental issue shouldn't be fixed. In the current security environment, from heartbleed to the NSA, it's becoming clear that security issues need to be systematically dealt with from an industry perspective or people will start to lose faith in secure Internet…

Are you also suggesting that the kernel provide a "great implementation" of SHA? Of SSL? Of https? Where do you draw the line? There's no reason for the kernel to provide standard library functions. In fact, I'd argue that syscalls should be reserved for only actions that cannot be done wholly in userspace (futex is a good example of this). The current model of "hardware randomness to seed a PRNG" makes sense. It is…

I would draw the line somewhere between SHA and SSL. Next question? ;)

Re: Linux /dev/urandom and concurrency

#55

Post by tptacek »

aidenn0 wrote: post #18

Seed a secure userspace PRNG from urandom, perhaps?

If you care about security, avoid this approach; it creates an additional single point of failure, which historically has also tended to be a very likely point of failure (see: Debian randomness, Android Java SecureRandom, &c).

Re: Linux /dev/urandom and concurrency

#56

Post by bch »

frankfarmer wrote: post #43
bch wrote: post #19

Earlier quoted context omitted.

Hard to say w/o seeing the data in question, but based on that, perhaps nscd or re-using curl handles could mitigate their frustration w/ runtime.

the c-ares init (which reads /dev/urandom) inside curl init happens even when DNS isn't used at all (even when making a request to 127.0.0.1), so it's pretty hard to avoid as long as curl is built with c-ares. The only way to mitigate is to remove c-ares or limit calls to curl init.

Ah... I've worked a lot w/ libcurl, and a bit w/ c-ares, but don't fully know how c-ares works w/i curl. Thanks for the rundown.

Re: "limit calls to curl init" -- do you mean curl_easy_init() ? In that case, reusing handles (eg:

  CURL  *handle
) would mitigate that, no?

Edit: This doesn't make sense. c-ares relationship must be in curl_easy_perform(). Now I'm curious:

  1) Am I correct re: c-ares / curl_easy_perform()
  2) Can one reuse CURL *handle and not invoke c-ares and /dev/urandom if one reuses the same domain name (but not necessarily the same URL) within a handle.

Re: Linux /dev/urandom and concurrency

#57

Post by acqq »

frankfarmer wrote: post #43
bch wrote: post #19

Earlier quoted context omitted.

Hard to say w/o seeing the data in question, but based on that, perhaps nscd or re-using curl handles could mitigate their frustration w/ runtime.

the c-ares init (which reads /dev/urandom) inside curl init happens even when DNS isn't used at all (even when making a request to 127.0.0.1), so it's pretty hard to avoid as long as curl is built with c-ares. The only way to mitigate is to remove c-ares or limit calls to curl init.

Isn't the solution exactly what is described here:

http://curl.haxx.se/libcurl/c/curl_easy_init.html

"If you did not already call curl_global_init(3), curl_easy_init(3) does it automatically. This may be lethal in multi-threaded cases, since curl_global_init(3) is not thread-safe, and it may result in resource problems because there is no corresponding cleanup."

I can imagine that only curl_global_init reads from urandom? Your application should do curl_global_init only once, then do other fetches each time using just curl_easy_init and cleanup.

Re: Linux /dev/urandom and concurrency

#58

Post by frankfarmer »

acqq wrote: post #57
frankfarmer wrote: post #43

Earlier quoted context omitted.

the c-ares init (which reads /dev/urandom) inside curl init happens even when DNS isn't used at all (even when making a request to 127.0.0.1), so it's pretty hard to avoid as long as curl is built with c-ares. The only way to mitigate is to remove c-ares or limit calls to curl init.

Isn't the solution exactly what is described here: http://curl.haxx.se/libcurl/c/curl_easy_init.html "If you did not already call curl_global_init(3), curl_easy_init(3) does it automatically. This may be lethal in multi-threaded cases, since curl_global_init(3) is not thread-safe, and it may result in resource problems because there is no corresponding cleanup." I can imagine that only curl_global_init reads from ura…

We're using libcurl via PHP (I know, I know) which doesn't expose curl_global_init at all.

Here's the stacktrace for the /dev/urandom read -- it's happening in curl_easy_init.

   Catchpoint 1 (call to syscall 'ioctl'), 0x0000003a74ecc4ba in tcgetattr () from /lib64/libc.so.6
   (gdb) backtrace
   #0  0x0000003a74ecc4ba in tcgetattr () from /lib64/libc.so.6
   #1  0x0000003a74ec7a1c in isatty () from /lib64/libc.so.6
   #2  0x0000003a74e60d51 in _IO_file_doallocate_internal () from /lib64/libc.so.6
   #3  0x0000003a74e6d6dc in _IO_doallocbuf_internal () from /lib64/libc.so.6
   #4  0x0000003a74e6ba7c in _IO_file_xsgetn_internal () from /lib64/libc.so.6
   #5  0x0000003a74e61dd2 in fread () from /lib64/libc.so.6
   #6  0x0000003341606414 in ares_init_options () from /usr/lib64/libcares.so.2
   #7  0x0000003d3404f0c9 in ?? () from /usr/lib64/libcurl.so.4
   #8  0x0000003d340242a5 in ?? () from /usr/lib64/libcurl.so.4
   #9  0x0000003d3402f9a6 in curl_easy_init () from /usr/lib64/libcurl.so.4
   #10 0x00002b35e0304fb0 in ?? () from /usr/lib64/php/modules/curl.so
   #11 0x0000000000606da9 in ?? ()
   #12 0x00000000006456b8 in execute_ex ()
   #13 0x00000000005d2bba in zend_execute_scripts ()
   #14 0x00000000005769ee in php_execute_script ()
   #15 0x000000000067e44d in ?? ()
   #16 0x000000000067ede8 in ?? ()
   #17 0x0000003a74e1d994 in __libc_start_main () from /lib64/libc.so.6
   #18 0x0000000000422b09 in _start ()
   (gdb)

Re: Linux /dev/urandom and concurrency

#59

Post by acqq »

throwaway2048 wrote: post #27
jerf wrote: post #17

Earlier quoted context omitted.

If you're using crypto /rand to yank a whole bunch of random numbers out for the purpose of deciding which DNS record to use when multiple DNS records were returned, yes, the Go application is misdesigned. Such applications should be using math /rand. Seeding your math/rand from crypto/rand isn't a bad idea, but you don't need to be hammering on /dev/urandom in such code.

This "some random data is more important then other random data" musical chair dance going on with /dev/random vs /dev/urandom vs userland [CS]PRNGs (often gathering from extremely poor sources, or using broken algos) has been nothing short of an unmitigated security and useability disaster. We have the abillity to make the /dev/urandom CSPRNG secure enough and fast enough for (almost) any randomness purpose. We need…

It's impossible (or just not worth the trade-off) to make one piece of software (this time, kernel) fast for any use case imaginable. In this case, kernel behaved correctly but with the speed degradation for extreme cases. That the author's Rube Goldberg machine then runs slow I don't consider kernel to be guilty.

The guy uses PHP and instead of built-in HTTPRequest he uses curl to make a request to "a bucketed key-value store built on PostgreSQL that speaks HTTP which uses Clojure and the Compojure web framework to provide a REST interface over HTTP." A bit of shooting the flies with cannons on every side?

On another side, if it can be proved that urandom has serious problems in reasonable use cases it should be checked what can be changed and how.

Re: Linux /dev/urandom and concurrency

#60

Post by mike-cardwell »

pbsd wrote: post #49
mike-cardwell wrote: post #33

Earlier quoted context omitted.

Ah. You appear to be right. I'm glad I asked now. [edit] I'm going to skip using the Mersenne twister engine and just use std::random_device for all random data, instead of as a seed. It seems on Linux at least that random_device is basically /dev/urandom. I assume the source will be sane on other OS's too.

The C++ standard gives little guarantees for the quality of std::random_device. As I remember, MinGW on Windows uses the Mersenne Twister with a hardcoded seed for it. boost::random_device, on the other hand, has better guarantees: it is only implemented where there's a decent entropy source.

Thanks for the info. Seeing as I'm already using boost, that sounds very useful.
Post reply on HN