Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux /dev/urandom and concurrency

drsnyder.us

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Linux /dev/urandom and concurrency

#61

Post by acqq »

frankfarmer wrote: post #58
acqq wrote: post #57

Earlier quoted context omitted.

Isn't the solution exactly what is described here: http://curl.haxx.se/libcurl/c/curl_easy_init.html "If you did not already call curl_global_init(3), curl_easy_init(3) does it automatically. This may be lethal in multi-threaded cases, since curl_global_init(3) is not thread-safe, and it may result in resource problems because there is no corresponding cleanup." I can imagine that only curl_global_init reads from ura…

We're using libcurl via PHP (I know, I know) which doesn't expose curl_global_init at all. Here's the stacktrace for the /dev/urandom read -- it's happening in curl_easy_init. Catchpoint 1 (call to syscall 'ioctl'), 0x0000003a74ecc4ba in tcgetattr () from /lib64/libc.so.6 (gdb) backtrace #0 0x0000003a74ecc4ba in tcgetattr () from /lib64/libc.so.6 #1 0x0000003a74ec7a1c in isatty () from /lib64/libc.so.6 #2 0x0000003a7…

I just guess, but maybe the things missing in the stack trace (marked with ?? -- maybe you miss some debug symbols?) do what is described in the manual entry I've quoted (calling global init which calls ares?).

And why curl at all? PHP has built in HTTPRequest?

Re: Linux /dev/urandom and concurrency

#62

Post by acqq »

bodyfour wrote: post #9
Glyptodon wrote: post #6

So why is there a lock for reads from urandom? I suppose if there weren't a lock concurrent reads would all get the same random values?

Yeah basically. That could be a disaster for, say, nonce generation. The solution would be to have multiple independent entropy pools and either bind them to cores(/sets of cores) or pick a non-busy one in a contention case.

Yes, if there is no a urandom generator per core, it would be convenient for some extreme cases to introduce such. The question is if it's worth the effort and the resulted "bloat" of the kernel code and memory usage. Linux runs on some very small devices too and even there decent user-space programmers can easily do their own per-thread generation in their programs. Normal uses of crypto are such: you initialize your own crypto once, then produce a lot of data in your own space.

If urandom is really "one for all cores" somebody should be able to demonstrate the speed drop by just writing some bash script? Volunteers?

Re: Linux /dev/urandom and concurrency

#63

Post by bch »

acqq wrote: post #57
frankfarmer wrote: post #43

Earlier quoted context omitted.

the c-ares init (which reads /dev/urandom) inside curl init happens even when DNS isn't used at all (even when making a request to 127.0.0.1), so it's pretty hard to avoid as long as curl is built with c-ares. The only way to mitigate is to remove c-ares or limit calls to curl init.

Isn't the solution exactly what is described here: http://curl.haxx.se/libcurl/c/curl_easy_init.html "If you did not already call curl_global_init(3), curl_easy_init(3) does it automatically. This may be lethal in multi-threaded cases, since curl_global_init(3) is not thread-safe, and it may result in resource problems because there is no corresponding cleanup." I can imagine that only curl_global_init reads from ura…

curl_global_init() really ought to be called on module load, before any (eg) set handle = curl::new is presented to a module user. And ea. curl::new call out to curl_easy_init(), will bypass global init again w/ this first line of code:

   if(initialized++)
    return CURLE_OK;
Correct?

Re: Linux /dev/urandom and concurrency

#64

Post by raverbashing »

tptacek wrote: post #53
raverbashing wrote: post #29

Earlier quoted context omitted.

1 - Rand is faster 2 - You don't need the crypto qualities of it and you're emptying the entropy pool for nothing 3 - You're doing much more work, especially if you're reading one byte at a time from /dev/urandom (doing a syscall, etc), while rand is just a calculation

There is in practice no such thing as "entropy depletion". The retail side of a CSPRNG is very similar to a stream cipher. The idea behind "entropy depletion" is structurally the same as the idea of a stream cipher "depleting its key". You can run AES-CTR as a stream cipher for several exbibytes before the output starts becoming distinguishable (which is not the same thing as "reveals the key").

True, unfortunately /dev/random blocking "soon" in Linux helps to propagate this myth. I stand corrected.

Re: Linux /dev/urandom and concurrency

#65

Post by raverbashing »

azinman2 wrote: post #44
raverbashing wrote: post #31

Earlier quoted context omitted.

Of course. But there are a lot of needs for random numbers that don't need the random numbers to be secure.

In which case rand and the like really should be renamed unsecure_random to prevent confusion.

Fine by me

Re: Linux /dev/urandom and concurrency

#66

Post by acqq »

azinman2 wrote: post #42
jerf wrote: post #38

Earlier quoted context omitted.

This is not one of them.

And while that's interesting from the perspective of fetching web content via curl (and kudos to the author tracing it down to that), that doesn't mean the fundamental issue shouldn't be fixed. In the current security environment, from heartbleed to the NSA, it's becoming clear that security issues need to be systematically dealt with from an industry perspective or people will start to lose faith in secure Internet…

Why does kernel have to be "fixed" if anything in the current "APIs/frameworks/deseign patterns" is not doing its own homework?

It seems the author admits in the comments: "All the application needs to do is open a socket and generate a GET request." So why complaining about the kernel?

If there's problem with urandom, demonstrate it on the reasonable use case example, don't try to impress anybody by showing how much different libraries, modules and programs you combine for one key-value query.

Re: Linux /dev/urandom and concurrency

#68

Post by ape4 »

frankfarmer wrote: post #45
ape4 wrote: post #8

Why does he need so much pseudorandomness. And why use /dev/urandom directly. Maybe using the random library from the programming environment would make more sense.

Simply initializing a curl handle causes the /dev/urandom read -- so a large number of parallel curl requests easily triggers this issue.

Thanks for the reply.

Re: Linux /dev/urandom and concurrency

#69

Post by nubs »

acqq wrote: post #59
throwaway2048 wrote: post #27

Earlier quoted context omitted.

This "some random data is more important then other random data" musical chair dance going on with /dev/random vs /dev/urandom vs userland [CS]PRNGs (often gathering from extremely poor sources, or using broken algos) has been nothing short of an unmitigated security and useability disaster. We have the abillity to make the /dev/urandom CSPRNG secure enough and fast enough for (almost) any randomness purpose. We need…

It's impossible (or just not worth the trade-off) to make one piece of software (this time, kernel) fast for any use case imaginable. In this case, kernel behaved correctly but with the speed degradation for extreme cases. That the author's Rube Goldberg machine then runs slow I don't consider kernel to be guilty. The guy uses PHP and instead of built-in HTTPRequest he uses curl to make a request to "a bucketed key-v…

Slightly off topic, but I wanted to clear this up:

> The guy uses PHP and instead of built-in HTTPRequest he uses curl to make a request

HTTPRequest is not built-in to PHP. It is a PECL extension that is usually installed separately from PHP.

Curl is more built-in to PHP - it's a PHP compile-time flag, and it is distributed with PHP source.

Re: Linux /dev/urandom and concurrency

#70

Post by acqq »

nubs wrote: post #69
acqq wrote: post #59

Earlier quoted context omitted.

It's impossible (or just not worth the trade-off) to make one piece of software (this time, kernel) fast for any use case imaginable. In this case, kernel behaved correctly but with the speed degradation for extreme cases. That the author's Rube Goldberg machine then runs slow I don't consider kernel to be guilty. The guy uses PHP and instead of built-in HTTPRequest he uses curl to make a request to "a bucketed key-v…

Slightly off topic, but I wanted to clear this up: > The guy uses PHP and instead of built-in HTTPRequest he uses curl to make a request HTTPRequest is not built-in to PHP. It is a PECL extension that is usually installed separately from PHP. Curl is more built-in to PHP - it's a PHP compile-time flag, and it is distributed with PHP source.

So the best approach for the given problem (just send the request, fetch the response, without too much overhead) seems to be using something a bit lower level:

http://at2.php.net/stream_socket_client

Post reply on HN