Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux /dev/urandom and concurrency

drsnyder.us

Post reply on HN
Print view View original

41–50 of 80 posts

Re: Linux /dev/urandom and concurrency

#41

Post by rcoh »

The stdlib rand() function on unix has a global lock around it provided by many versions of Linux. As such, if rand() is called in performance critical parallel code, performance will tank as each thread or process attempts to acquire this lock. Even if this lock is not acquired, you will still have a race condition on the state of the random number generator and may produce bad (non-random) randomness.

Use rand_r(unsigned int *state) instead in parallel and concurrent applications.

Sources: man 3 rand [unix command] http://unixhelp.ed.ac.uk/CGI/man-cgi?rand+3

Re: Linux /dev/urandom and concurrency

#42

Post by azinman2 »

jerf wrote: post #38
helper wrote: post #26

Earlier quoted context omitted.

There are lots of applications that require a cryptographically secure random number generator, which math/rand doesn't claim to be.

This is not one of them.

And while that's interesting from the perspective of fetching web content via curl (and kudos to the author tracing it down to that), that doesn't mean the fundamental issue shouldn't be fixed.

In the current security environment, from heartbleed to the NSA, it's becoming clear that security issues need to be systematically dealt with from an industry perspective or people will start to lose faith in secure Internet communication, which would undermine too much of what's valuable about the Internet.

What we need is great APIs/frameworks/design patterns to simplify cryptography so that a newbie ruby on rails programmer CAN create actually secure applications and not even realize that it was complicated in the first place.

In cryptography you make one misstep and the entire chain is broken. It's thus important for things like the linux kernel to provide great implementations so that people don't think twice about using it and never want their own PRNG.

Re: Linux /dev/urandom and concurrency

#43

Post by frankfarmer »

bch wrote: post #19
mike-cardwell wrote: post #15

Earlier quoted context omitted.

So you can randomise the ID in the request packet to help protect against cache poisoning. And also so you can apply 0x20 bit (x) encoding to the qname for further protection. (x) http://courses.isi.jhu.edu/netsec/papers/increased_dns_resis...

Hard to say w/o seeing the data in question, but based on that, perhaps nscd or re-using curl handles could mitigate their frustration w/ runtime.

the c-ares init (which reads /dev/urandom) inside curl init happens even when DNS isn't used at all (even when making a request to 127.0.0.1), so it's pretty hard to avoid as long as curl is built with c-ares. The only way to mitigate is to remove c-ares or limit calls to curl init.

Re: Linux /dev/urandom and concurrency

#44

Post by azinman2 »

raverbashing wrote: post #31
mcpherrinm wrote: post #24

Earlier quoted context omitted.

Userland PRNGs are one of the easiest ways to introduce security vulnerabilities into your programs. I would recommend being very, VERY careful before trying to do this, like the traditional "Don't roll your own crypto" advice.

Of course. But there are a lot of needs for random numbers that don't need the random numbers to be secure.

In which case rand and the like really should be renamed unsecure_random to prevent confusion.

Re: Linux /dev/urandom and concurrency

#45

Post by frankfarmer »

ape4 wrote: post #8

Why does he need so much pseudorandomness. And why use /dev/urandom directly. Maybe using the random library from the programming environment would make more sense.

Simply initializing a curl handle causes the /dev/urandom read -- so a large number of parallel curl requests easily triggers this issue.

Re: Linux /dev/urandom and concurrency

#46

Post by ekimekim »

azinman2 wrote: post #42
jerf wrote: post #38

Earlier quoted context omitted.

This is not one of them.

And while that's interesting from the perspective of fetching web content via curl (and kudos to the author tracing it down to that), that doesn't mean the fundamental issue shouldn't be fixed. In the current security environment, from heartbleed to the NSA, it's becoming clear that security issues need to be systematically dealt with from an industry perspective or people will start to lose faith in secure Internet…

Are you also suggesting that the kernel provide a "great implementation" of SHA? Of SSL? Of https? Where do you draw the line?

There's no reason for the kernel to provide standard library functions. In fact, I'd argue that syscalls should be reserved for only actions that cannot be done wholly in userspace (futex is a good example of this). The current model of "hardware randomness to seed a PRNG" makes sense. It is up to the userspace libraries to provide good implementations.

Re: Linux /dev/urandom and concurrency

#47

Post by ekimekim »

mike-cardwell wrote: post #33
aidenn0 wrote: post #28

Earlier quoted context omitted.

I don't know what DNS uses the randomness for, but if a malicious attacker can gain from guessing the randomness, don't use MT, as the state can be extracted from MT by observing a relatively small number of outputs.

Ah. You appear to be right. I'm glad I asked now. [edit] I'm going to skip using the Mersenne twister engine and just use std::random_device for all random data, instead of as a seed. It seems on Linux at least that random_device is basically /dev/urandom. I assume the source will be sane on other OS's too.

Please don't do this. This exact approach is what caused the problems described in the article.

If you need secure random numbers, do what you had above (though in light of other comments, perhaps consider a different algo besides MT).

Re: Linux /dev/urandom and concurrency

#48

Post by ekimekim »

rcoh wrote: post #41

The stdlib rand() function on unix has a global lock around it provided by many versions of Linux. As such, if rand() is called in performance critical parallel code, performance will tank as each thread or process attempts to acquire this lock. Even if this lock is not acquired, you will still have a race condition on the state of the random number generator and may produce bad (non-random) randomness. Use rand_r(un…

The problem you're describing is similar but not the same as the one in the article. What you describe is part of the libc implementation of rand(3), whereas the article is talking about reads from /dev/urandom, which has a lock inside the kernel code (for the same reasons as libc).

Re: Linux /dev/urandom and concurrency

#49

Post by pbsd »

mike-cardwell wrote: post #33
aidenn0 wrote: post #28

Earlier quoted context omitted.

I don't know what DNS uses the randomness for, but if a malicious attacker can gain from guessing the randomness, don't use MT, as the state can be extracted from MT by observing a relatively small number of outputs.

Ah. You appear to be right. I'm glad I asked now. [edit] I'm going to skip using the Mersenne twister engine and just use std::random_device for all random data, instead of as a seed. It seems on Linux at least that random_device is basically /dev/urandom. I assume the source will be sane on other OS's too.

The C++ standard gives little guarantees for the quality of std::random_device. As I remember, MinGW on Windows uses the Mersenne Twister with a hardcoded seed for it.

boost::random_device, on the other hand, has better guarantees: it is only implemented where there's a decent entropy source.

Re: Linux /dev/urandom and concurrency

#50

Post by ekimekim »

mcpherrinm wrote: post #24
hosay123 wrote: post #22

Earlier quoted context omitted.

Adding to aidenn0's comment, if you trust /dev/urandom to produce 4kb of random data, it follows that you trust it to produce 128 bits. 128 bits (32 bytes) is sufficient to initialize a PRNG into any one of 115792089237316195423570985008687907853269984665640564039457584007913129639936 states (that's 1 with 77 digits). Consequently, hitting the kernel constantly for so much data is utterly inefficient in the first ins…

Userland PRNGs are one of the easiest ways to introduce security vulnerabilities into your programs. I would recommend being very, VERY careful before trying to do this, like the traditional "Don't roll your own crypto" advice.

Only in this case you aren't rolling your own crypto, but (hopefully) using a popular, field-tested CSPRNG available as a library for your language of choice. Unless you mistakenly pick a non-cryptographic PRNG, and assuming the implementation is correct (which is just as true of kernel code as it is of user code), I fail to see how you would introduce a vulnerability.
Post reply on HN