Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Myths about /dev/urandom (2014)

2uo.de

Post reply on HN
Print view View original

51–60 of 70 posts

Re: Myths about /dev/urandom (2014)

#51

Post by jMyles »

tptacek wrote: post #48
jMyles wrote: post #47

Earlier quoted context omitted.

> there is exactly one time in the lifecycle of an entire system when you want a blocking read from the Linux KRNG Right. I'm glad to hear you phrasing it this way. And listen: I don't mean to be flamy here, and it's possible that my sense of these things is tainted by our many... colorful discussions here on HN, but the unradom orthodoxy, to which getrandom and python's os.urandom are in part a response, is somethin…

The blog post you're referring to contains exactly the advice I just gave in the preceding comment. I don't see a link to a Michigan study in this thread. If you post it again, I'll be happy to explain why it doesn't say that you should use /dev/random, or a userspace RNG.

Well, man, that's not the debate. As so often in these conversations, you're running away with the goalposts. If I may advise: these are exactly the antics that are costing you credibility with those of us out here in the field actually building cryptographic systems.

Good day.

Re: Myths about /dev/urandom (2014)

#52

Post by tptacek »

jMyles wrote: post #51
tptacek wrote: post #48

Earlier quoted context omitted.

The blog post you're referring to contains exactly the advice I just gave in the preceding comment. I don't see a link to a Michigan study in this thread. If you post it again, I'll be happy to explain why it doesn't say that you should use /dev/random, or a userspace RNG.

Well, man, that's not the debate. As so often in these conversations, you're running away with the goalposts. If I may advise: these are exactly the antics that are costing you credibility with those of us out here in the field actually building cryptographic systems. Good day.

I don't understand your objections, like, at all. Every "concession" you think you're extracting is right there, in a blog post I wrote 4 years ago. Those are the goalposts. They haven't moved since 2014.

Everything else you have to say just reads like word salad. I should be concerned about container RNGs, but not Linux containers, unless they're the Linux containers that don't share the KRNG? Or the IOT devices where blocking somehow solves the cold-start entropy problem, and at the same time can't be done in the init scripts? Even the Python issue thread you posted --- yes, I read it --- disagrees with you. I have no idea what you're talking about and, candidly, I don't know that you do either.

I've written (until now) perhaps 1 word on this thread for every 30 of yours, and I think my comments are straightforward and make obvious points. You respond to each of them with a wall of text. Can I suggest that maybe instead of a grand disagreement, you've instead just done a poor job communicating whatever your concern might be?

Regarding that last sentence of yours (er, the one before your "good day"): I am extremely comfortable with who does and doesn't find me credible. Thanks anyways.

Re: Myths about /dev/urandom (2014)

#53

Post by Taniwha »

tptacek wrote: post #41
ChrisSD wrote: post #28

Earlier quoted context omitted.

Does increasing the rate of external entropy risk swamping the system generated entropy? I.e. if the external source is compromised could this also compromise urandom such that the results could become predictable?

No! This is the thing everyone needs to understand: once you have achieved "entropy" with a modern CSPRNG (including T'so's), you never lose that property . Think of the RNG like a stream cipher like AES-CTR. How much data do you have to encrypt with AES-CTR to "lose" or "leak" the underlying AES key?

Here I have to disagree (in the small) essentially the problem of 'breaking' a RNG is discovering that hidden state (the entropy pool) behind the CSPRNG - if you believe that the NSA, and their competitors, don't each have a team of PhDs pouring over the linux random code looking for ways to do that you're dreaming. After all if you know its state you can predict the keys it will make in the future

My intuition is that every operation we do (start TCP connections, generate keys, backoff after a failed login, ....) reveals a tiny fraction of a bit about that hidden state, web assembly calling random over and over again probably reveals much much more, over time it builds up.

(anyone want to guess if there's a quantum computer attack here, it kind of looks like the sort of problem where there might be)

So I think that with a sophisticated enough attacker entropy does degrade, the trick is to replace that entropy at a faster rate that the rate at which you guess the worst case attacker might be able to guess.

For most people it probably doesn't matter, you're exposing so little state over time you're likely effectively as safe as the parent poster claims, it kind of depends on whether, in this age when we know that nation state actors are watching every packet that passes thru their hands, watching every thing we're doing online, are you interesting enough for them to spend the time and effort? does the US/5Is think you're not a Trump supporter? are the Chinese interested in your trade secrets? whi knows what the Russians think

As I mentioned above using /dev/urandom doesn't have a mechanism you can use to signal this guess about how much entropy it needs over time to be safe, prior to 4.8 we used to have a mechanism that said "give it some more entropy every N seconds, 60 by default" but that's now broken

(provided here a python applet that periodically drains a little bit of /dev/random entropy if it has plenty available to stir the input pool that /dev/urandom uses for its entropy source)

Re: Myths about /dev/urandom (2014)

#54

Post by Taniwha »

acqq wrote: post #24
Taniwha wrote: post #13

Earlier quoted context omitted.

BTW - a tiny bit of python to keep /dev/urandom fed post 4.8 - useful if you have a hardware RNG (either an external one or the one in your CPU): #! /usr/bin/python # # OneRNG /dev/urandom tickler # (C) Copyright Paul Campbell Moonbase Otago 2018 # Released under a GPL 3.0 license # import time with open("/dev/random", 'rb') as r: while True: with open("/proc/sys/kernel/random/entropy_avail", 'r') as e: iv = e.read()…

GPL 3.0 for a few obvious lines seems to be really an unfitting license. Non-licensed English language version, containing the same information: Once per second, read /proc/sys/kernel/random/entropy_avail and if the resulting number is more than 1500 perform a read of 1000 bytes from /dev/random.

sorry, just grabbed my standard disclaimer, can't release code without copyrighting it first. As others have pointed out it's trivial to reimplement in the language of your choice go for it

Re: Myths about /dev/urandom (2014)

#55

Post by tptacek »

Taniwha wrote: post #53
tptacek wrote: post #41

Earlier quoted context omitted.

No! This is the thing everyone needs to understand: once you have achieved "entropy" with a modern CSPRNG (including T'so's), you never lose that property . Think of the RNG like a stream cipher like AES-CTR. How much data do you have to encrypt with AES-CTR to "lose" or "leak" the underlying AES key?

Here I have to disagree (in the small) essentially the problem of 'breaking' a RNG is discovering that hidden state (the entropy pool) behind the CSPRNG - if you believe that the NSA, and their competitors, don't each have a team of PhDs pouring over the linux random code looking for ways to do that you're dreaming. After all if you know its state you can predict the keys it will make in the future My intuition is th…

The point is that the LRNG isn't unknowable, and if the NSA can break it, they can break a lot of other stuff. The LRNG isn't built on AES-CTR (but it just as easily could have been); it's (last I checked) essentially a sort of PRF built on SHA1. All the machinations in the LRNG with multiple pools and the elaborate mixing function and the counters are just plumbing around what is essentially a keystream generated from SHA1 "keyed" by "entropy". The "entropy updates" the kernel rekey that PRF, not because the PRF is weakening but because a vulnerability (orthogonal to the LRNG) could expose kernel memory and thus the LRNG's state, and the updates allow it to recover from that kind of compromise.

That's a lot of words because the LRNG is overly complicated. But let's make it clearer: what you're saying about every TCP sequence number "revealing" part of the "hidden state" is exactly equivalent to the argument that every byte of AES-CTR ciphertext is revealing the hidden AES key. If that were the case, AES would be grievously broken.

If you don't believe that AES-CTR and the LRNG are roughly equivalent, try this thought experiment: retain the initialization functionality from the current LRNG --- building a pool of mixed and hashed unpredictable data from kernel events --- and scrap the rest of the LRNG, replacing it with... AES-CTR, using a hash of the entropy pool as its key. Reads from random/urandom (or kernel random reads, as with your sequence numbers) just tap bytes from the AES-CTR keystream. Is every TCP sequence number still revealing a little bit of the RNG state?

The Python script seems entirely silly to me; once the first read from /dev/random succeeds, you're guaranteed to have a fully initialized LRNG, and every subsequent read from urandom will be unpredictable and indistinguishable from random. Nothing is served by updating the LRNG after that point.

Re: Myths about /dev/urandom (2014)

#56

Post by Taniwha »

ChrisSD wrote: post #28
Taniwha wrote: post #5

(disclaimer I sell a HW random number generator) I think both the diagrams here are simplistic - in reality pre 4.8 there were 3 pools (an input pool and 2 output pools, one of which blocked) and post 4.8 where there are 2 pools, an input pool and a blocking pool (urandom now pulls from the input pool thru a CSPRNG). One big downside of the new (post 4.8) architecture is that urandom_min_reseed_secs is ignored - pre-…

Does increasing the rate of external entropy risk swamping the system generated entropy? I.e. if the external source is compromised could this also compromise urandom such that the results could become predictable?

You have to be root to add entropy to /dev/*random so you have to trust root - mind you many distros (ubuntu for example) use the CPU's RNG to provide external entropy by default, they trust Intel, you have to decide if you do too.

However kernel entropy is real, adding more entropy to the entropy pool is always is always adding more entropy. If the pool had 1000 bits in it and you add 5 bits from the kernel and a fake 1000 bits from a bogus external source it still has 1005 bits of entropy (however /dev/random thinks it will be able to extract 2005 bits)

We actually encourage our customers who are particularly paranoid to lie to the kernel about the quality of the entropy our RNG generates - saying the entropy quality is lower than it really is causes more good quality data to be mixed into the kernel pool (of course the defaults don't do this)

The kernel pools are actually quite small (4k bits) so there's a real practical limit to how much you can play this sort of game

Re: Myths about /dev/urandom (2014)

#57

Post by Taniwha »

paavoova wrote: post #25
Taniwha wrote: post #13

Earlier quoted context omitted.

BTW - a tiny bit of python to keep /dev/urandom fed post 4.8 - useful if you have a hardware RNG (either an external one or the one in your CPU): #! /usr/bin/python # # OneRNG /dev/urandom tickler # (C) Copyright Paul Campbell Moonbase Otago 2018 # Released under a GPL 3.0 license # import time with open("/dev/random", 'rb') as r: while True: with open("/proc/sys/kernel/random/entropy_avail", 'r') as e: iv = e.read()…

entropy_avail returns bits of entropy to a maximum of 4096(?), but this reads 1000 bytes. Why check if it's greater than 1500 if you're going to use it all anyway? Am I missing something here?

remember that this is for people with external entropy sources, the kernel entropy will refill fast. 1500 is more a measure of "is /dev/random busy at the moment" if it is then it's pulling data from from the external entropy source and /dev/urandom is essentially already being fed with more entropy

The missing mechanism we're trying to emulate used to essentially read N bytes (whatever user space happened to want at the time) every M seconds our experience has been that that can be way more than 1000 bytes, but reduce it if you like

Re: Myths about /dev/urandom (2014)

#58

Post by Taniwha »

tptacek wrote: post #55
Taniwha wrote: post #53

Earlier quoted context omitted.

Here I have to disagree (in the small) essentially the problem of 'breaking' a RNG is discovering that hidden state (the entropy pool) behind the CSPRNG - if you believe that the NSA, and their competitors, don't each have a team of PhDs pouring over the linux random code looking for ways to do that you're dreaming. After all if you know its state you can predict the keys it will make in the future My intuition is th…

The point is that the LRNG isn't unknowable, and if the NSA can break it, they can break a lot of other stuff. The LRNG isn't built on AES-CTR (but it just as easily could have been); it's (last I checked) essentially a sort of PRF built on SHA1. All the machinations in the LRNG with multiple pools and the elaborate mixing function and the counters are just plumbing around what is essentially a keystream generated fr…

I think you misunderstand my point, we're continually revealing tiny fractions of a bit of the internal state = hopefully really tiny fractions, an AES stream doesn't reveal the key, but other places reveal the direct output of the random number generator (TCP sessions are a great example, various nonces), all those tiny fractions can add up, but provided we add entropy faster than we expose all those little bits we're probably OK

Re: Myths about /dev/urandom (2014)

#59

Post by Tomte »

rini17 wrote: post #45
shawn wrote: post #43

Earlier quoted context omitted.

128 bits can be expanded into an infinite stream of unpredictable output. If those 128 bits are not known, then it is guaranteed to be secure.

But you can't do this "those 128 bits are used but are kept not known" thing with software. Sure, it is being tried again and again, and it fails every time. You need separate hardware to safely prevent these 128bits from leaking.. but then why don't we ditch whole idea and implement true HW RNG instead?

Because we don't generate random numbers to look at them and marvel at their philosophical pureness, we use them. In software.

So those bits will be known to software, no matter where the generation happens.

Your argument would make a bit of sense if you were arguing for doing all cryptographic operations in hardware. Like on a smartcard. But that's not what we're talking about.

Re: Myths about /dev/urandom (2014)

#60

Post by wolf550e »

Taniwha wrote: post #58
tptacek wrote: post #55

Earlier quoted context omitted.

The point is that the LRNG isn't unknowable, and if the NSA can break it, they can break a lot of other stuff. The LRNG isn't built on AES-CTR (but it just as easily could have been); it's (last I checked) essentially a sort of PRF built on SHA1. All the machinations in the LRNG with multiple pools and the elaborate mixing function and the counters are just plumbing around what is essentially a keystream generated fr…

I think you misunderstand my point, we're continually revealing tiny fractions of a bit of the internal state = hopefully really tiny fractions, an AES stream doesn't reveal the key, but other places reveal the direct output of the random number generator (TCP sessions are a great example, various nonces), all those tiny fractions can add up, but provided we add entropy faster than we expose all those little bits we'…

If the output of the random number generator used in tcp sessions is aes-ctr keyed by hash of entropy pool and aes does not reveal the key, then the output of the random number generator used in tcp sessions does not reveal the entropy pool.

What you wrote is magical thinking.

Post reply on HN