Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hashpipe – Pipe iff the hash matches

jbenet.github.io

Post reply on HN
Print view View original

81–90 of 90 posts

Re: Hashpipe – Pipe iff the hash matches

#81

Post by digi_owl »

_prometheus wrote: post #57
mpeg wrote: post #52

I think this is a topic worth raising, I spend so much time auditing the "magic" in open source projects to try and find out how they work behind the scenes, and whether they are doing anything to protect me. For example: Pip (python package manager) tells you to run a "secure" script off the internet [0], where the only real protection is being behind SSL. But, surely I can just check the code of that script? Oh wai…

> I like what CoreOS does, they don't even provide SSL download links for their isos, it's all plain text http, so they stress that you need to verify the image signature against their GPG signing key which is distributed via github. Sounds very much like TRTTD. I'll see about this with `sigpipe` (hashpipe with signatures). The fundamental problem is PKI is weak. using github isn't great either. We need a chain of tr…

What is "sane" when you are potentially up against attackers that may have clusters to brute force collision with?

Re: Hashpipe – Pipe iff the hash matches

#82

Post by Nullabillity »

vortico wrote: post #25

Deployment one-liners with hashpipe will only work if hashpipe is installed, which would be equally difficult for users to install properly than the software itself. Then you'd need something like this: > Simply install using `curl hashpi.pe | bash`

Well, it'd be easier to get hashpipe into all the typical package managers than to get every piece of software ever into them (though that way would of course be preferable).

Re: Hashpipe – Pipe iff the hash matches

#83

Post by e12e »

hueving wrote: post #75
mpeg wrote: post #52

I think this is a topic worth raising, I spend so much time auditing the "magic" in open source projects to try and find out how they work behind the scenes, and whether they are doing anything to protect me. For example: Pip (python package manager) tells you to run a "secure" script off the internet [0], where the only real protection is being behind SSL. But, surely I can just check the code of that script? Oh wai…

>so they stress that you need to verify the image signature against their GPG signing key which is distributed via github. If someone has the ability to MiTM HTTPS connections, they could just as easily MiTM the victim's connection to github to return a different GPG signing key. Also, the protection requires active checking rather than coming 'for free' like it does by downloading via HTTPS, so 95%+ of the users won…

If you use CoreOS, you just need to get the Core OS key one time, and sign it with your own key, then let the keyservers mirror it for you. An adversary can still make sure you can't connect to any key servers -- but it's an order of magnitude easier to verify a single GPG key (even if that involves getting on a plane and talking to a CoreOS developer in person) -- than it is to verify every single CA cert shipped with your browser.

Just look at the latest CNNIC key removal: essentially Google and Firefox are saying that SSL was never secure, because we couldn't trust CNNIC, even though everyone shipped their keys.

Before that, no browser was secure, because a number of CAs were compromised, and or incompetent.

While GPG isn't some magic dust that makes trust easy -- it's crazy to claim that SSL is somehow more secure, or more importantly, easier to trust.

[edit: And that doesn't even touch on the number of lines of code involved, the directness of the path between what is signed, and who will use it (security of the web server, dns etc not an issue if you know you can trust the GPG signature...]

Re: Hashpipe – Pipe iff the hash matches

#84

Post by e12e »

pritambaral wrote: post #47

I think this one isn't too far from useability: curl -o filename url && sha256sum -c And works without requiring installation of third-party tools too!

Maybe we should just recommend that software is distributed as a (not detached) singed file. So:

   curl http://yolo.example.com/lulz.gpg| gpg -d - | sh
(Which of course doesn't work either, as one could replace lulz.gpg with "#/usr/bin/env sh;rm -rf /" signed by some key one trusts... which isn't so far fetched assuming some trusted entity publishes a utility for wiping a system automatically (think: shred /dev/sd*))...

Still, gpg -d, seems preferable to detached signatures (not to mention detached hashes) for this type of thing? The files can't be used without verifying, or running through gpg [ed: or other manual intervention, for shooting oneself in the foot] -- and there is a single, sane way to do that.

Re: Hashpipe – Pipe iff the hash matches

#85

Post by e12e »

tbrownaw wrote: post #63
comex wrote: post #53

Earlier quoted context omitted.

This is non-portable, as OS X has no sha256sum out of the box. But it does have shasum from Perl, and Linux distros typically come with Perl, so 'shasum -ba256' should work... I wonder if there is any concise way to do this without needing to save a file to disk. I can't think of one, as it requires splitting the input in two, which bash can do with >(command) but not sequentially or with the ability to communicate f…

You need to buffer the entire input somewhere . You don't get your hash without consuming the entire input. You can't start feeding the input to the shell, until you verify the hash. You don't want to download it twice, it case it comes back different the second time (or in case your network is slow). At the point that you are verifying the hash, the entire file must exist on your system somewhere . Disk, memory, OCR…

> Buffering to memory can only work if the input is "small", whatever that means.

Well, increasingly it means "up to 8GB or more". Or put another way, RAM is increasing faster than network speed (or the speed of light for that matter...). So, I'd say that, yes, for many things you'd want to download from the internet, caching in RAM is absolutely an option?

Re: Hashpipe – Pipe iff the hash matches

#86

Post by e12e »

_prometheus wrote: post #58
RegEx wrote: post #56

Earlier quoted context omitted.

This is a really, really interesting point! In addition to your point, (and perhaps this point has already been made elsewhere in the thread), but the type of people that would install hashpipe would be the kind of people to avoid the practice entirely. Perhaps there's some mechanism for convincing people to use the tool, and if that mechanism ends up being easier than convincing people to stop piping directly to bas…

yeah, this is just a stopgap for me. in the end i want: ipfs daemon --mount & sh /ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6 (this actually works, btw :)

Thank's for pimpong ipfs -- I hadn't heard of it before. Initial discussion on hn:

https://news.ycombinator.com/item?id=8069836

github(?): https://github.com/ipfs/go-ipfs

Recent re-submission of new(?) home page: https://news.ycombinator.com/item?id=9321209

Re: Hashpipe – Pipe iff the hash matches

#87

Post by mpeg »

hueving wrote: post #75
mpeg wrote: post #52

I think this is a topic worth raising, I spend so much time auditing the "magic" in open source projects to try and find out how they work behind the scenes, and whether they are doing anything to protect me. For example: Pip (python package manager) tells you to run a "secure" script off the internet [0], where the only real protection is being behind SSL. But, surely I can just check the code of that script? Oh wai…

>so they stress that you need to verify the image signature against their GPG signing key which is distributed via github. If someone has the ability to MiTM HTTPS connections, they could just as easily MiTM the victim's connection to github to return a different GPG signing key. Also, the protection requires active checking rather than coming 'for free' like it does by downloading via HTTPS, so 95%+ of the users won…

SSL does nothing to guarantee the integrity of code you download and run off the internet.

I'm not worried about MITM attacks, SSL provides adequate protection against those in most cases; but if someone gains access to the coreos systems SSL would not help.

In this case, I have a GPG key saved for the coreos image signing, which hopefully is done offline and I got that through a third party which I have a certain amount of trust on (github), and I can always verify it on other channels.

Re: Hashpipe – Pipe iff the hash matches

#89

Post by hueving »

mpeg wrote: post #87
hueving wrote: post #75

Earlier quoted context omitted.

>so they stress that you need to verify the image signature against their GPG signing key which is distributed via github. If someone has the ability to MiTM HTTPS connections, they could just as easily MiTM the victim's connection to github to return a different GPG signing key. Also, the protection requires active checking rather than coming 'for free' like it does by downloading via HTTPS, so 95%+ of the users won…

SSL does nothing to guarantee the integrity of code you download and run off the internet. I'm not worried about MITM attacks, SSL provides adequate protection against those in most cases; but if someone gains access to the coreos systems SSL would not help. In this case, I have a GPG key saved for the coreos image signing, which hopefully is done offline and I got that through a third party which I have a certain am…

But did you verify it on other channels? Or did you do like 99.9% of users and assume nobody would be attacking you? My point is mainly that gpg is such a usability nightmare that it's effectively a broken security model.

Re: Hashpipe – Pipe iff the hash matches

#90

Post by KMag »

_prometheus wrote: post #37
gojomo wrote: post #29

Earlier quoted context omitted.

When the just-over-the-next-hilltop promised-land nirvana of content-centric networking arrives, the hash will be enough to locate & download the content – so you shouldn't even need an URL: $ hashcurl QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 | sh Maybe it's even a special filesystem path, that contains (but does not list) everything-that's-nameable-and-findable: $ sh /everything/QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6…

this already works. install ipfs: http://ipfs.io/docs/install then: ipfs init ipfs daemon & sleep 20 # sorry this will go away ipfs mount sh /ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6 you can see it on the web at: http://gateway.ipfs.io/ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4...

If any of you are designing a system like ipfs, please use Merkle tree roots as the identifiers (and make sure leaf nodes are hashed differently than inner nodes as in THEX, or better yet Joan Daemen's Sakura construction).

The main reason is that at some point you probably want to support downloading from multiple untrusted sources. Properly implemented Merkle trees (such as using the Sakura construction) are provably as strong as the underlying hash algorithm and allow lightweight cryptographic proof that a given block belongs to the file in question and belongs in the claimed place.

Gnutella uses the SHA-1 of the whole file the identifier and then just assumes the first peer to give it a Merkle tree root is giving it the correct root. Because there isn't guaranteed consistency between the identifier and the tree root, this makes it vulnerable to an attack where an attacker tries hard to be first, and give it a root for a corrupted version of the file and the peer wastes a lot of time and bandwidth re-downloading uncorrupted blocks that fail to verify.

Alternatively, you could make the identifier the concatenation of a full-file hash and a Merkle tree root. However, since the Sakura construction (and several other constructions) are provably as strong as the underlying hash, this is a waste of space. Concatenating the SHA-256 of the file with a SHA-256 Merkle tree root only gives you as much strength as the first 257 bits of a SHA-512 Merkle tree root. If you can spare the space for the longer identifiers, you're better off using a longer hash instead of two shorter hashes.

In other words, it's a waste of space to concatenate a whole-file hash with a Merkle tree root if you use a tree construction that's provably as strong as the underlying hash function.

Post reply on HN