Hashpipe – Pipe iff the hash matches • Hacker News Board

Re: Hashpipe – Pipe iff the hash matches

#51

Post by kazinator » Sat, Apr 04, 2015, 12:43 AM UTC

tyho wrote: post #5

Be warned that this reads everything until EOF into memory, doing something like: hashpipe QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 will produce unexpected behaviour.

It doesn't have to read into memory; if that becomes a problem they can fix it. The hashing algorithm just needs a small, constant amount of state. While it is hashing, it can divert the data to a temporary file. Then it can just pass the file to its standard output.

A "hashexec" command could also be produced which passes the name of this temporary file to a subprogram:

   whatever | hashexec  arbitrary --command with args

This "arbitrary" command's standard input is redirected from a temporary file created by hashexec (so no cat-like loop has to execute to feed the data).

Also, since this is for scripts, there could be an argument which limits the size. This could have default value, say one megabyte. Anyone pulling down scripts which are anywhere near one megabyte has to add explicit overrides for the size:

   whatever | hashpipe --max=4M  | ...

The real problem isn't that hashpipe buffers everything, because the scripting language running the script will also do that; rather that because it buffers everything, it can be DDoS'ed with an infinite stream or whatever.

By incorporating a size limit, hashpipe could provide an additional protection measure to the next pipeline element: it protects against content which doesn't match the hash, and against content which is too large.

Idea: the size of the input could be encoded as a few digits of the hash. Then the argument is unnecessary. The hash itself tells you that the script is exactly 6059 bytes long; if you read 6060 bytes, the input is not the right one. Or the program could just stop reading at 6059 and check the hash at that point, and either pass on the 6059 bytes or error out.

Re: Hashpipe – Pipe iff the hash matches

#52

Post by mpeg » Sat, Apr 04, 2015, 12:47 AM UTC

I think this is a topic worth raising, I spend so much time auditing the "magic" in open source projects to try and find out how they work behind the scenes, and whether they are doing anything to protect me.

For example:

Pip (python package manager) tells you to run a "secure" script off the internet [0], where the only real protection is being behind SSL. But, surely I can just check the code of that script? Oh wait, it's embedding a zip file inside the python script, so I need to manually unzip it and look into it before I can start to consider it safe.

I like what CoreOS does, they don't even provide SSL download links for their isos, it's all plain text http, so they stress that you need to verify the image signature against their GPG signing key which is distributed via github.

[0] https://bootstrap.pypa.io/get-pip.py

Re: Hashpipe – Pipe iff the hash matches

#53

Post by comex » Sat, Apr 04, 2015, 12:57 AM UTC

pritambaral wrote: post #47

I think this one isn't too far from useability: curl -o filename url && sha256sum -c And works without requiring installation of third-party tools too!

This is non-portable, as OS X has no sha256sum out of the box. But it does have shasum from Perl, and Linux distros typically come with Perl, so 'shasum -ba256' should work...

I wonder if there is any concise way to do this without needing to save a file to disk. I can't think of one, as it requires splitting the input in two, which bash can do with >(command) but not sequentially or with the ability to communicate from the subshell to the outer one.

Re: Hashpipe – Pipe iff the hash matches

#54

Post by gojomo » Sat, Apr 04, 2015, 1:07 AM UTC

stouset wrote: post #35

gojomo wrote: post #29

Earlier quoted context omitted.

When the just-over-the-next-hilltop promised-land nirvana of content-centric networking arrives, the hash will be enough to locate & download the content – so you shouldn't even need an URL: $ hashcurl QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 | sh Maybe it's even a special filesystem path, that contains (but does not list) everything-that's-nameable-and-findable: $ sh /everything/QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6…

It's not save a few characters. It's to allow for flexibility of encoding. "sha1" all but requires ASCII. Sometimes you're limited to hex, base64, etc.

Not sure I understand; almost any context where "QmTpn…" can appear could also handle a "sha1:" human-readable prefix. Including, notably, this shell context.

If another context is sufficiently different, it would be fine to use a controlled fixed-length binary-vocabulary there. (But, the ASCII bytes for "sha1:" would still be a pretty robust and largely self-documenting choice.)

Re: Hashpipe – Pipe iff the hash matches

#55

Post by cheald » Sat, Apr 04, 2015, 1:24 AM UTC

I am rather amused that the author provides precompiled binaries without hash checksums. Heh.

Re: Hashpipe – Pipe iff the hash matches

#56

Post by RegEx » Sat, Apr 04, 2015, 1:43 AM UTC

vortico wrote: post #25

Deployment one-liners with hashpipe will only work if hashpipe is installed, which would be equally difficult for users to install properly than the software itself. Then you'd need something like this: > Simply install using `curl hashpi.pe | bash`

This is a really, really interesting point!

In addition to your point, (and perhaps this point has already been made elsewhere in the thread), but the type of people that would install hashpipe would be the kind of people to avoid the practice entirely.

Perhaps there's some mechanism for convincing people to use the tool, and if that mechanism ends up being easier than convincing people to stop piping directly to bash, it sounds like it would be worth pursuing :)

Re: Hashpipe – Pipe iff the hash matches

#57

Post by _prometheus » Sat, Apr 04, 2015, 1:44 AM UTC

mpeg wrote: post #52

I think this is a topic worth raising, I spend so much time auditing the "magic" in open source projects to try and find out how they work behind the scenes, and whether they are doing anything to protect me. For example: Pip (python package manager) tells you to run a "secure" script off the internet [0], where the only real protection is being behind SSL. But, surely I can just check the code of that script? Oh wai…

> I like what CoreOS does, they don't even provide SSL download links for their isos, it's all plain text http, so they stress that you need to verify the image signature against their GPG signing key which is distributed via github.

Sounds very much like TRTTD. I'll see about this with `sigpipe` (hashpipe with signatures). The fundamental problem is PKI is weak. using github isn't great either.

We need a chain of trust that's sane (CA system is not sane).

Re: Hashpipe – Pipe iff the hash matches

#58

Post by _prometheus » Sat, Apr 04, 2015, 1:45 AM UTC

RegEx wrote: post #56

vortico wrote: post #25

Deployment one-liners with hashpipe will only work if hashpipe is installed, which would be equally difficult for users to install properly than the software itself. Then you'd need something like this: > Simply install using `curl hashpi.pe | bash`

This is a really, really interesting point! In addition to your point, (and perhaps this point has already been made elsewhere in the thread), but the type of people that would install hashpipe would be the kind of people to avoid the practice entirely. Perhaps there's some mechanism for convincing people to use the tool, and if that mechanism ends up being easier than convincing people to stop piping directly to bas…

yeah, this is just a stopgap for me. in the end i want:

  ipfs daemon --mount &
  sh /ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6

(this actually works, btw :)

Re: Hashpipe – Pipe iff the hash matches

#59

Post by _prometheus » Sat, Apr 04, 2015, 1:49 AM UTC

aablkn wrote: post #50

You might want to `set -o pipefail` on your bash because a failing process does not stop things from getting piped: `echo OK | false | echo OK2` --> this command returns zero exit code even though false does return non-zero exit code. If you do `set -o pipefail` entire pipe will fail with non-zero exit code (of `false`).

great point. if you have a good idea of how hashpipe should do it better pls comment at: https://github.com/jbenet/hashpipe/issues

at least hashpipe stops the output, so this would not run the bad code. hashpipe would stop it:

  cat evil | hashpipe  | sh

without pipefail, it would still exit 0, but at least the evil would be contained.

Re: Hashpipe – Pipe iff the hash matches

#60

Post by _prometheus » Sat, Apr 04, 2015, 1:50 AM UTC

whyrusleeping wrote: post #48

gojomo wrote: post #41

Earlier quoted context omitted.

That's great! I hope that IPFS, or something of its ilk, can be the promised-land to which I alluded. If I were to install it, how hard/breaking would it be to change the access-path to something more grandiosely descriptive like '/everything/'?

its a simple matter of changing the ipfs config file

<3