Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hashpipe – Pipe iff the hash matches

jbenet.github.io

Post reply on HN
Print view View original

61–70 of 90 posts

Re: Hashpipe – Pipe iff the hash matches

#61

Post by _prometheus »

gojomo wrote: post #41
_prometheus wrote: post #37

Earlier quoted context omitted.

this already works. install ipfs: http://ipfs.io/docs/install then: ipfs init ipfs daemon & sleep 20 # sorry this will go away ipfs mount sh /ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6 you can see it on the web at: http://gateway.ipfs.io/ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4...

That's great! I hope that IPFS, or something of its ilk, can be the promised-land to which I alluded. If I were to install it, how hard/breaking would it be to change the access-path to something more grandiosely descriptive like '/everything/'?

also, we want to push towards `/` being everything, shoving all your local crap into `/local` or `/localhost`

Re: Hashpipe – Pipe iff the hash matches

#62

Post by serendipitous »

gojomo wrote: post #29
quicksnap wrote: post #20

Since the main use case for this utility is verifying network shell scripts, it would be interesting to see a query param convention, so we could use a tool such as: > hashcurl http://load.this/script?hash= QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 | sh

When the just-over-the-next-hilltop promised-land nirvana of content-centric networking arrives, the hash will be enough to locate & download the content – so you shouldn't even need an URL: $ hashcurl QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 | sh Maybe it's even a special filesystem path, that contains (but does not list) everything-that's-nameable-and-findable: $ sh /everything/QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6…

This sounds like DHT, which has existed for a while now. https://en.wikipedia.org/wiki/Distributed_hash_table

Re: Hashpipe – Pipe iff the hash matches

#63

Post by tbrownaw »

comex wrote: post #53
pritambaral wrote: post #47

I think this one isn't too far from useability: curl -o filename url && sha256sum -c And works without requiring installation of third-party tools too!

This is non-portable, as OS X has no sha256sum out of the box. But it does have shasum from Perl, and Linux distros typically come with Perl, so 'shasum -ba256' should work... I wonder if there is any concise way to do this without needing to save a file to disk. I can't think of one, as it requires splitting the input in two, which bash can do with >(command) but not sequentially or with the ability to communicate f…

You need to buffer the entire input somewhere.

You don't get your hash without consuming the entire input.

You can't start feeding the input to the shell, until you verify the hash.

You don't want to download it twice, it case it comes back different the second time (or in case your network is slow).

At the point that you are verifying the hash, the entire file must exist on your system somewhere. Disk, memory, OCR-friendly printout, whatever.

Buffering to memory can only work if the input is "small", whatever that means. And pipelines aren't meant to do this, so you'll have to do something a bit odd (ie, confusing to anyone trying to understand your code) to make it work.

So, buffer to disk.

.

Or, use Perl. Perl solves everything.

Re: Hashpipe – Pipe iff the hash matches

#65

Post by _prometheus »

gojomo wrote: post #54
stouset wrote: post #35

Earlier quoted context omitted.

It's not save a few characters. It's to allow for flexibility of encoding. "sha1" all but requires ASCII. Sometimes you're limited to hex, base64, etc.

Not sure I understand; almost any context where "QmTpn…" can appear could also handle a "sha1:" human-readable prefix. Including, notably, this shell context. If another context is sufficiently different, it would be fine to use a controlled fixed-length binary-vocabulary there. (But, the ASCII bytes for "sha1:" would still be a pretty robust and largely self-documenting choice.)

oh the "Qm..." string is the base58 representation. you can have a binary packed version as well. check out more at https://github.com/jbenet/multihash/

the important thing here is standardizing the encoding, i.e. there should be a valid repr in {binary, hex, b32, b58, b64, ... bemoji ...}

so that, say, if your input field _only takes hex_ you have a way to enter the hash. this problem surfaces when tools expect reprs to be in a particular encoding (all too common), and you dont have power or access to make the tool better (sadly all too common too).

Re: Hashpipe – Pipe iff the hash matches

#66

Post by _prometheus »

kazinator wrote: post #51
tyho wrote: post #5

Be warned that this reads everything until EOF into memory, doing something like: hashpipe QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 will produce unexpected behaviour.

It doesn't have to read into memory; if that becomes a problem they can fix it. The hashing algorithm just needs a small, constant amount of state. While it is hashing, it can divert the data to a temporary file. Then it can just pass the file to its standard output. A "hashexec" command could also be produced which passes the name of this temporary file to a subprogram: whatever | hashexec arbitrary --command with a…

All great ideas! mind adding some to https://github.com/jbenet/hashpipe/issues ? over time we can implement some

Re: Hashpipe – Pipe iff the hash matches

#67

Post by _prometheus »

languagehacker wrote: post #45

Getting this in before the gratuitous negativity brigade starts hammering down: This an implementation of a stupid joke looking for a problem. If someone actually needed this in their toolbelt they could use perl or even (amazingly) bash to handle this problem. I would personally not name anything I worked on after a song from one of Weezer's shittiest albums, but that's just me.

many execution contexts don't have perl. i have some <10MB VMs in mind.

Re: Hashpipe – Pipe iff the hash matches

#68

Post by kazinator »

_prometheus wrote: post #66
kazinator wrote: post #51

Earlier quoted context omitted.

It doesn't have to read into memory; if that becomes a problem they can fix it. The hashing algorithm just needs a small, constant amount of state. While it is hashing, it can divert the data to a temporary file. Then it can just pass the file to its standard output. A "hashexec" command could also be produced which passes the name of this temporary file to a subprogram: whatever | hashexec arbitrary --command with a…

All great ideas! mind adding some to https://github.com/jbenet/hashpipe/issues ? over time we can implement some

I don't use github; I have my own server with git and cgit.

Re: Hashpipe – Pipe iff the hash matches

#69

Post by _prometheus »

latchkey wrote: post #23

I love this idea, as it has bothered me for a while now that homebrew initial install just points to a script on github. That said, now we just need someone to hack gobuilder.me and replace the binary distribution of hashpipe with something that always returns the input.

Great point!! working towards fixing that (the general problem of safe execution of binaries). signed releases will help.

I will release as self-describing signed package for this soon.

Re: Hashpipe – Pipe iff the hash matches

#70

Post by gojomo »

_prometheus wrote: post #65
gojomo wrote: post #54

Earlier quoted context omitted.

Not sure I understand; almost any context where "QmTpn…" can appear could also handle a "sha1:" human-readable prefix. Including, notably, this shell context. If another context is sufficiently different, it would be fine to use a controlled fixed-length binary-vocabulary there. (But, the ASCII bytes for "sha1:" would still be a pretty robust and largely self-documenting choice.)

oh the "Qm..." string is the base58 representation. you can have a binary packed version as well. check out more at https://github.com/jbenet/multihash/ the important thing here is standardizing the encoding, i.e. there should be a valid repr in {binary, hex, b32, b58, b64, ... bemoji ...} so that, say, if your input field _only takes hex_ you have a way to enter the hash. this problem surfaces when tools expect repr…

'sha1:' (0x736861313a) and other self-describing labels can be represented in hex (etc) just fine.
Post reply on HN