Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft cURLs too

daniel.haxx.se

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Microsoft cURLs too

#91

Post by aplorbust »

aplorbust wrote: post #73
lettergram wrote: post #14

One thing I am always sure to share with collegues when we discuss curl, is the fact from the command line you can generate the underlying C code. This is pretty useful when creating a CLI for pretty much any app, and I've used it regularly to generate a CLI for an app. My post on how to do it: http://austingwalters.com/export-a-command-line-curl-command...

It wasnt until 2006 that curl added HTTP/1.1 pipelining support. Hence I always used netcat instead of curl because I utilised pipelining heavily for text/html retrieval (IME, most servers supported it). Imagine something like this with curl: curl where curl only opens a single connection. Alas, AFAIK, pipelining is still not enabled in the curl binary. As I understand it, the --libcurl option only generates code for…

To reproduce:

   curl --libcurl 1.c http://example.com/a.htm http://example.com/b.htm

   grep curl_multi_init 1.c
https://curl.haxx.se/mail/archive-2008-02/0036.html

Re: Microsoft cURLs too

#92

Post by jjnoakes »

shakna wrote: post #85
jjnoakes wrote: post #84

Earlier quoted context omitted.

Why would you be using an IP and not a URL? And why wouldn't you trust the source of the install script as much as any other installer? Do you audit the binary installers you use as well? I don't disagree about a minor difference between the methods, but I definitely disagree with piping to sh being "very unsafe". If you trust the site/author enough to run their code on your computer at all, the install method risk d…

> If you trust the site/author enough to run their code on your computer at all Piping curl, means you can't be sure it came from the author's site. It means you can't be sure you're getting the same software you've been considering installing. It means a broken connection is a broken install, with no cleanup and no idea what it has changed. > Do you audit the binary installers you use as well? Don't install random b…

You didn't answer the 'random ip' question.

> Piping curl, means you can't be sure it came from the author's site.

Assuming you take the same precautions you'd take with any other software download (like using https), there's no difference between curl-piped-to-sh and clicking on a link to a rpm, deb, exe, or anything else.

> It means a broken connection is a broken install

Not if the shell script is written correctly. And if you can't trust the source of your software to get that right, then you can't trust them to get the regular installers right either, so there's no difference here either.

> Don't install random binaries either

No one's advocating for random binaries, but you do have to install binaries from time to time, no? Or are you getting your CPU microcode updates in source form too?

I get it, it's a knee-jerk cargo-cult reaction to flame folks who don't see the huge issue with piping a https URL from the software's main web site to the shell, but if you actually think about it, it does not have the major flaws that you claim it does.

Re: Microsoft cURLs too

#93

Post by bagder »

lpgauth wrote: post #83
bagder wrote: post #79

Earlier quoted context omitted.

If you invoke "curl http://example.com/a.htm http://example.com/b.htm" , curl will only use a single connection.

Yes, but it will wait for Response A before sending Request B.

It probably isn't too obvious here, but yeah I know how it works =)

(I'm Daniel, who wrote the blog post discussed here and leads the curl development...)

Re: Microsoft cURLs too

#94

Post by shakna »

jjnoakes wrote: post #92
shakna wrote: post #85

Earlier quoted context omitted.

> If you trust the site/author enough to run their code on your computer at all Piping curl, means you can't be sure it came from the author's site. It means you can't be sure you're getting the same software you've been considering installing. It means a broken connection is a broken install, with no cleanup and no idea what it has changed. > Do you audit the binary installers you use as well? Don't install random b…

You didn't answer the 'random ip' question. > Piping curl, means you can't be sure it came from the author's site. Assuming you take the same precautions you'd take with any other software download (like using https), there's no difference between curl-piped-to-sh and clicking on a link to a rpm, deb, exe, or anything else. > It means a broken connection is a broken install Not if the shell script is written correctl…

> You didn't answer the 'random ip' question.

Just because you have a URL, doesn't mean it is connecting to the expected IP. Not everyone uses https yet. Not everyone uses HSTS to protect against downgrading.

> Assuming you take the same precautions you'd take with any other software download (like using https), there's no difference between curl-piped-to-sh and clicking on a link to a rpm, deb, exe, or anything else.

And that assumption isn't a guarantee. Remember who these kinds of installs target: developers with little experience. You can't ensure they'll notice a missing 's'. You can't ensure a worn-out admin will either.

> Not if the shell script is written correctly. And if you can't trust the source of your software to get that right, then you can't trust them to get the regular installers right either, so there's no difference here either.

No. I can't. Have a glance over Heroku's Ubuntu script.[0] It's not fenced, if that echo breaks, it could case some chaos. In fact, none of the commands are even checked for success, except the su.

> No one's advocating for random binaries, but you do have to install binaries from time to time, no? Or are you getting your CPU microcode updates in source form too?

If you look one level up, it's specifically binary installers. And secondly, I use a package manager, which has some review of this kind of thing.

I wouldn't curl a microcode update. That's asking for trouble.

[0] https://cli-assets.heroku.com/install-ubuntu.sh

Re: Microsoft cURLs too

#95

Post by shakna »

Spivak wrote: post #87
shakna wrote: post #85

Earlier quoted context omitted.

> If you trust the site/author enough to run their code on your computer at all Piping curl, means you can't be sure it came from the author's site. It means you can't be sure you're getting the same software you've been considering installing. It means a broken connection is a broken install, with no cleanup and no idea what it has changed. > Do you audit the binary installers you use as well? Don't install random b…

> Piping curl, means you can't be sure it came from the author's site. Up to the trustworthiness of the CA system yes you can. If the author's site is serving malicious downloads to the curl UA then you're probably hosed either way. It would be easier to just slip malicious code in the software itself. > It means a broken connection is a broken install, with no cleanup and no idea what it has changed. This is the rea…

> Up to the trustworthiness of the CA system yes you can. If the author's site is serving malicious downloads to the curl UA then you're probably hosed either way. It would be easier to just slip malicious code in the software itself.

If they have HSTS, otherwise you might end up using plain ol' http by accident. Like over at surge.sh, but at least they use a package manager.

> Nobody who is running curl|bash isn't installing a 'random' binary but downloading an installer from a source they trust.

But you can't trust it, because most shell scripts out there are woefully inadequate. So you're one broken connection, one WiFi drop, from corrupting your system. At least a binary needs to be complete to run.

Example: Heroku's CLI [0]

If it breaks on the echo, you could end up overwriting your entire source list.

e.g. It breaks to:

echo "deb https://cli-assets.heroku.com/branches/stable/apt ./" > /etc/apt/sources.list

instead of the intended

echo "deb https://cli-assets.heroku.com/branches/stable/apt ./" > /etc/apt/sources.list.d/heroku.list

And it'll work too, because the entire thing runs as the root user.

> The argument surrounding curl|bash should really focus on this rather than hand-wavy security.

They're the same thing. A broken connection with curl | sh is a security problem. As is downgraded https, because of an accidentally misconfigured host. As is running without even the basic check of seeing if you get the complete file before executing it.

Everything about curl | sh is inherently untrustworthy.

[0] https://cli-assets.heroku.com/install-ubuntu.sh

Re: Microsoft cURLs too

#96

Post by jjnoakes »

shakna wrote: post #94
jjnoakes wrote: post #92

Earlier quoted context omitted.

You didn't answer the 'random ip' question. > Piping curl, means you can't be sure it came from the author's site. Assuming you take the same precautions you'd take with any other software download (like using https), there's no difference between curl-piped-to-sh and clicking on a link to a rpm, deb, exe, or anything else. > It means a broken connection is a broken install Not if the shell script is written correctl…

> You didn't answer the 'random ip' question. Just because you have a URL, doesn't mean it is connecting to the expected IP. Not everyone uses https yet. Not everyone uses HSTS to protect against downgrading. > Assuming you take the same precautions you'd take with any other software download (like using https), there's no difference between curl-piped-to-sh and clicking on a link to a rpm, deb, exe, or anything else…

> Just because you have a URL, doesn't mean it is connecting to the expected IP.

Ignoring how weak of an argument that is, I don't see how that is any different of a risk between curl-pipe-to-sh and regular software downloads.

> Remember who these kinds of installs target: developers with little experience.

I think that's a little condescending. I image these kinds of installers target folks who want to get up and running quickly and conveniently, regardless of their experience. And I imagine, on average, folks pasting this into their shell have more than average experience already, since they (1) went out of their way to try this software and (2) know how to open a shell and copy commands into it.

> You can't ensure they'll notice a missing 's'. You can't ensure a worn-out admin will either.

You think it's more likely they will notice the missing 's' in the click-to-download-the-installer scenario than in the paste-a-command-into-the-shell scenario? I find that hard to believe.

> Have a glance over Heroku's Ubuntu script.[0] It's not fenced, if that echo breaks, it could case some chaos.

So file an issue. If their normal installer has bugs in it, things would break too. I don't see the difference. Buggy installers are buggy, which is just an argument against buggy installers, not against different install methods.

> I wouldn't curl a microcode update.

You missed the point. You can download it any way you like; unless you have its source, though, you can't audit it at all. So your claim that binary software is untrustable falls short in the practical world.

Re: Microsoft cURLs too

#97

Post by jjnoakes »

shakna wrote: post #95
Spivak wrote: post #87

Earlier quoted context omitted.

> Piping curl, means you can't be sure it came from the author's site. Up to the trustworthiness of the CA system yes you can. If the author's site is serving malicious downloads to the curl UA then you're probably hosed either way. It would be easier to just slip malicious code in the software itself. > It means a broken connection is a broken install, with no cleanup and no idea what it has changed. This is the rea…

> Up to the trustworthiness of the CA system yes you can. If the author's site is serving malicious downloads to the curl UA then you're probably hosed either way. It would be easier to just slip malicious code in the software itself. If they have HSTS, otherwise you might end up using plain ol' http by accident. Like over at surge.sh, but at least they use a package manager. > Nobody who is running curl|bash isn't i…

> Everything about curl | sh is inherently untrustworthy

Nope. Only one item is of minor concern (which I've covered many times in this thread) and it has an easy and known solution.

The rest of your objections are not specific to curl-piped-to-sh and are irrelevant to this discussion.

The sky is not falling, so I'm not sure what your agenda really is.

Re: Microsoft cURLs too

#98

Post by aplorbust »

aplorbust wrote: post #91
aplorbust wrote: post #73

Earlier quoted context omitted.

It wasnt until 2006 that curl added HTTP/1.1 pipelining support. Hence I always used netcat instead of curl because I utilised pipelining heavily for text/html retrieval (IME, most servers supported it). Imagine something like this with curl: curl where curl only opens a single connection. Alas, AFAIK, pipelining is still not enabled in the curl binary. As I understand it, the --libcurl option only generates code for…

To reproduce: curl --libcurl 1.c http://example.com/a.htm http://example.com/b.htm grep curl_multi_init 1.c https://curl.haxx.se/mail/archive-2008-02/0036.html

Here is how RFC 7230 defines "HTTP/1.1 pipelining":

"6.3.2. Pipelining

A client that supports persistent connections MAY "pipeline" its requests (i.e., send multiple requests without waiting for each response)."

AFAIK, the curl binary does not do pipelining.

And, AFAIK, it will not generate code to do pipelining by invoking it with --libcurl.

Re: Microsoft cURLs too

#99

Post by aplorbust »

aplorbust wrote: post #91
aplorbust wrote: post #73

Earlier quoted context omitted.

It wasnt until 2006 that curl added HTTP/1.1 pipelining support. Hence I always used netcat instead of curl because I utilised pipelining heavily for text/html retrieval (IME, most servers supported it). Imagine something like this with curl: curl where curl only opens a single connection. Alas, AFAIK, pipelining is still not enabled in the curl binary. As I understand it, the --libcurl option only generates code for…

To reproduce: curl --libcurl 1.c http://example.com/a.htm http://example.com/b.htm grep curl_multi_init 1.c https://curl.haxx.se/mail/archive-2008-02/0036.html

Here is how RFC 7230 defines "HTTP/1.1 pipelining":

"6.3.2. Pipelining

A client that supports persistent connections MAY "pipeline" its requests (i.e., send multiple requests without waiting for each response)."

AFAIK, the curl binary does not do pipelining by this definition.

And, AFAIK, it will not generate code to do pipelining by invoking it with --libcurl.

Re: Microsoft cURLs too

#100

Post by shakna »

jjnoakes wrote: post #96
shakna wrote: post #94

Earlier quoted context omitted.

> You didn't answer the 'random ip' question. Just because you have a URL, doesn't mean it is connecting to the expected IP. Not everyone uses https yet. Not everyone uses HSTS to protect against downgrading. > Assuming you take the same precautions you'd take with any other software download (like using https), there's no difference between curl-piped-to-sh and clicking on a link to a rpm, deb, exe, or anything else…

> Just because you have a URL, doesn't mean it is connecting to the expected IP. Ignoring how weak of an argument that is, I don't see how that is any different of a risk between curl-pipe-to-sh and regular software downloads. > Remember who these kinds of installs target: developers with little experience. I think that's a little condescending. I image these kinds of installers target folks who want to get up and ru…

> So your claim that binary software is untrustable falls short in the practical world.

Where the hell do you think I made that claim?

> So file an issue.

Are you just trying to ignore everything I say? I responded to a claim that shell installers would be written correctly, with evidence that a fairly sizeable company doesn't get it right.

Your response to that is that it doesn't matter.

> You think it's more likely they will notice the missing 's' in the click-to-download-the-installer scenario than in the paste-a-command-into-the-shell scenario? I find that hard to believe.

A giant green bar is a little bit more than a single letter.

---

You really don't seem interested in a conversation about the shortcomings that exist. You seem interested only picking holes and saying that you are correct.

I have no interest in responding to that kind of conversation.

Post reply on HN