Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory safe ‘curl’ for a more secure internet

daniel.haxx.se

Post reply on HN
Print view View original

151–160 of 210 posts

Re: Memory safe ‘curl’ for a more secure internet

#151

Post by remram »

nicoburns wrote: post #50
svnpenn wrote: post #32

I like this idea, but I dont know if Hyper is the best package to go with. Hyper occupies part of the Rust ecosystem that I think suffers from package bloat, like much of NPM. For example, currently Hyper requires 52 packages: autocfg, bitflags, bytes, cfg-if, fnv, fuchsia-zircon, fuchsia-zircon-sys, futures-channel, futures-core, futures-sink, futures-task, futures-util, h2, hashbrown, http, http-body, httparse, htt…

Part of this is just crates being broken up more in Rust. For example the `http` crate only contains trait (interface) definitions. They break down like so: Platform integration: libc, winapi, winapi-build, winapi-i686-pc-windows-gnu, winapi-x86_64-pc-windows-gnu, ws2_32-sys, fuchsia-zircon, fuchsia-zircon-sys, kernel32-sys, redox_syscall Primitive algorithms: itoa, memchr, unicode-xid Proc macro / pinning utilities:…

Honestly it still seems like a lot this way.

If those platform crates are just backends for libc (or similar to libc), why aren't they all folded in a single project? Having them as separate crates open the gate for supply-chain attacks without really allowing greater control or expressiveness. You are always going to pull all of them in, you are unlikely to ever use them directly, and they have no more impact on compilation than features.

Having to use proc-macro2+quote+syn for macros feels wrong, considering it's a language feature.

Having to pull in 4 crates for pinning also seems wrong. This could easily be a single utility crate, and if you really need all this cruft to use Pin (a language feature) most of this should probably be in std.

The async I/O feels like definite bloat. Not only is that a lot of futures-* crates, but I know from first-hand experience that those tend to implement multiple versions of some primitives (like streams) that are incompatible.

Re: Memory safe ‘curl’ for a more secure internet

#152

Post by steveklabnik »

remram wrote: post #151
nicoburns wrote: post #50

Earlier quoted context omitted.

Part of this is just crates being broken up more in Rust. For example the `http` crate only contains trait (interface) definitions. They break down like so: Platform integration: libc, winapi, winapi-build, winapi-i686-pc-windows-gnu, winapi-x86_64-pc-windows-gnu, ws2_32-sys, fuchsia-zircon, fuchsia-zircon-sys, kernel32-sys, redox_syscall Primitive algorithms: itoa, memchr, unicode-xid Proc macro / pinning utilities:…

Honestly it still seems like a lot this way. If those platform crates are just backends for libc (or similar to libc), why aren't they all folded in a single project? Having them as separate crates open the gate for supply-chain attacks without really allowing greater control or expressiveness. You are always going to pull all of them in, you are unlikely to ever use them directly, and they have no more impact on com…

> If those platform crates are just backends for libc (or similar to libc),

They're not, they're to the specific platform's APIs. The libc crate is a package that targets libc on every platform.

> Having to use proc-macro2+quote+syn for macros feels wrong, considering it's a language feature.

In order to ship procedural macros as a feature, they're pretty minimal. There's a tradeoff here; others could be chosen, but weren't for good reasons.

> Having to pull in 4 crates for pinning also seems wrong.

This is covered by https://news.ycombinator.com/item?id=24730280; that is, it's a transitive dependency issue.

> The async I/O feels like definite bloat.

Really depends, network calls are a textbook use-case for async I/O. And there's so many futures crates specifically so that you can only depend on the bits you need.

Re: Memory safe ‘curl’ for a more secure internet

#153

Post by benibela »

johnisgood wrote: post #110

I think Ada/SPARK would have been a much better choice, but oh well. Is it a licensing issue?

Or Wuffs

I do not know much about Wuffs, but it seems to be completely safe. No arithmetic overflows, no bound checking failures, no None unwrapping panics, no memory allocation failure panics.

Re: Memory safe ‘curl’ for a more secure internet

#154

Post by tick_tock_tick »

jdright wrote: post #117
dagenix wrote: post #94

Earlier quoted context omitted.

I don't believe that's accurate in general. Let's say you open up a new C codebase: What are its dependencies? You'll have to hunt through its README (hopefully it's up to date!), other build instructions, maybe CMake, maybe some custom build system, etc. What version of dependencies does it use? If the code has been vendored, you at least know what code its using - but where do you look for updates to that code? Do…

This. Take even CURL as an example, try to list its dependencies and you'll see how harder it is.

libc that's it. Every other dependency for curl is optional.....

Re: Memory safe ‘curl’ for a more secure internet

#155

Post by asveikau »

dagenix wrote: post #94
asveikau wrote: post #78

Earlier quoted context omitted.

> The only thing new about it is that programmers are exposed to more of the costs of it up-front. That's funny, because it's only "new" if your experiences primarily lie in newer languages and communities. There's a lot of criticism of C, but one thing it does is make dependencies pretty explicit. Some say that's good, some say that's bad, I guess it can be both at different times.

I don't believe that's accurate in general. Let's say you open up a new C codebase: What are its dependencies? You'll have to hunt through its README (hopefully it's up to date!), other build instructions, maybe CMake, maybe some custom build system, etc. What version of dependencies does it use? If the code has been vendored, you at least know what code its using - but where do you look for updates to that code? Do…

But each one of these dependencies is pretty consciously and manually added, most of the time. In new code, to introduce a new one usually requires thought, and that creates a culture of caution.

Also if you are dynamic linking, ldd(1) can give you a pretty good picture.

Re: Memory safe ‘curl’ for a more secure internet

#156

Post by scottlamb »

CUViper wrote: post #142
nindalf wrote: post #126

Earlier quoted context omitted.

Why does hyper pull in hashbrown? Isn't it identical to std::collections::HashMap?

The dependency goes hyper -> h2 -> indexmap -> hashbrown, and indexmap is built on hashbrown::raw::RawTable.

Looks like the indexmap -> hashbrown dependency was added recently: https://github.com/bluss/indexmap/pull/131

Re: Memory safe ‘curl’ for a more secure internet

#157

Post by burntsushi »

simias wrote: post #141
burntsushi wrote: post #111

Earlier quoted context omitted.

I switched ripgrep to clap 4 years ago. And that was well after clap had already become the popular "go to" solution. Some parts of the ecosystem are more stable than others. That's true. And it takes work to know which things are stable and which aren't. And yet, some things just take a longer time to improve. lazy_static has been stable and unchanged for a very long time and it works just fine. You don't need to sw…

Interesting that I missed clap when I wrote that program a few years ago then. In my defence "argparse" is a lot more explicit than "clap" for a such a library. Also argparse's last update was 2 years ago, so there's been quite a bit of overlap. I guess what I'm saying is that it's an other problem with the current package ecosystem: you often end up finding multiple packages purporting to do what you need, and it ca…

Seems like an unavoidable problem unless you buy into a curated ecosystem. Like, yeah, the cost of a decentralized ecosystem is that you have to do your due diligence on which crate to use, if any. (For example, I don't even bother with a log helper crate because it just isn't necessary for simple cases.)

Re: Memory safe ‘curl’ for a more secure internet

#158

Post by eptcyka »

skohan wrote: post #139
ordu wrote: post #122

Earlier quoted context omitted.

I think that it is the right tool. 1. CURL without https seems insufficient nowadays. 2. CURL could be improved by running multiple downloads at once. I'm not sure that curl command line utility could do it, but certainly libcurl.so has this ability, it allows client code to work with multiple connections. 3. Any application having UI could benefit from async: input/output and main task are async by nature. For examp…

1. So as to your first point, I totally agree CURL needs to support HTTPS. My point is that Hyper needs a runtime for HTTPS, and it doesn't necessarily make sense for CURL to have a runtime. 2. I'm not sure that CURL should necessarily support multiple concurrent downloads. It could also be argued it's more UNIX-y to make it just do one thing and allow the caller to run multiple CURL processes at the same time 3. You…

Hyper only brings in a single-threaded runtime, so it's not much of a runtime at all. That is to say, driving a future returned from Hyper in a blocking fashion and then dropping it is all that's required. Once you drop the client, the runtime will be dropped too. I usually take issue with runtimes due to the added complexity and the lack of clarity about resource usage - I'm mostly worried about superfluous memory usage and some rogue threads going off and doing a quest and a half doing god knows what, hogging or otherwise interfering with my application's threads. I don't think that's possible in this case. Do you know if there are any other concerns that I should be worried about when bringing in something that has a runtime?

Re: Memory safe ‘curl’ for a more secure internet

#159

Post by staticassertion »

geodel wrote: post #138
ori_b wrote: post #85

Earlier quoted context omitted.

Saying "The people complaining are right, but they've been right a long time" isn't a great endorsement of the situation.

It seems to me Rust folks have developed this habit of deflecting blame by pointing to shallow commentary. One of good examples is compiler slowness reasons, sometimes its LLVM, or it is lot of optimizations, or it is not really slow compare to C++ and so on. They could have said it straight "Guys highly optimized, safe compilation of medium size project will be in range of 20-30 min". And that would great and honest…

Your argument seems to be "people see problems and other people explain why those problems exist" and somehow you frame it like it's a bad thing.

Re: Memory safe ‘curl’ for a more secure internet

#160

Post by pixel_fcker »

asveikau wrote: post #155
dagenix wrote: post #94

Earlier quoted context omitted.

I don't believe that's accurate in general. Let's say you open up a new C codebase: What are its dependencies? You'll have to hunt through its README (hopefully it's up to date!), other build instructions, maybe CMake, maybe some custom build system, etc. What version of dependencies does it use? If the code has been vendored, you at least know what code its using - but where do you look for updates to that code? Do…

But each one of these dependencies is pretty consciously and manually added, most of the time. In new code, to introduce a new one usually requires thought, and that creates a culture of caution. Also if you are dynamic linking, ldd(1) can give you a pretty good picture.

And a simple ‘cargo tree‘ will show you the tree of dependencies for a crate, nicely formatted.
Post reply on HN