Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The New AWS Command-Line Interface

aws.typepad.com

Post reply on HN
Print view View original

61–70 of 71 posts

Re: The New AWS Command-Line Interface

#61

Post by jrochkind1 »

sivers wrote: post #35
gtaylor wrote: post #11

Earlier quoted context omitted.

> The syntax looks quite the same, but Amazon's awscli Python installer has loads of dependencies. I'll have to see if it's worth switching. Why are the dependencies a problem? By combining a handful of smaller, focused modules that each do something well, you can end up with something better than if you were to re-invent the wheel for every need. AWS and the Python dev team are doing a heck of a job on botocore, and…

I just like knowing what's on my system. `pip install awscli` just installed 26 other modules besides awscli. Now I feel a little obliged to go check out those 26, as well, to see what they are. I agree about not re-inventing the wheel. But the amount of stuff installed is definitely a considering factor when choosing between two seemingly identical scripts.

In a pre pip/rubygem auto-installer world, this would have been one package that had the 26 dependencies embedded as modules inside it. You wouldn't have known they were unless you looked at the source code. You also wouldn't be able to update them independently, you'd get updates when a new release of the original thing was available that included embedded updates.

Would this make you feel more like you knew what was installed on your system? Would you have felt the need to look at the source code, including the source code for any embedded dependencies, in, say, a 'vendor' or 'lib/dist' directory or something?

What are the plusses and minuses of each approach? There are some plusses and some minuses either way. After considering them, do you still have a problem with the 'new way' of doing things, where a program might install along with explicit dependencies via pip (or rubygems in ruby) separated out in a different place in the file system, vs. embedded/bundled dependencies?

Re: The New AWS Command-Line Interface

#62

Post by spullara »

the_mitsuhiko wrote: post #42
spullara wrote: post #32

Earlier quoted context omitted.

The issue is that they can't release them all compiled into one thing. Instead the dependencies have to pollute the rest of the operating system. This is one of those things that Java got right and virtually everything else except npm (local node_modules) got wrong out of the gate. I'd love to have a simple way to package python apps that depend on other python and native libraries without having to install things se…

> The issue is that they can't release them all compiled into one thing. Instead the dependencies have to pollute the rest of the operating system. Bullshit. Python supports having modules installed into local locations (see virtualenv). Just `virtualenv ~/.local/lib/aws; ~/.local/lib/aws/bin/pip install awscli; ln -s ~/.local/lib/aws/bin/aws ~/.local/bin/aws` and put `~/.local/bin` into your PATH.

Many Python programs have native dependencies as well. Also, what you suggest isn't at all compiled into one thing I can hand to a customer.

Re: The New AWS Command-Line Interface

#63

Post by spullara »

cmccabe wrote: post #43
spullara wrote: post #32

Earlier quoted context omitted.

The issue is that they can't release them all compiled into one thing. Instead the dependencies have to pollute the rest of the operating system. This is one of those things that Java got right and virtually everything else except npm (local node_modules) got wrong out of the gate. I'd love to have a simple way to package python apps that depend on other python and native libraries without having to install things se…

If you don't want the pip dependencies to "pollute the rest of the system," you can just use pip install with the --root option. Java's CLASSPATH causes enormous pain for end-users. Just read the Hadoop mailing list. The fact that Java doesn't have a sane default for where to put anything or how to manage dependencies is a huge flaw. Back when I was doing AWS, I just used the C binaries (I forget what they were calle…

If you are using CLASSPATH for Java you are doing it wrong. Full dependency jars with every class needed is really the only way to do this reliably. That requires only a single file to be passed around. You can even include native libraries in it. Hadoop is a nightmare, I agree. They are doing it wrong.

Re: The New AWS Command-Line Interface

#65

Post by brown9-2 »

spullara wrote: post #62
the_mitsuhiko wrote: post #42

Earlier quoted context omitted.

> The issue is that they can't release them all compiled into one thing. Instead the dependencies have to pollute the rest of the operating system. Bullshit. Python supports having modules installed into local locations (see virtualenv). Just `virtualenv ~/.local/lib/aws; ~/.local/lib/aws/bin/pip install awscli; ln -s ~/.local/lib/aws/bin/aws ~/.local/bin/aws` and put `~/.local/bin` into your PATH.

Many Python programs have native dependencies as well. Also, what you suggest isn't at all compiled into one thing I can hand to a customer.

It is if you tar up the directory and hand that to them.

Re: The New AWS Command-Line Interface

#66

Post by cmccabe »

spullara wrote: post #63
cmccabe wrote: post #43

Earlier quoted context omitted.

If you don't want the pip dependencies to "pollute the rest of the system," you can just use pip install with the --root option. Java's CLASSPATH causes enormous pain for end-users. Just read the Hadoop mailing list. The fact that Java doesn't have a sane default for where to put anything or how to manage dependencies is a huge flaw. Back when I was doing AWS, I just used the C binaries (I forget what they were calle…

If you are using CLASSPATH for Java you are doing it wrong. Full dependency jars with every class needed is really the only way to do this reliably. That requires only a single file to be passed around. You can even include native libraries in it. Hadoop is a nightmare, I agree. They are doing it wrong.

Hadoop is a framework, not a library, so user applications need to link against the jars they need. That implies using CLASSPATH to locate them. Whether or not there is 1 jar or 100, the fact that there's no standard place to install jars in Java is a problem.

Hadoop jars are hundreds of megabytes, and we have multiple daemons. Duplicating all those jars in each daemon would multiply the size of the installation many times over. That's also a nontrivial amount of memory to be giving up because jars can no longer be shared in the page cache.

Some of these problems could be mitigated by making Hadoop a library rather than a framework (as Google's MR is), or by pruning unnecessary dependencies.

Re: The New AWS Command-Line Interface

#67

Post by spullara »

cmccabe wrote: post #66
spullara wrote: post #63

Earlier quoted context omitted.

If you are using CLASSPATH for Java you are doing it wrong. Full dependency jars with every class needed is really the only way to do this reliably. That requires only a single file to be passed around. You can even include native libraries in it. Hadoop is a nightmare, I agree. They are doing it wrong.

Hadoop is a framework, not a library, so user applications need to link against the jars they need. That implies using CLASSPATH to locate them. Whether or not there is 1 jar or 100, the fact that there's no standard place to install jars in Java is a problem. Hadoop jars are hundreds of megabytes, and we have multiple daemons. Duplicating all those jars in each daemon would multiply the size of the installation many…

Most of these issues could be addressed by actually modularizing the core of Hadoop, some of which has been done in the latest code. Also, many things could be provided at runtime by the system with only the interfaces required to be in the jars that customers depend on, thus making their jars backwards compatible and more robust. BTW, let's say you didn't want to put the jars in one jar but didn't want a classpath. You can use META-INF/manifest to include those jars automatically as long as they are in a well defined place relative to the host jar. Redesign with the requirement that end users don't have to worry about CLASSPATH and you will find that there are solutions.

I do sympathize that something akin to the maven repository and dependency mechanism hasn't been integrated into the JDK. I was on the module JSR and continually pushed them to do something like that but it turns out IBM would rather have OSGI standardized and so it deadlocked. Maybe something will come in JDK 9.

Sad email thread from years ago: http://markmail.org/message/y2a6nzhcsp62p5yv

Re: The New AWS Command-Line Interface

#68

Post by kolev »

hendry wrote: post #44

aws-cli/0.16.0 gripes: • slow • crap ton of dependencies, making it a pain to install on Arch https://aur.archlinux.org/packages/aws-cli/ Hint: static build ffs • broken help. `aws s3 help` instead of `aws help s3` which is more natural to git users • no glacier support • no s3 progress for downloads or uploads • many jarring UX issues https://github.com/aws/aws-cli/issues/305 https://github.com/aws/aws-cli/issues/30…

No CloudFront support either!

Re: The New AWS Command-Line Interface

#69

Post by kolev »

I've been using a combination of aws-cli and the Java-based tools (as some features are unsupported by aws-cli yet). I've been patiently waiting for CloudFront support - no luck yet! Also, I'm switching my new projects to use botocore, but it lacks any documentation and it's been really painful!

Re: The New AWS Command-Line Interface

#70

Post by cmccabe »

spullara wrote: post #67
cmccabe wrote: post #66

Earlier quoted context omitted.

Hadoop is a framework, not a library, so user applications need to link against the jars they need. That implies using CLASSPATH to locate them. Whether or not there is 1 jar or 100, the fact that there's no standard place to install jars in Java is a problem. Hadoop jars are hundreds of megabytes, and we have multiple daemons. Duplicating all those jars in each daemon would multiply the size of the installation many…

Most of these issues could be addressed by actually modularizing the core of Hadoop, some of which has been done in the latest code. Also, many things could be provided at runtime by the system with only the interfaces required to be in the jars that customers depend on, thus making their jars backwards compatible and more robust. BTW, let's say you didn't want to put the jars in one jar but didn't want a classpath.…

Well, I work on Hadoop. I don't know what you mean by "modularizing the core." There was an abortive attempt a few years ago to split HDFS, MapReduce, and common off into separate source code repositories. At some point it became clear that this was not going to work (I wasn't contributing to the project at the time, so I don't have more perspective than that).

Right now, we have several Maven subprojects. Maven does seem to enforce dependency ordering-- you cannot depend on HDFS code in common, for example. So it's "modular" in that sense. But you certainly never could run HDFS without the code in hadoop-common.

None of this really has much to do with CLASSPATH. Well, I guess it means that the common jars are shared between potentially many daemons. Dependencies get a lot more complicated than that, but that's just one example.

Really, the bottom line here is that there should be reasonable, sane conventions for where things are installed on the system. This is a lesson that old UNIX people knew well. There are even conventions for how to install multiple different versions of C/C++ shared libraries at the same time, and a tool for finding out what depends on what (ldd). Java's CLASSPATH mechanism itself is just a version of LD_LIBRARY_PATH, which also has a very well-justified bad reputation.

I don't know of anyone who actually uses OSGI. I think it might be one of those technologies that just kind of passed some kind of complexity singularity and imploded on itself, like CORBA. But I have no direct experience with it, so maybe that is unfair.

I like what Golang is doing with build systems and dependency management. They still lack the equivalent of shared libraries, though. Hopefully, when they do implement that feature, they'll learn from the lessons of the past.

Post reply on HN