Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Python Standard Library - Where Modules Go To Die

leancrew.com

Post reply on HN
Print view View original

41–50 of 75 posts

Re: The Python Standard Library - Where Modules Go To Die

#41

Post by darklajid »

heretohelp wrote: post #21
nkoren wrote: post #19

I'm not really a Python programmer, but a hacker who occasionally has cause to pick up Python scripts and do stuff with them. Perhaps I've been unlucky, but every time I've done this, it's turned into a profoundly frustrating exercise. There have always been dependencies outside the standard library, and those have had dependencies -- which more often than not, are incompatible with whatever version of Python my envi…

I think you've really just been unlucky. I've had the same experience before, but chiefly with Ruby. Interestingly, my problems with this in both Python and Ruby have evaporated once I got in the habit of using virtualenv/rb-env/rvm for my development environment. Python is about the cleanest/nicest experience I have in any language, for the record. Only language that comes close is Clojure. Leiningen is...legendary.

What is Leinigen offering over Maven? Isn't it based on the same stuff?

That said: Leiningen & npm _are_ really nice solutions for me.

Re: The Python Standard Library - Where Modules Go To Die

#42

Post by xaa »

heretohelp wrote: post #21
nkoren wrote: post #19

I'm not really a Python programmer, but a hacker who occasionally has cause to pick up Python scripts and do stuff with them. Perhaps I've been unlucky, but every time I've done this, it's turned into a profoundly frustrating exercise. There have always been dependencies outside the standard library, and those have had dependencies -- which more often than not, are incompatible with whatever version of Python my envi…

I think you've really just been unlucky. I've had the same experience before, but chiefly with Ruby. Interestingly, my problems with this in both Python and Ruby have evaporated once I got in the habit of using virtualenv/rb-env/rvm for my development environment. Python is about the cleanest/nicest experience I have in any language, for the record. Only language that comes close is Clojure. Leiningen is...legendary.

I would argue that Python's approach is FAR better than Clojure/Leiningen's "no batteries included" approach.

Suppose you want to do a very common task like parse some XML. In Clojure, the workflow is:

  1. Go to Github or Clojars, find the latest version number  of clojure.data.xml

  2. Add this version number to your project.clj

  3. Lein deps and restart the repl

  4. Re-acquire whatever REPL data you had
In Python, it's:

  1. import xml.{sax,dom,etree}
And, paradoxically, the availability of all these different versions of libraries in Clojure leads to MORE conflicts between libraries than would otherwise be the case, not less. In Python, you may not agree that, say, the "os" or "subprocess" modules are optimal -- but by golly, they're consistent.

Re: The Python Standard Library - Where Modules Go To Die

#43

Post by agentultra »

I've heard core Python developers tell people not to worry about getting a module into the stdlib. The problem being that once the module is there it won't be able to change much. APIs have the exact same problem. If you change it, you're changing other peoples' software. Tight coupling.

Is it a terrible way to write software? Maybe... but perhaps that's a different discussion.

I think the requests library is amazing. It has a much more simple API than urllib/urllib2. Does it need to replace those modules in the stdlib? I hope not!

There are only three reasons I would write a module/package that depended solely on stdlib:

  1. The module/package would be distributed primarily through package management systems.

  2. The installation of my module needs to avoid depending on anything else outside of a base python installation.

  3. The module or package will need to be supported for a long time and will likely not be updated frequently.
The first case is because you can't control what versions of third-party libraries the package manager will make available. Some might run your setuptools script while others may not. It's just easier to live with the cruft/warts of the stdlib and be sure that they'll always be there.

The second case covers a very unique situation. Modules and libraries written with this constraint are typically targeting one of two different kinds of developers. The first are the beginners who may not know about development environments and versioning. The other are experienced developers who want a minimalist script for their little one-off utility. Both should require zero dependency installation if possible.

The final case is harder to define up front. If you're writing something that you expect to run for a long time and receive little maintenance (ie: cron scripts, tools, etc) then you don't want to deal with API updates breaking your code. Fire and forget is what a long-term stable API gets you.

Re: The Python Standard Library - Where Modules Go To Die

#44

Post by EvilTerran »

It's funny, I've had the opposite problem. I was trying to write an IRC bot in Python, noted there didn't seem to be a standard library module for the IRC protocol, and so found myself looking at this:

http://pypi.python.org/pypi?%3Aaction=search&term=IRC

That's 400+ results - at least 20 of which are actually IRC protocol modules. There's no way of telling how mature each one actually is 'til you download it. It turned out the first three I tried were undocumented, buggy, incomplete, or otherwise no good.

So I gave up on PyPi and hacked it as an xchat plugin instead.

----------------

Perhaps the way forward would be styling your package repo after, say, addons.mozilla.org -- add just enough community functionality (as in ratings/reviews/"times downloaded" counters/etc) to allow the occasional gems to rise to the top of the muck. Once one solution for a given problem has been established as the best (well, most popular), that'll get more eyeballs on its internals as well, and it'll only increase its lead until it's de facto standard -- but the possibility is still there for a newcomer to dethrone it if it's genuinely better. And meanwhile, both can exist side-by-side without causing ugly compatibility issues.

Re: The Python Standard Library - Where Modules Go To Die

#45

Post by raverbashing »

jvdongen wrote: post #32
jordanb wrote: post #26

Earlier quoted context omitted.

On the contrary, the vast majority of the time, when one is using SSL, they're using it because they want encryption, rather than identification. The certificate system surrounding SSL is a complete mess. It does virtually nothing other than trigger false positives for people who who haven't paid the appropriate "security partner." The very rare person who is actually using SSL for identification rather than just to…

I now see your 'PS' (perhaps added while I was adding my comment?) - however I cannot follow the argument you make. IF you assume that MITM-attacks are rare, you probably also assume that traffic snooping is rare (which is after all a form of a MITM-attack). If that's the case, why use encrypted communication channels at all? Security is never perfect - it always is about adding layer upon layer to make the bar high…

I agree with the original poster

"IF you assume that MITM-attacks are rare, you probably also assume that traffic snooping is rare (which is after all a form of a MITM-attack)"

Well, no! You can snoop traffic without being the MITM (WiFi, local network snooping, etc). Snooping is much more easier.

As you said, security is never perfect, and 'security implementers' less so.

If the other part of the communication uses a self signed certificate (or signed by "Bob's SSL") well, I can try to convince them to change, but it will be hard.

Sure, I'll never accept a self signed key from my bank or e-commerce, but there are several other uses.

And when using APIs to connect to https you should be able to tell it to ignore the certificate, it doesn't matter, way more often than the opposite, unless you don't trust your ISP.

Re: The Python Standard Library - Where Modules Go To Die

#46

Post by mas1n »

espeed wrote: post #29
nkoren wrote: post #19

I'm not really a Python programmer, but a hacker who occasionally has cause to pick up Python scripts and do stuff with them. Perhaps I've been unlucky, but every time I've done this, it's turned into a profoundly frustrating exercise. There have always been dependencies outside the standard library, and those have had dependencies -- which more often than not, are incompatible with whatever version of Python my envi…

There have always been dependencies outside the standard library, and those have had dependencies -- which more often than not, are incompatible with whatever version of Python my environment is set up for. Create a virtual environment for the Python version you want to use, and install the package with pip (pip will install all the dependencies in your project's local environment)... $ mkdir myproj $ cd myproj $ vir…

Or you can install virtualenvwrapper (http://www.doughellmann.com/projects/virtualenvwrapper/) and after a little configuration...

  $ mkvirtualenv env1
  (env1)$ pip install somepackage
Creating self containing python projects has never been so easy!

Re: The Python Standard Library - Where Modules Go To Die

#47

Post by leephillips »

jbri wrote: post #33
jordanb wrote: post #26

Earlier quoted context omitted.

On the contrary, the vast majority of the time, when one is using SSL, they're using it because they want encryption, rather than identification. The certificate system surrounding SSL is a complete mess. It does virtually nothing other than trigger false positives for people who who haven't paid the appropriate "security partner." The very rare person who is actually using SSL for identification rather than just to…

If MitM attacks are so rare, why bother encrypting your traffic in the first place? Packet-snooping attacks are also "extremely rare" by most metrics, so why protect against one but not the other? Either go all the way on security, or be obvious about not having any. Appearing secure when in actuality you're not is the worst option.

'Packet-snooping attacks are also "extremely rare"'

I think they're pretty common, even for fun and recreation (http://codebutler.com/firesheep). I know I could start reading people's emails in Starbucks with what's on my laptop now and the knowledge in my head, but if I wanted to mount a MITM attack I would need to do some research.

Re: The Python Standard Library - Where Modules Go To Die

#48

Post by jvdongen »

raverbashing wrote: post #45
jvdongen wrote: post #32

Earlier quoted context omitted.

I now see your 'PS' (perhaps added while I was adding my comment?) - however I cannot follow the argument you make. IF you assume that MITM-attacks are rare, you probably also assume that traffic snooping is rare (which is after all a form of a MITM-attack). If that's the case, why use encrypted communication channels at all? Security is never perfect - it always is about adding layer upon layer to make the bar high…

I agree with the original poster "IF you assume that MITM-attacks are rare, you probably also assume that traffic snooping is rare (which is after all a form of a MITM-attack)" Well, no! You can snoop traffic without being the MITM (WiFi, local network snooping, etc). Snooping is much more easier. As you said, security is never perfect, and 'security implementers' less so. If the other part of the communication uses…

If you are in the position to passively sniff the traffic, you're almost always also in the position to redirect and modify the traffic.

The only thing you're protected against with encryption sans authentication is passive sniffing. If that's all you care about, fine, do realize however how limited the protection is you gain.

Re: The Python Standard Library - Where Modules Go To Die

#49

Post by leephillips »

heretohelp wrote: post #21
nkoren wrote: post #19

I'm not really a Python programmer, but a hacker who occasionally has cause to pick up Python scripts and do stuff with them. Perhaps I've been unlucky, but every time I've done this, it's turned into a profoundly frustrating exercise. There have always been dependencies outside the standard library, and those have had dependencies -- which more often than not, are incompatible with whatever version of Python my envi…

I think you've really just been unlucky. I've had the same experience before, but chiefly with Ruby. Interestingly, my problems with this in both Python and Ruby have evaporated once I got in the habit of using virtualenv/rb-env/rvm for my development environment. Python is about the cleanest/nicest experience I have in any language, for the record. Only language that comes close is Clojure. Leiningen is...legendary.

I've had the same experience before, but chiefly with Ruby.

Tell me about it: http://lee-phillips.org/badruby/

Re: The Python Standard Library - Where Modules Go To Die

#50

Post by jmoiron »

anthonyb wrote: post #39
zeeg wrote: post #16

Earlier quoted context omitted.

Well it would work just fine, but you'd always end up requiring the external dependency even if you didn't need to. For example, let's say there was a new urllib released (its still called urllib). It's now version 2.0, but the stdlib version is 1.0. If your package said "I need urllib==1.0", it would have know way of understanding that the version was already included within the standard library. That said, it would…

> it would have know way of understanding that the version was already included within the standard library Other than by introspecting which packages are installed, that is. Most of them will have a VERSION, __version__ or _version attribute which tells you.

This is available via the pkg-info in all installed packages; iirc version is required for distutils, and there is pep386 for version number format, so it should be possible to determine version number as well as compare them for all well-behaved packages. There is even a package which will find and parse pkg-info for an installed package called pkginfo:

http://pypi.python.org/pypi/pkginfo

Post reply on HN