Live data from Hacker News

'New' Python modules of 2015

blog.rtwilson.com

111–120 of 137 posts

Re: 'New' Python modules of 2015

#111
post #91

This is a tangent, but the most annoying change in the latest Python versions is you can no longer write print "foo". Now it has to be print("foo"). Damn kids ruining my language.

I was always way more annoyed having to remove/add parans when I swapped a logging, Exception, or write to a print.

It was a glaring inconsistency.

Re: 'New' Python modules of 2015

#112
post #34

Earlier quoted context omitted.

So I have been considering this. does conda track pypi or does it lag it? I have been concerned about moving over my requirements.text for a webapp with lots of dependencies

We have to use a mix of pypi and conda since quite a few of our dependencies are not in conda. We have a script which checks conda first, then falls back to pypi, all from one requirements.txt

Any chance you can share that script? I'm looking for something similar, since I too am using both conda and pypi for dependencies.

Re: 'New' Python modules of 2015

#113
post #63

Earlier quoted context omitted.

For those of us more dictionary oriented, there is https://pypi.python.org/pypi/voluptuous (which is OK for the most part, as long as you are only trying to do validation, and nothing too crazy)

Similar, I've used this library with a lot of success: https://marshmallow.readthedocs.org/en/latest/

hear hear. Steve Loria, the author, has done tremendous work this year. Thanks Steve!

Re: 'New' Python modules of 2015

#114
post #12

I am using Scrapy a lot. http://scrapy.org/ It is very well designed web crawling library.

I love that the Python community is coming up with so many robust frameworks. I have used Scrapy, and it provides most boilerplate functionality out of the box. Gone are the days I would use wget for my scraping tasks. This being said, it's a bit disappointing that Scrapy still doesn't have native support for dynamic pages. I am hoping to see this feature in the upcoming releases. With more and more of the web becoming dynamic, this should be priority feature. Other than that, I have nothing but praise for Scrapy. Pomp is apparently a simplistic take on Scraping, but it doesn't handle redirects, caching, cookies, authentication etc. I wonder if it provides parallel processing out of the box; most likely not. This is a bit strange, because these are the features for which I would prefer using a framework over writing my own code, which makes me less inclined to try Pomp. Scrapy for the win :D

Re: 'New' Python modules of 2015

#115
post #13

The Python module I learned to love this year is Click. Gets me better command line interfaces fast. URL is http://click.pocoo.org . It does progress bars too...

I'm looking for a new CLI library.

I like Click because it does very simple things very well. It gets hairy when you want to build more complex CLIs. For example, value options and validators don't play nicely because Click doesn't distinguish between the absence of a value and an invalid value, so you wind up dropping Click features and rewriting your own plumbing for that kind of stuff. The alternative is writing a more verbose CLI grammar, which leads to a really clunky UI.

Re: 'New' Python modules of 2015

#116
post #63

Hands down, my favorite new library is schema: https://pypi.python.org/pypi/schema Here's a schema I use in production, see how readable it makes the parameters of the API and how quick all the validation and normalization is: https://www.pastery.net/mhwwnv/ At the end, you get an object called data, and you can do data.title, data.language, etc, and be sure that everything is as you expect.

For those of us more dictionary oriented, there is https://pypi.python.org/pypi/voluptuous (which is OK for the most part, as long as you are only trying to do validation, and nothing too crazy)

voluptuous is simple and useful

Re: 'New' Python modules of 2015

#117
post #85

Earlier quoted context omitted.

I love it on Windows but I wonder why you use it on Linux. What does it do better than the alternative (native packages and pip)?

One word - isolation. One issue with using your system's packaging system is that a lot of system utilities are written in Python which makes it harder to play around with new versions, bleeding edge libs, etc.

Virtualenv is pretty good for isolation.

Re: 'New' Python modules of 2015

#118

tqdm looks super promising. progressbar and progressbar2 end up being complicated and weird enough to use that my company ended up making wrappers. Why maintain that when you can just use a library that works out of the box. It would be great if it had ipython notebook support. I often end up doing long operations that scrape services for data but have no idea what their progress is. For me 2015 has been the year of…

tqdm is failing for me on Windows at the moment. To be fair, it might not be its fault (I'm mixing it with Blinker signals), but still I'm slightly disappointed.

A lot of these "magic" tools fall apart when you're trying to do something slightly more structured than "throwaway bunch o' functions".

Re: 'New' Python modules of 2015

#119

Hands down, my favorite new library is schema: https://pypi.python.org/pypi/schema Here's a schema I use in production, see how readable it makes the parameters of the API and how quick all the validation and normalization is: https://www.pastery.net/mhwwnv/ At the end, you get an object called data, and you can do data.title, data.language, etc, and be sure that everything is as you expect.

> Here's a schema I use in production, see how readable it makes the parameters of the API and how quick all the validation and normalization is: https://www.pastery.net/mhwwnv/

Thank you so much for providing this example. I couldn't grok what schema did, and your code made it make sense.

If people want to provide examples for the other libraries in this thread, you'll be popular :)

Post reply on HN