Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Python's batteries are leaking”

pyfound.blogspot.com

Post reply on HN
Print view View original

341–350 of 420 posts

Re: “Python's batteries are leaking”

#341

Post by RayDonnelly »

loeg wrote: post #278
bloaf wrote: post #248

Earlier quoted context omitted.

Where I work, there is currently a push to get python on the computers that manage physical equipment operation. These computers are not allowed to connect to the internet, and have extremely limited connectivity to the rest of the business network. Installing anything new on them requires risk assessments like you wouldn't believe, since the consequences of malicious code could easily hit 10s of millions of dollars…

If your risk assessment says that the exact same tkinter outside of Python stdlib is riskier than in Python stdlib, maybe your risk evaluation process needs reevaluation.

I disagree. tcl/tk is written in C and C can be compiled in very very badly indeed (from a security perspective).

Re: “Python's batteries are leaking”

#342

Post by gray_-_wolf »

sph wrote: post #283
sametmax wrote: post #282

Hawk Owl is a _fantastic_ dev. She was the main force behind the twisted 2->3 transition. But because she is, she is missing the point of batteries included. Asyncio is in the stdlib so that we have an official lib and API. The main benefit is that most people now, when looking for async, are not wondering about twisted or gevent or tornado. Most just go asyncio. Most dev efforts go to asyncio. It's the end of the gr…

Having a huge standard library also kills analysis paralysis and lets people be more productive. If you're in the flow and trying to hack together something, the last thing you need is to lose all momentum to pick a date time library. I've had this issue tons of times with Node and Rust, where I'm not up to date with the current meta and my 30 minute hack job is interrupted 5 minutes in by having to google which libr…

> I've been trying to find another scripting language to replace Python because I'm not a fan of it anymore (I won't get into it right now), and considering what I just wrote, there's not much that can replace it

Have you considered ruby?

Re: “Python's batteries are leaking”

#343

Post by SlowRobotAhead »

sametmax wrote: post #282

Hawk Owl is a _fantastic_ dev. She was the main force behind the twisted 2->3 transition. But because she is, she is missing the point of batteries included. Asyncio is in the stdlib so that we have an official lib and API. The main benefit is that most people now, when looking for async, are not wondering about twisted or gevent or tornado. Most just go asyncio. Most dev efforts go to asyncio. It's the end of the gr…

>She was the main force behind the twisted 2->3 transition.

Explain? I’ve hated the slow adoption of 3.x from 2.x, and generally how terrible it is to have apps that are 2.x on your 3.x system, and would like to know more about how that happened.

Re: “Python's batteries are leaking”

#344

Post by sebazzz »

postgeographic wrote: post #314
hn_throwaway_99 wrote: post #211

Earlier quoted context omitted.

> If I fight the battle to get centralized IT to install Python, I now have a guaranteed set of standard libraries as well. I'm never going to get permission to install anything other than default. Ever. Can you explain this more? What kind of place do you work? I've had some experience with large, bureaucratic companies, but nothing ever so far as "you can't install any other libraries."

Not who you asked, but I work for a large international company, a big 4 professional services firm. I wanted to install anaconda and Jupyter on my machine (data science is not a part of my 'official' job description, but I wanted to see how much of my data exploration workflow I could speed up or automate). I had to go up three separate hierarchy ladders to get sign off. First, my own team, then IT, then our risk an…

I also work at a big 4, and perhaps the same one as you, and I assure you, there is a way. There is always a procedure or some way you can follow. You just need to know how to look it up.

We have our own development team, our own servers, our own freedom to deliver to clients fast without the hassle of the main corporation. How? We talked to the right persons.

Re: “Python's batteries are leaking”

#345

Post by scottLobster »

hn_throwaway_99 wrote: post #211
bsder wrote: post #207

Earlier quoted context omitted.

Python cannot be atomized effectively, and the issue is political. The problem is that I cannot count on being able to install new software in many environments. If I fight the battle to get centralized IT to install Python, I now have a guaranteed set of standard libraries as well. I'm never going to get permission to install anything other than default. Ever. Consequently, the standard libraries need to be very com…

> If I fight the battle to get centralized IT to install Python, I now have a guaranteed set of standard libraries as well. I'm never going to get permission to install anything other than default. Ever. Can you explain this more? What kind of place do you work? I've had some experience with large, bureaucratic companies, but nothing ever so far as "you can't install any other libraries."

I work for a major defense contractor, and while we have vanilla python we are categorically not allowed to download software off the internet and install it without authorization, even on the unclassified internet-connected corporate network.

Theoretically there's a process for requesting new software and getting it approved, but actually pushing it through requires getting one of my program architects to care enough to file the request (As a mere level 2 engineer all I can do is write it, can't submit), then potentially weeks of followup, for 1 specific version of 1 specific package.

In the case of python packages, perl and the perl packages we need are already approved because a few senior devs got together and pushed them through 10 years ago (was before my time, but I understand it was with quite a bit of arm twisting). It's more time-efficient to just code perl than to fight for python.

It's one of the many reasons I intend to get myself another job for Christmas. :)

As for why the system exists: Cost cutting, in the sense of "the less we invest in infrastructure the more we can divert to sexy hardware for the cameras and shareholder dividends. So long as it's theoretically possible for you to do your work, we don't care how many hoops you have to jump through to do it. And our competition is even worse than us, so we don't have to worry about anyone undercutting."

As a result all our infrastructure is centralized. Programs have to jockey with each other for everything from virtual servers to physical workstations and monitors. Hell the only reason my program has our primary test server is because one of our architects literally overheard a hallway conversation about a program that was spinning down and getting rid of some servers, so he jumped on it.

Re: “Python's batteries are leaking”

#346

Post by 0xbadcafebee »

j88439h84 wrote: post #195
0xbadcafebee wrote: post #41

It's funny to me that they're making a point that PyPI is better than core, because actually I think PyPI has created a rather crap ecosystem. The non-hierarchial organization of packages, the lack of curation, lack of inheriting past functionality and extending it as more standard functionality, etc has resulted in a confusing sprawl of packages with duplicate, incompatible, buggy functionality. It's a bit like Linu…

What functionality would you like to have on PyPI, in addition to curation?

The big "function" I would like is just organizing the packages differently to get people to think about and use them differently.

Search engines are a "cool" technology that have become the de facto way to find what you're looking for. But if there's a lot of content related to what you're looking for, they can suck.

Go to PyPI and search for "semantic version". 10,000+ projects for "semantic version" found. As you go through page after page of different modules related to versioning, the one module you won't find immediately is Versio (https://pypi.org/project/Versio/), a well-documented and useful module which I ended up using. I have no idea how I found this module, but it certainly wasn't from PyPI's search engine.

Now go to CPAN (really metacpan) and search for "semantic version". Yes, you're still looking at thousands of results - but wait! There are only two modules here that look useful: Version::Dotted::Semantic, and SemVer. And the description comes straight from the docs' README, rather than being a short uninformative blurb. The first module, Version::Dotted::Semantic, is inheriting a separate module, Version::Dotted, and adding some extra functionality. Not only does the search page give more information about the module, but the hierarchy makes it easier to find (and later extend) useful modules in an intuitive way. Since the base module's functionality is boring, generic, and simple, it's less likely that people will make 20 different versions of it, so it'll be reused more often and thus remain stable for a long time.

A lot of CPAN's module names have sprawled over time and gotten less useful, but there's still a general convention that you name your module as a hierarchy of what it does (even if it's kind of verbose) and make small, reusable modules, rather than giant modules that are hard to extend. Not all modules measure up to this standard, and there's definitely room to improve, but I think Python modules could benefit greatly from a system like this.

As far as curation goes, PyPI is often filled with cruft. While searching for Jenkins packages, you will come across lots of entries like this: https://pypi.org/project/jenkins2api/. The homepage leads to a GitHub 404, it's only ever had one release, and it has no documentation. This project should probably not have been listed on the main search page, or at least sorted well down the list by default with intelligent filters and marked accordingly. (The "date last updated" and "trending" sorting just results in having virtually no Jenkins-related modules in the results at all)

Re: “Python's batteries are leaking”

#347

Post by rogerbinns »

tachyonbeam wrote: post #166
rogerbinns wrote: post #133

Earlier quoted context omitted.

I think the biggest problem is going from zero third-party dependencies to one and more. Adding that very first one is a huge pain since there are many ways of doing it with many different trade offs. It is also time consuming and tedious. The various tools like you mention are best at adding even more dependencies, but are hurdles for the very first one.

Not true. The more external dependencies you add, the more likely it is that one of them will break. I try to have as few external dependencies as possible, and to pick dependencies that are robust and reliably maintained. There is so much Python code on GitHub that is just broken out of the box. When people try your software and it fails to install because your nth dependency is broken or won't build on their system…

To add one more third party dependency when you already have some is as simple as adding one more to whatever solution you are already using (eg another line in requirements or running a command).

When you have no third-party dependencies, then adding the first one requires picking amongst trade offs and lots of work. A subset of choices include using virtualenv, using pip, using higher layer tools, copying the code to the project, using a Python distribution that includes them, writing code to avoid needing the first dependency ...

* You have to document to humans and to the computer which of the approaches is being used

* Compiled extensions are a pain

* You have to consider multiple platforms and operating systems

* You have to consider Python version compatibility (eg third party could support fewer Python versions than the current code base)

* And the version compatibility of the tools used to reference the dependency

* And a way of checking license compatibility

* The dependency may use different test, doc, type checking etc tools so they may have to be added to the project workflow too

* Its makes it harder for collaborators since there is more complexity than "install Python and you are done"

I stand by my claim that the first paragraph (adding another dependency) is way less work, than the rest which is adding the very first one.

Re: “Python's batteries are leaking”

#348

Post by vlovich123 »

lalaland1125 wrote: post #274
tachyonbeam wrote: post #241

Earlier quoted context omitted.

It's trickier than just pinning dependencies because some libraries also need to build C code, etc. Once you bring in external build tools, you have that many more potential points of failure. It's great. Also, what happens if your dependencies don't pin their dependencies? Possibly, uploading a package to pipy should require freezing dependencies or do it automatically.

Modern python tooling like pipenv pins the dependencies of your dependencies as well. This is no longer an issue

I used a requirements.in file to list out all the top-level direct dependencies & then used pip-compile from piptools to convert that into a frozen list of versioned dependencies. pip-compile is also nice because it doesn't upgrade unless explicitly asked to which makes collaboration really nice. I then used the requirements.txt & various supporting tooling to auto-create & keep updated a virtualenv (so that my peers didn't need to care about python details & just running the tool was reliable on any machine). It was super nice but there's no existing tooling out there to do anything like that & it took about a year or two to get the tooling into a nice place. It's surprisingly hard to create Python scripts that work reliably out-of-the-box on everyone's environments without the user having to do something (which always means in my experience that something doesn't work right). C modules were more problematic (needing Xcode installation on OSX, potentially precompiled external libraries not available via pip but also not installed by default), but I created additional scripts to help bring a new developer's machine to a "good state" to take manual config out of the equation. That works in a managed environment where "clean" machines all share the same known starting state + configs - I don't know how you'd tackle this problem in the wild.

I do think there's a lot of low-hanging fruit where Python could bake something in to auto-setup a virtualenv for a script entrypoint & have the developer just list the top-level dependencies & have the frozen dependency list also version controlled (+ if the virtualenv & frozen version-controlled dependency list disgaree rebuild virtualenv).

Re: “Python's batteries are leaking”

#349

Post by vpzom »

petre wrote: post #257
Aeolun wrote: post #222

Earlier quoted context omitted.

Node is ostensibly batteries included, but they’re weird, arcane batteries powered by tears. Therefore, to get anything done easily usually requires an external package.

Node doesn't even have a stdlib. You have to npm or yarn anything. And yes most of the time it will end in tears. That's not my definition of batteries included, but rather some batteries might explode, others might only ignite, we wish you the best of luck.

What's a stdlib then, if Node's isn't?

Re: “Python's batteries are leaking”

#350

Post by mkesper »

pornel wrote: post #323
dragonwriter wrote: post #106

Note that similar issues were raised with Ruby stdlib, which is being addressed in part with “Gemification” of stdlib, so that all of stdlib (targeted for 3.0, though it's been going on since 2.4)[0] is being moved out to externally-updatable packages that are included by default (default and bundled gems), so that it is still “batteries included” but the batteries are at least replaceable. Amber's suggestion seems t…

This is close to what Rust is doing, and it's working pretty well, apart from shocking newcomers who expect libstd to be useful on its own. In Rust, libstd is mainly for interfacing with the compiler and providing interoperability between packages (crates). The wider crate ecosystem is the real standard library, since external crates are as easy to use as the standard library. For example, the libstd doesn't even hav…

But then you have six versions to have security fixes, how do you do that?
Post reply on HN