Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pypi.org is running a survey on the state of Python packaging

pypi.org

Post reply on HN
Print view View original

161–170 of 193 posts

Re: Pypi.org is running a survey on the state of Python packaging

#161

Post by BerislavLopac »

bno1 wrote: post #105
IceHegel wrote: post #101

If I see some JS, Go, or Rust code online I know I can probably get it running on my machine in less than 5 min. Most of the time, it's a ‘git clone’ and a 'yarn' | 'go install' | 'cargo run', and it just works. With python, it feels like half the time I don't even have the right version of python installed, or it’s somehow not on the right path. And once I actually get to installing dependencies, there are often ver…

I wish pip had some package deduplication implemented. Even some basic local environments have >100MB of dependencies. ML environments go into the gigabytes range from what I remember.

I'm not sure what you mean - unlike npm, pip installs only one copy of a dependency in each environment.

Re: Pypi.org is running a survey on the state of Python packaging

#162

Post by BerislavLopac »

tibbon wrote: post #135
thrown_22 wrote: post #125

Earlier quoted context omitted.

Installing Python only applications is trivial. What you're complaining about is all the missing code in other languages which isn't controlled by Python and depends on the OS to provide. This is why we created Linux distributions in the first place. It is not the place of every language to reinvent the wheel - poorly.

Isn’t wheel one of pythons many packaging schemes (intended to replace eggs?)? > There should be one– and preferably only one –obvious way to do it. Oh no…

There are only two: wheel, which includes binaries, and sdist, which is Python-only. Eggs have been deprecated for nearly 10 years now.

> Oh no…

That sentence doesn't mean what you think you means. It is saying that there should be "only one way to do something".

Re: Pypi.org is running a survey on the state of Python packaging

#163

Post by electromech »

woodruffw wrote: post #24
MonkeyMalarky wrote: post #16

Earlier quoted context omitted.

Seems more oriented to (potential) contributors than end users of the packaging system. Who cares about mission statements and inclusivity, secure funding and pay developers to make the tools.

> Who cares about mission statements and inclusivity, secure funding and pay developers to make the tools. These are connected things. I maintain a PyPA member project (and contribute to many others), and the latter is aided by the former: the mission statement keeps the community organized around shared goals (such as standardizing Python's packaging tooling), and inclusivity insures a healthy and steady flow of new…

> I maintain a PyPA member project (and contribute to many others)

THANK YOU!

> keeps the community organized around shared goals (such as standardizing Python's packaging tooling)

Personally I felt some disconnect between "package manager for all" and the need for "standardizing Python's packaging tooling." Yes, communities should be welcoming and friendly to everyone, AND the community should have clear expectations for best practices that members should follow. E.g., is an experienced female developer more likely to give up on contributing because she couldn't find a local meetup, or because she didn't know whether to create pyproject.toml vs requirements.txt? In some sense, the bigger and more diverse the community, the greater the need for a clear, solid foundation. IDK if that's remotely clear; it's just a feeling I had going through some of those questions.

Re: Pypi.org is running a survey on the state of Python packaging

#164

Post by cozzyd »

orf wrote: post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

Many "Python" packages include native code in some form either as bindings or to workaround Python being agonizingly slow. Which means you often need to call make, or cmake or some other build system anyway... unless you want to build wheels for every possible configuration a user might have (which is virtually impossible, considering every combination of OS, architecture, debug options, etc. you may want to support). Plus you need a build system to build the wheels anyway...

Re: Pypi.org is running a survey on the state of Python packaging

#165

Post by dalke »

Spiritus wrote: post #158
orf wrote: post #81

Earlier quoted context omitted.

To be clear, I’m not suggesting we remove the ability to compile native extensions. I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py. It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or opti…

Well we're almost there I think. You can define dependencies and other metadata in pyproject.toml nowadays: https://setuptools.pypa.io/en/latest/userguide/pyproject_con...

How do I specify that I need gfortran installed?

Re: Pypi.org is running a survey on the state of Python packaging

#166

Post by dalke »

pabs3 wrote: post #149
dalke wrote: post #146

Earlier quoted context omitted.

I do that, using manual CPUID tests, along with allowing environment variables to override the default path choices. But if the compiler by default doesn't enable AVX2 then it will fail to compile the AVX2 intrinsics unless I add -mavx2. Even worse was ~10 years ago when I had an SSSE3 code path, with one file using SSSE3 intrinsics. I had to compile only that file for SSSE3, and not the rest of the package, as other…

See the wiki page, the function multi-versioning stuff means you can use AVX2 in select functions without adding -mavx2. And using SIMD Everywhere you can automatically port that to ARM NEON, POWER AltiVec etc.

EDIT: after I wrote the below I realize I could use automatic multi-versioning solely to configure the individual functions, along with with a stub function indicating "was compiled for this arch?" I think that might be more effective should I need to revisit how I support multiple processor architecture dispatch. I will still need the code generation step.

Automatic multi-versioning doesn't handled what I needed, at least not when I started.

I needed a fast way to compute the popcount.

10 years ago, before most machines supported POPCNT, I implemented a variety of popcount algorithms (see https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... ) and found that the fastest version depended on more that just the CPU instruction set.

I ended up running some timings during startup to figure out the fastest version appropriate to the given hardware, with the option to override it (via environment variables) for things like benchmark comparisons. I used it to generate that table I linked to.

Function multi-versioning - which I only learned about a few month ago - isn't meant to handle that flexibility. To my understanding.

I still have one code path which uses __popcountll built-in intrinsics and another which has inline POPCNT assembly, so I can identify when it's no longer useful to have the inline assembly.

(Though I used AVX2 if available, I've also read that some of the AMD processors have several POPCNT execution ports, so may be faster than using AVX2 for my 1024-bit popcount case. I have the run-time option to choose which to use, if I ever have access to those processors.)

Furthermore, my code generation has one path for single-threaded use and one code path for OpenMP, because I found single-threaded-using-OpenMP was slower than single-threaded-without-OpenMP and it would crash on multithreaded macOS programs, due to conflicts between gcc's OpenMP implementation and Apple's POSIX threads implementation.

The AVX2 popcount is from Muła, Kurz, and Lemire, https://academic.oup.com/comjnl/article-abstract/61/1/111/38... , with manually added prefetch instructions (implemented by Kurz). It does not appear that SIMD Everywhere is the right route for me.

Re: Pypi.org is running a survey on the state of Python packaging

#167

Post by pabs3 »

dalke wrote: post #166
pabs3 wrote: post #149

Earlier quoted context omitted.

See the wiki page, the function multi-versioning stuff means you can use AVX2 in select functions without adding -mavx2. And using SIMD Everywhere you can automatically port that to ARM NEON, POWER AltiVec etc.

EDIT: after I wrote the below I realize I could use automatic multi-versioning solely to configure the individual functions, along with with a stub function indicating "was compiled for this arch?" I think that might be more effective should I need to revisit how I support multiple processor architecture dispatch. I will still need the code generation step. Automatic multi-versioning doesn't handled what I needed, at…

If you implement your own ifunc instead of using the compiler-supplied FMV ifunc, you could do your benchmarks from your custom ifunc that runs before the program main() and choose the fastest function pointer that way. I don't think FMV can currently do that automatically, theoretically it could but that would require on additional modifications to GCC/LLVM. From the sounds of it, running an ifunc might be too early for you though, if you have to init OpenMP or something non-stateless before benchmarking.

SIMD Everywhere is for a totally different situation; if you want to automatically port your AVX2 code to ARM NEON/etc without having to manually rewrite the AVX2 intrinsics to ARM ones.

Re: Pypi.org is running a survey on the state of Python packaging

#168

Post by duckmysick »

BerislavLopac wrote: post #161
bno1 wrote: post #105

Earlier quoted context omitted.

I wish pip had some package deduplication implemented. Even some basic local environments have >100MB of dependencies. ML environments go into the gigabytes range from what I remember.

I'm not sure what you mean - unlike npm, pip installs only one copy of a dependency in each environment.

So if I have environments A, B, and C and each has a dependency ML-package-1.2 that's 100 MB, it means there's one copy of it in each environment? Meaning 3x100 MB?

Re: Pypi.org is running a survey on the state of Python packaging

#169

Post by dagw »

duckmysick wrote: post #168
BerislavLopac wrote: post #161

Earlier quoted context omitted.

I'm not sure what you mean - unlike npm, pip installs only one copy of a dependency in each environment.

So if I have environments A, B, and C and each has a dependency ML-package-1.2 that's 100 MB, it means there's one copy of it in each environment? Meaning 3x100 MB?

Meaning 3x100 MB?

By default, yes. It is possible to install ML-package-1.2 into your 'base' python and then have virtual environments A,B and C all use that instead of installing their own copies. However this is generally not considered best practice.

Re: Pypi.org is running a survey on the state of Python packaging

#170

Post by Spiritus »

dalke wrote: post #165
Spiritus wrote: post #158

Earlier quoted context omitted.

Well we're almost there I think. You can define dependencies and other metadata in pyproject.toml nowadays: https://setuptools.pypa.io/en/latest/userguide/pyproject_con...

How do I specify that I need gfortran installed?

You can't. But is that possible with any programming language specific package manager? How would that even work given that every flavour of OS/distro have their own way of providing gfortran?
Post reply on HN